The famous Monte Hall Problem poses the following question: “Suppose you’re on a game show, and you’re given the choice of three doors. Behind one door is a car, behind the others, goats. You pick a door, say #1, and the host, who knows what’s behind the doors, opens another door, say #3, which has a goat. He says to you, ‘Do you want to pick door #2?’ Is it to your advantage to switch your choice of doors?”
While the intuitive answers seems to be “no,” as one might argue that the two remaining doors are equally likely to contain the car, the correct answer is actually “yes.” As vos Savant points out later on in the above link, the probability of winning if you switch is actually 2/3.
For instance, if we run the simulation 1,000 times, we see a fair amount of volatility in the results over the first 250 and even 500 trials. As we add more trials, however, the average of the results begin to converge to the true expected values: 2/3 chance of winning if we switch doors, and 1/3 chance if we don’t.
The results are even more concrete if we consider 10,000 trials:
As some of you may recall, I kicked off the Fun with Excel series with a post on attraction, where I hoped to explore the mechanics of physical attraction from a statistical perspective. Due to the amount of feedback I have received (positive, skeptical, or otherwise), I have decided to write a follow-up post.
This couple is happy, but are you?
In the first part of The Laws of Attraction, I focused solely on physical attraction, and the impact of bias in our perception of attractiveness on seeking a compatible partner. In part two, I focus on the bigger picture: given a set of personal traits, what is the probability that you will find someone with those traits at the specific level that you desire?
Background: In the song One In A Million, Ne-Yo sings about a girl who he calls “one in a million.” Of course, not content with just enjoying the music, I wondered to myself what it actually meant to be “one in a million.” One way of measuring this is by breaking down attraction into a larger set of personality traits and trying quantify our desires, which is essentially what online dating services do with their “matching formulas.” For purposes of our exercise, let’s say you have a list of 10 distinct characteristics that you believe to be important and that you actively look for when searching for a partner. You might be more picky on some traits than others, but it isn’t too hard to quantify your objectives. Similar to my previous project, I quantify these objectives in terms of percentile, which, at least from a guy’s perspective, is pretty straightforward. For example, I might say, “I’m only interested in a girl who’s in the 80% percentile for Trait 1, 90th percentile for Trait 2, 50th percentile for Trait 3…” and so on and so forth. Now, the question is “what are the chances that such a girl exists?” A closely related question is “how many such girls are out there?”, followed by the not-so-fun reality-check of “what are the odds that I’ll actually find such a girl?”
The Model: While we won’t tackle the last question in this post, the first two are pretty straightforward to simulate from a mathematical standpoint. For each trait, the probability of finding someone who is at the X-percentile or higher of that trait is (100-X)%. For multiple traits, all we have to do is multiply these probabilities together, but the key assumption here is that all the traits are independent. Obviously, this isn’t true in real life, but we’ll revisit this point in a little while.
Assuming we start with a set of 10 traits, I will define a person having N “Perfect” Traits as someone who ranks at the 90th percentile or higher in N traits, and at the 50th percentile or higher in the remaining 10 minus N traits. Thus, assuming a world population of 7.12 billion, a male/female split of 50/50, and that you are heterosexual, the number of potential partners with 0 “Perfect” Traits walking on the planet is 3,476,563, or 1 in 1,024 (the mathematically inclined should immediately realize that 1,024 = 2^10). On the other hand of the spectrum, there are theoretically only 18 people with 9 “Perfect” Traits, or 1 in 200 million. Note that a person with 10 “Perfect” Traits technically doesn’t exist, as probability indicates a 1 in 10 billion chance. At this point, the astute reader will note one possible answer to Ne-Yo’s earlier problem: if you consider a smaller set of 6 traits rather than 10, a “one in a million” girl would simply be a girl who has 6 “Perfect” Traits (all 6 traits at the 90th percentile or higher) in that scenario.
The Results: I plotted the entire spectrum of N “Perfect” Traits in the scenario of 10 traits, to arrive at the following graph:
It should be no surprise that our graph strongly resembles a normal curve, as we are working with a binomial distribution.
I suppose the lesson here is that it doesn’t pay to be picky, but recall the very important (and incorrect) assumption we made earlier that all traits occur independently of one another. In the real world, however, this couldn’t be further from the truth. Creativity may be correlated with Curiosity, Honor may be correlated with Kindness, and Intelligence may be highly correlated with (or the cause of) all the other traits. Accounting for the dependencies between and among all 10 traits would require us to estimate both marginal and conditional probabilities, which would not only be difficult, but also complicate our model very quickly. Statistical mumbo jumbo aside, what this means is that the probabilities estimated by a simple binomial model are far too conservative (too low). This should be great news for all the picky daters out there.
An alternative way of tackling the dependent traits problem is to simply consider a smaller set of traits. For example, if we created a list of 10 traits, and then realized that two of them were very highly correlated with each other, then we could eliminate one of them and simply consider a 9 trait model, which in turn would be a more accurate simulation of what the actual probabilities might look like in real life. To that point, I also plotted out graphs for scenarios involving 7 traits and 5 traits:
Note that as we decrease the number of traits, the number of potential candidates increases exponentially. So if you only considered 5 main traits, and furthermore were only picky about 3 of them (3 “Perfect” Traits in the graph above), then you would only be looking at a probability of 1 in 400. Not bad.
Great Success!
Conclusion
At the end of the day, it is perhaps silly to attempt to model real life human dynamics with 50 lines in Excel. But that would also be missing the point of the exercise. Thinking about real world problems from a different perspective (whether it is psychologically, statistically, or otherwise) can shed new light on the issue, or simply affirm something we already knew or suspected. Even if it is only the latter, there is still value derived from being able to connect the dots between a variety of different frameworks.
My father was a big fan of the Chicago Bulls back in the ’80s and ’90s, so I had the good fortune of watching some of the best playoff basketball (i.e. Michael Jordan) that the NBA (and the world) has ever witnessed. Perhaps that is the same reason why the last decade or so of NBA basketball has seemed to pale noticeably in terms of excitement. It is generally agreed upon among basketball fans that the game as it is played today is (a lot) less physical (and perhaps less exciting) than it once was.
Officiating has also seemingly become a bigger determinant of results, and like virtually all professional team sports, the blame often lands on the referees. “If it weren’t for that call, they would have won the game,” is a phrase we hear all too often, and one that I am guilty of committing as well. However, have changes in officiating really been that significant over the last few decades, and if so, how would we measure such a phenomenon? The answer, of course, lies in the numbers.
Luckily, statistics for the NBA are readily available, but for the purposes of my project, I decided to look at playoff statistics from the 1983-84 season to the latest 2012-13 season. However, even if the data is easily accessible, oftentimes the most time-consuming aspect of a project is collecting the data and organizing it in a way that makes it easy to analyze. This was no exception. Luckily, with a little vlookup and text parsing (the latter is needlessly complex in Excel) magic, I was able to largely automate the process of converting 30 years of raw playoff data into something I could process more easily.
My first goal was to see if there were any high level trends in the NBA playoffs through time, in particular the number of games played and the point differential in each game. Moreover, I wanted to analyze these metrics by playoff round (e.g. first round (1R), conference semifinals (2R), conference finals (3R), and finals (4R)). If we were to believe that officiating actually had a measurable impact on playoff results, we may expect to find the following:
An overall longer playoff campaign
Smaller average point differentials, to convey the appearance of “closer” games
I’ll make a few observations, but the data really speaks for itself here. In the first chart, we see that after adjusting for the NBA’s change in playoff format since the 2002-03 campaign, both the average numbers of games played in the playoffs and the average number of games played per round has not shown any noticeable shift through time. The average points differential chart shows the same story, and in fact both charts seem to suggest some cyclical trends through time. Lastly, the average free throw attempts and fouls chart actually displays a noticeable decrease through time on a per game adjusted basis. Perhaps this is a testament to just how physical the game was back in the 1980s and 90s, which MJ himself has suggested on many an occasion.
Conclusion
The data doesn’t seem to indicate any obvious playoff trends that may have been caused by officiating. However, more granular foul data (which may not be available) may help clarify the story. In particular, even if the average number of fouls per game has trended down over the last 30 years, have the types of fouls called changed in any significant way? Perhaps more calls are coming during particularly tight stretches of games, or conversely, during blow outs, to ensure that the losing team is “still in it.” Of course, all of this is pure speculation, and without hard evidence, it is difficult to move forward. As Sir Arthur Conan Doyle once said through his most famous character Sherlock Holmes, “It is a capital mistake to theorize before one has data. Insensibly one begins to twist facts to suit theories, instead of theories to suit facts.” Until such facts are found, our theories will remain theories.
In this week’s (long overdue) Fun with Excel, I explore a social phenomenon mostly commonly known as the ridiculous long CVS receipt.
Full Disclosure: I’m a frequent shopper at CVS, and a long-time ExtraCare Card holder. I’ve been generally impressed by the company’s wide variety of product offerings (cheap generic drugs, am I right?) and convenient self-checkout kiosks. But the one thing that has left me scratching my head is the amazingly long receipt that the company seems to insist on pushing onto its customers. Case in point: I went to CVS last week to pick up some orange juice and soap, and ended up with a 3.5-foot-long receipt.
There are a few things that always cross my mind whenever I pick up my receipts from CVS:
The obvious: Why the hell is it so damn long?
Are any of the coupon on this thing actually useful? Sadly, the answer to this is no. Aside from the occasional extrabucks (woohoo!), most of the “coupons” on these long receipts sound something like this: “10% off any Beauty purchase & Join Beauty Club with this coupon!” (What is the CVS Beauty Club, and why would I join?), “$2 off any Vitamin purchase of $8 or more!” (Sorry, I don’t take vitamins), “$2.50 off $12 Shampoo, Conditioner, Hair Treatment or Styling” (That’s some really expensive hair product for a dude…), or “$2 off any Sunless Tanner or Sun Protection purchase $10 or more” (Again, I don’t tan, and I’m pretty sure the people who do wouldn’t get their stuff from CVS). What’s more frustrating is that these coupons almost always expire within 1 or 2 weeks, which makes redeeming them even less practical.
If long receipts tend to be nothing more than a big waste of paper, then why does CVS insist on continuing to print them? Well, here’s where a little analysis might come in.
With some help from CVS’s latest 10-K and a few articles on the internet, I was able to gather some basic information on the company’s financials and store information. From personal experience, I have only gotten long receipts when using the self-checkout kiosks, so I made the number of kiosk-equipped stores a key driver in my analysis. CVS currently has about 7,500 stores, and in the base case, I assumed that 1,000 of these had kiosks (according to this article, the company expected 420 such stores by the end of 2011, so this estimate may already be aggressive). I further assumed that on average each kiosk-equipped store had four kiosks, that the average number of unique visitors per store was 25 per hour, the average number of operational hours per day was 12, and that 75% of all visitors used the kiosk. On the cost side, I used this product off Amazon as a benchmark, assuming that CVS would achieve a 25% cost savings due to its ability to purchase wholesale. From there, I calculated the annual total cost of receipt paper in the base case (assuming 3.5′ to be the average), as well as the total cost of receipt paper assuming a more reasonable receipt length (6 inches).
The results lead to a clear answer to the question posed in #3 above. The bottom line is that the cost savings from using less receipt paper are likely outweighed by the potential increase in sales revenue (due to customers either redeeming their coupons, or returning to CVS due to the perception that they are getting a good deal). You can take a look at the simple spreadsheet I put together here. As usual, please be sure to provide proper attribution where appropriate if you plan to re-use it!
To put things in perspective, CVS had sales of $123 billion in 2012, and operating expenses of roughly $15 billion. In our base case, the total cost of receipts comes out to a measly $1.14 million, versus a “potential” cost of $0.16 million if we assumed that CVS started printing only 6” receipts. While a million bucks in savings might not sound like a bad idea (and indeed it isn’t), the costs savings represent only 0.006% of annual operating expenses. Even when using a very aggressive set of assumptions for both kiosk and visitor data, the potential cost savings still only amount to about $4 million a year, or 0.026% of operating expenses. At this point I’m assuming that management reasonably believes that the potential gains in revenues more than offset the costs, although I’m highly interested in their rationale for this, considering that the majority of the coupons offered tend to be irrelevant (see #2 above), and that the wasteful practice of printing long receipts in general may reduce the company’s integrity in the eyes of this generation’s more eco-conscious consumers.
This post is dedicated to my Dad, whose lifelong passion for learning has been an inspiration for my own never-ending pursuit of excellence and the truth. Happy birthday Pops!
—
Disclaimer: This post is the first of (hopefully) many in a new series called “Fun with Excel,” where I use Microsoft Excel to model out and explore interesting real world topics.
—
This week, I explore the topic of physical attraction from a statistical perspective.
A presumably happy couple.
Okay, I admit I was very tempted to insert a provocative picture of <Name of Hot Actress/Model>, but that would have been trying too hard. So what is physical attraction and how does it work? Again, I want to reiterate the fact that I am talking about physical attraction (aka Hot or Not), none of this lovey-dovey emotional stuff. So I don’t want to read a comment later that says, “But Jeff, you didn’t incorporate personality into model!” No, I didn’t, and that was on purpose.
Background: For starters, an assumption: attractiveness is (mostly) objective. Sure, we’ve all heard the phase “beauty is in the eye of the beholder,” and this saying certainly holds merit. Ask a group of men to rank a group of women by attractiveness (or vice versa), and it is highly unlikely that you will get two identical rankings. However, the correlation between the rankings should be statistically significant. Height, skin complexion, body proportion are just three of many physical traits that play a role in defining a person’s “objective” attractiveness. The ancient Greeks figured this out millenniums ago, but in case you’re not convinced, here’s a short expert from Malcolm Gladwell’s Blink. So if attractiveness is indeed objective, it seems reasonable that we can also assume that is is normally distributed. Data collected from the popular dating site OkCupid seems to suggest that this could be the case:
Ignoring the message distribution lines for a moment, we notice that while males rate females on a normal distribution, women seem to rate males on a log-normal distribution. Ouch. So does this mean that most men are just ugly? Not quite. Remember, both males and females are ranking the opposite sex based on their perceptions of attractiveness. But if we know that attractiveness is objective, what might cause the discrepancy between the perceived log-normal distribution and the actual normal distribution? One likely explanation is superiority bias, which is psychology speak for narcissism. Superiority bias states that humans tend to overestimate their positive qualities and underestimate their negative ones. If that sounds familiar, it’s because it is. Superiority bias is documented in almost everything we do, from our perception of our own intelligence to our driving skills (oh God). However, the superiority bias is nothing more than an illusion. 80% of people might rate themselves above average on driving skills, but this is a logical fallacy. By definition, 50% of the population must be above average drivers. The same principle should hold true when it comes to beauty: half of the population is above the average attractiveness level, while half is below. Clear? Ok, let’s move on to the model.
The Model: Throughout the model, I thought about attractiveness on a percentile basis rather than on a raw scale (1-10). Although these methodologies should theoretically yield the same results, it is often more natural for people to think on a linear scale. However, this tendency actually has the impact of embedding our biases into our ratings, causing us to be less objective for the reasons stated above. I found that thinking about things on a percentile basis forces us to consider the situation from a more objective perspective. Rather than ask “Is this person a 9.5 (out of 10)?” which leads us to question what a “9.5” constitutes in the first place, we can ask “Is this person 3 standard deviations above the mean?” The latter has an inherent meaning, namely, if you put this person in a room of 1,000 people is he/she the most attractive person in the room? In addition to assuming a normal distribution for the attractiveness of both men and women, I gave each group two additional characteristics: superiority bias (%) and seek range (%).
I incorporated superiority bias by applying it as a scaling factor for how people perceived their own attractiveness. For example, if you have a true attractiveness-percentile (a-perc) of 50% (i.e. you’re average) and a superiority bias of 0%, then you would perceive yourself as also having an a-perc of 50%. However, if you had a superiority bias of 20%, then you would perceive yourself as having an a-perc of 70%.
Seek range refers to how wide a person looks when looking for a potential partner. There are a few important things to be noted about how the seek range is actually incorporated into the model. First, the seek range is based on one’s perceived a-perc and not their true a-perc. Think about it. If we believe we are more attractive than we actually are, then it makes sense that we would attempt to seek out other people whom we believe to be around the same attractiveness. So returning to our example, if you have a true a-perc of 50% and a superiority bias of 20%, you would perceive your a-perc to be 70%. If you also had a seek range of 20%, you would look for potential partners with a true a-perc between 60% and 80% (I assume for simplicity that people will seek both upwards and downwards equally, except in boundary cases). The rationale for using true a-perc here rather than perceived a-perc is the observation that other people tend to perceive us more objectively than we do ourselves. In other words, superiority bias is something that affects your own perception and not the judgment of others.
By the default, the model assumes that the “seeker” is a male who is looking for a “target” female (it is quite easy to change this if desired). Furthermore the model has the option to customize the superiority bias and seek range of the male and female populations independently.
The Goal: Given a set of assumptions for the 4 input variables (2 biases and 2 seek ranges), the model includes a macro that iterates the seeker’s true a-perc from ~0% to ~100%, returning the compatibility range, which is the range of targets that is also interested in seeker. Remember that while attraction can be one or two-sided, we are only interested in how changing the input variables will impact the area of mutual attraction. Building on our example from earlier, recall that the seeker is a male with a true a-perc of 50% and a superiority bias of 20% (and therefore a perceived a-perc of 70%). With a seek range of 20%, he is looking for females with a true a-perc between 60% and 80%. Conversely, assuming that females also have a bias of 20% and range of 20%, we can back-solve to figure out that the set of females interested in the seeker have a self-perceived a-perc between 40% and 60% (don’t continue reading until this makes sense to you). This in turn corresponds to the set of females with a true a-perc between 28.6% and 42.9% (by reversing the superiority bias). However, recall that the seeker is only interested in females with a true a-perc between 60% and 80%. So it is obvious that in this case that the compatibility range is 0%, and the seeker goes home unhappy to eat his bowl of ramen noodles and cry himself to sleep.
The Results: I first explored the impact of the magnitude of the superiority bias by keeping the bias assumptions symmetrical. Here are the results (click on the charts to see the original image size):
The base case where the superiority biases = 0% paints an interesting picture of what happens at the two extremes. Due to the way that seek range is incorporated in the model, once a-perc reaches either the low-end or high-end, the seek range becomes asymmetrical since the range itself remains the same at all points. I won’t delve too deeply into the mathematical analysis of why the lines look exactly the way they do, but intuitively these results should make sense. People with very low a-percs have a smaller compatibility range since fewer people are interested in them, while the middle of the pack flattens out as expected. People at (and slightly above) the 80% level receive a wide range of interest from the opposite sex, but their compatibility range is still capped at their own seek range of 20%. Lastly, people with very high a-percs also experience a smaller compatibility range, due to the fact that there are simply fewer people pursuing them (more on this later).
Things get interesting as we increase the superiority biases. The middle of the curve becomes more V-like as the bias increases, until the whole curve becomes very bimodal at the 15% and 20% bias levels. In other words, significant superiority bias has a very disproportional negative impact on those of average attractiveness. Due to both their own bias and a symmetrical bias in their targets, these Average Joes will aim for women who won’t be interested in them. Similarly, the range of women who are interested in the Average Joe are below his seek range. In the scenario where both males and females hold a superiority bias of 20%, half of the men (with a-percs between 25% and 75%) end up with a compatibility range of 0%. Now, before you raise your hand and point out that 20% is a very high value to assign a superiority bias, ask yourself this: given a roomful of 100 of your peers, would you rank yourself in the top 30 in terms of attractiveness? If this doesn’t seem entirely ridiculous, then your superiority bias may be larger than you thought. Given the somewhat bleak picture painted by Chart 1, should we all just give up on love and dating if our chances of being attracted to someone who also happens to be attracted to us is so low?
A presumably unhappy couple.
Luckily, no. For one, the assumption that superiority bias is symmetric might not be correct. Remember the two OkCupid charts above, which seem to suggest that males perceive female attractiveness normally while females perceive male attractiveness log-normally? Well, one way to actually incorporate this discrepancy into our model is by making the superiority biases asymmetric. Thus, if we accept the findings of the OkCupid study to be valid for the general population, then we should give women a larger superiority bias than men.
In Chart 2, I’ve kept the female superiority bias constant at 20% for all the plots, while changing the male bias from 20% to 0%. Note that this has the impact of skewing the V-shape part of the plot to the right, while the boundary cases remain unchanged. From these plots, we see quite clearly that even if we make a conscious effort to reduce our superiority bias or even remove it entirely, it doesn’t get us too far if the other side doesn’t reciprocate. So now what?
There are still two things we haven’t considered. As Chart 3 demonstrates, increasing the seek range can in fact compensate for a high superiority bias in the opposite sex (compare the red plot to the orange one). However, note that the vast majority of the benefits resulting from increasing the seek range end up going to the high-end (those with high a-scores), with less impact on the lower end of the scale and almost no impact on the middle section. Finally, we must remember that people’s preferences (and even those of the entire population) can change over time. As we grow older, we gain a better understanding of ourselves as well as a better sense of what kind of partner we’re looking for. This may cause us to lower our own superiority bias while either increasing or decreasing our seek range. For example, the black plot in Chart 3 is my best guess at what the dating scene might look like around the age of 28-35, where most people (and perhaps women more so than men) are looking to get married. This plot looks more along the lines of the black “zero-bias” plot in Chart 1, and features a less serious V-shape, which means there is hope yet for our Average Joe 🙂
So What?
Statistics are interesting and modeling is fun (right Troy?). Our model seems to do a relatively decent job of demonstrating how attraction works, but it’s all somewhat meaningless if we can’t draw some larger conclusions to the real world. To that end, I offer the following points:
Be more flexible. Remember that the upper limit of your success will always be your seek range. It doesn’t matter if you are in the 90th or 10th percentile of attractiveness. If you only search within a 5% range for potential partners, you’re compatibility range is at most 5% as well! Although this point may seem obvious, it reinforces the idea that people of average attractiveness (particularly within 1 SD of the mean) should broaden their seek range in order to increase their chances of success. (Note that increasing your compatibility range is not the same as increasing your utility…but this topic is best left for another day).
It doesn’t hurt to aim high. Take advantage of the fact that people with very high a-percs actually have a smaller compatibility range. This is because there simply aren’t enough people who are able to pursue the high a-percs, since you yourself would need to have a very high a-perc in order for the very top echelons to fall within your seek range. You can differentiate from the crowd by either lowering your superiority bias or increasing your seek range (or both), and doing so will increase your probability of success.
Patience can pay off. After all, attraction involves two parties, and as our model has shown, you can only do so much on your side of the equation to impact the overall reality. However, people are neither homogeneous nor static in their preferences, both of which are fundamental assumptions in the model. So even if things aren’t working out at the moment, don’t give up, because they eventually will in the end.
—
Phew! What a journey. When the idea for the first topic in the Fun with Excel series popped into my head, I didn’t expect to end up this deep in the weeds. The Excel model was a product of a couple days’ thought process, and several hours of actually building out the model and stress testing it. My first version included a particularly nasty macro to continuously automate Excel’s goal seek function, but luckily I was able to figure out a way to reverse the implementation of the superiority bias using mathematics, which sped up the process of actually generating results. If you’re interested, you can take a look at the model here. You are free to play around with it as you like, but if you plan to use it or modify it for academic, commercial, or any other purpose that involves publication, I ask you to please provide the proper attribution.
I thoroughly enjoyed working on this project, and welcome all your questions and comments below. If you have any suggestions of future topics I could pursue in the Fun with Excel series, please let me know!
You must be logged in to post a comment.