Friday, December 21, 2018

Attention: Your Favorite Christmas Movies Suck

Ho ho ho there! The holidays are upon us, and with vacation and family time comes the inevitable re-watching of Christmas favorites from our childhood and beyond. There's only one problem - most of these movies are on the bad side of mediocre, and I'm going to use data (gathered from IMDb) to prove it to you.

PS- my favorite Christmas movie is Bad Santa, if that tells you anything.

Reason #1: Too many Christmas movies are unfunny"comedies"

Looking at only the primary movie genre, and excluding rom-coms (which I'll get to later, yuck), over HALF of the top 100 or so Christmas movies are classified as 'Comedy'. I'm not laughing, though: 

                                                 
You might say - what's wrong with some good, clean, family fun? Well, you may have a point. But the numbers disagree with you. For posterity, here are the average ratings for the top movie categories:



Reason #2: Not nearly enough cursing, violence, or nudity


Almost all the top movies are rated G, PG, or PG-13, as show in the pie chart below. We need more R-rated films, and Die Hard (yes, it counts) is a great example of this: sure, it's nice when the main character reunites with their family in time for Christmas, but it's that much more memorable when they blow stuff up along the way.




Reason #3: The most popular movies aren't necessarily good ones


The following scatterplot shows the most popular Christmas movies and if their popularity (measured by IMDb's traffic) has any correlation to how good they were rated by viewers. Spoiler alert: the correlation is weaker than a snowman in May.



Sure, some popular movies are also good. But if you actually enjoyed The Santa Clause 2, you should probably keep waiting for Easter to come around, because you need Jesus.


Reason #4: They haven't made a good one in years

I'll use a simple table to illustrate this - here are the top 25 Christmas movies by rating, with their release year. Notice anything? There are exactly 4 movies made after the year 2000. The last broad-release Christmas movie to feature in the top 100 is 2015's The Night Before (yes, the one with Seth Rogen) which earned a whopping 6.4 rating, so let's all reflect on how Hollywood has continually failed us.




Reason #5: Would it kill you to have some damn culture?

Sure, America has always dominated the holiday movie industry. But Christmas is celebrated worldwide, and while we do have some Canadian (Silent Night, Black Christmas) and British films (Love Actually,  A Christmas Carol ), these are still all in English. Here are the few non-English movies I could find that register a blip on IMDb:

- The Disappearance of Haruhi Suzumiya (Japan, 2010)
- Joyeux Noel (France, 2005)
- A Christmas Tale (France, 2008)

Reason #6: The best Christmas movies aren't even released on Christmas!

Although the vast majority of Christmas movies are released during November and December, none of the top favorites were released on Christmas Day itself (see the chart).


Imagine, you've spent the entire day with family, and all you want is to see the latest & greatest Christmas film in theaters. But what kind of movies are  released on Dec 25? That honor is reserved for lame money-grabbing flicks like Marley & Me, Catch Me If You Can, and As Good As It Gets. The end result of this being that you end up seeing the same November-release Christmas movie you've seen with your 5-year-old nephew 3 times already in the past month. BORING!


Reason #7: This actually exists


We're so bad at making real-life Christmas movies that we have to resort to milking holiday cheer from an always-grumpy OGRE, his ogre wife, their ogre baby (what?) and the sound of Eddie Murphy's career dying. I rest my case.






Tuesday, December 12, 2017

Don't be alone this Hanukkah! - a handy guide on the names of Jewish singles

It was one of those rare times I joined my friends at a bar in downtown Fort Lauderdale. After several dances with an especially attractive woman, my friend returns to our table, dejected.

"Not Jewish," he says.
"How do you know? Did you ask?" I reply
"Of course not! You can't do that," he retorted. "Anyway, her name is Tiffany. Have you ever met a Jewish woman named Tiffany?"
"Also, she's blonde." my other friend added thoughtfully.

Frequency of names always fascinated me. For better or worse, I was always the only Yoav in school, or even the entire world for all I knew. On the flip side, most of us have more than one friend named David, John, Ashley, Lauren, or Miguel. The Social Security Administration tracks names for babies born each year, and many data scientists have used this readily available information to note trends for certain names and decades. But, where can we find information specifically about Jewish names?

I'm glad you asked. The answer is JSwipe, a popular dating application for Jews. In what was no small feat, I manually collected the names of 1000 Jewish men and 1000 Jewish women (yes, there are at least this many gefilte fish in the sea), aged 18-40, in the South Florida area - from West Palm Beach to Boca Raton to Miami. I also noted the ethnicity of users, either by name origin or other profile details. Get ready to swipe right on my findings:

1. The top female Jewish name is Jessica. For men, Michael and David are tied at #1

The bar graph I constructed shows the top 20 names for both genders:


 


2. Israeli Men Really Need Your Love

While going through profiles, I noticed a definite peculiarity: there are almost twice as many Israeli men on the app than there are Israeli women. If you're a woman looking for men on JSwipe (or rather, having men aggressively look for you), there's a 1 in 7 chance you're swiping on an Israeli, with the most common names being Tal, Avi, and Yossi. If you're a man looking for women, about 1 in 13 are Israeli, with the most common name being Yael. 

While American followed by Israeli names were by far the most common, I also identified three other distinct groups of name origin: Latin (from Brazil and Spanish speaking countries), Russian and Eastern European, and Ashkenazi Orthodox (Hebrew names that are not popular in modern Israel, such as Bracha or Mendel). The breakdown is represented in these two graphs: 


























3. Men's names are slightly more top-heavy, and women have more unique names

Out of 1000 people, there were 417 unique names for women, and only 324 for men. The fact that there are seemingly "less" male names corresponds with the finding that, compared to women, the top 10 male names are more common (taking up 27% of all male names) than the top 10 female names (which constitute 22% of all female names):

Men:
 Women:

















This means that by yelling "hey David, Michael, Daniel!" into a synagogue of 100 men, you should expect an average of 11 people to turn around. In fact, according to the binomial distribution, there's a 99.9% chance that at least 3 people will answer.

For women, however, you might have to shout one more name to achieve the same feedback, and this disparity grows the more names you try to guess.

update: after discussion of these findings, I've had it pointed out to me that male names tend to be more traditional (and often biblical) as opposed to female names (of which there are less in the Bible), which change more quickly and are more likely to be considered "modern". This is certainly an interesting observation.

4. There are too many Bretts

Seriously. I had to swipe through 11 Bretts? Yeesh. 


5. It's interesting to compare Jewish names against the general population

Which names are more Jewish? Which names are less Jewish? I've analyzed data from the general population against the Jswipe data set to find this out. Here's a graphical summary of selected names:






















And there you have it.

----

I hope all of you have an extremely happy and joyful Hanukkah. And to my non-Jewish friends, which almost certainly include one Tiffany who danced with us at Rhythm & Vine last week, I'd like to wish a Merry Christmas and a Happy New Year.

Yours,
Yoav

Wednesday, June 7, 2017

It's Always Sunny in Miami, and Not So Much in Boston

My old roommate and I were involved in a vicious feud. You see, he liked to open the windows when it was warm out; however, he always neglected to go back and close them when it inevitably got cold again. This resulted in me shivering for a few hours before I realize that, while it was 75 degrees only a day ago, it's now 55 degrees and I should really put on some pants.

 The essence of the problem can be described in visual form:



This is a common problem for anyone who's lived in New England, where any Uber trip between April and June is likely to involve complaints about the weather. 

Just how common is this problem? I set out to find just how fickle the daily temperature is among the top 25 US cities. In choosing these cities, I used a combination of metrics: population, GDP, and net migration. The analysis was done using the statistical software R, with weather records from Weather Underground. (For a more detailed, technical view of how I worked with the data, click here)

I first set out to find the variance of the weather for these cities as a measure of volatility. I used daily high temperature data for all 366 days of 2016 (a leap year). I ranked the cities by their temperature variance, in descending order, below:

























But what does this really mean? Variance is defined as the squared sum of deviation from the mean. In other words, a high variance in this context would mean that a city has many days in which the temperature is much different than the average temperature for the entire year. This could mean many days of ping-pong hot and cold weather, or simply a very pronounced changing of the seasons. Let's look at a temperature graph of a few selected cities. I've ordered them from high to low variance:

       

We see that cities with high variance, such as Minneapolis and Denver, have temperatures that reach a wide range throughout the year, and while there's a definite pattern (cold winter, warm summer), there are still sharp week-to-week and sometimes day-to-day changes - see Boston's dip to a high of about 60 degrees in July and then a jump to the 90s a week after. On the flip side, cities with extremely low variance such as Miami and San Francisco have much flatter, lower-amplitude graphs. Amazingly, Miami's daily high temperature hovers between 72 and 92 degrees Fahrenheit a full 90% of the year!

This doesn't fully answer my original question - in which cities are you most likely to experience the frustration of making a completely new daily evaluation of your window openings, how many layers to wear, and whether or not to finally put away your winter clothes for good? To answer this, I've calculated the temperature swings of each day over the previous day. I've chosen a difference of +/- 10 degrees as an arbitrary metric for "pain-in-the-butt-weather". Let's see the results:



So, almost every third day in Boston seems to be a jump in temperature. However, in Miami, you can paint a pretty accurate weather forecast of tomorrow if you've been outside today. I guess now we know why Pitbull is so happy.

PS -I also calculated the number of days in a year with a 20+ degree temperature swings. Denver leads with 21, but almost all cities have 10 or less of these days (Miami and San Francisco have 0). So, if you wake up one morning and it's 20 degrees colder or hotter than the day before, do note that this is pretty rare, and therefore definitely merits complaining (I'll be here to listen!)




Tuesday, April 28, 2015

Are You Smarter Than a Rhode Islander? Analyzing Jeopardy! Data

Jeopardy! is an American trivia game show that has been running for over 30 seasons now. When it comes to game shows, it is a true battle of the brains: contestants have few seconds to answer difficult questions spanning from Greek mythology to Shakespeare to sports and entertainment.

Jeopardy! promo containing its infamous host, Alex Trebek


So what makes a good Jeopardy! player? If you had to choose the perfect contestant in order to win, say, a bet with your friends about the outcome of the next show, who or what would you choose? To replace such thoughts with hard numbers, I dove into historical data on the past 31 seasons of Jeopardy!:

Step 1: Find and extract data
Luckily, a dedicated fan-run website keeps meticulous tabs each show, including the contestants (name, origin, and occupation), the scores at the end of each round, and even the questions themselves. Copying and pasting this would be a chore even an unpaid intern couldn't finish. To speedily extract the information, i used import.io , an incredibly intelligent tool which can scrape data requested by the user over multiple pages automatically

Step 2: Cleaning the data
The fans aren't getting paid to do this, so naturally they missed a show here and there, or cut off a contestant's name or hometown. What do you do with empty or null values? What about data from different show variations such Teen Jeopardy! or Jeopardy! Kids Edition? My point is, data isn't perfect, and we have to make some choices about how to prune the bonsai tree of possible outliers.

Step 3: Asking questions
There are numerous potential variables which may determine a good contestant, but our choices are limited to the data collected. Interesting variables to consider which are currently untracked may be age, sex, or income, for example. I chose to separate Jeopardy! contestants by two factors: state of residence and occupation.

Step 4: Presenting data visually
So, does a state determine whether or not you'll be a winner? Let's take a look at these graphics, which I created using Tableau:


Figure 1: Average Jeopardy! score by contestant's state

It's quite easy to see Utah emerge as the top performer among the show's contestants. That may not be a fair comparison, considering that a huge majority of those data points came from one person - Ken Jennings, a teacher from Salt Lake City who strung together 74 consecutive wins. However, we can see color-coded data on the average score from each state. Note that the overall average winnings for a contestant on Jeopardy! is about $9500.

Let's explore another two charts:

Figure 2: Winning ratio for contestants from a given state


In this graphic, deep red color indicates a low chance of winning. The hypothetical chance for each contestant is 1/3, or 0.333, and you'll see only slight variation among most states. However, Utah again comes up ahead with a 78% chance to win(!), while a contestant hailing from Rhode Island has only a 15% chance to win, less than half the theoretical possibility. Is something in the water over there?

For completeness, we'll show a heat map of where Jeopardy! contestants come from. Counting repeat appearances, there have been over 12,000 guests on the show. 
Figure 3: Geographical frequency distribution of contestants
Unsurprisingly, there are large clusters of contestants from major US metro areas such as Los Angeles, New York, and Washington DC. It should be noted that the number of contestants in Utah and Rhode Island are relatively small (100 and 47, respectively, compared to over 2000 in California and 1200 in New York), but still relevant enough to draw conclusions, depending on the degree of confidence we want. Interestingly, the Law of Large Numbers seems to hold here: states with larger contestant populations approach the theoretical mark mentioned before of a 0.333 chance of winning. 


Lastly, I've identified a few major categories which have comprised the occupations listed for Jeopardy! contestants over the years. The average score for each as well as the count of appearances is listed below:

Figure 4: Average contestant winnings by occupation group. The number of contestants is labeled to the right of the bar.


Students and engineers at the top, managers and moms at the bottom. Is this what you expected? What other metrics do you think would be good for comparing Jeopardy! performance?

In any case, I hope you've enjoyed reading, and I really, really hope you're smarter than a Rhode Islander.

--
Tech notes: for those interested, I have the data files available for sharing in both .mdf (SQL Server) and .xlsx (Excel)

Tuesday, September 2, 2014

Brute-force: introduction to hacking

In late August of 2014, a large set of celebrity information was hacked, with the most "newsworthy" material being nude or explicit photos. Per this article, the security hole existed in Apple's iCloud (specifically, the Find My iPhone feature) which allowed potential hackers to use "brute-force" attacks to gain entry to user accounts.

So, what is brute-force? Stated simply, if you are trying to open a numerical combination lock with 4 digits (0-9 making up 10 possibilities) and you don't know the code, you can try any combination until it opens: 1111, 2918, 3345, etc
The number of possibilities, by using the concept of permutations, is 10*10*10*10 = 10^4 = 10,000
Meaning that given enough time and finger strength, you WILL break the code in 10,000 tries or less (5000 on average).
Brute-force hacking is the most simple form of hacking there is, and usually takes the longest. Other methodologies may or may not be detailed in the future.

10,000 tries is quite a lot - which is why bike thieves usually use a hammer instead


If this code were a digital password, one could use a computer program or internet script to automatically input the 10,000 different combinations to gain access to the protected content. A computer, being much more powerful and fast than the average typing human, could knock this task out in a few hours (a maximum of 10,000 seconds or about 2 hrs and 45 mins), if we assume 1 second per try. However, per Wikipedia, good "cracking" programs can submit attempted passwords at a rate of 100+ million per second.

Consider most websites which require you to have a password of a minimum of 8 characters, using lowercase (26), uppercase (26), digits (10) and special characters such as % ^ & @ * etc (let's say 15 - it can vary per website). Note that this is assuming the English/Latin language alphabet base. The amount of password combinations for a password of exactly 8 characters is thus:

(26+26+10+15)^8 = 1.2 x 10^15 combinations. Dividing by 100 million, or 1x10^8 =
1.2 x 10^7 seconds to break the combination = 143 days. This number further increases if you have the option of using 9, 10, 11 etc characters. Likewise, if you limit yourself to only 8 lowercase letters and no digits or special symbols, your password will take 35 minutes to crack, given that the program attempts only lowercase letters first. This underscores the need for a "strong password".

So does this mean every password can be hacked given enough time? Well, yes. But, like your normal phone screen lock, trying too many wrong passwords results in the user being locked out from trying again - an important security feature. Unfortunately, this feature was neglected in just ONE Apple application which required a sign-in. So, given a celebrity's AppleID (usernames and email address are not exactly private most of the time), the hackers went to work.

So, what have we learned here?
1. Buy Android
2. Use strong passwords
3. Read my blog

(See - Apple's rebuttal)

Sunday, August 10, 2014

Blood, sweat, tears, and cryptography

AES (Advanced Encryption Standard) is quite a bit more powerful than the methods described in my previous blog post. It is widely used among private companies and governments to encrypt text, passwords, and file contents. There are several other encryption standards, which I won't delve into. This website provides a free tool to code and decode messages using AES, and so do many others.

On a personal note, those close to me will know that the past two years have brought some of the biggest challenges in my life (don't mean to be dramatic, I swear!), and also some fun memorable moments To all who have been there in some way or another - I would like to thank you in the form of an encrypted message. All you have to do is use the links with your initials to find my individualized salute to you. Oh, and you all have to ask for your individualized link & passwords

JH
FP
PN
TS
JV
LH
NT
JG
ISO
VT



Cryptography, part one

In an alternate universe, after the British used careful analysis of social networks (great short article, do read) to narrow in on Paul Revere, the colonial hero did not have much time to deliver his famous message. Knowing he may be caught at any moment, he decided to encrypt his message. That is, to turn words into code

Paul Revere - silversmith, patriot, and amateur code boy


 His first attempt was rather weak - turning all letters to corresponding numbers.

A B C D E F G H I J K L M N O P Q R S T U V W X Y Z *
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27
Figure 1
His message now looked like this

T H E * B R I T I S H * A R E * C O M I N G
20 8 5 27 2 18 9 20 9 19 8 27 1 18 5 27 3 15 13 9 14 7
Figure 2

This "code" could easily be intercepted by anyone who read the numbers.  So he tried again, with a slightly more complex replacement - he split the alphabet into 3, and gave each letter a number and a symbol, going backwards:

A B C D E F G H I J K L M N O P Q R S T U V W X Y Z *
9* 8* 7* 6* 5* 4* 3* 2* 1* 9^ 8^ 7^ 6^ 5^ 4^ 3^ 2^ 1^ 9$ 8$ 7$ 6$ 5$ 4$ 3$ 2$ 1$
Figure 3

The message could now be written and passed on more discreetly, but it was still not secure. If the police caught Revere and intercepted his letter, it would be only a short matter of time until they figure out the one-to-one correspondence between letter and code. How could Revere ensure a more random coding and translation? Since he did not have access to modern computing, the answer was matrices.

Matrices can be used as a mathematical basis for cryptology. Using a numerical message, such as in Figure 2, we can use matrix multiplication to "jumble up" numbers to a substantial degree of randomness, making coded messages more difficult to decipher. Besides the message itself, the process requires an encoding matrix, which must be square in shape and invertible (if you are rusty on matrix algebra, don't worry about this part). The larger the square encoding matrix, the more secure the encryption. We will use the following 3x3 matrix below, with message matrix split into columns of 3 for multiplication purposes.

7 2 1
0 3 -1
-3 4 -2
With encoding matrix above, multiply by message matrix below. The message matrix is the original numerical message, [20,8,5,27,2,18...] transposed into columns of 3
20 27  9  19  1  27  13  7
8  2  20  8  18  3  19  27
5 18 9  27  5  15  14  27

The following matrix results: 
161 211 112 176 48 210 143 130
19 -12   51   -3   49   -6   43  54
-38 -109 35 -79  59  -99   9 33

Paul Revere can now write these numbers down on paper, and the code won't be as obvious. For one thing, we are not using 1-27 anymore, and there is no one-to-one correspondence for letters. However, to solve the code, Revere's compatriots will have the key, or the original 3x3 encoding matrix, and multiply its inverse by Revere's transformed new matrix. I won't bother showing these steps - the result, as we've said before, will alert them just in time:

T H E * B R I T I S H * A R E * C O M I N G
20 8 5 27 2 18 9 20 9 19 8 27 1 18 5 27 3 15 13 9 14 7

Happy New Year and Don't Blow Yourself Up: Exploring National Injury Data with Python

Every year, hundreds of thousands of Americans go to the Emergency Room for various injuries resulting from common consumer products. Do you...