Testing at the Super Bowl

The single largest retail event in the United Status in 2014 may just be the SuperBowl. Selling those products—everything from hats to jerseys and foam fingers too—means shipping logistics, websites, and yes, a lot of software. In this article, we thought we’d take a look at how that software is tested. You might be shocked at what we found.

Take the NFL Shop, the official providers of licensed NFL gear for fans, which ran an extensive and innovative “fan usability testing” effort at Super Bowl XLVIII. Shops in the stadium displayed slightly different designs of the official Super Bowl XLVIII Mohawk Short Thematic Hat side-by-side. In addition, the various NFL Shop stands around the stadium were set up in different ways.

For example, some had mirrors available so shoppers can see how they look in the hats. Post point-of-sale interviews will tell us if the scandalous use of foam fingers by Miley Cyruss in the much maligned duet performance and twerk-a-ganza will boost or sink sales versus those who did not see the televised foam-finger abuse. The sales results of all merchandise will be compared to inform setup for future NFL Shop merchandise stands, but the research goes much deeper than standard sales comparison. (That’s A/B split testing, isn’t it?)

You might have seen the hats or heard about them on TV, but did you know about the researchers wearing nano-sized webcams posing as vendors? Their job was to observe how fans react to the different models of hat. For example, does having a mirror available so that fans can see how they look improve sales or deter buying once the fans see how silly they look? They recorded everything, like buying patterns of different color palettes, including Day-Glo shades versus official team colors. The webcams created video, which will be analyzed for research for next year by answering questions like the following: How many hats do they pick up? How many do they try on?

Different displays will be tried to see if, for example, having the hats hanging up leads to more conversions than laying them out on tables.

The NFL Shop is also experimenting with a new “lean” approach to product development. Inspired by recent events in the news, they assembled souvenir models of the George Washington Bridge on the spot in special pop-up shops within the MetLife Stadium. Throughout the game, bridges of different color schemes were put out for sale to gauge their appeal with fans. Line length was measured to see how the wait deterred sales as a performance test. Accessories, such as a Governor Chris Christie bobble-head with Super Bowl XLVIII emblazoned on the front, were offered with some models.

Some empty boxes with photos were placed on the shelf based on “lean startup” recommendations, and when the products were brought to the register, the results were tallied to gauge customer interest. Product managers will use the sales results to decide whether to produce models of local landmarks for all NFL stadiums in future seasons.

An additional study was performed on Seattle Seahawks head coach Pete Carroll to determine what impact his gum chewing has, if any, on the overall team performance. Pete has been known to answer questions regarding his non-stop game time gum chewing once per season. Both the type of gum, the amount, and the pace of chewing (CPM=Chews Per Minute) were tracked. While pounds-of-pressure-per-bite as well as number of chews per game were suggested, the tracking method difficulty was the deciding factor resulting in leaving out these measures during the initial study. Chewing gum during practice and on game day was a habit Pete started early in his coaching career.

Pete began chewing gum as part of his game day “poker face” to avoid being caught by the media with his mouth open, as if he were catching flies. He does not chew gum at home. Pete’s preference for Bubble Yum is so strong that even with a financial incentive, study leaders were unable to convince him to change brands, eliminating brand competition from the measured factors. Early study results indicated an increased CPM with each Seahawk penalty call, as well as an increased piece per quarter for each minute the Seahawks are behind in the score overall. These behavior changes did not occur during practice sessions, providing an excellent control data point. While a correlation between team penalties and increased chewing pace and overall gum use has been established, there is not enough evidence to establish causation clearly.

On December 8, 2013, Denver Bronco’s kicker, Matt Prater set the NFL Record by kicking a 64-yard field goal! Prater’s kick came as time expired in the first half, to bring the Broncos within 21-20 at intermission in the team’s 51-28 rout of the Tennessee Titans, beating the previous record, held for 43 years by Tom Dempsey of the New Orleans Saints. Knowing the impressive kicking power of the Denver Broncos, the Seattle Kickers practiced with the intent to best the Broncos. “To be the best, you have to beat the best, and that is what we intend to do,” said Seahawks Backup Kicker, Ryan Longwell, who was recently added to the Seahawks roster after starting kicker Stephen Hauschka injured his planting leg in the first-round playoff victory against the Redskins.

Leading up to the super bowl, Seahawks kicker, Stephen Hauschka was seen working with coaches to prepare his field goal kicks for any contingency. When a local news team asked about the temporary orange target zone set up beyond the goal posts, Special Teams assistant coach Nick Sorensen explained that the Seahawks Field goal unit was practicing kicking more than the length of the field, to be prepared for any weather conditions including gale force winds. In order to achieve this kick of 110 yards, the coach would put an orange cloth target zone representing the width of the goal posts out in the stands. "I know Prater can hit it," Hauschka said, referring to the impressive kicks well over 50 yards practiced by Matt Pratter.  "I've seen him it several times." When asked about the possibility of the game relying field goal, Hauscha said, “I thought about it back in April, We sat down as a team back then and each player wrote down his goals. One of my goals was to kick the game-winning field goal in the Super Bowl."

In other NFL testing projects, leading up to yesterday’s game, coaches from both teams were reviewing tape, including past interviews, to discover any potential advantages. The interviews chosen specifically mention tangible and measurable actions. The numbers and actions were then compared against actual player stats to better predict game outcomes.  The players compared weren’t just the current Bronco and Seahawks 2013 rosters and practice teams. They also included previous championship teams to analyze the correlation between reported percentage of effort expended in interviews, and player performance during the playoff games. 

Based on the findings, goals were set for both teams, not only in preparation to play the Super Bowl, but to provide data to improve future coaching. The average reported percentage of effort expended up until yesterday’s game has been approximately 104 percent.  When asked in general terms, NFL Players have reported giving “their all.”

In this scenario, the reported effort would count as 100 percent. The second most popular effort expenditure reported by coaches and players alike was 110 percent.  Numbers where surprisingly stable when grouped by coaches, players, or a combination of both. Researchers were shocked at yesterday’s outcome at the Super Bowl to learn that they missed a metric that could be the key to championship wins.

The percentage effort of the Seattle Fans, also known by the nickname of The Twelfth Man was well over 1000 percent at getting to New Jersey and cheering on their team, thus leading to the historic first ever Super bowl win for Seattle!

What They Got Wrong
If you’ve read this far, you’ve probably realized that the research was too late for this year. With one big-ticket item per year, all the research and split testing needed to happen before the event in order to optimize sales. Likewise, Hauschka’s 110 percent kick didn’t offer any specific value to the team; it didn’t make sense, and the greater-than-once-hundred percent effort isn’t really a thing either.

But that’s okay. Because we made it all up.

The lessons for test, though? Those are for real.