Please note: This is an update to an ongoing experiment. More details will be published as the experiment progresses.
Summary. Infinitive has completed the first phase of an effort to predict viewership of NFL football games. Using publicly available data and the Databricks Data Lakehouse platform we have developed a layered model with a Root Mean Square Error (RMSE) of 3.917M across games with viewership ranging from 60m – 160m viewers. Given the encouraging findings, we plan to extend the experiment by integrating richer and more comprehensive data sources. Sports streaming is experiencing exponential growth, driven by the convenience of allowing fans to watch their favorite games from anywhere. This blog explores the growing trend of live-streamed sports and the need for more accurate ad inventory forecasting tools based on how many viewers are predicted to be watching the live stream. The dynamic nature of live sports, with its constantly shifting matchups and evolving storylines throughout a season, presents a challenge for traditional forecasting methods. To address this, we built and tested AI models, aiming to capture these complexities and deliver more accurate predictions. Our tests initially focused on NFL games due to the abundance of data. Our hypothesis was that incorporating a diverse range of NFL-related variables, such as Vegas odds, fantasy statistics, and win-loss records, would lead to more accurate forecasts. This proved true, as all models significantly outperformed the baseline metric, demonstrating the potential for a highly effective forecasting tool. Future improvements include incorporating proprietary data and predicting ad inventory forecasts at a more granular level (e.g., by quarter). to achieve accurate forecasting, ensuring the process seamlessly addressed the evolving needs throughout the ad inventory lifecycle. The goal is to improve forecast accuracy by 5-10% or more, which can translate into substantial revenue and cost-saving benefits. Overall, this blog dives into our exploration with an AI model designed to revolutionize NFL viewership forecasting in the dynamic world of live sports streaming.
Business Need. The live streaming industry is experiencing rapid growth, indicating that media companies are engaging in significantly more forecasting. Limitations of traditional forecasting methods hinder accurate ad inventory prediction for live streaming events. This necessitates the development of innovative forecasting solutions to unlock the full potential of demand-based dynamic pricing.
Media companies, which own the rights to broadcast or stream NFL games, pay for those rights by selling advertising. These advertisements are sold as what we know as “ad inventory.” For streaming companies, ad inventory encompasses the available spots for advertisements within their on-demand content, live streams, or in the interfaces of their platforms. Streaming platforms may offer targeted advertising based on user data, increasing the value of their ad inventory by allowing advertisers to reach specific audiences more effectively.
For streamers, falling short of your forecast means missing out on potential revenue, while overestimating can result in operational inefficiencies and the need to provide ‘make-goods’ to advertisers. Accurately predicting viewership, like for NFL games, plays a crucial role in determining the amount of ad inventory available for streamed content and its pricing. Overall, more accurate forecasting also allows dynamic pricing to maximize the value of inventory.
In the 2023-2024 NFL season, there were two games that were exclusively available for streaming, marking a significant moment in NFL broadcasting. One of these was an NFL regular-season game between the Buffalo Bills and the Los Angeles Chargers, which took place on Saturday, December 23, 2023, at 8:00 p.m. ET. The game aired exclusively on Peacock, marking the first time a regular-season NFL game was streamed solely on this platform. The second game was the first-ever exclusive live-streamed NFL Wild Card game, which also took place on Peacock, featuring the Miami Dolphins against the Kansas City Chiefs. This historic matchup occurred on Saturday, January 13, 2024.
The Potential. Most sports entertainment experts believe that the trend towards NFL games viewable exclusively through live streaming will only increase in the future. Those same experts also believe that this trend will hold true for other sports as well.
The data-rich environment of sports provides AI models with a fertile ground to learn complex patterns and predict game outcomes with greater accuracy. Beyond just predicting winners, AI can also adapt and learn in real-time, personalizing the viewing experience for fans based on their individual habits. Successfully building models for these predictions can unlock new opportunities to leverage AI for forecasting-related business questions across the media industry.
Our experiment faced the challenge of limited data availability for forecasting. Ad server data was not readily accessible, and historical viewership calculations were inconsistent and outdated. To address this, we utilized Nielsen data, acknowledging its limitations, which we compiled from publicly available resources. Despite these limitations, Nielsen data offered a valuable foundation for our models due to its broader scope, encompassing not just streaming but also other relevant activities and one-off events.
Our Approach. Before tackling a challenge, like predicting NFL viewership, we had to rely on our deep media domain knowledge to layer in a business-centric approach. From a business perspective, having predictions of viewership is crucial throughout the year. The challenge is that throughout a calendar year, different data points are available for when a season is and is not in-progress.
As a result, we built modeling approaches that will allow us flexibility in predicting viewership at different assumed points in time. While quantitative AI methods like statistical and machine learning models offer powerful capabilities, their dependence on specific data assumptions, like temporal or time-based characteristics, can limit their flexibility in adapting to unique needs. By combining our established business workflow with these newly developed models, we can ensure continuous forecasting throughout the year. As more data accumulates over time, the models will refine their accuracy, leading to increasingly reliable predictions.
Our assumption is that predictions before a season starts will provide a general ballpark and predictions during a season will capture important intra-season information that will lead to more accurate model output. The three models we produced to support the business approach are listed below and will be explored further:
- Total Weekly Viewership Time-Series Model
- Prior to Season Start Matchup Viewership Regression Model
- In-Season Matchup Viewership Regression Model
As previously mentioned, we compiled Nielsen viewership data to serve as the target variable in our experiment. The source of our viewership data also included data for when and where matchups aired, allowing us to derive the timeslot (ex. Monday Night Football, Sunday Afternoon). Additional features sourced for our model training included: matchups and scores, league information (divisions, conferences), league standings, major rivalries, social media following, and betting odds.
Our data pipelines ensure a steady stream of diverse publicly available data for model training. However, ongoing exploration of additional data sources remains crucial to potentially optimize model performance. To limit the scope of our efforts and to create more dependable models, we limited our data to regular season matchups, omitting playoff matchups, and removed matchups that were only streamed and not televised. Reliable data infrastructure is critical for moving from experimentation to production!
The Experiment. Our models leverage linear inventory forecasts as the target variable. However, the two pre-season models (1 time-series and 1 regression) are limited by the temporal nature of the data and cannot incorporate in-season data, such as updated win-loss records or futures odds. Only the in-season model, deployed and updated weekly, reflects these dynamic factors for enhanced accuracy.
We established a baseline metric for our two regression models by comparing their performance against the median inventory forecasts from the previous season for each airing window (e.g., Monday Night Football). The metric used in our experimentation was root mean squared error (RMSE), which gives us an idea of the average distance between the observed data values and the predicted data values in the original unit of the target variable (accurate inventory forecasts). For the 2023 regular season, the baseline RMSE that our models should beat is 4.966M.
- Total Weekly Viewership Time-Series Model
- We can look at the 2017-2022 data and make predictions on the most recent 2023 season at a weekly level, making assumptions on the number of matchups during each week.
- Through week 13 in the 2023 season, we are able to achieve an RMSE of about 8M. This means that our model can generally predict weekly NFL viewership within 8 million viewers of the true, observed viewership. When weekly viewership can vary between a massive range (60M-160M viewers), being able to predict within 8M viewers can add lots of value.
- Prior to Season Start Matchup Viewership Regression Model
- We can train a regression model on all information known before a season starts and make predictions for all matchups in each season where some information for a matchup may be assumed. For instance, we may not know when or where a given matchup will air. In that case, we can make predictions under a range of assumptions and note the range of our outputs. Our model is most sensitive to data around the airing window of a matchup, so changing an assumption or a schedule change can have a significant impact on the predicted inventory forecasts.
- We beat our baseline RMSE and achieved an RMSE of about 4.17M.
- In-Season Matchup Viewership Regression Model
- We can build upon the previous prior-to-season start model and include additional features that capture in season factors that drive inventory forecasting on a week-to-week basis.
- We beat both our baseline RMSE and RMSE of the other regression model by achieving an RMSE of 3.917M.
Achieving Results. Like most AI undertakings, results are not guaranteed, and this effort was no exception. To achieve the current performance that we are happy with, the following were important in the process:
Feature Engineering – Creating meaningful features from the data sourced, included a mix of creating Boolean and categorical fields as well as calculating numerical fields that included median viewership in previous seasons, ratios of odds and other data.
Model Choice – We built stacked regression models that included a linear regression meta model and a range of machine learning-based base models. Different models were able to perform well under a set of conditions but poor under others. A stacked model allows us to leverage what each model does well and avoid what it does poorly.
Next Steps. We managed to get encouraging results with our initial experimentation and believe that continued efforts alongside business-specific needs can drive value. Proprietary data is the key to unlocking the potential of this use case and being able to create models at a more granular level. A more granular level can be utilizing streaming viewership data that is captured throughout a broadcast and creating models that are able to predict viewership down to the quarter level. While these models are not yet perfect, they demonstrate promising results using only publicly available data.
