Improve On-Time Performance (OTP) for Airlines

Business Problem

Delays are costly for airlines and their passengers. A 2010 study commissioned by the Federal Aviation Administration estimated that flight delays cost the airline industry $8 billion a year, much of it due to increased spending on crews, fuel, and maintenance. This does not yet include the cost to the passenger and the cost of lost demand. Tardiness creates other problems including missed connections, lost bags, and short tempers among frustrated travelers. While the airline cannot avoid delays entirely, minimizing their magnitude definitely will save the company a lot of money. A daily departure delay prediction could aid in the decision-making process of the tactical teams as they assess Air Traffic Control (ATC), maintenance, crew connections, ground handling, and scheduling integrity even before a delay happens, or even before the day starts. Currently, if an aircraft is delayed, all subsequent flights connected to that aircraft will be tagged as delayed in the system.

Intelligent Solution

Airline and airport operations are the logical and best first areas for performance gains through the introduction of AI. The technology enables additional data points to be collected in a real-time situation, offering a bird’s eye view which helps the operators get the necessary equipment and people in place. This means that trends can be identified to predict future challenges for OTP and resolve them before they become problems.

There are three basic elements that can be used to create AI measures for OTP:

• Proportion of flight departures departing on time (usually within a specified time window)
• Proportion of flight arrivals arriving on time
• Proportion of flights cancelled

Through AI, OTP can be improved by predicting whether a flight will be delayed using information such as the scheduled departure time and the day of the flight, etc.

Value Estimation

How would I measure ROI for my use case? 

The cost of delay to airline operations comprises several components, such as costs of passenger delays to the airline and crew and maintenance costs. During the airborne phase, fuel costs and (in the future) emissions charges need to be considered. The total cost is often dominated by the passenger component, which can be split into ‘hard’ costs (such as those due to passenger rebooking, compensation and care), and ‘soft’ costs.

The bottom up formula mentioned below to calculate cost is only for reference and just covers those costs related to customer delay. Note that costs will vary by aircraft type, departure or arrival airport, and (typically) time of the year. 

As an example, the estimated cost to an airline due to delay from B737-300 (~3Mn USD)  = $13.22 (cost per minute for delay between 1-15 min) * 1163 (Avg number of flights per airline / month) * 12 (months in a year) * 15 (minutes)

About the Data  

For illustrative purposes, in this tutorial we are going to be using a dataset provided by the US Department of Transportation that tracks the on-time performance of domestic flights operated by large air carriers. The dataset for this problem is from Kaggle. 

Problem Framing

The target variable for this use case is whether or not a flight departure will have been delayed by more than 15 minutes (Binary; True or False, 1 or 0, etc.). This choice in target makes this a binary classification problem. The target variable in our data is not explicitly defined. Hence we did a bit of feature engineering to define our target. A new feature, time_to_take_off, is created to calculate the time taken for a flight to take off (wheels off) from its scheduled departure time.  The calculation for this is DEPARTURE_DELAY + TAXI_OUT. And finally the binary target variable (delay_more_then_15) is calculated as 1 if time_to_take_off is more than 15 otherwise 0.

The features below are the raw features pertaining to a flight history. They encompass each flight’s historical departure, arrival time, origination/destination airport, etc, which will help DataRobot find relevant patterns across the dataset. 

Other external data sources that may also be relevant include: Bureau of Transportation Statistics (U.S.), weather, holidays, social media etc. There can be additional features which come from the aircraft (e.g., number of seats), airport (e.g., gate number) and flights (i.e., statistics about previous departing flight from the same gate).

Sample Feature List

Data Preparation

There are 3 main tables to the dataset.

• Airlines: Mapping of airline code and airline names
• Airports: Information about an airport such as city, state, lat, long
• Flights: Information of a particular flight with details such as departure airport, time, runway time, arrival airport, arrival time etc.

The original raw data consisted of ~6 million flight details for a time period of 1 year (2015). For the final data prep, filtering is done on the original data. For the model development (below), we use only data from April and remove all cancelled flights.

1. Use only April data.
2. Remove all cancelled flights.

The 3 data files are joined using the following keys:

Model Training

DataRobot AutoML automates many parts of the modeling pipeline. Instead of having to hand-code and manually test dozens of models to find the one that best fits your needs, DataRobot automatically runs dozens of models and finds the most accurate one for you, all in a matter of minutes. In addition to training the models, DataRobot automates other steps in the modeling process such as processing and partitioning the dataset. 

We will jump straight to interpreting the model results. Take a look here to see how to use DataRobot from start to finish and how to understand the data science methodologies embedded in its automation.

Interpret Results

Features with flags for delay reasons are the top features for predicting on time performance. The most impactful feature is delay caused by an aircraft; as shown here, that feature is LATE_AIRCRAFT_DELAY. 

If the delay caused by an aircraft is high, then the probability of a delay is higher.

Delays due to air systems decrease by up to 14-18 minutes and then increase drastically afterwards.

Delays due to the airline increase the probability of flight delays.

As the scheduled time for a flight increases, so does the probability for delay.

The probability for delay decreases as distance increases.

Evaluate Accuracy

The best performing model’s AUC is 85.57 for the validation set and 84.71 for cross validation. 

This gives a good understanding that the model is well generalized and not overfitting on the data. 

The Lift Chart also shows that the predicted values (blue line) are very close to the actual values (orange line) in all deciles, thus giving confidence of the model performance. 

(The reason we use CV and not OTV for this scenario is because we are using data for one month in the domestic US market. )

Decision Environment

After you are able to find the right model that best learns patterns in your data to predict flight delays, DataRobot makes it easy to deploy the model into your desired decision environment. Decision environments are the ways in which the predictions generated by the model will be consumed by the appropriate stakeholders in your organization and how they will ultimately make decisions using the predictions that impact your process. 

This is a critical piece of implementing the use case as it ensures that predictions are used in the real world to reduce flight delays and reduce airline costs. 

Decision Maturity 

Automation | Augmentation | Blend 

While there are use cases where decisions can be automated in a data pipeline, a flight delay model is geared to augment the decisions of network control centers. It acts as an intelligent machine that, combined with the expertise of NCC manager, determines flights that are likely to be delayed for the day so that crewing, maintenance, and airports can make optimal decisions quicker and earlier.

Model Deployment

DataRobot provides various forms of deployment for its users. For this use case we developed a Tableau dashboard for users to consume flight delay predictions. Tableau provides an extension for DataRobot to build visualizations on the fly. (Check out this step-by-step guide to use the Tableau extension, https://www.datarobot.com/drstarterkit/.) 

Decision Stakeholders 

Decision Executors

Decision executors for airline OTP prediction would be a network control center team to identify which flights could delay, on a daily basis. 

Decision Managers

Decision managers are the executive stakeholders who will monitor and manage the program to analyze the airline’s on-time performance. 

• NCC manager

Decision Authors

Decision authors are the technical stakeholders who will set up the decision flow in place.    

• Ops analytics team

Decision Process

There are several decisions that could be taken by teams:

• Proactive scheduling of maintenance and staff based on flight schedules
• Manage the flight paths and routing
• Smart gate allocation
• Manage customers’ expectations and improve customer experience

Model Monitoring

With the OTP predictions, the team can then take preventive actions to manage on ground support, improve customer service, and eventually reduce costs.

• Decision Operators: Ops analytics team
• Prediction Cadence: Batch predictions generated on a daily basis 
• Model Retraining Cadence: Models retrained once data drift reaches an assigned threshold; otherwise, retrain the models at the beginning of every new operating quarter.

Implementation Risks

With the recent outbreak of COVID-19, several airlines have cancelled their operations. But as the curve flattens and airlines resume their operations, it is important to monitor and manage the models. Factors such as data drift and model accuracy become important to enable performance monitoring. Additionally, external data such as staff availability and the impact of COVID-19 in a country could also be considered to incorporate with the model.