Predict Likelihood of Loan Default and Credit Default Rates

Overview

Business Problem

After the 2008 financial crisis, the IASB (International Accounting Standard Board) and FASB (Financial Accounting Standards Board) reviewed accounting standards. As a result, they updated policies to require estimated Expected Credit Loss (ECL) to maintain enough regulatory capital to handle any Unexpected Loss (UL). Nowadays, every risk model undergoes tough scrutiny, and it is important to be aware of the regulatory guidelines while trying to deliver an AI model.

In this use case we focus on credit risk, which is defined as the likelihood that a borrower would not repay their lender. Calculating the ECL using ECL = PD * LGD * EAD is the most common approach. We will focus on building the PD model for the consumer loan portfolio and provide some suggestions related to LGD and EAD modeling. Credit risk can arise for individuals, SMEs, and large corporations. Depending on the asset class different companies take different strategies for ECL = PD * LGD * EAD.

Intelligent Solution

Many credit decisioning systems are driven by scorecards, which are very simplistic rules-based systems. These are built by end-user organizations through industry knowledge or through simple statistical systems. Some organizations go a step further and obtain scorecards from third parties which may not be customized for an individual organization’s book.

An AI-based approach can help financial institutions learn signals from their own book and assess risk at a more granular level. Once the risk is calculated, a strategy may be implemented to use this information for interventions. If you can predict someone is going to default, this may lead to intervention steps such as sending earlier notices or rejecting loan applications.

Value Estimation

How would I measure ROI for my use case? 

ROI for this use case varies on the size of the business and the portfolio. For example, the ROI for secured loans would be quite different to those for a credit card portfolio. 

Furthermore, if you are moving from one compliance framework to another you need to take the appropriate considerations – whether to model a new and existing portfolio separately and, if so, make appropriate adjustments to the ROI calculations.

ROI depends on the decisioning system. If it is a binary (yes or no) decision on loan approval, we can assign Dollar values to the amounts of true positives, false positives, true negatives, and false negatives. The sum total of that is the value at a given threshold. If there is an existing model, the difference in results between existing and new models is the ROI captured. 

If the decisioning is non binary, then at every decision point, evaluate the difference between loan provided and the collections done.

Technical Implementation

About the Data  

For illustrative purposes, this tutorial uses sample datasets on home credit provided by Home Credit Group, which are publicly available on Kaggle. 

Problem Framing

Banks deal with different types of risks like credit risk, market risk, and operational risk. Calculating the ECL using EL = PD * LGD * EAD is the most common approach nowadays. Risk is defined in financial terms as the chance that an outcome or investment’s actual gains will differ from an expected outcome or return. 

There are many ways you can position the problem, but for this specific use case we will be building Probability of Default (PD) models and will provide some guidance related to Loss Given Default (LGD) and Exposure at Default (EAD) modeling. For the PD model, the target variable we are predicting is whether or not a borrower will repay their debt.

Here is additional guidance on the definition of each component of the ECL equation. 

Probability of Default (PD) 

• The borrower’s inability to repay their debt in full or on time.
• Target is normally defined as 90 days delinquency.
• Machine learning models generally give good results if good enough data is available for a particular asset class.

Loss Given Default (LGD) 

• The proportion of the total exposure that cannot be recovered by the lender once a default has occurred. 
• Target is normally defined as the recovery rates and the value lies between 0 and 1. 
• Machine learning models required for this problem normally use Beta regression which is not very common and supported in a lot of statistical software. We can divide this into two stages since a lot of the values in the target are zero.
  • Stage 1—the model predicts the likelihood of recovery greater than zero.
  • Stage 2—the model predicts the rate for all loans with the likelihood of recovery greater than zero.

Exposure at Default (EAD) 

• The total value that a lender is exposed to when a borrower defaults. 
• Target is the proportion from the original amount of the loan that is still outstanding at the moment when the borrower defaulted. 
• Generally, machine learning models with MSE as loss are used.

Sample Feature List

Feature Name	Data Type	Description	Data Source	Example
Amount_Credit	Numeric	Credit taken by a person	Application	20,000
Flag_Own_Car	Categorical	Flag if applicant owns a car	Application	1
Age	Numeric	Age of the applicant	Application	25
CreditOverdue	Binomial	Whether credit is overdue	Bureau	TRUE
Channel	Categorical	Channel through which credit taken	PreviousApplication	Online
Balance	Numeric	Balance in credit card	CreditCard	2,500

Model Training

DataRobot Automated Machine Learning automates many parts of the modeling pipeline. Instead of having to hand-code and manually test dozens of models to find the one that best fits your needs, DataRobot automatically runs dozens of models and finds the best performing one for you, all in a matter of minutes. In addition to training the models, DataRobot automates other steps in the modeling process such as processing and partitioning the dataset. 

We will jump straight to interpreting the model results. We recommend watch this end-to-end demo to get a better understanding of the DataRobot AI Platform. Learn how to connect DataRobot to your data source, perform feature engineering, follow best practice data science techniques, and more by visiting the DataRobot Documentation. Alternatively, sign up for our 30-day free trial to experience the full breadth of DataRobot capabilities.

Interpret Results

Feature Impact

Feature Impact (for a selected machine learning model) reveals the association between each feature and the model target. 

Feature Effects (AMT_CREDIT – Loan Amount)

In order to understand the direction of impact and the default risk at different levels of the input feature, DataRobot provides partial dependence plots (within Feature Effects) to depict how the likelihood of default changes when the input feature takes different values. 

In this example, as the loan amount increases above to $300K, the default risk increases in a step manner from 6% to 7% and then in another step to 7.8% when the loan about is around $500K. 

Prediction Explanations

In order to turn the machine-made decision into human-interpretable rationale, DataRobot provides Prediction Explanations for each alert scored and prioritized by the machine learning model. In the example below, the record with ID=3606 gets a very high likelihood of turning into a loan default (prediction=51.2%), and the main reasons are due to information from external sources (EXT_SOURCE_3) and the source of income (NAME_INCOME_TYPE) being Pension. 

Prediction Explanations also help in maintaining regulatory compliance. It provides reasons why a particular loan decision was taken.

Evaluate Accuracy

Lift Chart

The Lift Chart shows you how effective the model is at separating the default and non-default applications. Each record in the out-of-sample partition gets scored by the trained model and assigned with a default probability. In the Lift Chart, records are sorted based on the predicted probability, broken down into 10 deciles, and displayed from lowest to the highest. For each decile, DataRobot computes the average predicted risk (blue line/plus) as well as the average actual risk (orange line/circle), and displays the two lines together. 

ROC Curve

Once we know the model is performing very well, we have to select an explicit threshold to make a binary decision based on the continuous default risk predicted by DataRobot. To pick up the optimal threshold, we follow following criteria:

• The false negative rate has to be as small as possible. False negatives are the applications that are flagged by DataRobot as not defaults but actually end up defaulting on payment. Missing a true default is dangerous.
• We also want to ensure the selected threshold is working not only on the seen data, but on the unseen data too.

Post-Processing

In some cases where there are fewer regulatory considerations, straight-through processing (SIP) may be possible, where an automated yes or no decision can be taken based on the predictions.

But the more common approach is to convert the risk probability into a score (i.e., a credit score determined by organizations like Experian and TransUnion). The scores are derived based on exposure of probability buckets and on SME knowledge.

Most of the machine learning models used for credit risk require approval from the Model Risk Management (MRM) team; to address this, the compliance documentation provides comprehensive evidence and rationale for each step in the model development process.

Business Implementation

Decision Environment 

After you are able to find the right model that best learns patterns in your data, DataRobot makes it easy to deploy the model into your desired decision environment. Decision environments are the ways in which the predictions generated by the model will be consumed by the appropriate stakeholders in your organization, and how these stakeholders will make decisions using the predictions to impact the overall process. 

Decision Maturity 

Automation | Augmentation | Blend 

Decisions can be a blend of automated / straight-through processing or manual interventions. The degree of automation depends on the portfolio and business maturity. For example, retail loans or peer-to-peer portfolios in banks and fintechs are highly automated. Some fintechs promote their low loan processing times. Unlike high ticket items like mortgages, corporate loans may be a case of manual intervention.

Model Deployment

Most banks will have a decision management system in place. Two of the major vendors are Experian and FICO. 

Experian

Experian has a decision management system called PowerCurve. It supports Java environments and also API calls. Based on our work with Experian, a Codegen-based deployment was proven to be the best solution. 

FICO

FICO is an analytics company with a proven track record in the risk analytics space. Their Blaze system supports Java-based models, like DataRobot Scoring Code. One possible approach would include a middle layer codebase that scores a DataRobot model and then feeds it into the Blaze decisioning system. (Consult a DataRobot Use Case Engineer if you are interested in such an implementation.)

Decision Stakeholders

Decision Executors

The underwriting team will be the direct consumers of the predictions. These can be direct systems in the case of straight-through processing or an underwriting team sitting in front offices in the case of manual intervention.

Decision Managers

The decisions would flow through to the Chief Risk Officer who is responsible for the ultimate risk of the portfolio. There would be, however, intermediate managers based on the structure of the organization.

Decision Authors

Data scientists in credit risk teams would be driving the modeling. The model risk monitoring team would also be a key stakeholder here

Decision Process

We generally do not see a direct yes or no decision being made, except in cases where models are used in less-regulated environments. We generally see the risk being converted to a score and, based on the score, the interest or credit amount offered to the customer. 

Model Monitoring

Predictions are done in real time or batch mode based on the nature of the business. Regular monitoring and alerting is critical for data drift. This is particularly important from a model risk perspective. These models are designed to be robust and last longer, so recalibration may be less frequent than in other industries. 

Implementation Risks

• Credit risk models normally require integrations with third party solutions like Experian and FICO. It is recommended to ask about deployment requirements and if it is possible to move away from legacy tools. 
• Credit risk models require approval from the validation team. It can take a really long time to convince them to adopt new methods of approving models if they have not previously approved machine learning models. 
• Model validation teams can have strict requirements for specific assets. For example, if models need to generate an equation, DataRobot should support export of the code for that model. Please make sure to discuss all these questions in advance with the modeling team before taking the final model to validation teams. 
• Discuss alternative approaches for assets where default rate was historically low as the model might not be accurate enough to prove ROI. 

Trusted AI

In addition to traditional risk analysis, the following elements of AI Trust may require attention in this use case. 

Target leakage: Target leakage describes information that should not be available at the time of prediction being used to train the model. That is, particular features make leak information about the eventual outcome that will artificially inflate the performance of the model in training. While the implementation outlined in this document involves relatively few features, it is important to be mindful of target leakage whenever merging multiple datasets due to improper joins. DataRobot supports robust target leakage detection in the second round of exploratory data analysis (EDA) and the selection of the Informative Features feature list during Autopilot.

Conclusion

People with years of experience can judge someone’s likelihood of default, usually with the help of scorecards, which are very simplistic rules-based systems. But that is not an efficient method for judgement that is often not based on a financial institution’s own book. Increasing the value and number of loans is difficult without a scalable method in place to judge the default risk of many applicants.

DataRobot uses past information about borrowers’ default rates to predict the likelihood of default for future borrowers. Incorporating the predictive models built with DataRobot into a real-time loan approval process is easy, allowing businesses to scale up and expand their loan portfolios.

DataRobot makes it easy to build and deploy default rate-based models, automating real-time applicant risk predictions.