Improve Patient Satisfaction Scores

Business Problem

To operate, Federally Qualified Health Centers (FQHC) rely on funding through programs such as Medicare and Medicaid. They are required to provide medical services on a sliding scale and are often governed by a board that includes patients. Success of an FQHC is measured in part on the satisfaction of patients having received care; this feedback plays a direct role in a center’s ability to receive funding and continue serving the local community.

In addition to understanding how hospital operations have led to poor satisfaction historically, it’s also important to have the ability to flag potential risks in real time. This can provide insight into which patients are potentially not having a great experience and can be used by hospital administration to intervene, talk to the patient, and rectify any issues. Increased patient satisfaction is a positive outcome for the community that the FQHC serves, and ensures that the provider can continue to operate to its fullest.

Intelligent Solution

With AI, hospital administrators can, in real time, understand which patients are likely to leave the hospital with a bad experience. Knowing which patients may be unsatisfied with their care, coupled with the primary reasons why, provides useful information to hospital administrators who can seek out patients, discuss the quality of their stay, and take steps to improve satisfaction. AI informs hospital administrators with the information they need to increase patient satisfaction scores: an outcome that serves the community and ensures that the provider can continue to operate.

There are two primary parts involved in this particular solution:

• Predict which patients are likely to respond to a survey request.
• Predict if that patient will respond with a positive or negative review of the care.

Value Estimation

How would I measure ROI for my use case? 

The value of increased patient satisfaction comes in two primary ways.  First, patient satisfaction is a key indicator of how well an FQHC is meeting the healthcare needs of the community that it serves.  Therefore, taking steps in increasing satisfaction is a positive outcome for the surrounding community.  Additionally, a one percent increase in satisfaction scores can substantially impact that funding that an FQHC receives, allowing it to expand services, hire more clinicians and continue providing quality care for those that need it most.

Problem Framing 

The target variable for this use case should be aligned to the survey output by which the provider is measured.  In this case, we’ll consider the Press Ganey patient satisfaction survey that buckets responses in three categories: top, middle, and bottom box, with top box being the best and bottom box being the worst.  Therefore, the target will be if the patient’s survey response was in the top, middle, or bottom box.  Furthermore, this means that we can frame this problem in two ways: 

• Multiclass with three classes: top, middle, and bottom box.  This will be the approach we’ll consider during this tutorial.
• It may also make sense to have three different binary models, one for each box score.

Additionally, it might be useful to also develop an initial model that predicts the likelihood of a response to a survey request. In this case, the target variable can be whether or not patients responded to surveys historically. 

Key features that are important in predicting patient satisfaction are listed below. They encompass information about the patient, their stay, diagnosis, and the interaction with clinicians.

• Demographic information
• Clinical data including diagnoses codes
• Clinician notes
• Clinician performance data and demographics
• Patient history information such as number of previous visits, recency, and frequency of visits and reasons
• Patient location information such as distance to provider

Beyond the feature categories listed above, we suggest incorporating additional data your organization may collect that could be relevant to readmissions. As you will see later, DataRobot is able to quickly differentiate important vs. unimportant features. 

Many of these features are generally stored across proprietary data sources available in an EMR system: Patient Data, Diagnosis Data, Clinician notes, Admissions Data. Examples of EMR systems are Epic and Cerner. 

Sample Feature List

Model Training

DataRobot Automated Machine Learning automates many parts of the modeling pipeline. Instead of hand-coding and manually testing dozens of models to find the one that best fits your needs, DataRobot automatically runs dozens of models and finds the most accurate one for you, all in a matter of minutes. In addition to training the models, DataRobot automates other steps in the modeling process such as processing and partitioning the dataset.

Take a look here to see how to use DataRobot from start to finish and how to understand the data science methodologies embedded in its automation.

There are a couple of key modeling decisions for this use case:

• Partitioning: It is possible that care procedures may change over time due to organizational or personnel changes.  Therefore, an OTV (out of time validation) partitioning scheme would be a good choice.  This approach will evaluate model performance on the most recent encounters and give a more accurate benchmark of how well the model will perform when deployed.
• Accuracy metrics: LogLoss

Decision Flow

After you are able to find the right model that best learns patterns in your data to predict patient satisfaction, DataRobot makes it easy to deploy the model into your desired decision environment. Decision environments are the ways in which the predictions generated by the model will be consumed by the appropriate stakeholders in your organization, and how these stakeholders will make decisions using the predictions to impact the overall process. 

Decision Maturity 

Automation | Augmentation | Blend

In practice, the output of these models can be consumed by a team focused on ensuring care satisfaction during a patient’s stay. A daily report or dashboard of patients and their predicted satisfaction scores can provide a guide for senior hospital administration and an understanding, in real time, of which patients are potentially dissatisfied with their care.  Furthermore, leveraging Prediction Explanations from DataRobot can be a useful way to understand the primary drivers of the dissatisfaction and allow the decision maker to address those issues directly.   

Model Deployment

All new applicants should be scored on a daily batch basis. Predictions and Prediction Explanations should be returned and saved in the database, which can be integrated into a BI dashboard or tool available to the hospital administrators. This allows them to continuously monitor which patients are likely to be dissatisfied with their stay. 

Decision Stakeholders

Decision Executors

Hospital administrators focused on patient experience. They are often under the organization of the Chief Patient Experience Officer of a provider or hospital.

Decision Managers

The Chief Patient Experience Officer is ultimately responsible for executing the strategy to improve patient experience and responsible for the use of the model output.

Decision Authors

Analytics professionals and data scientists with strong understanding of the patient data and processes are best positioned to develop these models as well as develop a meaningful representation of the output in the form of dashboards or reports. Data engineers and IT support is needed to ensure that the stakeholders can receive timely predictions reliably since these will require daily action from hospital administration.

Decision Process

• Coaching staff and clinicians are provided with information about potential issues.
• Prediction Explanations can guide staff on the top drivers for dissatisfaction.
• Staff can also speak directly to patients who are predicted to score low to understand what issues they may have regarding their experience and addressing those issues directly. 

Model Monitoring

Models should be retrained when data drift tracking shows significant deviations between the scoring and training data.