Introduction to a New System
The Problem with the Current System
Every time an Amsterdam resident applies for benefits, a caseworker reviews the application for irregularities. If an application looks suspicious, it can be sent to the city’s investigations department—which could lead to a rejection, a request to correct paperwork errors, or a recommendation that the candidate receive less money. Investigations can also happen later, once benefits have been dispersed; the outcome may force recipients to pay back funds, and even push some into debt.
Authority and Investigations
Officials have broad authority over both applicants and existing welfare recipients. They can request bank records, summon beneficiaries to city hall, and in some cases make unannounced visits to a person’s home. As investigations are carried out—or paperwork errors fixed—much-needed payments may be delayed. And often—in more than half of the investigations of applications, according to figures provided by Bodaar—the city finds no evidence of wrongdoing. In those cases, this can mean that the city has “wrongly harassed people,” Bodaar says.
A New Approach: Smart Check
The Smart Check system was designed to avoid these scenarios by eventually replacing the initial caseworker who flags which cases to send to the investigations department. The algorithm would screen the applications to identify those most likely to involve major errors, based on certain personal characteristics, and redirect those cases for further scrutiny by the enforcement team.
How Smart Check Works
If all went well, the city wrote in its internal documentation, the system would improve on the performance of its human caseworkers, flagging fewer welfare applicants for investigation while identifying a greater proportion of cases with errors. In one document, the city projected that the model would prevent up to 125 individual Amsterdammers from facing debt collection and save €2.4 million annually.
Development and Deployment
Smart Check was an exciting prospect for city officials like de Koning, who would manage the project when it was deployed. He was optimistic, since the city was taking a scientific approach, he says; it would “see if it was going to work” instead of taking the attitude that “this must work, and no matter what, we will continue this.” It was the kind of bold idea that attracted optimistic techies like Loek Berkers, a data scientist who worked on Smart Check in only his second job out of college.
The Algorithm Behind Smart Check
Smart Check made use of an algorithm called an “explainable boosting machine,” which allows people to more easily understand how AI models produce their predictions. Most other machine-learning models are often regarded as “black boxes” running abstract mathematical processes that are hard to understand for both the employees tasked with using them and the people affected by the results. The Smart Check model would consider 15 characteristics—including whether applicants had previously applied for or received benefits, the sum of their assets, and the number of addresses they had on file—to assign a risk score to each person.
Transparency and Evaluation
In an unusual step, the city has disclosed this information and shared multiple versions of the Smart Check model, effectively inviting outside scrutiny into the system’s design and function. With this data, we were able to build a hypothetical welfare recipient to get insight into how an individual applicant would be evaluated by Smart Check. This model was trained on a data set encompassing 3,400 previous investigations of welfare recipients. The idea was that it would use the outcomes from these investigations, carried out by city employees, to figure out which factors in the initial applications were correlated with potential fraud.
Conclusion
The Smart Check system represents a significant effort to improve the efficiency and fairness of the welfare application process in Amsterdam. By leveraging advanced algorithms and machine learning techniques, the city aims to reduce the number of unnecessary investigations and minimize the risk of errors. While the system is still in its early stages, it has the potential to make a positive impact on the lives of welfare recipients and applicants.
Frequently Asked Questions
What is Smart Check?
Smart Check is a system designed to screen welfare applications for potential errors or fraud. It uses an algorithm to assign a risk score to each applicant based on certain characteristics.
How does Smart Check work?
Smart Check uses an explainable boosting machine algorithm to evaluate applications and identify those that are most likely to involve major errors. It considers 15 characteristics, including previous applications and assets, to assign a risk score.
What are the benefits of Smart Check?
The Smart Check system aims to reduce the number of unnecessary investigations, minimize the risk of errors, and prevent debt collection. It also has the potential to save the city money and improve the overall efficiency of the welfare application process.
Is Smart Check transparent?
Yes, the city has disclosed information about the Smart Check model and has shared multiple versions of the system, allowing for outside scrutiny and evaluation.
What data is used to train the Smart Check model?
The Smart Check model is trained on a data set encompassing 3,400 previous investigations of welfare recipients. The idea is to use the outcomes from these investigations to figure out which factors in the initial applications are correlated with potential fraud.
