Scoring

A new initiative from the company that developed the first expert system for the collection industry

About Quantrax Corporation -

Quantrax Corporation is a software company that provides innovative technology solutions for the collection industry in the United States and Canada. Intelec, an expert system for collections was first installed in 1991. Today Quantrax supports over 100 clients in the US and Canada, has its own high-end, integrated predictive dialing platform and also offers economical, manual skip-tracing services for its clients.


Background and introduction

Scoring has intrigued and dazzled the collection industry for many years.  As automation has progressed and computers have become more powerful, scoring technologies and the application of scoring have also matured. In 1991, Quantrax started deploying an expert system called Intelec for managing collection operations. While intelligent software can offer tremendous gains in overall productivity, the greatest benefits are probably derived by trying to apply just the right amount of automated and human resources to every account. You can not do that using human expertise. The problem is far too complex and humans can not do what is required. This is truly a problem where computers are not simply useful, but are required. Few have described “collections” as an engineering problem. It really is! Consider the following.

  • Very large volumes of accounts are placed for collection

  • Collection experts are expensive

  • There are a large number or rules (client specified and based on legislation) to be considered when working and managing accounts

  • Automated options (e.g. letters, messages left by a dialer) cost less, but may not deliver the same results as human contact

  • The profitability of a collection operation depends on the fees generated (commission rate) and the costs of working account.

  • The quality of the resources applied to working an account is usually directly related to the results obtained. E.g. While a great collector could convince a less motivated debtor to pay, an inexperienced collector could fail to recover a very collectible account. Employing only experienced collectors is not viable and will always be cost prohibitive. As a result, proper account management becomes critical in controlling expenses and increasing profitability and unit yield.

  • While a reasonably professional company can successfully collect for most clients, true competitive advantage, great profitability and the ability to retain a client depends on one’s ability to do much more than everyone else. This usually takes more than the traditional tools such as collection software, collectors and management. Business intelligence and account scoring are options that must be considered if we are to
    • recover money faster
    • spend less time on uncollectible accounts
    • reduce the expense incurred working each account
    • use human resources in the most appropriate manner, using the right people for the right accounts and paying commissions to collectors only when it is necessary

As you can see, we face some significant challenges. Unfortunately there are no simple solutions to really complex problems and while we can do different things to shift it, complexity can never be eliminated. We believe that any viable solutions to the "collection problem" calls for an engineering methodology, which is the same approach Quantrax had taken when Intelec was initially conceived and designed.

Scoring refers to our ability to take a batch of accounts and determine which ones are most (or least) likely to pay. If you do not have a method of scoring your accounts, you have no option but to work every account the same way. If only 20% of bad debt is collected (much less with purchased debt) this means that you could be putting the same effort into the 80% that is not collected as the 20% that is recovered. This is a significant waste of valuable resources and a very significant expense. Practically, you can not afford to (and do not) put the same effort into every account. and usually give up early on many smaller balances, even though they may be collectable.

Today, credit bureaus and independent companies offer scores that you can be incorporated into business models and work plans within collection operations. For several years, Quantrax has considered the scoring of bad debt as a new and unconquered challenge. Quantrax was not convinced that any company had found the best way to score bad debt and believed that new approaches had great potential. This documents some of Quantrax’s research and results.


How are accounts scored?

There are many different approaches to scoring a batch of accounts. Typically, the “credit scores” that almost everyone refers, relate to new debt. As an example, a FICO score is used by a car dealer to evaluate your credit application. FICO comes from the Fair Isaac Company, which designed a technique to condense all of your credit information into one three-digit number! Most credit scores will use credit reports to analyze prior credit and payment history to determine a person’s ability to repay another new debt. A high score is good and a low score implies a greater risk in lending to the individual.

With regard to late stage collections, bad debt or purchased debt (sometimes out of statute), it is very likely that most of the scores obtained would fall within a  range where there are many accounts, with most falling within the uncertain or risky classification. FICO scores, for example range between 300 and 850. The following is how FICO scores are usually interpreted.

  • Excellent: Over 750
  • Very Good: 720 or more
  • Acceptable: 660 to 770
  • Uncertain: 620 to 660
  • Risky: less than 620

The average US credit score is supposed to be 678. While most people therefore have an acceptable credit score, we can see that the range of scores that are defined as uncertain or risky happens to be far greater than the range of acceptable or good scores. But if you have many accounts with lower scores, making good business and work plan decisions is not really possible. The debtors in the lower score range are too similar to make sound, different decisions on, and this is typically what you will face with bad debt. It is also the reason why several companies now offer you a “collection” or “recovery score”. And exactly how are these scores generated? Just as with all types of scores, different analytical tools are used. There is mathematics and modeling, but one fact emerges. No vendor clearly divulges exactly what they do. Marketing is always very powerful and is shrouded with technical terms and references to high technology. While we can easily conclude that accurate and meaningful scores are valuable, we usually do not understand exactly how the scores are generated and have to rely on marketing information and statements from a biased vendor, in order to place a value on the supplied scores.

Because scores promise such a great increase in productivity, a good collection operation is expected to apply scoring technology to their collection model. Today, first party lenders rely on scores and we do not dispute the value of the scored that are used. But why did the account go to bad debt? If credit was extended based on a favorable score, something did go wrong! If we want to use a score, it follows that we need a different score from the one that was initially used! We need a new method of scoring accounts than find their way into later stage collections. Assuming that we do find a solution to this problem, it is very likely that the same model will also work for early stage collections.


How do we evaluate the accuracy of a score?

Evaluating the accuracy of any scores you may obtain is not an easy task. Unfortunately, there is no independent evaluation system and no one we can go to and say “Tell me if these scores are accurate”. Let us consider an example where you have received a score on a batch of 20,000 new accounts. What would you now do with the scores you obtained? Most companies will put more effort into the accounts with a higher score. Less effort will be applied to the other accounts – That may mean more letters, less collector contact or less experienced collectors being assigned to the low-scoring accounts. What can we expect as a result of the decisions you make and the action taken? We would expect to collect more from the accounts into which we put more effort! We would analyze our results and find that we had collected more from the accounts with better scores! Was that because of the value of the score or because we put more effort into those accounts? Because you will almost always be able to justify any scores if you apply effort based on the scores, many refer to account scoring as a “self-fulfilling prophecy”. The objective way to evaluate a score is to work every account in the same way for a period of time, paying no attention to scores. After a reasonable period of time (say 3 months) we would check the higher scores and see if we were able to collect a very high percentage of those accounts. If we find a significant number of paying accounts from within the lower scores, the model is flawed. Unfortunately, most companies can not afford to carry out such an objective test by working every account for a long period. Some form of compromise is required if we are to objectively evaluate the accuracy and value of the scores supplied by a company.

There is another challenge we face. Scores must have a relationship to collectability of the accounts. We can not give high scores to the highest-ranked accounts in every batch that is processed - Only accounts that are likely to pay should obtain a high score. Take an example of 20,000 new accounts. Suppose 4,000 of those accounts were collectable based on the model selected. We would expect those 4,000 accounts to have higher scores than the other 16,000 accounts. Now consider 20,000 different accounts. Suppose only 2,000 of them were collectable based on the same model. We would expect those 2,000 accounts to have similar scores to the other 4,000 in the first batch. In the second batch, we would expect the other 18,000 accounts to have lower scores similar to the 16,000 less collectable accounts in the first batch of accounts. The important conclusion is that for a given batch, we can not assume that the accounts with the highest scores are collectable. The scoring system must also provide an indication of what scores indicate a collectable account. The scoring system could provide the equivalent of a “Collection probability” flag or say that accounts with scores above X have the best probability of being collected. That score could vary depending on your location, the types of accounts you collect, your staff, management, the time of the year etc. etc. It will need to be evaluated and adjusted from time to time.


Quantrax’s scoring model

After a prolonged obsession with the optimization of automated collection processes, Quantrax, in 2005, set out to develop and evaluate its own scoring technology. Since Intelec is probably the finest work-flow engine for account management, a reliable score would greatly enhance a company’s ability to optimally manage its resources and its bottom line. The following were considered as Quantrax evaluated this project as a technical challenge and a potential business opportunity.

  • We wanted to find a solution that was better and could be priced competitively when evaluated against other solutions
  • For Quantrax to build a better solution, they would need a better approach. We felt that our knowledge of the debt collection industry was superior to that of other companies. Since we were solving a collection problem, this could be a great advantage for Quantrax.
  • Traditional approaches used credit history and information from external databases. We were not aware of many companies using the collection history that was already available within your company. We believed that there was great value to using your data to create a model for scoring new accounts.
  • Rather than building models based on specific factors that we know affect the collectability of an account (e.g. age at placement or balance), we believed that we would obtain better results by making no assumptions. We decided to start with all of the data that we knew could affect collectability. We would then let our model figure out exactly what circumstances could make an account more collectable. (E.g. Debtors between the ages of 35 and 40 in a specific area were more likely to pay a credit card bill than a medical bill, but the results would be different for similar debtors and accounts in a different area!)
  • One of the most powerful technologies that are able to analyze and determine these complex relationships between many variables is “neural network” technology (neural networks are based on the parallel architecture of animal brains). Quantrax’s model was totally neural network-based and used the finest mathematical modeling software that is available today.
  • Quantrax’s approach combines the advantages of data mining and neural network technologies.
  • The key to developing successful neural networks is the selection and quality of the information that is used to train the network. One of the keys to Quantrax’s success is the selection and manipulation of the data entities, as well as the application of neuro-fuzzy techniques to the learning processes.

In the summer of 2005, Quantrax designed a model based on the above. From different clients, we downloaded 150,000 to 300,000 older accounts and used about half those accounts to build a model. Within these accounts there were paying accounts, accounts from different areas, accounts that had linked to other accounts, profitable clients and clients with low recovery rates. After we built the model, we looked at the rest of the accounts, “hiding” any payment information from the process. We then used our model to try to predict which of the new accounts would have paid.

Here is an example of a test we ran against Intelec data. We looked at about 84,000 accounts and sorted them by what we predicted as most collectable down to the least collectable. The data is analyzed and presented in 20 groups. The scores for each group were lower, as you went down the list of categories. As you will see, the majority of the paying accounts (any money received) were in the higher groups. 85% of the paying accounts were in the top 50%. If you had only worked 60% of your accounts, you would have collected from 91% of the accounts that would have paid! With regard to dollars collected, 92% of the money collected was in the top 60%.

Category
Group
Size
Paying Accounts
A(%)
B(%)
Net Placement ($)
Current
Balance ($)
Total
Payments ($)
Fraction Paid
C(%)
Cum%
Paid
1
4,196
3,154
75.17
20.24
1,021,672.78
639,617.17
382,055.61
37
13.66
13.7
2
4,196
2,229
53.12
34.55
2,033,843.12
1,597,070.94
436,772.18
21
15.62
29.3
3
4,196
1,676
39.94
45.31
1,661,814.07
1,339,274.23
322,539.84
19
11.54
40.8
4
4,196
1,331
31.72
53.85
1,683,873.72
1,289,182.13
394,691.59
23
14.12
54.9
5
4,196
1,133
27.00
61.12
1,290,179.24
1,116,879.08
173,300.16
13
6.20
61.1
6
4,196
973
23.19
67.36
1,358,490.46
1,222,712.98
135,777.48
10
4.86
66.0
7
4,196
892
21.26
73.09
1,608,806.53
1,440,347.54
168,458.99
10
6.02
72.0
8
4,196
748
17.83
77.89
1,647,421.57
1,491,165.48
156,256.09
9
5.59
77.6
9
4,196
637
15.18
81.98
1,536,401.11
1,414,015.35
122,385.76
8
4.38
82.0
10
4,196
528
12.58
85.37
1,684,516.51
1,581,591.76
102,924.75
6
3.68
85.7
11
4,196
480
11.44
88.45
1,473,890.70
1,363,807.77
110,082.93
7
3.94
89.6
12
4,196
391
9.32
90.96
1,456,767.25
1,389,289.45
67,477.80
5
2.41
92.0
13
4,196
295
7.03
92.85
1,577,515.07
1,538,012.40
39,502.67
3
1.41
93.4
14
4,196
248
5.91
94.44
1,748,119.16
1,710,461.58
37,657.58
2
1.35
94.8
15
4,196
236
5.62
95.96
1,648,348.50
1,623,110.28
25,238.22
2
0.90
95.7
16
4,196
214
5.10
97.33
2,134,065.01
2,091,798.28
42,266.73
2
1.51
97.2
17
4,196
140
3.34
98.23
2,582,533.86
2,560,791.31
21,742.55
1
0.78
98.0
18
4,196
126
3.00
99.04
2,027,115.68
2,008,138.60
18,977.08
1
0.68
98.6
19
4,196
86
2.05
99.59
2,300,094.26
2,286,959.14
13,135.12
1
0.47
99.1
20
4,182
64
1.53
100.00
3,545,868.02
3,521,019.95
24,848.07
1
0.89
100.0
83,906
15,581
18.57
36,021,336.62
33,225,245.42
2,796,091.20
8


Net placement = Total Placements - Total adjustments (for the category)

A =Number of paid accounts / Total number of paid accounts (for the category)

B = Number of paying accounts within the above category / Total paying accounts (for all categories up to and including that category)

Fraction paid = Total payments for category / Net placements (for the category)

C = Total payments within the category / Total payment amount (for all categories)

Cum% paid (Cumulative % paid) = Percentage (dollars) paid as a total of all payments, for all categories up to and including that category.

"Paying accounts" refers to accounts where some money was collected, not necessarily a payment in full. Our reasoning was that if you were able to collect anything from a debtor, the ability to receive payment in-full is often based on the collector's skills. Identifying the accounts that could generate any size of payment was the key to the classification of accounts.


It is important to note the following.

  • The results were obtained with only a company's own data! No credit data was used.
  • The first 10 groups had 50% of 83,906 accounts that were analyzed.
  • The first 10 groups (the top 50%) contained over 85% of the accounts that received a payment.
  • That top 50% also had 85% of the dollars collected.
  • 92% of the dollars collected were within the first 12 groups, which represents 60% of the accounts analyzed.

In the above example, you could have collected 92% of accounts paid on, by working only 60% of the accounts placed! That knowledge could translate to faster recoveries at significantly lower operating costs. Remember that these are real accounts and not analysis or numbers made up by Quantrax. Most experts would agree that these are excellent results considering the fact that no credit scores were used. How does Quantrax explain these outstanding numbers?

We believe that the following explains some of Quantrax’s early success.

  • Scoring companies do not understand collection data and processes as well as Quantrax.

  • Quantrax’s approach used the correct combination of engineering, mathematics and collection industry knowledge and experience. The research and development teams were comprised of some uniquely talented individuals from the US and Europe, including the universities of Oxford and London.

  • Quantrax's models use several data elements that are not considered by the other scoring companies.

  • Quantrax's proprietary analysis and use of demographic data is quite unique and contributes significantly to the accuracy of the models.

  • Quantrax’s solution was entirely based on neural network technology.

    It is generally accepted that neural networks can not be effectively applied to random data or circumstances. As a result, Quantrax initially set out to understand if “paying accounts” were a random occurrence or something that could be predicted because they  fell into what is described in statistics as a “normal distribution”. The logical conclusion is that if all accounts were worked the same way, some and only some of them would pay. That leads us to believe that payment patterns and results obtained are not random. Quantrax was able to prove this, since its predictions closely matched the actual results obtained. Quantrax’s single-minded focus on neural network technology was more than justified under the circumstances.

  • The product was developed using the most advanced technology that is available to solve complex problems that require advanced mathematical modeling tools. For example, Quantrax’s modeling uses the same software environment that is used by every major aerospace and defense organization in the world to develop air, naval, land and space systems.

Even though collections may not be rocket science, Quantrax approached the scoring problem by using technology from the company that has its products applied to rocket science! The image shown (courtesy of NASA/JPL/Caltech) is related to NASA’s generation of fault protection code for the spacecraft named Deep Space 1, using different products from the same technology company. Quantrax’s approach and results can be summarized by the following quote from another user of the same family of products. “These tools put a whole new spin on the way the design process can be done." Jim McGowan, Lucent Technologies.

And what if you had the benefit of a “credit score”? A reliable credit score (based on the type of accounts being worked) can be utilized in different ways to enhance the value of the model. We tested our results by adding such a score into our models. The results showed that we were able to identify a more paying accounts with the use of credit scores within the model. Scoring is not only a matter of identifying accounts are most or least likely to pay. It is about doing that in the most economical manner. Your reasons for scoring should be the following.

  • Collect what is possible as fast as is possible
  • Spend the least amount on obtaining the scores
  • Spend the least amount on resources (people, technology and expenses such as letter generation) working the accounts placed

There are different strategies to incorporate a scoring product into your business plan. Clearly, if two companies offer scoring products, you would evaluate their products and pick the one that offered the best combination of results and costs. Quantrax offers you an unusual solution and one that does not use any credit scores. Based on the outstanding results we have consistently obtained, a practical plan would be use Quantrax for your initial scoring effort. The 25% - 40% of the accounts that are placed in the mid-level categories of the results would fall between the very collectable and least collectable accounts. Obtaining a quality "collection score" or even a FICO score on this group of accounts could result in some of those accounts moving to a higher group or being classified as less collectable. This information can further reduce the efforts needed to effectively work and manage all your accounts.

Note that results could vary for different areas, types of accounts and based on your work patterns. In general, if we can create a model using a good sample of historical data, we expect results that will be close to (within 5% of) what we have described in our examples.


How does Quantrax’s model predict what accounts are collectable?

To understand the manner in which Quantrax approached this problem, some basics about neural networks will be helpful. Neural networks can be applied in any circumstances where there is a possible relationship between inputs and outputs. In the collection industry, the inputs could be account characteristics, client-related variables and debtor demographics, while the output would be a score that reflected the chances of collecting any money from the debtor. In a complex application such as credit and collections, the relationships between all of the different variables are very complex, often very difficult to comprehensively articulate and only neural networks are sufficiently sophisticated to model these systems. For those with a mathematical background, neural networks are nonlinear (as opposed to being linear and less complex) and can work with very large numbers of variables.

  • Neural networks learn by example. Training algorithms are used to structure the data presented to the network. Based on the way the network is trained, you could subsequently try to predict results based on new information that would be presented to the network. While it may be very complex technology, artificial neural networks are created using concepts similar to those that allow the brain to perform extremely complex tasks. We use the term “artificial” to differentiate a computer-based neural network from a natural neural network such as the human brain. While the human brain is associated with “unsupervised” learning, computer-based neural networks can be cleverly manipulated as a result of the fact that they use “supervised” learning.

  • Quantrax initially trains a network based on a broad range of accounts and history stored on your system. Only basic account data is used (the information analyzed may be different, depending on the type of account).

  • Along with relevant client information, we also cleverly manipulate demographic information pertaining to the debtor.

  • We do not use credit report data or any credit scores from credit bureau or other source.

  • The scores provided are "absolute" and not "relative" values. This means that the accounts placed at the top of the list are not automatically assigned high values. If most of the accounts in a given batch are not considered collectable, then there will be only be a few accounts with higher scores.

Quantrax has always believed that not enough thought has gone into what a company should do with scores obtained on all of their new accounts. Traditionally, collection accounts have been classified based on balance. With reliable scoring models, we can work new accounts based on collectability and potential profitability, or a combination of those factors, which is often the better approach. New placements that link to existing accounts may or may not go through the scoring system. You will probably make that decision based on the date the existing accounts were last worked and if the debtor has made any payments.

A good scoring model will allow you to classify your accounts based on potential collectability. Good collection software will allow you to apply expensive or scarce resources based on the information supplied by the scores. Since Intelec is an expert system that can be rapidly modified to adjust for changes in your business plans, it is the perfect companion to a good scoring model.

There is one other key consideration as Quantrax looks to offering its scoring models to collection operations. Phone numbers are vital for contacting debtors. This in turn will increase the chances of collecting an account. Often, phone information on an account is missing or inaccurate. Quantrax will offer a phone number lookup and verification options as well as bankruptcy and deceased checking on accounts selected for scoring. The services will be priced based on the options selected. If a quality phone lookup is available, collectors will not be required to contact directory inquiries, adding several hours to their productive work time. Quantrax is also offering economical, high quality skip-tracing services that can further add to a collector’s productive phone time.

As we face more competition and reduced fees, a new, consolidated and comprehensive approach to your collection model offers great potential. It must be evaluated and considered along with all of the other changes you will contemplate in the short to long term. Considering the fact that your people are your most expensive resource, scoring and skip-tracing can offer very significant gains in productivity and considerable savings to any collection operation.

Why you must consider scoring every account on your system

Most companies have not looked at their “cost model” for several years. By “cost model” we are referring to the primary cost structure of working an account. You are almost always paid based on what you collect, but you pay for all effort and activity regardless of whether it results in payments being generated or not. Your primary costs are associated with:

  • Letters sent, and paying for forwarding information
  • Phone calls connected, whether it is the right party or not (the cost of a phone call)
  • The cost of “working” an account (the value of a collector’s time when debtors are contacted, or any account-related contact takes place, right party or not)
  • Skip-tracing costs – the costs associated with different types of services, batched or interactive and the value of individual’s time on any work related to skip-tracing
  • Costs for obtaining phone numbers (Paying for electronic or other services as well as the value of an individual’s time when applicable. E.g. directory assistance)

Most of the above costs will apply to every account that you work. That includes the 80% you do not collect! As an example, sending an extra letter costs about 50 cents. Talking to someone who is unlikely to pay not only wastes money, but also uses scarce resources that could be applied to a more collectable account. Skip-tracing services usually charge by the look-up as do phone number services. How do you reduce the costs for these necessary services? Any quality scoring model will help you to spend less on every account with a lower collectability. Since 80% of accounts in bad debt are never collected, the savings are potentially large and very significant.

A combination of an early phone look-up, bankruptcy and deceased check, along with a reliable collection score, gives you an opportunity to re-engineer your work plans with a view to being more efficient with less people and while spending less. One may choose to not score certain types of accounts or accounts that link to others that are paying or are actively being worked. By scoring every other account that is placed, you have the opportunity for significant savings that will greatly exceed the fixed, one-time cost of scoring every account.


REFERENCES ON NEURAL NETWORKS

There are many papers and documentation about neural networks, the technology that Quantrax has extensively used in its scoring model. Some interesting background on neural networks can be found at the following sites.

http://www.doc.ic.ac.uk/~nd/surprise_96/journal/vol4/cs11/report.html#Contents

http://www.cs.stir.ac.uk/~lss/NNIntro/InvSlides.html


Quantrax Corporation Inc.
4300 Montgomery Avenue – Suite 106
Bethesda, MD    20814

www.quantrax.com

© Quantrax Corporation, Inc. 2006