Results, audit and tests
Results of the two models
| Metric | Value |
|---|---|
| Eligibility model (2016 hold-out) | |
| ROC AUC | 0.9097 |
| Accuracy / precision / recall at 50% | 0.885 / 0.831 / 0.966 |
| Agreement with rule FICO ≥ 660, DTI ≤ 35, loan < $40,000 | 99.3% |
| Accuracy of that rule on its own | 0.885 |
| Charge-off model (2015 test) | |
| ROC AUC | 0.6265 |
| Mean predicted vs actual default rate | 13.9% vs 14.8% |
| Brier score: model vs base rate | 0.1232 vs 0.1265 |
| Largest calibration gap across deciles | 1.9 percentage points |
The eligibility model has learned LendingClub's rule. A simple rule (FICO at least 660, DTI at most 35, loan below $40,000) scores an accuracy of 0.885 on its own, and the model agrees with it on 99.3% of applications. For a $15,000 loan with DTI 15, the score jumps from 27.0 at FICO 659 to 82.9 at FICO 660. It applies the rule consistently, but makes no independent judgment.
The charge-off model ranks modestly but its probabilities are good. An AUC of 0.626 means that of one charged-off and one repaid loan picked at random, it gives the charged-off one the higher risk about 63% of the time. Predicted and actual default rates rise together across the ten groups, from 5.7% predicted and 5.4% actual in the safest group to 23.2% and 25.1% in the riskiest. It slightly underestimates risk for the riskiest loans, by at most 1.9 points.
Predicted and actual default rate in each risk group
2015 test loans sorted into ten equal groups by predicted risk (1 = lowest)
- Predicted
- Observed
Where the charge-off limit comes from
| Algorithm (2015 test) | AUC |
|---|---|
| Gradient boosting | 0.6283 |
| Random forest | 0.6265 |
| Histogram gradient boosting (LendWise) | 0.6265 |
| Extra trees | 0.6258 |
| Logistic regression | 0.6247 |
| Neural network (MLP) | 0.6244 |
| Linear discriminant analysis | 0.6238 |
| Decision tree | 0.6219 |
| AdaBoost | 0.6217 |
| k-nearest neighbours (k=500, 60,000-loan sample) | 0.6216 |
| Quadratic discriminant analysis | 0.6188 |
| Gaussian naive Bayes | 0.6174 |
| Group | Loans | Default rate | AUC |
|---|---|---|---|
| FICO 660 to 679 | 37,804 | 19.3% | 0.5725 |
| FICO 680 to 699 | 26,529 | 15.8% | 0.5732 |
| FICO 700 to 719 | 17,574 | 11.8% | 0.5665 |
| FICO 720 to 749 | 11,368 | 8.3% | 0.5726 |
| FICO 750 to 850 | 6,725 | 5.1% | 0.5889 |
| Loan amount $0 to $4,999 | 11,456 | 15.2% | 0.5858 |
| Loan amount $5,000 to $14,999 | 53,963 | 15.4% | 0.6225 |
| Loan amount $15,000 to $24,999 | 24,088 | 14.0% | 0.6433 |
| Loan amount $25,000 to $40,000 | 10,493 | 13.2% | 0.6415 |
| DTI 0 to 9.99 | 17,689 | 11.2% | 0.6002 |
| DTI 10 to 19.99 | 40,975 | 13.2% | 0.6155 |
| DTI 20 to 29.99 | 30,611 | 16.8% | 0.6055 |
| DTI 30 to 100 | 10,725 | 21.2% | 0.5981 |
AUC is lowest inside FICO bands, which makes sense because FICO hardly changes inside a band. The average loss on a charged-off loan is 50.3%; a regression model for severity changed R² by only 0.003, so the average is used.
Comparison of variants
| Variant | Eligibility AUC | Jumpy | Charge-off AUC | Jumpy |
|---|---|---|---|---|
| A Baseline (no constraints) | 0.9843 | 5.4% | 0.6269 | 4.7% |
| B Monotonic constraints only | 0.9098 | 1.3% | 0.6264 | 3.1% |
| C Constraints + regularisation (chosen) | 0.9097 | 0.7% | 0.6265 | 0.2% |
| D Heavier regularisation | 0.9086 | 0.4% | 0.6259 | 0.0% |
Behaviour checks: each of the three inputs is swept across its range at four reference profiles, for both models, which gives 24 checks. A check fails if the score moves against the expected direction by more than 1 point. The unconstrained baseline (A) failed 24 of them and the constrained models (C) failed 0. A risk estimate that rises when the loan gets smaller would be very hard to justify to anyone.
Audit of the 0.99 AUC
The first eligibility model scored an AUC of 0.992. Retraining it with different inputs shows where that came from. Both files were already matched to 2016, so the time period was not the reason.
Employment length on its own reaches 0.9248, because 90.8% of rejected 2016 records say "< 1 year" against 7.4% of accepted loans. FICO alone reaches 0.9556, because the lender's screening is close to a fixed rule. Loan purpose added only 0.000.
FICO is recorded differently as well. Every accepted loan stores the low end of a 5-point range (660, 665 and so on), so 100% of accepted FICO values end in 0 or 5, against 19.5% of rejected ones. Without constraints the model uses this: a $5,000 loan with DTI 10 scores 88.2 at FICO 700 but 56.6 at FICO 701. The monotonic constraint rules this out.
The same three inputs without constraints reach 0.9814; the constrained, regularised model reaches 0.9097.
Eligibility AUC with different sets of inputs
First, unconstrained model on the 2016 cohort. Employment length is highlighted.
Acceptance tests
| ID | Input or action | Expected | Actual | Status |
|---|---|---|---|---|
| TC-01 | Loan 15000, FICO 700, DTI 15 | Both outputs within 0 to 100 | Eligibility 86.0, PD 10.34%, approve | Pass |
| TC-02 | FICO = 659.5 | Rejected: FICO must be a whole number | FICO score must be a whole number. | Pass |
| TC-03 | DTI = "18%" | Accepted as 18 | Accepted as 18 | Pass |
| TC-04 | DTI = True | Rejected as invalid type | DTI must be a number, not true or false. | Pass |
| TC-05 | Loan amount = 45000 | Rejected: out of range | Loan amount must be between $500 and $40,000. | Pass |
| TC-06 | Blank, NaN or infinity | One error message | Loan amount is required. | Pass |
| TC-07 | Missing input key | Validation error, no crash | DTI is missing. | Pass |
| TC-08 | FICO 659 vs 660 (loan 15000, DTI 15) | Eligibility falls below 40 at 659 | Eligibility 27.0 vs 82.9 | Pass |
| TC-09 | Save, reload and score again | Same outputs | Identical on 2,000 inputs | Pass |
| TC-10 | 200 quick submissions | No old results left on screen | 200 concurrent requests matched to their own inputs; web form state test passed | Pass |
The one warning is expected: the raw models accept NaN and infinity without complaining, so the validator must always sit in front of them, and it does.
Input validation25 of 25 passed
- loan_amount: rejects blank values4 values rejected
- loan_amount: rejects boolean values2 values rejected
- loan_amount: rejects not finite values8 values rejected
- loan_amount: rejects text values11 values rejected
- loan_amount: rejects wrong type values6 values rejected
- loan_amount: rejects out-of-range values8 values rejected
- fico: rejects blank values4 values rejected
- fico: rejects boolean values2 values rejected
- fico: rejects not finite values8 values rejected
- fico: rejects text values11 values rejected
- fico: rejects wrong type values6 values rejected
- fico: rejects out-of-range values6 values rejected
- dti: rejects blank values4 values rejected
- dti: rejects boolean values2 values rejected
- dti: rejects not finite values8 values rejected
- dti: rejects text values11 values rejected
- dti: rejects wrong type values6 values rejected
- dti: rejects out-of-range values7 values rejected
- fico: rejects values that are not whole numbers (TC-02)4 values rejected
- dti: accepts an optional % sign (TC-03)"18%" -> 18
- loan amount accepts numeric text"$15,000.50" -> 15000.5
- boundary values are accepted500/300/0 and 40,000/850/100 accepted
- missing keys give a validation error, not a crash (TC-07)3 keys checked
- wrong containers are rejected5 containers rejected
- extra keys are ignoredok
Saved models4 of 4 passed
- artifacts load and metadata is completelendwise-hgb-2026.11.19
- save, reload and score again gives the same outputs (TC-09)identical on 2,000 inputs
- monotonic constraints are stored in both models[-1, +1, -1] and [+1, -1, +1]
- both models use 3-fold sigmoid calibration3 calibrated folds each
Behaviour8 of 8 passed
- eligibility score stays within 0 to 100 on 18,720 inputs (TC-01)min 0.01, max 86.01
- default probability stays within 0 to 100 on 18,720 inputs (TC-01)min 2.85, max 31.47
- eligibility: grid never moves against the loan amount constraint18,252 neighbouring pairs, 0 violations
- eligibility: grid never moves against the FICO constraint18,200 neighbouring pairs, 0 violations
- eligibility: grid never moves against the DTI constraint17,280 neighbouring pairs, 0 violations
- charge-off: grid never moves against the loan amount constraint18,252 neighbouring pairs, 0 violations
- charge-off: grid never moves against the FICO constraint18,200 neighbouring pairs, 0 violations
- charge-off: grid never moves against the DTI constraint17,280 neighbouring pairs, 0 violations
Behaviour checks (24)24 of 24 passed
- eligibility: score can only fall with Loan amount (FICO score 700, DTI 10)never moves the wrong way
- eligibility: score can only fall with Loan amount (FICO score 680, DTI 20)never moves the wrong way
- eligibility: score can only fall with Loan amount (FICO score 720, DTI 25)never moves the wrong way
- eligibility: score can only fall with Loan amount (FICO score 760, DTI 30)never moves the wrong way
- eligibility: score can only rise with FICO score (Loan amount 5,000, DTI 10)never moves the wrong way
- eligibility: score can only rise with FICO score (Loan amount 15,000, DTI 20)never moves the wrong way
- eligibility: score can only rise with FICO score (Loan amount 25,000, DTI 25)never moves the wrong way
- eligibility: score can only rise with FICO score (Loan amount 35,000, DTI 30)never moves the wrong way
- eligibility: score can only fall with DTI (Loan amount 5,000, FICO score 700)never moves the wrong way
- eligibility: score can only fall with DTI (Loan amount 15,000, FICO score 680)never moves the wrong way
- eligibility: score can only fall with DTI (Loan amount 25,000, FICO score 720)never moves the wrong way
- eligibility: score can only fall with DTI (Loan amount 35,000, FICO score 760)never moves the wrong way
- charge-off: score can only rise with Loan amount (FICO score 700, DTI 10)never moves the wrong way
- charge-off: score can only rise with Loan amount (FICO score 680, DTI 20)never moves the wrong way
- charge-off: score can only rise with Loan amount (FICO score 720, DTI 25)never moves the wrong way
- charge-off: score can only rise with Loan amount (FICO score 760, DTI 30)never moves the wrong way
- charge-off: score can only fall with FICO score (Loan amount 5,000, DTI 10)never moves the wrong way
- charge-off: score can only fall with FICO score (Loan amount 15,000, DTI 20)never moves the wrong way
- charge-off: score can only fall with FICO score (Loan amount 25,000, DTI 25)never moves the wrong way
- charge-off: score can only fall with FICO score (Loan amount 35,000, DTI 30)never moves the wrong way
- charge-off: score can only rise with DTI (Loan amount 5,000, FICO score 700)never moves the wrong way
- charge-off: score can only rise with DTI (Loan amount 15,000, FICO score 680)never moves the wrong way
- charge-off: score can only rise with DTI (Loan amount 25,000, FICO score 720)never moves the wrong way
- charge-off: score can only rise with DTI (Loan amount 35,000, FICO score 760)never moves the wrong way
Policy9 of 9 passed
- loss severity matches the training data (about 50.3%)50.32%
- expected-loss thresholds are 11.07% and 6.04%reject above 11.07%, approve up to 6.04%
- eligibility below 40 is always rejected18 combinations
- expected loss above 11.07% is rejectedPD 22.01% rejected, 22.00% not
- a range-guard flag sends the case to review instead of approvalok
- approve only when eligibility >= 75 and expected loss <= 6.04%edges checked
- every decision on the 18,720-input grid follows the rulesapprove 2,249, review 3,407, reject 13,064
- tiers match the decision thresholdsLikely/Borderline/Unlikely and Low/Moderate/High
- charge-off risk is hidden when eligibility is Unlikelyok
Consistency5 of 5 passed
- eligibility: single and batch scores are identical300 rows
- charge-off: single and batch scores are identical300 rows
- range guard: single and batch flags are identical300 rows
- the interface result matches the batch scores60 rows
- repeated scoring is deterministicgrid scored twice
Extreme cases10 of 11 passed
- corner $500 / FICO 300 / DTI 0 gives a valid resultreject, eligibility 0.0
- corner $500 / FICO 300 / DTI 100 gives a valid resultreject, eligibility 0.0
- corner $500 / FICO 850 / DTI 0 gives a valid resultreview, eligibility 86.0
- corner $500 / FICO 850 / DTI 100 gives a valid resultreject, eligibility 3.2
- corner $40,000 / FICO 300 / DTI 0 gives a valid resultreject, eligibility 0.0
- corner $40,000 / FICO 300 / DTI 100 gives a valid resultreject, eligibility 0.0
- corner $40,000 / FICO 850 / DTI 0 gives a valid resultreject, eligibility 37.4
- corner $40,000 / FICO 850 / DTI 100 gives a valid resultreject, eligibility 1.8
- FICO 659 vs 660 shows LendingClub's cut-off (TC-08)27.0 vs 82.9
- high-FICO, low-DTI, small loan is approvedeligibility 86.0, PD 4.65%
- the raw models accept NaN and infinity, so the validator must sit in frontmodels returned scores for NaN/inf inputs; only the validator stops them
Stability2 of 2 passed
- eligibility: at most 1% of 551 profiles jump by more than 10 points0.7% of 551 profiles
- charge-off: at most 1% of 551 profiles jump by more than 3 points0.2% of 551 profiles
Reported metrics10 of 10 passed
- eligibility hold-out AUC matches the reported value0.9097
- eligibility accuracy, precision and recall at 50% match0.885 / 0.831 / 0.966
- eligibility agrees with the lender rule on at least 99% of applications99.3%
- charge-off AUC on the 2015 test year matches the reported value0.6265
- mean predicted default rate is within 2 points of the observed rate13.9% vs 14.8%
- Brier score beats the base-rate forecast0.1232 vs 0.1265
- no risk decile is off by more than 3 percentage pointslargest gap 1.9 points
- observed default rate rises across the risk deciles5.4% to 25.1%
- the risk model is trained on 2012 to 2014 and tested on 2015 only300,000 train, 100,000 test
- eligibility data is a 50/50 sample of 2016 with an 80:20 split100,000 + 100,000; 160,000 / 40,000
Range guard4 of 4 passed
- limits are the 0.1st and 99.9th percentiles of the training loansloan_amount 1000 to 35000; fico 660 to 825; dti 0.33 to 39.17
- inputs outside the training range are flagged$35,001 flagged, $15,000 not
- the Isolation Forest flags about 1% of training loans or fewer0.74%
- a flag never causes a rejection on its own2,263 flagged grid inputs went to review
Counter-offer3 of 3 passed
- offers are smaller, in $500 steps, and approved by the rules2 offers checked
- the offer is the largest approved amount$1,500 approved, $2,000 not
- approved applications get no counter-offerok
Interface13 of 13 passed
- TC-01: a valid application returns both outputs within 0 to 100eligibility 86.0, PD 10.34%
- TC-02: the API answers fico errors with one message and no scoresFICO score must be a whole number.
- TC-04: the API answers dti errors with one message and no scoresDTI must be a number, not true or false.
- TC-05: the API answers loan_amount errors with one message and no scoresLoan amount must be between $500 and $40,000.
- TC-06: the API answers loan_amount errors with one message and no scoresLoan amount is required.
- TC-07: the API answers dti errors with one message and no scoresDTI is missing.
- TC-08: the API shows the FICO cut-off659 reject, 660 likely
- TC-10: 200 quick submissions each get their own result200 concurrent requests matched to their own inputs; web form state test passed
- an application can be submitted, tracked and decided by an officerLW-2026-000001 approved by the officer
- a malformed request body is rejected with one messageSome fields are invalid. Please review and try again.
- officer pages require a login401 without a valid token
- median response time is under 50 msmedian 19.1 ms over 200 requests
- health check reports the models as livelendwise-hgb-2026.11.19
Comparison
| Aspect | Common practice | LendWise |
|---|---|---|
| Inputs | Many, sometimes from after the loan | Three application-time inputs |
| Data split | Random | By year for the risk model |
| Class imbalance | Oversampling (SMOTE) | Natural rate with calibration |
| Behaviour checks | Rarely done | Monotonic constraints and nudge tests |
| Data audit | Rarely done | Audit of the 0.99 AUC |
| Delivered as | Model only | Validator, rules, website and test suite |
| Feature | Manual screening | LendWise |
|---|---|---|
| Speed | Minutes to days | Milliseconds per application |
| Consistency | Can differ between officers | Same input always gives the same output |
| Risk estimate | Based on judgment | Calibrated default probability |
| Counter-offer | Worked out by hand | Automatic search in $500 steps |
| Input checking | Manual | Validator and range guard |