Skip to content

Results, audit and tests

Every number on this page is computed by the project's scripts from the saved models and the LendingClub data, and is served live from the model card.

Results of the two models

Eligibility is measured on a random 20% hold-out of the 2016 sample; charge-off on loans from 2015, a year the model never saw.
Results of the two models
MetricValue
Eligibility model (2016 hold-out)
ROC AUC0.9097
Accuracy / precision / recall at 50%0.885 / 0.831 / 0.966
Agreement with rule FICO ≥ 660, DTI ≤ 35, loan < $40,00099.3%
Accuracy of that rule on its own0.885
Charge-off model (2015 test)
ROC AUC0.6265
Mean predicted vs actual default rate13.9% vs 14.8%
Brier score: model vs base rate0.1232 vs 0.1265
Largest calibration gap across deciles1.9 percentage points

The eligibility model has learned LendingClub's rule. A simple rule (FICO at least 660, DTI at most 35, loan below $40,000) scores an accuracy of 0.885 on its own, and the model agrees with it on 99.3% of applications. For a $15,000 loan with DTI 15, the score jumps from 27.0 at FICO 659 to 82.9 at FICO 660. It applies the rule consistently, but makes no independent judgment.

The charge-off model ranks modestly but its probabilities are good. An AUC of 0.626 means that of one charged-off and one repaid loan picked at random, it gives the charged-off one the higher risk about 63% of the time. Predicted and actual default rates rise together across the ten groups, from 5.7% predicted and 5.4% actual in the safest group to 23.2% and 25.1% in the riskiest. It slightly underestimates risk for the riskiest loans, by at most 1.9 points.

Predicted and actual default rate in each risk group

2015 test loans sorted into ten equal groups by predicted risk (1 = lowest)

  • Predicted
  • Observed

Where the charge-off limit comes from

Twelve algorithms on the same three inputs all land between 0.6174 and 0.6283. Adding income, credit history and other bureau fields raises the AUC to 0.6684 with the same method. The limit comes from the inputs, not the algorithm.
Charge-off AUC by algorithm
Algorithm (2015 test)AUC
Gradient boosting0.6283
Random forest0.6265
Histogram gradient boosting (LendWise)0.6265
Extra trees0.6258
Logistic regression0.6247
Neural network (MLP)0.6244
Linear discriminant analysis0.6238
Decision tree0.6219
AdaBoost0.6217
k-nearest neighbours (k=500, 60,000-loan sample)0.6216
Quadratic discriminant analysis0.6188
Gaussian naive Bayes0.6174
Charge-off AUC inside smaller groups
GroupLoansDefault rateAUC
FICO 660 to 67937,80419.3%0.5725
FICO 680 to 69926,52915.8%0.5732
FICO 700 to 71917,57411.8%0.5665
FICO 720 to 74911,3688.3%0.5726
FICO 750 to 8506,7255.1%0.5889
Loan amount $0 to $4,99911,45615.2%0.5858
Loan amount $5,000 to $14,99953,96315.4%0.6225
Loan amount $15,000 to $24,99924,08814.0%0.6433
Loan amount $25,000 to $40,00010,49313.2%0.6415
DTI 0 to 9.9917,68911.2%0.6002
DTI 10 to 19.9940,97513.2%0.6155
DTI 20 to 29.9930,61116.8%0.6055
DTI 30 to 10010,72521.2%0.5981

AUC is lowest inside FICO bands, which makes sense because FICO hardly changes inside a band. The average loss on a charged-off loan is 50.3%; a regression model for severity changed R² by only 0.003, so the average is used.

Comparison of variants

Constraints and regularisation trade a little eligibility AUC for far fewer jumpy profiles: profiles whose score moves by more than 10 points (eligibility) or 3 points (risk) when the loan amount moves by 1%, FICO by 1 point or DTI by 0.5. Variant C is the one LendWise uses.
Comparison of variants
VariantEligibility AUCJumpyCharge-off AUCJumpy
A Baseline (no constraints)0.98435.4%0.62694.7%
B Monotonic constraints only0.90981.3%0.62643.1%
C Constraints + regularisation (chosen)0.90970.7%0.62650.2%
D Heavier regularisation0.90860.4%0.62590.0%

Behaviour checks: each of the three inputs is swept across its range at four reference profiles, for both models, which gives 24 checks. A check fails if the score moves against the expected direction by more than 1 point. The unconstrained baseline (A) failed 24 of them and the constrained models (C) failed 0. A risk estimate that rises when the loan gets smaller would be very hard to justify to anyone.

Audit of the 0.99 AUC

The first eligibility model scored an AUC of 0.992. Retraining it with different inputs shows where that came from. Both files were already matched to 2016, so the time period was not the reason.

Employment length on its own reaches 0.9248, because 90.8% of rejected 2016 records say "< 1 year" against 7.4% of accepted loans. FICO alone reaches 0.9556, because the lender's screening is close to a fixed rule. Loan purpose added only 0.000.

FICO is recorded differently as well. Every accepted loan stores the low end of a 5-point range (660, 665 and so on), so 100% of accepted FICO values end in 0 or 5, against 19.5% of rejected ones. Without constraints the model uses this: a $5,000 loan with DTI 10 scores 88.2 at FICO 700 but 56.6 at FICO 701. The monotonic constraint rules this out.

The same three inputs without constraints reach 0.9814; the constrained, regularised model reaches 0.9097.

Eligibility AUC with different sets of inputs

First, unconstrained model on the 2016 cohort. Employment length is highlighted.

Acceptance tests

The acceptance suite runs on the saved models and the real interface code. It checks 126 bad input values, a grid of 18,720 valid inputs for monotonic behaviour, the policy thresholds, single vs batch scoring, extreme cases, stability, the reported metrics and end-to-end runs of the interface. Last run Nov 20, 2026, 01:29 AM: 117 passed, 0 failed and 1 warning, in 17 seconds.
Test cases
IDInput or actionExpectedActualStatus
TC-01Loan 15000, FICO 700, DTI 15Both outputs within 0 to 100Eligibility 86.0, PD 10.34%, approvePass
TC-02FICO = 659.5Rejected: FICO must be a whole numberFICO score must be a whole number.Pass
TC-03DTI = "18%"Accepted as 18Accepted as 18Pass
TC-04DTI = TrueRejected as invalid typeDTI must be a number, not true or false.Pass
TC-05Loan amount = 45000Rejected: out of rangeLoan amount must be between $500 and $40,000.Pass
TC-06Blank, NaN or infinityOne error messageLoan amount is required.Pass
TC-07Missing input keyValidation error, no crashDTI is missing.Pass
TC-08FICO 659 vs 660 (loan 15000, DTI 15)Eligibility falls below 40 at 659Eligibility 27.0 vs 82.9Pass
TC-09Save, reload and score againSame outputsIdentical on 2,000 inputsPass
TC-10200 quick submissionsNo old results left on screen200 concurrent requests matched to their own inputs; web form state test passedPass

The one warning is expected: the raw models accept NaN and infinity without complaining, so the validator must always sit in front of them, and it does.

Input validation25 of 25 passed
  • loan_amount: rejects blank values4 values rejected
  • loan_amount: rejects boolean values2 values rejected
  • loan_amount: rejects not finite values8 values rejected
  • loan_amount: rejects text values11 values rejected
  • loan_amount: rejects wrong type values6 values rejected
  • loan_amount: rejects out-of-range values8 values rejected
  • fico: rejects blank values4 values rejected
  • fico: rejects boolean values2 values rejected
  • fico: rejects not finite values8 values rejected
  • fico: rejects text values11 values rejected
  • fico: rejects wrong type values6 values rejected
  • fico: rejects out-of-range values6 values rejected
  • dti: rejects blank values4 values rejected
  • dti: rejects boolean values2 values rejected
  • dti: rejects not finite values8 values rejected
  • dti: rejects text values11 values rejected
  • dti: rejects wrong type values6 values rejected
  • dti: rejects out-of-range values7 values rejected
  • fico: rejects values that are not whole numbers (TC-02)4 values rejected
  • dti: accepts an optional % sign (TC-03)"18%" -> 18
  • loan amount accepts numeric text"$15,000.50" -> 15000.5
  • boundary values are accepted500/300/0 and 40,000/850/100 accepted
  • missing keys give a validation error, not a crash (TC-07)3 keys checked
  • wrong containers are rejected5 containers rejected
  • extra keys are ignoredok
Saved models4 of 4 passed
  • artifacts load and metadata is completelendwise-hgb-2026.11.19
  • save, reload and score again gives the same outputs (TC-09)identical on 2,000 inputs
  • monotonic constraints are stored in both models[-1, +1, -1] and [+1, -1, +1]
  • both models use 3-fold sigmoid calibration3 calibrated folds each
Behaviour8 of 8 passed
  • eligibility score stays within 0 to 100 on 18,720 inputs (TC-01)min 0.01, max 86.01
  • default probability stays within 0 to 100 on 18,720 inputs (TC-01)min 2.85, max 31.47
  • eligibility: grid never moves against the loan amount constraint18,252 neighbouring pairs, 0 violations
  • eligibility: grid never moves against the FICO constraint18,200 neighbouring pairs, 0 violations
  • eligibility: grid never moves against the DTI constraint17,280 neighbouring pairs, 0 violations
  • charge-off: grid never moves against the loan amount constraint18,252 neighbouring pairs, 0 violations
  • charge-off: grid never moves against the FICO constraint18,200 neighbouring pairs, 0 violations
  • charge-off: grid never moves against the DTI constraint17,280 neighbouring pairs, 0 violations
Behaviour checks (24)24 of 24 passed
  • eligibility: score can only fall with Loan amount (FICO score 700, DTI 10)never moves the wrong way
  • eligibility: score can only fall with Loan amount (FICO score 680, DTI 20)never moves the wrong way
  • eligibility: score can only fall with Loan amount (FICO score 720, DTI 25)never moves the wrong way
  • eligibility: score can only fall with Loan amount (FICO score 760, DTI 30)never moves the wrong way
  • eligibility: score can only rise with FICO score (Loan amount 5,000, DTI 10)never moves the wrong way
  • eligibility: score can only rise with FICO score (Loan amount 15,000, DTI 20)never moves the wrong way
  • eligibility: score can only rise with FICO score (Loan amount 25,000, DTI 25)never moves the wrong way
  • eligibility: score can only rise with FICO score (Loan amount 35,000, DTI 30)never moves the wrong way
  • eligibility: score can only fall with DTI (Loan amount 5,000, FICO score 700)never moves the wrong way
  • eligibility: score can only fall with DTI (Loan amount 15,000, FICO score 680)never moves the wrong way
  • eligibility: score can only fall with DTI (Loan amount 25,000, FICO score 720)never moves the wrong way
  • eligibility: score can only fall with DTI (Loan amount 35,000, FICO score 760)never moves the wrong way
  • charge-off: score can only rise with Loan amount (FICO score 700, DTI 10)never moves the wrong way
  • charge-off: score can only rise with Loan amount (FICO score 680, DTI 20)never moves the wrong way
  • charge-off: score can only rise with Loan amount (FICO score 720, DTI 25)never moves the wrong way
  • charge-off: score can only rise with Loan amount (FICO score 760, DTI 30)never moves the wrong way
  • charge-off: score can only fall with FICO score (Loan amount 5,000, DTI 10)never moves the wrong way
  • charge-off: score can only fall with FICO score (Loan amount 15,000, DTI 20)never moves the wrong way
  • charge-off: score can only fall with FICO score (Loan amount 25,000, DTI 25)never moves the wrong way
  • charge-off: score can only fall with FICO score (Loan amount 35,000, DTI 30)never moves the wrong way
  • charge-off: score can only rise with DTI (Loan amount 5,000, FICO score 700)never moves the wrong way
  • charge-off: score can only rise with DTI (Loan amount 15,000, FICO score 680)never moves the wrong way
  • charge-off: score can only rise with DTI (Loan amount 25,000, FICO score 720)never moves the wrong way
  • charge-off: score can only rise with DTI (Loan amount 35,000, FICO score 760)never moves the wrong way
Policy9 of 9 passed
  • loss severity matches the training data (about 50.3%)50.32%
  • expected-loss thresholds are 11.07% and 6.04%reject above 11.07%, approve up to 6.04%
  • eligibility below 40 is always rejected18 combinations
  • expected loss above 11.07% is rejectedPD 22.01% rejected, 22.00% not
  • a range-guard flag sends the case to review instead of approvalok
  • approve only when eligibility >= 75 and expected loss <= 6.04%edges checked
  • every decision on the 18,720-input grid follows the rulesapprove 2,249, review 3,407, reject 13,064
  • tiers match the decision thresholdsLikely/Borderline/Unlikely and Low/Moderate/High
  • charge-off risk is hidden when eligibility is Unlikelyok
Consistency5 of 5 passed
  • eligibility: single and batch scores are identical300 rows
  • charge-off: single and batch scores are identical300 rows
  • range guard: single and batch flags are identical300 rows
  • the interface result matches the batch scores60 rows
  • repeated scoring is deterministicgrid scored twice
Extreme cases10 of 11 passed
  • corner $500 / FICO 300 / DTI 0 gives a valid resultreject, eligibility 0.0
  • corner $500 / FICO 300 / DTI 100 gives a valid resultreject, eligibility 0.0
  • corner $500 / FICO 850 / DTI 0 gives a valid resultreview, eligibility 86.0
  • corner $500 / FICO 850 / DTI 100 gives a valid resultreject, eligibility 3.2
  • corner $40,000 / FICO 300 / DTI 0 gives a valid resultreject, eligibility 0.0
  • corner $40,000 / FICO 300 / DTI 100 gives a valid resultreject, eligibility 0.0
  • corner $40,000 / FICO 850 / DTI 0 gives a valid resultreject, eligibility 37.4
  • corner $40,000 / FICO 850 / DTI 100 gives a valid resultreject, eligibility 1.8
  • FICO 659 vs 660 shows LendingClub's cut-off (TC-08)27.0 vs 82.9
  • high-FICO, low-DTI, small loan is approvedeligibility 86.0, PD 4.65%
  • the raw models accept NaN and infinity, so the validator must sit in frontmodels returned scores for NaN/inf inputs; only the validator stops them
Stability2 of 2 passed
  • eligibility: at most 1% of 551 profiles jump by more than 10 points0.7% of 551 profiles
  • charge-off: at most 1% of 551 profiles jump by more than 3 points0.2% of 551 profiles
Reported metrics10 of 10 passed
  • eligibility hold-out AUC matches the reported value0.9097
  • eligibility accuracy, precision and recall at 50% match0.885 / 0.831 / 0.966
  • eligibility agrees with the lender rule on at least 99% of applications99.3%
  • charge-off AUC on the 2015 test year matches the reported value0.6265
  • mean predicted default rate is within 2 points of the observed rate13.9% vs 14.8%
  • Brier score beats the base-rate forecast0.1232 vs 0.1265
  • no risk decile is off by more than 3 percentage pointslargest gap 1.9 points
  • observed default rate rises across the risk deciles5.4% to 25.1%
  • the risk model is trained on 2012 to 2014 and tested on 2015 only300,000 train, 100,000 test
  • eligibility data is a 50/50 sample of 2016 with an 80:20 split100,000 + 100,000; 160,000 / 40,000
Range guard4 of 4 passed
  • limits are the 0.1st and 99.9th percentiles of the training loansloan_amount 1000 to 35000; fico 660 to 825; dti 0.33 to 39.17
  • inputs outside the training range are flagged$35,001 flagged, $15,000 not
  • the Isolation Forest flags about 1% of training loans or fewer0.74%
  • a flag never causes a rejection on its own2,263 flagged grid inputs went to review
Counter-offer3 of 3 passed
  • offers are smaller, in $500 steps, and approved by the rules2 offers checked
  • the offer is the largest approved amount$1,500 approved, $2,000 not
  • approved applications get no counter-offerok
Interface13 of 13 passed
  • TC-01: a valid application returns both outputs within 0 to 100eligibility 86.0, PD 10.34%
  • TC-02: the API answers fico errors with one message and no scoresFICO score must be a whole number.
  • TC-04: the API answers dti errors with one message and no scoresDTI must be a number, not true or false.
  • TC-05: the API answers loan_amount errors with one message and no scoresLoan amount must be between $500 and $40,000.
  • TC-06: the API answers loan_amount errors with one message and no scoresLoan amount is required.
  • TC-07: the API answers dti errors with one message and no scoresDTI is missing.
  • TC-08: the API shows the FICO cut-off659 reject, 660 likely
  • TC-10: 200 quick submissions each get their own result200 concurrent requests matched to their own inputs; web form state test passed
  • an application can be submitted, tracked and decided by an officerLW-2026-000001 approved by the officer
  • a malformed request body is rejected with one messageSome fields are invalid. Please review and try again.
  • officer pages require a login401 without a valid token
  • median response time is under 50 msmedian 19.1 ms over 200 requests
  • health check reports the models as livelendwise-hgb-2026.11.19

Comparison

LendWise does not claim to be more accurate than published models. Many report very high accuracy on LendingClub data, but they predict something slightly different, split the data at random and use fields only known after the loan is issued. Our own first model reached 0.99 for the same reasons. So we compare the approach instead.
Comparison of approach
AspectCommon practiceLendWise
InputsMany, sometimes from after the loanThree application-time inputs
Data splitRandomBy year for the risk model
Class imbalanceOversampling (SMOTE)Natural rate with calibration
Behaviour checksRarely doneMonotonic constraints and nudge tests
Data auditRarely doneAudit of the 0.99 AUC
Delivered asModel onlyValidator, rules, website and test suite
Comparison with manual screening
FeatureManual screeningLendWise
SpeedMinutes to daysMilliseconds per application
ConsistencyCan differ between officersSame input always gives the same output
Risk estimateBased on judgmentCalibrated default probability
Counter-offerWorked out by handAutomatic search in $500 steps
Input checkingManualValidator and range guard