What this is

A technical post on Juejin recently caught our eye: the author pushed XGBoost to 0.93 accuracy on a breast cancer dataset, then spent most of the article on a more awkward fact — no matter how accurate the model, the first question from the business side is always "why was this case rejected?" The standard tool the post recommends is SHAP. It's not a new model — it's a mathematical method that decomposes a model's prediction into per-feature contributions, derived from the Shapley value in game theory, which answers the classic "how do you fairly divide a cake?" question. The core equation is simple: model output = baseline + each feature's contribution increment, and each increment can be attributed by name — "this feature pushed the score up by 0.32."

Industry view

We see two camps pulling in opposite directions. The pro side: explainability is already a compliance necessity. The EU AI Act took effect in 2024, and China's Interim Measures for the Management of Generative AI Services also requires algorithmic traceability; any bank, insurer, or hospital using AI for loan rejections, claim denials, or assisted diagnosis must explain "why" to regulators and to customers. The con side is equally sharp: many companies think installing SHAP is enough, but four common misreadings are creating new compliance risks — treating SHAP as causation (it's actually correlation), reading SHAP values without a baseline (increments without a baseline are nonsense), having correlated features dilute each other's contribution (two highly correlated features split the credit), and assuming TreeExplainer is exact for all models (it's only exact for tree models; everything else is an approximation). Tool bought, problem unsolved.

Impact on regular people

  • Enterprise IT: explainability platform procurement budgets will rise this year. Risk control, lending, healthcare, and HR are the hardest hit.
  • Individual careers: non-technical managers also need to get this. Stop chasing accuracy — start asking "why." This is the new baseline for working with AI.
  • Consumer market: when AI rejects your loan, claim, or job offer, you'll increasingly have the right to demand a plain-language explanation.