Metric Limitations & Assumptions

Scientific transparency demands a rigorous account of what a framework cannot do. TrustLens relies on specific statistical methodologies that carry inherent limitations and edge cases.

[!NOTE] Evidence Traceability: The limitations discussed on this page reflect the bounded scope of the trustlens_model_zoo_benchmark.ipynb and are derived from the empirical bounds of the framework’s diagnostic capabilities.

1. Calibration Score Assumptions

The primary driver of the TrustLens Calibration module is the Expected Calibration Error (ECE).

  • Limitation: ECE is highly sensitive to the number of bins chosen and the binning strategy (uniform vs. quantile). TrustLens uses a standard 10-bin uniform approach.

  • Edge Case: If a model outputs extremely clustered probabilities (e.g., heavily quantized outputs), the ECE might artificially inflate or deflate depending on where those clusters land relative to the bin edges.

  • Assumption: The metric assumes that the target variable is stationary. If the base rate of the underlying population changes drastically post-deployment, the pre-computed ECE becomes invalid.

2. Failure Score & The Confidence Gap

The Failure Score penalizes models that make incorrect predictions with high confidence.

  • Limitation: The metric assumes that the model’s output probabilities are somewhat correlated with epistemic uncertainty.

  • Edge Case: If a model is perfectly calibrated but inherently low-confidence (e.g., predicting 0.51 vs 0.49 for all samples), the Confidence Gap shrinks to zero. TrustLens will score this model highly on “Failure Stability” because it doesn’t fail confidently, even though it fails frequently. (The Trust Score balances this by penalizing the baseline accuracy/calibration).

3. Bias Score Dependencies

The Bias module evaluates disparity across defined subgroups (e.g., via Equalized Odds or Demographic Parity).

  • Limitation: TrustLens cannot detect bias against subgroups that are not explicitly defined in the sensitive_features array. It does not perform unsupervised intersectional bias discovery.

  • Edge Case: In scenarios with extremely small minority subgroups (e.g., N < 10), the variance of the subgroup metric (like TPR/FPR) becomes so high that the resulting Bias penalty may be statistically noisy rather than structurally meaningful.

4. Representation Score (Silhouette)

The Representation module utilizes Silhouette scores to assess the clustering quality of the model’s latent embeddings.

  • Limitation: The Silhouette score inherently prefers convex, hyper-spherical clusters.

  • Edge Case: Complex, non-linear manifold topologies (common in advanced vision models or LLMs) may receive artificially low Silhouette scores despite being highly separable by a downstream non-linear classifier.

  • Assumption: The module requires explicit access to internal model representations. For API-based black-box models, this module cannot be executed, and the framework gracefully degrades to omit the penalty.