When to Use Confidence Weighting: A Practitioner's Guide

TL;DR: Confidence weighting helps most with few classifiers (m < 8) in the quality sweet spot (55-75% ROC-AUC) with high diversity. With many classifiers (m > 12), simple averaging is surprisingly effective!

---

Notation

Throughout this document, we use the following notation:

Symbol	Meaning	Example
m	Number of base classifiers	m = 15 (you have 15 models)
n	Number of instances (data points)	n = 1000 (1000 patients, genes, etc.)
u	Classifier index	u ∈
i	Instance index	i ∈
R	Probability matrix, shape (m, n)	R[u, i] = probability that classifier u assigns to instance i
labels	True labels, shape (n,)	labels[i] = 1 (positive) or 0 (negative)
labeled_mask	Boolean mask for labeled data	labeled_mask[i] = True if instance i is labeled
y_true	True labels (labeled instances only)	y_true = labels[labeled_mask]

Example setup:

# You have:
m = 15                           # 15 classifiers (e.g., Random Forest, SVM, Neural Net, ...)
n = 1000                         # 1000 instances (e.g., patients, genomic sequences, ...)
R = np.array(shape=(15, 1000))   # Probability matrix: R[u, i] = classifier u's prediction for instance i
labels = np.array(shape=(1000,)) # Ground truth: labels[i] = 1 or 0 (may contain NaN for unlabeled)

# To evaluate classifier u on labeled data:
for u in range(m):  # Loop over classifiers u=0, 1, 2, ..., 14
    quality_u = compute_metric(labels[labeled_mask], R[u, labeled_mask])

---

Key Definitions

Classifier Quality (q)

Primary Metric: Depends on Your Data

For Imbalanced Data (Recommended): PR-AUC (Precision-Recall AUC) or F1-Score

from sklearn.metrics import average_precision_score, precision_recall_curve, auc, f1_score

# For a single classifier u (e.g., u=0 for the first classifier)
u = 0  # Classifier index

# PR-AUC (Precision-Recall AUC) - RECOMMENDED for imbalanced data
# R[u, labeled_mask] = predictions from classifier u on labeled instances
# y_true[labeled_mask] = ground truth labels for labeled instances
quality_prauc = average_precision_score(y_true[labeled_mask], R[u, labeled_mask])

# Or manually compute PR-AUC
precision, recall, _ = precision_recall_curve(y_true[labeled_mask], R[u, labeled_mask])
quality_prauc = auc(recall, precision)

# F1-Score (requires threshold, here we use 0.5)
y_pred = (R[u, labeled_mask] > 0.5).astype(int)
quality_f1 = f1_score(y_true[labeled_mask], y_pred)

For Balanced Data: ROC-AUC is acceptable

from sklearn.metrics import roc_auc_score

# ROC-AUC - Use only if classes are roughly balanced (e.g., 40/60)
u = 0  # Classifier index
quality_roc = roc_auc_score(y_true[labeled_mask], R[u, labeled_mask])

Interpretation (for PR-AUC or ROC-AUC): - 1.0 = Perfect classifier (no errors) - 0.9-0.95 = Excellent (our "ceiling" range) - 0.75-0.85 = Good (diminishing returns zone) - 0.55-0.75 = Moderate (sweet spot for confidence weighting) - 0.50 = Random baseline (varies by metric) - < 0.50 = Below random (something is wrong)

Why PR-AUC for Imbalanced Data? ✅ RECOMMENDED 1. ✅ Focuses on minority class (what you actually care about - e.g., splice sites) 2. ✅ Ignores TNs (abundant negatives don't inflate score) 3. ✅ Threshold-independent (evaluates ranking quality) 4. ✅ Sensitive to performance on positives (critical for biomedical data)

Why NOT ROC-AUC for severe imbalance? ⚠️ 1. ❌ Misleading with few positives - High TN count inflates score 2. ❌ Equal weight to FPR and TPR - But we care more about TPR! 3. ❌ Can look good while missing most positives - Dangerous in critical applications (e.g., disease detection, splice site prediction)

When ROC-AUC is okay: - Balanced datasets (e.g., 40/60 split) - When FPR and TPR are equally important - Comparing with literature that uses ROC-AUC

Average Ensemble Quality

The average quality across all m classifiers:

# Compute quality for each of the m classifiers
qualities = []
for u in range(m):  # u = 0, 1, 2, ..., m-1 (each classifier)
    # Evaluate classifier u on labeled data
    auc = roc_auc_score(y_true[labeled_mask], R[u, labeled_mask])
    qualities.append(auc)

# Average quality across all classifiers
avg_quality = np.mean(qualities)  # This is what we mean by "quality q"

Example: - m = 15 classifiers - Individual qualities: [0.65, 0.70, 0.58, 0.72, 0.68, ...] - Average quality q = 0.68 → We say "Quality 0.68" in this document

Metric Selection Guide

Metric	Use Case	Formula	Notes
PR-AUC ⭐	Imbalanced data (biomedical, rare events)	average_precision_score(y_true, y_pred_proba)	RECOMMENDED default
F1-Score	Imbalanced data, need single threshold	2 * (precision * recall) / (precision + recall)	Good for operational metrics
ROC-AUC	Balanced data, literature comparison	roc_auc_score(y_true, y_pred_proba)	⚠️ Misleading if severe imbalance
Accuracy	Balanced data, all errors equal cost	mean(y_true == y_pred)	❌ Avoid for imbalanced data
AP (Avg Precision)	Same as PR-AUC	average_precision_score(y_true, y_pred_proba)	AP ≈ PR-AUC in practice

⚠️ Important: - For imbalanced data (most biomedical applications): Use PR-AUC or F1-Score - For balanced data: ROC-AUC is acceptable - Throughout this document: When we say "quality 0.70", we mean quality metric ≈ 0.70 (adjust interpretation based on your chosen metric)

Diversity

Definition: Standard deviation of classifier qualities.

# qualities = [quality_0, quality_1, ..., quality_{m-1}]
# Example: qualities = [0.65, 0.70, 0.58, 0.72, 0.68] for m=5 classifiers
diversity = np.std(qualities)  # Higher = more diverse

Interpretation: - > 0.10 = High diversity (classifiers have very different strengths/weaknesses) - Example: qualities = [0.50, 0.70, 0.55, 0.80, 0.60] → std = 0.11 - 0.05-0.10 = Medium diversity - Example: qualities = [0.65, 0.70, 0.68, 0.72, 0.66] → std = 0.03 - < 0.05 = Low diversity (all classifiers perform similarly) - Example: qualities = [0.68, 0.69, 0.67, 0.68, 0.69] → std = 0.008

Why it matters: High diversity means classifiers make different errors on different instances, which confidence weighting can leverage to improve ensemble performance.

Ensemble Size (m)

Definition: Number of base classifiers in your ensemble.

m, n = R.shape  # m = number of classifiers, n = number of instances

Critical thresholds: - m < 5 = Very small (each classifier critical) - 5 ≤ m < 12 = Medium (sweet spot for confidence weighting) - m ≥ 12 = Large (simple averaging very effective) - m ≥ 15 = Very large (minimal gains from weighting)

Complete Example: Computing All Metrics

import numpy as np
from sklearn.metrics import roc_auc_score, average_precision_score, f1_score, precision_recall_curve, auc

def evaluate_ensemble_config(R, labels, labeled_idx=None, metric='auto'):
    """
    Evaluate ensemble configuration and quality.

    Parameters
    ----------
    R : np.ndarray, shape (m, n)
        Probability matrix (classifiers × instances)
    labels : np.ndarray, shape (n,)
        True labels (may contain NaN for unlabeled)
    labeled_idx : np.ndarray, optional
        Boolean mask or indices for labeled instances
    metric : str, default='auto'
        Quality metric: 'prauc' (recommended for imbalanced), 'roc_auc', 'f1', or 'auto'
        'auto' selects prauc if imbalance detected, otherwise roc_auc

    Returns
    -------
    dict with keys: m, n, avg_quality, diversity, qualities, baseline_score, metric_used
    """
    m, n = R.shape

    # Create labeled mask
    if labeled_idx is None:
        labeled_mask = ~np.isnan(labels)
    elif labeled_idx.dtype == bool:
        labeled_mask = labeled_idx
    else:
        labeled_mask = np.zeros(n, dtype=bool)
        labeled_mask[labeled_idx] = True

    y_true = labels[labeled_mask]

    # Auto-detect imbalance
    pos_rate = np.mean(y_true)
    is_imbalanced = (pos_rate < 0.3) or (pos_rate > 0.7)

    # Select metric
    if metric == 'auto':
        metric = 'prauc' if is_imbalanced else 'roc_auc'
        print(f"⚙️  Auto-selected metric: {metric.upper()} (positive rate: {pos_rate:.1%})")

    # Compute quality for each of the m classifiers
    qualities = []
    for u in range(m):  # Loop over classifiers: u = 0, 1, ..., m-1
        try:
            if metric == 'prauc':
                # Evaluate classifier u using PR-AUC
                score = average_precision_score(y_true, R[u, labeled_mask])
            elif metric == 'roc_auc':
                # Evaluate classifier u using ROC-AUC
                score = roc_auc_score(y_true, R[u, labeled_mask])
            elif metric == 'f1':
                # Evaluate classifier u using F1-Score (requires hard predictions)
                y_pred = (R[u, labeled_mask] > 0.5).astype(int)
                score = f1_score(y_true, y_pred)
            qualities.append(score)
        except ValueError:
            # Handle edge cases (e.g., only one class present)
            qualities.append(0.5 if metric in ['prauc', 'roc_auc'] else 0.0)

    qualities = np.array(qualities)

    # Baseline ensemble (simple averaging across all m classifiers)
    # R[:, labeled_mask] = all m classifiers' predictions on labeled instances
    # axis=0 means average across classifiers (m dimension)
    baseline_pred = np.mean(R[:, labeled_mask], axis=0)
    if metric == 'prauc':
        baseline_score = average_precision_score(y_true, baseline_pred)
    elif metric == 'roc_auc':
        baseline_score = roc_auc_score(y_true, baseline_pred)
    elif metric == 'f1':
        baseline_pred_binary = (baseline_pred > 0.5).astype(int)
        baseline_score = f1_score(y_true, baseline_pred_binary)

    return {
        'm': m,
        'n': n,
        'n_labeled': labeled_mask.sum(),
        'positive_rate': pos_rate,
        'is_imbalanced': is_imbalanced,
        'metric_used': metric,
        'avg_quality': np.mean(qualities),
        'min_quality': np.min(qualities),
        'max_quality': np.max(qualities),
        'diversity': np.std(qualities),
        'qualities': qualities,
        'baseline_score': baseline_score
    }

# Example usage
config = evaluate_ensemble_config(R, labels, labeled_idx, metric='auto')

print(f"\n📊 Ensemble Configuration:")
print(f"   Ensemble size: {config['m']} classifiers")
print(f"   Data: {config['n_labeled']} labeled, positive rate {config['positive_rate']:.1%}")
print(f"   {'⚠️  Imbalanced!' if config['is_imbalanced'] else '✓ Balanced'}")
print(f"\n📈 Quality Metrics ({config['metric_used'].upper()}):")
print(f"   Average quality: {config['avg_quality']:.3f}")
print(f"   Quality range: [{config['min_quality']:.3f}, {config['max_quality']:.3f}]")
print(f"   Diversity (std): {config['diversity']:.3f}")
print(f"\n🎯 Baseline Performance:")
print(f"   Simple averaging: {config['baseline_score']:.3f} {config['metric_used'].upper()}")
print(f"\n→ This is what we mean by 'quality {config['avg_quality']:.2f}'")

Output example (Imbalanced data - e.g., splice sites):

⚙️  Auto-selected metric: PRAUC (positive rate: 15.0%)

📊 Ensemble Configuration:
   Ensemble size: 15 classifiers
   Data: 200 labeled, positive rate 15.0%
   ⚠️  Imbalanced!

📈 Quality Metrics (PRAUC):
   Average quality: 0.52
   Quality range: [0.38, 0.68]
   Diversity (std): 0.095

🎯 Baseline Performance:
   Simple averaging: 0.74 PRAUC

→ This is what we mean by 'quality 0.52'

Key insights: 1. Imbalance detected (15% positives) → Auto-selected PR-AUC 2. Individual classifiers weak (avg 0.52 PR-AUC) but ensemble strong (0.74 PR-AUC) 3. This is the ensemble size effect - even weak classifiers become powerful when averaged!

Output example (Balanced data):

⚙️  Auto-selected metric: ROC_AUC (positive rate: 48.0%)

📊 Ensemble Configuration:
   Ensemble size: 15 classifiers
   Data: 200 labeled, positive rate 48.0%
   ✓ Balanced

📈 Quality Metrics (ROC_AUC):
   Average quality: 0.68
   Quality range: [0.55, 0.78]
   Diversity (std): 0.082

🎯 Baseline Performance:
   Simple averaging: 0.89 ROC-AUC

→ This is what we mean by 'quality 0.68'

---

Quick Decision Tree

Step 1: What's your minority class rate?

├─ 5-10% positives (rare disease, drug response):
│   └─ ✅✅✅ OPTIMAL for confidence weighting! (Expected: +1-4%)
│       → Proceed to Step 2
│
├─ 2-5% positives (very rare events):
│   └─ ✅ Good candidate (Expected: +0.5-4%, varies)
│       → Proceed to Step 2, test on your data
│
├─ 10-20% positives (moderate imbalance):
│   └─ ✅ Can help (Expected: +0.5-2%)
│       → Proceed to Step 2
│
└─ <1% positives (splice sites, extreme rarity):
    └─ ❌ Not recommended (Expected: < 0.5%)
        → Focus on: More data, better features, active learning

Step 2: How many classifiers do you have?

├─ m ≥ 15: Simple averaging very effective
│   └─ Expected gain from confidence weighting:
│       - 5% positives: +2-4% ⭐
│       - 10% positives: +0.5-1%
│       - Use if every % matters!
│
├─ 10 ≤ m < 15: Confidence weighting helpful
│   └─ Expected gain: +1-5% (depending on imbalance)
│       → Especially good at 5% positives
│
├─ 5 ≤ m < 10: Confidence weighting very helpful
│   └─ Expected gain: +2-8%
│       → Sweet spot for confidence weighting!
│
└─ m < 5: Confidence weighting critical!
    └─ Expected gain: +3-10%
        Individual classifier quality matters most

---

Experimental Evidence (2026-01-24)

Imbalanced Data Experiments ⭐ PRIMARY RESULTS

Setup: - 15 classifiers, high diversity - 3 trials per quality level - Primary metric: PR-AUC (appropriate for imbalanced data)

Three scenarios tested:

Imbalance	Random Baseline	Peak Improvement	Status
10% positives	0.10	+1.06%	✅ Recommended
5% positives ⭐	0.05	+3.94% 🏆	✅✅✅ OPTIMAL
1% positives	0.01	+0.10%	❌ Not recommended

Key Discovery: The 5% Sweet Spot

Most important finding: 5% positives (95% negatives) shows BEST gains!

Why? - Not too easy (10% has less room for improvement) - Not too hard (1% hits fundamental limits) - Just right - Challenging but learnable

Results at 5% positives:

Quality 0.158 PR-AUC (Best point):
  Baseline: 0.197 PR-AUC
  Learned:  0.237 PR-AUC
  Gain: +3.94% (+0.040 PR-AUC points)

  This is HUGE for rare disease detection!
  → 20% relative improvement in catching positives

Results by Imbalance Level

10% Positives (Disease Detection) - Quality range: 0.112 - 0.270 PR-AUC - Peak improvement: +1.06% at quality 0.270 - Baseline already decent (0.60 PR-AUC) → less room to improve

5% Positives (Rare Disease) ⭐ - Quality range: 0.050 - 0.158 PR-AUC
- Peak improvement: +3.94% at quality 0.158 🏆 - Optimal balance of challenge and learnability

1% Positives (Splice Sites) - Quality range: 0.029 - 0.097 PR-AUC - Peak improvement: +0.10% (negligible) - Extreme rarity makes improvements very difficult

Visualizations: - Individual results: results/quality_threshold_*/quality_threshold_analysis.png - Side-by-side comparison: results/imbalance_comparison.png

Earlier Experiments (Balanced/Mild Imbalance)

Setup: - 15 classifiers, high diversity - Quality range: 0.45-0.72 ROC-AUC - 5 trials per level - Data: Mild imbalance (60/40) with realistic complexity

⚠️ Note: These earlier experiments used ROC-AUC. The ensemble size effect and quality patterns hold across metrics, but absolute thresholds differ.

Quality (ROC-AUC)	Baseline	Label-Aware	Improvement
0.45	0.39	0.40	+0.44 pts
0.48	0.48	0.49	+0.49 pts
0.50	0.59	0.59	+0.47 pts
0.54	0.71	0.72	+0.40 pts
0.58	0.83	0.83	+0.28 pts
0.61	0.90	0.90	+0.16 pts
0.65	0.95	0.95	+0.13 pts
0.70	0.98	0.98	+0.06 pts

Key Finding: With 15 classifiers at quality 0.58, simple averaging already achieves 0.83 ROC-AUC! The law of large numbers is powerful.

---

The Ensemble Size Effect

Why Size Matters

Mathematical Intuition:

Individual error: e = 1 - quality (where quality = ROC-AUC)
Ensemble error: E ≈ e / √m

Example with quality = 0.70 ROC-AUC (e = 0.30):
  m = 3:  E ≈ 0.30 / √3  ≈ 0.17  → Ensemble ~0.83 ROC-AUC
  m = 5:  E ≈ 0.30 / √5  ≈ 0.13  → Ensemble ~0.87 ROC-AUC
  m = 10: E ≈ 0.30 / √10 ≈ 0.09  → Ensemble ~0.91 ROC-AUC
  m = 15: E ≈ 0.30 / √15 ≈ 0.08  → Ensemble ~0.92 ROC-AUC

Real Results (from experiments): - Quality 0.58, m=15 → Baseline 0.83 (very close to theory!) - Quality 0.70, m=15 → Baseline 0.98

Implication: With many classifiers, simple averaging is already near-optimal!

When Ensemble Size Doesn't Save You

❌ Systematic biases - All classifiers fail on same subgroup
❌ Low diversity - Classifiers make correlated errors
❌ Domain-specific expertise - Some classifiers excel on specific cases
❌ Severe miscalibration - Confidence scores meaningless

In these cases, confidence weighting can help even with m > 12.

---

Strategy Recommendations

For Large Ensembles (m ≥ 12)

Default: Simple Averaging

ensemble_pred = np.mean(R, axis=0)

When to try confidence weighting: - Classifiers have known domain expertise (e.g., algorithm A excels on subgroup X) - Very limited labeled data (n_labeled < 50) - You observe that some classifiers consistently fail on specific subgroups

Recommended strategy: LabelAwareConfidence (simple, consistent +0.3-0.5%)

For Medium Ensembles (5 ≤ m < 12)

⭐ Sweet spot for confidence weighting!

Quality 0.55-0.75 + High Diversity:

# Option 1: Label-aware (simple, robust)
confidence_strategy = LabelAwareConfidence()

# Option 2: Learned reliability (if systematic biases exist)
rel_model = ReliabilityWeightModel()
rel_model.fit(R, labels, labeled_idx, classifier_stats)
W_learned = rel_model.predict(R)

Expected gains: - Label-aware: +0.5-1.5% ROC-AUC - Learned reliability: +0.5-3% (if biases present)

For Small Ensembles (m < 5)

Confidence weighting is critical!

Individual classifier quality matters significantly. Use:

1. Evaluate each classifier carefully
2. Learn cell-level reliability
3. Consider removing weak classifiers (m=4 strong > m=6 mixed)

Expected gains: 1-5% ROC-AUC improvement

---

Class Imbalance Impact (Validated 2026-01-24)

The Goldilocks Principle of Imbalance

Key Finding: Confidence weighting effectiveness follows a non-monotonic relationship with imbalance!

Improvement vs Minority Class Rate:

 4% ┤        ╭────╮ ← 5% positives: BEST GAINS!
    │       ╱      ╲
 2% ┤      ╱        ╲
    │     ╱          ╲___
 1% ┤____╱               ╰___ 1% positives
    │   10% pos              
 0% ┼────┬────┬────┬────┬────┬────
    0%   5%   10%  15%  20%  25%
         Minority Class Rate

Recommendations by Imbalance Level

✅✅✅ 5-10% Positives: OPTIMAL RANGE

Scenarios: Rare disease (5-10% prevalence), drug response (10-20% responders)

Why optimal: - Challenging enough that confidence weighting matters - Tractable enough to learn meaningful patterns - Best balance of signal and difficulty

Expected Results (m=15): - Quality range: 0.15-0.27 PR-AUC - Improvements: +1-4% PR-AUC - 5% positives shows peak gains (+3.94%)

Action: ✅ Strong recommendation for confidence weighting!

✅ 2-5% Positives: Good Candidate

Scenarios: Very rare diseases, uncommon adverse events

Expected Results: - Variable gains: +0.5-4% (depends on exact rate) - Best around 5% (peak of curve)

Action: ✅ Recommended, test on your data first

⚠️ 10-20% Positives: Moderate Benefit

Scenarios: Moderate imbalance, common diseases

Expected Results: - Improvements: +0.5-1.5% - Baseline already decent due to more positives

Action: ⚠️ Optional - cost/benefit analysis needed

❌ <1% Positives: Not Recommended

Scenarios: Splice sites (0.1-1%), extremely rare events

Why not: - Fundamental scarcity limits learning - Confidence weighting: < 0.5% gain - Ensemble averaging already at limits

Expected Results (at 1% positives): - Quality range: 0.03-0.10 PR-AUC - Improvements: +0.1% (negligible)

Action: Focus on: 1. 🔴 More labeled data (especially positives!) 2. 🔴 Better features (domain expertise critical) 3. 🔴 Active learning (target rare positives) 4. 🔴 Cost-sensitive methods (penalize missing positives) 5. 🔴 Specialized algorithms (SMOTE, focal loss, etc.)

Then consider confidence weighting after improvements above.

---

Quality Thresholds (Validated)

Note: "Quality" = average of your chosen metric across all classifiers. - Imbalanced data: Use PR-AUC or F1-Score (recommended) - Balanced data: ROC-AUC is acceptable

The thresholds below were validated with ROC-AUC, but the patterns hold for other metrics: - Sweet spot exists (moderate quality) - Ceiling effect at high quality - Below-random performance indicates problems

See Key Definitions for how to compute your metric.

❌ Below 0.55 ROC-AUC: Fix Classifiers First

• Individual quality too low: Barely better than random (0.50)
• Too noisy for confidence weighting
• Expected gain: < 0.3% ROC-AUC improvement
• Action: Improve base classifiers first
• Better features / feature engineering
• Hyperparameter tuning
• Try different algorithms
• Get more training data

✅ 0.55-0.75 ROC-AUC: Optimal Range ⭐

• Reliable enough for meaningful confidence signals
• Significant room for improvement
• Expected gain: 0.5-2% ROC-AUC (depends on m and diversity)
• Action: Apply confidence weighting - This is the sweet spot!
• With m < 8: Expect 1-2% gains
• With m = 8-12: Expect 0.5-1% gains
• With m > 12: Expect 0.3-0.5% gains

⚠️ 0.75-0.85 ROC-AUC: Diminishing Returns

• Already good performance
• Less room for improvement
• Expected gain: 0.2-0.8% ROC-AUC
• Action: Optional - test if worth the complexity
• May help if systematic biases exist
• Consider cost vs. benefit

⚠️ Above 0.85 ROC-AUC: Ceiling Effect

• Near-optimal performance
• Minimal improvement possible (approaching theoretical limit)
• Expected gain: < 0.1% ROC-AUC
• Action: Skip confidence weighting
• Simple averaging is sufficient
• Focus effort elsewhere (data quality, feature engineering)

---

Common Misconceptions

❌ "More classifiers → Always use confidence weighting"

Reality: With m ≥ 15, simple averaging is already excellent. Confidence weighting provides minimal gains (<0.3%) unless systematic biases exist.

❌ "Confidence weighting always helps"

Reality: It helps most with: - Fewer classifiers (m < 8) - Moderate quality (0.55-0.75) - High diversity (different strengths/weaknesses) - Systematic biases (domain-specific failures)

❌ "Low quality → Confidence weighting can save it"

Reality: Below 0.55 ROC-AUC, classifiers are too noisy. Fix them first!

✅ "Large ensembles + simple averaging = powerful"

Truth: The law of large numbers is remarkably effective. With 15 diverse classifiers at 0.70 quality, you already get ~0.98 ROC-AUC from simple averaging!

---

Diagnostic Checklist

Before implementing confidence weighting (see Key Definitions for metric details):

from sklearn.metrics import roc_auc_score, average_precision_score

# 1. Check ensemble size
m, n = R.shape
print(f"Ensemble size: {m}")
if m >= 12:
    print("→ Simple averaging likely sufficient")

# 2. Detect imbalance and choose metric
y_true_labeled = y_true[mask]
pos_rate = np.mean(y_true_labeled)
is_imbalanced = (pos_rate < 0.3) or (pos_rate > 0.7)

if is_imbalanced:
    print(f"⚠️  Imbalanced data detected (positive rate: {pos_rate:.1%})")
    print("→ Using PR-AUC as quality metric")
    # Compute PR-AUC for each of the m classifiers
    quality_scores = [average_precision_score(y_true_labeled, R[u, mask]) 
                      for u in range(m)]  # u = 0, 1, ..., m-1
    metric_name = "PR-AUC"
else:
    print(f"✓ Balanced data (positive rate: {pos_rate:.1%})")
    print("→ Using ROC-AUC as quality metric")
    # Compute ROC-AUC for each of the m classifiers
    quality_scores = [roc_auc_score(y_true_labeled, R[u, mask]) 
                      for u in range(m)]  # u = 0, 1, ..., m-1
    metric_name = "ROC-AUC"

# 3. Check quality
avg_quality = np.mean(quality_scores)
print(f"Average quality ({metric_name}): {avg_quality:.3f}")

if avg_quality < 0.55:
    print("→ Too weak, fix classifiers first")
elif avg_quality > 0.85:
    print("→ Already excellent, minimal gains expected")

# 4. Check diversity
diversity = np.std(quality_scores)
print(f"Diversity (std): {diversity:.3f}")

if diversity < 0.05:
    print("→ Low diversity, increase variety first")

# 5. Check baseline ensemble
baseline = np.mean(R, axis=0)
if is_imbalanced:
    baseline_score = average_precision_score(y_true_labeled, baseline[mask])
else:
    baseline_score = roc_auc_score(y_true_labeled, baseline[mask])

print(f"Baseline ensemble {metric_name}: {baseline_score:.3f}")

# Decision
if m >= 12 and baseline_score > 0.90:
    print("\n✓ Simple averaging already excellent!")
elif 5 <= m < 12 and 0.55 <= avg_quality <= 0.75 and diversity > 0.08:
    print("\n⭐ OPTIMAL for confidence weighting!")
else:
    print("\n⚠️  Confidence weighting may have limited benefit")

Example output (Imbalanced biomedical data):

Ensemble size: 10
⚠️  Imbalanced data detected (positive rate: 12.0%)
→ Using PR-AUC as quality metric
Average quality (PR-AUC): 0.58
Diversity (std): 0.095
Baseline ensemble PR-AUC: 0.78

⭐ OPTIMAL for confidence weighting!

---

Recommended Reading

1. base_classifier_quality_analysis.md - Full mathematical and experimental analysis
2. polarity_models_tutorial.md - How to learn cell-level reliability
3. theory_vs_empirics.md - What can be proven vs. what requires experiments

---

Summary

The Golden Rule:

Confidence weighting is most effective with few, diverse, moderately-performing classifiers. With many classifiers, simple averaging is surprisingly powerful!

Practical Threshold: - m < 8: Consider confidence weighting (expected +0.5-2%) - m ≥ 12: Simple averaging preferred (expected +0.1-0.5%)

Quality Sweet Spot: - 0.55-0.75 ROC-AUC → Maximum gains

Don't Forget: - Diversity matters! High diversity amplifies gains - Check for systematic biases - They justify confidence weighting even with large ensembles - Label scarcity - Confidence weighting helps more when n_labeled << n

---

Last Updated: 2026-01-24
Based on: Quality threshold experiments with 15 classifiers, 5 trials, quality range 0.45-0.72