CUBSO

Catálogo CUBSO · read-only

NATIVE · CANONICAL_COMPLETE

CUBSO_A_SCALE_PROCRUSTES_BGE5K_RRF_CALIBRATION_V1

Track CUBSO_A_ONLY_RETRIEVAL_FUSION

Experimentos

Pregunta científica

Can fixed reciprocal-rank fusion exploit complementary Procrustes and BGE-M3-5k candidate rankings on CALIBRATION without assuming cross-model score calibration?

Full-catalog RRF(k=60) improves CALIBRATION Recall@100 by at least 2 pp with significant paired hit@100 evidence and no target-unseen Recall@100 loss, or improves target-unseen Recall@100 by at least 2 pp without lowering global Recall@100.

Cohort

CUBSO_A_SCALE_CALIBRATION

DEVELOPMENT

Exposure: INFORMED_HYPOTHESIS · Sealed: SEALED

86082d60be22556b4b3347dc8d3c9ce96351705c8d59c03792f3bac1ac88f1a3

Baseline

Retained full-data Procrustes ranking on A-scale CALIBRATION.

Treatment

Full 85,711-candidate reciprocal-rank fusion of retained Procrustes and the frozen BGE-M3-5k checkpoint with k=60 and equal source contribution.

Variable causal

rank aggregation only: full-catalog Procrustes + BGE5K -> RRF(k=60)

Modelo y ejecución

fixed reciprocal-rank fusion over retained Procrustes and BGE-M3-5k

Procrustes:CUBSO_A_SCALE_QUERY_PROJECTION_LC_V1;BGE5K:ca27492deb375a01a369c549cac486450770752cc3d06d8f6b26c6ce597579b4
COMPLETE
PASS_18_OF_18_CANONICAL_CPU_RRF_PROBES

Runtime: CUBSO_A_SCALE_FULL_RRF_GPU_EVALUATION_V1 · Provider: kaggle

3333d8f3dd41e900465cd772ff6544246218f98c

Decisión científica

KEEP

CUBSO_A_SCALE_PROCRUSTES_BGE5K_RRF_CALIBRATION_V1 was evaluated only on the preregistered CALIBRATION development cohort; TEST and TEMPORAL_TEST remain sealed.

Regla: KEEP_PROCRUSTES_BGE5K_RRF60

Resultado: matched preregistered CALIBRATION screening decision gate

External validation: no · Replication required: sí

Métricas

Sólo se visualizan métricas primarias que comparten semántica de display; el gráfico no autoriza comparaciones entre familias incompatibles.

MétricaValor exacto
DeltaMRR-0.00934453103932
DeltaRecall@1000.0415816574371
DeltaRecall@200.0289517147576
DeltaRecall@500.0453706402409
DeltaTargetUnseenMRR0.00777013095091
DeltaTargetUnseenRecall@1000.0665770006725
DeltaTargetUnseenRecall@200.0437121721587
DeltaTargetUnseenRecall@500.0585070611970
DeltaTargetUnseenTop10.000672494956288
DeltaTop1-0.0193335276401
McNemar_R100_p1.95561196101e-66
MRR0.386564395376
Recall@1000.795977849024
Recall@200.661226075974
Recall@500.751481589430
TargetUnseenMRR0.230847522609
TargetUnseenRecall@1000.652992602555
TargetUnseenRecall@200.473436449227
TargetUnseenRecall@500.585743106927
TargetUnseenTop10.156018829859
Top10.288545613524

Tests estadísticos

overall_recall_at_100_losses
119
overall_recall_at_100_mcnemar_exact_p
1.9556119610123373e-66
overall_recall_at_100_ties
9627
overall_recall_at_100_wins
547
overall_recall_at_20_losses
394
overall_recall_at_20_mcnemar_exact_p
1.196141997637048e-19
overall_recall_at_20_ties
9207
overall_recall_at_20_wins
692
overall_recall_at_50_losses
159
overall_recall_at_50_mcnemar_exact_p
2.90562053455355e-66
overall_recall_at_50_ties
9508
overall_recall_at_50_wins
626
target_unseen_recall_at_100_losses
20
target_unseen_recall_at_100_mcnemar_exact_p
2.436195528507787e-18
target_unseen_recall_at_100_ties
1348
target_unseen_recall_at_100_wins
119
target_unseen_recall_at_20_losses
58
target_unseen_recall_at_20_mcnemar_exact_p
0.000001534087396557496
target_unseen_recall_at_20_ties
1306
target_unseen_recall_at_20_wins
123
target_unseen_recall_at_50_losses
38
target_unseen_recall_at_50_mcnemar_exact_p
4.95559410776739e-12
target_unseen_recall_at_50_ties
1324
target_unseen_recall_at_50_wins
125

Incertidumbre

DeltaMRR
[-0.01604255705439739,-0.0026918880057031163]
DeltaRecall@100
[0.03575008765007713,0.04742588873625721]
DeltaRecall@20
[0.020534421036965844,0.03723470567158277]
DeltaRecall@50
[0.03854450081629366,0.052445526431712894]
DeltaTop1
[-0.027562004640292643,-0.010895990609353586]
TargetUnseenDeltaMRR
[-0.005508146985265241,0.021293063368526415]
TargetUnseenDeltaRecall@100
[0.048606327544828555,0.08378166765458148]
TargetUnseenDeltaRecall@20
[0.0223453249165778,0.06445763554706313]
TargetUnseenDeltaRecall@50
[0.03848652858569512,0.07875166965292332]
TargetUnseenDeltaTop1
[-0.015254082961313467,0.01690417195754107]

Lineage

Informed by

CUBSO_A_SCALE_QUERY_PROJECTION_LC_V1
CUBSO_A_SCALE_BGE_M3_DENSE_FINETUNE_5K_V1

Same population as

Code bindings

workspaces/conosce/conosce/cubso/experiments/retrieval/a_scale/rrf_evaluation.py

GIT_BLOB

3333d8f3dd41e900465cd772ff6544246218f98c
0d78e0d5731df75149904c343a09b86dbff96b04e54b3fad584e7eec23712816

workspaces/conosce/conosce/cubso/experiments/retrieval/a_scale/rrf_gpu_worker.py

GIT_BLOB

3333d8f3dd41e900465cd772ff6544246218f98c
a84b31bf5459373afc28c12bb074c190a47f2d1741bd7d6d29cb8d0bed4c021f

Artefactos y evidencia

  • BASE_C0_EMBEDDINGS

    artifacts/derived/procurement_need/tasks/cubso/experiments/a_scale_procrustes_bge_rrf_calibration_v1_20260824/rrf_inputs/base_candidate_embeddings.parquet

    6e663819fb44c92759603a557d8900da04bfae9b75b656fc3bfab048609801c5

  • BGE5K_C0_EMBEDDINGS

    artifacts/derived/procurement_need/tasks/cubso/experiments/a_scale_procrustes_bge_rrf_calibration_v1_20260824/rrf_inputs/bge5k_candidate_embeddings.parquet

    72a3af050a43762b4d2cc418180894dc2b31239c19d4ff65c547a1725c0972e6

  • BGE5K_CALIBRATION_EMBEDDINGS

    artifacts/derived/procurement_need/tasks/cubso/experiments/a_scale_procrustes_bge_rrf_calibration_v1_20260824/rrf_inputs/bge5k_query_embeddings.parquet

    935cf9eeceb3f252b3623744e938912eda16a6f8ef3700fc9491fcc45155f92f

  • CALIBRATION_MAPPING

    artifacts/derived/procurement_need/tasks/cubso/experiments/a_scale_procrustes_bge_rrf_calibration_v1_20260824/rrf_inputs/query_mapping.parquet

    7cf69c85d77bb720b15bef2daff1e8f22337617b91b3f8b5db17be9ddc58e72d

  • EXACT_CALIBRATION_RANKING

    artifacts/derived/procurement_need/tasks/cubso/experiments/a_scale_procrustes_bge_rrf_calibration_v1_20260824/providers/kaggle-rrf/da404a3b3d54/results-v2/rrf_ranking.parquet

    d3c7abf93b49516a1f4fec6a0442bba98969d4bc369d48c1f548d8818eea899f

  • GPU_SUMMARY

    artifacts/derived/procurement_need/tasks/cubso/experiments/a_scale_procrustes_bge_rrf_calibration_v1_20260824/providers/kaggle-rrf/da404a3b3d54/results-v2/job_summary.json

    65f1a789c5016a2e076afd5e7269194e3fff3f6fccc707a7287bc54d11e16758

  • PAIRED_CALIBRATION_EVALUATION

    artifacts/derived/procurement_need/tasks/cubso/experiments/a_scale_procrustes_bge_rrf_calibration_v1_20260824/evaluation.json

    9e7c2e21d03eee53261fa3a7c1a833e605e81aa3df86de5b4753267b22edc4a5

  • PREREGISTRATION

    docs/research/cubso/preregistrations/CUBSO_A_SCALE_PROCRUSTES_BGE5K_RRF_CALIBRATION_V1.json

    2c26d6fa208e9e960877f623d9d197918919672756bc6964913ef714f302a34c

  • PROCRUSTES_CALIBRATION_EMBEDDINGS

    artifacts/derived/procurement_need/tasks/cubso/experiments/a_scale_procrustes_bge_rrf_calibration_v1_20260824/rrf_inputs/procrustes_query_embeddings.parquet

    d6bd9ac62bd3697c3815e084dd4aae3636ba33596251e196bc92fd7db6a6af82

Provenance relacionado

Entities

a-scale:rrf:preregistration
PREREGISTRATION
a-scale:screening-result:CUBSO_A_SCALE_PROCRUSTES_BGE5K_RRF_CALIBRATION_V1:decision
SCIENTIFIC_DECISION_EVIDENCE
a-scale:screening-result:CUBSO_A_SCALE_PROCRUSTES_BGE5K_RRF_CALIBRATION_V1:evaluation
EVALUATION

Activities

a-scale:rrf:preregistration-activity
PREREGISTRATION_GIT_BOUND · b4926b787e6fe32213b17a4906a6642c6dd8b9f1 · 2026-08-24T06:10:00Z · 2026-08-24T06:10:00Z
a-scale:screening-result:CUBSO_A_SCALE_PROCRUSTES_BGE5K_RRF_CALIBRATION_V1:binding-activity
RESULT_BINDING · 3333d8f3dd41e900465cd772ff6544246218f98c · 2026-08-24T06:49:00Z · 2026-08-24T06:49:00Z

Edges

a-scale:rrf:edge:prereg-generated
WAS_GENERATED_BY: a-scale:rrf:preregistration → a-scale:rrf:preregistration-activity
a-scale:screening-result:CUBSO_A_SCALE_PROCRUSTES_BGE5K_RRF_CALIBRATION_V1:edge:1
WAS_GENERATED_BY: a-scale:screening-result:CUBSO_A_SCALE_PROCRUSTES_BGE5K_RRF_CALIBRATION_V1:evaluation → a-scale:screening-result:CUBSO_A_SCALE_PROCRUSTES_BGE5K_RRF_CALIBRATION_V1:binding-activity
a-scale:screening-result:CUBSO_A_SCALE_PROCRUSTES_BGE5K_RRF_CALIBRATION_V1:edge:2
WAS_GENERATED_BY: a-scale:screening-result:CUBSO_A_SCALE_PROCRUSTES_BGE5K_RRF_CALIBRATION_V1:decision → a-scale:screening-result:CUBSO_A_SCALE_PROCRUSTES_BGE5K_RRF_CALIBRATION_V1:binding-activity

Source authority SHA-256

ab2d1066ba1a483a08a5bff24bee85f68197ebef177d93d999a26c059b93f088