Semantic-Aligned Block-Sparse Featurizers:
One Percent of Labels Reorganizes a Visual Dictionary

Edward Lu

Research notes · August 2026 · not peer reviewed · combined write-up: BSF beyond fixed widths · related: adaptive-dimensional BSF

Abstract On a controlled synthetic toy model, weakly supervising which block a concept lands in was strictly harmful: the unsupervised assignment was already one-to-one, so labels had nothing to add. That failure predicted its own reversal. On real features, where block assignment is genuinely underdetermined, the same idea should help. We test it on DINOv2 patch tokens with ADE20K segmentation classes as labels: a supervised-contrastive pull on block-support vectors, scaled by the reconstruction loss, applied to a labeled fraction of tokens. With 1% of images labeled, mean block purity more than doubles (0.25 to 0.54) and the number of blocks carrying 90% of a class falls from 626 to 173. A matched-reconstruction control shows the gain is supervision, not undertraining, and a protocol-identical 0% run shows zero artifact from the training procedure itself. Costs are real and reported: a small reconstruction drop, generic high-traffic blocks stay generic, and global class-block NMI degrades past 1%.

1Setup: the regime where supervision should matter

Prior result (toy model, known factors, near-orthogonal subspaces): supervising block identity hurt at every labeled fraction. The unsupervised model already assigned one block per concept; label gradients only perturbed training.
Prediction from that null: supervision helps exactly when assignment is underdetermined, which is the situation in real, polysemantic features.
This test: DINOv2-base patch tokens on ADE20K scenes; every patch gets a segmentation class, so labels are dense and free at any fraction. Vanilla BSF at G=4096, b=4, k=16 vs the same model plus a semantic loss, at 0/1/5/10% labeled images. 15k steps, seed 42, five runs, zero failures.

2Method: a contrastive pull on block support

For labeled tokens, build the normalized pre-TopK block-support vector (which blocks want to fire, before selection). Apply a supervised-contrastive loss: tokens of the same class — from different images — pull their support vectors together, different classes push apart (SupCon, Khosla et al. 2020).
No class-to-block assignment anywhere: no Hungarian matching, no fixed targets. The loss only asks that same-class tokens use similar blocks; which blocks is left to the model, and within-block coordinates stay free.
The semantic term is scaled by the current reconstruction loss and held off for a 15% warmup — both lessons from the toy-model failure (an unscaled loss overwhelmed the MSE; early pressure disturbed dictionary formation).
Two guards built into the design. A 0% run with the identical protocol and init isolates any artifact of the procedure itself. A matched-reconstruction control saves Vanilla checkpoints every 2,500 steps and scores the one nearest each semantic run's R², separating supervision effects from the small reconstruction cost.

3Result: one percent of labels reorganizes the dictionary

Table 1: Semantic supervision on DINOv2 + ADE20K. Purity = mean class purity over live blocks; n90 = blocks needed to cover 90% of a class's firing mass, averaged over common classes; entropy = class-to-block firing entropy.

Run	Global R²	Live blocks	Mean purity	n90	Entropy
Vanilla BSF	0.774	4096	0.252	626	5.69
Sem 0% (protocol twin)	0.774	4096	0.252	617	5.67
Sem 1%	0.742	3716	0.541	173	4.57
Sem 5%	0.743	3285	0.576	87	4.09
Sem 10%	0.750	3169	0.586	97	4.18

Mean purity more than doubles at 1% (0.25 → 0.54); a class's firing mass concentrates from 626 blocks to 173 (87 at 5%). Split-half consistency of a class's block usage rises from 0.982 to 0.993 and above.
The 0% twin lands on Vanilla within the pre-computed seed-noise yardstick (purity within 0.003, NMI within 0.001): the procedure itself does nothing without labels.
The reconstruction cost is 0.024–0.032 R², and the label appetite saturates fast: 5% → 10% buys little (n90 87 → 97).

Purity, concentration, matched-recon control — **Figure 1: One percent of labels doubles purity and concentrates classes; the gain survives reconstruction matching.** **Left:** mean block purity vs labeled fraction (dashed = Vanilla). **Middle:** blocks needed for 90% of a class's mass. **Right:** at equal reconstruction (R² = 0.74), the nearest Vanilla checkpoint reaches purity 0.21 vs 0.54 for Sem 1%.

4The decisive control: matched reconstruction

Objection: semantic runs reconstruct slightly worse, so maybe the purity gain is just an earlier point on the Vanilla trajectory.
Test: score the Vanilla checkpoint whose R² is nearest each semantic run (step 2,500, R² = 0.745).
Result: purity 0.209 and n90 965, against 0.541 and 173 for Sem 1% at R² = 0.742. At equal reconstruction the semantic model is about 2.6× purer and 5.6× more concentrated. The reorganization is bought by labels, not by giving up reconstruction.

5What supervision actually changes, and what it does not

Concentration: the most shattered common class (person) needs 1,158 blocks for 90% of its mass under Vanilla and 316 under Sem 1% (Figure 2, top row).
Consistency: the same block ids carry a class across very different images; wall, building and sky each reuse a small stable block set scene after scene (Figure 3).
What does not change: high-traffic generic blocks stay generic — usage-weighted purity is flat (0.34 → 0.28–0.33), so supervision organizes the long tail of blocks, not the busy head.
A metric that degrades: global class–block NMI falls past 1% (0.164 → 0.122 at 5%): pulling classes onto compact block sets also merges some class distinctions the unsupervised code kept. Purity, concentration and NMI move differently — report all three, pick by use case.
Live blocks drop from 4,096 to ~3,200–3,700: alignment pressure retires part of the dictionary as classes consolidate.

Shattering before and after supervision — **Figure 2: Shattering collapses under weak supervision.** Per-class block distributions for the most shattered common classes, Vanilla (top) vs Sem 1% (bottom): the same class's firing mass concentrates onto a small block set.

Same class across images and its active blocks — **Figure 3: The same class activates the same blocks across different images.** Rows = ADE20K classes; for each of four different validation images, the class's patches (highlighted) and the top-5 blocks they activate — under Sem 1% the block bars repeat across images.

Metrics vs supervision fraction — **Figure 4: Metric-by-metric view of the supervision dial.** Usage-weighted purity, class–block NMI, split-half consistency, and reconstruction R² against labeled fraction; dashed = the differently-seeded Vanilla run, whose gap to Sem 0% doubles as a live seed-noise yardstick.

6Limits and what's next

DINOv2 fallback (DINOv3 weights gated in this run); 10-coarse-label Imagenette probes elsewhere in the series do not apply here, but ADE20K's 150 classes are still scene-level proxies — attribute-level evaluation (TAPAScore/synCUB) is the planned upgrade.
Single seed per run; the seed-noise yardstick (|Vanilla − Sem 0%| ≈ 0.003 purity) is measured, and the headline gaps exceed it by ~100×, but multi-seed replication of Vanilla and Sem 1% remains queued.
Purity is computed over live blocks; combined with block retirement this can flatter the semantic runs — the matched-budget comparisons in the combined write-up address the analogous effect for adaptive widths.
Open: does the reorganized dictionary steer better? The toy steering null (see the LLM paper) makes ADE20K-aligned blocks the natural next steering substrate.

7Sources

Fel et al. Structuring Sparsity: Block-Sparse Featurizers Capture Visual Concept Manifolds. arXiv:2606.25234 · official code
Khosla et al. Supervised Contrastive Learning. arXiv:2004.11362
Zhou et al. Scene Parsing through ADE20K. CVPR 2017.
Oquab et al. DINOv2. arXiv:2304.07193
TAPAScore / synCUB concept-annotation benchmark. arXiv:2606.24716

All numbers from a self-contained Colab notebook with cached runs, recorded deviations, and an exported archive · this page was reconstructed from its published text after an accidental deletion; all figures and numbers are unchanged from the original run

Semantic-Aligned Block-Sparse Featurizers:One Percent of Labels Reorganizes a Visual Dictionary