Replicating the Vision Transformer on the Terracotta Temples of Bishnupur
ViT-Base/16 in PyTorch on a 16GB M1, pointed at a hand-built dataset of ten Malla-era temples in eastern India. Trained from scratch it learned nothing; the same 85.8M parameters starting from ImageNet weights scored 88.9%.
![86.5% on IndicGLUE sna.bn, above mBERT and IndicBERT — from 7.5M parameters trained 28 hours on one 16GB M1. On the right, the six-layer encoder stack: a [CLS] A [SEP] B [SEP] pair goes in at the top, a pooled [CLS] classifier reads it out at the bottom.](/_astro/bert_cover.ChghBB-Z_Z16YI7L.webp)
