I wanted to know if I could train a real vision-language model from scratch instead of fine-tuning someone else's, and do it on hardware that costs nothing. The whole thing ran on Kaggle, two T4 GPUs, free tier. That single constraint shaped almost every decision that came after.
The design pairs two ideas. One is I-JEPA style masked prediction, where the model predicts missing image patches in latent space rather than reconstructing pixels. The other is SigLIP style contrastive alignment between an image and its caption. I trained on COCO 2017, around 591k image-caption pairs, and the final model lands at 154M parameters with 83M of them trainable.
The first serious runs collapsed. Not crashed, collapsed. The contrastive part of the loss kept pushing every image and caption toward nearly the same point in embedding space, so the numbers that mattered, the retrieval scores, went to garbage while the validation loss sat there looking calm. That was the trap. I had been picking the "best" checkpoint by validation loss, and validation loss was lying to me. A model that maps everything to one blob has a perfectly stable loss and zero useful behaviour.
Fixing it took a few rounds. I moved the contrastive objective over to a SigLIP style sigmoid loss, dropped the text encoder learning rate to a twentieth of the vision side so the language tower stopped dragging the alignment around, and gave the early epochs a proper cold start. The bigger change was in how I judged progress. I stopped trusting val loss and started selecting checkpoints by mean recall on the retrieval task. Once the thing I optimised for matched the thing I actually cared about, the runs stopped collapsing.
Getting two GPUs to cooperate was its own job. The contrastive loss needs to see negatives from across the whole batch, so under distributed training I had to gather embeddings across both ranks in a way that still flows gradients. Get that wrong and each GPU quietly trains on a quarter of the signal. I wired up torchrun, a distributed sampler, rank-zero-gated file writes so the two processes don't fight over the same checkpoint, and a small two-rank smoke test so I'd catch a broken gather before burning a nine hour session on it.
The headline numbers: 50.30 percent image-to-text recall@1 and 68.04 percent mean recall on COCO 5K, trained in about 10.8 hours on the dual T4 setup. ViT-Tiny scale CLIP-style baselines usually sit in the 30 to 40 percent range, so beating that from scratch on free hardware felt like the point was proven. There are 64 tests in the repo, most of them written after some subtle bug ate a run, which is the honest reason tests exist.
The lesson I keep from this one is simple. Your loss is not your goal. They are correlated until they aren't, and the moment they diverge is exactly the moment a model looks healthy and is actually dead. Watch the metric you ship on, not the one that's easy to log.