Vision language models are not trained from scratch
Nobody trains a vision language model from a blank slate. You freeze a ViT, train a Q-Former to bridge image features into the text embedding space, and wrap the language model in LoRA. This is how those three pieces fit and where the GPU hours go.