Well, to be honest these are my guesses. As without any guidance, VQGAN divagates graphically, as GPT2 does often... as CLIP is which tells to VQGAN what to do... makes sense for me. But my technical knowledge about AI's is very limited. In fact had to google some of the stuff you mentioned in your comment hehe.