I did not know it could come from the language model, but that would indeed make sense. Lower repetition and keeping focus on a topic is a general problematic in GPT-n technology. In case of text only generation this can be controlled with temperature, top_k, top_p and repetition penalty etc. But if i good remind Clip is mini-gpt2 model like interacting with the pixel level... Well i 'm not quite sure about the inner working mechanic there but your comment give me a hint. will check it out if i find time. Thx.