Like fat-tail phenomena, LLMs act as frequency-based systems that can't extrapolate beyond their training samples. Their knowledge is limited to what's visible in the data.