LLM outputs are optimized like ultra-processed food

Aug 23, 2025. 203 words. 1 minute. In Quotes. AI LLMs RL

From andy99:

Even though I think most people know this deep down, I still don’t think we actively realize how optimized LLMs are towards sounding good. It’s the ultra processed food version of information consumption. People are super lazy (economical if you like) and rlhf et al have optimized LLM output to being easy to digest.

Consequence is you get a bunch of output that looks really good as long as you don’t think about it (and they actively promotes not thinking about it) that you don’t really understand, and that if you did dig into you’d realize is empty fluff or actively wrong.

It’s worse than not learning, it’s actively generating unthinking but palatable garbage that’s the opposite of learning.

This is a great observation and I love the analogy to processed food.

Reinforcement Learning with Human Feedback (RLHF) has made ChatGPT possible in the first place. However, the result is an extremely aligned model (not alignment for safety, but for maximizing engagement). It may feel good “talking” to these models, but the answers are optimized to feel good. If the user is aware of that, it can still be a useful tool, but I doubt that most users are aware of this fact.