Scientists Have Found a New Way to Control AI’s Behavior

Models are rewarded regardless of how they completed their task, incentivizing shortcuts or other questionable impulses, like accruing power.

A version of this problem helps explain why AI is often a suck-up. During training, models are rewarded for answers most likely to receive a “thumbs up” from human graders. Known as “reinforcement learning from human feedback,” or RLHF, the technique was heralded as a safety breakthrough in 2022 for its ability to turn unruly language models into well-mannered chatbots, curbing their tendency to offer racist and misogynistic responses. But it can also make models pathologically agreeable, sometimes indulging and reinforcing users’ delusions.

The underlying riddle is how to train AI to do something without inadvertently teaching it the wrong lesson—known among researchers as the “alignment problem.” 

Similar to how neuroscientists use real-time magnetic resonance imaging to identify which regions of brain activity correlate with certain states, interpretability researchers have developed tools for recognizing which patterns of digital neuron activations correspond with concepts like “the Golden Gate Bridge” or “reward hacking,” the technical term for AI cheating. A tiny AI model can then be trained to spot these patterns. The result is known as a “probe,” a narrow window into the model, which fires when it “thinks” about a particular concept.