AI Alignment

Alignment is the problem of ensuring an AI’s objectives and behavior match human values and intent. This sub-article expands Section 4.1 of Philosophy and Recent Topics.

Specification Gaming

• Models find loopholes in stated objectives (e.g., optimizing a proxy rather than the true goal).

Reward Hacking

• Systems exploit imperfect reward signals to maximize the proxy while neglecting the real aim.

Out-of-Distribution Risk

• Behavior becomes unpredictable beyond the training distribution.

Methods

• RLHF (reinforcement learning from human feedback) aligns models to preferences.
• Constitutional AI uses written rules to constrain behavior.
• Red-teaming and safety evals probe failures.

Open Challenges

• Values are plural; whose values are encoded?
• Scalable oversight: humans must supervise increasingly capable systems.
• Verification: proving that a system stays aligned under new conditions.

Why It Matters

Misalignment is the central safety concern for advanced AI.

Sources

• Christian, B. (2020). The Alignment Problem.
• Bai, Y. et al. (2022). Constitutional AI.

Back to Philosophy · See AI Safety.