# AI Alignment

Alignment is the problem of ensuring an AI's objectives and behavior match human values and intent. This sub-article expands [Section 4.1 of Philosophy](../philosophy.md) and [Recent Topics](../recent.md).

## Specification Gaming
- Models find loopholes in stated objectives (e.g., optimizing a proxy rather than the true goal).

## Reward Hacking
- Systems exploit imperfect reward signals to maximize the proxy while neglecting the real aim.

## Out-of-Distribution Risk
- Behavior becomes unpredictable beyond the training distribution.

## Methods
- **RLHF** (reinforcement learning from human feedback) aligns models to preferences.
- Constitutional AI uses written rules to constrain behavior.
- Red-teaming and safety evals probe failures.

## Open Challenges
- Values are plural; whose values are encoded?
- Scalable oversight: humans must supervise increasingly capable systems.
- Verification: proving that a system stays aligned under new conditions.

## Why It Matters
Misalignment is the central safety concern for advanced AI.

## Sources
- Christian, B. (2020). *The Alignment Problem.*
- Bai, Y. et al. (2022). *Constitutional AI.*

*Back to [Philosophy](../philosophy.md) · See [AI Safety](ai-safety.md).*