Part 2/3:
The Cost of Training
At the moment, the pre-training stage remains the most expensive part of the overall training process. However, as post-training techniques continue to evolve, it's possible that the balance could shift, with post-training becoming the more costly component. This would likely involve scaling up methods that rely on human interaction, such as debate or iterated amplification, rather than direct human feedback.
Constitutional AI: Aligning Models with Principles
The concept of "Constitutional AI," as described in a 2020 paper, introduces the idea of embedding a set of principles or a "constitution" into the model's decision-making process. This allows the AI system to evaluate its own responses against these principles, effectively engaging in a form of self-play to improve its alignment with the specified criteria.
[...]