Train your own decision model in AC2

AC2 now supports decision model training. Teams can build custom Jev-like models for use cases like request routing, content classification, and anomaly detection. Start with a question, its possible answers, and labeled examples. Then post-train a model that returns a probability for each answer.

A decision model pairs a language-model backbone with a small learned output layer called a decision head. The backbone processes the input, question, and answer choices; the head produces scores, and a softmax over the supplied choices turns those scores into probabilities. Instead of generating a free-form response, the model directly scores your options. You can easily add a variable-sized decision head to existing frontier open-source models through AC2. Choose any open source model, and train it as a decision model.

Training supports hard labels and probability targets with two losses:

• Brier loss⌝ minimizes the sum of squared differences between predicted and target probabilities across answer choices.
• Cross-entropy⌝, or negative log likelihood, minimizes the negative log probability of the correct answer for hard labels, or the target-weighted average for probability targets.

Training updates both the decision head and the language backbone.

We tried it on Civil Comments⌝, a public toxicity-classification dataset. Starting from Perplexity’s pplx-decider-v1-27b⌝, we trained for one epoch on 50,000 sampled training comments using Brier loss and annotator toxicity fractions as soft targets.

On a held-out set of 20,000 sampled comments from the official test split, F1 for detecting toxic comments increased from 0.482 to 0.677, and average precision improved from 0.531 to 0.771. Both evaluations used a fixed 0.5 threshold for targets and predictions when computing F1, with no test-set tuning. The training job completed in less than an hour on four B300 GPUs.

See the decision model training guide⌝ to get started with your own classification task.