TAU-HOME.COM
LOADING

Karotte, a Framework for Building RL Environments, Goes Open Source

Preference Model has open-sourced Karotte, its framework for building RL environments. This covers its reward-hack-resistant defaults, validation through 1M+ ev

tau · October 8, 2026

#Karotte #RLEnvironments #OpenSource #PreferenceModel #AIAlignment

Karotte, a Framework for Building RL Environments, Goes Open Source

Preference Model open-sourced Karotte, its framework for building RL environments, on October 7, 2026 (UTC; October 8 KST). The release was confirmed simultaneously in a launch thread by Jennifer Zhou (@chem_safety) and the official blog post "Introducing Karotte". It is the first public component of the stack the team says it used internally over the past year to build MLE RL environments for frontier labs.

Karotte is a framework for building robust RL environments for training aligned AI. Its core is correct reward signals and graders, plus secure defaults against reward hacking. According to the official blog and the a16z investment announcement, it bakes in defenses such as killing stray processes before grading and rejecting files designed to crash the grader, and it has been hardened through more than a million internal evaluation runs plus controlled red-teaming.

What it does and who it is for

Karotte targets AI researchers and ML engineers who design and operate RL environments directly. The official technical deep dive takes a practical approach: it builds a small environment by hand, shows how each part can break, and then contrasts that with how Karotte prevents those failure modes.

In the public thread, the author states directly that training on even small amounts of faulty RL environments can misalign models. Karotte starts with strict, opinionated defaults to reduce such failures, while still letting users modify files such as the Containerfile when they need finer control.

When asked how Karotte compares with Scale AI's AgentEnv, the author answered directly. AgentEnv is closer to building a realistic virtual world, while Karotte focuses on getting graders and reward signals correct and free from reward hacks. The docs reflect that difference: same word "environment", different problem.

Verified install path and stated limits

The official documentation site is karotte.dev, and the repository is github.com/preferencemodel/karotte. The PyPI package karotte (v3.0.52 series at verification time) requires Python 3.12+ and uv. The verified flow is: karotte run builds the image, runs the task, and writes the result to out/transcript.json, then karotte dashboard out/ displays it. Runs use a VM by default: Apple container on macOS, Firecracker on Linux, with docker and podman also supported.

The limits are explicit. The author recommends Firecracker for sandboxing but states that lower-stack security issues, such as container or KVM escapes, are not solved by Karotte itself. Those belong to a lower level of the stack, such as Linux kernel bug fixes and automated rollout monitoring, so production deployments need separate lower-stack security measures.

Seed-funding and recent-revenue figures mentioned in the thread ($16M and ~$15M respectively) are the author's self-reported claims, so this article does not treat them as facts. The verifiable facts here are limited to the release itself, the design direction, the docs/repository/package paths, and the design focus and limits the author stated directly.

Sources