Meta FAIR Launched Code World Mannequin (CWM): A 32-Billion-Parameter Open-Weights LLM, to Advance Analysis on Code Era with World Fashions

Meta FAIR launched Code World Mannequin (CWM), a 32-billion-parameter dense decoder-only LLM that injects world modeling into code technology by coaching on execution traces and long-horizon agent–surroundings interactions—not simply static supply textual content.

What’s new: studying code by predicting execution?

CWM mid-trains on two giant households of statement–motion trajectories: (1) Python interpreter traces that file native variable states after every executed line, and (2) agentic interactions inside Dockerized repositories that seize edits, shell instructions, and take a look at suggestions. This grounding is meant to show semantics (how state evolves) somewhat than solely syntax.

To scale assortment, the analysis workforce constructed executable repository photographs from 1000’s of GitHub initiatives and foraged multi-step trajectories by way of a software-engineering agent (“ForagerAgent”). The discharge experiences ~3M trajectories throughout ~10k photographs and three.15k repos, with mutate-fix and issue-fix variants.

Mannequin and context window

CWM is a dense, decoder-only Transformer (no MoE) with 64 layers, GQA (48Q/8KV), SwiGLU, RMSNorm, and Scaled RoPE. Consideration alternates native 8k and world 131k sliding-window blocks, enabling 131k tokens efficient context; coaching makes use of document-causal masking.

Coaching recipe (pre → mid → publish)

• Common pretraining: 8T tokens (code-heavy) at 8k context.
• Mid-training: +5T tokens, long-context (131k) with Python execution traces, ForagerAgent knowledge, PR-derived diffs, IR/compilers, Triton kernels, and Lean math.
• Publish-training: 100B-token SFT for instruction + reasoning, then multi-task RL (~172B-token) throughout verifiable coding, math, and multi-turn SWE environments utilizing a GRPO-style algorithm and a minimal toolset (bash/edit/create/submit).
• Quantized inference suits on a single 80 GB H100.

Benchmarks

The analysis workforce cites the next cross@1 / scores (test-time scaling famous the place relevant):

• SWE-bench Verified: 65.8% (with test-time scaling).
• LiveCodeBench-v5: 68.6%; LCB-v6: 63.5%.
• Math-500: 96.6%; AIME-24: 76.0%; AIME-25: 68.2%.
• CruxEval-Output: 94.3%.

The analysis workforce place CWM as aggressive with equally sized open-weights baselines and even with bigger or closed fashions on SWE-bench Verified.

For context on SWE-bench Verified’s process design and metrics, see the official benchmark resources.

Why world modeling issues for code?

The discharge emphasizes two operational capabilities:

1. Execution-trace prediction: given a operate and a hint begin, CWM predicts stack frames (locals) and the executed line at every step by way of a structured format—usable as a “neural debugger” for grounded reasoning with out dwell execution.
2. Agentic coding: multi-turn reasoning with device use in opposition to actual repos, verified by hidden assessments and patch similarity rewards; the setup trains the mannequin to localize faults and generate end-to-end patches (git diff) somewhat than snippets.

Some particulars value noting

• Tokenizer: Llama-3 household with reserved management tokens; reserved IDs are used to demarcate hint and reasoning segments throughout SFT.
• Consideration format: the 3:1 native:world interleave is repeated throughout the depth; long-context coaching happens at giant token batch sizes to stabilize gradients.
• Compute scaling: learning-rate/batch measurement schedules are derived from inner scaling-law sweeps tailor-made for long-context overheads.

Abstract

CWM is a realistic step towards grounded code technology: Meta ties a 32B dense transformer to execution-trace studying and agentic, test-verified patching, releases intermediate/post-trained checkpoints, and gates utilization underneath the FAIR Non-Business Analysis License—making it a helpful platform for reproducible ablations on long-context, execution-aware coding with out conflating analysis with manufacturing deployment.

Try the Paper, GitHub Page, and Model on Hugging Face. Be happy to take a look at our GitHub Page for Tutorials, Codes and Notebooks. Additionally, be at liberty to comply with us on Twitter and don’t neglect to hitch our 100k+ ML SubReddit and Subscribe to our Newsletter.