Anthrogen Introduces Odyssey: A 102B Parameter Protein Language Mannequin that Replaces Consideration with Consensus and Trains with Discrete Diffusion

Anthrogen has launched Odyssey, a household of protein language fashions for sequence and construction technology, protein enhancing, and conditional design. The manufacturing fashions vary from 1.2B to 102B parameters. The Anthrogen’s analysis crew positions Odyssey as a frontier, multimodal mannequin for actual protein design workloads, and notes that an API is in early entry.

What downside does Odyssey goal?

Protein design {couples} amino acid sequence with 3D construction and with useful context. Many prior fashions undertake self consideration, which mixes info throughout the complete sequence without delay. Proteins observe geometric constraints, so lengthy vary results journey via native neighborhoods in 3D. Anthrogen frames this as a locality downside and proposes a brand new propagation rule, referred to as Consensus, that higher matches the area.

Enter illustration and tokenization

Odyssey is multimodal. It embeds sequence tokens, construction tokens, and light-weight useful cues, then fuses them right into a shared illustration. For construction, Odyssey makes use of a finite scalar quantizer, FSQ, to transform 3D geometry into compact tokens. Consider FSQ as an alphabet for shapes that lets the mannequin learn construction as simply as sequence. Purposeful cues can embody area tags, secondary construction hints, orthologous group labels, or brief textual content descriptors. This joint view offers the mannequin entry to native sequence patterns and lengthy vary geometric relations in a single latent house.

Spine change, Consensus as an alternative of self consideration

Consensus replaces international self consideration with iterative, locality conscious updates on a sparse contact or sequence graph. Every layer encourages close by neighborhoods to agree first, then spreads that settlement outward throughout the chain and phone graph. This transformation alters compute. Self consideration scales as O(L²) with sequence size L. Anthrogen studies that Consensus scales as O(L), which retains lengthy sequences and multi area constructs inexpensive. The corporate additionally studies improved robustness to studying price selections at bigger scales, which reduces brittle runs and restarts.

Coaching goal and technology, discrete diffusion

Odyssey trains with discrete diffusion on sequence and construction tokens. The ahead course of applies masking noise that mimics mutation. The reverse time denoiser learns to reconstruct constant sequence and coordinates that work collectively. At inference, the identical reverse course of helps conditional technology and enhancing. You’ll be able to maintain a scaffold, repair a motif, masks a loop, add a useful tag, after which let the mannequin full the remaining whereas preserving sequence and construction in sync.

Anthrogen studies matched comparisons the place diffusion outperforms masked language modeling throughout analysis. The web page notes decrease coaching perplexities for diffusion versus advanced masking, and decrease or comparable coaching perplexities versus easy masking. In validation, diffusion fashions outperform their masked counterparts, whereas a 1.2B masked mannequin tends to overfit to its personal masking schedule. The corporate argues that diffusion fashions the joint distribution of the complete protein, which aligns with sequence plus construction co design.

Key takeaways

1. Odyssey is a multimodal protein mannequin household that fuses sequence, construction, and useful context, with manufacturing fashions at 1.2B, 8B, and 102B parameters.
2. Consensus replaces self consideration with locality conscious propagation that scales as O(L) and reveals strong studying price habits at bigger scales.
3. FSQ converts 3D coordinates into discrete construction tokens for joint sequence and construction modeling.
4. Discrete diffusion trains a reverse time denoiser and, in matched comparisons, outperforms masked language modeling throughout analysis.
5. Anthrogen studies higher efficiency with about 10x much less information than competing fashions, which addresses information shortage in protein modeling.

Odyssey is spectacular mannequin as a result of it operationalizes joint sequence and construction modeling with FSQ, Consensus, and discrete diffusion, enabling conditional design and enhancing underneath sensible constraints. Odyssey scales to 102B parameters with O(L) complexity for Consensus, which lowers price for lengthy proteins and improves learning-rate robustness. Anthrogen studies diffusion outperforming masked language modeling in matched evaluations, which aligns with co-design targets. The system targets multi-objective design, together with efficiency, specificity, stability, and manufacturability. The analysis crew emphasizes information effectivity close to 10x versus competing fashions, which is materials in domains with scarce labeled information.

Try the Paper, and Technical details. Be at liberty to take a look at our GitHub Page for Tutorials, Codes and Notebooks. Additionally, be at liberty to observe us on Twitter and don’t overlook to affix our 100k+ ML SubReddit and Subscribe to our Newsletter. Wait! are you on telegram? now you can join us on telegram as well.