TII Releases Falcon Notion: A 0.6B-Parameter Early-Fusion Transformer for Open-Vocabulary Grounding and Segmentation from Pure Language Prompts

Within the present panorama of pc imaginative and prescient, the usual working process entails a modular ‘Lego-brick’ strategy: a pre-trained imaginative and prescient encoder for characteristic extraction paired with a separate decoder for activity prediction. Whereas efficient, this architectural separation complicates scaling and bottlenecks the interplay between language and imaginative and prescient.

The Know-how Innovation Institute (TII) analysis crew is difficult this paradigm with Falcon Notion, a 600M-parameter unified dense Transformer. By processing picture patches and textual content tokens in a shared parameter house from the very first layer, TII analysis crew has developed an early-fusion stack that handles notion and activity modeling with excessive effectivity.

The Structure: A Single Stack for Each Modality

The core design of Falcon Notion is constructed on the speculation {that a} single Transformer can concurrently study visible representations and carry out task-specific era^.

Hybrid Consideration and GGROPE

Not like commonplace language fashions that use strict causal masking, Falcon Notion employs a hybrid consideration technique^{. Picture tokens attend to one another bidirectionally to construct a world visible context, whereas textual content and activity tokens attend to all previous tokens (causal masking) to allow autoregressive prediction.}

To keep up 2D spatial relationships in a flattened sequence, the analysis crew makes use of 3D Rotary Positional Embeddings. This decomposes the top dimension right into a sequential element and a spatial element utilizing Golden Gate ROPE (GGROPE). GGROPE permits consideration heads to take care of relative positions alongside arbitrary angles, making the mannequin sturdy to rotation and facet ratio variations.

Minimalist Sequence Logic

The fundamental architectural sequence follows a Chain-of-Notion format:

[Image] [Text] ... ^.

This ensures that the mannequin resolves spatial ambiguity (place and measurement) as a conditioning sign earlier than producing the ultimate segmentation masks^.

Engineering for Scale: Muon, FlexAttention, and Raster Ordering

TII analysis crew launched a number of optimizations to stabilize coaching and maximize GPU utilization for these heterogeneous sequences.

• Muon Optimization: The analysis crew report that using the Muon optimizer for specialised heads (coordinates, measurement, and segmentation) led to decrease coaching losses and improved efficiency on benchmarks in comparison with commonplace AdamW.
• FlexAttention and Sequence Packing: To course of photographs at native resolutions with out losing compute on padding, the mannequin makes use of a scatter-and-pack technique. Legitimate patches are packed into fixed-length blocks, and FlexAttention is used to limit self-attention inside every picture pattern’s boundaries.
• Raster Ordering: When a number of objects are current, Falcon Notion predicts them in raster order (top-to-bottom, left-to-right). This was discovered to converge sooner and produce decrease coordinate loss than random or size-based ordering.

The Coaching Recipe: Distillation to 685GT

The mannequin makes use of multi-teacher distillation for initialization, distilling information from DINOv3 (ViT-H) for native options and SigLIP2 (So400m) for language-aligned options^{. Following initialization, the mannequin undergoes a three-stage notion coaching pipeline totaling roughly 685 Gigatokens (GT):}

1. In-Context Itemizing (450 GT): Studying to ‘listing’ the scene stock to construct world context.
2. Activity Alignment (225 GT): Transitioning to independent-query duties utilizing Question Masking to make sure the mannequin grounds every question solely on the picture.
3. Lengthy-Context Finetuning (10 GT): Brief adaptation for excessive density, growing the masks restrict to 600 per expression.

Throughout these levels, the task-specific serialization is used:

expr1 expr2 ^.

The and tokens power the mannequin to decide to a binary choice on an object’s existence earlier than localization^.

PBench: Profiling Capabilities Past Saturated Baselines

To measure progress, TII analysis crew launched PBench, a benchmark that organizes samples into 5 ranges of semantic complexity to disentangle mannequin failure modes.

Major Outcomes: Falcon Notion vs. SAM 3 (Macro-F1)

Falcon Notion considerably outperforms SAM 3 on advanced semantic duties, significantly exhibiting a +21.9 level acquire on spatial understanding (Stage 3)^.

FalconOCR: The 300M Doc specialist

TII crew additionally prolonged this early-fusion recipe to FalconOCR, a compact 300M-parameter mannequin initialized from scratch to prioritize fine-grained glyph recognition. FalconOCR is aggressive with a number of bigger proprietary and modular OCR programs:

• olmOCR: Achieves 80.3% accuracy, matching or exceeding Gemini 3 Professional (80.2%) and GPT 5.2 (69.8%).
• OmniDocBench: Reaches an general rating of 88.64, forward of GPT 5.2 (86.56) and Mistral OCR 3 (85.20), although it trails the highest modular pipeline PaddleOCR VL 1.5 (94.37).

Key Takeaways

• Unified Early-Fusion Structure: Falcon Notion replaces modular encoder-decoder pipelines with a single dense Transformer that processes picture patches and textual content tokens in a shared parameter house from the primary layer. It makes use of a hybrid consideration masks—bidirectional for visible tokens and causal for activity tokens—to behave concurrently as a imaginative and prescient encoder and an autoregressive decoder.
• Chain-of-Notion Sequence: The mannequin serializes occasion segmentation right into a structured sequence $(langle coordrangle rightarrow langle sizerangle rightarrow langle segrangle)$, which forces it to resolve spatial place and measurement as a conditioning sign earlier than producing the pixel-level masks.
• Specialised Heads and GGROPE: To handle dense spatial information, the mannequin makes use of Fourier Characteristic encoders for high-dimensional coordinate mapping and Golden Gate ROPE (GGROPE) to allow isotropic 2D spatial consideration. The Muon optimizer is employed for these specialised heads to steadiness studying charges towards the pre-trained spine.
• Semantic Efficiency Positive aspects: On the brand new PBench benchmark, which disentangles semantic capabilities (Ranges 0-4), the 600M mannequin demonstrates important positive aspects over SAM 3 in advanced classes, together with a +13.4 level lead in OCR-guided queries and a +21.9 level lead in spatial understanding.
• Excessive-Effectivity OCR Extension: The structure scales all the way down to Falcon OCR, a 300M-parameter mannequin that achieves 80.3% on olmOCR and 88.64 on OmniDocBench. It matches or exceeds the accuracy of a lot bigger programs like Gemini 3 Professional and GPT 5.2 whereas sustaining excessive throughput for large-scale doc processing.

Take a look at the Paper, Model Weight, Repo and Technical details.  Additionally, be happy to observe us on Twitter and don’t neglect to hitch our 120k+ ML SubReddit and Subscribe to our Newsletter. Wait! are you on telegram? now you can join us on telegram as well.