Roei Herzig

Hi there! I am Roei, a Postdoctoral Scholar in the Berkeley AI Research Lab (BAIR) at UC Berkeley, working with Prof. Trevor Darrell. Additionally, I recently worked under the supervision of Prof. Amir Globerson at Tel Aviv University between April 2019 and April 2023. I'm also affiliated as a research scientist at IBM Research AI.

Previously, I graduated magna cum laude from Tel Aviv University with MSc (CS), BSc (CS), and BSc (Physics), and worked as a Machine Learning & Deep Learning researcher at Nexar and Trax Image Recognition for 5 years.

I was awarded the 2023 IAAI Best PhD thesis award for the outstanding thesis in the field of Artificial Intelligence in Israel.

I'm looking for strong Master's and senior undergrads to collaborate and publish in top-tier conferences on Vision & Language, Vision & Robotics, and Video Understanding.

Email  /  Twitter  /  BlueSky  /  Github  /  LinkedIn  /  CV  /  Google Scholar

Short Bio

Short Research Statement

• [June 2025] Outstanding Reviewer at CVPR 2025 (ranked in the top 5%, 711 out of 12,593 reviewers).
• [June 2025] Two invited talks on "Structured Physical Intelligence" at the T4V and Agents workshops, CVPR 2025.
• [June 2025] Serving as panel moderator at the MMFM3 Workshop, CVPR 2025.
• [May 2025] Our Paper ARM4R, was accepted to ICML 2025
• [April 2025] Our Workshop MMFM4, was accepted to ICCV 2025.
• [April 2025] I gave a talk on "Structured Robotic Visual Intelligence" at the Spring 2025 BAIR Robotics Workshop.

Pre-training Auto-regressive Robotic Models with 4D Representations

We propose ARM4R, an Autoregressive Robotic Model that leverages low-level 4D Representations learned from human video data to yield a better robotic model.

In-Context Learning Enables Robot Action Prediction in LLMs

We introduce RoboPrompt, a framework that enables off-the-shelf text-only LLMs to directly predict robot actions through ICL without training.

We propose LLARVA, a model trained with a novel instruction tuning method that leverages structured prompts to unify a range of robotic configurations and introduces the concept of visual traces to further align the vision and action spaces.

Activation Reward Models for Few-Shot Model Alignment

SAVs is a finetuning-free method that leverages sparse attention head activations (fewer than 5% of heads) in LMMs as powerful feature representations for vision-language classification tasks, achieving state-of-the-art performance compared to both few-shot and finetuned baselines.

We introduce Granite Vision, a lightweight large language model with vision capabilities, specifically designed to excel in enterprise use cases, particularly in visual document understanding

We demonstrate the existence of multimodal task vectors--compact implicit representations of many-shot in-context examples compressed in the model’s attention head-- and leverage them for many-shot in-context learning in LMMs.

We present TraveLER, a modular multi-LMM agent framework for video question-answering that does not require task-specific fine-tuning or annotations. Through interactive question-asking using several agents with different roles, our framework aims to answer the question by collecting relevant information from keyframes.

We propose Compositional Chain-of-Thought (CCoT), a novel zero-shot Chain-of-Thought prompting method that utilizes scene graph representations in order to extract compositional knowledge from an LMM.

We propose to improve pretrained VLMs, which are usually trained on large-scale image-text pairs, by designing a specialized model architecture and a new training method that utilizes a small set of scene graph annotations from the Visual Genome dataset that are richer and reflect structured visual and textual information.

We propose a fine-tuning approach for automatically treating two factors limiting VL models’ compositional reasoning performance: (i) the caption quality, or in other words 'image alignment', of the texts; and (ii) the level of caption density, which refers to the number of details that appear in the image.

We demonstrate language augmentation techniques for teaching language structure to VL models.

We present SViT (for Structured Video Tokens), a model that utilizes the structure of a small set of images, whether they are within or outside the domain of interest, available only during training for a video downstream task.

We present ORViT, an object-centric approach that extends video transformer layers with a block that directly incorporates object representations.

We introduce the formalism of Action Graphs, a natural and convenient structure representing the dynamics of actions between objects over time. We show we can synthesize goal-oriented videos on the CATER and Something Something datasets and generate novel compositions of unseen actions.

We present a novel model that can inherently learn canonical graph representations and show better robustness to graph size, adversarial attacks, and semantic equivalent, thus generating superior images of complex visual scenes.

We propose a novel compositional action recognition task where the training combinations of verbs and nouns do not overlap with the test set. We show the effectiveness of our approach on the proposed compositional task and a few-shot compositional setting which requires the model to generalize across both object appearance and action category.

We propose a latent inter-object graph representation for activity recognition that explores the visual interaction between the objects in a self-supervised manner.

We collect a new SKU-110K dataset which takes detection challenges to unexplored territories, and propose a novel mechanism to learn deep overlap rates for each detection.

We propose a novel invariant graph network for mapping images to scene graphs using the permutation invariant property, which achieves a new state-of-the-art results on Visual Genome dataset.