Roei Herzig

Hi there! I am Roei, a Postdoctoral Scholar in the Berkeley AI Research Lab (BAIR) at UC Berkeley, working with Prof. Trevor Darrell. Additionally, I recently worked under the supervision of Prof. Amir Globerson at Tel Aviv University between April 2019 and April 2023. I'm also affiliated as a research scientist at IBM Research AI.

Previously, I graduated magna cum laude from Tel Aviv University with MSc (CS), BSc (CS), and BSc (Physics), and worked as a Machine Learning & Deep Learning researcher at Nexar and Trax Image Recognition for 5 years.

I was awarded the 2023 IAAI Best PhD thesis award for the outstanding thesis in the field of Artificial Intelligence in Israel.

I'm looking for strong Master's and senior undergrads to collaborate and publish in top-tier conferences on Vision & Language, Vision & Robotics, and Video Understanding.

Email  /  Twitter  /  Github  /  LinkedIn  /  CV  /  Google Scholar

Compositionality is a fundamental aspect of human cognition and language understanding. At its core, it is the ability to decompose complex entities or concepts into simpler, more basic components, and then combine these components in various ways in order to create more complex entities. Despite their impressive capabilities, AI models are limited in this area, and they have difficulty solving tasks that require generalization beyond the training distribution. My research goal is to develop compositionality into intelligent machines in order to improve robustness and generalization across a wide range of fields, such as vision, language, and robotics.

I'm a proud father of Adam and Liam and happily married to Esti, my amazing wife and my partner for life. When I'm not working, I'm also a history buff and love learning about science, politics, music, and the two World Wars.

Multimodal Task Vectors Enable Many-Shot Multimodal In-Context Learning

We demonstrate the existence of multimodal task vectors--compact implicit representations of many-shot in-context examples compressed in the model’s attention head-- and leverage them for many-shot in-context learning in LMMs.

TraveLER: A Multi-LMM Agent Framework for Video Question-Answering

We present TraveLER, a modular multi-LMM agent framework for video question-answering that does not require task-specific fine-tuning or annotations. Through interactive question-asking using several agents with different roles, our framework aims to answer the question by collecting relevant information from keyframes.

LLARVA: Vision-Action Instruction Tuning Enhances Robot Learning

We propose LLARVA, a model trained with a novel instruction tuning method that leverages structured prompts to unify a range of robotic configurations and introduces the concept of visual traces to further align the vision and action spaces.

Recursive Visual Programming

We present Recursive Visual Programming (RVP), a new approach of visual programming that simplifies generated routines, allows more efficient problem solving, and manages more complex data structures.

We propose Compositional Chain-of-Thought (CCoT), a novel zero-shot Chain-of-Thought prompting method that utilizes scene graph representations in order to extract compositional knowledge from an LMM.

We present U2Seg, a unified framework for Unsupervised Universal image Segmentation that consistently outperforms previous state-of-the-art methods.

We propose to improve pretrained VLMs, which are usually trained on large-scale image-text pairs, by designing a specialized model architecture and a new training method that utilizes a small set of scene graph annotations from the Visual Genome dataset that are richer and reflect structured visual and textual information.

We propose a fine-tuning approach for automatically treating two factors limiting VL models’ compositional reasoning performance: (i) the caption quality, or in other words 'image alignment', of the texts; and (ii) the level of caption density, which refers to the number of details that appear in the image.

We present PromptonomyViT, a model that leverages a multi-task prompt learning approach for video transformers, where a shared transformer backbone is enhanced with task-specific prompts.

We demonstrate language augmentation techniques for teaching language structure to VL models.

We present SViT (for Structured Video Tokens), a model that utilizes the structure of a small set of images, whether they are within or outside the domain of interest, available only during training for a video downstream task.

We present FETA, a novel benchmark for evaluating and improving Foundation Vision and Language Models performance on expert data tasks, such as technical document understanding.

We present ORViT, an object-centric approach that extends video transformer layers with a block that directly incorporates object representations.

We present a novel self-supervised cross-domain learning method based on semantically aligning all the domains to a common BrAD domain - a learned auxiliary bridge domain as an edge map with image-to-image mappings.

Pretraining transformers to localize potential objects improves object detection.

We introduce the formalism of Action Graphs, a natural and convenient structure representing the dynamics of actions between objects over time. We show we can synthesize goal-oriented videos on the CATER and Something Something datasets and generate novel compositions of unseen actions.

We present a novel model that can inherently learn canonical graph representations and show better robustness to graph size, adversarial attacks, and semantic equivalent, thus generating superior images of complex visual scenes.

We propose a novel compositional action recognition task where the training combinations of verbs and nouns do not overlap with the test set. We show the effectiveness of our approach on the proposed compositional task and a few-shot compositional setting which requires the model to generalize across both object appearance and action category.

We propose an intermediate “graph-like” representation (DSGs) that can be learned in an end-to-end manner from the supervision for a downstream visual reasoning task, which achieves a new state-of-the-art results on Referring Relationships task.

We propose a latent inter-object graph representation for activity recognition that explores the visual interaction between the objects in a self-supervised manner.

We collect a new SKU-110K dataset which takes detection challenges to unexplored territories, and propose a novel mechanism to learn deep overlap rates for each detection.

We propose a novel invariant graph network for mapping images to scene graphs using the permutation invariant property, which achieves a new state-of-the-art results on Visual Genome dataset.

Template: Jon Barron, Last update: 12/2023.