SocialOmni: Audio-Visual Social Interactivity

Social interaction is naturally multimodal. Facial expressions, gaze, body movement, speech tone, timing, and scene context all matter. A model that reads only text or isolated frames can miss the social signals people use to understand what is happening.

SocialOmni is built around this gap. It evaluates omni models on audio-visual social interactivity, asking whether a system can reason over synchronized cues rather than treating each modality as a separate input stream.

Evaluation scope

• Audio-visual alignment in social scenes.
• Interaction understanding beyond static object recognition.
• Practical evaluation for omni models that claim broad multimodal capability.
• Dataset and code artifacts that make the benchmark easy to inspect and reuse.

Links

• arXiv
• Code
• Dataset