Projects

SGLang-Omni

I contribute audio-visual input handling and evaluation infrastructure. My work covers media preprocessing, mixed-modality input alignment, streaming error handling, and reproducible benchmark execution.

Status

Merged contributions and ongoing development.

vLLM-Omni

I develop evaluation tooling for omni-modal model serving, including multimodal request construction, asynchronous execution, metric accounting, and automated regression tests. The work also covers reproducible run records and continuous integration.

Status

Upstream contribution under review.

Training-Free Multimodal LLM Orchestration

How can an interactive omni-modal assistant coordinate heterogeneous expert models without retraining a monolithic model?

A large language model (LLM) controller schedules vision and audio experts without additional integration training. Shared cross-modal memory supports context reuse, while the interaction layer supports simultaneous listening and response generation, including interruptions.

Role

First author; scheduling, full-duplex speech interaction, and cross-modal memory

Status

Paper, code, and research note available

SocialOmni

Existing omni-model benchmarks underrepresent time-sensitive audio-visual social interaction.

Query-time-bounded audio-visual evidence supports separate evaluation of speaker identification, turn-entry decisions, and response generation. The current release includes 2,000 perception items, 200 interaction-generation items, and results for 11 omni models.

Role

First author

Status

arXiv 2026; under review; code and dataset available

Performance varies across perception, turn entry, and response generation. This offline protocol does not measure persistent streaming state or wall-clock latency.

Long-video Frame Selection

Long-video models spend substantial computation on redundant frames while missing sparse semantic events.

Event anchors and wavelet-detected semantic boundaries guide adaptive frame allocation without training an additional selector.

Role

Fourth author on WFS-SB

Status

WFS-SB accepted; EFS preprint available

Correct While Verifying

Speculative decoding discovers incorrect draft tokens only after final verification, then discards the entire suffix following the first rejection.

Intermediate target-model states predict correct continuations early, seed correction branches, and verify original and corrected drafts in parallel.

Role

Fifth author

Status

AAAI 2027 submission; project note available

SCOPE

Semantic span-level credit assignment for prompt rewriting, with training and image-generation evaluation code.

RoTA-Bench

Interactive 3D projection alignment, with evaluation code and the fixed 674-task dataset.

A2RBench

Generation, formal verification, augmentation, and evaluation of abstract reasoning tasks.

WaveZip

Training-free video token compression, with model integrations, benchmark loaders, and evaluation launchers.

CrystaL

Visual latent reasoning, with training scripts, evaluation tools, and a released model checkpoint.

MLLMEmbed-ReID

Unified cross-modal retrieval and teacher-to-student distillation for edge deployment.

Duplex-MPE

Multi-party full-duplex dialogue evaluation. The project page provides audio examples, results, and benchmark construction details.