A2RBench: Abstract Reasoning with Formal Verification

Reasoning benchmarks are useful only when the answer signal can be trusted. For abstract reasoning tasks, manual construction is expensive, while uncontrolled generation can introduce ambiguous or unverifiable examples.

A2RBench explores an automatic paradigm for generating formally verifiable abstract reasoning benchmarks. The goal is to scale task construction while keeping correctness explicit and auditable.

Approach

• It separates benchmark generation from informal human intuition.
• Formal verification makes answer checking more reliable.
• Automatic generation expands coverage across reasoning patterns.
• The paradigm is useful for evaluating models that claim abstract reasoning ability.

Link

• arXiv