Abstract
Large language models are increasingly applied to materials science reasoning, yet their behavior under physically structured distribution shifts remains poorly understood. We introduce SCALAR, a benchmark for evaluating geometric scale generalization and its connection to structural hallucination, consistency, and reasoning in materials foundation models. Starting from canonical crystal representations, models reason about derived nanoparticle structures obtained through supercell expansion and spherical carving from literature-grounded bulk unit cells with documented provenance across length scales spanning 25 to 18 123 atoms: 1743 base structures (83 materials × 21 radii), augmented by split-aware rotations to ≈100 000 structures across training, in-distribution and out-of-distribution evaluation splits. SCALAR defines three tasks: (i) CIF → property prediction; (ii) a chain-of-thought variant with explicit physics-grounded reasoning; and (iii) inverse retrieval identifying crystals from candidates given target properties. Outputs are evaluated via structured metrics capturing numeric error, hallucination, cross-prompt consistency, scale-trend reasoning, output validity, and retrieval fidelity. Experiments across diverse foundation models reveal large, model-dependent shifts under explicit reasoning, often reducing hallucination and error, but frequently destabilizing consistency or validity, demonstrating that scale generalization cannot be inferred from accuracy alone.
| Original language | English |
|---|---|
| Journal | Digital Discovery |
| Early online date | Oct 2026 |
| DOIs | |
| Publication status | E-pub ahead of print - Oct 2026 |
Fingerprint
Dive into the research topics of 'SCALAR: quantifying structural hallucination, consistency, and reasoning gaps in materials foundation models'. Together they form a unique fingerprint.Cite this
- APA
- Author
- BIBTEX
- Harvard
- Standard
- RIS
- Vancouver