Skip to main navigation Skip to search Skip to main content

SCALAR: quantifying structural hallucination, consistency, and reasoning gaps in materials foundation models

  • Can Polat
  • , Mustafa Kurban*
  • , Erchin Serpedin
  • , Hasan Kurban*
  • *Corresponding author for this work
  • Texas A&M University
  • Texas A&M University at Qatar
  • Ankara University

Research output: Contribution to journal › Article › peer-review

Abstract

Large language models are increasingly applied to materials science reasoning, yet their behavior under physically structured distribution shifts remains poorly understood. We introduce SCALAR, a benchmark for evaluating geometric scale generalization and its connection to structural hallucination, consistency, and reasoning in materials foundation models. Starting from canonical crystal representations, models reason about derived nanoparticle structures obtained through supercell expansion and spherical carving from literature-grounded bulk unit cells with documented provenance across length scales spanning 25 to 18 123 atoms: 1743 base structures (83 materials × 21 radii), augmented by split-aware rotations to ≈100 000 structures across training, in-distribution and out-of-distribution evaluation splits. SCALAR defines three tasks: (i) CIF → property prediction; (ii) a chain-of-thought variant with explicit physics-grounded reasoning; and (iii) inverse retrieval identifying crystals from candidates given target properties. Outputs are evaluated via structured metrics capturing numeric error, hallucination, cross-prompt consistency, scale-trend reasoning, output validity, and retrieval fidelity. Experiments across diverse foundation models reveal large, model-dependent shifts under explicit reasoning, often reducing hallucination and error, but frequently destabilizing consistency or validity, demonstrating that scale generalization cannot be inferred from accuracy alone.

Original languageEnglish
JournalDigital Discovery
Early online dateOct 2026
DOIs
Publication statusE-pub ahead of print - Oct 2026

Fingerprint

Dive into the research topics of 'SCALAR: quantifying structural hallucination, consistency, and reasoning gaps in materials foundation models'. Together they form a unique fingerprint.

Cite this