I have a technical concern regarding the reported Test-Suite Accuracy (TS) metric and its implications for semantic generalization.
According to the methodology, TS was evaluated using “distilled test databases” derived from the MySQL classicmodels schema to check semantic equivalence between gold and model-generated queries. However, no details were provided about the degree of schema or data perturbation in these distilled databases, nor about the specific classes of semantic variations tested (e.g., logical equivalence under aliasing, aggregation equivalence, or filter commutativity).
Can the authors clarify what types of semantic variance these test suites covered, and whether they go beyond simple lexical or syntactic variation? Without such specification, the reported TS values, especially the high scores for GPT-4 and LLaMA-3.3, may reflect surface-level consistency rather than deep generalization to unseen or logically equivalent cases.