Dear Huayang Li, Richard Sproat, Thank you for submitting your manuscript 4006, "Last Words: Boring Problems Revisited", to Computational Linguistics. I regret to inform you that after careful consideration of the reviews and the action editor's recommendation, we have concluded that the paper does not make a sufficiently significant contribution to be accepted for publication in Computational Linguistics. The action editor's meta-review is given below, followed by the individual reviews. I hope you will find the reviewers' feedback useful in developing this work further. Meta-review: While both reviewers acknowledge the merit of the work, they raise serious and complementary concerns about this manuscript. Reviewer 1 raises the limitations of the current experimental setup in terms of the number of models used (currently one), and emphasizes that the findings cannot be fully trusted unless replicated using model variations with different sizes and architectures. Also all tested number systems use a base of 10, which again limits the scope of the work. Reviewer 2 on the other hand raises fundamental questions about the formulation of the study and its relevance to natural languages in general, and questions the impact of the findings and the insight they generate. -- We appreciate your submitting your work to Computational Linguistics and wish you the best in your future research. Sincerely, Professor Wei Lu Nanyang Technological University, Singapore Editor-in-Chief, Computational Linguistics editor@cljournal.org on behalf of Afra Alishahi Action Editor Computational Linguistics Reviewer 1: Recommendation: Revisions Required 1. Bottom-line Decision My recommendation for this paper is: Revise and resubmit for review: Needs more than minor changes If you selected "Revise and resubmit for review" please indicate what kind of revisions are required: Further Work Required 2. What is this paper about? This is a follow-up to a 2022 paper from one of the authors where they argued that some problems such as text normalization and specifically number reading can’t be solved with purely neural models as they suffer from catastrophic errors. The new paper revisits that question and tests two Claude models on a set of artificial number systems of different complexities. They find that Sonnet performs poorly, but Opus was able to learn the number systems near-perfectly from few-shot prompts. 3. Strengths: The difference between Sonnet and Opus is striking. The paper is quite timely given the recent release of Opus 4.8. Designing artificial number reading systems is a neat idea to avoid train/eval set contamination. Critically revisiting their own work after 4 years is something that doesn’t happen often, but it should be encouraged as it sets both papers in a long-term context that is often overlooked. 4. Weaknesses: I get that this paper merely addresses the “feasibility” question, so finding a single successful model (Opus 4.8) is technically enough to answer it. Still, extending the experimental setup to more model versions, model families, and different sizes would make the results less anecdotal and more useful — is Opus 4.8 just a lucky shot, or are all models bigger or newer than Opus 4.8 fine? All tested number systems use a base of 10. Have you considered a) other naturally occurring bases, or b) bases that are not used in any human language to understand the nature of Opus’ generalization capabilities when more than 10 single digit words are used? 5. Substantive Revisions Complete this section if either "Revise and Resubmit" or "Reject" has been recommended. Substantive Revisions to be Required: Test a broader set of models Substantive Revisions to be Encouraged: Test number systems with different bases 6. Minor Revisions Minor Revisions to be Required: Minor Revisions to be Encouraged: 7. Typographic Errors 8. Revision time How much time do you think the authors might need to make the required changes you have requested? 1 month Some other time (please specify): Reviewer 2: Recommendation: Revisions Required 1. Bottom-line Decision My recommendation for this paper is: Accept with minor revisions: Only minor points to be addressed 2. What is this paper about? I read this piece with interest, as it is not merely a scientific article but a reflection on the capabilities of language models to solve problems that, *a priori*, seem uninteresting to the vast majority of the NLP community. In particular, the authors previously argued—and to some extent demonstrated—that language models could not solve certain text normalization problems, specifically the issue of unrecoverable errors arising from verbalizations that are fluent yet semantically incorrect. 3. Strengths: In this new piece, utilizing more advanced models, the authors present several instances of the problem of naming (reading) numerals (in Arabic notation) in an artificial language; as I understand it, these languages possess an associated grammar that could potentially be learned. Here, the authors report the success of certain new models on the proposed languages. 4. Weaknesses: A question that immediately comes to mind is whether the authors conducted experiments aimed at generating a grammar for the proposed languages (using traditional techniques) and exposing the rule-based model solely to a set of examples. A question that immediately comes to mind is whether the authors conducted experiments aimed at generating a grammar for the proposed languages (using traditional techniques) and exposing the rule-based model solely to a set of examples. The second question concerns the universe of discourse: what evidence demonstrates that the problem has been solved when the universe is infinite? The third issue relates to the artificial nature of the languages—a problem that might be better addressed within automata or formal language theory rather than NLP. In this regard, one might ask whether there are not more interesting—and linguistically realistic—problems that could be proposed. Why the new models did better now? How do you explain nesides speculation? The conclusion could go further and point to existing problems we should concentrate on, even if of interest for a small community. 5. Substantive Revisions Complete this section if either "Revise and Resubmit" or "Reject" has been recommended. Substantive Revisions to be Required: Substantive Revisions to be Encouraged: From a formal standpoint, there is room for improvement, particularly regarding the description of the grammar; it appears neither exhaustive nor clearly explained. 6. Minor Revisions Minor Revisions to be Required: Several errors were found during your experiments which you acknowledge, but it is little clear how they affected. Minor Revisions to be Encouraged: 7. Typographic Errors 8. Revision time How much time do you think the authors might need to make the required changes you have requested? 2 months Some other time (please specify):