This Week's 5 Most Notable AI Research Papers - Week 46

1. Who Evaluates AI’s Social Impacts? Mapping Coverage and Gaps in First- and Third-Party Evaluations

Key points:

1. First-party social-impact evaluations are shallow and shrinking.
2. Independent evaluators cover more ground but lack access to provider data.
3. Key dimensions like moderation labour and compute cost are under-reported.
4. The current evaluation ecosystem leaves gaps in governance and accountability.
5. Multi-stakeholder roadmap proposed: strengthening disclosure rules, supporting independent evaluators and creating a shared infrastructure for comparing assessments.

As AI models become more embedded in daily life, there is an increasing reliance on evaluations to measure their societal effects, ranging from fairness and environmental impact to labour and privacy risks. This research provides a first comprehensive overview of how these evaluations are conducted, comparing reports from developers at the time of model release with independent evaluations carried out afterwards. The authors examined almost 200 documents from AI companies and a similar number from external organisations.

Their analysis reveals a consistent trend: companies tend to publish limited information about social harms, while external researchers cover a broader range of issues, yet still lack the necessary data for a comprehensive overview. Crucial indicators, such as the origin of training data, the human labour involved in moderation and the computational resources used, are rarely disclosed. The paper argues that this fragmented landscape weakens accountability and makes it harder for policymakers to understand real-world risks.

Read the full article here

Authors: Anka Reuel, Avijit Ghosh, Jenny Chim, Andrew Tran, Yanan Long, Jennifer Mickel, Usman Gohar, Srishti Yadav et al.

2. Introducing Nested Learning: A new ML paradigm for continual learning

Key points:

1. Nested Learning views model architecture + optimisation as multi-level nested problems.
2. The HOPE architecture shows improved long-context and continual-learning performance.
3. Continuum Memory Systems allow modules to update at different tempos, reducing forgetting.
4. AI systems may increasingly shift from static training to continual learning regimes.
5. Governance and lifecycle design of models will matter more as AI becomes adaptive over time.

Google Research has presented Nested Learning, a new framework designed to help machine-learning models update themselves over time without forgetting what they have previously learnt. Unlike conventional approaches, where the architecture is fixed first and then optimised afterwards, this method treats a model as a collection of multiple learning processes operating at different speeds and levels of detail.

The team introduced a prototype system called HOPE, which demonstrates that this structure can mitigate catastrophic forgetting and process long, information-rich sequences more efficiently than traditional designs. These results suggest a promising path towards AI systems that can adapt continuously and at multiple levels, more closely resembling the way humans learn over different timescales.

Read the full article here

Authors: Ali Behrouz, Vahab Mirrokni

3. From Memorization to Reasoning in the Spectrum of Loss Curvature

Key points:

1. Loss curvature distinguishes between memorisation vs reasoning in LLMs.
2. Sharper curvature = memorisation; flatter curvature = reasoning & generalisation.
3. The transition to reasoning aligns with deeper layers and training evolution.
4. Curvature metrics can serve as new tools for model evaluation, interpretability and governance.

As LLMs increase in size and capability, a key question arises: how do they transition from memorising training data to reasoning about new inputs?

This study introduces the concept of loss curvature, a mathematical measure of how a model’s loss landscape behaves, as a means of exploring this transition. By analysing how loss curvature varies across tasks and layers, the authors demonstrate that memorisation and reasoning correspond to distinct curvature regimes. Their experiments reveal that deeper layers with sharper curvature are associated with memorisation, whereas flatter curvature is associated with more generalisable reasoning. These findings suggest that training dynamics and architectural choices significantly impact where and how reasoning emerges in LLMs, providing an insight for designing more robust and trustworthy AI systems.

Read the full article here

Authors: Jack Merullo, Srihita Vatsavaya, Lucius Bushnaq, Owen Lewis

4. How Do AI Agents Do Human Work? Comparing AI and Human Workflows Across Diverse Occupations

Key points:

1. First direct workflow comparison: 48 workers vs 4 agent systems across 16 occupations.
2. AI agents excel at structured, rule-based tasks but struggle with ambiguity and adaptation.
3. Workflow integration (how tasks are decomposed, how errors are handled, and how humans intervene) is key to deployment success.
4. Overlooking workflow dynamics can lead to hidden failure risks when deploying AI in workplaces.
5. Policies and business strategies must emphasise hybrid human-agent workflows, not full automation blanket assumptions.

This paper provides a comprehensive analysis of the similarities and differences between the work processes of AI agents and humans across a wide range of occupations. The researchers collected detailed data from 48 individuals and four AI systems spanning 16 job categories. Their analysis shows that, while AI agents can perform highly efficiently in tasks with a clear structure, they still fall short in areas that demand flexibility, troubleshooting or situational judgement. The authors argue that integrating AI agents into workplaces requires more than merely assessing task performance; it also necessitates careful consideration of how these systems align with existing workflows, how responsibilities are distributed and where potential breakdowns may arise.

Read the full article here

Authors: Zora Zhiruo Wang, Yijia Shao, Omar Shaikh, Daniel Fried, Graham Neubig, Diyi Yang

5. GTIG AI Threat Tracker: Advances in Threat Actor Usage of AI Tools

Key points:

1. Malicious actors are using generative AI mid-execution in malware.
2. Illicit AI-tool markets are emerging, lowering the barrier to sophisticated attacks.
3. AI is now embedded across reconnaissance, phishing, malware and exfiltration.
4. Standard defenses need to evolve; governance and public awareness are critical.

Threat actors’ use of artificial intelligence is undergoing a notable transformation, according to the Google Cloud analysis. Instead of relying on AI simply to enhance performance, attackers are now integrating AI-driven malware directly into their activities. Google’s Threat Intelligence Group (GTIG) reports that families like PROMPTFLUX and PROMPTSTEAL draw on large language models in real time to craft malicious scripts and disguise their activity to avoid detection. The update also explains that adversaries are increasingly turning to tailored social-engineering strategies to circumvent the protective safeguards built into generative-AI systems such as Gemini.

Read the full article here

Authors: Google Threat Intelligence Group