I am closing this project as it has been subsumed into the ARIA Opportunity Seed on Mathematics for Safe AI: ARIA Opportunity Seeds – SafePlanBench & Logically Constrained Reinforcement Learning.

The funds from this Manifund grant were primarily used for compute costs, in particular API credits supporting initial experimental evaluations of LLM-based agents.

Since the project’s inception, its core direction has evolved into a broader research agenda focused on formal guarantees for safety in AI systems. In line with this shift, a concrete outcome of this work is a paper accepted at the Symposium on AI Verification: Value Functions as Supermartingale Certificates. In this work, we introduce a method for generating proof certificates that guarantee a reinforcement learning policy satisfies a specified linear temporal logic (LTL) specification.

We are currently extending this line of work to more directly address the original motivation of the project, namely safety guarantees for LLM-based agents.

Overall, while SafePlanBench in its original form is no longer the central focus, the project has contributed to a broader line of work on mathematically grounded safety guarantees for learning-based and agentic systems.

I consider Integral Altruism to be in the process of developing a coherent set of critiques of existing EA approaches to positive impact. These critiques were not structurally organised before and seem essential to finding wiser ways of making the world a better place. Alongside this, it has been building a network and community around these ideas.

I met Euan during the 2024 PIBBSS fellowship and briefly engaged with Integral Altruism's activities last year, and I think it would be of great value to have him running the activities full time.

What progress have you made since your last update?

We have developed an advanced version of our problem generation tool, which enables the creation of planning problem instances in a "gripper-like" environment, modeled after the classical STRIPS planning domain, where a robot moves between rooms and picks up, drops, or interacts with objects, with configurable numbers of objects, locations, and safety constraints.

This tool allows us to systematically vary the size of the problem and the number of safety constraints, supporting the construction of a flexible and scalable benchmark.

Initial experimental runs using this setup have also provided us with important conceptual clarity. In particular, we've identified a promising direction for contribution: characterizing the computational complexity of safety constraints. Our aim is to link different classes of constraints to known complexity classes in automated planning — and to use this connection to better understand and empirically predict how likely it is that state-of-the-art frontier models will violate these constraints, depending on their complexity.

What are your next steps?

• Formalize our theoretical framing around safety constraint complexity and its empirical implications, with the goal of producing a framework that connects symbolic planning theory with LLM behavior in practice.

• Finalize the SafePlanBench benchmark by expanding the set of safety constraint types and further diversifying problem templates.

• Begin large-scale evaluation of instruction-tuned and reasoning LLMs using the benchmark.