Agentic Code Execution instead of tool calls
Elixir and function docs are perfect for agentic code generation instead of tool call chaining
This post is my own words and not generated by LLM.
TL;DR
LLM generates code instead of calling tools one by one.
Elixir docs contain the samples needed by LLM to generate code.
Virtual file system for agent to browse sample code.
Security is achieved with function wrapping and Dune library.
Working with LLM agents, most of us have found tool calling slow and wastage of LLM tokens. Moreover, building the tools usually requires adding another layer to encapsulate the business logic with a request/response schema and then maintaining it.
Lately, there has been a lot of buzz around code execution with MCP to speed up multi-step tool calling. The idea is to simply let LLM generate code using samples provided and execute it directly instead of sequential tool calling. This optimises the speed, reliability and token economics.
Elixir and Agentic Code Generation
I had already written an agentic framework in Elixir as it is a very suitable ecosystem for LLM Agents. Apparently, Elixir also really shines in agentic code generation as the documentation of functions comes with examples as to how to call that code. These examples can be tested using doctest to make sure the code changes and documentation don’t deviate. This coupling reduces the need to provide samples separately and maintain them.
Having examples meant that I could simply look for ## Examples heading in the function’s @doc tag, thus avoiding the need to write separate tool files with sample code.
Virtual File System
I ended up with a framework where I created a virtual file system that agent could browse to list resources and tool definitions.
Note: I decided to go with .exs extension to add a hint to the LLM that this is elixir code. I have no proof if it increased accuracy of the generated code.
File Browsing Tools
Next I created 3 tools list_resources, list_files and read_file. These tools allowed agent to browse the virtual file system and read code in each file.
list_resources
list_files
read_file
Security in Code Execution
Code generation and evaluation by LLM Agents brings a huge security risk, where a malicious prompt can perform unwanted activities on the system.
I devised 4 layers of security to reduce chances of malicious code execution.
Layer 1 - Guard rails in the system prompt to disallow importing/aliasing of other modules or accepting ```elixir tag code.
Layer 2 - Make sure all functions being called accept Auth Scope so that the actions are performed within that organisation or tenant. Use %MyApp.Accounts.Scope{} pattern on the function call signatures.
Layer 3 - Wrap the generated code inside an anonymous function to provide Scope thus making sure that scope can be passed by the the code executor and not defined during code generation.
Layer 4 - Sandbox the generated code using Dune and maintain a strict allow list. Dune does not allow creating of structs which means that %Scope{} cannot be called and can only be passed by executor.
Amazing read!