TL;DR

• LLM generates code instead of calling tools one by one.

• Elixir docs contain the samples needed by LLM to generate code.

• Virtual file system for agent to browse sample code.

• Security is achieved with function wrapping and Dune library.

  ---

Working with LLM agents, most of us have found tool calling slow and wastage of LLM tokens. Moreover, building the tools usually requires adding another layer to encapsulate the business logic with a request/response schema and then maintaining it.

Lately, there has been a lot of buzz around code execution with MCP to speed up multi-step tool calling. The idea is to simply let LLM generate code using samples provided and execute it directly instead of sequential tool calling. This optimises the speed, reliability and token economics.

Elixir and Agentic Code Generation

I had already written an agentic framework in Elixir as it is a very suitable ecosystem for LLM Agents. Apparently, Elixir also really shines in agentic code generation as the documentation of functions comes with examples as to how to call that code. These examples can be tested using doctest to make sure the code changes and documentation don’t deviate. This coupling reduces the need to provide samples separately and maintain them.

Having examples meant that I could simply look for ## Examples heading in the function’s @doc tag, thus avoiding the need to write separate tool files with sample code.

Virtual File System

I ended up with a framework where I created a virtual file system that agent could browse to list resources and tool definitions.

Note: I decided to go with .exs extension to add a hint to the LLM that this is elixir code. I have no proof if it increased accuracy of the generated code.

File Browsing Tools

Next I created 3 tools list_resources, list_files and read_file. These tools allowed agent to browse the virtual file system and read code in each file.

list_resources

list_files

read_file

Security in Code Execution

Code generation and evaluation by LLM Agents brings a huge security risk, where a malicious prompt can perform unwanted activities on the system.

I devised 4 layers of security to reduce chances of malicious code execution.

Layer 1 - Guard rails in the system prompt to disallow importing/aliasing of other modules or accepting ```elixir tag code.

Layer 2 - Make sure all functions being called accept Auth Scope so that the actions are performed within that organisation or tenant. Use %MyApp.Accounts.Scope{} pattern on the function call signatures.

Layer 3 - Wrap the generated code inside an anonymous function to provide Scope thus making sure that scope can be passed by the the code executor and not defined during code generation.

Layer 4 - Sandbox the generated code using Dune and maintain a strict allow list. Dune does not allow creating of structs which means that %Scope{} cannot be called and can only be passed by executor.

Share

Leave a comment

This project made us nervous as safety of the passengers was of extreme importance. The nervousness was amplified by a hard deadline of 6 weeks after which the video shooting had to be done. The issue of tight deadlines is very common for most experiential projects. The creative teams usually take a lot of time in refining and negotiating which leaves very little time for the actual execution ( Or may be they just like to see us suffer).

We quickly went into action and started planning the development.

Quest to find a BCI device

Our first challenge was finding a BCI device that we could easily program. Neither of us had ever worked with a BCI (Brain Computer Interface) device and it was our understanding that they are pretty expensive to purchase.

EMOTIV to the rescue

We googled for affordable BCI devices and within a short time found out about EMOTIV. The pricing was reasonable, around 300 USD for EMOTIV insight product. EMOTIV was fairly new at that time, just out of kickstarter. I quickly read the SDK documentation and decided to go for EMOTIV.

We ordered 5 of EMOTIV Insight devices. 2 of those were delivered to me in Amsterdam while the other 3 went to TheAlphabetCollective in London.

Mock it till you make it

As there was going to be some lead time in the delivery of devices, we started working on the code and hardware assembly by simulating the BCI device. In other words, made more buttons and notches in the UI :)

Arduino and Actuators

Matt took care of the hardware. He procured actuators that were more powerful than needed and could possibly destroy the car. I guess he didn’t listen to the advice from Spiderman’s uncle about power and responsibility. He connected those actuators to Arduino controller, which sends the commands on the serial port.

Software for BCI

My part was to write the software for interaction with the BCI device and the Arduino controller. We had a few Windows laptops lying around from previous jobs and decided to aim for Windows OS. I decided to go with Python as a language for faster development, plus EMOTIV SDK support for Python was excellent. We also wanted to create a GUI control panel for manual safety overrides and visualisation of the overall working of the software (e.g., actuator position, BCI metrics). The UI framework seemed a bit tricky to get it right at first but I settled on wxPython framework. In retrospect, I would not choose Python for such a realtime application due to Python GIL. I ended up using queues to get some level of concurrency and a responsive UI.

Safety concerns

Safety was of high importance as we couldn’t risk car going out of control if software misbehaved. We created 4 levels of failsafes.

1. Bake/Release buttons on the GUI on the Windows laptop that would release accelerator and press the brake.

2. A bigger reset button on the Arduino.

3. A switch that sat between Arduino and Actuators.

4. Lastly, the rig with actuators could be released manually with one hand.

If bugs could kill

Once the simulated code was completed, I flew a few days before the event to test integration with real actuators. While testing we realised that every few minutes the actuators went crazy and pushed their full length of 12 cm out. If we had tested with a real car, it would have accelerated to max which could have threatened the life of the passengers. Luckily, we figured out the issue quickly. Apparently, we were sending commands to the actuators faster than they could accept on the serial bitrate. This caused corruption of data on the serial port towards actuators. We fixed it by making Python code wait for an ACK from the Arduino code before sending the next accelerate/brake value.

Shooting day

Shooting took place at an airfield. We configured 2 cars, a day before the shooting and drove them around with the power of our minds. Even though, we knew how the whole system worked, it was still a very strange feeling in the stomach when car moved on its own due to our focus.

We had a separate rig on a table to train the talent (advertising term for the main characters) on the system before sitting in the car. The rest of the shooting went well and we had a lot of fun.

How it all started?

I occasionally help my friend Matt from TheAlphabetCollective agency/studio in their experiential marketing projects. In 2019 they were asked via BrandFuel to help build one of the Android installations for MobileWorldCongress in Barcelona. My friends asked me if I could help in building the software side.

The requirement of the project was to play a video in loop on ~140 devices during the whole of MWC 2019 event that lasted for 4 days. The videos had to be switched between day and night.

We had about 2 months to build out the software, hardware and the frame to place all the devices. A friend designed the iron rig and it was assembled by us at the convention. This images shows the rig assembled and being installed.

Challenges and Solutions

We knew right from the start that this was not going to be an easy project. My friend and I come from a telecom background. We built together long-range WiFi to GSM networks as well as media distribution systems.

We faced following technical challenges during the project.

1 - Fragmented Android Devices and OS versions:

We were given a list of android devices that included Android TV, Tablets, Smart Phones and Smart Watches. The devices were from various manufacturers including Huawei, Samsung, Nokia, LG, Xiaomi, Motorola and a few other lesser known brands.

Android is a highly fragmented platform and we knew this was going to be a problem for software development.

Solution: Android SDK 7

We decided early on that we will have to build apps using Android SDK instead of going with ReactNative or Xamarin like tools to stay close to the OS to be able to support devices even if it meant creating various versions. A good thing about these projects is that they are short lived and you don’t have to support it for years to come. We also constrained our android app to Android 7 (API level 24) to support all the devices.

We ended up creating only 3 versions of the app mainly due to UI differences all in the same project with different UI code. TV, Mobile and SmartWatch.

Here is a picture of software displaying DEBUG screen.

2 - Connectivity

We knew from previous experience that connectivity provided by the venues in such events is never stable enough for a realtime application.

Solution - Our own LAN

We decided to build our own local network. We checked all devices that we could connect via ethernet and the rest had to go on WiFi. We created 3 networks with a mix of WiFi and LAN to provide reliability and capacity for all the devices.

3 - Content delivery to the devices

The playback on all devices had to be in sync to create a unified bigger screen effect. We went through a few approaches to get the best result.

Discarded Solution - WebRTC(RTSP) playback

We discussed for a while the possibility to stream videos to the phones using WebRTC or RTSP streaming. In this approach, the devices would connect to the streaming server on the LAN network and stream videos realtime. We discarded this initial idea as we were concerned about the connection reliability of the devices on WiFi. Moreover, the videos had to be played in a loop and if one device started a second later to play the video due to its Hardware speed, the video would have been off by a second and over time, would have gone completely out of sync.

Solution - Pre-loaded files

We decided to pre-load video files on the devices as they were not going to change that often. We also decided that we would get the videos cropped for each device for its position on the grid. This way the video size would be smaller and software would not have to crop the video realtime/offline. This info was provided to our video editing partner who created the grid in Unity3D to crop and export the videos.

4 - Playback position deviation

We had the local files playback idea but we still had to make sure that devices played files synchronously and didn’t deviate over time.

Solution - Central Synchronisation Server

The solution was simply to fake video playback with a timer on the server and broadcast the expected frame of the video to all the devices. Devices would then seek the playback to the correct position. If it is done consistently the users would not see the difference. The accepted deviation in the playback of all the devices was set to within 50-100ms.

This server also acted as web server and controller of devices allowing us to upload new videos with a single command and switch between night/day videos.

5 - Device Time

It seemed simple at first to just broadcast a packet to the devices forcing them to seek but not all packets would reach the devices the same time. This meant that network latency even local could affect the playback as a device reading the packet after 1 second would seek 1 second later and hence will be always behind.

Solution - NTP

We added timestamp of the server to each playback position broadcast packet. This meant that now devices could calculate the latency and account for that in the playback seek. This could only work if all the devices had their timing synced from a single source. We noticed that Android devices had a few seconds time skew. We decided to setup NTP server on the same synchronisation server and make sure that devices used it to sync their timing. Android (at that time at least) didn’t have the option to set custom NTP server and we ended up using TrueTime library for this.

6 - Transport protocol for broadcasting

Early on it seemed plausible to just use UDP broadcast from server to send the new position to all of the devices. This proved very difficult due to different versions of OS and security implementation by manufacturers.

Solution - TCP sockets with Actor Model

After all failed with UDP, we decided that TCP sockets were the best way to manage devices. The devices would act as clients and connect to the synchronisation server configured in the app. We decided to use GO as the language for the server side. To keep track of the devices connected we decided to use Proto.Actor framework. This way we got the goodness of supervisors and stateful actors to manage each device’s life cycle.

7 - Irregular Keyframe positions in the video

Video encoders optimise the video size by placing keyframes only when necessary. We realised that seek function tries to seek to the closest keyframe instead of the exact location specified. This caused the videos on devices to go out of sync by a second or more, sometimes.

Solution - Export videos with consistent keyframes

We requested our video editing partner to export keyframes at consistent positions ignoring the size optimisation. We didn’t care about the size as we were going to pre-load the videos anyway. This was the last key in keeping video playbacks synchronised.

Final Result

It took us about 3 days of working 16 hours at the site to assemble and test the whole system.

This is how it looked like at 4 am before the event in the morning.

TL;DR

Built app for Android 7 the lowest supported Android OS on the devices.

Own private network with mostly Ethernet for connectivity.

Pre-loaded files on the devices pushed using the central server also running on the LAN.

Used TCP sockets instead of UDP broadcast.

GO with Proto.Actor for synchronisation server.

Local NTP server for device time synchronisation.

Consistent keyframes in the video.

Expected playback position broadcast from sync server to all the devices to keep videos in sync.

Business Case Study and Pictures.