Running a CNN on the Olympus E-PL3

Prometheus stole the fire from the Olympians in a fennel stalk.

Anyway. In a previous post I converted my E-PL3 to full spectrum. Now, this is fun and all, but the camera is realistically a first-generation mirrorless body lacking multiple comfort firmware features.

Luckily these days, doing the entirety of the reverse engineering into mod loader dance is made trivial.

I decided to crack its firmware open and see where it takes me.

I started with adding some features I always wanted, such as live focus distance readout, proceeded to add a convenient red-blue channel in-camera swap option and finally got distracted into coercing a "modern" object detection neural network to run on the camera.

Structure of the project#

I wanted this to end up more as a general modding framework, called fenykl ("fennel" in Czech), with potential for adding my E-M10 Mark II to the mix later as I get more confident about the process.

The central idea is that we start by modifying a stock firmware image with a relatively small and easy to sanity check loader, which then loads a binary image from the SD card into memory and executes it. This means there is no need to re-flash the firmware during development. The loader is inert if no binary is present on the SD card.

As GCC has support for the MN10300 upstream, the plugin code can be written in C as one would desire, with a shared SDK adding abstracted helpers such as logging to the SD card.

The central goal is to develop one "main" plugin that has a native configuration menu where the various subfeatures can be configured. But I also want this to be a playground for experimenting with the hardware.

At the moment, the main plugin contains a bunch of goodies improving the HUD, with the more complex features to be merged later.

Getting the neural network working#

It's worth orienting ourselves in time here. The E-PL3 is a camera announced in June 2011. AlexNet, the first really "practical" convolutional neural network, was published in September of 2012, over a year later.

Over a decade of architectural improvements has gotten to the point where CNNs for image classification are commoditized and can be run on a microcontroller.

Having a neural network subject detector on the E-PL3 was not impossible because of hardware limitations, but the machine learning research just was not there yet at the time.

The architecture is born out of agent-driven hyperparameter search loosely based on FOMO and is fairly straightforward:

On the input, we snag a buffer that is originally destined for the LCD live view. This is a 540 x 480 block of 4:2:2 YUYV data.

Reusing the hardware downscaler, we push this block down to 96 x 72 of YUYV.

Cascades of convolution / maxpool layers, most of those are restricted to power-of-two weights turn this into a 12 x 9 logit heatmap.

Maximum of this heatmap figures out the rough location of the target.

Center-of-mass on the cells near the peak locate the target more precisely.

It has about 53k parameters and evaluates at around 6 FPS on the camera, which is borderline for something that feels real time. The built-in C-AF tracker takes to its output reasonably well, but I plan to replace it with something more finely tuned eventually.

It's worth appreciating what the neural network actually ingests. An example from the WetlandBirds (CC BY 4.0) dataset below illustrates:^*Blue is the predicted center, the reticle is the true bounding box center.

The dataset is a lazily filtered mixture of Open Images, iNaturalist and WetlandBirds dataset. I suspect that this is a limiting factor of sorts, since pictures that make it into iNaturalist are likely going to be keepers and not what the camera would see, on average, in the wild. On the test set it puts the point on the bird 84% of the time, but it also fires on 39% of frames with no bird in them. I decided to push the threshold far towards recall. My eyebulbs are plenty good at judging whether the frame actually contains a bird.

What is the hardware actually#

The good news is that this is a dual-core CPU and one of the cores is pretty much free in normal operation. Usually it performs eye detection^*This is not the same step as face detection! and video encoding hardware blocks, but this can be disabled.

The cores are basically a simple single issue CPU one would expect from a microcontroller from this era. Running at 160 MHz with 8kB of I-cache to spice things up. There is 256MB of DRAM attached, most of it allocated to the frame burst buffer.

One of the instructions declared "user-defined" in the ISA documentation I acquired is actually a single-cycle integer multiply. I think this is actually a common extension of the architecture since the binutils port also interprets this instruction as a multiply.

How do the power-of-two layers work#

It's worth noting that the power-of-two layers are actually compiled into executable instructions, as a sequence of adds/subs. The result is a sequence of instructions that just looks straight up like this:

6ea81158     fa 00 9c 43     mov (17308,a0),d0
6ea8115c     fa 04 a4 43     mov (17316,a0),d1
6ea81160     f1 02   sub d0,d2
6ea81162     f1 07   sub d1,d3
6ea81164     ea      add d2,d2
6ea81165     ef      add d3,d3

Zero-weights get dropped completely. Power-of-two weights within reasonable range are essentially free: start by applying the largest weight, double your running total, apply the next power of two in line.

All of this processing is done on four horizontally adjacent pixels at once: each 32-bit register holds two 16-bit totals and two output registers are computed in an interleaved fashion.

Splitting these pre-generated kernels into chunks <8kB improves throughput by around 50% as they can be kept warm in the I-cache.

Ultimately, this is several times faster than a generic convolution kernel, even with the single cycle multiply.

Piloting the loop#

I run a relatively vanilla terminal instance of Claude Code with some utilities for task management. The main instance I interact with serves as the coordinator, managing tasks and dispatching subagents to handle them, merging the results. Pretty standard stuff, nothing exotic.

To get started, I directed a subagent to investigate a source code dump of a firmware packer from an old Russian download site and reimplement that in Python. In parallel, another agent chipped away at translating a leaked(?) manual for the MN103S-ISA core used in the TruePic VI SoC of the E-PL3 into a Ghidra module so that disassembly may proceed without extra pain.

Constructing the loader required some careful manipulation of the safety filter. The coordinator was not careful enough, so the subagent it dispatched got killed almost immediately. Opening a separate session and asking the agent a bit more carefully yielded the desired result without a hitch.

After this point, I unleashed a bunch of parallel agents statically analyzing various parts of the firmware to map it out, producing comprehensive but possibly wrong documentation. Eventually, the coordinator started assigning tasks to my humble self to verify some hypotheses on the real camera :).

As the camera capabilities became clearer, I started directing the work on fenykl a bit more, forcing a certain project structure and abstraction layers I had in mind to get it in the shape I wanted.

What just happened?#

I actually tried this first in Spring with Opus 4.8 and it could not even get through the zeroth step, requiring a lot of babysitting to even get something resembling a working Ghidra SLEIGH module for the MN103S. Even with the GCC toolchain as a reference, it was just unable to get this to completion.

With Opus 5.5, it was smooth sailing from the start. It did the grunt work at superhuman speed, oneshotting the SLEIGH module and getting the loader right on the first try. The static analysis results were mostly correct, with reasonably designed on-camera live tests that figured out the details.

The project can be found on GitHub. This is an extremely in-progress piece of work, everything is highly in flux as I explore where I want to take it.