I've been exploring the midpoints of AI technology and software engineering, that is AI engineering. Simply put, I want to focus more on integrating models into real, production-shaped system with control over most of the stack.

Product Description

Therefore, I'm building a simple web app that allows users to track expenses more easily. It tries to remove the effort needed to track expenses by implementing speech-to-text technology and an SLM that would help record and standardize the information. I'm also planning to use IndexedDb as user facing data persistence, to keep the structured financial records private and local-first. Another prominent part of this system is user corrections, where users can give corrections if the SLM misinterprets the information, these corrections will also be used to retrain our models if the user gives consent to do so.

The stack

1. For the web-app we'll use React, as we will mainly use SPA and fewer concerns about SEO.
2. IndexedDB as mentioned earlier for the user facing data persistence.
3. Whisper for STT technology, for easier cross-browser implementation.
4. SLM (chose Llama 3.2 1B for its balance of size vs. quality for on-device inference via WebGPU).
5. PostgreSQL for the global data persistence. Mainly used for training data and metadata for models.
6. R2 for object storage, for cost efficiencies.
7. Backend written in Python chosen for familiarity and strong fit for data orchestration, AI related development and minimum API work.

High-level e2e flow

Design decisions and tradeoffs

1. Familiar stacks, enable easier development and future collaboration (if any). So far, I don't see any challenges that would nudge us towards using more advanced and nuanced stacks, at least at this current scope.
2. Local-first principles, allow users to have explicit control over their financial records, whilst also reducing latency between processes. The tradeoffs would be relatively big memory consumption and initial data load. I've got some ideas for workarounds like lazy loading, caching and compression (Quantization). We'll see if it becomes major pain later on.
3. Evaluation gate before model promotion. This allows us better control over the model lifecycle. The tradeoffs would be the requirement of human in the loop, which in my opinion is justified for better model quality.
4. Whisper over native WebAPI. It introduces heavier initial load and memory consumption, but it gives us control over the STT part. Furthermore, native WebAPI does not have wide enough support for various browsers (e.g., Firefox).

What I'd do differently at scale

One option worth exploring is offloading the SLM to a dedicated GPU server, which would improve accessibility for users on low-to-medium-end devices, but it directly trades away the on-device privacy guarantee that's core to this project's design. A middle path worth exploring would be to keep local inference as the default, and offer server-side inference as an explicit opt-in for devices that can't handle it, mirroring the same opt-in pattern already used for training data sync.

There's also an idea to extend the model improvement pipeline, to also improve localized versions of the SLM, both the STT and the Interpreter SLM. If this had real multilingual usage, extending the pipeline to localized STT and extraction models might better serve users across different languages, though it's a meaningful amount of additional work. The work in question would involve maintaining multiple versions of data and evaluation processes, separated by language. Achievable, but highly dependent on real usage patterns at scale.

---

I'll be writing regularly about the progress, cases, and plans throughout the development. I'd really appreciate hearing your thoughts.

Cheers!