Journey to AI Engineering (Part 2): Finding an Indonesian ASR Model for Edge Deployment

In Part 1, I introduced the expense tracker I’m building and why I want to keep its processing local.

This time, I want to focus on one specific problem: finding a speech recognition model that can reliably transcribe Indonesian financial conversations and eventually run close to the user’s device.

The idea sounds straightforward. I speak about an expense, the application transcribes it, and another component interprets the text into structured financial data.

In practice, getting the first step right is already a challenge.

The problem with Indonesian financial speech

Speech recognition is not just about understanding ordinary conversations. For an expense tracker, the transcription needs to preserve details such as amounts, currencies, and other information that determines whether a transaction is recorded correctly.

For example, consider a sentence containing an amount of Rp15,000. A transcription error in that amount is not merely a spelling mistake. It can change the financial data the application eventually processes.

That makes general speech recognition quality an incomplete measure of success. I need a model that works well for the specific language and context in which the application will be used.

I also have a deployment constraint: I want to run inference locally rather than depend on an external speech recognition service.

Trying existing models first

Before training my own model, I wanted to see what existing options could do.

I tried several candidates:

• Chrome's native Web Speech API
• Whisper Tiny
• Whisper Small
• An Indonesian-specific Whisper model from Hugging Face
• OmniCTC

This was an exploratory test rather than a formal benchmark. I used the same audio recording to compare the transcriptions and inspect their differences. It was useful for narrowing down the options, but it was not enough to establish that one model was generally better than another.

Chrome's Web Speech API produced a very good transcription in my test. However, that result alone did not make it a suitable production choice. Its recognition may rely on an external service, which conflicts with my local-first requirement.

The other models still needed to prove that they could handle the kind of Indonesian speech I care about.

Whisper Small did not perform well enough in my initial test. I also tried an Indonesian-specific Whisper model from Hugging Face, but it struggled with a simple financial example involving the phrase "15,000 ribu."

That was enough to show me that choosing a model because it supports Indonesian does not guarantee that it will transcribe Indonesian financial speech correctly.

I also evaluated OmniCTC as another candidate. The results and deployment compatibility still need more investigation, so I won't draw a conclusion about it yet.

At this point, continuing to search for a ready-made model seemed less promising than testing whether I could adapt one myself.

Fine-tuning Whisper Small

I decided to fine-tune Whisper Small for Indonesian speech recognition.

The training is still running, so I don't have results to report yet. This is an ongoing experiment, not a claim that fine-tuning has already solved the problem.

The training pipeline uses Hugging Face Transformers, starting from openai/whisper-small. It configures the model for Indonesian transcription and uses a sequence-to-sequence training process.

The current setup also accounts for the limited GPU memory available during training. It uses a small per-device batch size with gradient accumulation, and it includes an out-of-memory recovery strategy that initially freezes the encoder before attempting to unfreeze it.

These are implementation decisions, not evidence of improved recognition. I'll need to evaluate the resulting model before deciding whether the extra training effort is worthwhile.

Using FLEURS as the starting dataset

I didn't want to begin by collecting and transcribing a large dataset manually. That would introduce another substantial task before I could even evaluate the model.

Instead, I'm using Google FLEURS, a multilingual speech dataset available through Hugging Face. For Indonesian, the configuration is id_id.

My preparation script loads the training and test splits, converts the audio into mono, 16 kHz, 16-bit PCM WAV files, and writes a manifest containing the audio paths and transcripts.

Keeping the test split separate is important. I want to evaluate the fine-tuned model against the original Whisper Small checkpoint on the same test examples, rather than rely on a few hand-picked recordings.

There is a limitation, though: FLEURS provides general speech data, not a dataset specifically designed for expense tracking. Improving performance on general Indonesian speech does not necessarily mean the model will become reliable at recognizing financial amounts.

That is why I need a second evaluation set.

The golden dataset

My initial idea was to record myself saying many financially related phrases and manually write down the correct transcripts.

That would give me examples closer to the actual use case, but collecting and transcribing enough recordings would take time. I wanted to make progress without turning dataset creation into the entire project.

The current approach is to use FLEURS as the starting point while building a smaller, domain-specific dataset that I call the golden dataset.

The golden dataset is intended to contain recordings that reflect the application's real requirements. Each example includes an audio file, its reference transcript, and expected structured information such as the amount, currency, category, merchant, and date.

I'm also separating examples into clear and hard cases. The goal is not just to collect recordings, but to have a consistent set of examples that can reveal where the system succeeds and where it fails.

This gives me two complementary ways to evaluate the model:

1. General speech recognition: Does fine-tuning improve transcription on the held-out FLEURS test set?
2. Expense-specific correctness: Does the model correctly preserve the financial information that matters to the application?

I don't have enough results yet to know how well this approach will work. But keeping those evaluation goals separate should make the eventual results more informative than a single overall score.

Training from Google Colab

To make the experiment easier to run, I prepared a Jupyter Notebook for Google Colab.

The notebook acts as a wrapper around my pasak Python package. The training and dataset operations live in the package, while the notebook provides a convenient way to execute the workflow in a Colab environment.

The backend is split into several responsibilities:

• Preparing the ASR dataset
• Fine-tuning Whisper Small
• Evaluating the original and fine-tuned checkpoints
• Exporting a quantized ONNX version for further runtime experiments
• Collecting, importing, packaging, and validating golden-dataset examples

I want the process to be repeatable. If I change the dataset or training configuration, I should be able to run the same preparation and evaluation steps again instead of relying on manual experimentation.

The ONNX export is also part of the deployment investigation. A model that trains successfully is not automatically suitable for browser inference. Model size, runtime compatibility, memory usage, and latency still need to be tested in the intended environment.

For now, the export pipeline is groundwork for that next stage, not proof that browser deployment is already working.

What I still need to learn

There are three questions I want this experiment to answer.

First, does fine-tuning actually improve recognition?

I'll compare the fine-tuned checkpoint with the original Whisper Small model on the held-out FLEURS test set, using word error rate (WER). I'll also evaluate the expense-specific golden set separately.

WER is useful for measuring transcription differences, but it doesn't fully capture whether a financial amount was interpreted correctly. That's why the domain-specific evaluation matters.

Second, does the model work better on financial speech?

A model can improve its general transcription score without solving the errors that matter most to an expense tracker. I need to look specifically at amounts and currencies rather than assuming that a better WER means better financial data.

Third, can the model run efficiently at the edge?

Even a more accurate model may be impractical if its runtime requirements are too high. I'll need to investigate quantization, model size, browser compatibility, and inference latency before making a deployment decision.

These questions are related, but they are not interchangeable. A successful training run is only one part of the solution.

Where the experiment stands

At this stage, I have tested several existing ASR options, prepared a fine-tuning pipeline, and started training Whisper Small with Indonesian data.

The early comparisons helped me identify the gap between general Indonesian transcription and the more specific requirements of financial speech. The golden dataset is my attempt to make that gap measurable without waiting until I have collected a large custom dataset.

The next step is to finish training and evaluate the results. Only then can I decide whether fine-tuning Whisper Small is a worthwhile direction or whether I need to revisit the model and deployment choices.

For now, I'm treating this as an engineering investigation: start with existing models, identify the failure cases, build a repeatable experiment, and let the results guide the next decision.