Retain Quran

General speech recognition transcribes what it hears into words. That is not the task here. Recitation evaluation has to judge whether what was recited was correct - pronunciation, articulation and the rules of tajweed - which is closer to assessment than to transcription.

Off-the-shelf speech recognition is not built for that. It is trained to be robust to accent variation, which means it will helpfully normalise away precisely the differences that constitute an error here. Custom models were not a preference; the task required them.

Users recite in Arabic from all over the world, with substantial regional variation, at every age from young children to adults, into whatever microphone their phone has, in rooms with televisions on.

The failure mode to design against was a model that performs well on clear adult recitation from one region and materially worse for a child in another - the sort of gap an overall accuracy figure hides completely. Evaluating performance across those groups separately, rather than reporting a single number, was treated as an acceptance criterion rather than a refinement.

The two errors are not equivalent. Missing a genuine mistake means a user practises an error uncorrected. Flagging correct recitation as wrong is worse in a different way: it undermines the user's confidence in something they hold sacred, and it is the kind of experience that ends usage permanently.

Where the model is uncertain, the system says so rather than asserting. A tool that occasionally admits it is unsure is more trustworthy than one that is always confident, and in this domain that trade is not close.

Usage does not grow smoothly. It rises sharply during Ramadan and peaks around particular times of day within it, and the date is known months ahead - which makes it one of the rare capacity problems that can be prepared for properly rather than absorbed.

Cloud Run and Kubernetes on GCP handle the elasticity, with the important detail being what the scaling signal is. Inference is memory and accelerator bound while CPU sits low, so autoscaling on CPU would let the service saturate and queue while reporting comfortable utilisation. Scaling on queue depth and on latency against the actual target is what keeps it responsive at peak.

A model that has degraded still returns confident answers with normal latency and a zero error rate. Conventional monitoring reports perfect health while the product gets worse.

Prometheus and Grafana therefore track the distribution of evaluation outcomes, not only service health - if the proportion of recitations flagged as incorrect shifts materially, something has changed in the model, the audio pipeline or the user base, and all three are worth knowing about. Ground truth arrives too slowly to be the alerting signal, so the output distribution serves as the early one.

Audio payloads are large and the inference service is called on every submission. gRPC's binary encoding and streaming meaningfully reduce transfer overhead compared with JSON over HTTP at that volume, and the generated contracts keep the API surface between the application and the model service explicit as both change.