Optimization work often starts with a plausible assumption: the model is slow because it needs more compute. We profiled OmniVoice on an NVIDIA L4 and found the opposite. The GPU had ample compute available, but spent most of its time waiting for data and for the next piece of work to be launched. That evidence changed both the technical direction and where we invested engineering time.
A profiler turns assumptions into evidence
We traced a complete 32 step voice generation with PyTorch Profiler and inspected the timeline in Perfetto. The trace contained roughly 58,000 CUDA kernel calls. At a high level, the repeated model passes looked busy. At a closer level, the GPU was idle for more than 80 percent of the timeline in critical regions.
Individual matrix operations took only about 12 microseconds, while the gap before the next operation could reach 100 microseconds. The application was paying more for scheduling thousands of small tasks than for executing many of them. Without a profiler, this could easily be mistaken for insufficient GPU compute.
The bottleneck was movement, not mathematics
The profiler breakdown showed that elementwise operations consumed 33 percent of kernel time, nearly matching matrix multiplication at 36 percent. There were more than 43,000 small elementwise kernels. Launch overhead alone accounted for an estimated 307 milliseconds across the generation process.
A roofline check confirmed the broader issue: OmniVoice's relatively small Qwen3 backbone was memory bound on the NVIDIA L4. The GPU was waiting for weights and activations to move through memory rather than running out of arithmetic capacity. Buying a GPU with more theoretical compute would not directly address that constraint.
Profiling pointed to two practical actions
First, reduce the amount of data moved for the repeated linear operations. We used W4A4 quantization based on Nunchaku, reducing both weight and activation width. The implementation also combined related operations so intermediate data could be reused instead of repeatedly travelling to GPU memory. The kernel details matter to the implementation team, but the business logic is straightforward: move less data and avoid unnecessary round trips.
Second, reduce scheduling overhead. OmniVoice repeats hundreds of GPU operations across 32 generation steps. CUDA Graph allowed those operations to be captured once and replayed as a group. Because request shapes vary, we prepared a small set of input size buckets in advance, avoiding expensive graph capture during a live request.
The result and its tradeoff
Across ten English and Vietnamese voice cloning samples on an NVIDIA L4, mean inference time decreased from 1,654 milliseconds with PyTorch FP16 to 561 milliseconds with W4A4 and CUDA Graph. Real Time Factor improved from 0.145 to 0.056, which is 2.6 times faster than the baseline. In practical terms, one second of audio required about 56 milliseconds of generation time.
Performance was not free. INT4 activations introduced light noise or garbled segments in some samples. The quality was acceptable for the evaluated realtime and streaming scenarios, but not yet sufficient for every high fidelity voice cloning use case. Reporting this tradeoff is as important as reporting speed, because the right production choice depends on the client's quality threshold.
The management lesson
Profiling is not a final debugging step. It is a tool for deciding which work deserves investment. In this project, it prevented us from optimizing compute that was not saturated, connected low level evidence to two concrete engineering actions, and gave us a baseline for validating both improvement and quality impact.
For an AI engineering team, the repeatable process matters more than a specific kernel: measure representative workloads, identify where time and resources actually go, choose the smallest intervention that addresses that constraint, then benchmark again under the same conditions.