OmniVoice is a multilingual text to speech model with voice cloning, voice design, and fine grained control. Its generation path combines a Qwen3 transformer with custom audio components and an iterative decoding process. Our goal was not simply to make one model benchmark faster. We wanted a maintainable serving path that could use NVIDIA's optimized transformer stack while preserving the behavior and quality controls already built into OmniVoice.
Why TensorRT LLM instead of TensorRT alone
TensorRT is the compiler and runtime underneath the solution. It can optimize a general neural network graph, but using it directly would leave our team responsible for implementing and maintaining transformer specific attention, normalization, rotary position handling, quantized linear layers, and engine integration. That is possible, but it creates more custom infrastructure than the project needs.
TensorRT LLM adds the transformer building blocks, quantization aware linear layers, FP8 GEMM support, checkpoint conventions, and engine tooling we needed. We could reuse those maintained components while defining only the part unique to OmniVoice. This reduced implementation risk and left a clearer path for future NVIDIA runtime improvements.
The choice was therefore not TensorRT versus TensorRT LLM as two competing runtimes. TensorRT LLM runs on TensorRT. The decision was whether to build directly on the lower level runtime or use its transformer focused layer and customize the model boundary. For this workload, the second option provided the right balance of performance, control, and maintainability.
A custom boundary for an unusual model
The standard Qwen3 engine accepts token IDs and returns language model logits. OmniVoice instead sends a combination of text and audio embeddings into the transformer and needs hidden states back for its audio heads. It also uses iterative unmasking rather than conventional autoregressive token generation.
We therefore accelerated only the 28 Qwen3 transformer blocks. Text and audio embeddings, audio heads, the generation loop, and the HiggsAudioV2 codec remain in the original PyTorch path. This hybrid boundary limits the change to the compute intensive component, preserves existing product behavior, and retains a PyTorch fallback for validation and recovery.
Converting the checkpoint
Our converter reads the OmniVoice safetensors checkpoint, extracts the transformer layers and final normalization, then maps their names to the custom TensorRT LLM model. Embeddings, audio heads, and the unused language model head are deliberately skipped because they stay in PyTorch.
The FP16 checkpoint compiles into an engine of about 840 MB. More importantly, this explicit mapping makes the conversion inspectable. We can see which weights are accelerated and which parts of the original model remain untouched.
How the FP8 conversion works
FP8 targets seven linear layers in every transformer block: the four attention projections and the three MLP projections. Across 28 blocks, 196 weight tensors are converted to FP8 E4M3. Each tensor uses a scale calculated as its maximum absolute value divided by 448, the largest finite E4M3 value.
The converter stores the FP8 weights and their per tensor scaling factors, keeps normalization weights in FP16, and records FP8 in the TensorRT LLM configuration. Building with the FP8 GEMM plugin produces an engine of about 420 MB, half the size of FP16, and enables FP8 Tensor Core matrix multiplication on the L4.
The current implementation uses a default activation scale of 1.0 without a calibration dataset. This keeps conversion simple and worked for the tested voices, but it is also a known tradeoff. Representative calibration should be the next step if a broader evaluation reveals quality loss on specific speakers, languages, or voice design prompts.
Results on NVIDIA L4
We benchmarked nine English and Vietnamese voice cloning samples with output lengths from roughly 5 to 25 seconds. Mean server inference fell from about 1,377 ms in PyTorch to 989 ms with the FP16 engine and 779 ms with FP8. That makes FP8 1.77 times faster than PyTorch and 1.27 times faster than the FP16 TensorRT LLM engine for the measured server workload.
End to end latency tells a more nuanced story: PyTorch measured 2.69 seconds, FP16 measured 2.38 seconds, and FP8 measured 2.42 seconds. This measurement includes Python process startup, HTTP, inference, and file I/O, so its noise can hide the engine level gain. At FP8, server inference represents only about 32 percent of total time.
What the benchmark taught us
Customizing the engine boundary was more valuable than replacing the whole application. It let us accelerate the compute intensive transformer while preserving OmniVoice behavior and keeping a safe fallback path. The benchmark also makes the next optimization target clear: audio processing, codec decoding, request handling, and batching now matter more than another isolated transformer improvement.