Model Card for VITS-CSMSC (IB-Robot, hybrid CPU + Ascend 310P)

PaddleSpeech VITS-CSMSC — single-speaker Mandarin text-to-speech, 22.05 kHz — packaged for the IB-Robot framework as a hybrid deployment: the text encoder and duration predictor run as ONNX on CPU, and the flow decoder (which contains the vocoder and dominates compute) runs as an .om on Ascend 310P1.

This is the speech-output stage of the robot pipeline. Accepts arbitrary Chinese text; the voice is the fixed CSMSC speaker.

Deployments

deployment	backend	flow artifact	frames	notes
ascend_310p	ONNX CPU + Ascend ACL	vits_flow_decoder_dynamic_fp16.om	variable	recommended, any utterance length
torch_cpu	ONNX Runtime CPU	artifacts/onnx/vits_flow_decoder.onnx	variable	full CPU reference path

Two fixed-shape flow decoders also ship in the repo without their own deployment entry, matching the convention across this organisation that one hardware target carries one deployment:

file	frames	audio	notes
vits_flow_decoder_t639_fp16.om	639	7.42 s	~20 % lower flow latency than dynamic, 44 % of its size
vits_flow_decoder_t681_fp16.om	681	7.91 s	second fixed-shape build

Both are declared in bundle.files and load by pointing an ACL session at the file directly. They accept exactly their frame count, so an utterance has to be padded or trimmed to fit; the dynamic build takes any length, which is why it is the one wired to ascend_310p. Accuracy for all three is in Validation — they are within 1e-5 cosine of each other.

The ascend_310p row is a composite deployment in the manifest: there is no single runtime_profile, because there is no single backend. role_identities and role_runtime_profiles name the backend per role — onnx on CPU for text_encoder and duration, ascend/acl for flow_decoder. torch_cpu runs all three roles on ONNX Runtime and so keeps one shared profile.

The two host-side steps between duration and flow (duration_expand_path, sample_z_p) are listed in each deployment's metadata.host_stages; they are not model roles and have no artifact.

Shared contract: tensor_model/vits/synthesize (tts.phone_ids [1,-1] -> tts.audio [1,-1]), preprocessing paddlespeech-zh-frontend-jieba-pypinyin-phoneid-v1, output mono-float32-pcm-22050hz-v1, logical revision paddlespeech-vits-csmsc@onnx-export-v1.

ascend_310p uses the dynamic-shape flow decoder, so no padding or trimming is needed. For latency-sensitive use with a known utterance length, load one of the two fixed-shape builds above directly.

Pipeline

Chinese text
  → PaddleSpeech frontend / pypinyin      → phone ids
  → text encoder     (ONNX, CPU)          → x, m_p, logs_p (192-d), x_mask
  → duration predictor (ONNX, CPU)        → logw
  → host: alignment expansion + z_p       (see below)
  → flow decoder     (.om, Ascend 310P1)  → waveform
  → 22.05 kHz mono wav

Host stage (between duration and flow)

This is not a thin glue layer — it is part of the model's semantics and must be reproduced exactly. See assets/reference_impl/run_vits_hybrid_310p.py:

dur    = ceil(exp(logw) * float32(x_mask))  # per-phone frame counts
path   = generate_path(dur, ...)           # [1, T_text, T_frames] alignment
m      = (path.transpose @ m_p.transpose).transpose      # expand to frames
logs   = (path.transpose @ logs_p.transpose).transpose
y_mask = ones([1, 1, T_frames])
z_p    = m + randn_like(m) * exp(logs) * 0.667           # noise_scale 0.667

The reference uses seed 1234 for reproducibility; the sampling of z_p is genuinely stochastic, so runs with different seeds produce different (equally valid) audio, and bit-exact comparison against a reference requires pinning it.

Tensor contract

text encoder   text        int64  [1, -1]      phone ids
               text_lengths int64  [1]
             → x           float32 [1, 192, -1]   outputs are positional and
               m_p         float32 [1, 192, -1]   come back in this order
               logs_p      float32 [1, 192, -1]
               x_mask      bool    [1, 1, -1]

duration       x           float32 [1, 192, -1]
               x_mask      bool    [1, 1, -1]     bool, not float
             → logw        float32 [1, 1, -1]

flow decoder   z_p         float32 [1, 192, T]   T = 639 / 681 / -1
               y_mask      float32 [1, 1, T]
             → wav         float32 [1, T*256]    256 samples per frame

Sample rate 22050 Hz, hop 256 → one frame ≈ 11.61 ms.

Repository Structure

• inference_manifest.json — deployment routing (schema v3)
• assets/adapter.json — adapter identity (vits / synthesize)
• assets/phone_id_map.txt — 268-entry phone→id map used by the frontend
• assets/snapshot_iter_150000.pdz — the upstream Paddle training checkpoint (1.1 GB), byte-identical to the one in vits_csmsc_ckpt_1.4.0.zip
• assets/default.yaml — that checkpoint's own training config
• assets/reference_impl/run_vits_hybrid_310p.py — the hybrid runner, and the normative source for the host stage above
• assets/reference_impl/synthesize_vits_310p.py — text → wav CLI
• artifacts/onnx/*.onnx — the three ONNX graphs (CPU stages + CPU reference)
• artifacts/ascend_310p/*.om — three flow decoder builds

Source Model

Upstream is PaddleSpeech VITS-CSMSC, released checkpoint vits_csmsc_ckpt_1.4.0. The manifest's logical revision is paddlespeech-vits-csmsc@onnx-export-v1 — it names the ONNX export this bundle runs, not the upstream release tag, which is 1.4.0.

wget https://paddlespeech.cdn.bcebos.com/Parakeet/released_models/vits/\
vits_csmsc_ckpt_1.4.0.zip
# 1,027,863,302 bytes
# sha256 994340036777b24f3bf02967bf37f570d40e5514b54ff0001c5b4732921ae064

The archive holds exactly three files, and all three are shipped here byte-for-byte (sha256 verified against a fresh download and extraction):

shipped as	in archive	bytes	sha256
assets/snapshot_iter_150000.pdz	snapshot_iter_150000.pdz	1,116,898,380	ef0faad8186cdd1aa278b61e0b2a3cb7dee9216d1b84f1bc760ec887bcd935c7
assets/default.yaml	default.yaml	7,835	7eebdd044a663f2485b6abd54f6179e90da64c0c7c109778295f9a23be6dcbfc
assets/phone_id_map.txt	phone_id_map.txt	2,128	ca00d60618a00a2dc1e42002faaa9024c9e5057e04b1e834325950111cee59c3

phone_id_map.txt pins the phone inventory, and therefore the text frontend contract, to this exact release. default.yaml is the checkpoint's own training config and independently confirms the numbers this bundle's tensor contract depends on: fs: 22050, n_shift: 256 (the 256 samples per frame above) and hidden_channels: 192 (the 192-d z_p).

Shipping the checkpoint follows the convention in the rest of this organisation, where the upstream weights live under assets/. Note that it is not the artifact any deployment loads — every deployment runs the ONNX graphs below. It is there so the bundle carries its own provenance.

The three ONNX graphs were exported from that checkpoint in separate passes, which is why their opsets differ (12 / 13 / 11):

file	bytes	opset	sha256
vits_text_encoder_clean.onnx	55,329,745	12	15c5fb01538ce38c4748c9bc5892fc0af9bee7c65b228cd116651237a3f72536
vits_duration.onnx	2,644,862	13	f75fd325801a62976ecea0f52ba03a384197f574fedf0486598e5c0fc840c89c
vits_flow_decoder.onnx	86,596,889	11	3cb9dfe036b843abce71b867626f23d60745c0f0f35f29787ecec7fbfaf0f7fa
vits_flow_decoder_t639_fp16.om	91,708,818	—	b7da515895536b7baf75314b908344164efb916ffc8cc2246c6dac0e4c4cf91b
vits_flow_decoder_t681_fp16.om	91,706,422	—	df192baf48cefb732f52945cd411d3c88bbd310cac784dc51e11a8a7ed0d2b61
vits_flow_decoder_dynamic_fp16.om	210,986,822	—	c8b26160a0bc7e4de987424fab228bcc8dfd3fb053bd910737443999412c40d8

Shipping the checkpoint does not make the ONNX byte-reproducible from it: the export script is not part of this bundle, and PaddleSpeech static-model export behaviour changed with PaddlePaddle ≥ 2.5. What is established is the accuracy of the export — cosine 0.9999999999998159 against the Paddle reference, below — plus the exact checkpoint and config it came from.

Validation

All measured on a real Ascend 310P1, against ONNX Runtime CPU fp32.

ONNX export vs Paddle reference (text encoder + duration) — the export itself is lossless:

cosine   0.9999999999998159
max_abs  4.291534423828125e-06

Flow decoder OM vs ONNX, on the flow subgraph alone:

build	cosine	max_abs	latency mean	p95	subgraph RTF
t639	0.9999842557218173	0.01094835	96.17 ms	96.38 ms	0.01296
t681	0.9999692545131995	0.01450025	102.72 ms	—	0.01299

Those are the conversion-time numbers. They have since been independently reproduced on the shipped artifacts — same board, same artifacts/onnx/ vits_flow_decoder.onnx reference, but with z_p built through the real encoder → duration → host-alignment path rather than synthetic tensors (_tools/board/verify_vits_flow_om_vs_onnx.py):

build	frames	cosine	max_abs	rel_l2	latency mean	p95	subgraph RTF
t639	639	0.99998847	0.008860	0.00480	95.89 ms	96.00 ms	0.01292
t681	681	0.99998923	0.008666	0.00464	102.47 ms	102.71 ms	0.01296
dynamic	399	0.99998542	0.006929	0.00540	77.32 ms	77.90 ms	0.01669

Latency matches the conversion report to within 0.3 ms on every build, and accuracy comes out slightly better on realistic z_p than on the synthetic input used at conversion time. The dynamic build's higher RTF at 399 frames is the fixed per-call overhead spread over less audio, not a slower kernel — at 639 frames it lands within ~40 ms of the fixed build (see below).

End-to-end hybrid pipeline, 20 runs, 7.42 s of output audio. Measured with the fixed t639 build, which is what the reference runner loads (assets/reference_impl/run_vits_hybrid_310p.py:15); the same pipeline on the ascend_310p dynamic build is the baseline row two tables down, 367.63 ms:

cosine_vs_onnx   0.9999853236698978
max_abs          0.008778512477874756
latency  mean    327.38 ms   p50 323.81 ms   p95 355.89 ms
RTF              0.0441

Per-stage breakdown (mean, ms):

stage	backend	ms	share
text encoder	ONNX CPU	48.71	15%
duration	ONNX CPU	25.38	8%
host alignment + z_p	numpy CPU	156.64	48%
flow decoder	Ascend OM	96.53	29%

The host alignment stage is now the single largest cost — larger than the NPU work it feeds. It is pure numpy (generate_path plus two matmuls) and is the obvious next optimisation target; moving it onto the NPU or vectorising it would cut end-to-end latency by roughly a third.

Dynamic build, variable text length:

sample	phones	frames	audio	latency mean	RTF
baseline	101	639	7.4188 s	367.63 ms	0.04955
new_text	144	915	10.6231 s	414.97 ms	0.03906

Latency grows sub-linearly with length, so RTF improves on longer utterances. The dynamic build costs ~40 ms over the fixed t639 build at the same length.

Usage

from inference_manifest import load_inference_manifest

validated = load_inference_manifest("models/vits-csmsc", "ascend_310p")

The hybrid path needs both an ONNX Runtime CPU session (encoder, duration) and an ACL session (flow), plus the host stage between them — see assets/reference_impl/run_vits_hybrid_310p.py, which is the normative implementation.

Known Limitations

• Not a pure-NPU TTS. Text encoder and duration remain on CPU; only the flow decoder is compiled to OM.
• Chinese only. English and other languages are not a validation target.
• Fixed speaker. Single-speaker CSMSC voice, no speaker conditioning.
• The two extra fixed-shape OMs require exact frame counts (639 / 681). The ascend_310p deployment uses the dynamic build and has no such limit; the fixed builds are for callers who know their length and want the ~40 ms.
• Output is stochastic (noise_scale = 0.667) unless the RNG seed is pinned.

Environment

Validated on: Ascend 310P1 with CANN and ais_bench, Python 3.11.6, ONNX Runtime 1.23.2.

License

Code and packaging: Apache-2.0. Model weights follow the upstream PaddleSpeech license (Apache-2.0); the CSMSC corpus is subject to its own terms.

Citation

@inproceedings{vits2021,
  title     = {Conditional Variational Autoencoder with Adversarial Learning for End-to-End Text-to-Speech},
  author    = {Kim, Jaehyeon and Kong, Jungil and Son, Juhee},
  booktitle = {arXiv preprint arXiv:2106.06103},
  year      = {2021}
}
@article{paddlespeech2022,
  title   = {PaddleSpeech: An Easy-to-Use All-in-One Speech Toolkit},
  author  = {Zhang, Hui and Yuan, Tian and Chen, Junkun and Li, Xintong and Zheng, Renjie and Huang, Yuxin and Chen, Xiaojie and Gong, Enlei and Chen, Zeyu and Hu, Xiaoguang and Yu, Dianhai and Ma, Yanjun and Huang, Liang},
  journal = {arXiv preprint arXiv:2205.12007},
  year    = {2022}
}
@software{ib_robot,
  title   = {IB-Robot: Intelligence Boom Robot},
  url     = {https://atomgit.com/openeuler/IB_Robot},
  license = {Apache-2.0}
}