NagaYu/bleep-0.09b

Bleep — 0.09B frame tagger for sensitive speech regions

Find where sensitive speech is, without transcribing what was said.

Bleep reads audio and outputs (category, start, end) — a spoken digit string here, a letter-by-letter spelling there — and nothing else. It never produces a transcript, and it has no mechanism that could: the network takes log-mel features and emits nine per-frame channels, so there is no path by which a digit or a name could leave it.

• 🤗 Dataset: NagaYu/bleep-spans
• 🎛️ Demo: NagaYu/bleep

---

⚠️ Headline: this is a published negative result

At this training budget the model does not work. That is the finding, and it belongs at the top rather than in a limitations section.

On the held-out test split, at the ASR baseline's own measured false-alarm rate (31.4 spurious regions/hour):

arm	recall
A — ASR (whisper-tiny.en) + text PII	0.826
B — ASR (whisper-small.en) + text PII	0.861
C — Bleep alone	0.156
D — A ∪ Bleep	0.851 (+0.021 over A)
oracle — perfect transcript + the same text rules	0.986

Bleep recovered 12% of the regions arm A missed. The union is therefore almost all arm A.

Three separable reasons, and the benchmark separates them

1. The failure mode the idea targets is real, but the corpus did not put it where it was expected. On the training-condition test split the ASR baseline is essentially flat — 0.818 clean, 0.857 at 8 kHz (i.e. better than clean), 0.720 under 6 dB overlap. A band-limit plus G.711 is not a hard condition for a modern ASR, especially on hyper-articulated TTS speech. The collapse only appears in the harsher held-out conditions: 0.271 at 0 dB three-way overlap and 0.490 at narrowband-plus-overlap. The premise survives; the training conditions were too mild to exercise it.

2. The text rules were never the bottleneck. The oracle arm scores 0.96–1.00 in every condition, so every baseline miss is an ASR word error rather than a detector gap. That is precisely what the oracle arm exists to establish, and it means "use a better ASR" (arm B, +0.035) is a real if partial answer.

3. The model is badly undertrained. 1500 steps, batch 4, ~1.8 epochs over 9.15 hours — 70 minutes on one consumer GPU. Validation loss was still falling monotonically at the final step (0.218 → 0.183): a stopped run, not a converged one.

One directional hint

Bleep's per-category recall has the shape the acoustic argument predicts. Its best categories are SPELLED_OUT (0.475) and GROUPED_DIGITS (0.432) — the two with the most distinctive rhythm — against ~0.00 for DIGIT_SEQUENCE, MONEY and IDENTIFIER. The cue family it managed to learn is the predicted one. There is just far too little of it to be useful.

What did hold up, independent of model quality

• Speed. 0.0562 real-time factor on CPU with the front end included — 17.88× real time, 2.6× faster than the cheapest ASR arm and 17× faster than arm B. Core ML on the Neural Engine: 0.00157 RTF.
• The union containment guarantee. Zero violations across every utterance in both splits: adding Bleep never removed a region the baseline found.
• The no-content invariant and the review-gated redactor, both enforced by tests rather than asserted.

One gate the checkpoint fails, recorded rather than relaxed

The boundary-accuracy test asserts a 600 ms 90th-percentile onset error; this model measures 1092 ms. The decoder-only tests pass under 45 ms on synthetic scores, which localises the problem to the model — at 1500 steps it fires on the edges of a sensitive region rather than across it, which is also why span-level recall (0.156) trails frame-level recall (0.709). The gate is left where it is and marked a known gap; a threshold moved to fit a result stops being a test.

---

Model

parameters	86,967,945 (0.087 B — inside the 0.05–0.15 B target band)
architecture	dilated depthwise-separable 1-D convolutions, 24 blocks, d_model 768
input	64-band log-mel, 16 kHz, 25 ms / 10 ms
output	50 Hz (20 ms resolution) × 9 channels: 8 categories + ANY
receptive field	8.98 s (capped deliberately — see below)
CPU RTF	0.0562 (17.88× real time), peak RSS 1082 MB

Convolutional rather than attention-based, because the requirement is long recordings on a device: cost is O(T) in time and O(1) in memory per streamed chunk, with no KV cache. Chunked inference is bit-exact against whole-file inference, so constant-memory processing of a two-hour recording is not an approximation.

The receptive field is capped near 9 s on purpose. On a synthetic corpus a model that can see a whole utterance can recognise the template it was generated from instead of the sensitive region inside it — and the benchmark would score that as success.

The eight categories

DIGIT_SEQUENCE, GROUPED_DIGITS, SPELLED_OUT, DATETIME, ADDRESS, PERSON_NAME, IDENTIFIER, MONEY.

---

Files

file	what it is
bleep-base.pt	PyTorch checkpoint (weights + architecture + front-end config)
bleep-base.config.json	human-readable architecture/cost summary
operating_points.json	thresholds calibrated on the disjoint calib split
bleep-base.train_report.json	training config, validation curve, measured speed
exports/bleep-base.onnx (+ .data)	ONNX, opset 20, dynamic time axis. Verified vs PyTorch to 1.53e-05
exports/bleep-base.mlpackage	Core ML ML Program. Verified to 1.36e-02, 0.00157 RTF
exports/bleep-base.f16.gguf	GGUF tensor container + architecture metadata
benchmarks/	full result JSON and rendered tables
figures/	per-condition, per-category, union and ROC figures (light + dark)

On GGUF, plainly: it is a tensor container with architecture metadata for quantisation tooling and custom runtimes. llama.cpp has no graph for a dilated depthwise-separable convolution tagger and cannot run this file. Saying otherwise would be the kind of claim this project is trying not to make.

---

Usage

The bleep/ package is bundled in this repository (and in the Space), so everything you need is in one place.

huggingface-cli download NagaYu/bleep-0.09b --local-dir bleep-model
cd bleep-model && pip install numpy scipy torch

import json
from huggingface_hub import hf_hub_download

from bleep.audio import load_audio
from bleep.tagger import load_model, predict_scores
from bleep.threshold import OperatingPoint, RecallFirstThresholding

ckpt = hf_hub_download("NagaYu/bleep-0.09b", "bleep-base.pt")
ops = hf_hub_download("NagaYu/bleep-0.09b", "operating_points.json")

model, feat, _ = load_model(ckpt)
audio = load_audio("call.wav")

# (T, 9) per-frame probabilities: 8 categories + an "any sensitive" channel.
scores = predict_scores(model, audio.samples, feat)

# "high" favours recall. Thresholds were calibrated on a speaker-disjoint
# split, never on the data any number is reported on.
op = OperatingPoint.from_dict(json.load(open(ops))["high"])
rf = RecallFirstThresholding(op, model.cfg.frame_rate)

# decode_any() reads the "any sensitive" channel -- this is the recall-first
# path, and the one the operating point is calibrated on. Category attribution
# is a separate, optional step (decode_typed), kept separate so that naming the
# wrong category cannot cost recall.
for span in rf.decode_any(scores, duration=audio.duration):
    print(f"{span.start:7.2f} {span.end:7.2f}  {span.label}  {span.score:.2f}")

for span in rf.decode_typed(scores, duration=audio.duration):
    print(f"  category: {span.label} at {span.start:.2f}-{span.end:.2f}")

A Span carries start, end, label, score, source — and nothing else. There is no field a transcript could live in.

Command line:

bleep find call.wav --recall high        # category + time, never content
bleep redact call.wav                    # writes a REVIEW PLAN, not audio
bleep union call.wav --asr-spans yours.json

bleep redact does not redact by default. It writes a reviewable plan with a decision field per region, because muting audio is irreversible and this detector is fallible. Applying a plan with undecided regions is refused.

---

Intended use, and what this is not

Intended: an additional signal alongside a transcribe-then-detect pipeline you already run, to direct a human's attention. The UnionCombiner guarantees the combined output contains every span your existing detector produced, so adopting it can add coverage but never remove any.

Not a redaction guarantee. On this checkpoint it misses most things. An empty result means "nothing was flagged", never "nothing is there".

Not a replacement for your ASR-based detector. The measurements above say plainly that it is not currently competitive with one.

Not trained on real audio. Every recording is synthetic. No real call, no clinical recording, nothing derived from either, at any stage. That is a deliberate constraint and also a real limitation: TTS speech is far easier to transcribe than conversational audio, which is part of why the baseline looks so strong here.

English only. Japanese generators exist in the codebase and are wired end to end, but are not benchmarked and no Japanese number should be assumed.

---

Measured results in full

Split test: 250 utterances (0.89 h), 288 sensitive regions. A region counts as detected when at least 50% of it is covered.

Primary: recall at a matched false-alarm rate

Arm A's measured false-alarm rate on this split is 31.4 spurious regions/hour. Bleep's threshold (0.812) was calibrated on the disjoint calib split to meet that same budget.

arm	recall	FA/hour	redaction overhead	RTF
A ASR(whisper-tiny.en) + text PII [standard configuration]	0.826	31.4	0.0494	0.146
B ASR(whisper-small.en) + text PII [better ASR]	0.861	32.5	0.0622	0.932
C Bleep alone [no transcription] @ matched FA (thr 0.812)	0.156	30.3	0.0310	0.080
D Arm A union Bleep	0.851	58.3	0.0761	0.227
oracle perfect transcript + text PII [diagnostic, not deployable]	0.986	24.7	0.0465	—

Complementarity

Of 288 sensitive regions: 39 found by both, 199 by arm A only, 6 by Bleep only, 44 by neither.

• recall — arm A 0.826, Bleep 0.156, union 0.847 (+0.021)
• of the regions arm A missed, Bleep found 0.120
• union containment verified on every utterance: True (0 violations)

Recall by degradation condition

condition	n	A	B	C	D	unseen
clean	44	0.818	0.864	0.295	0.864
mic_variation	43	0.837	0.837	0.233	0.884
noisy_babble_10db	35	0.800	0.771	0.000	0.800
noisy_street_5db	26	0.885	0.885	0.000	0.885
overlap_6db	25	0.720	0.760	0.040	0.760
reverb_moderate	39	0.846	0.872	0.282	0.897
telephony_8k	28	0.857	0.929	0.357	0.857
telephony_noisy	48	0.833	0.938	0.000	0.833

Recall by category

category	n	A	B	C	D	reported failure mode
DATETIME	43	0.860	0.884	0.116	0.860	yes
GROUPED_DIGITS	37	0.946	1.000	0.432	0.973	yes
IDENTIFIER	44	0.727	0.795	0.000	0.727	yes
SPELLED_OUT	40	0.775	0.775	0.475	0.875	yes
ADDRESS	25	0.720	0.880	0.040	0.760
DIGIT_SEQUENCE	34	0.971	0.941	0.000	0.971
MONEY	32	1.000	0.969	0.000	1.000
PERSON_NAME	33	0.606	0.667	0.121	0.636

Recall on the reported failure modes

The cases transcription-first detection is reported to handle worst. These carry the argument.

subset	n	A	B	C	D
hard:datetime	66	0.879	0.894	0.106	0.879
hard:grouped_digits	62	0.871	0.952	0.306	0.919
hard:identifier	65	0.785	0.831	0.092	0.800
hard:overlap	25	0.720	0.760	0.040	0.760
hard:spelled	66	0.803	0.818	0.303	0.879

Category attribution: of the 23 regions Bleep detected, it named the right category for 0.652 of them. Reported separately from recall, because a mis-typed region still gets redacted.

Boundary accuracy (Bleep)

Onset error median -183.1 ms (negative = early, the safe direction), 90th percentile absolute 1092.4 ms. 32.6% of onsets were late, which is the direction that leaks the head of a word.

Speed and size

bleep-base: 87.0 M parameters, CPU real-time factor 0.0562 (17.88x real time, front end included), peak RSS 1081.9 MB, 347.9 MB fp32 weights.

Real-time factors are only meaningful on an otherwise-idle machine; a measurement taken under load understates speed by an arbitrary amount.

Unseen degradation conditions (test_heldout)

arm	recall	FA/hour
A	0.649	25.4
C	0.087	43.8
D	0.688	63.6

Generated from benchmarks/results.json (2026-09-18T14:56:09Z) by scripts/update_readme.py.

---

Reproducing

Everything is scripted: corpus build, training, export, benchmark, figures, and the regeneration of the results block above from the benchmark's own JSON. See the repository README. 328 tests cover the no-content invariant (structurally, via an AST walk over the module graph), the union containment guarantee (randomised), the time-preservation of every degradation, and the on-device speed budget.

Licence

Apache-2.0.