Inference controls and reproducible measurements

Olenji exposes optional device, precision, thread and native-backend requests through the CLI, HTTP API and MCP. These requests reach the same owned inference workers for file transcription, buffered live recognition and refinement. No default model, runtime or dependency is silently replaced to satisfy a request.

Discover support before selecting hardware

olenji models inspect faster-whisper.tiny --json
olenji models inspect whisper.tiny.q8_0 --json
olenji models inspect fluid-speech.mini.onnx --json

Each result includes inference_options.supported, defaults and an availability description. MCP's olenji_list_models retains the same capability object. A supported option describes the adapter's request interface; it does not establish that a GPU, driver, acceleration provider or compatible binary exists on the host.

device is cpu or cuda. cpu_threads is an integer from 1 through 64; booleans and fractional values are rejected. Faster Whisper precision choices are int8, int8_float32, int8_float16, int8_bfloat16, int16, float16, bfloat16 and float32. The installed CTranslate2 runtime validates the actual device/precision combination before loading weights. For example, a CPU may support int8 and float32 but reject float16.

The pinned transcribe.cpp binding accepts auto, cpu, metal, vulkan, cpu_accel, cuda and rocm. Its installed build determines availability. An explicit native selection must match the backend the loaded model actually reports; a mismatch closes that model and returns an error. auto remains an intentional automatic policy and can honor the native TRANSCRIBE_BACKEND environment setting. The Python layer does not promise every backend on every platform.

ONNX CUDA requests require an active CUDA execution provider in the encoder and frontend sessions and disable runtime provider fallback in those sessions. The decoder and joint network retain their existing CPU execution; graph placement remains the runtime's responsibility. A compiled or registered provider alone is insufficient evidence of usable acceleration. This control does not install CUDA packages, drivers or libraries. Hardware acceptance remains a native-host task in NATIVE_HANDOFF.md.

CLI, API and MCP

olenji transcribe recording.wav --model faster-whisper.tiny \
  --device cpu --compute-type int8 --cpu-threads 2 --json

olenji live start --source browser --model whisper.tiny.q8_0 \
  --native-backend cpu --cpu-threads 4 --open --json

olenji sessions refine SESSION_ID --inference '{"cpu_threads":4}' --json
olenji sessions refine SESSION_ID --inference '{}' --json

Use individual flags or --inference JSON, --inference @file.json, or --inference - for stdin in one request. Combining the JSON form with individual flags is rejected before transport. The JSON reader has the same bounded-input contract as other CLI document inputs.

The HTTP file-path, live and refinement bodies, and their corresponding MCP tools, accept:

{
  "inference": {
    "device": "cpu",
    "compute_type": "int8",
    "cpu_threads": 2
  }
}

Multipart file upload accepts an optional inference form field containing the JSON object as text. Schema and backend-field validation occurs before a new session, job, capture owner or upload destination is created. Syntactically valid fields unsupported by that backend return UNSUPPORTED_INFERENCE_OPTION with HTTP 422. Invalid typed API/MCP input retains the normal INVALID_ARGUMENT envelope; malformed upload option objects and direct engine-option validation use INVALID_INFERENCE_OPTIONS. Actual runtime availability failures are reported separately and can occur when the admitted job starts inference.

Persistence, refinement and cache ownership

Session metadata and each recognized source turn's provenance record the explicit request under inference_options. JSON/JSONL exports retain it. An empty object means adapter defaults were requested. These fields are not measurements of actual GPU utilization, effective compute precision or installed driver identity.

For refinement, omitted or null inference inherits the saved request only when the newly selected model uses the same backend. Selecting a different backend without options restores its defaults. Any explicit object replaces the entire request; {} resets it. A failed refinement preserves the old transcript and restores its prior model, language and inference metadata.

Faster Whisper and ONNX cache identities include their device, precision where applicable, and thread configuration. transcribe.cpp caches the model by native backend and applies the thread request to each recognition session. A loaded model or session cannot satisfy a new explicit selection by reusing incompatible old options. Installation identity checks, model leases, cancellation and owned worker cleanup retain their existing behavior. Unavailable device/precision checks do not invalidate a known-good installation, and a later supported request can recover.

Default requests preserve their previous persisted idempotency fingerprints. Repeating a completed pre-option request with its original idempotency key returns the original result. Adding explicit inference settings changes that request and cannot reuse the same key as though the selection were unchanged.

Measure on the intended host

The opt-in driver requires an already installed model, its managed runtime and an explicitly selected local speech fixture. It downloads nothing. Run it in disposable development state and write a new report:

python tests/benchmark_inference.py --help
python tests/benchmark_store_search.py --help

The inference driver compares Olenji's persistent managed worker with direct Faster Whisper in that same runtime, using the same model, decoding options, thread count and waveform. It separates the first inference from paired warm samples, compares complete normalized segment/word records, verifies worker reuse and closes owned children. The storage driver uses a fresh synthetic fixture and an explicitly selected trusted baseline Store module; it compares complete response bytes and coordinates a concurrent writer at the search boundary.

Use the exact commands and results in current validation. A short single-voice Tiny-model fixture does not establish accuracy or optimal thread counts for larger models, Turkish, overlapping voices or sustained live capture. Preserve model/artifact/runtime identity and compare recognition quality with speed. The competitive review defines the remaining corpus and native-streaming evidence.