Add ONNX Runtime neural-net backend (OpenVINO EP, v1.17.1)

[seniorfish]

Add ONNX Runtime neural-net backend (OpenVINO EP)

Draft PR. Feedback welcome — see notes at the bottom.

Summary

This PR adds a new USE_BACKEND=ONNX option that runs KataGo's neural-net inference through ONNX Runtime, with pluggable execution providers.

Relation to #1164: an ONNX Runtime backend PR (#1164 by @ChinChangYang) already exists. This PR is offered as an alternative with two differences: (1) it tracks current master (v1.17.0 + v1.17.1) and supports the transformer trunk models introduced there (the existing PR predates that release); and (2) it reuses master's existing OnnxModelBuilder (shared with the TensorRT backend) instead of introducing a separate graph emitter, so the new backend is a single file plus wiring and touches no other backend.

What changed:

• cpp/neuralnet/onnxbackend.cpp (new) — the full NeuralNet::* backend implementation behind #ifdef USE_ONNX_BACKEND. It serializes the model graph via the existing OnnxModelBuilder into an ONNX ModelProto and feeds it to an Ort::Session; input filling and output decode are copied line-for-line from trtbackend, so the IO protocol and post-processing are identical to TensorRT.
• cpp/neuralnet/onnxmodelbuilder.{h,cpp} — adds an opt-in alignInputsToConsumptionOrder parameter to OnnxModelBuilder::build() (default false). When true, graph inputs are declared in consumption order (InputSpatial, InputGlobal, InputMask) instead of the default order. This is a no-op for backends that bind inputs by name, but is required for the OpenVINO EP under ORT, which builds its name→index map from declaration order while the runtime feeds the EP kernel ports in consumption order — with the default order the EP misroutes the mask tensor into the InputSpatial port. The ONNX backend requests it only for the OpenVINO provider.
• cpp/CMakeLists.txt — ONNX branch in the USE_BACKEND chain (links ONNX Runtime + protobuf + zlib, generates onnx.proto, defines USE_ONNX_BACKEND, copies ORT DLLs).
• cpp/program/setup.cpp — registers the onnx backend prefix; ONNX defaults to NCHW (inputsUseNHWC = false), otherwise createComputeHandle throws.
• cpp/main.cpp — version-info #elif USE_ONNX_BACKEND.
• cpp/program/gtpconfig.cpp — multi-GPU onnxDeviceToUseThread handling.
• cpp/configs/gtp_example.cfg — ONNX settings example block.
• Compiling.md — a new section documenting how to build ONNX Runtime from source with the desired EP and compile KataGo against it.

Compatibility with existing code:

The backend is non-invasive. It reuses master's existing OnnxModelBuilder (shared with the TensorRT backend) rather than introducing a parallel graph emitter, and the only shared-source change — the alignInputsToConsumptionOrder parameter — is an additive default parameter. The TensorRT call site is unchanged and produces a byte-identical graph (declaration order unchanged), so TRT behavior is zero-diff. Other backends are untouched. IO contract and output decode are identical to TRT, so nneval/nninputs/loadmodel need no changes. Verified to build on master at v1.17.1 and to run both convnet (v15) and transformer (v17) models.

Extensibility:

The architecture is provider-agnostic: onnxProvider selects among cpu / openvino / cuda / tensorrt / migraphx / coreml, and adding a future EP requires no code change — only config. The alignInputsToConsumptionOrder escape hatch is generic: any EP that exhibits the same declaration-vs-consumption-order quirk can be opted in at the single build() call site. New onnx* config keys are auto-recognized by the onnx prefix in setup.cpp, so provider-specific options can be added without touching the backend core. Only the OpenVINO provider has been tested end-to-end in this PR; the other providers are wired in and should work, but are unverified.

How To Use

A prebuilt Windows x64 binary with all runtime DLLs bundled is available here:

https://github.com/seniorfish/KataGo/releases/tag/onnx-backend-v1

Download katago-onnx-backend-v1.zip, unzip, drop in a .bin.gz model, and point your config at it.

The backend adds a family of onnx* keys to the config file (see configs/gtp_example.cfg for the full list). The essential ones:

Key	Default	Purpose
onnxProvider	cpu	Execution provider: cpu/openvino/cuda/tensorrt/migraphx/coreml
onnxOpenVINODeviceType	GPU	OpenVINO device: GPU/CPU/NPU/AUTO:.../MULTI:...
onnxOpenVINOCacheDir	empty	Compile cache dir — speeds up cold starts
onnxOpenVINONumStreams	empty	OpenVINO execution streams
onnxOpenVINOPrecision	empty	FP16/FP32/ACCURACY
onnxTransformerNHWC	false	NHWC trunk for transformer models

To use an Intel GPU you must set both:

onnxProvider = openvino
onnxOpenVINODeviceType = GPU

Tip (measured): if you are on a discrete Intel GPU rather than an integrated one, setting onnxOpenVINONumStreams = 2 noticeably improves throughput — two parallel OpenVINO streams keep the GPU saturated at higher thread counts (see the ablation rows in the Benchmark).

Benchmark

katago benchmark, -visits 60, default config otherwise noted. EloDiff is relative to the first row of each group.

Desktop — Intel Arc B580

OpenCL (baseline). Model b11c768h12nbt3tflrs-fson-silu.bin.gz (transformer, v17):

numSearchThreads	visits/s	nnEvals/s	nnBatches/s	avgBatchSize	time	EloDiff
2	14.61	14.11	14.06	1.00	41.7s	baseline
4	15.39	15.12	7.76	1.95	40.9s	−4
8	16.74	16.72	4.67	3.58	40.0s	−20

ONNX Runtime (OpenVINO EP). Model b11c768h12nbt3tflrs-fson-silu.bin.gz, onnxOpenVINONumStreams = 2, onnxTransformerNHWC = true:

numSearchThreads	visits/s	nnEvals/s	nnBatches/s	avgBatchSize	time	EloDiff
4	273.22	216.37	108.51	1.99	29.4s	baseline
8	362.81	290.80	73.26	3.97	22.2s	+87
12	394.78	321.04	54.35	5.91	20.5s	+102

Model kata1-zhizi-b40c768nbt-s11472M-d5982M.bin.gz (convnet, v15), onnxOpenVINONumStreams = 2:

numSearchThreads	visits/s	nnEvals/s	nnBatches/s	avgBatchSize	time	EloDiff
4	122.82	96.05	48.20	1.99	65.4s	baseline
8	156.49	125.23	31.55	3.97	51.6s	+62
12	169.99	134.65	22.75	5.92	47.7s	+66

Ablation (b11 transformer, numSearchThreads = 12), isolating NumStreams / transformerNHWC:

onnxOpenVINONumStreams	onnxTransformerNHWC	visits/s	nnEvals/s	avgBatchSize	time
1	true	126.72	103.04	5.91	63.9s
2	true	394.78	321.04	5.91	20.5s
2	false	185.39	150.18	5.91	43.7s

Both knobs matter: NumStreams = 2 ≈ 3× throughput vs 1, and transformerNHWC = true ≈ 2× vs false on this model.

Laptop — Intel Core Ultra 9 285H, Intel Arc 140T

ONNX Runtime (OpenVINO EP), numSearchThreads = 8:

numSearchThreads	visits/s	nnEvals/s	nnBatches/s	avgBatchSize	time	EloDiff
8	58.22	57.88	11.80	4.90	14.9s	baseline

---

Notes

• Verified hardware: runs correctly on an Intel Arc B580 desktop and an Intel Core Ultra 9 285H laptop (Arc 140T).
• Tested scope: only the OpenVINO provider has been tested. Other providers (cpu/cuda/tensorrt/migraphx/coreml) are implemented but unverified — treat them as best-effort until someone confirms them.
• NPU: not yet available via this backend (onnxOpenVINODeviceType = NPU is not working at this time).
• Status: functionality has been tested with no anomalies on the above hardware. Feedback and suggestions are very welcome — this is opened as a draft precisely to gather them.
• Maintenance: I intend to keep this rebased on master and to address review feedback as time allows. As a solo contributor I can't guarantee indefinite long-term stewardship, but I'm glad to coordinate with anyone — maintainer or fellow contributor — who wants to help carry it forward.

Co-authored-by: glm5.2

[seniorfish]

Prebuilt binaries with ONNX Runtime + OpenVINO 2026.2.1 for Intel GPUs: https://github.com/seniorfish/KataGo/releases/tag/onnx-backend-v2 ~20% faster than v1 on transformer models (numStreams=2, 10 threads, 800 visits, 347.49 visits/s -> 416.52 visits/s).

[kaorahi]

Thank you for sharing your work! For some reason, it runs at only about 36% of the speed of #1171 in my environment (Intel Core Ultra 7 255U, Linux).

• OpenCL: 25.16 visits/s with kata1-b18c384nbt
• PR 1171 (iGPU): 76.33
• PR 1171 (NPU): 87.30
• PR 1171 + 3ed8922 (NPU + iGPU): 142.14
• PR 1222 (NPU): 31.52

Also, what do you think about supporting the combined use of the NPU and iGPU? Simply setting onnxOpenVINODeviceType = MULTI:NPU,GPU didn't make much difference. For a significant speedup, we may need "per-thread OpenVINO device and batch size configuration" like the patch linked above.

[kaorahi]

(1) katago benchmark fails with the older small model g170-b6c96-s175395328-d26788732.txt.gz when onnxOpenVINODeviceType = NPU. It works with CPU and GPU.

terminate called after throwing an instance of 'StringError'
  what():  Got nonfinite for policy sum

(2) It also fails with the model kata1-b18c384nbt-s9996604416-d4316597426.bin.gz when onnxOpenVINODeviceType = GPU. It works with CPU and NPU.

ERROR: Async bot thread failed: Got nonfinite for policy sum

(3) Removing modelDesc.applyScale8ToReduceActivations(); from cpp/neuralnet/onnxbackend.cpp makes NPU inference run at 4× the original speed (= 1.5× the speed of #1171).

• PR 1222 - Scale8 (NPU): 129.25 visits/s with kata1-b18c384nbt

[seniorfish]

@kaorahi, thank you. That's a very insightful observation. The patch has been committed. The new version gives me roughly a 4x performance improvement here as well (Core Ultra 9 285H reaching 54 visits/s, NPU).

Regarding your second issue, kata1-b18c384nbt-s9996604416-d4316597426 seems to run fine on my end, both on the B580 and the 285H, with onnxOpenVINODeviceType = GPU.
I'll test the first issue later.

As for your suggestion about supporting the combined use of the NPU and iGPU, I'll try to implement it.