triton-onnx-model-repository
Deploy a user's ONNX file as a working Triton Inference Server endpoint, end to end. Use when the user has a .onnx file and wants it served, wants a config.pbtxt generated from the ONNX graph, or wants to debug a Triton load or inference failure on an ONNX model. Use triton-pytorch-model-repository for .pt files and triton-tensorrt-model-repository for .plan engines.
Try your first task
A source-grounded starter task has not been verified for this skill yet.
SKILL.md
name: triton-onnx-model-repositorydescription: Deploy a user's ONNX file as a working Triton Inference Server endpoint, end to end. Use when the user has a .onnx file and wants it served, wants a config.pbtxt generated from the ONNX graph, or wants to debug a Triton load or inference failure on an ONNX model. Use triton-pytorch-model-repository for .pt files and triton-tensorrt-model-repository for .plan engines.license: Apache-2.0metadata: author: Vinya Kestur <vinyak@nvidia.com> tags: - triton - inference - onnx - model-repository - config-pbtxtTriton ONNX Model Repository
When to Use This Skill
Trigger when the user:
- Has a
.onnxfile and wants it served on Triton Inference Server. - Wants a
config.pbtxtgenerated from an ONNX graph. - Is debugging a Triton load or inference failure on an ONNX model.
- Is writing a Python or REST client for a served ONNX model.
Do not trigger for .pt (TorchScript, use triton-pytorch-model-repository), .plan (TensorRT engines), LLM serving on Triton, Kubernetes deployment, or ensembles. Do not trigger for training, fine-tuning, or exporting a model to .onnx from another framework; this skill serves an already-exported .onnx file only. Decline the request and point to the framework's training docs.
Purpose
Turn a user's .onnx file into a running Triton Inference Server endpoint the user's code can call. Read the graph, build the repository, write config.pbtxt only when needed, start the server, verify the load, return a working client snippet.
Prerequisites
- Read access to the
.onnxfile. - Docker installed. For GPU targets, the NVIDIA Container Toolkit set up so
docker run --rm --gpus all nvcr.io/nvidia/tritonserver:26.10-py3 nvidia-smi -Lprints a device. - Python 3.10+ with the
onnxpackage. - Either
tritonclientorcurlfor the smoke test. - Host ports 8000, 8001, 8002 free, or overrides supplied in
ports.
Required Inputs
Collect or infer these before changing anything. Ask only for what cannot be inferred:
onnx_path: absolute path to the.onnxfile.model_name: default is the file's stem in lowercase, hyphenated.repo: absolute path to the model repository root. Default:${PWD}/triton-repo.target:cpuorgpu. Default:gpuif the host has one and the user did not say otherwise, elsecpu.batching:none|auto|explicit:<max_batch_size>. Default:autoif the ONNX graph has a dynamic leading axis, elsenone.versioning:single|canary:<v1,v2>. Default:single.cache:on|off. Default:off.accelerator:none|tensorrt|openvino. Default:none. Settensorrtfor GPU production to get the ORT-TRT 2x speedup. Setopenvinofor Intel CPU.ports: three integers<http>,<grpc>,<metrics>. Default:8000,8001,8002. Change if the defaults are occupied.execute:trueto run commands here,falseto only produce the exact commands. Default:true.review:trueto pause and show the generatedconfig.pbtxtbefore starting the server,falseto skip the pause. Default:true.
Instructions
1. Preflight
Run read-only checks first. Do not mutate anything yet.
test -f "${onnx_path}"
python3 -c 'import sys, onnx; onnx.checker.check_model(sys.argv[1])' "${onnx_path}"
docker version >/dev/null
# GPU target only:
docker run --rm --gpus all nvcr.io/nvidia/tritonserver:26.10-py3 nvidia-smi -L
Stop and ask the user if any check fails.
2. Inspect the ONNX graph
Read names, dtypes, and shapes from the graph. Do not assume them.
import onnx
m = onnx.load(onnx_path)
inputs = [(i.name, i.type.tensor_type.elem_type, [d.dim_value or d.dim_param for d in i.type.tensor_type.shape.dim]) for i in m.graph.input]
outputs = [(o.name, o.type.tensor_type.elem_type, [d.dim_value or d.dim_param for d in o.type.tensor_type.shape.dim]) for o in m.graph.output]
A graph is batchable only if EVERY input AND EVERY output has a string leading axis that resolves to the same symbol (e.g., "N"). A graph where inputs are [N, 4] and [4] is NOT batchable. Walk every (name, type, shape) tuple from Step 2 and confirm the leading axis on all of them before treating the graph as batchable.
3. Build the model repository
${repo}/
${model_name}/
1/
model.onnx
The file must be named model.onnx. The version directory must be a positive integer.
For external weights, either keep the sidecar files next to model.onnx in the version directory, or wrap the whole set in a directory also named model.onnx/ with the main file inside:
${model_name}/
1/
model.onnx/
model.onnx
<external weight files>
For custom operators, copying the shared library into the version directory does NOT register the ops. Add a model_operations block to config.pbtxt:
model_operations { op_library_filename: "/path/to/libmycustom.so" }
The path must be reachable from inside the server process (a path in the mounted repository works).
4. Write config.pbtxt only if required
Skip when all of these hold: batching is auto, versioning is single, cache is off, accelerator is none, no graph name contains :0 or /, and the graph has no custom ops. Auto-complete handles the rest.
When batching is none on a batchable graph, DO write a complete config.pbtxt with max_batch_size: 0 plus full input and output blocks. A complete config leaves nothing for auto-complete to fill in; auto-complete would otherwise set max_batch_size: 4 (from the backend's default-max-batch-size) and attach a dynamic_batching { } block, overriding the user's intent. Do not use --disable-auto-complete-config for this: it is a server-wide flag and would break other models in the repository.
Otherwise, write the file using references/config-pbtxt.md. Use backend: "onnxruntime", not platform:. Match input and output names to the graph exactly.
When accelerator is tensorrt, append:
optimization { execution_accelerators {
gpu_execution_accelerator: [{ name: "tensorrt"
parameters { key: "precision_mode" value: "FP16" }
parameters { key: "max_workspace_size_bytes" value: "1073741824" }
}]
}}
ORT-TRT compiles the graph on first load, which can take minutes for large models. Pair with model_warmup so the compile happens at load time, not on the first request.
When accelerator is openvino, append:
optimization { execution_accelerators {
cpu_execution_accelerator: [{ name: "openvino" }]
}}
When this step writes a config.pbtxt, print the full file contents to the user and pause for confirmation before moving to Step 5. Skip the pause when review is false or execute is false.
5. Start Triton
HTTP_PORT="$(echo ${ports} | cut -d, -f1)"
GRPC_PORT="$(echo ${ports} | cut -d, -f2)"
METRICS_PORT="$(echo ${ports} | cut -d, -f3)"
docker run -d --rm --name triton \
--gpus all \
-p "127.0.0.1:${HTTP_PORT}":8000 -p "127.0.0.1:${GRPC_PORT}":8001 -p "127.0.0.1:${METRICS_PORT}":8002 \
-v "${repo}:/models:ro" \
nvcr.io/nvidia/tritonserver:26.10-py3 \
tritonserver --model-repository=/models --exit-on-error=false \
$( [ "${cache}" = "on" ] && echo --cache-config local,size=1048576 )
Ports bind to loopback by default. Drop the 127.0.0.1: prefix only when remote access is an explicit requirement and the deployment target can authorize external callers. For CPU targets, drop --gpus all. Wait for /v2/health/live to return 200 before continuing.
6. Verify the load
curl -s http://localhost:"${HTTP_PORT}"/v2/health/ready
curl -s -X POST http://localhost:"${HTTP_PORT}"/v2/repository/index
curl -s http://localhost:"${HTTP_PORT}"/v2/models/"${model_name}"/config | jq
Confirm the model appears with "state": "READY". On any other state, open references/errors.md.
7. Smoke test from a client
Generate a Python snippet from the graph read in Step 2. Use the graph's exact names, the matching numpy dtype, and shape with a leading 1 when the model is batchable.
import numpy as np, tritonclient.http as hc
c = hc.InferenceServerClient("localhost:<http_port>")
x0 = np.ones(<input0_shape>, dtype=np.float32)
inp = [hc.InferInput("<input0_name>", x0.shape, "FP32")]
inp[0].set_data_from_numpy(x0)
r = c.infer("<model_name>", inp)
print(r.as_numpy("<output_name>"))
For gRPC, swap tritonclient.http for tritonclient.grpc and use <grpc_port> (default 8001) in place of <http_port> (default 8000). InferInput, set_data_from_numpy, infer, and as_numpy are identical. Both frontends are started every time tritonserver boots (src/main.cc:268,280), so every model is reachable over both.
Use the KFServing v2 datatype spelling ("FP32", not TYPE_FP32) on the client side. See references/datatypes.md.
8. Return the Output Contract
Produce the fields listed under "Output Contract" below. If execute is false, produce the commands without running them.
Output Contract
Return to the user:
- Model repository path. Absolute path on disk.
- config.pbtxt. The file contents if written. "Auto-completed" otherwise.
- Server command. The literal
docker runortritonservercommand. - Client snippet. Python code that matches the graph's names and shapes exactly.
- Smoke-test result. The output tensor (shape, dtype, first few values). "Not executed" if
executeisfalse. - Metrics URL.
http://<host>:<metrics_port>/metrics, using the host metrics port fromports. - Blockers. Any step that failed, with the exact error and the row from
references/errors.mdthat matches.
Available Scripts
None. Use built-in Python and shell. For config validation, consult references/config-pbtxt.md against the Step 2 graph read.
Examples
Fixed-shape CPU, no config.
mkdir -p repo/add/1 && cp model.onnx repo/add/1/model.onnx
docker run -d --rm --name triton -p 8000:8000 -v $(pwd)/repo:/models:ro \
nvcr.io/nvidia/tritonserver:26.10-py3 tritonserver --model-repository=/models
curl -s http://localhost:8000/v2/models/add/config | jq
Dynamic-shape GPU, batching, cache. Write repo/mymodel/config.pbtxt with backend: "onnxruntime", max_batch_size: 16, dynamic_batching { preferred_batch_size: [8] }, instance_group [{ kind: KIND_GPU }], response_cache { enable: true }. Start with --cache-config local,size=1048576 --gpus all.
Canary v1 + v3. Place 1/model.onnx and 3/model.onnx. Add version_policy: { specific: { versions: [1, 3] } } to config.pbtxt. Client picks via model_version="3".
Limitations
- ONNX only. For PyTorch (
.pt), usetriton-pytorch-model-repository. For TensorRT (.plan), usetriton-tensorrt-model-repository. - Serving only. Training, fine-tuning, and ONNX export from another framework (PyTorch, TensorFlow, JAX, etc.) are out of scope. Decline and point the user to the framework's training/export docs.
- Single-model focus. Ensembles and the Triton model orchestrator are out of scope.
- Development setup. Does not configure TLS, authentication, rate limiting, or multi-tenant isolation.
- Does not install Docker, the NVIDIA Container Toolkit, the
onnxPython package, ortritonclient. Prerequisites must already be satisfied. - Does not run the model repository on object storage (S3/GCS). Local filesystem only.
- Does not create or manage Kubernetes resources. For Kubernetes deployment, use a separate Triton Kubernetes skill.
Troubleshooting
Short pointers only; the full table with exact error strings is in references/errors.md.
| Symptom | Where to look |
|---|---|
Model missing entirely from /v2/repository/index | references/errors.md row "silent parse failure" |
UNAVAILABLE with failed to stat file ... | File is not named model.onnx. Rename, or set default_model_filename. |
UNAVAILABLE with unexpected inference input/output ... | Name in config.pbtxt does not match the ONNX graph. Re-read step 2. |
[400] unexpected shape ... Expected [-1,N], got [N] | Request shape missing batch dim. Reshape client input. |
[404] no available versions | Model failed to load. Check the index. |
is_server_ready() False while models work | Server ready is strict. Use is_server_live() and per-model readiness. |
Load fails with NOT_IMPLEMENTED : Could not find an implementation for <OpType>(<opset>) | The op at that opset is beyond what Triton's bundled ONNX Runtime covers. Re-export at a lower opset, or upgrade Triton. |
Load fails with ONNXRuntimeError ... not implemented for ... | Graph contains an operator the backend cannot handle on the target device. Re-export without the op, or run on CPU. |
Benchmark
See BENCHMARK.md. Auto-generated by the NVSkills CI pipeline on /nvskills-ci.
References
references/config-pbtxt.md: everyconfig.pbtxtfield this skill may write, with the exact rule and example.references/datatypes.md: theTYPE_FP32to"FP32"tonp.float32crosswalk.references/errors.md: exact Triton 26.10 error strings, cause, and fix.docs/user_guide/metrics.mdin the Triton server repo: per-model and per-GPU counters exposed on port 8002.- Triton docs in the server repo:
docs/user_guide/model_repository.md,docs/user_guide/model_configuration.md. - Working examples in the server repo:
qa/L0_rhel_tutorial_cpu/gen_onnx.pyand itsmodels/add_onnx/config.pbtxt.