← LeaderboardThe Skill Leaderboard

triton-pytorch-model-repository

Deploy a user's PyTorch model as a working Triton Inference Server endpoint, end to end. Use when the user has a TorchScript .pt file or a PT2 archive (.pt2 from torch.export + AOT Inductor) and wants it served, wants a config.pbtxt written from scratch, or wants to debug a Triton load or inference failure on a PyTorch model. Use triton-onnx-model-repository for .onnx files and triton-tensorrt-model-repository for .plan engines.

Published by NVIDIA ↗NVIDIA/skills

Try your first task

A source-grounded starter task has not been verified for this skill yet.

SKILL.md

name: triton-pytorch-model-repositorydescription: Deploy a user's PyTorch model as a working Triton Inference Server endpoint, end to end. Use when the user has a TorchScript .pt file or a PT2 archive (.pt2 from torch.export + AOT Inductor) and wants it served, wants a config.pbtxt written from scratch, or wants to debug a Triton load or inference failure on a PyTorch model. Use triton-onnx-model-repository for .onnx files and triton-tensorrt-model-repository for .plan engines.license: Apache-2.0metadata:   author: Vinya Kestur <vinyak@nvidia.com>  tags:     - triton    - inference    - pytorch    - torchscript    - torch-aoti    - model-repository    - config-pbtxt

Triton PyTorch Model Repository

When to Use

Trigger on .pt, .pt2, config.pbtxt for a PyTorch model, or a Triton load or inference failure on a PyTorch model. Hand off .onnx to triton-onnx-model-repository and .plan to triton-tensorrt-model-repository. Not for training.

Hard Constraints

  1. No auto-complete. Every input, output, datatype, and shape must be declared in config.pbtxt. Source: pt/model_state.cc, pt2/model_state.cc.
  2. No BF16, no FP8, no UINT16+. Cast to FP16 or FP32, or use the ONNX or TensorRT backend. Source: libtorch_utils.cc. See references/datatypes.md.
  3. PT2 archives are device-baked. CPU export cannot serve on KIND_GPU and vice versa. Re-export to change device. See references/torch-aoti.md.
  4. Misnamed inputs load but produce wrong results. If names match neither forward() nor <name>__<index>, the backend falls back to strict config order. See references/naming.md.
  5. Cache-cleaning parameter is named differently per platform. ENABLE_CACHE_CLEANING on TorchScript, CACHE_CLEANING_ENABLED on PT2. See references/parameters.md.

Platform Form

Formconfig.pbtxtModel file
TorchScript, C++ runtimeplatform: "pytorch_libtorch"model.pt
TorchScript, Python runtimebackend: "pytorch" + runtime: "model.py"model.pt (the runtime: "model.py" value names the backend's shipped Python runtime, not a user file; if a user authors their own model.py in the version directory it takes priority over model.pt)
AOT Inductor (PT2)backend: "pytorch" + platform: "torch_aoti"model.pt2

Default filenames overridable with default_model_filename.

Deploy TorchScript

Layout:

model_repository/
  <name>/
    config.pbtxt
    1/model.pt

Minimal config.pbtxt:

name: "my_model"
platform: "pytorch_libtorch"
max_batch_size: 8
input  [ { name: "INPUT__0"  data_type: TYPE_FP32 dims: [ 3, 224, 224 ] } ]
output [ { name: "OUTPUT__0" data_type: TYPE_FP32 dims: [ 1000 ] } ]
instance_group [ { kind: KIND_GPU } ]

Launch:

tritonserver --model-repository=/models

--disable-auto-complete-config is optional and only silences the PyTorch backend's "auto-complete not supported" warning at startup. Do not add it when the same model repository also holds ONNX, TensorFlow, or TensorRT models that rely on auto-complete: the flag is server-wide and those models will fail to load.

Deploy PT2

Build the archive with a dynamic batch dim; sample input must have batch > 1 so torch.export does not specialize:

import torch

m = Model().eval()                # add .cuda() for KIND_GPU
sample = (torch.randn(2, 4),)     # one entry per forward() argument; .cuda() each for KIND_GPU
batch = torch.export.Dim("batch", min=1, max=8)
exported = torch.export.export(
    m, sample, dynamic_shapes=({0: batch},)
)
torch._inductor.aoti_compile_and_package(exported, package_path="model.pt2")

Minimal config.pbtxt:

name: "my_model"
backend: "pytorch"
platform: "torch_aoti"
max_batch_size: 8
input  [ { name: "INPUT__0" data_type: TYPE_FP32 dims: [ 4 ] } ]
output [ { name: "OUTPUT__0" data_type: TYPE_FP32 dims: [ 4 ] } ]
instance_group [ { kind: KIND_CPU } ]

Dim("batch", max=N) must be >= max_batch_size. instance_group kind must match the export device.

Build vs Serve Containers

nvcr.io/nvidia/tritonserver:26.10-py3 ships libtorch but no Python torch. Build artifacts in nvcr.io/nvidia/pytorch:26.10-py3 or install torch before building. Match build and serve to 26.10 to avoid opset drift.

Inference (HTTP KFServing v2)

curl -s -X POST -H "Content-Type: application/json" \
  -d '{"inputs":[{"name":"INPUT__0","shape":[1,4],"datatype":"FP32","data":[1,1,1,1]}]}' \
  http://localhost:8000/v2/models/my_model/infer

Large tensors: use the binary tensor extension. See references/config-pbtxt.md.

Debug a Failed Load

Validation order: file exists → input count matches → datatype supported → naming resolves. Fix in order; each later check only runs if earlier ones pass.

Error (server log)Fix
unable to find '/models/<name>/1/model.pt'Place model.pt at 1/ or set default_model_filename.
PyTorch inductor model file "..." is unreachable or inaccessible.Place model.pt2 at 1/.
unsupported datatype TYPE_<X> (TorchScript)See references/datatypes.md. Cast or switch backend.
Unsupported datatype TYPE_<X> ... model instance "<name>_0_0". (PT2)Same recovery; note PT2 wording differs.
configuration expects N inputs, but model expects M inputs.Match input count in config.pbtxt to the exported call spec.
Incorrect device passed to aoti_runner_cpuRe-export PT2 on the target device.
PyTorch execute failure: output must be of type Tensor, List[str] or Tuple ... (inference-time)Return Tensor, 1-D List[str], or a Tuple. Wrap scalars with torch.tensor(...).
PyTorch execute failure: output ... received List[Tensor] (inference-time)Return a Tuple of Tensors, not a List.
failed to split the output tensor '<name>' in responses: expected batch size of at least N ... (PT2 inference-time)PT2 archive lacks dynamic batch Dim. Re-export with torch.export.Dim.
sequence batching with state but not all ... <name>__<index>Rename every input, output, and initial_state.name to <name>__<index>.

Full catalog: references/errors.md.

Performance

  • Backend is BLOCKING (libtorch.cc). Add instance-group entries, not threads.
  • INTRA_OP_THREAD_COUNT and INTER_OP_THREAD_COUNT are process-global; first model wins (pt/model_state.cc).
  • INFERENCE_MODE default true is stronger than NoGrad; leave on.
  • TorchScript JIT compiles on first invocation; use model_warmup to amortize.
  • KIND_MODEL lets the model's own .to(device) win; inputs land on CPU, output path forces cudaStreamSynchronize (pt/model_instance_state.cc).

See references/parameters.md.

Metrics

Standard Triton metrics at GET /metrics on port 8002. No PyTorch-specific metrics. See docs/user_guide/metrics.md.

Hand Off

  • User needs BF16, FP8, UINT16+: ONNX or TensorRT backend.
  • User wants Dynamo (vLLM, TRT-LLM router): Dynamo skills.
  • User wants TorchServe features: different product.
  • CUDA OOM or device-side assertion: model issue, not Triton config.

References

triton-pytorch-model-repository by NVIDIA · The Skill Leaderboard