← LeaderboardThe Skill Leaderboard

nixl-docs

Find and read NIXL documentation for APIs, architecture, backends, installation and builds, environment variables, telemetry, troubleshooting, examples, benchmarks, and contribution.

Published by NVIDIA ↗NVIDIA/skills

Try your first task

A source-grounded starter task has not been verified for this skill yet.

SKILL.md

name: nixl-docsdescription: Find and read NIXL documentation for APIs, architecture, backends, installation and builds, environment variables, telemetry, troubleshooting, examples, benchmarks, and contribution.license: Apache-2.0metadata:   version: "1.0.0"  author: NVIDIA NIXL Team <sronen@nvidia.com>  tags:     - nixl    - docs

NIXL Documentation Index

Purpose

Index of NIXL documentation for agents working in a NIXL repository clone.

Prerequisites

A clone of the NIXL repository.

Instructions

Find relevant files in the index and read them. Paths are relative to this skill directory; ../../ is the repository root.

Repository root

Project overview, contribution and community policies, security reporting, and third-party license attributions.

Core documentation

Architecture, backend plugin API, C++ style, Python API, telemetry, and tracing.

  • Backend Plugin Interface: Southbound backend plugin API, plugin manager, capabilities, memory registration, transfers, notifications, and UCX.
  • NIXL Code Style: C++ naming, class layout, headers, formatting, Doxygen comments, and coding practices.
  • NIXL Doxygen Overview: Doxygen architecture overview of agents, memory sections, backends, metadata exchange, etcd, and Redis.
  • NIXL Architecture: Architecture, agents, memory sections, backends, metadata exchange with etcd and Redis, and TOML configuration.
  • NIXL Python API: Python API features, installation, backend parameters via get_plugin_params, QueryMem, UCX, and GDS.
  • NIXL Telemetry System: Telemetry events, metrics, shared-memory buffer, Prometheus, prometheus_mp, DOCA exporters, and readers.
  • NIXL Tracing System: Span tracing with NVTX, Nsight Systems, sampling, backend plugins, and correlation.
  • NIXL Usage Examples: Designs and frameworks that use NIXL to move model weights during RL post-training.

Getting Started (Fern)

NIXL overview, component architecture, and install-to-transfer quick start.

  • Overview: NIXL is a high-performance point-to-point data transfer library for AI inference frameworks.
  • Architecture: How NIXL's core components work together to enable high-performance data transfers.
  • Quick Start: Install NIXL and learn the complete transfer workflow -- from agent initialization to transfer completion.

User Guide (Fern)

Transfer backends, etcd metadata exchange, telemetry events and exporters, NIXLBench, and KVBench.

  • Backend Selection: How to choose the right NIXL transfer backend for your memory types and use case.
  • UCX: UCX transfer backend for high-performance VRAM and DRAM transfers via RDMA and TCP.
  • Libfabric: Libfabric (OFI) network backend for VRAM-to-VRAM, VRAM-to-DRAM, and DRAM-to-DRAM transfers as an alternative to UCX.
  • Mooncake: Mooncake Transfer Engine backend for VRAM and DRAM transfers in Mooncake-enabled deployments.
  • UCCL-P2P: UCCL-P2P network backend for VRAM and DRAM transfers.
  • DOCA GPUNetIO: DOCA GPUNetIO backend for high-performance GPU-to-GPU transfers using GPUDirect Async.
  • GPUDirect Storage: GPUDirect Storage backend for direct GPU-to-file transfers without CPU bounce buffers.
  • GPUDirect Storage MT: Multi-threaded GPUDirect Storage backend for higher throughput on parallel file operations.
  • POSIX: Standard POSIX I/O backend for DRAM-to-file transfers.
  • HF3FS: HuggingFace 3FS backend for file operations on 3FS fuse-based filesystems.
  • OBJ: Object storage backend supporting S3-compatible stores with multiple executor variants.
  • Azure Blob: Azure Blob Storage backend for DRAM-to-object-storage transfers via Azure REST API.
  • INFINIA: DDN INFINIA backend for asynchronous transfers between DRAM or VRAM and object storage.
  • DOCA_MEMOS: DOCA_MEMOS backend for key-value transfers between DRAM and an NVMe KV device on BlueField.
  • GUSLI: Block-level I/O backend for transfers between block storage devices and DRAM.
  • Metadata Exchange with etcd: Configure etcd-based distributed metadata exchange for NIXL agents -- key prefix scheme, watcher lifecycle, and Prometheus integration.
  • Telemetry Guide: Understanding NIXL's telemetry system -- event types, metrics, configuration, and monitoring with Prometheus.
  • Logging Guide: Configuring NIXL's logging -- verbosity, per-process log files, filename templates, size limits, and failure behavior.
  • NIXLBench: A benchmarking tool for measuring NIXL data transfer performance across network and storage backends.
  • Building NIXLBench: Build NIXLBench using Docker or natively from source with Meson.
  • Using NIXLBench: How to run NIXLBench benchmarks and troubleshoot common issues.
  • KVBench: A KV cache benchmarking utility that generates and runs NIXLBench commands to profile LLM inference transfer performance.
  • Building KVBench: Install KVBench using the NIXLBench Docker container or a Python virtual environment.
  • Using KVBench: KVBench command reference, model configuration guide, and LLM example configurations.

Developer Guide (Fern)

C++, Python, Rust, and Docker source builds, plus contribution guidance.

  • Building NIXL from Source: Build NIXL from source -- C++ library, Python bindings, Rust bindings, or Docker container.
  • Docker: Build and run NIXL in a Docker container.
  • NIXL C++ (Meson): Build the NIXL C++ library from source using the Meson build system.
  • Python Bindings: Build and install NIXL Python bindings from source.
  • Rust Bindings: Build NIXL Rust bindings from source using Meson or Cargo.
  • Contribution Guide: How to contribute to the NIXL project -- development setup, code style guidelines, testing requirements, and the contribution process.

Resources (Fern)

Environment variables for core, etcd, telemetry, and backends, plus troubleshooting.

  • Environment Variables: Complete reference for all NIXL environment variables covering core configuration, etcd, telemetry, and backend-specific settings.
  • Troubleshooting: Common errors, debugging steps, and solutions for NIXL build, runtime, and transfer issues.

Fern snippets and placeholder pages

Placeholder Fern pages and reusable backend environment-variable tables.

Network and peer-transfer plugins

RDMA, GPU-initiated, and peer-to-peer transfer backends.

  • DOCA GPUNetIO Plugin: DOCA GPUNetIO GDAKI backend for GPU-initiated RDMA on attached or pooled CUDA streams.
  • GPUDirect TCPXO Plugin: Build and run the GPUDirect TCPXO backend for GPU-to-GPU transfers on Google Cloud A3 VMs.
  • NIXLBench TCPXO Guide: Run NIXLBench on two Google Cloud A3 VMs to check a GPUDirect TCPXO setup.
  • Libfabric Plugin: Libfabric OFI multi-rail RDMA backend with GPU Direct, EFA, Neuron, and NUMA-aware rails.
  • Mooncake Backend Plugin: Mooncake Transfer Engine backend for TCP, RDMA, CXL shared memory, and NVMe-oF transfers.
  • UCCL Backend Plugin: UCCL P2P backend for RDMA, TCP, TCP-X, EFA, intranode transfers, and vLLM KV-cache disaggregation.

Storage backend plugins

File, block, object, and key-value storage backends.

  • Azure Blob Storage Plugin: Azure Blob backend using Azure SDK, Entra ID, Azurite, and asynchronous blob I/O.
  • DOCA_MEMOS Plugin: Build, configure, and troubleshoot the DOCA_MEMOS key-value plugin on BlueField.
  • GDS Plugin: GPUDirect Storage GDS and GDS_MT backends using cuFile batch and TaskFlow engines.
  • GUSLI Plugin: GUSLI client-server block I/O backend for shared-memory SPDK, networked, and kernel block devices.
  • HF3FS Plugin: HF3FS backend using hf3fs_usrbio for 3FS file I/O and shared-memory transfers.
  • Infinia Plugin: DDN Infinia coroutine object storage backend for DRAM, VRAM, and OBJ transfers.
  • Object Storage Plugin: AWS S3 OBJ backend with standard and CRT clients plus vendor-accelerated engines.
  • CUObject Client Config for S3 Accelerated: Configure CUObject RDMA addresses for accelerated S3 engines and speed up their startup.
  • Dell ObjectScale S3 Engine: Dell ObjectScale S3-over-RDMA OBJ engine using CUDA CUObject for DRAM and VRAM.
  • POSIX Plugin: POSIX file backend using Linux AIO, libaio, io_uring, liburing, and FILE_SEG.
  • Redis Plugin: Redis key-value backend using hiredis, libevent, connection pooling, asynchronous reads and writes, and queryMem.

Telemetry exporter plugins

Custom exporter development plus DOCA and Prometheus telemetry plugins.

Bindings and utilities

Python and Rust bindings, utility tests, device proxy, and file backend helpers.

  • nixl Python Meta Package: PyPI meta package selecting CUDA backends at runtime through PyTorch.
  • Rust Bindings for NIXL: Idiomatic Rust bindings, Cargo and stub-API builds, test suite, and CUDA memory-view tests.
  • Utility Tests: Unit tests for serialization, deserialization, UCX utility functions, and UCX active messages.
  • Device Proxy: Host CPU proxy turning GPU command rings into backend transfers; proxyTransport ownership, threading, and requests.
  • File Utilities: Shared QueryMem and path-mode FILE_SEG helpers for POSIX, HF3FS, GDS, and GDS_MT.

Examples

Runnable Python, C++, Rust, storage, and expert-parallel transfer examples.

  • NIXL Examples: Python, C++, and Rust overview; two-peer registration, prepped and combined transfers, reposting, and notifications.
  • C++ Examples: C++ examples for nixlAgent transfers, etcd metadata exchange, and telemetry reading.
  • NIXL EP: Expert-Parallel Communication Example: Device-API MoE dispatch and combine over RDMA and NVLink, Buffer, connect_ranks, and elastic scaling.
  • Elastic Test Suite: NIXL EP elastic.py single-node and multi-node rank expansion and contraction test plans.
  • Python Examples: Python two-peer tensor transfers, GDS, partial metadata, QueryMem, notifications, and telemetry.
  • NIXL Storage Transfer Sample: Client-server POSIX and GDS storage transfers with UCX networking and pipelined remote I/O.
  • Rust Examples: Single-process NIXL transfer example using Rust bindings.

Benchmarks

KVBench workloads, NIXLBench backend benchmarks, and etcd coordination runtime.

  • NIXL KVBench: LLM KV-cache transfer workloads, custom traffic matrices, RDMA, storage, and etcd coordination.
  • Adding a New Model Architecture: KVBench BaseModelArch subclasses, factory registration, YAML configuration, and KV-cache sizing.
  • Creating a Model Configuration: KVBench YAML for parallelism, quantization, sequence lengths, hardware, and access patterns.
  • Custom Traffic Performance Test: ct-perftest and sequential-ct-perftest traffic matrices for asymmetric KV-cache flows.
  • KVCache GDS Profiling: KVBench and NIXLBench GDS profiling for KV-cache transfers between files and GPU memory.
  • NIXL Benchmark: ASIO- or etcd-coordinated UCX, GPUNETIO, GDS, POSIX, and OBJ benchmarks with NIXL/NVSHMEM workers and communication patterns.
  • ETCD Python Runtime: etcd-based MPI replacement for KVBench via C++, pybind11, and _EtcdDistUtils; point-to-point, collectives, barriers, ranks.

CI, tests, contrib, and GitHub

CI jobs, crash debugging, GPU RDMA setup, tests, Conda builds, AWS testing, and pull requests.

  • Debugging CI Core Dumps: Finding CI sqsh exports, Enroot import via Slurm, locating core files, gdb, and debug symbols.
  • Build Wheel Matrix CI: Jenkins manylinux wheel matrix, Docker stages, UCX plugins, auditwheel, and Artifactory publishing.
  • NIXL CI Overview: GitHub Actions, Blossom-CI, Jenkins jobs, triggers, PR gating, nightly runs, and image caching.
  • CI Crash Debug Reference: Slurm crash-triggered Enroot exports to NFS, gdb availability, and cleanup.
  • GPU RDMA Test Setup: Ubuntu GPU host setup: drivers, MLNX_OFED, nvidia-peermem, GDRCopy, GDS, MPS, and Docker.
  • Test Descriptions: Agent, descriptor, metadata, UCX, Python binding, telemetry, and Google Test suites.
  • DOCA_MEMOS Unit Tests: Run the DOCA_MEMOS unit tests with mocked DOCA calls, so no BlueField hardware is needed.
  • DOCA GPUNetIO Unit Test: DOCA GPUNetIO GDAKI GPU transfer test with attached and pooled CUDA streams.
  • Conda Build Options: Conda C++ and Python binding builds and installs, plus manylinux wheel builds with OpenSSL 4.
  • NIXL AWS Testing: AWS Batch, EKS, and GitHub Actions tests for EFA and the S3 plugin.
  • Pull Request Template: What, Why, and How prompts for the change, its justification, and design.

C++ and Python APIs

Public C++ headers and Python modules for agents, descriptors, backends, telemetry, and logging.