nixl-docs
Find and read NIXL documentation for APIs, architecture, backends, installation and builds, environment variables, telemetry, troubleshooting, examples, benchmarks, and contribution.
Try your first task
A source-grounded starter task has not been verified for this skill yet.
SKILL.md
name: nixl-docsdescription: Find and read NIXL documentation for APIs, architecture, backends, installation and builds, environment variables, telemetry, troubleshooting, examples, benchmarks, and contribution.license: Apache-2.0metadata: version: "1.0.0" author: NVIDIA NIXL Team <sronen@nvidia.com> tags: - nixl - docsNIXL Documentation Index
Purpose
Index of NIXL documentation for agents working in a NIXL repository clone.
Prerequisites
A clone of the NIXL repository.
Instructions
Find relevant files in the index and read them. Paths are relative to this skill directory; ../../ is the repository root.
Repository root
Project overview, contribution and community policies, security reporting, and third-party license attributions.
- NVIDIA Inference Xfer Library (NIXL): Project overview, supported platforms, source builds, Python wheel, Rust bindings, Docker, ROCm, examples, and etcd.
- Contribution Guidelines: Development environment, code standards, contribution process, plugin development, testing, documentation, pull requests, and DCO.
- Contributor Covenant Code of Conduct: Community behavior standards, scope, reporting, and enforcement guidelines.
- Report a Security Vulnerability: Security vulnerability reporting via NVIDIA PSIRT email, web form, PGP key, and required details.
- C++ third-party attributions: Third-party C and C++ libraries, repository links, and license texts.
- Python third-party attributions: Generated third-party Python package attributions, project links, and license texts.
- Rust third-party attributions: Generated third-party Rust crate attributions, repository links, and license texts.
Core documentation
Architecture, backend plugin API, C++ style, Python API, telemetry, and tracing.
- Backend Plugin Interface: Southbound backend plugin API, plugin manager, capabilities, memory registration, transfers, notifications, and UCX.
- NIXL Code Style: C++ naming, class layout, headers, formatting, Doxygen comments, and coding practices.
- NIXL Doxygen Overview: Doxygen architecture overview of agents, memory sections, backends, metadata exchange, etcd, and Redis.
- NIXL Architecture: Architecture, agents, memory sections, backends, metadata exchange with etcd and Redis, and TOML configuration.
- NIXL Python API: Python API features, installation, backend parameters via get_plugin_params, QueryMem, UCX, and GDS.
- NIXL Telemetry System: Telemetry events, metrics, shared-memory buffer, Prometheus, prometheus_mp, DOCA exporters, and readers.
- NIXL Tracing System: Span tracing with NVTX, Nsight Systems, sampling, backend plugins, and correlation.
- NIXL Usage Examples: Designs and frameworks that use NIXL to move model weights during RL post-training.
Getting Started (Fern)
NIXL overview, component architecture, and install-to-transfer quick start.
- Overview: NIXL is a high-performance point-to-point data transfer library for AI inference frameworks.
- Architecture: How NIXL's core components work together to enable high-performance data transfers.
- Quick Start: Install NIXL and learn the complete transfer workflow -- from agent initialization to transfer completion.
User Guide (Fern)
Transfer backends, etcd metadata exchange, telemetry events and exporters, NIXLBench, and KVBench.
- Backend Selection: How to choose the right NIXL transfer backend for your memory types and use case.
- UCX: UCX transfer backend for high-performance VRAM and DRAM transfers via RDMA and TCP.
- Libfabric: Libfabric (OFI) network backend for VRAM-to-VRAM, VRAM-to-DRAM, and DRAM-to-DRAM transfers as an alternative to UCX.
- Mooncake: Mooncake Transfer Engine backend for VRAM and DRAM transfers in Mooncake-enabled deployments.
- UCCL-P2P: UCCL-P2P network backend for VRAM and DRAM transfers.
- DOCA GPUNetIO: DOCA GPUNetIO backend for high-performance GPU-to-GPU transfers using GPUDirect Async.
- GPUDirect Storage: GPUDirect Storage backend for direct GPU-to-file transfers without CPU bounce buffers.
- GPUDirect Storage MT: Multi-threaded GPUDirect Storage backend for higher throughput on parallel file operations.
- POSIX: Standard POSIX I/O backend for DRAM-to-file transfers.
- HF3FS: HuggingFace 3FS backend for file operations on 3FS fuse-based filesystems.
- OBJ: Object storage backend supporting S3-compatible stores with multiple executor variants.
- Azure Blob: Azure Blob Storage backend for DRAM-to-object-storage transfers via Azure REST API.
- INFINIA: DDN INFINIA backend for asynchronous transfers between DRAM or VRAM and object storage.
- DOCA_MEMOS: DOCA_MEMOS backend for key-value transfers between DRAM and an NVMe KV device on BlueField.
- GUSLI: Block-level I/O backend for transfers between block storage devices and DRAM.
- Metadata Exchange with etcd: Configure etcd-based distributed metadata exchange for NIXL agents -- key prefix scheme, watcher lifecycle, and Prometheus integration.
- Telemetry Guide: Understanding NIXL's telemetry system -- event types, metrics, configuration, and monitoring with Prometheus.
- Logging Guide: Configuring NIXL's logging -- verbosity, per-process log files, filename templates, size limits, and failure behavior.
- NIXLBench: A benchmarking tool for measuring NIXL data transfer performance across network and storage backends.
- Building NIXLBench: Build NIXLBench using Docker or natively from source with Meson.
- Using NIXLBench: How to run NIXLBench benchmarks and troubleshoot common issues.
- KVBench: A KV cache benchmarking utility that generates and runs NIXLBench commands to profile LLM inference transfer performance.
- Building KVBench: Install KVBench using the NIXLBench Docker container or a Python virtual environment.
- Using KVBench: KVBench command reference, model configuration guide, and LLM example configurations.
Developer Guide (Fern)
C++, Python, Rust, and Docker source builds, plus contribution guidance.
- Building NIXL from Source: Build NIXL from source -- C++ library, Python bindings, Rust bindings, or Docker container.
- Docker: Build and run NIXL in a Docker container.
- NIXL C++ (Meson): Build the NIXL C++ library from source using the Meson build system.
- Python Bindings: Build and install NIXL Python bindings from source.
- Rust Bindings: Build NIXL Rust bindings from source using Meson or Cargo.
- Contribution Guide: How to contribute to the NIXL project -- development setup, code style guidelines, testing requirements, and the contribution process.
Resources (Fern)
Environment variables for core, etcd, telemetry, and backends, plus troubleshooting.
- Environment Variables: Complete reference for all NIXL environment variables covering core configuration, etcd, telemetry, and backend-specific settings.
- Troubleshooting: Common errors, debugging steps, and solutions for NIXL build, runtime, and transfer issues.
Fern snippets and placeholder pages
Placeholder Fern pages and reusable backend environment-variable tables.
- NIXL Documentation placeholder: Placeholder Fern page containing only the NIXL Documentation title.
- Second Page placeholder: Placeholder Fern page containing only the Second Page title.
- Azure Blob environment variables: AZURE_STORAGE account, container, and connection-string variables, plus AZURE_CA_BUNDLE.
- GDS environment variables: CUFILE_ENV_PATH_JSON for the GPUDirect Storage cuFile configuration path.
- Libfabric environment variables: NIXL_LIBFABRIC_MAX_BW_PER_DRAM_SEG for NUMA-aware DRAM rail bandwidth and dynamic provider-specific NIXL_LIBFABRIC_ variables.
- Mooncake environment variables: NIXL_MOONCAKE_IP_ADDR override for the Mooncake Transfer Engine network interface.
- OBJ environment variables: AWS_DEFAULT_BUCKET, AWS_ENDPOINT_OVERRIDE for S3-compatible stores, and AWS SDK credential and region variables.
- UCX environment variables: NIXL_UCX_WARNING_TIMEOUT for UCX memory registration warnings.
Network and peer-transfer plugins
RDMA, GPU-initiated, and peer-to-peer transfer backends.
- DOCA GPUNetIO Plugin: DOCA GPUNetIO GDAKI backend for GPU-initiated RDMA on attached or pooled CUDA streams.
- GPUDirect TCPXO Plugin: Build and run the GPUDirect TCPXO backend for GPU-to-GPU transfers on Google Cloud A3 VMs.
- NIXLBench TCPXO Guide: Run NIXLBench on two Google Cloud A3 VMs to check a GPUDirect TCPXO setup.
- Libfabric Plugin: Libfabric OFI multi-rail RDMA backend with GPU Direct, EFA, Neuron, and NUMA-aware rails.
- Mooncake Backend Plugin: Mooncake Transfer Engine backend for TCP, RDMA, CXL shared memory, and NVMe-oF transfers.
- UCCL Backend Plugin: UCCL P2P backend for RDMA, TCP, TCP-X, EFA, intranode transfers, and vLLM KV-cache disaggregation.
Storage backend plugins
File, block, object, and key-value storage backends.
- Azure Blob Storage Plugin: Azure Blob backend using Azure SDK, Entra ID, Azurite, and asynchronous blob I/O.
- DOCA_MEMOS Plugin: Build, configure, and troubleshoot the DOCA_MEMOS key-value plugin on BlueField.
- GDS Plugin: GPUDirect Storage GDS and GDS_MT backends using cuFile batch and TaskFlow engines.
- GUSLI Plugin: GUSLI client-server block I/O backend for shared-memory SPDK, networked, and kernel block devices.
- HF3FS Plugin: HF3FS backend using hf3fs_usrbio for 3FS file I/O and shared-memory transfers.
- Infinia Plugin: DDN Infinia coroutine object storage backend for DRAM, VRAM, and OBJ transfers.
- Object Storage Plugin: AWS S3 OBJ backend with standard and CRT clients plus vendor-accelerated engines.
- CUObject Client Config for S3 Accelerated: Configure CUObject RDMA addresses for accelerated S3 engines and speed up their startup.
- Dell ObjectScale S3 Engine: Dell ObjectScale S3-over-RDMA OBJ engine using CUDA CUObject for DRAM and VRAM.
- POSIX Plugin: POSIX file backend using Linux AIO, libaio, io_uring, liburing, and FILE_SEG.
- Redis Plugin: Redis key-value backend using hiredis, libevent, connection pooling, asynchronous reads and writes, and queryMem.
Telemetry exporter plugins
Custom exporter development plus DOCA and Prometheus telemetry plugins.
- Custom Telemetry Plugin Development Guide: Custom telemetry exporter interface, built-in events, metric filtering, CSV example, and Meson build.
- DOCA Telemetry Exporter: DOCA exporter with Prometheus scrape and DTS IPC delivery backends, histograms, labels, and metric filtering.
- Prometheus Telemetry Exporter: Single Prometheus endpoint aggregating all ranks via mmap stores, flock election, and stale-store reaping.
Bindings and utilities
Python and Rust bindings, utility tests, device proxy, and file backend helpers.
- nixl Python Meta Package: PyPI meta package selecting CUDA backends at runtime through PyTorch.
- Rust Bindings for NIXL: Idiomatic Rust bindings, Cargo and stub-API builds, test suite, and CUDA memory-view tests.
- Utility Tests: Unit tests for serialization, deserialization, UCX utility functions, and UCX active messages.
- Device Proxy: Host CPU proxy turning GPU command rings into backend transfers; proxyTransport ownership, threading, and requests.
- File Utilities: Shared QueryMem and path-mode FILE_SEG helpers for POSIX, HF3FS, GDS, and GDS_MT.
Examples
Runnable Python, C++, Rust, storage, and expert-parallel transfer examples.
- NIXL Examples: Python, C++, and Rust overview; two-peer registration, prepped and combined transfers, reposting, and notifications.
- C++ Examples: C++ examples for nixlAgent transfers, etcd metadata exchange, and telemetry reading.
- NIXL EP: Expert-Parallel Communication Example: Device-API MoE dispatch and combine over RDMA and NVLink, Buffer, connect_ranks, and elastic scaling.
- Elastic Test Suite: NIXL EP elastic.py single-node and multi-node rank expansion and contraction test plans.
- Python Examples: Python two-peer tensor transfers, GDS, partial metadata, QueryMem, notifications, and telemetry.
- NIXL Storage Transfer Sample: Client-server POSIX and GDS storage transfers with UCX networking and pipelined remote I/O.
- Rust Examples: Single-process NIXL transfer example using Rust bindings.
Benchmarks
KVBench workloads, NIXLBench backend benchmarks, and etcd coordination runtime.
- NIXL KVBench: LLM KV-cache transfer workloads, custom traffic matrices, RDMA, storage, and etcd coordination.
- Adding a New Model Architecture: KVBench BaseModelArch subclasses, factory registration, YAML configuration, and KV-cache sizing.
- Creating a Model Configuration: KVBench YAML for parallelism, quantization, sequence lengths, hardware, and access patterns.
- Custom Traffic Performance Test: ct-perftest and sequential-ct-perftest traffic matrices for asymmetric KV-cache flows.
- KVCache GDS Profiling: KVBench and NIXLBench GDS profiling for KV-cache transfers between files and GPU memory.
- NIXL Benchmark: ASIO- or etcd-coordinated UCX, GPUNETIO, GDS, POSIX, and OBJ benchmarks with NIXL/NVSHMEM workers and communication patterns.
- ETCD Python Runtime: etcd-based MPI replacement for KVBench via C++, pybind11, and _EtcdDistUtils; point-to-point, collectives, barriers, ranks.
CI, tests, contrib, and GitHub
CI jobs, crash debugging, GPU RDMA setup, tests, Conda builds, AWS testing, and pull requests.
- Debugging CI Core Dumps: Finding CI sqsh exports, Enroot import via Slurm, locating core files, gdb, and debug symbols.
- Build Wheel Matrix CI: Jenkins manylinux wheel matrix, Docker stages, UCX plugins, auditwheel, and Artifactory publishing.
- NIXL CI Overview: GitHub Actions, Blossom-CI, Jenkins jobs, triggers, PR gating, nightly runs, and image caching.
- CI Crash Debug Reference: Slurm crash-triggered Enroot exports to NFS, gdb availability, and cleanup.
- GPU RDMA Test Setup: Ubuntu GPU host setup: drivers, MLNX_OFED, nvidia-peermem, GDRCopy, GDS, MPS, and Docker.
- Test Descriptions: Agent, descriptor, metadata, UCX, Python binding, telemetry, and Google Test suites.
- DOCA_MEMOS Unit Tests: Run the DOCA_MEMOS unit tests with mocked DOCA calls, so no BlueField hardware is needed.
- DOCA GPUNetIO Unit Test: DOCA GPUNetIO GDAKI GPU transfer test with attached and pooled CUDA streams.
- Conda Build Options: Conda C++ and Python binding builds and installs, plus manylinux wheel builds with OpenSSL 4.
- NIXL AWS Testing: AWS Batch, EKS, and GitHub Actions tests for EFA and the S3 plugin.
- Pull Request Template: What, Why, and How prompts for the change, its justification, and design.
C++ and Python APIs
Public C++ headers and Python modules for agents, descriptors, backends, telemetry, and logging.
- nixlAgent (nixl.h): Declares nixlAgent for backends, memory registration, transfers, memory views, notifications, and peer and etcd metadata.
- Descriptors (nixl_descriptors.h): Declares nixlBasicDesc, nixlStrideDesc, nixlBlobDesc, nixlRemoteDesc, nixlDescList, and descriptor-list aliases.
- nixlAgentConfig (nixl_params.h): Declares nixlAgentConfig for progress and listener threads, synchronization, telemetry, and etcd.
- Core types (nixl_types.h): Declares nixl_mem_t, nixl_xfer_op_t, nixl_status_t, nixl_thread_sync_t, nixlAgentOptionalArgs, nixlMemViewH, and nixlXferTelemetry.
- Backend auxiliary types (backend_aux.h): Declares nixlBackendInitParams, nixlBackendOptionalArgs, nixlBackendReqH, nixlBackendMD, nixlMetaDesc, and nixlMetaStrideDescList.
- nixlBackendEngine (backend_engine.h): Declares nixlBackendEngine and its capability flags for registration, connections, transfers, notifications, queryMem, and telemetry events.
- nixlBackendPlugin (backend_plugin.h): Declares nixlBackendPlugin, creator template, plugin API, and dynamic loading entry points.
- nixlTelemetryExporter (telemetry_exporter.h): Declares nixlTelemetryExporter, exporter initialization, batch hooks, and bind-failure handling.
- nixlTelemetryPlugin (telemetry_plugin.h): Declares nixlTelemetryPlugin, exporter creator, plugin API, and dynamic loading entry points.
- Python package exports (init.py): Re-exports nixl_agent, nixl_agent_config, nixl_backend_handle, nixl_prepped_dlist_handle, nixl_xfer_handle, nixl_thread_sync_t, and DEFAULT_COMM_PORT.
- Python nixl_agent API (_api.py): Implements nixl_agent, configuration, descriptor helpers, memory registration, transfers, notifications, and metadata.
- Python logging (logging.py): Configures Python logging, get_logger, NIXL_LOG_LEVEL, and NIXL_DEBUG_LOGGING.