diff --git a/container-toolkit/cdi-support.md b/container-toolkit/cdi-support.md index 1e9dd2273..e548d107d 100644 --- a/container-toolkit/cdi-support.md +++ b/container-toolkit/cdi-support.md @@ -161,6 +161,59 @@ directly: $ sudo nvidia-ctk cdi generate --output=/var/run/cdi/nvidia.yaml ``` +## JIT-CDI Mode + +Just-in-time CDI (JIT-CDI) mode generates an in-memory CDI specification for the +NVIDIA devices that a container requests. The NVIDIA Container Runtime uses this +specification to update the container configuration. JIT-CDI mode does not write +a persistent specification to `/var/run/cdi` or `/etc/cdi`. + +When `nvidia-container-runtime.mode` is set to `auto`, the runtime selects JIT-CDI +mode on systems that use the NVIDIA Management Library (NVML) or Windows +Subsystem for Linux 2 (WSL2). Native CDI-enabled runtimes still use the +persistent specifications described in +[Automatic CDI Specification Generation](#automatic-cdi-specification-generation). + +### Disabling Hooks in JIT-CDI Mode + +You can prevent JIT-CDI mode from adding specific hooks to an in-memory specification. +The supported hook names are `create-symlinks`, +`disable-device-node-modification`, `enable-cuda-compat`, +`update-application-profile`, and `update-ldcache`. +Use `all` to disable every hook. + +The following command disables the application-profile and dynamic-linker cache +hooks. Separate multiple values with a colon: + +```console +$ sudo nvidia-ctk config --in-place \ + --set nvidia-container-runtime.modes.jit-cdi.nvcdi-disable-hooks=update-application-profile:update-ldcache +``` + +The resulting configuration contains the following settings: + +```toml +[nvidia-container-runtime.modes.jit-cdi] +nvcdi-disable-hooks = ["update-application-profile", "update-ldcache"] +``` + +```{warning} +Disable only the hook that conflicts with your environment. Hooks configure +libraries, links, device behavior, and GPU visibility in the container. Disabling +a required hook can prevent an application from starting or can expose more GPUs +to EGL and Vulkan applications than the container requested. +``` + +For a persistent specification, pass `--disable-hook` once for each hook when you +[generate the CDI specification manually](#manual-cdi-specification-generation): + +```console +$ sudo nvidia-ctk cdi generate \ + --disable-hook update-application-profile \ + --disable-hook update-ldcache \ + --output=/var/run/cdi/nvidia.yaml +``` + ## Running a Workload with CDI Using CDI to inject NVIDIA devices can conflict with using the NVIDIA Container Runtime hook. @@ -191,6 +244,13 @@ $ podman run --rm \ The preceding sample command requests the full GPU with index 0 and the first MIG device on GPU 1. The output should show only the UUIDs of the requested devices. +### IMEX Channels + +Containers can request specific NVIDIA IMEX channels with the +`NVIDIA_IMEX_CHANNELS` environment variable. For supported values, an example, +and validation errors, refer to +[Requesting IMEX Channels](docker-specialized.md#requesting-imex-channels). + ## Using CDI with Non-CDI-Enabled Runtimes To support runtimes that do not natively support CDI, you can configure the NVIDIA Container Runtime in a `cdi` mode. @@ -237,4 +297,4 @@ $ docker run --rm -ti --runtime=nvidia \ - [Container Device Interface](https://github.com/cncf-tags/container-device-interface) (CDI) specification from the Container Device Interface repository on GitHub. - [How to configure CDI](https://github.com/cncf-tags/container-device-interface#how-to-configure-cdi) from the GitHub repository provides an overview of manual configuration for CRI-O, containerd, and Podman. - The NVIDIA Container Toolkit performs the configuration for you. \ No newline at end of file + The NVIDIA Container Toolkit performs the configuration for you. diff --git a/container-toolkit/docker-specialized.md b/container-toolkit/docker-specialized.md index 6b88c1c5e..4d134b0a9 100644 --- a/container-toolkit/docker-specialized.md +++ b/container-toolkit/docker-specialized.md @@ -111,6 +111,34 @@ The following examples show common usage: nvidia/cuda nvidia-smi ``` +(requesting-imex-channels)= + +### Requesting IMEX Channels + +Use the `NVIDIA_IMEX_CHANNELS` environment variable to request NVIDIA IMEX +channels for a container. Specify one or more numeric channel IDs as a +comma-separated list. + +The following command requests channels 0 and 1: + +```console +$ docker run --rm --runtime=nvidia \ + -e NVIDIA_VISIBLE_DEVICES=all \ + -e NVIDIA_IMEX_CHANNELS=0,1 \ + +``` + +In CDI and JIT-CDI mode, each channel ID must meet both requirements: + +- The ID is in the range from 0 through 1,048,575. +- The corresponding `/dev/nvidia-caps-imex-channels/channel` device exists + on the host. + +If either requirement is not met, container creation fails with an error that +identifies the invalid or missing channel. Inspect +`/dev/nvidia-caps-imex-channels/` on the host and request only the channel IDs +that are present. + ### Driver Capabilities The `NVIDIA_DRIVER_CAPABILITIES` variable controls which driver libraries and binaries are mounted inside the container. @@ -146,13 +174,14 @@ The following table describes the supported driver capabilities: - Description * - ``compute`` - - required for CUDA and OpenCL applications. + - Required for CUDA and OpenCL applications. When present on the host, + the NVIDIA OpenCL ICD file is also available in the container. * - ``compat32`` - required for running 32-bit applications. * - ``graphics`` - - required for running OpenGL and Vulkan applications. + - Required for running OpenGL, EGL, and Vulkan applications. * - ``utility`` - required for using ``nvidia-smi`` and NVML. @@ -178,6 +207,26 @@ For example, to allow usage of CUDA and NVML, specify the `compute` and `utility > nvidia/cuda:12.5.0-base-ubuntu22.04 nvidia-smi > ``` +To run an OpenGL, EGL, or Vulkan application on a selected GPU, include the +`graphics` capability. The following command makes GPU 0 and the graphics and +utility driver components available to the container: + +```console +$ docker run --rm --runtime=nvidia \ + -e NVIDIA_VISIBLE_DEVICES=0 \ + -e NVIDIA_DRIVER_CAPABILITIES=graphics,utility \ + +``` + +Legacy mode limits EGL and Vulkan visibility to the physical GPUs assigned to +a container when the `graphics` or `display` driver capability is enabled. +The 1.20.0 release adds the same behavior to CDI and JIT-CDI modes through the +`update-application-profile` hook. This change provides feature parity across +the modes. + +If the application-profile hook conflicts with an application, refer to +[Disabling Hooks in JIT-CDI Mode](cdi-support.md#disabling-hooks-in-jit-cdi-mode). + ### Constraints The NVIDIA runtime also lets you define constraints on the configurations that the container supports. diff --git a/container-toolkit/install-guide.md b/container-toolkit/install-guide.md index 56282dab4..81480f170 100644 --- a/container-toolkit/install-guide.md +++ b/container-toolkit/install-guide.md @@ -149,6 +149,49 @@ where `systemd` cgroup drivers are used that cause containers to lose access to * You installed a supported container engine (Docker, Containerd, CRI-O, Podman). * You installed the NVIDIA Container Toolkit. +### Managing `config.toml` + +The `nvidia-ctk config` command reads +`/etc/nvidia-container-runtime/config.toml` by default. Without `--in-place` or +`--output`, the command writes the updated configuration to standard output and +does not change the source file. + +Preview a change before you write it to the host: + +```console +$ sudo nvidia-ctk config \ + --set nvidia-container-runtime.log-level=debug +``` + +Review the output. Then, add `--in-place` to update the source file: + +```console +$ sudo nvidia-ctk config --in-place \ + --set nvidia-container-runtime.log-level=debug +``` + +You can specify `--set` more than once. The following command configures a log +level and a log file in one update: + +```console +$ sudo nvidia-ctk config --in-place \ + --set nvidia-container-runtime.log-level=debug \ + --set nvidia-container-runtime.debug=/var/log/nvidia-container-runtime.log +``` + +For a list setting, separate elements with a colon. The following command makes +`crun` the first low-level runtime candidate and retains `runc` as a fallback: + +```console +$ sudo nvidia-ctk config --in-place \ + --set nvidia-container-runtime.runtimes=crun:runc +``` + +When the same key appears more than once, the last value takes effect. The +command preserves existing settings when you change a different setting. Review +the preview before every production update, especially when you use a custom +file with `--config-file`. + (setting-up-docker)= ### Configuring Docker diff --git a/container-toolkit/release-notes.md b/container-toolkit/release-notes.md index 29521bfe9..cd70b5046 100644 --- a/container-toolkit/release-notes.md +++ b/container-toolkit/release-notes.md @@ -8,6 +8,75 @@ This document describes the new features, improvements, fixes and known issues for the NVIDIA Container Toolkit. +## NVIDIA Container Toolkit 1.20.0 + +This release of the NVIDIA Container Toolkit `v1.20.0` is a feature release. + +### Fixes and Features + +- CDI specifications can now include an application-profile hook that limits EGL and Vulkan visibility to the GPUs assigned to the container. + Graphics applications no longer see unassigned host GPUs through these APIs. + For more information, refer to [issue #1899](https://github.com/NVIDIA/nvidia-container-toolkit/issues/1899) and [PR #1939](https://github.com/NVIDIA/nvidia-container-toolkit/pull/1939). +- CUDA compatibility handling now uses `libcuda.so` ELF metadata whenever it is available. + This improves CUDA minor-version compatibility by selecting the container's compatibility libraries only when they are appropriate for the installed driver. +- Driver file discovery now supports libraries spread across multiple directories, as occurs on distributions such as Debian, and matches graphics libraries against the exact installed driver version. + This fix avoids both missing required libraries and injecting libraries from another installed driver version. + For information about multiple-directory discovery, refer to [issue #1559](https://github.com/NVIDIA/nvidia-container-toolkit/issues/1559) and [PR #1820](https://github.com/NVIDIA/nvidia-container-toolkit/pull/1820). For information about exact-version matching, refer to [PR #1948](https://github.com/NVIDIA/nvidia-container-toolkit/pull/1948). +- CDI specifications now include the NVIDIA OpenCL ICD file and the legacy `libnvidia-nvvm70.so` library when present. + OpenCL loaders can locate the NVIDIA implementation, and workloads that depend on the legacy NVVM library receive it automatically. + For information about OpenCL support, refer to [issue #682](https://github.com/NVIDIA/nvidia-container-toolkit/issues/682) and [PR #1893](https://github.com/NVIDIA/nvidia-container-toolkit/pull/1893). For information about legacy NVVM support, refer to [issue #1875](https://github.com/NVIDIA/nvidia-container-toolkit/issues/1875) and [PR #1876](https://github.com/NVIDIA/nvidia-container-toolkit/pull/1876). +- On WSL2, CDI discovery now includes additional `.so`, `.bin`, and `.dll` files from the NVIDIA driver store instead of relying only on a fixed file list. + This fix enables containers to receive driver components introduced by newer Windows driver releases without waiting for a toolkit-specific allowlist update. + For more information, refer to [issue #1864](https://github.com/NVIDIA/nvidia-container-toolkit/issues/1864) and [PR #1890](https://github.com/NVIDIA/nvidia-container-toolkit/pull/1890). +- IMEX channel requests in CDI and JIT-CDI mode are now validated for both the supported ID range and the presence of the corresponding host device. + Invalid requests fail with a clear error. + For more information, refer to [issue #1309](https://github.com/NVIDIA/nvidia-container-toolkit/issues/1309) and [PR #1913](https://github.com/NVIDIA/nvidia-container-toolkit/pull/1913). +- JIT-CDI mode now honors the `nvidia-container-runtime.modes.jit-cdi.nvcdi-disable-hooks` configuration option. + You can disable individual CDI hooks for environments where a generated hook is unnecessary or incompatible. +- Updating `config.toml` no longer reverts previously modified options to their defaults. +- CDI generation no longer adds an `update-ldcache` hook when it discovers no driver libraries. + This fix prevents containers from running an unnecessary hook and avoids failures on systems or modes that do not inject libraries. + For more information, refer to [issue #373](https://github.com/NVIDIA/nvidia-container-toolkit/issues/373) and [PR #1894](https://github.com/NVIDIA/nvidia-container-toolkit/pull/1894). +- Fixed an issue where NVIDIA runtime handlers in a generated containerd + drop-in configuration could omit `runtime_type` when the base configuration + did not define it. + The affected containers failed to start with a + `container.Runtime.Name must be set` error. + The toolkit now sets the default runtime type when the field is missing or empty. + For more information, refer to + [issue #1956](https://github.com/NVIDIA/nvidia-container-toolkit/issues/1956) + and [PR #1969](https://github.com/NVIDIA/nvidia-container-toolkit/pull/1969). + +### Packaging Changes + +- RPMs rebuilt by the toolkit packaging image now use XZ payload compression instead of zstd. + The resulting packages can be installed on older Linux distributions that do not support zstd compression, such as Amazon Linux 2. +- Source package builds can use Podman by setting `DOCKER=podman`. + The build handles Podman's local image naming, SELinux volume labeling, and artifact-directory creation automatically. + +#### Enhancements to container-toolkit Container Images + +- The `container-toolkit` image now uses the non-development distroless base and includes a static BusyBox shell. + Init-container wrappers and lifecycle hooks retain the shell commands they need without depending on the development image. +- The NRI plugin can inject ordinary workload CDI devices outside the toolkit namespace. + For management devices, you can authorize additional namespaces with the `--nri-management-cdi-device-namespaces` option or the `NRI_MANAGEMENT_CDI_DEVICE_NAMESPACES` environment variable, enabling centralized management workloads without granting access cluster-wide. +- The toolkit installer now installs `nvidia-cdi-hook` instead of wrapping it with a shell script. + NRI-based deployments can invoke the hook on hosts that do not provide a shell. + +### Included Packages + +The following packages are included: + +- `nvidia-container-toolkit 1.20.0` +- `nvidia-container-toolkit-base 1.20.0` +- `libnvidia-container-tools 1.20.0` +- `libnvidia-container1 1.20.0` + +The following `container-toolkit` containers are included: + +- `nvcr.io/nvidia/k8s/container-toolkit:v1.20.0` +- `nvcr.io/nvidia/k8s/container-toolkit:v1.20.0-packaging` + ## NVIDIA Container Toolkit 1.19.1 This release of the NVIDIA Container Toolkit `v1.19.1` is a bugfix release. @@ -49,7 +118,7 @@ v0.7.0 of the CDI spec schema. The container runtime support for v0.7.0 of the C - podman >= 5.1.0 - [containers/podman@a40cf31](https://github.com/containers/podman/commit/a40cf3195acb6ac5fea5ab4617afb99006a3bed7) - crio >= 1.30.0 - [cri-o/cri-o@fd9aa76](https://github.com/cri-o/cri-o/commit/fd9aa76250fe05625d8c968b922cd1a0ae88eb1b) -If you are using a container runtime version that does not support v0.7.0 of the CDI spec schema, +If you are using a container runtime version that does not support v0.7.0 of the CDI spec schema, it is recommended to set the `no-additional-gids-for-device-nodes` CDI feature flag so that an older version of the CDI spec schema is used for spec file generation: diff --git a/container-toolkit/versions1.json b/container-toolkit/versions1.json index 24b46d827..b1f6d6c16 100644 --- a/container-toolkit/versions1.json +++ b/container-toolkit/versions1.json @@ -1,6 +1,10 @@ [ { "preferred": "true", + "url": "../1.20/", + "version": "1.20" + }, + { "url": "../1.19.1/", "version": "1.19.1" }, @@ -23,49 +27,5 @@ { "url": "../1.17.8/", "version": "1.17.8" - }, - { - "url": "../1.17.7/", - "version": "1.17.7" - }, - { - "url": "../1.17.6/", - "version": "1.17.6" - }, - { - "url": "../1.17.5/", - "version": "1.17.5" - }, - { - "url": "../1.17.4/", - "version": "1.17.4" - }, - { - "url": "../1.17.3/", - "version": "1.17.3" - }, - { - "url": "../1.17.2/", - "version": "1.17.2" - }, - { - "url": "../1.17.1/", - "version": "1.17.1" - }, - { - "url": "../1.17.0/", - "version": "1.17.0" - }, - { - "url": "../1.16.2/", - "version": "1.16.2" - }, - { - "url": "../1.16.1/", - "version": "1.16.1" - }, - { - "url": "../1.16.0/", - "version": "1.16.0" } ] diff --git a/repo.toml b/repo.toml index 374e1d164..458924fd9 100644 --- a/repo.toml +++ b/repo.toml @@ -110,8 +110,8 @@ project_build_order = [ docs_root = "${root}/container-toolkit" project = "container-toolkit" name = "NVIDIA Container Toolkit" -version = "1.19.1" -source_substitutions = {version = "1.19.1"} +version = "1.20" +source_substitutions = {version = "1.20"} copyright_start = 2020 redirects = [ { path="concepts.html", target="index.html" },