Fix NVIDIA driver installation on Debian 12 security kernels - #1400
Conversation
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request adds a repository-based installation path for the NVIDIA GPU driver on Debian 12 to address compatibility issues with kernel 6.1.0-52. The feedback recommends using curl with retries and downloading the keyring package to /tmp instead of using wget in the current working directory, ensuring consistency with the rest of the script and preventing workspace pollution.
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
Signed-off-by: liyuan <yuali@nvidia.com>
|
CC @cjac could you help check thx |
|
/gcbrun |
Signed-off-by: liyuan <yuali@nvidia.com>
|
update the plugin version to 26.08.1, could you help trigger /gcbrun again? thx @cjac |
|
/gcbrun |
1 similar comment
|
/gcbrun |
|
The 2.2-debian12 run didn't work. I'm bringing it up in my lab now |
|
I just double checked the current PR head on a fresh 2.2-debian12 cluster (2.2.84-debian12), it successfully init |
|
Hi @cjac , not sure could we merge this? |
|
Sorry, I haven't had time to finish quite yet. Next week, I think, I'll have enough time complete the work. |
|
Sorry for the continued delay. I've got a lot on my plate but I may find myself with time this afternoon. |
|
taking a look now |
|
/gcbrun |
1 similar comment
|
/gcbrun |
|
oof. Debian 11 just went out of support. |
|
The 2.1-debian11 and 2.2-debian12 clusters aren't transitioning to RUNNING ; I'm debugging what might be going wrong. I'll continue this tomorrow |
|
/gcbrun |
This commit introduces DKMS module caching in GCS to drastically improve
cluster startup latency on identical nodes (especially in CI or scaling
events). It also addresses several critical compatibility issues with
Debian distributions.
### Changes Details:
- **DKMS Module Caching**:
- Implemented GCS-based caching for locally compiled DKMS modules using
`dkms mktarball` and `ldtarball`.
- Unique cache paths isolate builds by OS, Kernel, Arch, CUDA version,
and Optional Signing Key Modulus.
- Added support for mirroring signed DKMS cache to unsigned paths to
support systems booting in insecure mode.
- Added fallback support for standard `modulus_md5sum` metadata attribute.
- **Debian Compatibility Fixes**:
- **Debian 12**: Override incompatible NVIDIA 550 driver with version
`580.95.05`. The Debian 12 security kernel includes API changes
incompatible with the 550 series.
- **Debian 11**: Explicitly downgrade `libglapi-mesa` and `ca-certificates`
to resolve conflicts with standard `bullseye/main` versions required
by CUDA dependencies.
- **APT Maintenance for EOL Distros**:
- Set `Acquire::Check-Valid-Until "false"` to ignore expired release files.
- Remove defunct `debian-security` lines for Debian 10 and 11 to prevent
`404 Not Found` errors during `apt-get update`.
- **Build Pipeline Fix**:
- Updated Bazel GPG key URL to `releases.bazel.build` in CloudBuild Dockerfile.
cjac
left a comment
cjac
left a comment
There was a problem hiding this comment.
Code Review: Feature Branch fix-debian12-nvidia-580-dkms
Overview
This branch adds DKMS module caching to Dataproc and applies fixes for Debian compatibility (Debian 11 and 12) in NVIDIA driver installation paths.
Technical Details
DKMS Cache Implementation
- Function:
get_dkms_cache_pathderives paths based on host and kernel details. - Storage: Retains standard
dataproc-temp-bucketusage. - Recovery:
restore_dkms_cachechecks for cache existence and loads viadkms ldtarball. Execution proceeds if cache is missing. - Save Path:
save_dkms_cachecreates tarball viadkms mktarballand uploads to GCS. Supports fallback if--archiveflag fails.
OS Overrides
- Debian 11: Forces installation of
libglapi-mesa=20.3.5-1andca-certificates=20210119to resolve dependency conflicts with CUDA packages. - Debian 12: Overrides default driver to
580.95.05if550series is detected, avoiding kernel API incompatibilities.
Security
- Path Construction: Paths are derived from VM introspection and non-secret GCE metadata.
- Local Keys: Script relies on standard local permissions for key access.
Verdict
Code addresses startup latency via caching and resolves documented EOL repo blockers.
ffc4812 to
5519740
Compare
2 participants
Summary
nvidia-kernel-open-dkmsandnvidia-driver-cudapackages to avoid desktop EGL/Wayland dependencies.Root cause
Jenkins build 1631 failed after the Dataproc Debian 12 image picked up kernel
6.1.0-52-cloud-amd64(6.1.178). That security kernel uses the four-argumentpci_resize_resourceAPI, while the bundled NVIDIA 550 open module calls the old three-argument API and fails DKMS compilation.Test plan
bash -n spark-rapids/spark-rapids.shgit diff --check2.3-debian12(2.3.34-debian12) with the patched init action.6.1.0-50-cloud-amd64loaded NVIDIA driver580.178.04./usr/local/cuda/bin/nvccreports CUDA 12.4 on both workers.Validation artifacts are under
yuanli@spark-yuanli:~/work/csps/dataproc/v26.06/debian12-580-fix.