diff --git a/docs/cuopt/source/convex-features.rst b/docs/cuopt/source/convex-features.rst index 82de2dbc78..d1239d6345 100644 --- a/docs/cuopt/source/convex-features.rst +++ b/docs/cuopt/source/convex-features.rst @@ -325,4 +325,7 @@ By default, PDLP operates in the native precision of the problem type (FP64 for Multi-GPU Mode -------------- -Users can use multiple GPUs to solve a problem by specifying the ``num_gpus`` parameter. The feature is restricted to LP problems that uses concurrent mode and supports up to 2 GPUs at the moment. Using this mode will run PDLP and barrier in parallel on different GPUs to avoid sharing single GPU resources. +Users can use multiple GPUs to solve a problem by specifying the ``num_gpus`` parameter, in one of two ways: + +- **Concurrent mode**: restricted to LP problems that use concurrent mode and supports up to 2 GPUs. Using this mode will run PDLP and barrier in parallel on different GPUs to avoid sharing single GPU resources. +- **Multi-GPU PDLP**: restricted to LP problems solved with the ``PDLP`` method. Setting ``num_gpus`` to ``-1`` or to a value greater than 1, together with ``use_distributed_pdlp`` (Python: ``use_mpdlp``) set to true, shards a single PDLP solve across multiple GPUs. ``-1`` selects all GPUs visible to the process, which may resolve to a single GPU on a single-GPU host; multi-GPU sharding only happens when more than one GPU is actually selected. Use ``distributed_pdlp_partitioner`` (Python: ``mpdlp_partitioner``) to control how the problem is split across GPUs. See :ref:`distributed-pdlp-partitioner-constants` for the available strategies. diff --git a/docs/cuopt/source/convex-settings.rst b/docs/cuopt/source/convex-settings.rst index 22d290c7c2..66ae2d1c45 100644 --- a/docs/cuopt/source/convex-settings.rst +++ b/docs/cuopt/source/convex-settings.rst @@ -129,7 +129,38 @@ cuOpt will stop at the first limit (iteration or time) reached. Number of GPUs ^^^^^^^^^^^^^^ -``CUOPT_NUM_GPUS`` controls the number of GPUs to use for the solve. This setting is only relevant for LP problems that uses concurrent mode and supports up to 2 GPUs at the moment. Using this mode will run PDLP and barrier in parallel on different GPUs to avoid sharing single GPU resources. +``CUOPT_NUM_GPUS`` controls the number of GPUs to use for the solve. + +For LP problems solved with ``Concurrent`` method, this setting supports up to 2 GPUs. Using this mode will run +PDLP and barrier in parallel on different GPUs to avoid sharing single GPU resources. + +For LP problems solved with ``PDLP`` method, setting ``CUOPT_NUM_GPUS`` to ``-1`` or to a value greater than 1, +together with ``CUOPT_USE_DISTRIBUTED_PDLP`` set to true, distributes the PDLP solve across multiple GPUs (this +is cuOpt's multi-GPU PDLP). A value of ``-1`` uses all GPUs visible to the process, which may resolve +to a single GPU on a single-GPU host; multi-GPU sharding only happens when more than one GPU is actually +selected. + +Multi-GPU PDLP +^^^^^^^^^^^^^^ + +``CUOPT_USE_DISTRIBUTED_PDLP`` controls whether PDLP should be distributed across multiple GPUs. It requires +``CUOPT_METHOD`` to be ``PDLP`` and ``CUOPT_NUM_GPUS`` to be ``-1`` or greater than 1 (as above, ``-1`` may +resolve to a single visible GPU, in which case the solve still runs but is not actually sharded). + +``CUOPT_DISTRIBUTED_PDLP_PARTITIONER`` controls how the problem is partitioned across the GPUs used by multi-GPU PDLP: + +* ``0``: Auto (default) - picks ``RoundRobin`` on a single GPU and ``KaMinPar`` otherwise +* ``1``: KaMinPar - a multi-threaded graph partitioner that generally produces better balanced shards at the cost of + extra partitioning time +* ``2``: RoundRobin - assigns rows/columns across GPUs in round-robin fashion, without building a partitioning graph + +With a single GPU there is nothing to partition, so both strategies are equivalent no-ops; Auto picks +RoundRobin there because it skips KaMinPar's graph-partitioning work for no benefit. + +.. note:: The default value is ``false`` for ``CUOPT_USE_DISTRIBUTED_PDLP`` and ``0`` (Auto) for + ``CUOPT_DISTRIBUTED_PDLP_PARTITIONER``. C API users should use the constants defined in + :ref:`distributed-pdlp-partitioner-constants`. Python API users can also use the ``use_mpdlp`` and + ``mpdlp_partitioner`` parameter aliases. Infeasibility Detection diff --git a/docs/cuopt/source/cuopt-c/convex/convex-c-api.rst b/docs/cuopt/source/cuopt-c/convex/convex-c-api.rst index 0718361b81..068cdefe26 100644 --- a/docs/cuopt/source/cuopt-c/convex/convex-c-api.rst +++ b/docs/cuopt/source/cuopt-c/convex/convex-c-api.rst @@ -214,6 +214,8 @@ These constants are used as parameter names in the :c:func:`cuOptSetParameter`, .. doxygendefine:: CUOPT_SOLUTION_FILE .. doxygendefine:: CUOPT_NUM_CPU_THREADS .. doxygendefine:: CUOPT_NUM_GPUS +.. doxygendefine:: CUOPT_USE_DISTRIBUTED_PDLP +.. doxygendefine:: CUOPT_DISTRIBUTED_PDLP_PARTITIONER .. doxygendefine:: CUOPT_USER_PROBLEM_FILE .. doxygendefine:: CUOPT_PDLP_PRECISION @@ -230,6 +232,17 @@ These constants are used to configure `CUOPT_PDLP_SOLVER_MODE` via :c:func:`cuOp .. doxygendefine:: CUOPT_PDLP_SOLVER_MODE_METHODICAL1 .. doxygendefine:: CUOPT_PDLP_SOLVER_MODE_FAST1 +.. _distributed-pdlp-partitioner-constants: + +Distributed PDLP Partitioner Constants +--------------------------------------- + +These constants are used to configure `CUOPT_DISTRIBUTED_PDLP_PARTITIONER` via :c:func:`cuOptSetIntegerParameter`. + +.. doxygendefine:: CUOPT_DISTRIBUTED_PDLP_PARTITIONER_AUTO +.. doxygendefine:: CUOPT_DISTRIBUTED_PDLP_PARTITIONER_KAMINPAR +.. doxygendefine:: CUOPT_DISTRIBUTED_PDLP_PARTITIONER_ROUND_ROBIN + .. _pdlp-precision-constants: PDLP Precision Constants diff --git a/python/cuopt/cuopt/linear_programming/solver_settings/solver_settings.pyx b/python/cuopt/cuopt/linear_programming/solver_settings/solver_settings.pyx index a5dcc78d18..154d18ef5d 100644 --- a/python/cuopt/cuopt/linear_programming/solver_settings/solver_settings.pyx +++ b/python/cuopt/cuopt/linear_programming/solver_settings/solver_settings.pyx @@ -1,4 +1,4 @@ -# SPDX-FileCopyrightText: Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved. # noqa +# SPDX-FileCopyrightText: Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved. # SPDX-License-Identifier: Apache-2.0 # cython: profile=False @@ -52,6 +52,16 @@ cpdef get_solver_parameter_names(): solver_params = get_solver_parameter_names() for param in solver_params: globals()["CUOPT_"+param.upper()] = param +# Public spelling for the two distributed-PDLP parameters, mapped to the +# underlying C++ parameter names. Accepted as aliases by get_parameter and +# set_parameter; the CUOPT_* constants below mirror the auto-generated ones. +PARAMETER_ALIASES = { + "use_mpdlp": "use_distributed_pdlp", + "mpdlp_partitioner": "distributed_pdlp_partitioner", +} +for alias, canonical in PARAMETER_ALIASES.items(): + globals()["CUOPT_" + alias.upper()] = globals()["CUOPT_" + canonical.upper()] + class SolverMethod(IntEnum): """ @@ -163,6 +173,7 @@ cdef class SolverSettings: For a list of availabe parameters, their descriptions, default values, and acceptable ranges, see the cuOpt documentation `parameter.rst`. """ + name = PARAMETER_ALIASES.get(name, name) if name not in solver_params: raise ValueError("Invalid parameter. Please check documentation") if name in self.settings_dict: @@ -188,6 +199,7 @@ cdef class SolverSettings: For a list of availabe parameters, their descriptions, default values, and acceptable ranges, see the cuOpt documentation `parameter.rst`. """ + name = PARAMETER_ALIASES.get(name, name) if name not in solver_params: raise ValueError("Invalid parameter. Please check documentation") self.settings_dict[name] = value diff --git a/python/cuopt/cuopt/tests/linear_programming/test_lp_solver.py b/python/cuopt/cuopt/tests/linear_programming/test_lp_solver.py index ae58752ffb..c558120a25 100644 --- a/python/cuopt/cuopt/tests/linear_programming/test_lp_solver.py +++ b/python/cuopt/cuopt/tests/linear_programming/test_lp_solver.py @@ -19,11 +19,13 @@ CUOPT_ABSOLUTE_DUAL_TOLERANCE, CUOPT_ABSOLUTE_GAP_TOLERANCE, CUOPT_ABSOLUTE_PRIMAL_TOLERANCE, + CUOPT_DISTRIBUTED_PDLP_PARTITIONER, CUOPT_DUAL_INFEASIBLE_TOLERANCE, CUOPT_INFEASIBILITY_DETECTION, CUOPT_ITERATION_LIMIT, CUOPT_METHOD, CUOPT_MIP_HEURISTICS_ONLY, + CUOPT_NUM_GPUS, CUOPT_PDLP_SOLVER_MODE, CUOPT_PRIMAL_INFEASIBLE_TOLERANCE, CUOPT_RELATIVE_DUAL_TOLERANCE, @@ -31,6 +33,7 @@ CUOPT_RELATIVE_PRIMAL_TOLERANCE, CUOPT_SOLUTION_FILE, CUOPT_TIME_LIMIT, + CUOPT_USE_DISTRIBUTED_PDLP, CUOPT_USER_PROBLEM_FILE, CUOPT_PRESOLVE, ) @@ -374,6 +377,25 @@ def test_solver_settings_basic(): PDLPSolverMode.Methodical1 ) + # Multi-GPU PDLP settings, via the canonical (C++) parameter names + settings.set_parameter(CUOPT_NUM_GPUS, -1) + settings.set_parameter(CUOPT_USE_DISTRIBUTED_PDLP, True) + # 0=Auto, 1=KaMinPar, 2=RoundRobin + settings.set_parameter(CUOPT_DISTRIBUTED_PDLP_PARTITIONER, 2) + + assert settings.get_parameter(CUOPT_NUM_GPUS) == -1 + assert settings.get_parameter(CUOPT_USE_DISTRIBUTED_PDLP) is True + assert settings.get_parameter(CUOPT_DISTRIBUTED_PDLP_PARTITIONER) == 2 + + # Same settings, via the public mpdlp aliases + settings.set_parameter("use_mpdlp", True) + settings.set_parameter("mpdlp_partitioner", 1) + + assert settings.get_parameter("use_mpdlp") is True + assert settings.get_parameter("mpdlp_partitioner") == 1 + # Aliases resolve to the same underlying parameter as the canonical name + assert settings.get_parameter(CUOPT_DISTRIBUTED_PDLP_PARTITIONER) == 1 + def test_solver_settings(tmp_path): """Push every registered parameter to the C++ layer via set_c_solver_settings.""" diff --git a/python/cuopt_self_hosted/cuopt_sh_client/__init__.py b/python/cuopt_self_hosted/cuopt_sh_client/__init__.py index 5a5481b6a6..d6d3b4e0d0 100644 --- a/python/cuopt_self_hosted/cuopt_sh_client/__init__.py +++ b/python/cuopt_self_hosted/cuopt_sh_client/__init__.py @@ -1,4 +1,4 @@ -# SPDX-FileCopyrightText: Copyright (c) 2023-2025 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-FileCopyrightText: Copyright (c) 2023-2025, NVIDIA CORPORATION & AFFILIATES. All rights reserved. # SPDX-License-Identifier: Apache-2.0 from cuopt_sh_client._version import __git_commit__, __version__ diff --git a/python/cuopt_server/cuopt_server/tests/test_lp_conversion.py b/python/cuopt_server/cuopt_server/tests/test_lp_conversion.py index 667c16672c..bb6ab3c8e9 100644 --- a/python/cuopt_server/cuopt_server/tests/test_lp_conversion.py +++ b/python/cuopt_server/cuopt_server/tests/test_lp_conversion.py @@ -58,6 +58,25 @@ def test_create_solver_limits(): assert int(solver_settings.get_parameter("iteration_limit")) == 100 +def test_create_solver_mpdlp(): + data = get_lp_json() + data["solver_config"]["method"] = 1 # PDLP + data["solver_config"]["num_gpus"] = -1 + data["solver_config"]["use_mpdlp"] = True + data["solver_config"]["mpdlp_partitioner"] = 2 # RoundRobin + + warnings, solver_settings = conversion.create_solver( + LPData.parse_obj(data), None + ) + + assert warnings == [] + assert int(solver_settings.get_parameter("num_gpus")) == -1 + assert bool(solver_settings.get_parameter("use_distributed_pdlp")) is True + assert ( + int(solver_settings.get_parameter("distributed_pdlp_partitioner")) == 2 + ) + + def test_create_solver_limits_clamped_by_environment(monkeypatch): monkeypatch.setenv("CUOPT_LP_TIME_LIMIT_SEC", "2") monkeypatch.setenv("CUOPT_LP_ITERATION_LIMIT", "10") diff --git a/python/cuopt_server/cuopt_server/utils/linear_programming/conversion.py b/python/cuopt_server/cuopt_server/utils/linear_programming/conversion.py index fb130f1579..f4bcca148b 100644 --- a/python/cuopt_server/cuopt_server/utils/linear_programming/conversion.py +++ b/python/cuopt_server/cuopt_server/utils/linear_programming/conversion.py @@ -17,6 +17,13 @@ from cuopt_server.utils.linear_programming.data_definition import WarmStartData +# solver_params entries whose SolverConfig attribute uses a different +# (public-facing) name than the underlying C++ parameter. +SOLVER_CONFIG_FIELD_ALIASES = { + "use_distributed_pdlp": "use_mpdlp", + "distributed_pdlp_partitioner": "mpdlp_partitioner", +} + def ignored_warning(field): return f"solver config {field} ignored in the cuopt service" @@ -98,7 +105,8 @@ def create_solver(LP_data, warmstart_data): if param.endswith("tolerance"): param_value = getattr(solver_config.tolerances, param, None) else: - param_value = getattr(solver_config, param, None) + attr_name = SOLVER_CONFIG_FIELD_ALIASES.get(param, param) + param_value = getattr(solver_config, attr_name, None) if param_value is not None and param_value != "": solver_settings.set_parameter(param, param_value) diff --git a/python/cuopt_server/cuopt_server/utils/linear_programming/data_definition.py b/python/cuopt_server/cuopt_server/utils/linear_programming/data_definition.py index 7f25808372..b4ae9b66c4 100644 --- a/python/cuopt_server/cuopt_server/utils/linear_programming/data_definition.py +++ b/python/cuopt_server/cuopt_server/utils/linear_programming/data_definition.py @@ -474,7 +474,32 @@ class SolverConfig(BaseModel): ) num_gpus: Optional[int] = Field( default=None, - description="Set the number of GPUs to use for LP solve.", + description="Set the number of GPUs to use for LP solve. For " + "multi-GPU PDLP, use -1 to use all visible GPUs, or a value " + "greater than 1 to use that many GPUs; multi-GPU sharding only " + "happens when more than one GPU is actually selected.", + ) + use_mpdlp: Optional[bool] = Field( + default=None, + description="Set True to distribute the PDLP solve of an LP " + "problem across multiple GPUs. Requires method to be PDLP and " + "num_gpus to be -1 or greater than 1.", + ) + mpdlp_partitioner: Optional[int] = Field( + default=None, + description="Partitioner used to split the problem across GPUs " + "when use_mpdlp is set:" + "
" + "- Auto: 0, pick automatically (RoundRobin on 1 GPU, " + "KaMinPar otherwise)" + "
" + "- KaMinPar: 1, multi-threaded KaMinPar graph partitioner" + "
" + "- RoundRobin: 2, round-robin assignment, no graph" + "
" + "With 1 GPU there is nothing to partition, so both strategies " + "are equivalent no-ops; Auto picks RoundRobin there because it " + "skips KaMinPar's graph-partitioning work for no benefit.", ) augmented: Optional[int] = Field( default=-1,