Await Bokeh dashboard shutdown

[jacobtomlinson-bot]

Closes #9350.

Bokeh 3.10 added an asynchronous dashboard shutdown path. Scheduler.close
and Worker.close run on the dashboard's event loop, so they must await it.
Keep the public synchronous ServerNode.stop_services() API by bridging it to
the private async helper; use Bokeh's legacy stop() when stop_async() is
not available.

Reproduction

The following was run with CPython 3.12.3, pytest 9.1.1, and Bokeh 3.10.0.
The selected tests already cover the synchronous in-process close and the two
async dashboard-close paths. The fourth selected test is marked slow and is
therefore skipped without --runslow.

On upstream main at b6d33471 (before this PR), from the repository root:

$ python -m pytest -q --tb=short -k 'test_quiet_client_close or test_dashboard_link_inproc or test_async_with' distributed/tests/test_client.py
============================= test session starts ==============================
platform linux -- Python 3.12.3, pytest-9.1.1, pluggy-1.6.0
rootdir: /tmp/distributed-issue9350-before
configfile: pyproject.toml
plugins: xdist-3.8.0, timeout-2.4.0, rerunfailures-16.6
collected 509 items / 505 deselected / 4 selected

distributed/tests/test_client.py FEsFF                                   [100%]

.../distributed/scheduler.py:4513: in close
    self.stop_services()
.../distributed/node.py:79: in stop_services
    application.stop()
.../site-packages/bokeh/server/tornado.py:761: in stop
    raise RuntimeError("Cannot synchronously wait on a running event loop; use 'await stop_async()'")
E   RuntimeError: Cannot synchronously wait on a running event loop; use 'await stop_async()'

=========================== short test summary info ============================
SKIPPED [1] distributed/tests/test_client.py:5074: need --runslow option to run
FAILED distributed/tests/test_client.py::test_quiet_client_close - RuntimeError: Cannot synchronously wait on a running event loop; use 'await...'
FAILED distributed/tests/test_client.py::test_dashboard_link_inproc - RuntimeError: Cannot synchronously wait on a running event loop; use 'await...'
FAILED distributed/tests/test_client.py::test_async_with - RuntimeError: Cannot synchronously wait on a running event loop; use 'await...'
ERROR distributed/tests/test_client.py::test_quiet_client_close - ValueError: ('Unclosed Comms', ...)
============ 3 failed, 1 skipped, 505 deselected, 1 error in 1.67s =============
$ echo $?
1

On this PR branch at 6767da58, with the same environment and command:

$ python -m pytest -q --tb=short -k 'test_quiet_client_close or test_dashboard_link_inproc or test_async_with' distributed/tests/test_client.py
============================= test session starts ==============================
platform linux -- Python 3.12.3, pytest-9.1.1, pluggy-1.6.0
rootdir: /tmp/distributed-issue9350-rebase
configfile: pyproject.toml
plugins: xdist-3.8.0, timeout-2.4.0, rerunfailures-16.6
collected 509 items / 505 deselected / 4 selected

distributed/tests/test_client.py .s..                                    [100%]

=========================== short test summary info ============================
SKIPPED [1] distributed/tests/test_client.py:5074: need --runslow option to run
================= 3 passed, 1 skipped, 505 deselected in 1.07s =================
$ echo $?
0

pre-commit run --all-files also passes on the PR branch.

[github-actions]

Unit Test Results

See test report for an extended history of previous test failures. This is useful for diagnosing flaky tests.

    40 files  ± 0      40 suites  ±0   14h 19m 38s ⏱️ + 8m 55s
 4 160 tests + 1   3 981 ✅ + 8    178 💤 ±0  1 ❌  - 5 
80 959 runs  +19  76 720 ✅ +25  4 238 💤 +1  1 ❌  - 5 

For more details on these failures, see this check.

Results for commit 114e15f. ± Comparison against base commit b6d3347.

♻️ This comment has been updated with latest results.

[bdice]

Thanks! This is roughly what I expected a fix would look like. I don’t use async very frequently but I think this is likely a proper fix. I see the fix was tested with Bokeh 3.10. I would want to verify that things pass with an earlier version as well.

[jacobtomlinson]

I tested an older bokeh version with this fix locally and things work as expected. The mindeps CI should also verify this.