Vortex runtime stalls forever when driven from inside `tokio::Runtime::block_on`

[joseph-isaacs]

Summary

A Vortex CurrentThreadRuntime driven from a thread that sits inside tokio::runtime::Runtime::block_on permanently loses wakeups delivered by Tokio's blocking pool. After a few hundred such wakeups the driving thread parks in epoll_wait and never resumes.

This is not specific to local file reads, and it is not a thread-count problem — a multi-threaded Tokio runtime stalls identically.

Minimal reproduction

No Vortex I/O involved:

let rt = tokio::runtime::Builder::new_current_thread().enable_all().build()?;
rt.block_on(async {
    CurrentThreadRuntime::new().block_on(async {
        for i in 0..1000 {
            assert_eq!(tokio::task::spawn_blocking(move || i).await.unwrap(), i);
        }
    });
});

This never completes. It stalls somewhere between iteration ~130 and ~200; the exact point varies, which is consistent with a lost-wakeup race rather than a capacity limit.

What does and does not trigger it

Scenario	Result
spawn_blocking awaited inside rt.block_on, current-thread runtime	stalls
spawn_blocking awaited inside rt.block_on, multi-thread runtime	stalls
Same runtime and tasks, Handle::spawn_blocking from a thread outside block_on	passes
Foreign std::thread + futures::channel::oneshot wakeups, 1000 iterations	passes
No Tokio runtime present at all	passes

The executor doing the blocking does not matter: smol::block_on, futures::executor::block_on and CurrentThreadRuntime::block_on all stall. The single discriminating variable is whether the thread is inside Runtime::block_on.

Thread dump at the stall

Thread 4 "seq_256"   vortex_io::runtime::current::block_on
                     -> async_io::driver::block_on
                     -> async_io::reactor::ReactorLock::react
                     -> polling::Poller::wait  (epoll_wait, no timeout)

Thread 2 "async-io"  async_io::driver::main_loop
                     -> async_io::reactor::Reactor::lock
                     -> Mutex::lock_contended   (blocked on the lock held by Thread 4)

Thread 3 "tokio-rt-worker"
                     tokio::runtime::blocking::pool::Inner::run
                     -> Condvar::wait_timeout   (idle: its work already finished)

The blocking task has completed, but its wakeup never reaches the driver. The driver has taken the async-io reactor lock and is asleep in epoll_wait with no timeout; the dedicated async-io thread that would otherwise notice is blocked on that same lock. Because futures::executor::block_on — which has no reactor at all — stalls the same way, the reactor is where it comes to rest, not the cause.

How this reaches Vortex

object_store's LocalFileSystem routes reads through tokio::task::spawn_blocking whenever tokio::runtime::Handle::try_current() succeeds on the calling thread. So any local read through ObjectStoreFileSystem (vortex-io/src/object_store/filesystem.rs) inherits the stall whenever the Vortex runtime is being driven from inside Runtime::block_on:

current_thread_runtime().block_on(async move {
    let vortex_rt = CurrentThreadRuntime::new();
    let fs = ObjectStoreFileSystem::local(vortex_rt.handle());
    vortex_rt.block_on(async move {
        let source = fs.open_read(&path).await?;
        // ~256 sequential 4 KiB reads: stalls
    })
});

Without an ambient Tokio runtime, LocalFileSystem reads inline, there is no cross-runtime wakeup, and the identical workload completes — which is why this has not shown up under the DuckDB CLI. It does affect any Rust embedder that calls vortex-duckdb from async code, and it is what wedges the DuckDB SLT suite, whose runner drives each file inside rt.block_on (vortex-sqllogictest/bin/sqllogictests-runner.rs:56).

Repro tests

vortex-io/tests/tokio_context_blocking_stall.rs carries the cases above. The three failing ones are #[ignore]d so CI stays green; they fail on a timeout rather than hanging:

cargo test -p vortex-io --features object_store,tokio \
    --test tokio_context_blocking_stall -- --ignored --test-threads=1

test local_object_store_reads_inside_block_on ... FAILED
test tokio_blocking_tasks_under_current_thread_runtime ... FAILED
test tokio_blocking_tasks_under_multi_thread_runtime ... FAILED

Versions

tokio 1.53.1, smol 2.0.2, object_store 0.13.2.

Notes on fixes

Two things that do not fix it, both worth recording because they look like they should:

• Giving the SLT runner a multi-threaded Tokio runtime. Proven above to stall identically.
• Special-casing local reads onto FileReadAt inside ObjectStoreFileSystem. That removes one route into spawn_blocking, so it hides this instance, but the underlying runtime interaction is untouched and any other Tokio-backed wakeup reaching a Vortex block_on will stall the same way.

What does work in every test above is keeping the thread that drives a Vortex runtime out of Runtime::block_on entirely.