Skip to content

Fix potential bvar deadlock by running describe()/dump() outside the global VarMap lock - #3470

Open
chenBright wants to merge 1 commit into
apache:masterfrom
chenBright:fix_bvar_deadlock
Open

Fix potential bvar deadlock by running describe()/dump() outside the global VarMap lock#3470
chenBright wants to merge 1 commit into
apache:masterfrom
chenBright:fix_bvar_deadlock

Conversation

@chenBright

Copy link
Copy Markdown
Contributor

What problem does this PR solve?

Issue Number: resolve #2888

Problem Summary:

bvar's global VarMap is guarded by a pthread mutex. Variable::describe_exposed()
(and describe_series_exposed(), dump_exposed(), plus the multi-dimension
MVariableBase counterparts) used to invoke var->describe() while holding that lock.
For PassiveStatus, describe() runs a user-provided callback; if the callback yields
the bthread (e.g. by acquiring a bthread::Mutex), the pthread mutex is never released
and the process deadlocks.

What is changed and the side effects?

Changed:

Run user callbacks OUTSIDE the global map lock via a small indirection handle:

  • New bvar/detail/exposed_ref.h: ExposedRef<T> (a reference-counted handle
    guarding an exposed object). It uses butil::Mutex + butil::ConditionVariable.
  • describe_exposed() / describe_series_exposed() / get_exposed(): under the
    map lock they now only seek + acquire() (ref-count +1, serialized with hide()'s
    erase); the lock is released, describe() is called outside the lock, then release().
  • hide() now also invalidates the handle and blocks (hide_and_wait()) until
    all in-flight readers finish, so a Variable cannot be destroyed while a concurrent
    describe() is still using it. Each expose() rebuilds a fresh handle (the old one
    is single-use once hidden).
  • MVariableBase::describe_exposed() / dump_exposed() get the same treatment;
    dump() in particular is moved outside the lock because Dumper is a
    user-overridable interface that may yield.
  • Since callbacks no longer run under the lock, the recursive VarMap mutex is no
    longer needed and is reverted to a plain mutex (now consistent with MVarMap).

Side effects:

  • Performance effects:

  • Breaking backward compatibility:


Check List:

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR addresses a potential deadlock in bvar caused by invoking user-defined describe() / dump() callbacks while holding the global VarMap (pthread) mutex. It introduces an indirection handle so readers can safely call user callbacks outside the map lock, while hide() waits for in-flight readers to finish to prevent use-after-free.

Changes:

  • Add bvar/detail/exposed_ref.h implementing detail::ExposedRef<T> (ref-count + hide-and-wait) and wire it into both Variable and MVariableBase exposure paths.
  • Update Variable::{describe_exposed, describe_series_exposed, get_exposed} and MVariableBase::{describe_exposed, dump_exposed} to acquire a safe handle under the map lock, then call user code after releasing the lock.
  • Add regression/unit tests covering (1) bthread-yield deadlock reproduction and (2) destructor waiting for in-flight describe_exposed() to complete.

Reviewed changes

Copilot reviewed 7 out of 7 changed files in this pull request and generated no comments.

Show a summary per file
File Description
test/bvar_variable_unittest.cpp Adds a test ensuring hide()/destruction waits for an in-flight describe_exposed() callback to finish.
test/bthread_unittest.cpp Adds a regression test reproducing issue #2888 (many bthreads + yielding callback) and ensuring no deadlock.
src/bvar/variable.h Adds an exposed-handle member (SharedExposedRef) to Variable for safe describe/get outside the global map lock.
src/bvar/variable.cpp Reworks VarMap entries to store the handle, moves describe()/get_value()/describe_series() out of the map lock, and makes hide() wait for in-flight readers.
src/bvar/mvariable.h Adds the same exposed-handle mechanism to MVariableBase.
src/bvar/mvariable.cpp Updates describe_exposed()/dump_exposed() to acquire under lock and invoke user callbacks outside the lock; hide() waits for readers.
src/bvar/detail/exposed_ref.h New shared indirection/ref-count + hide-and-wait primitive used to prevent deadlocks and UAF.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

bvar 的全局 VarMap 锁为 pthread mutex, 可能引发死锁

2 participants