Skip to content

Use a native __getattr__ hook for AttributeError suggestions (GIL-safe re-land of the miss-only hook)#129

Merged
jhonabreul merged 2 commits into
QuantConnect:masterfrom
jhonabreul:bug-attributeerror-hint-gil
Jul 6, 2026
Merged

Use a native __getattr__ hook for AttributeError suggestions (GIL-safe re-land of the miss-only hook)#129
jhonabreul merged 2 commits into
QuantConnect:masterfrom
jhonabreul:bug-attributeerror-hint-gil

Conversation

@jhonabreul

@jhonabreul jhonabreul commented Jul 6, 2026

Copy link
Copy Markdown
Collaborator

What does this implement/fix?

Re-lands the miss-only __getattr__ hook for AttributeError member-name suggestions (the #124 approach, which #126 had replaced with a tp_getattro override), and fixes the off-GIL defect that crashed Lean's CI on 2.0.55.

Why re-land the hook: with the hook, successful attribute accesses go straight through CPython's native generic getattr with no managed transition — only an actual miss enters managed code. The #126 tp_getattro override put a managed round-trip on every attribute access, hit or miss.

The 2.0.55 crash (see first commit): the original hook exposed the hint builder to Python as a .NET delegate (Func<PyObject, string, string>). Python invoked it through DelegateObject.tp_callMethodBinder.Invoke, and MethodBinder releases the GIL around every reflected invocation (allow_threads defaults to true). The callback therefore ran CPython C-API calls without holding the GIL on every attribute miss (hasattr, getattr with default, typos). It usually survived by luck, but segfaulted whenever the pythonnet Finalizer fired mid-callback: Finalizer.DisposeAll() starts with PyErr_Fetch, which dereferences the current thread state — NULL when the GIL is not held. This is what crashed Lean's CI test host after all 35k tests passed.

The fix (second commit): the __getattr__ installed on reflected types is now a native method descriptor — a process-lifetime PyMethodDef (METH_VARARGS) around a managed thunk, created with PyDescr_NewMethod (newly bound in Runtime.Delegates). CPython's slot_tp_getattr_hook calls the managed hook directly as a native method call with the GIL held; MethodBinder is never involved.

User-facing messages are unchanged from 2.0.55/2.0.56:

AttributeError: 'Version' object has no attribute 'majr' Did you mean: 'major'?

Does this close any currently open issues?

No open issue; this addresses the Lean CI test-host crash observed with 2.0.55 (PyErr_Fetch(tstate=NULL) under Finalizer.DisposeAll under BuildMissingAttributeMessage).

Verification performed

  • Instrumented build with PyGILState_Check() inside BuildMissingAttributeMessage: the delegate-based hook (first commit alone) reports GIL held: False on every miss; with the fix it reports GIL held: True.
  • 20s stress run: 6 threads doing concurrent attribute misses + CLR object churn (to keep the Finalizer queue busy) — no crash.
  • Embedding tests: 909 passed. Python tests: 443 passed (one pre-existing, environment-only failure from quantconnect-stubs shadowing the Microsoft namespace, identical on master).
  • Lean's Python/Pandas unit tests run against this build with the test host surviving (the 2.0.55 failure mode was a host crash after tests passed).
  • Behavioral checks: hasattr/getattr-with-default, dunder probes (pickle/copy machinery), Python subclasses defining their own __getattr__, and suggestions inherited by Python subclasses of .NET types all behave as before.

A new regression test (test_missing_attribute_hook_is_native) asserts the installed __getattr__ is a native method_descriptor, the property that keeps the callback out of MethodBinder's GIL-releasing path.

jhonabreul and others added 2 commits July 6, 2026 11:06
Revert the code changes of QuantConnect#126, restoring the AttributeErrorHint
miss-only __getattr__ hook approach from QuantConnect#124: successful attribute
accesses go straight through CPython's native generic getattr with no
managed transition; only a miss enters managed code to build the
"Did you mean ...?" suggestions.

The package <Version> is left at 2.0.56 (not reverted).

Note: as restored here, the hook still has the off-GIL callback defect
that crashed Lean's CI on 2.0.55 (the __clr_attr_msg__ delegate is
invoked through MethodBinder, which releases the GIL); it is fixed in
the follow-up commit.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The __getattr__ hook restored from QuantConnect#124 exposed the hint builder to
Python as a .NET delegate (Func<PyObject, string, string>). Python
invoked it through DelegateObject.tp_call -> MethodBinder.Invoke, and
MethodBinder releases the GIL around every reflected invocation
(allow_threads defaults to true). The callback therefore ran CPython
C-API calls (GetPythonType, GetManagedString, ...) without holding the
GIL on every attribute miss (hasattr, getattr with default, typos). It
usually survived by luck, but segfaulted whenever the pythonnet
Finalizer fired mid-callback: Finalizer.DisposeAll() starts with
PyErr_Fetch, which dereferences the current thread state - NULL when
the GIL is not held. This is what crashed Lean's CI test host on
2.0.55 after all 35k tests passed.

Replace the Python-function-plus-delegate pair with a native method
descriptor: a PyMethodDef (METH_VARARGS) around a managed thunk,
turned into __getattr__ via PyDescr_NewMethod (newly bound). CPython's
slot_tp_getattr_hook now calls the managed hook directly as a native
method call with the GIL held - MethodBinder is never involved. The
PyMethodDef and thunk are allocated once and kept for the process
lifetime, since descriptors reference them and can outlive engine
shutdown bookkeeping.

Verified with an instrumented build (PyGILState_Check inside
BuildMissingAttributeMessage): the delegate-based hook reports
"GIL held: False" on every miss; this version reports "GIL held: True".
Also survives a 20s stress run of concurrent attribute misses plus
finalizer churn across 6 threads, and behaves identically for hasattr/
getattr-with-default, dunder probes, Python subclasses with their own
__getattr__, and suggestion messages.

Add a regression test asserting the installed __getattr__ is a native
method_descriptor, which is the property that keeps the callback out of
MethodBinder's allow-threads path.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@jhonabreul jhonabreul merged commit 93ee21a into QuantConnect:master Jul 6, 2026
4 checks passed
@jhonabreul jhonabreul deleted the bug-attributeerror-hint-gil branch July 6, 2026 17:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants