Natvis: implement view() specifier, IncludeView/ExcludeView filtering, and CustomListItems loop engine by lugerard · Pull Request #1580 · microsoft/MIEngine

lugerard · 2026-06-10T13:04:56Z

This PR enables MIEngine to evaluate two natvis features:
view-based filtering (IncludeView/ExcludeView on DisplayString
and Expand elements, activated via the view() format specifier) and
<CustomListItems> loop bodies (<Loop>, <Item>, <Exec>,
<If>/<ElseIf>/<Else>, <Break>), which are the standard way to walk
pointer-based data structures such as linked lists and trees.

The two commits are bundled together because <CustomListItems> loop bodies
can contain IncludeView/ExcludeView guards on their <Item> elements,
so commit 1 is a functional prerequisite for commit 2.

Commit 1 adds view() specifier parsing and IncludeView/ExcludeView
filtering, plus fixes to EvalCondition error handling and expression
pre-processing that are prerequisites for correct evaluation on GDB/LLDB.

Commit 2 implements the <CustomListItems> loop engine, including
expression-growth prevention after <Exec> assignments and a $i
word-boundary fix.

All new code is covered by unit tests.

Add view support to the MIEngine natvis evaluator (steps 1, 2, and 3): 1. DisplayString IncludeView/ExcludeView filtering - Add IncludeView and ExcludeView properties to DisplayStringType (NatvisXsdTypes.cs) so the XML attributes are deserialized. - Filter DisplayString entries in FormatDisplayString() using the new IsIncludeViewMatch() and IsExcludeViewMatch() helpers before evaluating the Condition. 2. {expr,view(name)} inline specifier in DisplayString text - ExtractViewName() detects a view() format specifier in an inline expression block such as {this,view(RecZone)na}. - When detected, FormatValue() calls GetExpressionValue() with the view name, which re-enters FormatDisplayString() selecting only DisplayString entries whose IncludeView matches. - ExtractViewName() returns null for view() with an empty name (not a valid specifier). 3. View support on Expand elements - Add IncludeView and ExcludeView properties to ExpandedItemType, ItemType, and CustomListItemsType in NatvisXsdTypes.cs. - Thread currentView through Expand() and ExpandVisualized() so that view context propagates into recursive expand calls. - Filter Item and ExpandedItem elements by IncludeView/ExcludeView before evaluating their Condition. - Strip a view() specifier from an ExpandedItem expression before evaluating it, and pass the extracted view name into the recursive Expand() call so that IncludeView guards on the target type's Expand elements match correctly. - Add a CustomListItemsType stub case that applies IncludeView/ ExcludeView filtering; loop body execution to follow. 4. EvalCondition fallback on failure - Wrap condition evaluation in try/catch: MIException (debugger rejected the expression, e.g. too long) is caught silently and returns false; other exceptions are caught and logged at Warning before also returning false, so natvis never surfaces errors as debug-session failures. - Prerequisite: natvis files may use a lightweight condition as a platform probe (always true for valid data, but too long to expand on GDB/LLDB); without this fallback the condition would surface as an error rather than falling through to the next DisplayString. 5. Strip format specifier before name substitution in GetExpressionValue() - A specifier such as ",d" was being matched by ProcessNamesInString as a child-variable name, corrupting the expression. Strip it first, then re-attach after substitution. - Prerequisite: without this fix, an expression such as {call(),d} would have its specifier matched as a variable name, corrupting the expression before evaluation. 6. Intrinsic expansion before dll! stripping in ReplaceNamesInExpression() - Intrinsic bodies can contain dll!-qualified type casts. Moving expansion before the dll!-strip regex ensures those references are also cleaned up. - Prerequisite: without this fix, a dll!-qualified cast inside an intrinsic body would survive into the expression sent to GDB/LLDB. Note: items 4, 5, and 6 are bugfixes that are prerequisites for the view() feature to work correctly on GDB/LLDB. They are bundled in this commit because they share the same test infrastructure and were discovered during view() development. Unit tests: Add 18 tests in NatvisFormatSpecifierTest covering the three new static helpers (IsIncludeViewMatch, IsExcludeViewMatch, ExtractViewName) and the view() specifier parsing patterns used by Expand elements, including the empty-name edge case for ExtractViewName.

Add full execution support for the <CustomListItems> expand element, which was previously deserialized but produced no children. NatvisXsdTypes.cs: - Add seven new types covering the loop body: CustomListLoopType (<Loop>), CustomListLoopItemType (<Item>), CustomListBreakType (<Break>), CustomListExecType (<Exec>), CustomListIfType (<If>), CustomListElseIfType (<ElseIf>), CustomListElseType (<Else>). If/ElseIf/Else bodies and Loop bodies share the same element set, so nested loops and conditional branches are fully supported. - Extend CustomListItemsType with a Loop[] field. - Add Condition attribute to CustomListLoopType: acts as a while-guard, stopping the loop as soon as the condition evaluates to false. - Add missing [GeneratedCode], [Serializable] and [DebuggerStepThrough] attributes to the seven new types for consistency with the rest of the file. Natvis.cs: - Replace the view-filter stub with a complete execution engine: * <Variable> declarations initialise a local-variable table (name -> current expression string) via ReplaceNamesInExpression. * Optional <Size> sets the totalSize upper bound for pagination. * The loop driver evaluates an optional Loop Condition before each iteration (while-guard), then calls ExecuteCustomListBody() once per iteration. * <Break>: stops the loop when the condition holds. * <Item>: substitutes local vars, resolves field names and intrinsics, emits the child; increments the $i counter (GlobalIndex). * <Exec>: supports "varName = rhs" assignment and ++/-- shorthand (prefix and postfix) to update the local-variable table. After each update, the new expression is evaluated and normalised to a scalar literal to prevent unbounded expression growth across iterations (e.g. i++ stays "1", "2", "3" rather than accumulating nested parentheses). * <If>/<ElseIf>/<Else>: evaluates conditions in order and executes the first matching branch; all siblings are consumed in one pass so the idx cursor never re-processes them. * Pagination: fast-forwards through startIndex items, then collects up to MAX_EXPAND children; adds a [More...] node when the page is full and more items remain. * Infinite-loop guard: caps iterations at min(startIndex+51, 10000). - Add helpers (internal static for testability): ExecuteCustomListBody, SubstituteLocalVars, ApplyExecToLocalVars, FormatCustomListItemName ($i / {$i} in names with word-boundary guard to avoid corrupting tokens like $item), CustomListLoopContext (shared mutable loop state), s_execAssignment static Regex (with =(?!=) to avoid matching == operators), s_execIncrDecr static Regex for ++/-- shorthand, s_dollarI static Regex (\$i\b) for safe bare-$i replacement. NatvisFormatSpecifierTest.cs: - 25 new unit tests covering SubstituteLocalVars (empty, no-vars, single substitution, word-boundary, multi-var, parens), ApplyExecToLocalVars (assignment, unknown LHS, empty, counter increment, == not matched, prefix/postfix ++/--, unknown var with ++, whitespace tolerance), and FormatCustomListItemName (null template, {$i}, bare $i, no special tokens, local-var substitution).

Copilot

Pull request overview

This PR extends MIEngine’s Natvis evaluator to support view-based formatting/expansion (view() specifier plus IncludeView/ExcludeView filtering) and adds a CustomListItems loop execution engine to enable pointer-based collection traversal during expansion.

Changes:

Adds parsing/propagation of view(name) and applies IncludeView/ExcludeView filtering for DisplayString, Item, ExpandedItem, and CustomListItems.
Improves Natvis expression handling (specifier stripping before name substitution, intrinsic/module-prefix preprocessing ordering, and safer EvalCondition failure handling).
Implements CustomListItems loop-body execution (Loop/Item/Exec/If/ElseIf/Else/Break) with helper utilities and unit tests.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 4 comments.

File	Description
src/MIDebugEngineUnitTests/NatvisFormatSpecifierTest.cs	Adds unit tests for `view()` extraction, view matching helpers, and CustomListItems helper utilities.
src/MIDebugEngine/Natvis.Impl/NatvisXsdTypes.cs	Extends Natvis XSD-derived types with view attributes and new CustomListItems loop-body element models.
src/MIDebugEngine/Natvis.Impl/Natvis.cs	Implements view filtering, `view()` propagation through expansion/formatting, expression preprocessing fixes, and the CustomListItems loop engine.

lugerard · 2026-07-03T14:58:51Z

+                        {
+                            if (string.IsNullOrEmpty(v.Name)) continue;
+                            string initVal = v.InitialValue ?? "0";
+                            // Resolve field names, template parameters and intrinsics in the initial value.
+                            localVars[v.Name] = ReplaceNamesInExpression(initVal, variable, visualizer.ScopedNames, visualizer.Intrinsics);
+                        }


lugerard · 2026-07-03T14:58:54Z

+                            string normalized = GetExpressionValue(localVars[updatedVar], variable, visualizer.ScopedNames, visualizer.Intrinsics);
+                            if (!string.IsNullOrEmpty(normalized) &&
+                                !normalized.TrimStart().StartsWith("0x", StringComparison.OrdinalIgnoreCase))
+                            {
+                                localVars[updatedVar] = normalized;
+                            }


gregg-miskelly · 2026-06-21T05:16:11Z

+        }
+
+        /// <remarks/>
+        [System.Xml.Serialization.XmlElementAttribute("Item",   typeof(CustomListLoopItemType))]


CustomListLoopItemType

Can you rename this to CustomListItemType to match the main natvis.xsd

gregg-miskelly · 2026-06-21T05:16:41Z

+
+        /// <remarks/>
+        [System.Xml.Serialization.XmlElementAttribute("Item",   typeof(CustomListLoopItemType))]
+        [System.Xml.Serialization.XmlElementAttribute("Break",  typeof(CustomListBreakType))]


CustomListBreakType

Similar request for the rest. Here is how they are declared in the .xsd:

<xs:element name="Loop" minOccurs="0" maxOccurs="unbounded" type="LoopType"></xs:element> <xs:sequence minOccurs="0" maxOccurs="unbounded"> <xs:element name="If" minOccurs="1" maxOccurs="1" type="IfType"></xs:element> <xs:element name="Elseif" minOccurs="0" maxOccurs="unbounded" type="IfType"></xs:element> <xs:element name="Else" minOccurs="0" maxOccurs="1" type="ElseType"></xs:element> </xs:sequence> <xs:element name="Exec" minOccurs="0" maxOccurs="unbounded" type="ExecType"></xs:element> <xs:element name="Break" minOccurs="0" maxOccurs="unbounded" type="BreakType"></xs:element>

gregg-miskelly · 2026-06-21T05:22:03Z

+        [System.Xml.Serialization.XmlElementAttribute("Break",  typeof(CustomListBreakType))]
+        [System.Xml.Serialization.XmlElementAttribute("Exec",   typeof(CustomListExecType))]
+        [System.Xml.Serialization.XmlElementAttribute("If",     typeof(CustomListIfType))]
+        [System.Xml.Serialization.XmlElementAttribute("ElseIf", typeof(CustomListElseIfType))]


ElseIf

The casing of this is incorrect. It should be Elseif

gregg-miskelly · 2026-06-21T05:48:08Z

Can you add an end-to-end test that uses this? Example: #1559

gregg-miskelly · 2026-06-21T05:53:35Z

+                            progress = true;
+                        }
+                    }
+                }


}

else -> write a warning for unexpected element?

gregg-miskelly · 2026-06-21T05:54:16Z

+                            bool innerProgress = ExecuteCustomListBody(nestedLoop.Items, ctx, variable, visualizer, localVars, children);
+                            if (!innerProgress && !ctx.Done) break;
+                            progress = true;
+                        }


Can we share more code with the top level loop?

gregg-miskelly · 2026-06-21T06:01:06Z

+                            continue;
+                    }
+                    string execExpr = SubstituteLocalVars(exec.Value?.Trim() ?? "", localVars);
+                    string updatedVar = ApplyExecToLocalVars(execExpr, localVars);


ApplyExecToLocalVars

Not sure if you want to handle this, but the .natvis files that ship with VS will often assign to multiple variables in an <Exec> block. Example: <Exec>++idx, ++statptr</Exec>. Feels like we need to at least log a warning, but it probably isn't too hard to support.

gregg-miskelly · 2026-06-21T06:03:12Z

+                            }
+                        }
+                        catch (Exception) { /* keep the expression as-is if evaluation fails */ }
+                    }


}

Should we be logging a warning here and breaking?

gregg-miskelly

Thanks for doing this. Aside from that, this seems about right to me.

Rename the <CustomListItems> loop-body types to the names used in the main natvis.xsd, and fix the <Elseif> element casing: CustomListLoopItemType -> CustomListItemType CustomListBreakType -> BreakType CustomListExecType -> ExecType CustomListIfType -> IfType CustomListElseType -> ElseType CustomListLoopType -> LoopType element name "ElseIf" -> "Elseif" IfType change: the xsd declares <Elseif type="IfType">, so it shares IfType with <If> instead of having its own type. A single object[] member cannot map two element names ("If" and "Elseif") to the same type; XmlSerializer throws at construction. The generated form of the schema solves this with an XmlChoiceIdentifier, so this commit does the same: - <Elseif> reuses IfType (the separate CustomListElseIfType is removed). - IfType/ElseType/LoopType gain a parallel ItemsChoiceType[] ItemsElementName array, declared via [XmlChoiceIdentifier], that records which element name produced each Items entry. - ExecuteCustomListBody now receives that choice array and uses it to tell an <If> from an <Elseif>, since both deserialize to IfType. This is what regenerating NatvisXsdTypes.cs from natvis.xsd would produce, keeping the hand-maintained file consistent with the schema.

…d <Exec> normalization Two robustness fixes in the CustomListItems loop engine: - ExecuteCustomListBody now ends with an else branch that logs a warning naming any unrecognised loop-body element, instead of silently skipping it. - The <Exec> normalization catch now logs a warning and stops the loop. We log and break in the second case because the normalization evaluates the loop variable we just updated; the loop body and its conditions use that same variable, so if it can't be evaluated the iteration can't continue meaningfully. Rather than press on with a variable we know is broken (which would emit error items or run to the iteration cap) we log the failure and stop the loop.

.natvis files shipped with VS often assign to several variables in one <Exec> block (e.g. "<Exec>++idx, ++statptr</Exec>"), which we previously did not handle. - Multi-variable support: ApplyExecToLocalVars splits the <Exec> text on top-level commas (commas inside parentheses/brackets are left intact, so "f(a, b)" is not split) and applies each assignment in turn, returning all updated variable names so each is normalised. - Warning: a non-empty segment we cannot apply (an unsupported expression such as "idx += 2" or a call, or an undeclared left-hand side) is now logged at Warning instead of being silently dropped. - Fix: the <Exec> value was being substituted by SubstituteLocalVars *before* ApplyExecToLocalVars parsed it, turning "i = i + 1" into "(0) = (0) + 1" so the "(\w+)" left-hand-side match failed and the assignment was silently dropped; no <Exec> assignment took effect at runtime (the unit tests passed only because they call the helper with raw text). The helper is now given the raw <Exec> text and does the right-hand-side substitution itself. Tests cover multiple increments, multiple assignments in order, the returned name list, a comma inside parentheses, a trailing empty segment, unhandled segments reported for an unsupported form and an undeclared left-hand side, and SplitTopLevelCommas.

The top-level loop driver (in ExpandVisualized) and the nested <Loop> branch in ExecuteCustomListBody were near-identical copies of the same iteration logic: the iteration cap, the optional while-Condition guard, the per-pass ExecuteCustomListBody call, and the no-progress break. Extract them into a single DriveLoop helper used by both. No behaviour change.

- Sequential <Variable> init - Pointer-safe <Exec> normalization Adds unit tests for IsScalarLiteral.

CppTests integration test exercising the <CustomListItems> loop engine - CustomListContainer.h: a NULL-terminated singly-linked list - main.cpp: a customList(10, 20, 30) before the return breakpoint - Simple.natvis: a <CustomListItems> visualizer (Variable/Loop/Break/Item/Exec) - NatvisTests.cs: TestCustomListItems asserts Count=3 and [0..2]=10/20/30, and bumps the breakpoint line constants for the grown main.cpp

CppTests integration test for IncludeView/ExcludeView and a ",view(name)" specifier invoked from inside the natvis: - main.cpp: a ViewObject{x, y, z} and a ViewHolder wrapping one, before the return breakpoint - Simple.natvis: ViewObject tags a DisplayString and Expand items with IncludeView/ExcludeView; ViewHolder invokes the view via {inner,view(simple)} and <ExpandedItem>inner,view(simple)</ExpandedItem> - NatvisTests.cs: TestViewFiltering asserts the default view (tagged DisplayString and Item skipped) and the simple view through ViewHolder (IncludeView DisplayString selected, view propagated into the expansion, ExcludeView item filtered out)

A watch expression like "obj,view(simple)" previously reached the debugger unchanged and failed to evaluate (lldb: "use of undeclared identifier"). The view() specifier is a natvis concept, so the debugger should never see it: - VariableInformation.ProcessFormatSpecifiers now recognizes a view() specifier, strips it from the evaluated expression, and records the view name. It is handled before the modifier stripping, which could otherwise corrupt a view name containing a modifier letter pair (e.g. "view(second)"). - IVariableInformation exposes the recorded view, and AD7Property passes it to FormatDisplayString and Expand, so the named view selects the DisplayString and filters IncludeView/ExcludeView items in the expansion. The test harness could so far only read the one-line result of a watch expression, not what it expands to: the debugger's reply includes a handle for expanding the result (variablesReference), but the harness discarded it. It now keeps the handle, and the new FrameInspector.EvaluateChildren uses it to return the child rows of an evaluated expression. TestViewFiltering uses both: "vo,view(simple)" must summarize as the IncludeView DisplayString and expand to the view-filtered items.

…natvis-view-customlistitems

lugerard · 2026-07-03T14:59:28Z

Can you add an end-to-end test that uses this? Example: #1559

done

lugerard and others added 3 commits June 10, 2026 14:38

Merge branch 'main' into natvis-view-customlistitems

8337b7f

gregg-miskelly requested a review from Copilot June 21, 2026 00:49

Copilot started reviewing on behalf of gregg-miskelly June 21, 2026 00:50 View session

Copilot AI reviewed Jun 21, 2026

View reviewed changes

gregg-miskelly reviewed Jun 21, 2026

View reviewed changes

gregg-miskelly requested changes Jun 21, 2026

View reviewed changes

lugerard and others added 11 commits July 2, 2026 15:08

Natvis: fix CustomListItems local variable handling

5de0de3

- Sequential <Variable> init - Pointer-safe <Exec> normalization Adds unit tests for IsScalarLiteral.

Merge remote-tracking branch 'fork/natvis-view-customlistitems' into …

da46726

…natvis-view-customlistitems

Merge branch 'microsoft:main' into natvis-view-customlistitems

3b07336

Merge remote-tracking branch 'fork/natvis-view-customlistitems' into …

a150bc6

…natvis-view-customlistitems

Uh oh!

Conversation

lugerard commented Jun 10, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

gregg-miskelly Jun 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

gregg-miskelly commented Jun 21, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

gregg-miskelly Jun 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

gregg-miskelly Jun 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

gregg-miskelly left a comment

Choose a reason for hiding this comment

Uh oh!

lugerard commented Jul 3, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

gregg-miskelly Jun 21, 2026 •

edited

Loading

gregg-miskelly Jun 21, 2026 •

edited

Loading

gregg-miskelly Jun 21, 2026 •

edited

Loading