Skip to content

fix: select Shapley Value column before aggregating in plot_global_feat - #63

Open
daniel-n-silva wants to merge 1 commit into
feedzai:mainfrom
daniel-n-silva:fix/groupby-pandas2-compat
Open

daniel-n-silva wants to merge 1 commit into
feedzai:mainfrom
daniel-n-silva:fix/groupby-pandas2-compat

Conversation

@daniel-n-silva

Copy link
Copy Markdown

Summary

plot_global_feat() raises TypeError: agg function failed [how->mean,dtype->object] under pandas >= 2.0.

Root cause

feat_data.groupby('Feature').mean()['Shapley Value'] aggregates every column first and selects Shapley Value afterwards. Global feature explanations produced by feat_explain_all always carry an Entity column, which ends up as object dtype whenever the entity identifier is a string (as in the AReM.ipynb example notebook) — feat_data.astype(...) never casts it. filter_dataset forwards this column unchanged into plot_global_feat. Older pandas silently dropped non-numeric columns before averaging; pandas >= 2.0 raises instead.

Fix

Select the Shapley Value column before aggregating, so the other columns are never touched:

avg_df = feat_data.groupby('Feature')['Shapley Value'].mean()

Identical result when no extra columns are present; works on both old and new pandas without needing numeric_only=True.

Fixes #62.

Test plan

  • Reproduced the TypeError locally with pandas 2.3.3 using the real column set produced by feat_explain_all (Random Seed, NSamples, Feature, Shapley Value, Entity, Tolerance) with a string Entity column
  • Confirmed the fix resolves it and returns the same values/Series.name/index.name as before
  • Checked the other groupby(...).mean() call sites in src/ (event_level.py:104, pruning.py:92) — both already select/aggregate safely, no change needed there

feat_data.groupby('Feature').mean() raises
"TypeError: agg function failed [how->mean,dtype->object]" under
pandas >= 2.0. Global feature explanations produced by feat_explain_all
carry an 'Entity' column, which is of object dtype whenever the entity
identifier is a string (as in the AReM example notebook), and
filter_dataset forwards it unchanged to plot_global_feat.

Selecting the column before aggregating skips the other columns
entirely, working on both old and new pandas without numeric_only=True.
The result is identical when no extra columns are present.

Fixes feedzai#62.
@daniel-n-silva
daniel-n-silva force-pushed the fix/groupby-pandas2-compat branch from 3938db7 to d1c6500 Compare September 11, 2026 18:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Global Reports failing

1 participant