Skip to content

feat(assets): align Facebook AdsStats with CampaignsStats, allow BigQuery field addition under RECONCILE - #305

Open
aaaaahaaaaa wants to merge 2 commits into
mainfrom
feat/facebook-ads-stats-parity
Open

feat(assets): align Facebook AdsStats with CampaignsStats, allow BigQuery field addition under RECONCILE#305
aaaaahaaaaa wants to merge 2 commits into
mainfrom
feat/facebook-ads-stats-parity

Conversation

@aaaaahaaaaa

Copy link
Copy Markdown
Contributor

What

Two commits, in the order they must land:

  1. fix(google-cloud): BigQuery loads may add declared columns under RECONCILE. One helper decides schema_update_options for both load paths (Parquet DataFrame and JSON rows). It returns ALLOW_FIELD_ADDITION only when the destination's materialization strategy is RECONCILE and a declared schema drives the load. AUTO/STRICT are unchanged; schema-less loads never let inferred columns grow a table.
  2. feat(assets): Facebook AdsStats aligned with CampaignsStats. AdsStats grows from 75 to 200 fields (the 122 action-type columns CampaignsStats already declares, copied verbatim, plus attribution_setting, quality_ranking, converted_product_quantity). CampaignsStats gains the same three scalars (193 to 196). Existing fields, order and style untouched.

Why

Both Facebook stats assets request the same actions / action_values / cost_per_action_type / unique_actions arrays from Meta; the pivot yields one column per action type and the schema is the allowlist the RECONCILE step keeps. With 75 declared columns, purchase, add_to_cart, initiate_checkout, view_content, search and the omni / onsite_web / pixel breakdowns were fetched and silently dropped at ad level. The 75-vs-193 gap dates from how the two schemas were sampled in April 2024, not from any API or size constraint (identical requests, sparse columns cost roughly 8 % per row).

Widening a schema was not releasable before the first commit: the load job carries the declared schema with WRITE_APPEND and no schema_update_options, so an existing table rejects the first append with "Cannot add fields".

Verification

  • TestSchemaParity (assets) fails if AdsStats ever lacks a campaign action column or if a requested field is left undeclared; TestSchemaUpdateOptions (google-cloud) covers RECONCILE with and without schema, and AUTO.
  • Live ad hoc run against a Swarovski production account (403584573777711) for 2026-08-28, both assets, local destination: rows 26,885 / 2,686, impressions 3,613,356, spend 11,673.22, purchases 704, all identical to the previous stack; 69 newly populated ad-level columns; nothing undeclared in the output.
  • ruff check, ty, full pytest via pre-commit on both commits.

Reviewer notes

  • Merge with rebase; the fix commit must not land after the feat commit.
  • ruff format --check flags two pre-existing hunks in destination.py unrelated to this change; formatting is not enforced by pre-commit, so they were left alone.
  • account_id stays str on AdsStats and int on CampaignsStats; changing a type on existing tables is a separate decision.
  • Per-account custom conversions (offsite_conversion_custom_<id>) are out of scope of a shared catalog schema.

By Digitl

…NCILE

An asset schema that gains a field could not reach an existing BigQuery
table: the load job carries the declared schema with WRITE_APPEND and no
schema_update_options, so BigQuery rejects the append with "Cannot add
fields". Both load paths (Parquet DataFrame and JSON rows) now set
ALLOW_FIELD_ADDITION, decided by one helper, only when the destination's
materialization strategy is RECONCILE and a declared schema drives the
load. AUTO and STRICT are unchanged, and schema-less loads never let
inferred columns grow a table.

By Digitl
Both Facebook stats assets request the same action arrays from the Meta
Insights API and pivot them into one column per action type; the schema
then acts as the allowlist the RECONCILE step keeps. AdsStats declared 75
columns against 193 on CampaignsStats, so purchase, add_to_cart,
initiate_checkout, view_content, search and the omni/onsite_web/pixel
breakdowns were fetched and silently dropped at ad level.

AdsStats now declares the 122 action columns CampaignsStats has (types
and descriptions copied verbatim); both schemas also declare the
attribution_setting, quality_ranking and converted_product_quantity
scalars their assets already request. Two regression tests pin the parity
and fail when a requested field is left undeclared.

Verified against a live Swarovski account for 2026-08-28: row counts,
impressions, spend and purchases identical to the previous stack, with 69
newly populated ad-level columns.

By Digitl
@codecov-commenter

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants