feat(assets): align Facebook AdsStats with CampaignsStats, allow BigQuery field addition under RECONCILE - #305
Open
aaaaahaaaaa wants to merge 2 commits into
Open
feat(assets): align Facebook AdsStats with CampaignsStats, allow BigQuery field addition under RECONCILE#305aaaaahaaaaa wants to merge 2 commits into
aaaaahaaaaa wants to merge 2 commits into
Conversation
…NCILE An asset schema that gains a field could not reach an existing BigQuery table: the load job carries the declared schema with WRITE_APPEND and no schema_update_options, so BigQuery rejects the append with "Cannot add fields". Both load paths (Parquet DataFrame and JSON rows) now set ALLOW_FIELD_ADDITION, decided by one helper, only when the destination's materialization strategy is RECONCILE and a declared schema drives the load. AUTO and STRICT are unchanged, and schema-less loads never let inferred columns grow a table. By Digitl
Both Facebook stats assets request the same action arrays from the Meta Insights API and pivot them into one column per action type; the schema then acts as the allowlist the RECONCILE step keeps. AdsStats declared 75 columns against 193 on CampaignsStats, so purchase, add_to_cart, initiate_checkout, view_content, search and the omni/onsite_web/pixel breakdowns were fetched and silently dropped at ad level. AdsStats now declares the 122 action columns CampaignsStats has (types and descriptions copied verbatim); both schemas also declare the attribution_setting, quality_ranking and converted_product_quantity scalars their assets already request. Two regression tests pin the parity and fail when a requested field is left undeclared. Verified against a live Swarovski account for 2026-08-28: row counts, impressions, spend and purchases identical to the previous stack, with 69 newly populated ad-level columns. By Digitl
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Two commits, in the order they must land:
fix(google-cloud): BigQuery loads may add declared columns underRECONCILE. One helper decidesschema_update_optionsfor both load paths (Parquet DataFrame and JSON rows). It returnsALLOW_FIELD_ADDITIONonly when the destination's materialization strategy isRECONCILEand a declared schema drives the load.AUTO/STRICTare unchanged; schema-less loads never let inferred columns grow a table.feat(assets): FacebookAdsStatsaligned withCampaignsStats.AdsStatsgrows from 75 to 200 fields (the 122 action-type columnsCampaignsStatsalready declares, copied verbatim, plusattribution_setting,quality_ranking,converted_product_quantity).CampaignsStatsgains the same three scalars (193 to 196). Existing fields, order and style untouched.Why
Both Facebook stats assets request the same
actions/action_values/cost_per_action_type/unique_actionsarrays from Meta; the pivot yields one column per action type and the schema is the allowlist theRECONCILEstep keeps. With 75 declared columns, purchase, add_to_cart, initiate_checkout, view_content, search and the omni / onsite_web / pixel breakdowns were fetched and silently dropped at ad level. The 75-vs-193 gap dates from how the two schemas were sampled in April 2024, not from any API or size constraint (identical requests, sparse columns cost roughly 8 % per row).Widening a schema was not releasable before the first commit: the load job carries the declared schema with
WRITE_APPENDand noschema_update_options, so an existing table rejects the first append with "Cannot add fields".Verification
TestSchemaParity(assets) fails ifAdsStatsever lacks a campaign action column or if a requested field is left undeclared;TestSchemaUpdateOptions(google-cloud) covers RECONCILE with and without schema, and AUTO.ruff check,ty, full pytest via pre-commit on both commits.Reviewer notes
ruff format --checkflags two pre-existing hunks indestination.pyunrelated to this change; formatting is not enforced by pre-commit, so they were left alone.account_idstaysstronAdsStatsandintonCampaignsStats; changing a type on existing tables is a separate decision.offsite_conversion_custom_<id>) are out of scope of a shared catalog schema.By Digitl