Skip to content

Sanitize and parse logged TLDs using JSON arrays#3164

Merged
weiminyu merged 1 commit into
google:masterfrom
weiminyu:flowreporter_extractTld
Jul 17, 2026
Merged

Sanitize and parse logged TLDs using JSON arrays#3164
weiminyu merged 1 commit into
google:masterfrom
weiminyu:flowreporter_extractTld

Conversation

@weiminyu

@weiminyu weiminyu commented Jul 17, 2026

Copy link
Copy Markdown
Collaborator

In FlowReporter, we extract TLDs from EPP domain commands and log them under 'tld' and 'tlds' metadata fields to generate ICANN activity reports. Previously, invalid or extremely long TLD names (such as email addresses or long domain labels in non-validated XML payloads) could break downstream log parsing.

To prevent this issue, this change does the following:

  1. Java Sanitization: Introduces toLogSafeLabel(...) in DomainFlowUtils, which converts ASCII to lowercase, replaces any character outside of [a-z0-9.-] with -, and limits the length to 63 characters (appending '...' if truncated). FlowReporter now uses this method on guessed TLDs before logging them.

  2. SQL Modernization: Upgrades epp_metrics.sql and epp_metrics_test.sql to use BigQuery's native JSON_EXTRACT_STRING_ARRAY(json, '$.tlds') instead of the legacy SPLIT(REGEXP_EXTRACT(JSON_EXTRACT(...))) workaround. Because the native function extracts clean string elements, it removes the need to strip quotation marks with additional regexes and prevents parsing failures on commas, spaces, or nested structures.

SQL change tested in BigQuery.

BUG=http://b/535230985


This change is Reviewable

@weiminyu
weiminyu requested a review from gbrodman July 17, 2026 16:43

@gbrodman gbrodman left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@gbrodman reviewed 5 files and all commit messages, and made 1 comment.
Reviewable status: all files reviewed, 1 unresolved discussion (waiting on weiminyu).


core/src/main/java/google/registry/flows/domain/DomainFlowUtils.java line 187 at r1 (raw file):

  /// This method is mainly for use by `FlowReporter#recordToLogs()` to ensure that the logs
  /// can be safely and correctly queried by SQL queries. See b/534931586 for more information.
  public static Optional<String> toLogSafeLabel(String label) {

need to move the toLowerCase call to this method to satisfy the commit message

In FlowReporter, we extract TLDs from EPP domain commands and log them under 'tld' and 'tlds' metadata fields to generate ICANN activity reports. Previously, invalid or extremely long TLD names (such as email addresses or long domain labels in non-validated XML payloads) could break downstream log parsing.

To prevent this issue, this change does the following:

1. Java Sanitization:
   Introduces `toLogSafeLabel(...)` in `DomainFlowUtils`, which converts ASCII to lowercase, replaces any character outside of `[a-z0-9.-]` with `-`, and limits the length to 63 characters (appending '...' if truncated). FlowReporter now uses this method on guessed TLDs before logging them.

2. SQL Modernization:
   Upgrades `epp_metrics.sql` and `epp_metrics_test.sql` to use BigQuery's native `JSON_EXTRACT_STRING_ARRAY(json, '$.tlds')` instead of the legacy `SPLIT(REGEXP_EXTRACT(JSON_EXTRACT(...)))` workaround. Because the native function extracts clean string elements, it removes the need to strip quotation marks with additional regexes and prevents parsing failures on commas, spaces, or nested structures.

SQL change tested in BigQuery.

BUG=http://b/535230985
@weiminyu
weiminyu force-pushed the flowreporter_extractTld branch from 4b5bc5b to 356e31f Compare July 17, 2026 18:48

@weiminyu weiminyu left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@weiminyu made 1 comment.
Reviewable status: 3 of 5 files reviewed, 1 unresolved discussion (waiting on gbrodman).


core/src/main/java/google/registry/flows/domain/DomainFlowUtils.java line 187 at r1 (raw file):

Previously, gbrodman wrote…

need to move the toLowerCase call to this method to satisfy the commit message

Done.

@gbrodman gbrodman left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@gbrodman reviewed 2 files and all commit messages, and resolved 1 discussion.
Reviewable status: :shipit: complete! all files reviewed, all discussions resolved (waiting on weiminyu).

@weiminyu
weiminyu enabled auto-merge July 17, 2026 19:00
@weiminyu
weiminyu added this pull request to the merge queue Jul 17, 2026
Merged via the queue into google:master with commit 7e0489e Jul 17, 2026
16 checks passed
@weiminyu
weiminyu deleted the flowreporter_extractTld branch July 17, 2026 20:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants