Sanitize and parse logged TLDs using JSON arrays#3164
Conversation
gbrodman
left a comment
There was a problem hiding this comment.
@gbrodman reviewed 5 files and all commit messages, and made 1 comment.
Reviewable status: all files reviewed, 1 unresolved discussion (waiting on weiminyu).
core/src/main/java/google/registry/flows/domain/DomainFlowUtils.java line 187 at r1 (raw file):
/// This method is mainly for use by `FlowReporter#recordToLogs()` to ensure that the logs /// can be safely and correctly queried by SQL queries. See b/534931586 for more information. public static Optional<String> toLogSafeLabel(String label) {
need to move the toLowerCase call to this method to satisfy the commit message
In FlowReporter, we extract TLDs from EPP domain commands and log them under 'tld' and 'tlds' metadata fields to generate ICANN activity reports. Previously, invalid or extremely long TLD names (such as email addresses or long domain labels in non-validated XML payloads) could break downstream log parsing. To prevent this issue, this change does the following: 1. Java Sanitization: Introduces `toLogSafeLabel(...)` in `DomainFlowUtils`, which converts ASCII to lowercase, replaces any character outside of `[a-z0-9.-]` with `-`, and limits the length to 63 characters (appending '...' if truncated). FlowReporter now uses this method on guessed TLDs before logging them. 2. SQL Modernization: Upgrades `epp_metrics.sql` and `epp_metrics_test.sql` to use BigQuery's native `JSON_EXTRACT_STRING_ARRAY(json, '$.tlds')` instead of the legacy `SPLIT(REGEXP_EXTRACT(JSON_EXTRACT(...)))` workaround. Because the native function extracts clean string elements, it removes the need to strip quotation marks with additional regexes and prevents parsing failures on commas, spaces, or nested structures. SQL change tested in BigQuery. BUG=http://b/535230985
4b5bc5b to
356e31f
Compare
weiminyu
left a comment
There was a problem hiding this comment.
@weiminyu made 1 comment.
Reviewable status: 3 of 5 files reviewed, 1 unresolved discussion (waiting on gbrodman).
core/src/main/java/google/registry/flows/domain/DomainFlowUtils.java line 187 at r1 (raw file):
Previously, gbrodman wrote…
need to move the toLowerCase call to this method to satisfy the commit message
Done.
gbrodman
left a comment
There was a problem hiding this comment.
@gbrodman reviewed 2 files and all commit messages, and resolved 1 discussion.
Reviewable status:complete! all files reviewed, all discussions resolved (waiting on weiminyu).
In FlowReporter, we extract TLDs from EPP domain commands and log them under 'tld' and 'tlds' metadata fields to generate ICANN activity reports. Previously, invalid or extremely long TLD names (such as email addresses or long domain labels in non-validated XML payloads) could break downstream log parsing.
To prevent this issue, this change does the following:
Java Sanitization: Introduces
toLogSafeLabel(...)inDomainFlowUtils, which converts ASCII to lowercase, replaces any character outside of[a-z0-9.-]with-, and limits the length to 63 characters (appending '...' if truncated). FlowReporter now uses this method on guessed TLDs before logging them.SQL Modernization: Upgrades
epp_metrics.sqlandepp_metrics_test.sqlto use BigQuery's nativeJSON_EXTRACT_STRING_ARRAY(json, '$.tlds')instead of the legacySPLIT(REGEXP_EXTRACT(JSON_EXTRACT(...)))workaround. Because the native function extracts clean string elements, it removes the need to strip quotation marks with additional regexes and prevents parsing failures on commas, spaces, or nested structures.SQL change tested in BigQuery.
BUG=http://b/535230985
This change is