Stop asserting exact location-version-id equality after replication

[delthas]

The Bucket Replication location stripping scenario asserted that the <location>-version-id stamped in source metadata equals the destination's current version id. Replication is at-least-once, so that is not an invariant the product offers, and the test fails on correct behaviour.

What it caught. In one census run the object was replicated twice: two oplog populators were alive during a rolling restart and each allocated a Kafka-Connect connector for the same bucket, producing three entries for one source version. The status processor kept the first stamp — "entry replication is already COMPLETED for this location, skipping metadata update" — while the destination retained the version written by the second replication. Both objects were present and identical in content; only the recorded version id disagreed. Replication itself had succeeded, in 59 s.

The change. Assert on content rather than identity: compare ETags (normalising the surrounding quotes, as get-object-attributes.ts:68 already does) and require only that the version-id stamp is present. ContentLength was already compared. The neighbouring scal-version-id, scal-replication-status and <location>-replication-status assertions are untouched — scal-version-id records the source version, which both copies share, so the double replication does not affect it.

Census confirmation. Across the two 30-run censuses since:

• location stripping failures: 2/30 → 0/30
• the changed assertion ran 420 times with zero failures in the latest census

Issue: ZENKO-5339

[bert-e]

Hello delthas,

My role is to assist you with the merge of this
pull request. Please type @bert-e help to get information
on this process, or consult the user documentation.

Available options

name	description	privileged	authored
/after_pull_request	Wait for the given pull request id to be merged before continuing with the current one.
/bypass_author_approval	Bypass the pull request author's approval	⭐
/bypass_build_status	Bypass the build and test status	⭐
/bypass_commit_size	Bypass the check on the size of the changeset TBA	⭐
/bypass_incompatible_branch	Bypass the check on the source branch prefix	⭐
/bypass_jira_check	Bypass the Jira issue check	⭐
/bypass_peer_approval	Bypass the pull request peers' approval	⭐
/bypass_leader_approval	Bypass the pull request leaders' approval	⭐
/approve	Instruct Bert-E that the author has approved the pull request.		✍️
/create_pull_requests	Allow the creation of integration pull requests.
/create_integration_branches	Allow the creation of integration branches.
/no_octopus	Prevent Wall-E from doing any octopus merge and use multiple consecutive merge instead
/unanimity	Change review acceptance criteria from one reviewer at least to all reviewers
/wait	Instruct Bert-E not to run until further notice.

Available commands

name	description	privileged
/help	Print Bert-E's manual in the pull request.
/status	Print Bert-E's current status in the pull request TBA
/clear	Remove all comments from Bert-E from the history TBA
/retry	Re-start a fresh build TBA
/build	Re-start a fresh build TBA
/force_reset	Delete integration branches & pull requests, and restart merge process from the beginning.
/reset	Try to remove integration branches unless there are commits on them which do not appear on the source branch.

Status report is not available.

[bert-e]

Waiting for approval

The following approvals are needed before I can proceed with the merge:

• the author

• 2 peers

[SylvainSenechal]

Ok for me although when you say this :

"What it caught. In one census run the object was replicated twice: two oplog populators were alive during a rolling restart and each allocated a Kafka-Connect connector for the same bucket, producing three entries for one source version" :

• It would be nice to at least have a ticket to think about this behavior, and decide if its an acceptable design or if its possible to fix it on the backbeat side
• I think i would still keep the versionID assertion, and maybe when they are not equal, instead of failing the test, log a warning "test replication find non matching version ID likely due to backbeat oplog populator restarts..."

[delthas]

Both points done.

Version-id check kept as a warning (7dee82f80f). It still compares the stamped <location>-version-id against the destination's current version; on mismatch it logs a warning naming the likely cause instead of failing:

replicated object version id does not match the source stamp
  location, stampedVersionId, destinationVersionId,
  hint: the object was likely replicated more than once, for instance by two
        oplog populators both allocating a connector for the bucket after a restart

That keeps the signal visible without failing on something at-least-once replication is allowed to do.

Ticket filed: BB-834, for the populator behaviour itself — fence bucket-to-connector ownership so a terminating and a starting instance cannot both allocate the same bucket. It records the traced occurrence (three replication entries for one source version, the status processor dropping the second stamp with "already COMPLETED for this location, skipping metadata update"), and explicitly notes that after this PR a mismatch no longer fails CI, so nothing will catch it regressing. Suggested looking at it together with the populator's scan/watch ordering hole, since both are about ownership across a restart.

Worth adding for context: the later 30-run census did not reproduce it, so there is no rate estimate beyond that single occurrence. The trigger is rollout frequency, which in CI is inflated by the operator re-rendering every backbeat deployment on each location CRUD (ZKOP-568).

[DarkIsDude]

Same as Sylvain. I'm not sure it's the right approach. We should fix that in backbeat instead of here and keep the hard comparison here (it's what we want finally). Or it's a product decision and we are align with it, then your fix is the right one (but it seems not the case, at least in the scenario tested where one replication should happens), or it's not and we should fix it in backbeat.

Moreover, I would approve this PR if the occurence is high enough to be painful. Seems not the case regarding your last comment (30 runs didn't reproduce it).

We can also ask to the team what do they think, but IMO it's not the right approach here.

[DarkIsDude]

@delthas also in your ticket https://scality.atlassian.net/browse/BB-834 add a link to this thread to have the full context ?

[delthas]

@DarkIsDude you're right on the frequency, and I checked rather than assumed. The single occurrence came from one census run (Bucket Replication location stripping, 4 MiB row); no census since has reproduced it. The 30-run July census still had the hard equality assert in place and produced 6 replication-related failures, all of them timeouts, with no version-id assertion failure among the annotations. So the rate is low enough that relaxing a CI assertion is not worth it.

Closing in favour of BB-834.

[delthas]

Also I linked the thread in the ticket.