Skip to content

HDDS-16387. ContainerBalancer status --verbose throws IllegalStateException after a rejected start. - #11275

Open
navinko wants to merge 5 commits into
apache:masterfrom
navinko:HDDS-16387
Open

navinko wants to merge 5 commits into
apache:masterfrom
navinko:HDDS-16387

Conversation

@navinko

@navinko navinko commented Sep 19, 2026

Copy link
Copy Markdown
Contributor

What changes were proposed in this pull request?

Updated ContainerBalancer#start() and ContainerBalancer#startBalancer() code to assign startedAt after validation runs.

In the existing ContainerBalancer code:

ContainerBalancer#start() and ContainerBalancer#startBalancer() assign startedAt = OffsetDateTime.now() at the very top of the method, before any validation runs. If a start attempt is rejected during validation, startedAt is still advanced even though no new balancing run actually begins.

After a previous run has completed (so stoppedAt is set), a subsequent rejected start pushes startedAt past stoppedAt. This yields a negative "balancing duration".

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-16387

How was this patch tested?

  • Successful CI : https://github.com/navinko/ozone/actions/runs/35459414407

  • Unit testing with new test - TestContainerBalancer#testRejectedStartDoesNotModifyStartedAt which verifies that a start request rejected during validation does not update startedAt of last run.

  • Tested ContainerBalancer locally.

Before Fix

bash-5.1$ ozone admin containerbalancer start -d 10
Failed to start Container Balancer. max-datanodes-percentage-to-involve-per-iteration=10 allows at most 0 datanode(s) per iteration with 5 eligible datanode(s), but at least 2 are required for a source and target datanode pair.

bash-5.1$ ozone admin containerbalancer status --verbose
ContainerBalancer is Not Running.
Stop reason: INITIALIZATION_FAILED
Message: Failed to initialize a container balancer iteration. Details: iteration number 1, Did not find any unbalanced Datanodes.
Started at: 2026-09-05 19:12:37
Stopped at: 2026-09-05 15:25:14
java.lang.IllegalStateException: Provided duration is incorrect: PT-3H-47M-23S
at org.apache.hadoop.hdds.util.DurationUtil.getPrettyDuration(DurationUtil.java:49)

After Fix

bash-5.1$ ozone admin containerbalancer start -d=100
Container Balancer started successfully.

bash-5.1$ ozone admin containerbalancer status --verbose
ContainerBalancer is Not Running.
Stop reason: INITIALIZATION_FAILED
Message: Failed to initialize a container balancer iteration. Details: iteration number 1, Did not find any unbalanced Datanodes.
Started at: 2026-09-19 13:03:18
Stopped at: 2026-09-19 13:03:18
Balancing duration: 0s

bash-5.1$ ozone admin containerbalancer start -d 10
Failed to start Container Balancer. max-datanodes-percentage-to-involve-per-iteration=10 allows at most 0 datanode(s) per iteration with 5 eligible datanode(s), but at least 2 are required for a source and target datanode pair.
bash-5.1$ ozone admin containerbalancer status --verbose
ContainerBalancer is Not Running.
Stop reason: INITIALIZATION_FAILED
Message: Failed to initialize a container balancer iteration. Details: iteration number 1, Did not find any unbalanced Datanodes.
Started at: 2026-09-19 13:03:18
Stopped at: 2026-09-19 13:03:18
Balancing duration: 0s

Started at from last run did not get overridden.

@sreejasahithi
sreejasahithi self-requested a review September 20, 2026 17:01

@sravani-revuri sravani-revuri left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the patch @navinko ! Few minor nits below.

*/
@Test
public void testRejectedStartDoesNotModifyStartedAt() throws Exception {
// The test first starts and immediately stops the balancer normally.This creates a realistic "previous state"

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit:

Suggested change
// The test first starts and immediately stops the balancer normally.This creates a realistic "previous state"
// The test first starts and immediately stops the balancer normally. This creates a realistic "previous state"

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added the space.

// The test first starts and immediately stops the balancer normally.This creates a realistic "previous state"
// in system memory and later trigger a bad request and verify that startedAt remains identical to startedAtBefore,
// rather than being wrongly overwritten with "now", which would make startedAt come after stoppedAt which resulted
// negative balancing duration.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit:

Suggested change
// negative balancing duration.
// in a negative balancing duration.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

corrected the comment.


// A start that is rejected during validation. Here we used an invalid config this is just one of many rejection
// paths (safe mode, non-leader SCM, already running, etc.).
OzoneConfiguration invalidConf = new OzoneConfiguration();

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: consider building the invalid config similar to testRejectInvalidStartupConfiguration ( use setMoveReplicationTimeout / setMoveTimeout on balancerConfiguration ) instead of creating a new OzoneConfiguration .

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you! Ah i missed reusing the existing pattern, updated now.

@navinko

navinko commented Sep 21, 2026

Copy link
Copy Markdown
Contributor Author

Thanks @sreejasahithi @sravani-revuri for the review . I tried addressing the review suggestions.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants