HDDS-16179. Dry run command for container balancer with profiles

[sreejasahithi]

What changes were proposed in this pull request?

This PR adds a dry-run subcommand to ozone admin containerbalancer so users can estimate how much data the Container Balancer would move, how many iterations it would need, and how long it could take(upper bound), without starting the balancer.

Flow:

• Fetch datanode usage from SCM(via the existing getDatanodeUsageInfo RPC)
• Reads balancer limits, timeouts, and profile presets from local Ozone configuration (with optional CLI overrides)
• Analyze the cluster and estimate one or more throttling profiles

For each profile is reports:

• bytes to move
• per-iteration throughput
• estimated iterations
• upper-bound duration (with a +30% planning buffer)

Key changes:

• ozone admin containerbalancer dry-run shows estimates for only MEDIUM profile.
• ozone admin containerbalancer dry-run with optional --profile slow|medium|fast, only the requested profile is estimated.
• ozone admin containerbalancer dry-run --all all three profiles are estimated.
• Introduces SLOW, MEDIUM, and FAST throttling profiles with new hdds.container.balancer.profile.* config keys for datanode involvement and per-node move limits.
• Adds ContainerBalancerAdvisor and ContainerBalancerEstimation for dry-run cluster analysis and estimation.
• Extracts shared CLI options into ContainerBalancerConfigOptions, reused by start and dry-run (start behaviour unchanged).
• When all three profiles are estimated, a validation failure in one profile (e.g. FAST) does not block SLOW/MEDIUM results.

Note: start does not support --profile yet.

What is the link to the Apache JIRA

HDDS-16179

How was this patch tested?

Unit tests: TestContainerBalancerAdvisor, TestContainerBalancerSubCommand
Manual tested the dry-run command in a cluster:

1. ozone admin containerbalancer dry-run

Profile: MEDIUM
 Based on:
   Datanode involvement:     40%
   Max entering target:      26 GB / node
   Max leaving source:       26 GB / node
   Max per iteration:        500 GB
   Move timeout:             65 min
   Balancing interval:       70 min
 Bytes to move:            108.13 GB
 Per iteration (estimate): ~26 GB
 Estimated iterations:     5 (planning estimate: 7, includes +30% buffer)
 Estimated duration:       upper bound ~11.3 hours (planning estimate: ~15.8 hours, includes +30% buffer)
                           (assumes full move timeout + interval each cycle)

2. ozone admin containerbalancer dry-run --threshold 5

Profile: MEDIUM
Based on:
  Datanode involvement:     40%
  Max entering target:      26 GB / node
  Max leaving source:       26 GB / node
  Max per iteration:        500 GB
  Move timeout:             65 min
  Balancing interval:       70 min
Bytes to move:            146.10 GB
Per iteration (estimate): ~26 GB
Estimated iterations:     6 (planning estimate: 8, includes +30% buffer)
Estimated duration:       upper bound ~13.5 hours (planning estimate: ~18.0 hours, includes +30% buffer)
                          (assumes full move timeout + interval each cycle)

3. ozone admin containerbalancer dry-run --profile SLOW

Profile: SLOW
Based on:
 Datanode involvement:     40%
 Max entering target:      10 GB / node
 Max leaving source:       10 GB / node
 Max per iteration:        500 GB
 Move timeout:             65 min
 Balancing interval:       70 min
Bytes to move:            108.13 GB
Per iteration (estimate): ~10 GB
Estimated iterations:     11 (planning estimate: 15, includes +30% buffer)
Estimated duration:       upper bound ~1.0 days (planning estimate: ~1.4 days, includes +30% buffer)
                         (assumes full move timeout + interval each cycle)

4. ozone admin containerbalancer dry-run --all

Profile: SLOW
 Based on:
   Datanode involvement:     40%
   Max entering target:      10 GB / node
   Max leaving source:       10 GB / node
   Max per iteration:        500 GB
   Move timeout:             65 min
   Balancing interval:       70 min
 Bytes to move:            108.13 GB
 Per iteration (estimate): ~10 GB
 Estimated iterations:     11 (planning estimate: 15, includes +30% buffer)
 Estimated duration:       upper bound ~1.0 days (planning estimate: ~1.4 days, includes +30% buffer)
                           (assumes full move timeout + interval each cycle)

Profile: MEDIUM
 Based on:
   Datanode involvement:     40%
   Max entering target:      26 GB / node
   Max leaving source:       26 GB / node
   Max per iteration:        500 GB
   Move timeout:             65 min
   Balancing interval:       70 min
 Bytes to move:            108.13 GB
 Per iteration (estimate): ~26 GB
 Estimated iterations:     5 (planning estimate: 7, includes +30% buffer)
 Estimated duration:       upper bound ~11.3 hours (planning estimate: ~15.8 hours, includes +30% buffer)
                           (assumes full move timeout + interval each cycle)

Profile: FAST
 Based on:
   Datanode involvement:     40%
   Max entering target:      100 GB / node
   Max leaving source:       100 GB / node
   Max per iteration:        500 GB
   Move timeout:             65 min
   Balancing interval:       70 min
 Bytes to move:            108.13 GB
 Per iteration (estimate): ~100 GB
 Estimated iterations:     2 (planning estimate: 3, includes +30% buffer)
 Estimated duration:       upper bound ~4.5 hours (planning estimate: ~6.8 hours, includes +30% buffer)
                           (assumes full move timeout + interval each cycle)

5. ozone admin containerbalancer dry-run --threshold 5 --max-size-entering-target-in-gb 15 --max-size-leaving-source-in-gb 15 --balancing-iteration-interval-minutes 30 --profile SLOW

Profile: SLOW
 Based on:
   Datanode involvement:     40%
   Max entering target:      15 GB / node
   Max leaving source:       15 GB / node
   Max per iteration:        500 GB
   Move timeout:             65 min
   Balancing interval:       30 min
 Bytes to move:            146.10 GB
 Per iteration (estimate): ~15 GB
 Estimated iterations:     10 (planning estimate: 13, includes +30% buffer)
 Estimated duration:       upper bound ~15.8 hours (planning estimate: ~20.6 hours, includes +30% buffer)
                           (assumes full move timeout + interval each cycle)

6. ozone admin containerbalancer dry-run --max-size-to-move-per-iteration-in-gb 70 --threshold 5 --all

Profile: SLOW
 Based on:
   Datanode involvement:     40%
   Max entering target:      10 GB / node
   Max leaving source:       10 GB / node
   Max per iteration:        70 GB
   Move timeout:             65 min
   Balancing interval:       70 min
 Bytes to move:            146.10 GB
 Per iteration (estimate): ~10 GB
 Estimated iterations:     15 (planning estimate: 20, includes +30% buffer)
 Estimated duration:       upper bound ~1.4 days (planning estimate: ~1.9 days, includes +30% buffer)
                           (assumes full move timeout + interval each cycle)

Profile: MEDIUM
 Based on:
   Datanode involvement:     40%
   Max entering target:      26 GB / node
   Max leaving source:       26 GB / node
   Max per iteration:        70 GB
   Move timeout:             65 min
   Balancing interval:       70 min
 Bytes to move:            146.10 GB
 Per iteration (estimate): ~26 GB
 Estimated iterations:     6 (planning estimate: 8, includes +30% buffer)
 Estimated duration:       upper bound ~13.5 hours (planning estimate: ~18.0 hours, includes +30% buffer)
                           (assumes full move timeout + interval each cycle)

Profile: FAST
 Based on:
   Datanode involvement:     40%
   Max entering target:      100 GB / node
   Max leaving source:       100 GB / node
   Max per iteration:        70 GB
   Move timeout:             65 min
   Balancing interval:       70 min
 Estimation failed: max-size-entering-target must be less than or equal to max-size-to-move-per-iteration.

7. ozone admin containerbalancer dry-run --max-size-entering-target-in-gb 50 --max-size-leaving-source-in-gb 50

Profile: MEDIUM
 Based on:
   Datanode involvement:     40%
   Max entering target:      50 GB / node
   Max leaving source:       50 GB / node
   Max per iteration:        500 GB
   Move timeout:             65 min
   Balancing interval:       70 min
 Bytes to move:            108.13 GB
 Per iteration (estimate): ~50 GB
 Estimated iterations:     3 (planning estimate: 4, includes +30% buffer)
 Estimated duration:       upper bound ~6.8 hours (planning estimate: ~9.0 hours, includes +30% buffer)
                           (assumes full move timeout + interval each cycle)

The usageinfo for this test is:
ozone admin datanode usageinfo -c=100 -l

Usage Information (5 Datanodes)

ID                      : 5ff6b32f-9186-4d4b-bb04-a41025ffe8c5 
IP Address              : X.X.X.12 
Hostname                : datanode5 
Ozone Capacity          : 274850419154 B (255.97 GB) 
Ozone Used              : 4460985 B (4.25 MB) 
Ozone Used %            : 0.00% (Ozone Used/Ozone Capacity) 
Ozone Available         : 228511166464 B (212.82 GB) 
Ozone Available %       : 83.14% (Ozone Available/Ozone capacity) 
Pipeline(s)             : 1 
Container(s)            : 4 
Container Pre-allocated : 0 B (0 B) 
Remaining Allocatable   : 228511166464 B (212.82 GB) 
Free Space To Spare     : 21474836480 B (20 GB) 
Reserved                : 0 B (0 B) 

ID                      : 212ab8c7-ccbe-4d7a-98da-11924816db3b 
IP Address              : X.X.X.196 
Hostname                : datanode4 
Ozone Capacity          : 274850419154 B (255.97 GB) 
Ozone Used              : 4460985 B (4.25 MB) 
Ozone Used %            : 0.00% (Ozone Used/Ozone Capacity) 
Ozone Available         : 228195598336 B (212.52 GB) 
Ozone Available %       : 83.03% (Ozone Available/Ozone capacity) 
Pipeline(s)             : 1 
Container(s)            : 4 
Container Pre-allocated : 0 B (0 B) 
Remaining Allocatable   : 228195598336 B (212.52 GB) 
Free Space To Spare     : 21474836480 B (20 GB) 
Reserved                : 0 B (0 B) 

ID                      : 18aa7a4c-fafe-40cd-9002-f95c2a74bcad 
IP Address              : X.X.X.199 
Hostname                : datanode3 
Ozone Capacity          : 274850419154 B (255.97 GB) 
Ozone Used              : 156347144916 B (145.61 GB) 
Ozone Used %            : 56.88% (Ozone Used/Ozone Capacity) 
Ozone Available         : 72413556673 B (67.44 GB) 
Ozone Available %       : 26.35% (Ozone Available/Ozone capacity) 
Pipeline(s)             : 3 
Container(s)            : 37 
Container Pre-allocated : 24557647337 B (22.87 GB) 
Remaining Allocatable   : 47855909336 B (44.57 GB) 
Free Space To Spare     : 21474836480 B (20 GB) 
Reserved                : 0 B (0 B) 

ID                      : 317d55fe-f0ba-435e-b96f-44f1b3ce4214 
IP Address              : X.X.X.141 
Hostname                : datanode2
Ozone Capacity          : 270339965924 B (251.77 GB) 
Ozone Used              : 152572193303 B (142.09 GB) 
Ozone Used %            : 56.44% (Ozone Used/Ozone Capacity) 
Ozone Available         : 57404882881 B (53.46 GB) 
Ozone Available %       : 21.23% (Ozone Available/Ozone capacity) 
Pipeline(s)             : 3 
Container(s)            : 36 
Container Pre-allocated : 28332520937 B (26.39 GB) 
Remaining Allocatable   : 29072361944 B (27.08 GB) 
Free Space To Spare     : 21474836480 B (20 GB) 
Reserved                : 0 B (0 B) 

ID                      : fd57f332-990f-4359-934d-e688217bb3cf 
IP Address              : X.X.X.72 
Hostname                : datanode1
Ozone Capacity          : 270339965924 B (251.77 GB) 
Ozone Used              : 152467335955 B (142.00 GB) 
Ozone Used %            : 56.40% (Ozone Used/Ozone Capacity) 
Ozone Available         : 56859631553 B (52.95 GB) 
Ozone Available %       : 21.03% (Ozone Available/Ozone capacity) 
Pipeline(s)             : 3 
Container(s)            : 39 
Container Pre-allocated : 30429672937 B (28.34 GB) 
Remaining Allocatable   : 26429958616 B (24.61 GB) 
Free Space To Spare     : 21474836480 B (20 GB) 
Reserved                : 0 B (0 B) 

Green CI: https://github.com/sreejasahithi/ozone/actions/runs/35199786999