HDDS-16174. Add container balancer assessment command to report cluster imbalance by sravani-revuri · Pull Request #11256 · apache/ozone

sravani-revuri · 2026-09-18T05:55:11Z

What changes were proposed in this pull request?

Provide a one-liner summary of the changes in the PR Title field above.
It should be in the form of HDDS-1234. Short summary of the change.

Please describe your PR in detail:
he users need a way to understand cluster imbalance before starting the balancer — without changing anything or getting config suggestions. The assessment command answers: how imbalanced is the cluster, which datanodes are over/under utilized, and roughly how much data would need to move.

Add ozone admin containerbalancer assessment command which fetches datanode usage info via the existing getDatanodeUsageInfo RPC, runs the cluster analysis from HDDS-16173

and shows a human-readable report. It should provide a simple category label (e.g. large cluster, high imbalance, medium movement ratio) can be derived in the CLI from the returned summary from analyzer.
Output example:

CLUSTER BALANCE ASSESSMENT
 ═════════════════════════════════════════

  Drift:             23.5%  (max utilization 91.2% − min utilization 67.7%)

  Mean Utilization:  78.4%

  Eligible datanodes: 120 datanodes

  Category:          Large cluster, High imbalance, Medium movement ratio

 

 ─── Source Nodes (over-utilized) ─────────────────────

  14 datanodes above threshold

  Top 5:

    dn-worker-017.example.com    91.2%   (+13.2% above mean)

    dn-worker-004.example.com    89.8%   (+11.4% above mean)

    dn-worker-022.example.com    88.1%   (+9.7% above mean)

    dn-worker-011.example.com    87.6%   (+9.2% above mean)

    dn-worker-039.example.com    86.9%   (+8.5% above mean)

 

 ─── Target Nodes (under-utilized) ─────────────────────

  11 datanodes below threshold

  Bottom 5:

    dn-worker-118.example.com    67.7%   (−10.7% below mean)

    dn-worker-119.example.com    68.2%   (−10.2% below mean)

    dn-worker-120.example.com    69.1%   (−9.3% below mean)

    dn-worker-115.example.com    70.4%   (−8.0% below mean)

    dn-worker-116.example.com    71.0%   (−7.4% below mean)

 

 ─── Movement Summary ─────────────────────────

  Total bytes to move:       420 TB

  Movement ratio:            3.4% of cluster capacity

Threshold defaults to 10% from config, with an optional --threshold override. Include/exclude datanode lists should also be supported. limit option (-n) is provided to show top(n) source and target nodes in display (defaults to 5 / min number of nodes in cluster)

Assessment does not show iteration counts, duration estimates, recommended config, or container move details. Those belong to other commands.

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-16174

How was this patch tested?

Written Unit test and manual testing.

Normal command :

bash-5.1$ ozone admin containerbalancer assessment
CLUSTER BALANCE ASSESSMENT
Key                                                Value
Drift                                              28.2% (max utilization 29.0% - min utilization 0.8%)
Mean Utilization                                   14.9%
Eligible datanodes                                 6 datanodes
Category                                           Small cluster, High imbalance, Medium movement ratio


Source Nodes (over-utilized):
Datanodes above threshold                          3

Top 5:
ozone-balancer-datanode6-1.ozone-balancer_default  29.0% (+14.1% above mean)
ozone-balancer-datanode4-1.ozone-balancer_default  29.0% (+14.1% above mean)
ozone-balancer-datanode3-1.ozone-balancer_default  29.0% (+14.1% above mean)


Target Nodes (under-utilized):
Datanodes below threshold                          3

Bottom 5:
ozone-balancer-datanode5-1.ozone-balancer_default  0.8% (-14.1% below mean)
ozone-balancer-datanode2-1.ozone-balancer_default  0.8% (-14.1% below mean)
ozone-balancer-datanode1-1.ozone-balancer_default  0.8% (-14.1% below mean)


Movement Summary:
Total bytes to move                                252.08 MB
Movement ratio                                     2.1% of cluster capacity

Comparing with datanode usage command:

Usage Information (6 Datanodes)

ID                      : c999add5-2c55-4c9c-91ca-13f32d1f18fe 
IP Address              : 172.18.0.13 
Hostname                : ozone-balancer-datanode6-1.ozone-balancer_default 
Filesystem Capacity     : 2147483648 B (2 GB) 
Filesystem Used         : 622723072 B (593.88 MB) 
Filesystem Used %       : 29.00% (Filesystem Used/Filesystem Capacity) 
Filesystem Available    : 1524760576 B (1.42 GB) 
Filesystem Available %  : 71.00% (Filesystem Available/Filesystem Capacity) 
Ozone Capacity          : 2147268899 B (2.00 GB) 
Ozone Used              : 314576896 B (300.00 MB) 
Ozone Used %            : 14.65% (Ozone Used/Ozone Capacity) 
Ozone Available         : 1524760576 B (1.42 GB) 
Ozone Available %       : 71.01% (Ozone Available/Ozone capacity) 
Pipeline(s)             : 1 
Container(s)            : 3 
Container Pre-allocated : 0 B (0 B) 
Remaining Allocatable   : 1524760576 B (1.42 GB) 
Free Space To Spare     : 104857600 B (100 MB) 
Reserved                : 214749 B (209.72 KB) 

ID                      : f196a9e4-1906-4994-a489-a1d892d9fdff 
IP Address              : 172.18.0.12 
Hostname                : ozone-balancer-datanode4-1.ozone-balancer_default 
Filesystem Capacity     : 2147483648 B (2 GB) 
Filesystem Used         : 622723072 B (593.88 MB) 
Filesystem Used %       : 29.00% (Filesystem Used/Filesystem Capacity) 
Filesystem Available    : 1524760576 B (1.42 GB) 
Filesystem Available %  : 71.00% (Filesystem Available/Filesystem Capacity) 
Ozone Capacity          : 2147268899 B (2.00 GB) 
Ozone Used              : 314576896 B (300.00 MB) 
Ozone Used %            : 14.65% (Ozone Used/Ozone Capacity) 
Ozone Available         : 1524760576 B (1.42 GB) 
Ozone Available %       : 71.01% (Ozone Available/Ozone capacity) 
Pipeline(s)             : 1 
Container(s)            : 3 
Container Pre-allocated : 0 B (0 B) 
Remaining Allocatable   : 1524760576 B (1.42 GB) 
Free Space To Spare     : 104857600 B (100 MB) 
Reserved                : 214749 B (209.72 KB) 

ID                      : 68d912f2-53ce-49b1-a16d-0a2d0d706b8f 
IP Address              : 172.18.0.8 
Hostname                : ozone-balancer-datanode3-1.ozone-balancer_default 
Filesystem Capacity     : 2147483648 B (2 GB) 
Filesystem Used         : 622723072 B (593.88 MB) 
Filesystem Used %       : 29.00% (Filesystem Used/Filesystem Capacity) 
Filesystem Available    : 1524760576 B (1.42 GB) 
Filesystem Available %  : 71.00% (Filesystem Available/Filesystem Capacity) 
Ozone Capacity          : 2147268899 B (2.00 GB) 
Ozone Used              : 314576896 B (300.00 MB) 
Ozone Used %            : 14.65% (Ozone Used/Ozone Capacity) 
Ozone Available         : 1524760576 B (1.42 GB) 
Ozone Available %       : 71.01% (Ozone Available/Ozone capacity) 
Pipeline(s)             : 1 
Container(s)            : 3 
Container Pre-allocated : 0 B (0 B) 
Remaining Allocatable   : 1524760576 B (1.42 GB) 
Free Space To Spare     : 104857600 B (100 MB) 
Reserved                : 214749 B (209.72 KB) 

ID                      : a5230136-1116-40e7-8ed1-af552b4813ea 
IP Address              : 172.18.0.6 
Hostname                : ozone-balancer-datanode5-1.ozone-balancer_default 
Filesystem Capacity     : 2147483648 B (2 GB) 
Filesystem Used         : 17051648 B (16.26 MB) 
Filesystem Used %       : 0.79% (Filesystem Used/Filesystem Capacity) 
Filesystem Available    : 2130432000 B (1.98 GB) 
Filesystem Available %  : 99.21% (Filesystem Available/Filesystem Capacity) 
Ozone Capacity          : 2147268899 B (2.00 GB) 
Ozone Used              : 4096 B (4 KB) 
Ozone Used %            : 0.00% (Ozone Used/Ozone Capacity) 
Ozone Available         : 2130432000 B (1.98 GB) 
Ozone Available %       : 99.22% (Ozone Available/Ozone capacity) 
Pipeline(s)             : 2 
Container(s)            : 0 
Container Pre-allocated : 0 B (0 B) 
Remaining Allocatable   : 2130432000 B (1.98 GB) 
Free Space To Spare     : 104857600 B (100 MB) 
Reserved                : 214749 B (209.72 KB) 

ID                      : f155baaf-d464-4efc-8d2e-f53da1faf36b 
IP Address              : 172.18.0.9 
Hostname                : ozone-balancer-datanode2-1.ozone-balancer_default 
Filesystem Capacity     : 2147483648 B (2 GB) 
Filesystem Used         : 17051648 B (16.26 MB) 
Filesystem Used %       : 0.79% (Filesystem Used/Filesystem Capacity) 
Filesystem Available    : 2130432000 B (1.98 GB) 
Filesystem Available %  : 99.21% (Filesystem Available/Filesystem Capacity) 
Ozone Capacity          : 2147268899 B (2.00 GB) 
Ozone Used              : 4096 B (4 KB) 
Ozone Used %            : 0.00% (Ozone Used/Ozone Capacity) 
Ozone Available         : 2130432000 B (1.98 GB) 
Ozone Available %       : 99.22% (Ozone Available/Ozone capacity) 
Pipeline(s)             : 2 
Container(s)            : 0 
Container Pre-allocated : 0 B (0 B) 
Remaining Allocatable   : 2130432000 B (1.98 GB) 
Free Space To Spare     : 104857600 B (100 MB) 
Reserved                : 214749 B (209.72 KB) 

ID                      : f601fca9-fab8-44d6-906f-0518991fae3d 
IP Address              : 172.18.0.4 
Hostname                : ozone-balancer-datanode1-1.ozone-balancer_default 
Filesystem Capacity     : 2147483648 B (2 GB) 
Filesystem Used         : 17051648 B (16.26 MB) 
Filesystem Used %       : 0.79% (Filesystem Used/Filesystem Capacity) 
Filesystem Available    : 2130432000 B (1.98 GB) 
Filesystem Available %  : 99.21% (Filesystem Available/Filesystem Capacity) 
Ozone Capacity          : 2147268899 B (2.00 GB) 
Ozone Used              : 4096 B (4 KB) 
Ozone Used %            : 0.00% (Ozone Used/Ozone Capacity) 
Ozone Available         : 2130432000 B (1.98 GB) 
Ozone Available %       : 99.22% (Ozone Available/Ozone capacity) 
Pipeline(s)             : 2 
Container(s)            : 0 
Container Pre-allocated : 0 B (0 B) 
Remaining Allocatable   : 2130432000 B (1.98 GB) 
Free Space To Spare     : 104857600 B (100 MB) 
Reserved                : 214749 B (209.72 KB)

include and exclude:

bash-5.1$ ozone admin containerbalancer assessment \
  --include-datanodes ozone-balancer-datanode6-1.ozone-balancer_default,ozone-balancer-datanode1-1.ozone-balancer_default
CLUSTER BALANCE ASSESSMENT
Key                                                Value
Drift                                              28.4% (max utilization 29.0% - min utilization 0.6%)
Mean Utilization                                   14.8%
Eligible datanodes                                 2 datanodes
Category                                           Small cluster, High imbalance, Medium movement ratio


Source Nodes (over-utilized):
Datanodes above threshold                          1

Top 5:
ozone-balancer-datanode6-1.ozone-balancer_default  29.0% (+14.2% above mean)


Target Nodes (under-utilized):
Datanodes below threshold                          1

Bottom 5:
ozone-balancer-datanode1-1.ozone-balancer_default  0.6% (-14.2% below mean)


Movement Summary:
Total bytes to move                                86.03 MB
Movement ratio                                     2.1% of cluster capacity

bash-5.1$ ozone admin containerbalancer assessment \
  --exclude-datanodes ozone-balancer-datanode6-1.ozone-balancer_default
CLUSTER BALANCE ASSESSMENT
Key                                                Value
Drift                                              28.4% (max utilization 29.0% - min utilization 0.6%)
Mean Utilization                                   12.0%
Eligible datanodes                                 5 datanodes
Category                                           Small cluster, High imbalance, Medium movement ratio


Source Nodes (over-utilized):
Datanodes above threshold                          2

Top 5:
ozone-balancer-datanode4-1.ozone-balancer_default  29.0% (+17.0% above mean)
ozone-balancer-datanode3-1.ozone-balancer_default  29.0% (+17.0% above mean)


Target Nodes (under-utilized):
Datanodes below threshold                          3

Bottom 5:
ozone-balancer-datanode5-1.ozone-balancer_default  0.6% (-11.4% below mean)
ozone-balancer-datanode2-1.ozone-balancer_default  0.6% (-11.4% below mean)
ozone-balancer-datanode1-1.ozone-balancer_default  0.6% (-11.4% below mean)


Movement Summary:
Total bytes to move                                288.39 MB
Movement ratio                                     2.8% of cluster capacity

node limit:

bash-5.1$ ozone admin containerbalancer assessment -n 2 
CLUSTER BALANCE ASSESSMENT
Key                                                Value
Drift                                              28.2% (max utilization 29.0% - min utilization 0.8%)
Mean Utilization                                   14.9%
Eligible datanodes                                 6 datanodes
Category                                           Small cluster, High imbalance, Medium movement ratio


Source Nodes (over-utilized):
Datanodes above threshold                          3

Top 2:
ozone-balancer-datanode6-1.ozone-balancer_default  29.0% (+14.1% above mean)
ozone-balancer-datanode4-1.ozone-balancer_default  29.0% (+14.1% above mean)


Target Nodes (under-utilized):
Datanodes below threshold                          3

Bottom 2:
ozone-balancer-datanode5-1.ozone-balancer_default  0.8% (-14.1% below mean)
ozone-balancer-datanode2-1.ozone-balancer_default  0.8% (-14.1% below mean)


Movement Summary:
Total bytes to move                                252.08 MB
Movement ratio                                     2.1% of cluster capacity

…er imbalance

sravani-revuri · 2026-09-18T05:57:10Z

@ashishkumar50 , @sreejasahithi could you please review these changes?

sreejasahithi

Thanks @sravani-revuri for the PR,
left few comments

sreejasahithi · 2026-09-21T08:08:23Z

+      return;
+    }
+    builder.append(System.lineSeparator())
+        .append(aboveMean ? "Top " + nodeLimit + ":" : "Bottom " + nodeLimit + ":")


The header always uses nodeLimit (default 5), even when fewer nodes are listed (your output shows "Top 5:" with 3 rows). Use the actual list size when list size is < limit

sreejasahithi · 2026-09-21T08:10:41Z

The command applies -t or config default, but the printed report never shows it. For better understanding, add something like:
Threshold: 10.0%
Optionally Upper limit / Lower limit (values already exist on the snapshot)
Without this, users cannot verify why nodes were classified as source/target without doing mental math.

sreejasahithi · 2026-09-21T08:12:48Z

we can also add a --json option.

sreejasahithi · 2026-09-21T08:21:36Z

+    if (nodes.isEmpty()) {
+      return;


When there are no over- or under-utilized datanodes (Datanodes above threshold: 0 / Datanodes below threshold: 0), the command still prints the "Source Nodes" and "Target Nodes" section headers, but no node list appears underneath only a blank line before the next section.

That is correct behaviour, but it can look like output was cut off. Consider adding an explicit line such as (none) when the count is 0, so users can tell the command finished successfully and simply found no matching nodes.

sravani-revuri added 2 commits September 18, 2026 10:04

HDDS-16174. Add container balancer assessment command to report clust…

116bb26

…er imbalance

pmd

fcb3136

sreejasahithi requested review from ashishkumar50 and sreejasahithi September 18, 2026 05:59

sravani-revuri marked this pull request as ready for review September 18, 2026 07:19

Merge remote-tracking branch 'origin/master' into HDDS-16174

cea7beb

sreejasahithi reviewed Sep 21, 2026

View reviewed changes

Navigation Menu

HDDS-16174. Add container balancer assessment command to report cluster imbalance - #11256

HDDS-16174. Add container balancer assessment command to report cluster imbalance#11256
sravani-revuri wants to merge 3 commits into
apache:masterfrom
sravani-revuri:HDDS-16174

sravani-revuri commented Sep 18, 2026

Uh oh!

sravani-revuri commented Sep 18, 2026

Uh oh!

sreejasahithi left a comment

Uh oh!

sreejasahithi Sep 21, 2026

Uh oh!

sreejasahithi Sep 21, 2026

Uh oh!

sreejasahithi Sep 21, 2026

Uh oh!

sreejasahithi Sep 21, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

sravani-revuri commented Sep 18, 2026

What changes were proposed in this pull request?

What is the link to the Apache JIRA

How was this patch tested?

Normal command :

Comparing with datanode usage command:

include and exclude:

node limit:

Uh oh!

sravani-revuri commented Sep 18, 2026

Uh oh!

sreejasahithi left a comment

Choose a reason for hiding this comment

Uh oh!

sreejasahithi Sep 21, 2026

Choose a reason for hiding this comment

Uh oh!

sreejasahithi Sep 21, 2026

Choose a reason for hiding this comment

Uh oh!

sreejasahithi Sep 21, 2026

Choose a reason for hiding this comment

Uh oh!

sreejasahithi Sep 21, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants