DOC-13858 Server File-Based Rebalance for Data Service

modules/ROOT/nav.adoc

@@ -364,6 +364,7 @@ include::cli:partial$cbcli/nav.adoc[]
     **** xref:rest-api:rest-cluster-removenode.adoc[Removing Nodes from Clusters]
 
    *** xref:rest-api:rest-rebalance-overview.adoc[Rebalance]
+    **** xref:rest-api:file-based-data-rebalance.adoc[]
     **** xref:rest-api:rest-retrieve-cluster-rebalance-reason-codes.adoc[Getting Rebalance Reason Codes]
     **** xref:rest-api:rest-cluster-rebalance.adoc[Rebalancing the Cluster]
     **** xref:rest-api:rest-get-rebalance-progress.adoc[Getting Rebalance Progress]

modules/introduction/partials/new-features-81.adoc

@@ -15,7 +15,32 @@ TBD
 
 Couchbase Server 8.5 introduces several new features for the Data Service.
 
-TBD
+==== File-Based Rebalance (FBR)
+[.edition]#{enterprise}#
+
+Couchbase Server Enterprise Edition 8.5 introduces File-Based Rebalance (FBR) for the Data Service.
+FBR accelerates cluster rebalance by copying vBucket storage files between nodes rather than streaming individual documents using Database Change Protocol (DCP) replication.
+It eliminates the serialization and pipeline overhead of DCP backfill for large, disk-resident datasets.
+
+The following changes apply to the Data Service rebalance behavior in 8.5:
+
+* *Enabled by default*: FBR is enabled by default for Couchbase Server Enterprise Edition, both for self-managed deployments and Couchbase Capella.
+If you choose to, you can turn it off globally using the new `dataServiceFileBasedRebalanceEnabled` setting.
+You can also override the global setting for specific buckets using the new `dataServiceRebalanceType` setting when creating or editing buckets.
+
+* *Automatic rebalance type selection*: Couchbase Server automatically determines if it can use FBR to move a vBucket.
+If it cannot, it automatically falls back to copying the vBucket using DCP.
+
+* *Separate vBucket move concurrency for FBR*: A new setting, `dataServiceFileBasedRebalanceMovesPerNode`, controls the maximum number of concurrent file-based vBucket moves per node.
+This is independent of the existing `rebalanceMovesPerNode` setting, which applies to DCP rebalance.
+
+* *Data transfer rate throttle*: The new `snapshot_download_throttle_bytes` setting lets you limit FBR's data transfer rate between nodes.
+Setting this value too high (or setting it to 0, which makes those transfers unlimited) can increase the latency of KV operations.
+
+NOTE: FBR is a Couchbase Server Enterprise Edition feature.
+Couchbase Server Community Edition continues to use DCP-based rebalance for all vBucket moves.
+
+For more information, see xref:learn:clusters-and-availability/rebalance.adoc#file-based-rebalance[File-Based Rebalance].
 
 
 === Non-Data Services

modules/manage/pages/manage-nodes/add-node-and-rebalance.adoc

@@ -182,10 +182,14 @@ The screen now appears as follows:
 [#servers-screen-with-node-added-after-rebalance]
 image::manage-nodes/twoNodeClusterAfterRebalance.png[,800,align=middle]
 +
-Note that the figure in the *Items* column for node `10.142.181.101` is `31.5 K/31.6 K`, which indicates that 31.5 K items are stored on the node in _active_ vBuckets, and 31.6 K in _replica_ vBuckets.
+The figure in the Items column for node `10.142.181.101` is `31.5 K/31.6 K`.
+This value indicates that there are 31.5 K items stored on the node in active vBuckets and 31.6 K in replica vBuckets.
 The figure for `10.142.181.102` indicates the converse.
 Therefore, replication has successfully distributed the contents of `travel-sample` across both nodes, providing a single replica vBucket for each active vBucket.
 
+NOTE: By default, Couchbase Server Enterprise Edition automatically uses File-Based Rebalance (FBR) to move data for eligible vBuckets during node addition.
+For information, see xref:learn:clusters-and-availability/rebalance.adoc#file-based-rebalance[File-Based Rebalance (FBR)].
+
 [#node-information-within-the-ui]
 ==== Node Information Within the UI
 

modules/manage/pages/manage-settings/general-settings.adoc

@@ -174,24 +174,54 @@ See xref:learn:data/durability.adoc[Durability], for information.
 [#rebalance-settings]
 === Rebalance Settings
 
-_Rebalance_ redistributes data, indexes, event processing, and query processing among available nodes.
+Rebalance redistributes data, indexes, event processing, and query processing among available nodes.
 For an overview, see xref:learn:clusters-and-availability/rebalance.adoc[Rebalance].
-Fully open, the panel appears as follows:
+Expand the *Rebalance Settings* section to see all of the rebalance settings:
 
-image::manage-settings/rebalance-settings.png["The Rebalance Settings panel",548,align=center]
+image::manage-settings/rebalance-settings.png["The Rebalance Settings panel",align=center]
 
-The *Retry rebalance* option allows rebalance to be _retried_, in cases where it has failed.
-Check the checkbox, to enable.
-The specifiable, _maximum number of retries_ must be in the range of 1 to 3, inclusive.
-The specifiable, _maximum number of seconds_ must be in the range of 5 to 3600, inclusive.
+The settings in the section are:
 
-Note that this option should _not_ be enabled if the cluster is managed by _Couchbase Autonomous Operator_, or if custom scripts are already being used to trigger rebalance.
-Note also that no administrative tasks should be attempted when rebalance-retries are pending.
-However, pending rebalance-retries can be cancelled: see xref:manage:manage-nodes/add-node-and-rebalance.adoc#automated-rebalance-failure-handling[Automated Rebalance-Failure Handling], for information.
+Retry rebalance::
+Controls whether Couchbase Server retries failed rebalances.
+This setting is off by default.
+Select *Retry rebalance* to enable it.
+You can choose to have Couchbase Server retry a rebalance between 1 and 3 times.
+You also choose how long it waits between rebalance attempts.
+The minimum is 5 seconds and the maximum is 1 hour (3600 seconds).
 
-The *Max moves per node during rebalance* option establishes the maximum number of concurrent vBucket moves permitted on every individual node.
-The minimum value for the parameter is `1`, the maximum `64`, the default `4`.
-For information, see xref:learn:clusters-and-availability/rebalance.adoc#limiting-concurrent-vbucket-moves[Limiting Concurrent vBucket Moves].
++
+Do not enable this option when Couchbase Autonomous Operator manages your cluster or if you're using custom scripts to trigger rebalances.
+
++
+NOTE: Do not perform administrative tasks when rebalance-retries are pending.
+If necessary, you can cancel pending rebalance retries.
+For more information, see xref:manage:manage-nodes/add-node-and-rebalance.adoc#automated-rebalance-failure-handling[Automated Rebalance-Failure Handling].
+
+[#fbr-throttle]
+File-Based Rebalance Throttle Rate::
+Limits the rate of data transfer for File-Based Rebalance (FBR) on a per node basis.
+This limit applies to the total of incoming and outgoing data per second.
+The default value of `0` lets the node use unlimited bandwidth when transferring data during rebalance backfills.
+For more information, see xref:learn:clusters-and-availability/rebalance.adoc#fbr-throttle[FBR Backfill Network Throttle].
+
+
+Max moves per node during rebalance::
+Sets the maximum number of concurrent vBucket moves permitted on each node during rebalance backfills.
+The *For DCP-based rebalance* field sets the limit for Data Change Protocol (DCP) backfills.
+This is the only backfill type supported by Couchbase Server Community Edition.
+The range for this setting is from `1` to `64`.
+The default is `4`.
+
++
+Couchbase Server Enterprise Edition clusters have an additional field named *For file-based rebalance*.
+It controls the number of vBucket files a node can copy concurrently when performing an FBR.
+The range for this setting is from `1` to `64`.
+The default is `4`.
+For information about FBR, see xref:learn:clusters-and-availability/rebalance.adoc#file-based-rebalance[File-Based Rebalance (FBR)].
+
++
+For information about these settings, see xref:learn:clusters-and-availability/rebalance.adoc#limiting-concurrent-vbucket-moves[Limiting Concurrent vBucket Moves].
 
 [#data-settings]
 === Data Settings
@@ -871,33 +901,38 @@ For additional information, refer to xref:n1ql-rest-settings:index.adoc[].
 [#rebalance-settings-via-rest]
 === Rebalance Settings via REST
 
-By means of the REST API, both _rebalance retries_ and _maximum concurrent moves per node_ can be configured.
+You can change rebalance settings such as rebalance retries and maximum concurrent moves per node using REST API calls.
 
 ==== Rebalance Retries via REST
 
-To obtain the cluster's current settings for _rebalance retries_ by means of the REST API, use the `GET /settings/retryRebalance` HTTP method and URI, as follows:
+To obtain the cluster's current settings for rebalance retries using the REST API, use the `GET /settings/retryRebalance` HTTP method and URI, as follows:
 
 [source,shell]
 ----
-curl -X GET -u Administrator:password \
-http://10.143.192.101:8091/settings/retryRebalance
+curl -v -X GET -u $USERNAME:$PASSWORD \
+     http://node1.example.com:8091/settings/retryRebalance | jq
 ----
 
 If successful, the command returns the following object:
 
 [source,json]
 ----
-{"enabled":true,"afterTimePeriod":200,"maxAttempts":3}
+{
+  "enabled": true,
+  "afterTimePeriod": 300,
+  "maxAttempts": 3
+}
+
 ----
 
-This output shows that rebalance retry is enabled, with `200` seconds required to elapse before a retry is attempted, and a maximum of `3` retries possible.
+This output shows that rebalance retry is enabled, with `300` seconds required to elapse before a retry is attempted, and a maximum of `3` retries possible.
 
 To change the rebalance settings, use the `POST` method with the same URI, specifying appropriate values:
 
 [source,shell]
 ----
-curl -X POST -u Administrator:password \
-http://10.143.192.101:8091/settings/retryRebalance \
+curl -v -X POST -u $USERNAME:$PASSWORD \
+http://node1.example.com:8091/settings/retryRebalance \
 -d enabled=false \
 -d afterTimePeriod=100 \
 -d maxAttempts=2
@@ -910,45 +945,150 @@ If successful, the command returns the following object:
 {"enabled":false,"afterTimePeriod":100,"maxAttempts":2}
 ----
 
-This verifies that rebalance retry has been disabled, the required period between retries changed to `100` seconds, and the maximum number of retries changed to `2`.
+This verifies that rebalance retry has been turned off, the required period between retries changed to `100` seconds, and the maximum number of retries changed to `2`.
+
+For more information about getting and setting the rebalance retry status, see xref:rest-api:rest-configure-rebalance-retry.adoc[Configure Rebalance Retries], xref:rest-api:rest-get-rebalance-retry.adoc[Get Rebalance-Retry Status], and xref:rest-api:rest-cancel-rebalance-retry.adoc[Cancel Rebalance Retries].
+
+[#fbr-settings]
+==== Configure File-Based Rebalance via REST
+
+The Data Service defaults to using File-Based Rebalance (FBR) when possible to transfer vBucket data between nodes.
+For more information, see xref:learn:clusters-and-availability/rebalance.adoc#file-based-rebalance[File-Based Rebalance (FBR)].
+
+You can use the REST API to view the status of FBR in the cluster by making a `GET` request to the `/internalSettings` endpoint to view the `dataServiceFileBasedRebalanceEnabled` setting.
+The following example uses `jq` to filter out all other values:
+
+[source,shell]
+----
+curl -X GET -u $USERNAME:$PASSWORD \
+      http://node1.example.com:8091/internalSettings \
+      | jq '{dataServiceFileBasedRebalanceEnabled}'
+----
+
+The result of running this command returns a JSON object:
+
+[source,json]
+----
+{
+  "dataServiceFileBasedRebalanceEnabled": true
+}
+----
+
+To turn off FBR, make a `POST` request to set `dataServiceFileBasedRebalanceEnabled` to `false`:
+
+[source,bash]
+----
+curl -X POST -u $USERNAME:$PASSWORD \
+     http://node1.example.com:8091/internalSettings \
+     -d dataServiceFileBasedRebalanceEnabled=false
+----
+
+You can also override the FBR use on a per-bucket basis using the bucket-level xref:rest-api:rest-bucket-create.adoc#dataservicerebalancetype[`dataServiceRebalanceType`] setting.
+The following example sets the `travel-sample` bucket to use DCP rebalance instead of FBR.
 
-For more information on getting and setting the rebalance retry status, see xref:rest-api:rest-configure-rebalance-retry.adoc[Configure Rebalance Retries], xref:rest-api:rest-get-rebalance-retry.adoc[Get Rebalance-Retry Status], and xref:rest-api:rest-cancel-rebalance-retry.adoc[Cancel Rebalance Retries].
+[source,bash]
+----
+curl -X POST http://node1.example.com:8091/pools/default/buckets/travel-sample \
+     -u $USERNAME:$PASSWORD \
+     -d dataServiceRebalanceType=preferDcp
+----
 
+[#rest_rebalance_concurrent_moves]
 ==== Maximum Concurrent vBucket Moves via REST
 
-To inspect the current maximum number of concurrent vBucket moves permitted for every node, use the `GET /settings/rebalance` HTTP method and URI, with the `rebalanceMovesPerNode` parameter, as follows:
+Couchbase Server has 2 settings to control the number of vBucket moves a node can take part in concurrently: 1 for data change protocol (DCP) backfills and another for File-Based Rebalance (FBR) backfills.
+
+For more information about these backfill modes, see xref:learn:clusters-and-availability/rebalance.adoc#file-based-rebalance[File-Based Rebalance (FBR)].
+
+To find the current settings, make a `GET` request to the `/settings/rebalance` REST API endpoint:
 
 [source,shell]
 ----
-curl -v -X GET http://10.143.201.101:8091/settings/rebalance \
--u Administrator:password
+curl -v -X GET -u $USERNAME:$PASSWORD \
+     http://node1.example.com:8091/settings/rebalance | jq
 ----
 
-This returns an object, confirming the current setting as being `4` (which is the default value):
+This request returns an object containing the current settings:
 
 [source,json]
 ----
-{"rebalanceMovesPerNode":4}
+{
+  "rebalanceMovesPerNode": 4,
+  "dataServiceFileBasedRebalanceMovesPerNode": 4
+}
+
 ----
 
-To _set_ a new value for the parameter use the `POST` method with the same URI, and with the `rebalanceMovesPerNode` parameter.
-Note that the minimum value is `1`, and the maximum `64`.
+`rebalanceMovesPerNode` is the concurrency limit for DCP backfills and `dataServiceFileBasedRebalanceMovesPerNode` is the limit for FBR.
+Both are set to the default value of `4`.
+
+To set a new values, send a `POST` request to the same endpoint setting the parameters to a new values.
+The valid range for these settings are from `1` to `64`.
 
 [source,shell]
 ----
-curl -v -X POST http://10.143.201.101:8091/settings/rebalance \
--u Administrator:password \
--d rebalanceMovesPerNode=10
+curl -X POST http://node1.example.com:8091/settings/rebalance \
+     -u $USERNAME:$PASSWORD \
+     -d rebalanceMovesPerNode=8 \
+     -d dataServiceFileBasedRebalanceMovesPerNode=6 \
+     | jq
 ----
 
 If successful, the call returns an object confirming the new setting:
 
 [source,json]
 ----
-{"rebalanceMovesPerNode":10}
+{
+  "rebalanceMovesPerNode": 8,
+  "dataServiceFileBasedRebalanceMovesPerNode": 6
+}
+----
+
+For more information about limiting concurrent backfills, see xref:rest-api:rest-limit-rebalance-moves.adoc[Limiting Concurrent vBucket Moves].
+
+[#fbr-throttle-rest-api]
+==== Set FBR Bandwidth Throttle with the REST API
+
+By default, Couchbase Server does not limit the network bandwidth FBR uses when copying vBucket data between nodes.
+These unlimited transfers can increase latency for other Couchbase Server features, such as key-value operations.
+You can limit FBR's network use by setting the `snapshot_download_throttle_bytes` option using the REST API.
+For more information about limiting FBR's network use, see xref:learn:clusters-and-availability/rebalance.adoc#fbr-throttle[FBR Backfill Network Throttle].
+
+To view any existing FBR network throttle setting, make a `GET` request to the `pools/default/settings/memcached/global` REST API endpoint:
+
+[source,bash]
+----
+curl -X GET -u $USERNAME:$PASSWORD \
+     http://node1.example.com:8091/pools/default/settings/memcached/global | jq
+----
+
+If there are no overrides of any storage thread allocations (including `snapshot_download_throttle_bytes`), the command returns an empty JSON object.
+
+[source,json]
+----
+{}
+----
+
+To set a limit, make a `POST` request to the endpoint, setting `snapshot_download_throttle_bytes` to the maximum number of bytes per second FBR should use to transfer files.
+This example limits FBR transfers to 600MiB/s per node.
+
+[source,bash]
+----
+curl -X POST -u $USERNAME:$PASSWORD \
+     http://node1.example.com:8091/pools/default/settings/memcached/global \
+     -d snapshot_download_throttle_bytes=629145600 | jq
+----
+
+If successful, the result returns the current overrides to the settings:
+
+[source,json]
+----
+{
+  "snapshot_download_throttle_bytes": 629145600
+}
 ----
 
-For more information, see the REST reference page xref:rest-api:rest-limit-rebalance-moves.adoc[Limiting Concurrent vBucket Moves].
+See xref:rest-api:rest-reader-writer-thread-config.adoc[] for more information about overriding reader and writer thread settings.
 
 [#xdcr-process-setting-via-rest]
 === XDCR Process Setting via REST