diff --git a/modules/ROOT/nav.adoc b/modules/ROOT/nav.adoc index bb3c9b6ecc..a8b1ac3b74 100644 --- a/modules/ROOT/nav.adoc +++ b/modules/ROOT/nav.adoc @@ -364,6 +364,7 @@ include::cli:partial$cbcli/nav.adoc[] **** xref:rest-api:rest-cluster-removenode.adoc[Removing Nodes from Clusters] *** xref:rest-api:rest-rebalance-overview.adoc[Rebalance] + **** xref:rest-api:file-based-data-rebalance.adoc[] **** xref:rest-api:rest-retrieve-cluster-rebalance-reason-codes.adoc[Getting Rebalance Reason Codes] **** xref:rest-api:rest-cluster-rebalance.adoc[Rebalancing the Cluster] **** xref:rest-api:rest-get-rebalance-progress.adoc[Getting Rebalance Progress] diff --git a/modules/introduction/partials/new-features-81.adoc b/modules/introduction/partials/new-features-81.adoc index f062e7f4a0..058dd178b4 100644 --- a/modules/introduction/partials/new-features-81.adoc +++ b/modules/introduction/partials/new-features-81.adoc @@ -15,7 +15,32 @@ TBD Couchbase Server 8.5 introduces several new features for the Data Service. -TBD +==== File-Based Rebalance (FBR) +[.edition]#{enterprise}# + +Couchbase Server Enterprise Edition 8.5 introduces File-Based Rebalance (FBR) for the Data Service. +FBR accelerates cluster rebalance by copying vBucket storage files between nodes rather than streaming individual documents using Database Change Protocol (DCP) replication. +It eliminates the serialization and pipeline overhead of DCP backfill for large, disk-resident datasets. + +The following changes apply to the Data Service rebalance behavior in 8.5: + +* *Enabled by default*: FBR is enabled by default for Couchbase Server Enterprise Edition, both for self-managed deployments and Couchbase Capella. +If you choose to, you can turn it off globally using the new `dataServiceFileBasedRebalanceEnabled` setting. +You can also override the global setting for specific buckets using the new `dataServiceRebalanceType` setting when creating or editing buckets. + +* *Automatic rebalance type selection*: Couchbase Server automatically determines if it can use FBR to move a vBucket. +If it cannot, it automatically falls back to copying the vBucket using DCP. + +* *Separate vBucket move concurrency for FBR*: A new setting, `dataServiceFileBasedRebalanceMovesPerNode`, controls the maximum number of concurrent file-based vBucket moves per node. +This is independent of the existing `rebalanceMovesPerNode` setting, which applies to DCP rebalance. + +* *Data transfer rate throttle*: The new `snapshot_download_throttle_bytes` setting lets you limit FBR's data transfer rate between nodes. +Setting this value too high (or setting it to 0, which makes those transfers unlimited) can increase the latency of KV operations. + +NOTE: FBR is a Couchbase Server Enterprise Edition feature. +Couchbase Server Community Edition continues to use DCP-based rebalance for all vBucket moves. + +For more information, see xref:learn:clusters-and-availability/rebalance.adoc#file-based-rebalance[File-Based Rebalance]. === Non-Data Services diff --git a/modules/learn/assets/images/clusters-and-availability/rebalance-stages-fbr-1.svg b/modules/learn/assets/images/clusters-and-availability/rebalance-stages-fbr-1.svg new file mode 100644 index 0000000000..306d966e26 --- /dev/null +++ b/modules/learn/assets/images/clusters-and-availability/rebalance-stages-fbr-1.svg @@ -0,0 +1,527 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + 1. Backfill + + + + + + + + 3. Book-keeping + + + + + + + + + + + + + + + + + + + + + + + + + + + Copy the vBucket file snapshots from the origin to the destination node. + + + + + + Copies the items in the source cache via DCP.Transfer is memory-to-memory without writing to disk. + + + + + + Perform various ancillary tasks. + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + Persistence Time + + + Persistence Time + + + + + + + + Backfill Time + + + Backfill Time + + + + + + + + Move Time + + + Move Time + + + + + + + + Start + + + Start + + + + + + + + Finish + + + Finish + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + Persistence Time + + + Persistence Time + + + + + + + + Task Persistence + + + + + + + + + + + + + + + + + + + + + 2. Cache Transfer + + + + \ No newline at end of file diff --git a/modules/learn/assets/images/clusters-and-availability/rebalance-stages-fbr-2.svg b/modules/learn/assets/images/clusters-and-availability/rebalance-stages-fbr-2.svg new file mode 100644 index 0000000000..b15d805bad --- /dev/null +++ b/modules/learn/assets/images/clusters-and-availability/rebalance-stages-fbr-2.svg @@ -0,0 +1,661 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + 1. Backfill + + + + + + + + 3. Book-keeping + + + + + + + + + + + + + + + + + + + + + + + + + + + Move the active vBucket data from the node of origin to the memory of the destination node. + + + + + + Copies the items in the source cache via DCP.Transfer is memory-to-memory without writing to disk. + + + + + + Perform various ancillary tasks. + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + Persistence Time + + + Persistence Time + + + + + + + + Backfill Time + + + Backfill Time + + + + + + + + Move Time + + + Move Time + + + + + + + + 4. Active Takeover + + + + + + + + + + + + + Assign control to the new active vBucket, on the destination node. + + + + + + + + 5. Book-keeping + + + + + + + + + + + + + Perform various ancillary tasks. + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + Takeover Time + + + Takeover Time + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + Persistence Time + + + Persistence Time + + + + + + + + Task Persistence + + + + + + + + + + + + + + + + + + + + + Start + + + Start + + + + + + + + Finish + + + Finish + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + 2. Cache Transfer + + + + \ No newline at end of file diff --git a/modules/learn/assets/source/cbArchDiagrams/clustersAndAvailability/nodes/rebalance/rebalanceStagesFBR.odg b/modules/learn/assets/source/cbArchDiagrams/clustersAndAvailability/nodes/rebalance/rebalanceStagesFBR.odg new file mode 100644 index 0000000000..1ed739c3aa Binary files /dev/null and b/modules/learn/assets/source/cbArchDiagrams/clustersAndAvailability/nodes/rebalance/rebalanceStagesFBR.odg differ diff --git a/modules/learn/pages/clusters-and-availability/rebalance.adoc b/modules/learn/pages/clusters-and-availability/rebalance.adoc index 798acba9d7..f6c8ffd783 100644 --- a/modules/learn/pages/clusters-and-availability/rebalance.adoc +++ b/modules/learn/pages/clusters-and-availability/rebalance.adoc @@ -1,5 +1,6 @@ = Rebalance -:description: pass:q[_Rebalance_ redistributes data, indexes, event processing, and query processing among available nodes.] +:description: pass:q[Rebalance redistributes data, indexes, event processing, and query processing among available nodes.] +:totoro: codename Totoro release :page-aliases: clustersetup:rebalance :page-toclevels: 3 @@ -9,32 +10,33 @@ [#understanding-rebalance] == Understanding Rebalance -When one or more nodes have been _brought into_ a cluster (either by xref:learn:clusters-and-availability/nodes.adoc#node-addition[adding] or xref:learn:clusters-and-availability/nodes.adoc#node-joining[joining]), or have been _taken out_ of a cluster (either through xref:learn:clusters-and-availability/removal.adoc[Removal] or xref:learn:clusters-and-availability/failover.adoc[Failover]), _rebalance_ redistributes data, indexes, event processing, and query processing among available nodes. -The _cluster map_ is correspondingly updated and distributed to clients. -The process occurs while the cluster continues to service requests for data. +When you add 1 or more nodes to a cluster by xref:learn:clusters-and-availability/nodes.adoc#node-addition[adding] or xref:learn:clusters-and-availability/nodes.adoc#node-joining[joining], or remove nodes by xref:learn:clusters-and-availability/removal.adoc[removal] or xref:learn:clusters-and-availability/failover.adoc[failover], you must rebalance the cluster. +Rebalance redistributes data, indexes, event processing, and query processing among available nodes. +Rebalance also updates the cluster map and distributes it to clients. +This process occurs while the cluster continues to service requests for data. -See xref:learn:clusters-and-availability/cluster-manager.adoc[Cluster Manager], for information on the cluster map. +For information about the cluster map, see xref:learn:clusters-and-availability/cluster-manager.adoc[Cluster Manager]. See xref:manage:manage-nodes/node-management-overview.adoc[Manage Nodes and Clusters], for practical examples of using rebalance. [#rebalance-bucket-rank] == Bucket Rank -In Couchbase Server Version 7.6 and later, each bucket on the cluster (Couchbase or Ephemeral) can be assigned a _rank_. +In Couchbase Server Version 7.6 and later, each bucket on the cluster (Couchbase or Ephemeral) can be assigned a rank. The value is an integer from `0` (the default) to `1000`, inclusive. -Buckets with higher ranks are handled by the rebalance process _before_ buckets with lower ranks. -For example, if a cluster hosts four buckets, which are named _A_, _B_, _C_, and _D_; and bucket _A_ is explicitly assigned a rank of `10`, while buckets _B_, _C_, and _D_ are left with the default rank of `0`; when rebalance occurs, the vBuckets for bucket _A_ are addressed first; then, vBuckets for the other buckets are addressed, with the Cluster Manager making determinations as to the appropriate handling-order for those other buckets. +Buckets with higher ranks are handled by the rebalance process before buckets with lower ranks. +For example, if a cluster hosts 4 buckets, which are named A, B, C, and D; and bucket A is explicitly assigned a rank of `10`, while buckets B, C, and D are left with the default rank of `0`; when rebalance occurs, the vBuckets for bucket A are addressed first; then, vBuckets for the other buckets are addressed, with the Cluster Manager making determinations as to the appropriate handling-order for those other buckets. This assignment of `rank` allows a cluster's most mission-critical data to be rebalanced with top priority. -Bucket _rank_ can be established with either the CLI (see xref:cli:cbcli/couchbase-cli-bucket-create.adoc[bucket-create] and xref:cli:cbcli/couchbase-cli-bucket-edit.adoc[bucket-edit]) or the REST API (see xref:rest-api:rest-bucket-create.adoc[Creating and Editing Buckets]). +Bucket rank can be established with either the CLI (see xref:cli:cbcli/couchbase-cli-bucket-create.adoc[bucket-create] and xref:cli:cbcli/couchbase-cli-bucket-edit.adoc[bucket-edit]) or the REST API (see xref:rest-api:rest-bucket-create.adoc[Creating and Editing Buckets]). [#rebalance-stages] == Rebalance Stages -Each rebalance proceeds in sequential _stages_. +Each rebalance proceeds in sequential stages. Each stage corresponds to a Couchbase Service, deployed on the cluster. -Therefore, if all services have been deployed, there are _seven_ stages in all -- one each for the _Data_, _Query_, _Index_, _Search_, _Eventing_, _Backup_, and _Analytics_ services. -When all stages have been completed, the rebalance process itself is complete. +If you have deployed all services, rebalance has a stage for each service: Data, Query, Index, Search, Eventing, Backup, and Analytics. +When all stages finish, the rebalance process is complete. [#rebalancing-the-data-service] == Rebalance and the Data Service @@ -55,8 +57,8 @@ See xref:learn:clusters-and-availability/intra-cluster-replication.adoc[Intra-Cl [#data-service-rebalance-phases] === Data-Service Rebalance Phases -During the Data Service rebalance stage, vBuckets are moved in _phases_. -The phases -- which differ, depending on whether the vBucket is an _active_ or a _replica_ vBucket -- are described below. +During the Data Service rebalance stage, vBuckets are moved in phases. +The phases -- which differ, depending on whether the vBucket is an active or a replica vBucket -- are described below. [#rebalance-phases-for-replica-vbuckets] ==== Rebalance Phases for Replica vBuckets @@ -65,17 +67,22 @@ The phases through which rebalance moves a replica vBucket are shown by the foll image::clusters-and-availability/replicaVbucketMove.png[,640,align=left] -The move has two principal phases. Phase 1 is _Backfill_. Phase 2 is _Book-keeping_. +The move has 2 principal phases. +Phase 1 is Backfill. +Phase 2 is Book-keeping. -Phase 1, _Backfill_, itself consists of two subphases. +Phase 1, Backfill, itself consists of 2 subphases. The first subphase comprises the movement of the replica vBucket data from its node of origin to the memory of the destination node. The second subphase comprises the writing of the replica vBucket data from the memory to the disk of the destination node. -The time required for this second subphase, which only applies to Couchbase Buckets, is termed _Persistence Time_. -The time required for the entire _Backfill_ process, including _Persistence Time_, is termed _Backfill Time_. +The time required for this second subphase, which only applies to Couchbase Buckets, is termed Persistence Time. +The time required for the entire Backfill process, including Persistence Time, is termed Backfill Time. -Phase 2, _Book-keeping_, comprises various ancillary tasks required for move-completion. +Phase 2, Book-keeping, comprises various ancillary tasks required for move-completion. -The total time required for the move is calculated by adding _Backfill Time_ to the time required for Phase 2, _Book-keeping_; and is termed _Move Time_. +The total time required for the move is calculated by adding Backfill Time to the time required for Phase 2, Book-keeping; and is termed Move Time. + +NOTE: File-Based Rebalance (FBR) changes these phases. +For more information, see <<#file-based-rebalance>>. [#rebalance-phases-for-active-vbuckets] ==== Rebalance Phases for Active vBuckets @@ -84,39 +91,176 @@ The phases in which rebalance moves an active vBucket are shown by the following image::clusters-and-availability/activeVbucketMove.png[,780,align=left] -The move has four principal phases. -Phase 1, _Backfill_, and Phase 2, _Book-keeping_, are identical to those required for replica vBuckets; except that the _Book-keeping_ phase includes additional _Persistence Time_. +The move has 4 principal phases. +Phase 1, Backfill, and Phase 2, Book-keeping, are identical to those required for replica vBuckets; except that the Book-keeping phase includes additional Persistence Time. + +Phase 3, Active Takeover, comprises the operations required to establish the relocated vBucket as the new active copy. +The time required for Phase 3 is termed Takeover Time. + +Phase 4, Book-keeping, comprises a final set of ancillary tasks, required for move-completion. + +The total time for the move is termed Move Time. + +NOTE: File-Based Rebalance (FBR) also changes the phases for active vBuckets. +For more information, see <<#file-based-rebalance>>. + +[#file-based-rebalance] +=== File-Based Rebalance (FBR) +[.edition]#{enterprise}# + +File-Based Rebalance (FBR) for the Data Service copies the underlying Couchstore or Magma storage files directly from the source node to the destination node during the backfill phase of a vBucket move. +Copying the files reduces CPU overhead on both nodes and decouples rebalance time from item count, making rebalance time proportional to data size on disk rather than document count. + +FBR only works on Couchstore or Magma buckets. +It does not work on ephemeral buckets. + +FBR is enabled by default. +You can choose to turn it off globally or on a bucket-by-bucket basis. +For more about configuring FBR using the REST API, see xref:manage:manage-settings/general-settings.adoc#fbr-settings[Configure File-Based Rebalance via REST]. + +NOTE: FBR is a Couchbase Server Enterprise Edition feature. +Couchbase Server Community Edition only uses DCP-based rebalance for all vBucket moves. + +[#fbr-vs-dcp] +==== DCP Rebalance vs File-Based Rebalance + +Prior to Couchbase Server {totoro}, the Data Service always rebalanced using Data Change Protocol (DCP) backfills. +A DCP backfill reads each document from the vBucket on the source node. +It then transmits it via the network using the DCP streaming protocol to the destination node. +The destination node then writes the document into the target vBucket. +This approach is reliable, but its overhead is proportional to the number of documents in the dataset. +The DCP backfill has to deserialize, transmit, and re-serialize each document. + +Instead of performing a DCP backfill, FBR moves eligible vBuckets by copying the storage files directly. +Directly copying the vBucket files reduces CPU usage and improves network throughput by avoiding deserializing and reserializing each document. + +FBR adds an additional cache transfer phase beyond DCP's 2-step process for replica vBucket transfers: + +image::clusters-and-availability/rebalance-stages-fbr-1.svg[align=left] + +FBR also adds the cache transfer phase for active vBucket transfers: + +image::clusters-and-availability/rebalance-stages-fbr-2.svg[align=left] + +For more information about rebalance phases, see <<#data-service-rebalance-phases>>. + +The following table summarizes the differences between DCP and FBR. + +[width="100%",cols="25%,36%,39%",options="header"] +|=== +|Aspect |DCP Rebalance |File-Based Rebalance (FBR) + +|Transfer mechanism +|Stream documents through DCP pipeline +|Copy storage files directly over the network + +|Time scales with +|Number of items in the dataset +|Size of data on disk -Phase 3, _Active Takeover_, comprises the operations required to establish the relocated vBucket as the new active copy. -The time required for Phase 3 is termed _Takeover Time_. +|CPU overhead +|Higher — serialization on source, deserialization on destination +|Lower — file copy with no document processing -Phase 4, _Book-keeping_, comprises a final set of ancillary tasks, required for move-completion. +|Best suited for +|Small datasets, storage migration, ephemeral buckets +|Large disk-resident (DGM) datasets, swap rebalance, rebalance-in -The total time for the move is termed _Move Time_. +|Enterprise Edition only +|No — available in all editions +|Yes — EE only + +|Default from {totoro} onward +|Fallback when FBR is not applicable +|Default for all eligible vBucket moves +|=== + +[#fbr-backfill-takeover] +==== FBR Backfill and Takeover Phases + +FBR applies only to the Backfill phase of a vBucket move, as described in xref:learn:clusters-and-availability/rebalance.adoc#data-service-rebalance-phases[Data Service Rebalance Phases]. +The Active Takeover phase always uses DCP, regardless of whether FBR was used for backfill. + +[#fbr-automatic-selection] +==== Automatic Rebalance Method Selection + +The server automatically uses FBR (when enabled) to backfill vBuckets when it can. +If it cannot directly copy the vBucket files, Couchbase Server falls back to using DCP to move the data. +Situations where Couchbase Server cannot directly copy the vBucket files include: + +Storage engine migration between Couchstore and Magma:: +Migrating the storage format requires a full data reload, which is only possible through DCP. + +Eviction policy changes:: +Changing a bucket's eviction policy requires data to be reprocessed during rebalance, which requires DCP. + +Ephemeral buckets:: +Ephemeral buckets store data entirely in memory and have no persistent storage files for FBR to copy. + +[#fbr-performance] +==== Performance + +The primary goal of FBR is to improve rebalance speed for large datasets. +Rebalance time scales proportionally with the amount of data on disk and is independent of item count. +Throughput depends on the available network bandwidth, disk IOPS, and CPU resources on the participating nodes. + +NOTE: Workloads with lower resident ratios (the ratio of data stored on disk vs. in memory) show the greatest benefit from FBR. [#limiting-concurrent-vbucket-moves] === Limiting Concurrent vBucket Moves -Since vBucket moves are highly resource-intensive, Couchbase Server allows the concurrency of such moves to be _limited_: a setting is provided that determines the maximum number of concurrent vBucket moves permitted on any node. -The minimum value for the setting is `1`, the maximum `64`, the default `4`. +Because copying vBuckets via DCP or FBR is resource-intensive, Couchbase Server lets you limit the number of concurrent moves for both types of backfills. +DCP and FBR rebalances have their own concurrency settings. +These settings limit the maximum number of simultaneous vBucket moves allowed on a node (as either the source or destination) at a time. +They apply to both active and replica vBuckets. + +For example, suppose a node is the source for 2 moves that are in the DCP backfill phase. +Also, it's the target for 2 additional DCP backfill moves. +With the default setting of `4`, any additional backfills involving the node (either as a source or destination) have to wait until 1 of its current moves completes its backfill phase. + +Both settings default to `4` and have a minimum value of `1`. +You can set DCP concurrency to a maximum of `64` and FBR to a maximum of `1024`. + +They only have an effect on rebalances in the backfill phase. +For more about rebalance phases, see <<#data-service-rebalance-phases>> and <<#fbr-backfill-takeover>>. + +You can change the concurrency settings using the xref:manage:manage-settings/general-settings.adoc#rebalance-settings[Couchbase Web Console], the xref:manage:manage-settings/general-settings.adoc#rebalance-settings-via-cli[Couchbase CLI], or the xref:manage:manage-settings/general-settings.adoc#rest_rebalance_concurrent_moves[REST API]. + +Increasing the vBucket move concurrency can improve rebalance performance. +However, higher concurrency uses more CPU, memory, disk, and network bandwidth resources. +Conversely, a lower setting can degrade rebalance performance, while reducing the resources used by rebalance. + +Other factors can affect rebalance performance. +Therefore, changing these parameters does not always result in a faster rebalance. +Also consider that higher concurrency settings can degrade the performance of other database processes such the Data Service. + +[#fbr-throttle] +==== FBR Backfill Network Throttle + +FBR copies vBucket snapshots via the network during the rebalance backfill phase. +Directly copying vBucket files is efficient, but it can consume more network bandwidth than DCP backfills. +Depending on available network bandwidth and the size of the vBuckets, these transfers can add latency to other database activities such as key-value retrievals. + +If you see additional latency during rebalances, you can monitor the amount of bandwidth FBR file transfers are consuming using the following metrics: + +* xref:metrics-reference:data-service-metrics.adoc#kv_ep_snapshot_read_bytes[`kv_ep_snapshot_read_bytes`]: the number of bytes a node has read from other node's files for FBR. +* xref:metrics-reference:data-service-metrics.adoc#kv_ep_dcp_cache_transfer_read_bytes[`kv_ep_dcp_cache_transfer_read_bytes`]: the number of bytes a node has read from other nodes caches. +This metric measures the transfers made by other features in addition to FBR. -A _move_ counts toward this restriction only when in the _backfill_ phase, as described above, in xref:learn:clusters-and-availability/rebalance.adoc#data-service-rebalance-phases[Data Service Rebalance Phases]. -The move may be of either an _active_ or a _replica_ vBucket. -A node's participation in the move may be as either a source or a target. +For more information about monitoring metrics, see xref:manage:monitor/set-up-prometheus-for-monitoring.adoc[]. -For example, if a node is at a given time the source for two moves in backfill phase, and is the target for two additional moves in backfill phase, and the setting stands at `4`, the node may participate in the backfill phase of no additional moves, until at least one of its current moves has completed its backfill phase. +If you find that the FBR transfers are consuming too much bandwidth, you can limit the bandwidth FBR can use to transfer vBuckets. -The setting may be established by means of the xref:manage:manage-settings/general-settings.adoc#rebalance-settings[Couchbase Web Console], the xref:manage:manage-settings/general-settings.adoc#rebalance-settings-via-cli[Couchbase CLI], or the xref:manage:manage-settings/general-settings.adoc#rebalance-settings-via-rest[REST API]. +You can set the limit using the *File-Based Rebalance Throttle Rate* setting in the Couchbase Server Web Console's *Settings* page. +For more information, see xref:manage:manage-settings/general-settings.adoc#rebalance-settings[Rebalance Settings]. -A higher setting may improve rebalance performance, at the cost of higher resource consumption; in terms of CPU, memory, disk, and bandwidth. -Conversely, a lower setting may degrade rebalance performance, while freeing up such resources. -Note, however, that rebalance performance can be affected by many additional factors; and that in consequence, changing this parameter may not always have the expected effects. -Note also that a higher setting, due to its additional consumption of resources, may degrade the performance of other systems, including the Data Service. +To change this setting via the REST API, alter the xref:rest-api:rest-reader-writer-thread-config.adoc#fbr-throttle[`snapshot_download_throttle_bytes`] setting in the `/pools/default/settings/memcached/global` REST API endpoint. +For more information, see xref:manage:manage-settings/general-settings.adoc#fbr-throttle-rest-api[Set FBR Bandwidth Throttle with the REST API]. [#rebalance-reporting] === Accessing Rebalance Reports -Couchbase Server creates a _report_ on every rebalance that occurs. +Couchbase Server creates a report on every rebalance that occurs. The report contains a JSON document, which can be inspected in any browser or editor. The document provides summaries of the concluded rebalance activity, as well as details for each of the vBuckets affected: in consequence, the report may be of considerable length. @@ -126,8 +270,8 @@ On conclusion of a rebalance, its report can be accessed in any of the following * By means of the REST API, as described in xref:rest-api:rest-get-cluster-tasks.adoc[Getting Cluster Tasks]. -* By accessing the directory `/opt/couchbase/var/lib/couchbase/logs/rebalance` on _any_ of the cluster nodes. -A rebalance report is maintained here for (up to) the last _five_ rebalances performed. +* By accessing the directory `/opt/couchbase/var/lib/couchbase/logs/rebalance` on any of the cluster nodes. +A rebalance report is maintained here for (up to) the last 5 rebalances performed. Each report is provided as a `*.json` file, whose name indicates the time at which the report was run -- for example, `rebalance_report_2020-03-17T11:10:17Z.json`. A complete account of the report-content is provided in the xref:rebalance-reference:rebalance-reference.adoc[Rebalance Reference]. @@ -139,7 +283,7 @@ Rebalance affects each service differently. The following sections describe how rebalance affects non-Data Services. [#rebalancing-the-index-service] -== Index Service +=== Index Service The Index Service maintains a cluster-wide set of index definitions and metadata, which allows the redistribution of indexes and index replicas during a rebalance. @@ -156,14 +300,14 @@ For more information about the rebalance operation on Index Service, see xref:le The Search Service automatically partitions its indexes across all Search nodes in the cluster, ensuring optimal distribution, following rebalance. -To achieve this, in versions of Couchbase Server prior to 7.1, by default, partitions needing to be newly created were entirely _built_, on their newly assigned nodes. -In 7.1 and later versions, by default, new partitions are instead created by the _transfer_ of partition files from old nodes to new nodes: this significantly enhances performance. -This is an Enterprise-only feature, which requires all Search Service nodes _either_ to be running 7.1 or later; _or_ to be running 7.0.2, with the feature explicitly switched on. +To achieve this, in versions of Couchbase Server prior to 7.1, by default, partitions needing to be newly created were entirely built, on their newly assigned nodes. +In 7.1 and later versions, by default, new partitions are instead created by the transfer of partition files from old nodes to new nodes: this significantly enhances performance. +This is an Enterprise-only feature, which requires all Search Service nodes either to be running 7.1 or later; or to be running 7.0.2, with the feature explicitly switched on. Community Edition clusters that are upgraded to Enterprise Edition 7.1 and later versions thus gain this feature in its default setting. Community Edition clusters that are upgraded to Enterprise Edition 7.0.2 can have this feature switched on, subsequent to upgrade. -During file transfer, should an unresolvable error occur, file transfer is automatically abandoned, and _partition build_ is used instead. +During file transfer, should an unresolvable error occur, file transfer is automatically abandoned, and partition build is used instead. The file-transfer feature can be enabled and disabled by means of the REST API. See xref:fts-rest-manage:index.adoc[Search Manager Options]. @@ -190,26 +334,26 @@ If needed, you can retry these requests on another Query node that is still in t [#rebalancing-the-eventing-service] === Eventing Service -When an Eventing Service node has been added or removed, rebalance causes the mutation (_vBucket_ processing ownership) and timer event processing workload to be redistributed among available Eventing Service nodes. +When an Eventing Service node has been added or removed, rebalance causes the mutation (vBucket processing ownership) and timer event processing workload to be redistributed among available Eventing Service nodes. The Eventing Service continues to process mutations both during and after rebalance. Checkpoint information ensures that no mutations are lost. [#rebalancing-the-analytics-service] === Analytics Service -The Analytics Service uses _shadow data_, which is a copy of all or some of the data maintained by the Data Service. +The Analytics Service uses shadow data, which is a copy of all or some of the data maintained by the Data Service. By default, the shadow data is not replicated; however, it may be partitioned across all cluster nodes that run the Analytics Service. Starting with Couchbase Server 7.1, the shadow data and its partitions may be replicated up to 3 times. Each replica resides on an Analytics node: a given Analytics node can host a replica partition, or the active partition on which replicas are based. -If there are _no_ Analytics replicas, and an Analytics node fails over, the Analytics Service stops working cluster-wide: ingestion of shadow data stops and no Analytics operations can be run. +If there are no Analytics replicas, and an Analytics node fails over, the Analytics Service stops working cluster-wide: ingestion of shadow data stops and no Analytics operations can be run. In this case: * If the Analytics node is recovered, the Analytics Service is resumed and ingestion of shadow data resumes from the point before the node failed over. * If the Analytics node is removed, the Analytics Service becomes active again after rebalance, but ingestion of shadow data must begin again from scratch. -If there _are_ Analytics replicas, and an Analytics node fails over, the Analytics Service continues to work: one of the replicas is promoted to serve the shadow data that was stored on the failed over node. +If there are Analytics replicas, and an Analytics node fails over, the Analytics Service continues to work: 1 of the replicas is promoted to serve the shadow data that was stored on the failed over node. The Analytics Service only needs to rebuild any shadow data that isn't already ingested from the Data Service, depending on the state of the promoted replica. In this case: @@ -218,7 +362,7 @@ In this case: * If the Analytics node is removed, the shadow data is redistributed among the remaining Analytics nodes in the cluster. If no Analytics Service node has been removed or replaced, shadow data is not affected by rebalance. -In consequence of rebalance, the Analytics Service receives an updated _cluster map_, and continues to work with the modified vBucket-topology. +In consequence of rebalance, the Analytics Service receives an updated cluster map, and continues to work with the modified vBucket-topology. [#rebalancing-the-backup-service] === Backup Service @@ -227,12 +371,13 @@ A rebalance causes the scheduler for the Backup Service to stop running. This means that no new backup tasks are triggered until the rebalance has concluded; at which point, the scheduler restarts, and reconstructs the task schedule. Then, the triggering of Backup Service tasks is resumed. -Note that a rebalance has the effect of _restarting_ the Backup Service whenever the service has previously been stopped, due to loss of its _leader_: for information, see xref:learn:services-and-indexes/services/backup-service.adoc#backup-service-architecture[Backup-Service Architecture]. +Rebalance restarts the Backup Service if it stopped due to its leader being lost. +For more information, see xref:learn:services-and-indexes/services/backup-service.adoc#backup-service-architecture[Backup-Service Architecture]. [#rebalance-failure-handling] == Rebalance Failure-Handling -Rebalance failures can optionally be responded to automatically, with up to 3 _retries_. +Rebalance failures can optionally be responded to automatically, with up to 3 retries. The number of seconds required to elapse between retries can also be configured. For information on configuration options, see xref:manage:manage-settings/general-settings.adoc[General Settings]. For information on failure-notifications, and options for cancelling rebalance-retries, see xref:manage:manage-nodes/add-node-and-rebalance.adoc#automated-rebalance-failure-handling[Automated Rebalance Failure Handling]. diff --git a/modules/manage/assets/images/manage-settings/rebalance-settings.png b/modules/manage/assets/images/manage-settings/rebalance-settings.png index 3e6efaaf6e..2ccd2f8f2f 100644 Binary files a/modules/manage/assets/images/manage-settings/rebalance-settings.png and b/modules/manage/assets/images/manage-settings/rebalance-settings.png differ diff --git a/modules/manage/pages/manage-nodes/add-node-and-rebalance.adoc b/modules/manage/pages/manage-nodes/add-node-and-rebalance.adoc index d323773ea5..c976c5e07f 100644 --- a/modules/manage/pages/manage-nodes/add-node-and-rebalance.adoc +++ b/modules/manage/pages/manage-nodes/add-node-and-rebalance.adoc @@ -182,10 +182,14 @@ The screen now appears as follows: [#servers-screen-with-node-added-after-rebalance] image::manage-nodes/twoNodeClusterAfterRebalance.png[,800,align=middle] + -Note that the figure in the *Items* column for node `10.142.181.101` is `31.5 K/31.6 K`, which indicates that 31.5 K items are stored on the node in _active_ vBuckets, and 31.6 K in _replica_ vBuckets. +The figure in the Items column for node `10.142.181.101` is `31.5 K/31.6 K`. +This value indicates that there are 31.5 K items stored on the node in active vBuckets and 31.6 K in replica vBuckets. The figure for `10.142.181.102` indicates the converse. Therefore, replication has successfully distributed the contents of `travel-sample` across both nodes, providing a single replica vBucket for each active vBucket. +NOTE: By default, Couchbase Server Enterprise Edition automatically uses File-Based Rebalance (FBR) to move data for eligible vBuckets during node addition. +For information, see xref:learn:clusters-and-availability/rebalance.adoc#file-based-rebalance[File-Based Rebalance (FBR)]. + [#node-information-within-the-ui] ==== Node Information Within the UI diff --git a/modules/manage/pages/manage-settings/general-settings.adoc b/modules/manage/pages/manage-settings/general-settings.adoc index cd5b2601f4..bd9b02d5aa 100644 --- a/modules/manage/pages/manage-settings/general-settings.adoc +++ b/modules/manage/pages/manage-settings/general-settings.adoc @@ -174,24 +174,54 @@ See xref:learn:data/durability.adoc[Durability], for information. [#rebalance-settings] === Rebalance Settings -_Rebalance_ redistributes data, indexes, event processing, and query processing among available nodes. +Rebalance redistributes data, indexes, event processing, and query processing among available nodes. For an overview, see xref:learn:clusters-and-availability/rebalance.adoc[Rebalance]. -Fully open, the panel appears as follows: +Expand the *Rebalance Settings* section to see all of the rebalance settings: -image::manage-settings/rebalance-settings.png["The Rebalance Settings panel",548,align=center] +image::manage-settings/rebalance-settings.png["The Rebalance Settings panel",align=center] -The *Retry rebalance* option allows rebalance to be _retried_, in cases where it has failed. -Check the checkbox, to enable. -The specifiable, _maximum number of retries_ must be in the range of 1 to 3, inclusive. -The specifiable, _maximum number of seconds_ must be in the range of 5 to 3600, inclusive. +The settings in the section are: -Note that this option should _not_ be enabled if the cluster is managed by _Couchbase Autonomous Operator_, or if custom scripts are already being used to trigger rebalance. -Note also that no administrative tasks should be attempted when rebalance-retries are pending. -However, pending rebalance-retries can be cancelled: see xref:manage:manage-nodes/add-node-and-rebalance.adoc#automated-rebalance-failure-handling[Automated Rebalance-Failure Handling], for information. +Retry rebalance:: +Controls whether Couchbase Server retries failed rebalances. +This setting is off by default. +Select *Retry rebalance* to enable it. +You can choose to have Couchbase Server retry a rebalance between 1 and 3 times. +You also choose how long it waits between rebalance attempts. +The minimum is 5 seconds and the maximum is 1 hour (3600 seconds). -The *Max moves per node during rebalance* option establishes the maximum number of concurrent vBucket moves permitted on every individual node. -The minimum value for the parameter is `1`, the maximum `64`, the default `4`. -For information, see xref:learn:clusters-and-availability/rebalance.adoc#limiting-concurrent-vbucket-moves[Limiting Concurrent vBucket Moves]. ++ +Do not enable this option when Couchbase Autonomous Operator manages your cluster or if you're using custom scripts to trigger rebalances. + ++ +NOTE: Do not perform administrative tasks when rebalance-retries are pending. +If necessary, you can cancel pending rebalance retries. +For more information, see xref:manage:manage-nodes/add-node-and-rebalance.adoc#automated-rebalance-failure-handling[Automated Rebalance-Failure Handling]. + +[#fbr-throttle] +File-Based Rebalance Throttle Rate:: +Limits the rate of data transfer for File-Based Rebalance (FBR) on a per node basis. +This limit applies to the total of incoming and outgoing data per second. +The default value of `0` lets the node use unlimited bandwidth when transferring data during rebalance backfills. +For more information, see xref:learn:clusters-and-availability/rebalance.adoc#fbr-throttle[FBR Backfill Network Throttle]. + + +Max moves per node during rebalance:: +Sets the maximum number of concurrent vBucket moves permitted on each node during rebalance backfills. +The *For DCP-based rebalance* field sets the limit for Data Change Protocol (DCP) backfills. +This is the only backfill type supported by Couchbase Server Community Edition. +The range for this setting is from `1` to `64`. +The default is `4`. + ++ +Couchbase Server Enterprise Edition clusters have an additional field named *For file-based rebalance*. +It controls the number of vBucket files a node can copy concurrently when performing an FBR. +The range for this setting is from `1` to `64`. +The default is `4`. +For information about FBR, see xref:learn:clusters-and-availability/rebalance.adoc#file-based-rebalance[File-Based Rebalance (FBR)]. + ++ +For information about these settings, see xref:learn:clusters-and-availability/rebalance.adoc#limiting-concurrent-vbucket-moves[Limiting Concurrent vBucket Moves]. [#data-settings] === Data Settings @@ -871,33 +901,38 @@ For additional information, refer to xref:n1ql-rest-settings:index.adoc[]. [#rebalance-settings-via-rest] === Rebalance Settings via REST -By means of the REST API, both _rebalance retries_ and _maximum concurrent moves per node_ can be configured. +You can change rebalance settings such as rebalance retries and maximum concurrent moves per node using REST API calls. ==== Rebalance Retries via REST -To obtain the cluster's current settings for _rebalance retries_ by means of the REST API, use the `GET /settings/retryRebalance` HTTP method and URI, as follows: +To obtain the cluster's current settings for rebalance retries using the REST API, use the `GET /settings/retryRebalance` HTTP method and URI, as follows: [source,shell] ---- -curl -X GET -u Administrator:password \ -http://10.143.192.101:8091/settings/retryRebalance +curl -v -X GET -u $USERNAME:$PASSWORD \ + http://node1.example.com:8091/settings/retryRebalance | jq ---- If successful, the command returns the following object: [source,json] ---- -{"enabled":true,"afterTimePeriod":200,"maxAttempts":3} +{ + "enabled": true, + "afterTimePeriod": 300, + "maxAttempts": 3 +} + ---- -This output shows that rebalance retry is enabled, with `200` seconds required to elapse before a retry is attempted, and a maximum of `3` retries possible. +This output shows that rebalance retry is enabled, with `300` seconds required to elapse before a retry is attempted, and a maximum of `3` retries possible. To change the rebalance settings, use the `POST` method with the same URI, specifying appropriate values: [source,shell] ---- -curl -X POST -u Administrator:password \ -http://10.143.192.101:8091/settings/retryRebalance \ +curl -v -X POST -u $USERNAME:$PASSWORD \ +http://node1.example.com:8091/settings/retryRebalance \ -d enabled=false \ -d afterTimePeriod=100 \ -d maxAttempts=2 @@ -910,45 +945,150 @@ If successful, the command returns the following object: {"enabled":false,"afterTimePeriod":100,"maxAttempts":2} ---- -This verifies that rebalance retry has been disabled, the required period between retries changed to `100` seconds, and the maximum number of retries changed to `2`. +This verifies that rebalance retry has been turned off, the required period between retries changed to `100` seconds, and the maximum number of retries changed to `2`. + +For more information about getting and setting the rebalance retry status, see xref:rest-api:rest-configure-rebalance-retry.adoc[Configure Rebalance Retries], xref:rest-api:rest-get-rebalance-retry.adoc[Get Rebalance-Retry Status], and xref:rest-api:rest-cancel-rebalance-retry.adoc[Cancel Rebalance Retries]. + +[#fbr-settings] +==== Configure File-Based Rebalance via REST + +The Data Service defaults to using File-Based Rebalance (FBR) when possible to transfer vBucket data between nodes. +For more information, see xref:learn:clusters-and-availability/rebalance.adoc#file-based-rebalance[File-Based Rebalance (FBR)]. + +You can use the REST API to view the status of FBR in the cluster by making a `GET` request to the `/internalSettings` endpoint to view the `dataServiceFileBasedRebalanceEnabled` setting. +The following example uses `jq` to filter out all other values: + +[source,shell] +---- +curl -X GET -u $USERNAME:$PASSWORD \ + http://node1.example.com:8091/internalSettings \ + | jq '{dataServiceFileBasedRebalanceEnabled}' +---- + +The result of running this command returns a JSON object: + +[source,json] +---- +{ + "dataServiceFileBasedRebalanceEnabled": true +} +---- + +To turn off FBR, make a `POST` request to set `dataServiceFileBasedRebalanceEnabled` to `false`: + +[source,bash] +---- +curl -X POST -u $USERNAME:$PASSWORD \ + http://node1.example.com:8091/internalSettings \ + -d dataServiceFileBasedRebalanceEnabled=false +---- + +You can also override the FBR use on a per-bucket basis using the bucket-level xref:rest-api:rest-bucket-create.adoc#dataservicerebalancetype[`dataServiceRebalanceType`] setting. +The following example sets the `travel-sample` bucket to use DCP rebalance instead of FBR. -For more information on getting and setting the rebalance retry status, see xref:rest-api:rest-configure-rebalance-retry.adoc[Configure Rebalance Retries], xref:rest-api:rest-get-rebalance-retry.adoc[Get Rebalance-Retry Status], and xref:rest-api:rest-cancel-rebalance-retry.adoc[Cancel Rebalance Retries]. +[source,bash] +---- +curl -X POST http://node1.example.com:8091/pools/default/buckets/travel-sample \ + -u $USERNAME:$PASSWORD \ + -d dataServiceRebalanceType=preferDcp +---- +[#rest_rebalance_concurrent_moves] ==== Maximum Concurrent vBucket Moves via REST -To inspect the current maximum number of concurrent vBucket moves permitted for every node, use the `GET /settings/rebalance` HTTP method and URI, with the `rebalanceMovesPerNode` parameter, as follows: +Couchbase Server has 2 settings to control the number of vBucket moves a node can take part in concurrently: 1 for data change protocol (DCP) backfills and another for File-Based Rebalance (FBR) backfills. + +For more information about these backfill modes, see xref:learn:clusters-and-availability/rebalance.adoc#file-based-rebalance[File-Based Rebalance (FBR)]. + +To find the current settings, make a `GET` request to the `/settings/rebalance` REST API endpoint: [source,shell] ---- -curl -v -X GET http://10.143.201.101:8091/settings/rebalance \ --u Administrator:password +curl -v -X GET -u $USERNAME:$PASSWORD \ + http://node1.example.com:8091/settings/rebalance | jq ---- -This returns an object, confirming the current setting as being `4` (which is the default value): +This request returns an object containing the current settings: [source,json] ---- -{"rebalanceMovesPerNode":4} +{ + "rebalanceMovesPerNode": 4, + "dataServiceFileBasedRebalanceMovesPerNode": 4 +} + ---- -To _set_ a new value for the parameter use the `POST` method with the same URI, and with the `rebalanceMovesPerNode` parameter. -Note that the minimum value is `1`, and the maximum `64`. +`rebalanceMovesPerNode` is the concurrency limit for DCP backfills and `dataServiceFileBasedRebalanceMovesPerNode` is the limit for FBR. +Both are set to the default value of `4`. + +To set a new values, send a `POST` request to the same endpoint setting the parameters to a new values. +The valid range for these settings are from `1` to `64`. [source,shell] ---- -curl -v -X POST http://10.143.201.101:8091/settings/rebalance \ --u Administrator:password \ --d rebalanceMovesPerNode=10 +curl -X POST http://node1.example.com:8091/settings/rebalance \ + -u $USERNAME:$PASSWORD \ + -d rebalanceMovesPerNode=8 \ + -d dataServiceFileBasedRebalanceMovesPerNode=6 \ + | jq ---- If successful, the call returns an object confirming the new setting: [source,json] ---- -{"rebalanceMovesPerNode":10} +{ + "rebalanceMovesPerNode": 8, + "dataServiceFileBasedRebalanceMovesPerNode": 6 +} +---- + +For more information about limiting concurrent backfills, see xref:rest-api:rest-limit-rebalance-moves.adoc[Limiting Concurrent vBucket Moves]. + +[#fbr-throttle-rest-api] +==== Set FBR Bandwidth Throttle with the REST API + +By default, Couchbase Server does not limit the network bandwidth FBR uses when copying vBucket data between nodes. +These unlimited transfers can increase latency for other Couchbase Server features, such as key-value operations. +You can limit FBR's network use by setting the `snapshot_download_throttle_bytes` option using the REST API. +For more information about limiting FBR's network use, see xref:learn:clusters-and-availability/rebalance.adoc#fbr-throttle[FBR Backfill Network Throttle]. + +To view any existing FBR network throttle setting, make a `GET` request to the `pools/default/settings/memcached/global` REST API endpoint: + +[source,bash] +---- +curl -X GET -u $USERNAME:$PASSWORD \ + http://node1.example.com:8091/pools/default/settings/memcached/global | jq +---- + +If there are no overrides of any storage thread allocations (including `snapshot_download_throttle_bytes`), the command returns an empty JSON object. + +[source,json] +---- +{} +---- + +To set a limit, make a `POST` request to the endpoint, setting `snapshot_download_throttle_bytes` to the maximum number of bytes per second FBR should use to transfer files. +This example limits FBR transfers to 600MiB/s per node. + +[source,bash] +---- +curl -X POST -u $USERNAME:$PASSWORD \ + http://node1.example.com:8091/pools/default/settings/memcached/global \ + -d snapshot_download_throttle_bytes=629145600 | jq +---- + +If successful, the result returns the current overrides to the settings: + +[source,json] +---- +{ + "snapshot_download_throttle_bytes": 629145600 +} ---- -For more information, see the REST reference page xref:rest-api:rest-limit-rebalance-moves.adoc[Limiting Concurrent vBucket Moves]. +See xref:rest-api:rest-reader-writer-thread-config.adoc[] for more information about overriding reader and writer thread settings. [#xdcr-process-setting-via-rest] === XDCR Process Setting via REST diff --git a/modules/rest-api/pages/file-based-data-rebalance.adoc b/modules/rest-api/pages/file-based-data-rebalance.adoc new file mode 100644 index 0000000000..3473b75600 --- /dev/null +++ b/modules/rest-api/pages/file-based-data-rebalance.adoc @@ -0,0 +1,191 @@ += Configure File-Based Data Rebalance +:description: pass:q[You can turn Data Service File-Based Rebalance (FBR) on or off using the `/internalSettings` REST API endpoint.] +:page-edition: Enterprise Edition +:page-topic-type: reference +:page-toclevels: 3 + +[abstract] +{description} + +== Description + +The File-Based Rebalance (FBR) option for the Data Service directly copies vBucket data files between nodes during rebalance. +It's enabled by default. +FBR usually improves rebalance speed compared to the Data Change Protocol (DCP) backfill method. + +For more information about FBR and how it compares to DCP rebalance, see xref:learn:clusters-and-availability/rebalance.adoc#file-based-rebalance[File-Based Rebalance]. + + + +== HTTP Methods + +This API endpoint supports the following methods: + +* <> +* <> + +[#get-settings] +== Determine if FBR is Enabled + +You can get the current status of FBR by making a `GET` request to the `/internalSettings` endpoint. +The FBR settings are part of the internal settings, so this API returns all internal settings. + +.List Internal Settings +---- +GET /internalSettings +---- + +=== curl Syntax + +[source,bash] +---- +curl -u ${USER}:${PASSWORD} -X GET \ + http[s]://${HOST}:${PORT}/internalSettings +---- + +.Path Parameters +:required-privileges: get-privs +include::partial$user_pwd_host_port_params.adoc[] + + +[#get-privs] +=== Required Privileges + +You must have at least 1 of the following roles to get the internal settings, including the FBR settings: + +* xref:learn:security/roles.adoc#full-admin[Full Admin] +* xref:learn:security/roles.adoc#cluster-admin[Cluster Admin] +* xref:learn:security/roles.adoc#ro-security-admin[Read-Only Security Admin] +* xref:learn:security/roles.adoc#security-admin[Security Admin] +* xref:learn:security/roles.adoc#local-user-security-admin[Local User Admin] + +=== Responses + +`200 OK`:: +Returns the internal settings. +See examples for an example of the settings. + +`401 Unauthorized`:: +Returned when authentication fails, such as when the password is incorrect. + +`403 Forbidden`:: +Returned if you do not have 1 of the roles listed in <>. + +[#get-settings-example] +=== Examples + +The following example gets the current value of the FBR setting. +It filters the internal settings using the `jq` command to show just the setting that controls whether FBR is enabled. + +[source,bash] +---- +curl -s -u $USERNAME:$PASSWORD \ + -X GET http://node1.example.com:8091///internalSettings \ + | jq '{dataServiceFileBasedRebalanceEnabled}' +---- + +Running the example returns a JSON object containing the current FBR settings, such as the following: + +[source,json] +---- +{ + "dataServiceFileBasedRebalanceEnabled": true, +} +---- + +The key returned by the previous example controls whether FBR is enabled. +When `dataServiceFileBasedRebalanceEnabled` is `true`, eligible vBucket moves use FBR. +When it is false`, all vBucket moves use DCP. + +[#set-settings] +== Turn FBR On or Off + +You can turn FBR on or off by making a `POST` request to `dataServiceFileBasedRebalanceEnabled`. + + +.Change FBR Settings +---- +POST /internalSettings +---- + +=== curl Syntax + +[source,bash] +---- +curl -sS -u ${USER}:${PASSWORD} \ + -X POST http[s]://${HOST}:${PORT}/internalSettings \ + [-d dataServiceFileBasedRebalanceEnabled={true|false}] \ +---- + +=== Path Parameters +:required-privileges: settings-privs +include::partial$user_pwd_host_port_params.adoc[] + +=== Parameters + +`dataServiceFileBasedRebalanceEnabled`:: +Set to `true` (the default) to allow the Data Service to use FBR when rebalancing data in eligible vBuckets. +Set to `false` to force the Data Service to use DCP when rebalancing data. + ++ +NOTE: This setting can be overridden on a per-bucket basis using the bucket's `dataServiceRebalanceType` setting. +For more information, see xref:manage:manage-buckets/edit-bucket.adoc#bucket-rebalance-type[Bucket-Level Rebalance Type]. + +[#settings-privs] +=== Required Privileges + +You must have 1 of the following roles to change the FBR settings: + +* xref:learn:security/roles.adoc#full-admin[Full Admin] +* xref:learn:security/roles.adoc#cluster-admin[Cluster Admin] + +=== Responses + +`200 OK`:: +Updating the value or values succeeded. + +`400 Bad Request`:: +Returned if you specify an invalid value for `dataServiceFileBasedRebalanceEnabled`. + ++ +Also returns a JSON object that describes the error, such as the following: + ++ +[source,json] +---- +{ + "errors": [ + "dataServiceFileBasedRebalanceEnabled - Accepted values are 'true' and 'false'." + ] +} + +---- + +`401 Unauthorized`:: +Returned when authentication fails, such as when the password is incorrect. + +`403 Forbidden`:: +Returned if you do not have the proper roles to call this API. +See <<#settings-privs>>. + + +[#settings-examples] +=== Examples + +[#disable-fbr-example] +.Disable FBR + +The following example prevents the Data Service from using FBR cluster-wide: + +[source,bash] +---- +curl -u $USERNAME:$PASSWORD \ + -X POST http://node1.example.com:8091/internalSettings \ + -d dataServiceFileBasedRebalanceEnabled=false +---- + +=== See Also + +* xref:learn:clusters-and-availability/rebalance.adoc#file-based-rebalance[File-Based Rebalance (FBR)]. +* xref:manage:manage-settings/general-settings.adoc#rebalance-settings[Rebalance Settings]. +* xref:manage:manage-buckets/edit-bucket.adoc#bucket-rebalance-type[Bucket-Level Rebalance Type]. \ No newline at end of file diff --git a/modules/rest-api/pages/rest-bucket-create.adoc b/modules/rest-api/pages/rest-bucket-create.adoc index adfad21272..09141fce6a 100644 --- a/modules/rest-api/pages/rest-bucket-create.adoc +++ b/modules/rest-api/pages/rest-bucket-create.adoc @@ -94,6 +94,7 @@ curl -X POST -u : -d warmupBehavior=[ background | blocking | none ] -d memoryLowWatermark= -d memoryHighWatermark= + -d dataServiceRebalanceType=[ auto | preferFileBased | preferDcp ] -d continuousBackupEnabled=[ true | false] -d continuousBackupLocation= -d continuousBackupCloudStorageCredId= @@ -103,7 +104,7 @@ curl -X POST -u : -d continuousBackupKmCredId= ---- -All parameters are described in the following subsections. +The following subsections describe these parameters. NOTE: The `threadsNumber` parameter, which sets the number of threads for the bucket, has not had any effect since version Couchbase Server 7.0.0. It's deprecated and is no longer listed in the syntax. @@ -146,6 +147,7 @@ All other parameters are optional and have a default value. ** <> ** <> ** <> +** <> ** <> ** <> ** <> @@ -614,6 +616,62 @@ curl -v -X POST http://localhost:8091/pools/default/buckets/testBucket / Success returns `200 OK`, and changes the `rank` of `testBucket` to `200`. +[#dataservicerebalancetype] +=== dataServiceRebalanceType +[.edition]#{enterprise}# + +Controls the rebalance method used for this bucket's vBucket moves during Data Service rebalance. +This setting overrides the cluster-level `dataServiceFileBasedRebalanceEnabled` setting for the bucket you set it on. + +The valid values are: + +* `auto` (default): The server uses File-Based Rebalance (FBR) to transfer data between nodes if it can. +If it cannot, it falls back to using data-change protocol (DCP) to transfer vBuckets. +This is the recommended setting for most workloads. + +* `preferFileBased`: This setting is a synonym for `auto`. +If the server can use FBR to move vBucket files, it does. +Otherwise it uses DCP. + +* `preferDcp`: All vBucket moves for this bucket use DCP, regardless of the cluster-level FBR setting. +Couchbase Server never uses FBR for the bucket when you select this setting. + +You can set this parameter when creating or modifying the bucket. + +For more information about FBR and how the `dataServiceRebalanceType` bucket-level setting interacts with the cluster-level `dataServiceFileBasedRebalanceEnabled` setting, see xref:learn:clusters-and-availability/rebalance.adoc#file-based-rebalance[File-Based Rebalance (FBR)]. + +[#example-dataservicerebalancetype-create] +==== Example: Setting a Rebalance Type During Bucket Creation + +The following creates a bucket named `testBucket` with `preferFileBased` rebalance. + +[source,bash] +---- +curl -X POST http://localhost:8091/pools/default/buckets \ + -u $USERNAME:$PASSWORD \ + -d name=testBucket \ + -d ramQuota=125 \ + -d dataServiceRebalanceType=preferFileBased +---- + +A successful call returns `202 Accepted`. + +[#example-dataservicerebalancetype-edit] +==== Example: Updating the Rebalance Type + +The following updates the rebalance type for `testBucket` to `auto`: + +[source,bash] +---- +curl -v -X POST http://localhost:8091/pools/default/buckets/testBucket \ + -u Administrator:password \ + -d dataServiceRebalanceType=auto +---- + +Success returns `200 OK`. + +For conceptual information on FBR and the bucket-level setting, see xref:learn:clusters-and-availability/rebalance.adoc#file-based-rebalance[File-Based Rebalance (FBR)] and xref:manage:manage-buckets/edit-bucket.adoc#bucket-rebalance-type[Bucket-Level Rebalance Type]. + [#replicanumber] === replicaNumber @@ -1682,9 +1740,9 @@ The name of a credential in the Couchbase Server credential store that grants th ==== Continuous Backup Example -The following example enables continuous backup for the `travel-sample` bucket. +The following example enables continuous backup for the `travel-sample` bucket. It sets the backup interval to 10 minutes. -The backup location is an NFS share mounted at `/shared/continuous-backup` on all nodes: +The backup location is an NFS share mounted at `/shared/continuous-backup` on all nodes: [source,bash] ``` diff --git a/modules/rest-api/pages/rest-get-internal-setting.adoc b/modules/rest-api/pages/rest-get-internal-setting.adoc index 2d01ed7722..1be8f04e2d 100644 --- a/modules/rest-api/pages/rest-get-internal-setting.adoc +++ b/modules/rest-api/pages/rest-get-internal-setting.adoc @@ -1,67 +1,97 @@ = Managing Internal Settings -:description: Couchbase-Server internal settings can be retrieved and modified.. +:description: Couchbase Server lets you retrieve and modify internal settings that control cluster performance. :page-topic-type: reference [abstract] {description} -== HTTP methods and URIs +== HTTP Method and URI +.Get General Internal Settings +[source,url] ---- GET /internalSettings +---- +.Set General Internal Settings +[source] +---- POST /internalSettings +---- +.Get Max Parallel Indexers Setting +[source,url] +---- GET /settings/maxParallelIndexers +---- +.Set Max Parallel Indexers Setting +[source] +---- POST /settings/maxParallelIndexers ---- +[#description] == Description -Couchbase-Server internal settings can be inspected and modified, in order to control cluster-performance. +Couchbase Server lets you inspect and modify internal settings to control cluster performance. -WARNING: The settings detailed on this page are meant for performance tuning. -Rigorous and careful testing of any changes to these settings should be made in a non-production environment that is representative, in load and scale, of production data before implementing in production. -Incorrect settings can cause severe impacts to cluster performance or operation due to resource consumption. -Couchbase makes no recommendations for tuning beyond the default settings. -Any changes to these settings not explicitly recommended by Couchbase Employees are not supported by Couchbase. +[#warning] +[WARNING] +.Use With Care +==== +Couchbase intends the settings on this page for performance tuning. +Always test any changes to these settings in a non-production environment that represents the load and scale of your production data before you implement them in production. +Incorrect settings can severely degrade cluster performance and operation through excessive resource consumption. +Couchbase does not recommend tuning beyond the default settings. +Couchbase does not support any changes to these settings unless a Couchbase employee explicitly recommends them. +==== -[#curl-syntax] -== Curl Syntax +== Getting Internal Settings + +To get the current general internal settings for Couchbase Server, send a GET request to the `/internalSettings` endpoint. + +[#get-curl-syntax] +=== curl Syntax +[source,bash] +---- +curl -X GET http[s]://{host}:{port}/internalSettings + -u $USER:$PASSWORD ---- -curl -u : -X GET \ - http://:8091/internalSettings - curl -u : -X POST \ - http://:8091/internalSettings - [-d =]* +==== Path Parameters +:required-privileges: required-privileges-get-internal +include::partial$user_pwd_host_port_params.adoc[] -curl -u : -X GET \ - http://:8091/internalSettings/maxParallelIndexers +[#required-privileges-get-internal] +=== Required Privileges -curl -u : -X POST \ - http://:8091/internalSettings/maxParallelIndexers - -d globalValue= ----- +Your account must have at least 1 of the following roles to make a GET request to this endpoint: -The `/internalSettings/maxParallelIndexers` URI relates to _View_ index configuration only. -Note that it does _not_ apply to the _GSI_ indexes provided by the Index Service. -When `POST` is used with this URI, the `integer-value` specified by means of the `globalValue` parameter must be between `1` and `1024`. -The default is `4`. +* xref:learn:security/roles.adoc#full-admin[Full Admin] +* xref:learn:security/roles.adoc#cluster-admin[Cluster Admin] +* xref:learn:security/roles.adoc#ro-security-admin[Read-Only Security Admin] +* xref:learn:security/roles.adoc#security-admin[Security Admin] +* xref:learn:security/roles.adoc#local-user-security-admin[Local User Admin] -All commands require either the Full Admin or Cluster Admin role. +=== Responses -[#responses] -== Responses +200 OK:: +The request was successful. +Returns a JSON object containing all current general internal settings. -If successful, `GET` and `POST` commands return `200 OK`; and a JSON object containing all current _general_ internal settings. +401 Unauthorized:: +Authentication failed. -Failure to authenticate returns `401 Unauthorized`. -A URI featuring an incorrect hostname or port returns `404 Object Not Found` and an object containing an error message. -For example: +403 Forbidden:: +The account making the request does not have 1 of the roles listed in <>. +404 Not Found:: +The URI specified an incorrect hostname or port. +Also returns a JSON object containing an error message, such as the following: ++ +[source,json] ---- { "error": "not_found", @@ -69,124 +99,326 @@ For example: } ---- -Incorrect specification of a setting-change returns `400 Bad Request` and an object containing an error message, such as the following: - ----- -{"errors":["Unknown key rebalanceMovesPerNoede"]} ----- - -If a cluster contains any node running a version of Couchbase Server less that 7.1, an attempt to change the `eventLogLimit` produces the following error message: `not supported in mixed version clusters`. - -[#examples] -== Examples +=== Example -The following example returns general internal settings for the cluster. -Note that the output is piped to the https://stedolan.github.io/jq/[jq^] command, to facilitate readability. +The following example gets the general internal settings for the cluster. +It pipes the output to the https://stedolan.github.io/jq/[jq^] command to make it easier to read. +[source,bash] ---- -curl -u Administrator:password -X GET http://10.144.220.101:8091/internalSettings | jq '.' +curl -X GET -u $USERNAME:$PASSWORD \ +http://node1.example.com:8091/internalSettings | jq '.' ---- -If successful, the command returns output such as the following: +If successful, the call returns a JSON object such as the following: +[source,json] ---- { - "enforceLimits": false, "indexAwareRebalanceDisabled": false, "rebalanceIndexWaitingDisabled": false, "rebalanceIndexPausingDisabled": false, "rebalanceIgnoreViewCompactions": false, - "rebalanceMovesPerNode": 4, + "internalTlsMinVersion": "tlsv1.3", "rebalanceMovesBeforeCompaction": 64, "maxParallelIndexers": 4, "maxParallelReplicaIndexers": 2, "maxBucketCount": 30, + "magmaMinMemoryQuota": 100, "eventLogsLimit": 10000, "gotraceback": "single", - "indexAutoFailoverDisabled": true, + "failoverBulkJanitorFactor": 1, "certUseSha1": false, - "httpNodeAddition": false + "clientCertNotAfterDurationSeconds": 71193600, + "httpNodeAddition": false, + "appTelemetryMinScrapeIntervalSeconds": 60, + "appTelemetryMaxScrapeIntervalSeconds": 600, + "argon2idTime": 3, + "argon2idMem": 524288, + "pbkdf2HmacSha512Iterations": 10000, + "scramShaIterations": 15000, + "useRelativeWebRedirects": true, + "maxDocsSkip": 1000, + "maxDocsLimit": 1000, + "includeUsernameInUICookie": false, + "dataServiceFileBasedRebalanceEnabled": true } ---- -To change a setting, use the `POST` method, specifying the setting as an argument, and specifying its new value. -For example: +== Setting Internal Settings ----- -curl -u Administrator:password -X POST http://10.144.210.101:8091/internalSettings -d rebalanceMovesPerNode=4 ----- +To change a general internal setting, send a `POST` to the `/internalSettings` endpoint, specifying the setting as an argument along with its new value. -If successful, the command returns `200 OK`, and an empty array. -To check the result, use the `GET` method again: +[#curl-syntax] +=== curl Syntax +[source,bash] ---- -curl -u Administrator:password -X GET http://10.144.210.101:8091/internalSettings | jq '.' | grep rebalanceMovesPer +curl -X POST http[s]://{host}:{port}/internalSettings + -u $USER:$PASSWORD + [-d eventLogsLimit=] + [-d dataServiceFileBasedRebalanceEnabled=] + [-d =] ---- -This returns the following output: +==== Path Parameters +:required-privileges: required-privileges-post-internal +include::partial$user_pwd_host_port_params.adoc[] ----- - "rebalanceMovesPerNode": 4, ----- +==== Parameters -This confirms that the change was successful. -To inspect the current setting for `maxParallelIndexers`, enter the following: +`eventLogsLimit`:: +(optional) Sets the maximum number of entries Couchbase Server retains in the xref:learn:clusters-and-availability/system-events.adoc[system event log]. +Valid values are integers from `3000` to `20000`. ----- -curl -u Administrator:password -X GET http://10.144.210.101:8091/settings/maxParallelIndexers ----- ++ +You can only change this value when every node in the cluster runs Couchbase Server 7.1 or higher. +If a cluster contains any node running a version of Couchbase Server less than 7.1, an attempt to change `eventLogsLimit` produces the following error message: `not supported in mixed version clusters`. -If successful, this returns output such as the following: ----- -{"globalValue":4,"nodes":{"ns_1@10.144.210.101":4}} ----- +`dataServiceFileBasedRebalanceEnabled`:: +(optional) Controls whether the data service's rebalance can use file-based backfills. +This value defaults to `true`. +For more information, see xref:learn:clusters-and-availability/rebalance.adoc#file-based-rebalance[File-Based Rebalance (FBR)]. -To change the value, specify the new value by means of the `globalValue` argument: +``:: +(optional) The name of any general internal setting, set to its new value. +See the <<#warning,warning at the top of this page>> before changing any settings in this REST API endpoint. ----- -curl -u Administrator:password -X POST http://10.144.210.101:8091/settings/maxParallelIndexers -d globalValue=3 ----- -The output displays that the value has been successfully changed: +[#required-privileges-post-internal] +=== Required Privileges + +Your account must have at least 1 of the following roles to make a POST request to this endpoint: + +* xref:learn:security/roles.adoc#full-admin[Full Admin] +* xref:learn:security/roles.adoc#cluster-admin[Cluster Admin] +=== Responses + +200 OK:: +The request was successful. +Returns an empty array. + +400 Bad Request:: +The request specified an unknown setting or an invalid value. +Also returns a JSON object containing an error message, such as the following: ++ +[source,json] ---- -{"globalValue":3,"nodes":{"ns_1@10.144.210.101":3}} +{"errors":["Unknown key rebalanceMovesPerNoede"]} ---- -Note that the value is thus established cluster-wide. +401 Unauthorized:: +Authentication failed. + +403 Forbidden:: +The account making the request does not have 1 of the roles listed in <>. + +404 Not Found:: +The URI specified an incorrect hostname or port. + +=== Examples -To change the current size for the _system log_ (which is described in xref:learn:clusters-and-availability/system-events.adoc[System Events]), use the `eventLogsLimit` argument. -The value should be an integer, specifying the number of logs. -The smallest acceptable value is `3000`, and the highest `20000`. -Note that this value can only be changed when every node in the cluster is running Couchbase Server Version 7.1 or higher. +The following example changes the size of the system log to 15,000 entries: + +[source,bash] ---- -curl -u Administrator:password -X POST http://10.144.220.101:8091/internalSettings -d eventLogsLimit=15000 +curl -X POST -u $USERNAME:$PASSWORD \ +http://node1.example.com:8091/internalSettings \ +-d eventLogsLimit=15000 ---- If successful, the call returns an empty array. -The modified value can be checked by means of the `GET` method, which returns output such as the following: +Use the `GET` method to verify the modified value, which returns output such as the following: +[source,json] ---- { - "enforceLimits": false, "indexAwareRebalanceDisabled": false, "rebalanceIndexWaitingDisabled": false, "rebalanceIndexPausingDisabled": false, "rebalanceIgnoreViewCompactions": false, - "rebalanceMovesPerNode": 4, + "internalTlsMinVersion": "tlsv1.3", "rebalanceMovesBeforeCompaction": 64, "maxParallelIndexers": 4, "maxParallelReplicaIndexers": 2, "maxBucketCount": 30, + "magmaMinMemoryQuota": 100, "eventLogsLimit": 15000, "gotraceback": "single", - "indexAutoFailoverDisabled": true, + "failoverBulkJanitorFactor": 1, "certUseSha1": false, - "httpNodeAddition": false + "clientCertNotAfterDurationSeconds": 71193600, + "httpNodeAddition": false, + "appTelemetryMinScrapeIntervalSeconds": 60, + "appTelemetryMaxScrapeIntervalSeconds": 600, + "argon2idTime": 3, + "argon2idMem": 524288, + "pbkdf2HmacSha512Iterations": 10000, + "scramShaIterations": 15000, + "useRelativeWebRedirects": true, + "maxDocsSkip": 1000, + "maxDocsLimit": 1000, + "includeUsernameInUICookie": false, + "dataServiceFileBasedRebalanceEnabled": true } ---- -This confirms that the current size of the `eventLogsLimit` is now 15k. +The output confirms that `eventLogsLimit` is now `15000`. + +== Getting the Max Parallel Indexers Setting + +To get the current setting for the maximum number of parallel indexers, send a GET request to the `/settings/maxParallelIndexers` endpoint. + +This setting applies only to View index configuration. +It does not apply to the GSI indexes that the Index Service provides. + +[#get-mpi-curl-syntax] +=== curl Syntax + +[source,bash] +---- +curl -X GET http[s]://{host}:{port}/settings/maxParallelIndexers + -u $USER:$PASSWORD +---- + +==== Path Parameters +:required-privileges: required-privileges-get-mpi +include::partial$user_pwd_host_port_params.adoc[] + +[#required-privileges-get-mpi] +=== Required Privileges + +Your account must have at least 1 of the following roles to make a GET request to this endpoint: + +* xref:learn:security/roles.adoc#full-admin[Full Admin] +* xref:learn:security/roles.adoc#backup-full-admin[Backup Full Admin] +* xref:learn:security/roles.adoc#bucket-admin[Bucket Admin] +* xref:learn:security/roles.adoc#cluster-admin[Cluster Admin] +* xref:learn:security/roles.adoc#eventing-full-admin[Eventing Full Admin] +* xref:learn:security/roles.adoc#query-list-index[Query List Index] +* xref:learn:security/roles.adoc#query-manage-index[Query Manage Index] +* xref:learn:security/roles.adoc#query-system-catalog[Query System Catalog] +* xref:learn:security/roles.adoc#xdcr-admin[XDCR Admin] +* xref:learn:security/roles.adoc#read-only-admin[Read-Only Admin] +* xref:learn:security/roles.adoc#ro-security-admin[Read-Only Security Admin] +* xref:learn:security/roles.adoc#security-admin[Security Admin] +* xref:learn:security/roles.adoc#external-user-security-admin[External User Admin] +* xref:learn:security/roles.adoc#local-user-security-admin[Local User Admin] +* xref:learn:security/roles.adoc#views-admin[Views Admin (deprecated)] + + +=== Responses + +200 OK:: +The request was successful. +Returns a JSON object containing the current global value and the value for each node. + +401 Unauthorized:: +Authentication failed. + +403 Forbidden:: +The account making the request does not have 1 of the roles listed in <>. + +404 Not Found:: +The URI specified an incorrect hostname or port. + +=== Example + +The following example gets the current setting for `maxParallelIndexers`: + +[source,bash] +---- +curl -X GET -u $USERNAME:$PASSWORD \ +http://node1.example.com:8091/settings/maxParallelIndexers +---- + +If successful, the call returns a JSON object such as the following: + +[source,json] +---- +{"globalValue":4,"nodes":{"ns_1@node1.example.com":4}} +---- + +== Setting the Max Parallel Indexers Setting + +To change the maximum number of parallel indexers, send a `POST` to the `/settings/maxParallelIndexers` endpoint, specifying the new value with the `globalValue` parameter. + +[#set-mpi-curl-syntax] +=== curl Syntax + +[source,bash] +---- +curl -X POST http[s]://{host}:{port}/settings/maxParallelIndexers + -u $COUCHBASE_USER:$PASSWORD + -d globalValue= +---- + +==== Path Parameters +:required-privileges: required-privileges-post-mpi +include::partial$user_pwd_host_port_params.adoc[] + +==== Parameters + +`globalValue`:: +(required) Sets the maximum number of parallel indexers cluster-wide. + ++ +Valid values are integers from `1` to `1024`. +The default value is `4`. + +[#required-privileges-post-mpi] +=== Required Privileges + +Your account must have at least 1 of the following roles to make a POST request to this endpoint: + +* xref:learn:security/roles.adoc#full-admin[Full Admin] +* xref:learn:security/roles.adoc#backup-full-admin[Backup Full Admin] +* xref:learn:security/roles.adoc#cluster-admin[Cluster Admin] + + +=== Responses + +200 OK:: +The request was successful. +Returns a JSON object containing the new global value and the value for each node. + +400 Bad Request:: +The request specified a `globalValue` outside the valid range. + +401 Unauthorized:: +Authentication failed. + +403 Forbidden:: +The account making the request does not have 1 of the roles listed in <>. + +404 Not Found:: +The URI specified an incorrect hostname or port. + +=== Example + +The following example sets `maxParallelIndexers` to `3`: + +[source,bash] +---- +curl -X POST -u $USERNAME:$PASSWORD \ +http://node1.example.com:8091/settings/maxParallelIndexers \ +-d globalValue=3 +---- + +If successful, the call returns a JSON object confirming the new value cluster-wide: + +[source,json] +---- +{"globalValue":3,"nodes":{"ns_1@node1.example.com":3}} +---- + +[#see-also] +== See Also + +* xref:rest-api:rest-limit-rebalance-moves.adoc[Limiting Concurrent vBucket Moves] +* xref:learn:clusters-and-availability/system-events.adoc[System Events] +* xref:manage:manage-settings/manage-settings.adoc[Manage Settings] diff --git a/modules/rest-api/pages/rest-limit-rebalance-moves.adoc b/modules/rest-api/pages/rest-limit-rebalance-moves.adoc index 7c14c94218..73b321c8e8 100644 --- a/modules/rest-api/pages/rest-limit-rebalance-moves.adoc +++ b/modules/rest-api/pages/rest-limit-rebalance-moves.adoc @@ -1,86 +1,206 @@ = Limiting Concurrent vBucket Moves -:description: Couchbase Server places a limit on the number of concurrent vBucket moves that can occur during rebalance, applying the limit to every node in the cluster. +:description: You can set a limit on the number of vBucket moves a node can take part in concurrently during rebalance. :page-topic-type: reference [abstract] {description} -This limit can be inspected and reset by means of the REST API. +Increasing the limit can reduce the duration of a rebalance. +However, it can also affect other database operations such as KV operations. -== HTTP methods and URI +== HTTP Method and URI +.Get vBucket Move Limit +[source,url] ---- GET /settings/rebalance +---- +.Set vBucket Move Limit +[source] +---- POST /settings/rebalance ---- [#description] == Description -These respectively retrieve and establish the setting whereby the number of concurrent vBucket moves that any node can handle during rebalance is limited. -See xref:learn:clusters-and-availability/rebalance.adoc#limiting-vbucket-moves[Limiting vBucket Moves], for detailed information. +Couchbase Server limits the number of vBucket moves a node can take part in concurrently during rebalance. +This endpoint lets you inspect and change that limit. +For more information, see xref:learn:clusters-and-availability/rebalance.adoc#limiting-concurrent-vbucket-moves[Limiting Concurrent vBucket Moves]. -[#curl-syntax] -== Curl Syntax +== Getting the Current vBucket Move Limit ----- -curl -X GET -u : - http://:/settings/rebalance +To get the current vBucket move limit for Couchbase Server, send a GET request to the `/settings/rebalance` endpoint. + +[#get-curl-syntax] +=== curl Syntax -curl -X POST -u : - http://:/settings/rebalance - -d rebalanceMovesPerNode= +[source,bash] ---- +curl -X GET http[s]://{host}:{port}/settings/rebalance + -u $USER:$PASSWORD +---- + +==== Path Parameters +:required-privileges: required-privileges-get +include::partial$user_pwd_host_port_params.adoc[] + +[#required-privileges-get] +=== Required Privileges + +Your account must have at least 1 of the following roles to make a GET request to this endpoint: + -The `positive-integer` used as the value for the `rebalanceMovesPerNode` parameter must be between `1` and `64` inclusive, with `4` being the default. +* xref:learn:security/roles.adoc#full-admin[Full Admin] +* xref:learn:security/roles.adoc#backup-full-admin[Backup Full Admin] +* xref:learn:security/roles.adoc#bucket-admin[Bucket Admin] +* xref:learn:security/roles.adoc#cluster-admin[Cluster Admin] +* xref:learn:security/roles.adoc#eventing-full-admin[Eventing Full Admin] +* xref:learn:security/roles.adoc#xdcr-admin[XDCR Admin] +* xref:learn:security/roles.adoc#read-only-admin[Read-Only Admin] +* xref:learn:security/roles.adoc#ro-security-admin[Read-Only Security Admin] +* xref:learn:security/roles.adoc#security-admin[Security Admin] +* xref:learn:security/roles.adoc#external-user-security-admin[External User Admin] +* xref:learn:security/roles.adoc#local-user-security-admin[Local User Admin] +* xref:learn:security/roles.adoc#views-admin[Views Admin] -[#responses] -== Responses +=== Responses -If successful, `200 OK` is given, and an object containing group-related information is returned. +200 OK:: +The request was successful. +Returns a JSON object containing the current vBucket move limit. -A malformed URI gives `400 Object Not Found`. -Failure to authenticate gives `401 Unauthorized`. +401 Unauthorized:: +Authentication failed. -If an inappropriate value is specified, the following error-notification is returned: `{"errors":{"rebalanceMovesPerNode":"The value must be in range from 1 to 64"}}`. +403 Forbidden:: +The account making the request does not have 1 of the roles listed in <>. -[#examples] -== Examples +=== Example -The following example retrieves the current setting for vBucket move limitation: +The following example gets the current vBucket move limit for Couchbase Server: +[source,bash] ---- -curl -X GET -u Administrator:password \ -http://localhost:8091/settings/rebalance +curl -X GET -u $USERNAME:$PASSWORD \ +http://localhost:8091/settings/rebalance | jq ---- -If successful, the call returns an object such as the following: +If successful, the call returns a JSON object containing the current setting: +[source,json] ---- -{"rebalanceMovesPerNode":4} +{ + "rebalanceMovesPerNode": 4 +} ---- -This indicates that the current setting is `4`, which is the default. +This shows that the current setting is `4`, which is the default. + +== Setting the vBucket Move Limit + +To limit the number of vBucket moves a node can perform concurrently, send a `POST` to the `/settings/rebalance` endpoint. + +[#curl-syntax] +=== curl Syntax + +[source,bash] +---- +curl -X POST http[s]://{host}:{port}/settings/rebalance + -u $USER:$PASSWORD + [-d rebalanceMovesPerNode=] + [-d dataServiceFileBasedRebalanceMovesPerNode=] +---- + +==== Path Parameters +:required-privileges: required-privileges-post +include::partial$user_pwd_host_port_params.adoc[] + +==== Parameters -The following example establishes a new value for the setting: +`rebalanceMovesPerNode`:: +(optional) Sets the maximum number of vBucket moves any node can take part in concurrently during a DCP rebalance. ++ +Valid values are integers from `1` to `64`. +The default value is `4`. + ++ +Increasing this value can reduce the duration of a rebalance. +However, it can also affect other database operations, such as KV operations, so test any changes to this setting before applying them to production systems. + +`dataServiceFileBasedRebalanceMovesPerNode`:: +(optional) Sets the number of simultaneous transfers a node can participate in when performing a File-Based Rebalance (FBR) backfill. +This limit applies to vBucket file transfers where the node is the source or target of the transfer. + ++ +Valid values are integers from `1` to `64`. +The default value is `4`. + ++ +For more information, see xref:learn:clusters-and-availability/rebalance.adoc#file-based-rebalance[File-Based Rebalance (FBR)]. + +[#required-privileges-post] +=== Required Privileges + +Your account must have at least 1 of the following roles to make a POST request to this endpoint: + +* xref:learn:security/roles.adoc#full-admin[Full Admin] +* xref:learn:security/roles.adoc#backup-full-admin[Backup Full Admin] +* xref:learn:security/roles.adoc#cluster-admin[Cluster Admin] + + +=== Responses + +200 OK:: +The request was successful. +Returns a JSON object containing the new vBucket move limit. + +400 Bad Request:: +The request had errors or an invalid value for `rebalanceMovesPerNode` or `dataServiceFileBasedRebalanceMovesPerNode`. +Also returns a JSON object containing an error message. + +401 Unauthorized:: +Authentication failed. + +403 Forbidden:: +The account making the request does not have 1 of the roles listed in <>. + +=== Example + +The following example sets the vBucket move limit to `12`: + +[source,bash] ---- -curl -X POST -u Administrator:password \ +curl -X POST -u $USERNAME:$PASSWORD \ http://localhost:8091/settings/rebalance \ --d rebalanceMovesPerNode=12 +-d rebalanceMovesPerNode=12 | jq ---- -If successful, the call returns an object such as the following: +If successful, the call returns a JSON object confirming the new setting: +[source,json] ---- -{"rebalanceMovesPerNode":12} +{ + "rebalanceMovesPerNode": 12 +} ---- -This confirms that the value has been successfully reset to `12`. +If you specify a value outside the valid range, Couchbase Server returns an error such as the following: + +[source,json] +---- +{ + "errors": { + "rebalanceMovesPerNode": "The value must be in range from 1 to 64" + } +} +---- [#see-also] == See Also -A complete conceptual overview of rebalance is provided in xref:learn:clusters-and-availability/rebalance.adoc[Rebalance]. -Information on inspecting and modifying Couchbase-Server settings is provided in xref:manage:manage-settings/manage-settings.adoc[Manage Settings]. -Various examples of performing rebalance and other node-management operations can be found in xref:manage:manage-nodes/node-management-overview.adoc[Manage Nodes and Clusters]. +* xref:learn:clusters-and-availability/rebalance.adoc[Rebalance] +* xref:rest-api:file-based-data-rebalance.adoc[Configure File-Based Data Rebalance] +* xref:manage:manage-settings/manage-settings.adoc[Manage Settings] +* xref:manage:manage-nodes/node-management-overview.adoc[Manage Nodes and Clusters] diff --git a/modules/rest-api/pages/rest-reader-writer-thread-config.adoc b/modules/rest-api/pages/rest-reader-writer-thread-config.adoc index 797f2b369c..323d064077 100644 --- a/modules/rest-api/pages/rest-reader-writer-thread-config.adoc +++ b/modules/rest-api/pages/rest-reader-writer-thread-config.adoc @@ -30,6 +30,8 @@ For more information, see xref:learn:data/durability.adoc[Durability]. However, allocating too many threads can reduce performance on nodes with limited resources. Test any changes to thread allocation before applying them to production systems. +This endpoint also lets you limit the bandwidth each node uses when downloading vBucket snapshots during a file-based rebalance (FBR) backfill. +For more information, see xref:learn:clusters-and-availability/rebalance.adoc#file-based-rebalance[File-Based Rebalance (FBR)]. == Getting Current Thread Settings @@ -75,11 +77,13 @@ The following example shows how to get the current global thread settings for Co [source,bash] ---- -curl -X GET -u Administrator:password \ -http://localhost:8091/pools/default/settings/memcached/global | jq +curl -X GET -u $USERNAME:$PASSWORD \ +http://node1.example.com:8091/pools/default/settings/memcached/global | jq ---- -If successful, the call returns a JSON object containing any overrides to the default global thread settings: +If successful, the call returns a JSON object containing any overrides to the default global thread settings. +If there are no overrides, the command returns an empty JSON object. +The following example shows the result after 2 settings have been overridden. [source,json] ---- @@ -106,6 +110,7 @@ curl -X POST http[s]://{host}:{port}/pools/default/settings/memcached/global [-d num_auxio_threads=] [-d num_storage_threads=] [-d magma_flusher_thread_percentage=] + [-d snapshot_download_throttle_bytes=] ---- @@ -165,6 +170,22 @@ NOTE: `num_storage_threads` and `magma_flusher_thread_percentage` are advanced s Contact Couchbase Support before making changes to them. Support can help you determine the best settings for your workload and hardware. +[#fbr-throttle] +`snapshot_download_throttle_bytes`:: +(optional) Limits the number of bytes per second each node can read to download vBucket snapshots during a xref:learn:clusters-and-availability/rebalance.adoc#file-based-rebalance[File-Based Rebalance] (FBR) backfill. +The default value is 0, which does not limit the download rate. +The maximum value is the maximum unsigned 64-bit integer value (2^64^ - 1). + ++ +[NOTE] +==== +Setting this value to too high (or leaving it set at the default value of 0, which lets FBR use unlimited bandwidth) can increase the latency of KV operations. +If KV latency during rebalance become an issue, consider adjusting this value to limit the network bandwidth available to FBR. + +You can monitor the xref:metrics-reference:data-service-metrics.adoc#kv_ep_snapshot_read_bytes[`kv_ep_snapshot_read_bytes`] metric to find the bandwidth FBR is consuming when transferring vBucket snapshots. +For more information about monitoring metrics, see xref:manage:monitor/set-up-prometheus-for-monitoring.adoc[]. +==== + [#required-privileges-post] === Required Privileges @@ -192,8 +213,8 @@ To set the numbers of reader, writer, NonIO, and AuxIO threads for Couchbase Ser [source,bash] ---- -curl -X POST -u Administrator:password \ -http://localhost:8091/pools/default/settings/memcached/global \ +curl -X POST -u $USERNAME:$PASSWORD \ +http://node1.example.com:8091/pools/default/settings/memcached/global \ -d num_reader_threads=12 \ -d num_writer_threads=8 \ -d num_nonio_threads=6 \ @@ -217,8 +238,8 @@ The following example increases the Magma storage engine's thread pool to 30 thr [source,bash] ---- -curl -X POST -u Administrator:password \ - http://localhost:8091/pools/default/settings/memcached/global \ +curl -X POST -u $USERNAME:$PASSWORD \ + http://node1.example.com:8091/pools/default/settings/memcached/global \ -d num_storage_threads=30 \ -d magma_flusher_thread_percentage=25 | jq ---- @@ -233,6 +254,30 @@ If successful, the call returns an object confirming your new settings: } ---- +[#fbr-throttle-example] +This example limits FBR file downloads to 600MiB/s. + +[source,bash] +---- +curl -X POST -u $USERNAME:$PASSWORD \ + http://node1.example.com:8091/pools/default/settings/memcached/global \ + -d snapshot_download_throttle_bytes=629145600 | jq +---- + +If successful, the result returns the current settings: + +[source,json] +---- +{ + "num_reader_threads": "balanced", + "num_writer_threads": "balanced", + "num_storage_threads": "default", + "snapshot_download_throttle_bytes": 629145600, + "throttle_enabled": false, + "node_capacity": 18446744073709551615 +} +---- + [#see-also] == See Also diff --git a/modules/rest-api/pages/rest-rebalance-overview.adoc b/modules/rest-api/pages/rest-rebalance-overview.adoc index ebbe8b5c2c..11ef6746b5 100644 --- a/modules/rest-api/pages/rest-rebalance-overview.adoc +++ b/modules/rest-api/pages/rest-rebalance-overview.adoc @@ -1,5 +1,5 @@ = Rebalance -:description: pass:q[When one or more nodes have been brought into or taken out of a cluster, _rebalance_ redistributes data, indexes, event processing, and query processing among available nodes.] +:description: pass:q[When 1 or more nodes have been brought into or taken out of a cluster, rebalance redistributes data, indexes, event processing, and query processing among available nodes.] :page-topic-type: reference [abstract] @@ -8,7 +8,7 @@ Rebalance can be performed and configured by means of the REST API. == APIs in this Section -_Rebalance_ must be performed whenever the number of nodes in a cluster have changed, and whenever buckets have been added or removed. +Rebalance must be performed whenever the number of nodes in a cluster have changed, and whenever buckets have been added or removed. A complete overview is provided in xref:learn:clusters-and-availability/rebalance.adoc[Rebalance]. The REST API for rebalance is as follows: diff --git a/modules/rest-api/partials/rest-rebalance-table.adoc b/modules/rest-api/partials/rest-rebalance-table.adoc index 3bb6fb516e..b284ee1719 100644 --- a/modules/rest-api/partials/rest-rebalance-table.adoc +++ b/modules/rest-api/partials/rest-rebalance-table.adoc @@ -42,4 +42,12 @@ | `/internalSettings` | xref:rest-api:rest-cluster-disable-query.adoc[Disabling Consistent View Query Results on Rebalance] +| `GET` +| `/internalSettings` +| xref:rest-api:file-based-data-rebalance.adoc#get-settings[Get File-Based Rebalance (FBR) Settings] + +| `POST` +| `/internalSettings` +| xref:rest-api:file-based-data-rebalance.adoc#set-settings[Configure FBR] + |=== diff --git a/modules/rest-api/partials/user_pwd_host_port_params.adoc b/modules/rest-api/partials/user_pwd_host_port_params.adoc index 6f52d5f4eb..302ebabaac 100644 --- a/modules/rest-api/partials/user_pwd_host_port_params.adoc +++ b/modules/rest-api/partials/user_pwd_host_port_params.adoc @@ -1,14 +1,14 @@ // Be sure to set the 'required-privileges' attribute before including this partial. // It must be the anchor to the Required Privileges section for the specific endpoint. -`host`:: -Hostname or IP address of a Couchbase Server. +`HOST`:: +Hostname or IP address of a Couchbase Server node. -`port`:: +`PORT`:: Port number for the REST API. Defaults are 8091 for unencrypted and 18901 for encrypted connections. -`$USER`:: +`USER`:: The name of a user who has at least 1 of the roles listed in xref:{required-privileges}[Required Privileges]. -`$PASSWORD`:: +`PASSWORD`:: The password for the user.