diff --git a/content/embeds/rdi-tls-secrets.md b/content/embeds/rdi-tls-secrets.md index 147e76b516..4b6c74c70f 100644 --- a/content/embeds/rdi-tls-secrets.md +++ b/content/embeds/rdi-tls-secrets.md @@ -1,4 +1,4 @@ -When creating secrets for TLS or mTLS, ensure that all certificates and keys are in `PEM` format. The only exception to this is that for PostgreSQL, the private key `SOURCE_DB_KEY` secret must be in `DER` format. If you have a key in `PEM` format, you must convert it to `DER` before creating the `SOURCE_DB_KEY` secret using the command: +When creating secrets for TLS or mTLS, ensure that all certificates and keys are in `PEM` format. The only exception to this is that for PostgreSQL, the source's private key secret must be in `DER` format. If you have a key in `PEM` format, you must convert it to `DER` before creating the secret using the command: ```bash openssl pkcs8 -topk8 -inform PEM -outform DER \ @@ -6,4 +6,4 @@ openssl pkcs8 -topk8 -inform PEM -outform DER \ -out /path/to/myclient.pk8 -nocrypt ``` -This command assumes that the private key is not encrypted. See the [`openssl` documentation](https://docs.openssl.org/master/) to learn how to convert an encrypted private key. \ No newline at end of file +This command assumes that the private key is not encrypted. See the [`openssl` documentation](https://docs.openssl.org/master/) to learn how to convert an encrypted private key. diff --git a/content/embeds/rdi-when-to-use-dec-tree.md b/content/embeds/rdi-when-to-use-dec-tree.md index f5219bef37..10d4f558ae 100644 --- a/content/embeds/rdi-when-to-use-dec-tree.md +++ b/content/embeds/rdi-when-to-use-dec-tree.md @@ -32,21 +32,6 @@ questions: label: "⚠️ Check deployment options to see if RDI is suitable for your needs before proceeding" id: deploymentMismatch sentiment: "indeterminate" - yes: - value: "Yes" - nextQuestion: singleSource - singleSource: - text: | - Are you transferring data from a single source database? - whyAsk: | - RDI is designed to work with a single source database. Multiple sources create conflicting change events. - answers: - no: - value: "No" - outcome: - label: "❌ RDI won't work with multiple source databases" - id: multipleSourcesOrActiveActive - sentiment: "negative" yes: value: "Yes" nextQuestion: consistency diff --git a/content/embeds/rdi-when-to-use.md b/content/embeds/rdi-when-to-use.md index 3251321552..7ef4734d35 100644 --- a/content/embeds/rdi-when-to-use.md +++ b/content/embeds/rdi-when-to-use.md @@ -3,7 +3,7 @@ RDI is a good fit when: - You want your app/micro-services to read from Redis to scale reads at speed. -- You want to transfer data to Redis from a *single* source database. +- You want to transfer data to Redis from one or more source databases. - You must use a slow database as the system of record for the app. - The app must always *write* its data to the slow database. - Your app can tolerate *eventual* consistency of data in the Redis cache. diff --git a/content/integrate/redis-data-integration/architecture/_index.md b/content/integrate/redis-data-integration/architecture/_index.md index f9d09f0453..c49e54701a 100644 --- a/content/integrate/redis-data-integration/architecture/_index.md +++ b/content/integrate/redis-data-integration/architecture/_index.md @@ -20,13 +20,13 @@ weight: 30 ## Overview -RDI implements a [change data capture](https://en.wikipedia.org/wiki/Change_data_capture) (CDC) pattern that tracks changes to the data in a -non-Redis *source* database and makes corresponding changes to a Redis +RDI implements a [change data capture](https://en.wikipedia.org/wiki/Change_data_capture) (CDC) pattern that tracks changes to the data in one or +more non-Redis *source* databases and makes corresponding changes to a Redis *target* database. You can use the target as a cache to improve performance because it will typically handle read queries much faster than the source. To use RDI, you define a *dataset* that specifies which data items -you want to capture from the source and how you want to +you want to capture from each source and how you want to represent them in the target. For example, if the source is a relational database then you specify which table columns you want to capture but you don't need to store them in an equivalent table @@ -35,15 +35,15 @@ representation is most suitable for your app. To convert from the source to the target representation, RDI applies *transformations* to the data after capture. -RDI synchronizes the dataset between the source and target using +RDI synchronizes the dataset between the sources and the target using a *data pipeline* that implements several processing steps in sequence: -1. A *CDC collector* captures changes to the source database. RDI - currently uses an open source collector called +1. A *CDC collector* per source captures changes to the source + databases. RDI currently uses an open source collector called [Debezium](https://debezium.io/) for this step. -1. The collector records the captured changes using +1. Each collector records the captured changes using [Redis streams]({{< relref "/develop/data-types/streams" >}}) in the RDI database. @@ -60,12 +60,12 @@ its state and configuration data and also the change data streams in a Redis dat {{< image filename="images/rdi/ingest/ingest-dataflow.webp" >}} When you first start RDI, the target database is empty and so all -of the data in the source database is essentially "change" data. +of the data in the source databases is essentially "change" data. RDI collects this data in a phase called *initial cache loading*, which can take minutes or hours to finish, depending on the size of the source data. Once the initial cache loading is complete, there is a *snapshot* dataset in the target that will gradually -change when new data gets captured from the source. At this point, +change when new data gets captured from the sources. At this point, RDI automatically enters a second phase called *change streaming*, where changes in the data are captured as they happen. Changes are usually added to the target within a few seconds after capture. @@ -83,7 +83,7 @@ overall state). ## Checkpointing RDI uses Redis streams to store the sequence of change events -captured from the source. The events are then retrieved in order +captured from the sources. The events are then retrieved in order from the streams, processed, and written to the target. The stream processor uses a *checkpoint* mechanism to keep track of the last event in the sequence that it has successfully processed and stored. If the processor fails @@ -96,11 +96,11 @@ face of failures. Sometimes, data records can get added to the streams faster than RDI can process them. This can happen if the target is slowed or disconnected -or simply if the source quickly generates a lot of change data. +or simply if a source quickly generates a lot of change data. If this continues, then the streams will eventually occupy all the available memory. When RDI detects this situation, it applies a *backpressure* mechanism to slow or stop the flow of incoming data. -Change data is held at the source until RDI clears the backlog and has +Change data is held at the sources until RDI clears the backlog and has enough free memory to resume streaming. {{}}The Debezium log sometimes reports that RDI has run out @@ -128,7 +128,7 @@ It includes: and exports them as [Prometheus](https://prometheus.io/) metrics. The *data plane* contains the processes that actually move the data. -It includes the *CDC collector* and the *stream processor* that implement +It includes a *CDC collector* for each source and the *stream processor*, which implement the two phases of the pipeline lifecycle (initial cache loading and change streaming). The *management plane* provides tools that let you interact @@ -148,8 +148,8 @@ The diagram below shows all RDI components and the interactions between them: RDI provides two implementations of the stream processor, *classic* and *Flink*. You select the implementation per pipeline through the [`processors.type`]({{< relref "/integrate/redis-data-integration/data-pipelines/pipeline-config#processors" >}}) -property in `config.yaml`. The default is `classic`, so existing pipelines -keep their behavior unchanged. +property in `config.yaml`. The default is `flink`, so set `classic` explicitly to run a +pipeline on the classic processor. See [Differences between the classic and Flink processors]({{< relref "/integrate/redis-data-integration/architecture/classic-vs-flink" >}}) @@ -164,7 +164,7 @@ deploy RDI. ### RDI on your own VMs -For this deployment, you must provide two VMs. The collector and stream processor +For this deployment, you must provide two VMs. The collectors and stream processor are active on one VM, while on the other they are in standby to provide high availability. The two operators running on both VMs use a leader election algorithm to decide which VM is the active one (the "leader"). diff --git a/content/integrate/redis-data-integration/architecture/classic-vs-flink.md b/content/integrate/redis-data-integration/architecture/classic-vs-flink.md index 3818dfe91b..40dc1fe45d 100644 --- a/content/integrate/redis-data-integration/architecture/classic-vs-flink.md +++ b/content/integrate/redis-data-integration/architecture/classic-vs-flink.md @@ -69,7 +69,7 @@ for VM installations. The two processors share the same `config.yaml` envelope and the same `connections`, `sources`, `targets`, and `jobs` sections. The only differences are inside the `processors:` block, which is selected via -`processors.type` (`classic` or `flink`, default `classic`). Properties +`processors.type` (`classic` or `flink`, default `flink`). Properties that apply to only one implementation are annotated with **Classic processor only.** or **Flink processor only.** in the [pipeline configuration reference]({{< relref "/integrate/redis-data-integration/data-pipelines/pipeline-config#processors" >}}), @@ -102,6 +102,12 @@ for examples and [`redis.lookup`]({{< relref "/integrate/redis-data-integration/reference/data-transformation/lookup" >}}) for the full property list. +The Flink processor also accepts an advanced matcher syntax in a job's `source` section, where +`server_name`, `db`, `schema`, and `table` each accept a list of names, and an entry prefixed +with `regex:` selects all names that match the regular expression. One job can then process +multiple tables, potentially from different sources, databases, or schemas. See +[Job files]({{< relref "/integrate/redis-data-integration/data-pipelines/transform-examples" >}}) for more information. + ## Metrics The two processors expose different Prometheus metric sets and use diff --git a/content/integrate/redis-data-integration/data-pipelines/_index.md b/content/integrate/redis-data-integration/data-pipelines/_index.md index 62f7f255b6..0f364c4e61 100644 --- a/content/integrate/redis-data-integration/data-pipelines/_index.md +++ b/content/integrate/redis-data-integration/data-pipelines/_index.md @@ -28,9 +28,9 @@ deploy them. ## How a pipeline works -An RDI pipeline captures change data records from the source database, and transforms them -into Redis data structures. It writes each of these new structures to a Redis target -database under its own key. +An RDI pipeline captures change data records from one or more source databases, and +transforms them into Redis data structures. It writes each of these new structures to a +Redis target database under its own key. By default, RDI transforms the source data into [hashes]({{< relref "/develop/data-types/hashes" >}}) or @@ -94,6 +94,10 @@ to the snapshot phase. When this is complete, the pipeline continues with CDC as Follow the steps described in the sections below to prepare and run an RDI pipeline. +The following example uses a single-source pipeline. See +[Multiple sources in one pipeline]({{< relref "/integrate/redis-data-integration/data-pipelines/multiple-sources" >}}) +for details on how to capture multiple sources in one pipeline. + ### 1. Prepare the source database Before using the pipeline you must first prepare your source database to use diff --git a/content/integrate/redis-data-integration/data-pipelines/deploy.md b/content/integrate/redis-data-integration/data-pipelines/deploy.md index d126291b45..da1ed7eb93 100644 --- a/content/integrate/redis-data-integration/data-pipelines/deploy.md +++ b/content/integrate/redis-data-integration/data-pipelines/deploy.md @@ -21,33 +21,39 @@ The sections below explain how to deploy a pipeline after you have created the r ## Set secrets -Before you deploy your pipeline, you must set the authentication secrets for the -source and target databases. Each secret has a name that you pass to the +Before you deploy your pipeline, you must set the authentication secrets for the source +and target databases. Every secret belongs to one database: a source, identified by its +name in `config.yaml`, or the target. You name that database with the `--db` option of the [`redis-di set-secret`]({{< relref "/integrate/redis-data-integration/reference/cli/redis-di-set-secret" >}}) -command to set the secret value. -You can then refer to these secrets in the `config.yaml` file using the syntax "`${SECRET_NAME}`" -(the sample -[config.yaml file]({{< relref "/integrate/redis-data-integration/data-pipelines/pipeline-config#example" >}}) -shows these secrets in use). +command. -The table below lists all valid secret names. Note that the -username and password are required for the source and target, but the other -secrets are only relevant for TLS/mTLS connections. +The table below lists the available secret keys. The username and password are required, +while the other keys are only relevant for TLS/mTLS connections. -| Secret name | Description | +| Secret key | Description | | :-- | :-- | -| `SOURCE_DB_USERNAME` | Username for the source database | -| `SOURCE_DB_PASSWORD` | Password for the source database | -| `SOURCE_DB_CACERT` | (For TLS only) Source database CA certificate | -| `SOURCE_DB_CERT` | (For mTLS only) Source database client certificate | -| `SOURCE_DB_KEY` | (For mTLS only) Source database private key | -| `SOURCE_DB_KEY_PASSWORD` | (For mTLS only) Source database private key password | -| `TARGET_DB_USERNAME` | Username for the target database | -| `TARGET_DB_PASSWORD` | Password for the target database | -| `TARGET_DB_CACERT` | (For TLS only) Target database CA certificate | -| `TARGET_DB_CERT` | (For mTLS only) Target database client certificate | -| `TARGET_DB_KEY` | (For mTLS only) Target database private key | -| `TARGET_DB_KEY_PASSWORD` | (For mTLS only) Target database private key password | +| `USERNAME` | Username for the database | +| `PASSWORD` | Password for the database | +| `CACERT` | (For TLS only) CA certificate | +| `CERT` | (For mTLS only) Client certificate | +| `KEY` | (For mTLS only) Private key | +| `KEY_PASSWORD` | (For mTLS only) Private key password | + +You can reference a secret in `config.yaml` using an environment variable that is derived from +the secret key and the database name. The variable name consists of the database name in +uppercase (with each dash replaced by an underscore), followed by `_DB_`, followed by the key. +For example, if you set `PASSWORD` with `--db mysql` the corresponding environment variable +is `MYSQL_DB_PASSWORD`, which the source references as +`${MYSQL_DB_PASSWORD}`. If you set `PASSWORD` with `--db target`, the environment variable is `TARGET_DB_PASSWORD`. The sample +[config.yaml file]({{< relref "/integrate/redis-data-integration/data-pipelines/pipeline-config#example" >}}) +shows these references in use, and +[Multiple sources in one pipeline]({{< relref "/integrate/redis-data-integration/data-pipelines/multiple-sources" >}}) +covers the source naming rules. + +{{< note >}}The scope-prefixed keys `SOURCE_DB_*` and `TARGET_DB_*` are also accepted, and can be +used without specifying `--db`. RDI accepts a `SOURCE_DB_*` key only for a pipeline with exactly +one source. Prefer the per-database keys with `--db`, which work for any pipeline. See +[Existing names are kept after an upgrade]({{< relref "/integrate/redis-data-integration/data-pipelines/multiple-sources#existing-names-are-kept-after-an-upgrade" >}}) for more information.{{< /note >}} {{< note >}} {{< embed-md "rdi-tls-secrets.md" >}} @@ -58,38 +64,38 @@ secrets are only relevant for TLS/mTLS connections. Use [`redis-di set-secret`]({{< relref "/integrate/redis-data-integration/reference/cli/redis-di-set-secret" >}}) to set secrets for any installation type (VM, Kubernetes, or Redis Cloud). -The specific command lines for source secrets are as follows: +The command lines for a source named `mysql` are shown below. If your pipeline has multiple sources, you should run the command once for each source, using the appropriate database name. ```bash # For username and password -redis-di set-secret SOURCE_DB_USERNAME yourUsername -redis-di set-secret SOURCE_DB_PASSWORD yourPassword +redis-di set-secret USERNAME --db mysql yourUsername +redis-di set-secret PASSWORD --db mysql yourPassword # With source TLS, in addition to the above -redis-di set-secret SOURCE_DB_CACERT /path/to/myca.crt +redis-di set-secret CACERT --db mysql /path/to/myca.crt # With source mTLS, in addition to the above -redis-di set-secret SOURCE_DB_CERT /path/to/myclient.crt -redis-di set-secret SOURCE_DB_KEY /path/to/myclient.key -# Use this only if SOURCE_DB_KEY is password-protected -redis-di set-secret SOURCE_DB_KEY_PASSWORD yourKeyPassword +redis-di set-secret CERT --db mysql /path/to/myclient.crt +redis-di set-secret KEY --db mysql /path/to/myclient.key +# Use this only if the private key is password-protected +redis-di set-secret KEY_PASSWORD --db mysql yourKeyPassword ``` The corresponding command lines for target secrets are: ```bash # For username and password -redis-di set-secret TARGET_DB_USERNAME yourUsername -redis-di set-secret TARGET_DB_PASSWORD yourPassword +redis-di set-secret USERNAME --db target yourUsername +redis-di set-secret PASSWORD --db target yourPassword # With target TLS, in addition to the above -redis-di set-secret TARGET_DB_CACERT /path/to/myca.crt +redis-di set-secret CACERT --db target /path/to/myca.crt # With target mTLS, in addition to the above -redis-di set-secret TARGET_DB_CERT /path/to/myclient.crt -redis-di set-secret TARGET_DB_KEY /path/to/myclient.key -# Use this only if TARGET_DB_KEY is password-protected -redis-di set-secret TARGET_DB_KEY_PASSWORD yourKeyPassword +redis-di set-secret CERT --db target /path/to/myclient.crt +redis-di set-secret KEY --db target /path/to/myclient.key +# Use this only if the private key is password-protected +redis-di set-secret KEY_PASSWORD --db target yourKeyPassword ``` By default, `set-secret` waits for the pipeline to apply the change before returning. When you set @@ -103,14 +109,17 @@ never returns secret values, these commands show only the secret keys and whethe the stored values. ```bash -# List all the secrets of a pipeline and whether each one is set +# List all the secrets of a pipeline, with the database each one belongs to redis-di list-secrets +# List only the secrets of one database +redis-di list-secrets --db mysql + # Show a single secret and whether it is set -redis-di describe-secret SOURCE_DB_PASSWORD +redis-di describe-secret PASSWORD --db mysql # Delete a secret (prompts for confirmation unless you add --force) -redis-di delete-secret SOURCE_DB_CACERT +redis-di delete-secret CACERT --db mysql ``` See the reference pages for @@ -135,9 +144,12 @@ kubectl create secret generic \ --from-literal== ``` -Where `` is either `source-db` for source secrets or `target-db` for target secrets. +Where `` is `-db` for the secrets of a source, or `target-db` for target secrets. +The examples below use a source named `mysql`, so its secret is `mysql-db`. -If you use TLS or mTLS for either the source or target databases, you also need to create the `source-db-ssl` and/or `target-db-ssl` K8s secrets that contain the certificates used to establish secure connections. The general pattern of the commands is: +If you use TLS or mTLS for either the source or target databases, you also need to create the +`-db-ssl` and/or `target-db-ssl` K8s secrets that contain the certificates used +to establish secure connections. The general pattern of the commands is: ```bash kubectl create secret generic -ssl \ @@ -149,36 +161,37 @@ The specific command lines for source secrets are as follows: ```bash # Without source TLS -# Create or update source-db secret -kubectl create secret generic source-db --namespace=rdi \ ---from-literal=SOURCE_DB_USERNAME=yourUsername \ ---from-literal=SOURCE_DB_PASSWORD=yourPassword \ +# Create or update mysql-db secret +kubectl create secret generic mysql-db --namespace=rdi \ +--from-literal=MYSQL_DB_USERNAME=yourUsername \ +--from-literal=MYSQL_DB_PASSWORD=yourPassword \ --save-config --dry-run=client -o yaml | kubectl apply -f - # With source TLS -# Create of update source-db secret -kubectl create secret generic source-db --namespace=rdi \ ---from-literal=SOURCE_DB_USERNAME=yourUsername \ ---from-literal=SOURCE_DB_PASSWORD=yourPassword \ ---from-literal=SOURCE_DB_CACERT=/etc/certificates/source_db/ca.crt \ +# Create or update mysql-db secret +kubectl create secret generic mysql-db --namespace=rdi \ +--from-literal=MYSQL_DB_USERNAME=yourUsername \ +--from-literal=MYSQL_DB_PASSWORD=yourPassword \ +--from-literal=MYSQL_DB_CACERT=/etc/certificates/mysql_db/ca.crt \ --save-config --dry-run=client -o yaml | kubectl apply -f - -# Create or update source-db-ssl secret -kubectl create secret generic source-db-ssl --namespace=rdi \ +# Create or update mysql-db-ssl secret +kubectl create secret generic mysql-db-ssl --namespace=rdi \ --from-file=ca.crt=/path/to/myca.crt \ --save-config --dry-run=client -o yaml | kubectl apply -f - # With source mTLS -# Create or update source-db secret -kubectl create secret generic source-db --namespace=rdi \ ---from-literal=SOURCE_DB_USERNAME=yourUsername \ ---from-literal=SOURCE_DB_PASSWORD=yourPassword \ ---from-literal=SOURCE_DB_CACERT=/etc/certificates/source_db/ca.crt \ ---from-literal=SOURCE_DB_CERT=/etc/certificates/source_db/client.crt \ ---from-literal=SOURCE_DB_KEY=/etc/certificates/source_db/client.key \ ---from-literal=SOURCE_DB_KEY_PASSWORD=yourKeyPassword \ # add this only if SOURCE_DB_KEY is password-protected +# Create or update mysql-db secret. Include the MYSQL_DB_KEY_PASSWORD line +# only if the private key is password-protected. +kubectl create secret generic mysql-db --namespace=rdi \ +--from-literal=MYSQL_DB_USERNAME=yourUsername \ +--from-literal=MYSQL_DB_PASSWORD=yourPassword \ +--from-literal=MYSQL_DB_CACERT=/etc/certificates/mysql_db/ca.crt \ +--from-literal=MYSQL_DB_CERT=/etc/certificates/mysql_db/client.crt \ +--from-literal=MYSQL_DB_KEY=/etc/certificates/mysql_db/client.key \ +--from-literal=MYSQL_DB_KEY_PASSWORD=yourKeyPassword \ --save-config --dry-run=client -o yaml | kubectl apply -f - -# Create or update source-db-ssl secret -kubectl create secret generic source-db-ssl --namespace=rdi \ +# Create or update mysql-db-ssl secret +kubectl create secret generic mysql-db-ssl --namespace=rdi \ --from-file=ca.crt=/path/to/myca.crt \ --from-file=client.crt=/path/to/myclient.crt \ --from-file=client.key=/path/to/myclient.key \ @@ -196,7 +209,7 @@ kubectl create secret generic target-db --namespace=rdi \ --save-config --dry-run=client -o yaml | kubectl apply -f - # With target TLS -# Create of update target-db secret +# Create or update target-db secret kubectl create secret generic target-db --namespace=rdi \ --from-literal=TARGET_DB_USERNAME=yourUsername \ --from-literal=TARGET_DB_PASSWORD=yourPassword \ @@ -208,14 +221,15 @@ kubectl create secret generic target-db-ssl --namespace=rdi \ --save-config --dry-run=client -o yaml | kubectl apply -f - # With target mTLS -# Create or update target-db secret +# Create or update target-db secret. Include the TARGET_DB_KEY_PASSWORD line +# only if the private key is password-protected. kubectl create secret generic target-db --namespace=rdi \ --from-literal=TARGET_DB_USERNAME=yourUsername \ --from-literal=TARGET_DB_PASSWORD=yourPassword \ --from-literal=TARGET_DB_CACERT=/etc/certificates/target_db/ca.crt \ --from-literal=TARGET_DB_CERT=/etc/certificates/target_db/client.crt \ --from-literal=TARGET_DB_KEY=/etc/certificates/target_db/client.key \ ---from-literal=TARGET_DB_KEY_PASSWORD=yourKeyPassword \ # add this only if TARGET_DB_KEY is password-protected +--from-literal=TARGET_DB_KEY_PASSWORD=yourKeyPassword \ --save-config --dry-run=client -o yaml | kubectl apply -f - # Create or update target-db-ssl secret kubectl create secret generic target-db-ssl --namespace=rdi \ @@ -225,11 +239,12 @@ kubectl create secret generic target-db-ssl --namespace=rdi \ --save-config --dry-run=client -o yaml | kubectl apply -f - ``` -Note that the certificate paths contained in the secrets `SOURCE_DB_CACERT`, `SOURCE_DB_CERT`, and `SOURCE_DB_KEY` (for the source database) and `TARGET_DB_CACERT`, `TARGET_DB_CERT`, and `TARGET_DB_KEY` (for the target database) are internal to RDI, so you *must* use the values shown in the example above. You should only change the certificate paths when you create the `source-db-ssl` and `target-db-ssl` secrets. +Note that the certificate paths contained in the `CACERT`, `CERT`, and `KEY` secrets are internal to RDI, so you *must* use the values shown in the example above. Each source has its own certificate directory, named after the source (for example, `/etc/certificates/mysql_db/` for a source named `mysql`). You should only change the certificate paths when you create the `-db-ssl` and `target-db-ssl` secrets. -Secrets that you create directly with `kubectl` must also be labeled so that the RDI operator -discovers them as pipeline secrets. Each secret needs the following labels, where the -`app.kubernetes.io/instance` label is the pipeline name (`default` for the default pipeline): +You must also label any secrets that you create directly with `kubectl` so that the RDI operator +discovers them as pipeline secrets. Give each secret the following labels, where the +`app.kubernetes.io/instance` label corresponds to the pipeline name (the name is just +`default` for the default pipeline): | Label | Value | | :-- | :-- | @@ -240,7 +255,7 @@ discovers them as pipeline secrets. Each secret needs the following labels, wher Apply the labels to each secret with [`kubectl label`](https://kubernetes.io/docs/reference/kubectl/generated/kubectl_label/): ```bash -kubectl label secret source-db --namespace=rdi --overwrite \ +kubectl label secret mysql-db --namespace=rdi --overwrite \ app.kubernetes.io/name=pipeline \ app.kubernetes.io/instance=default \ product=rdi @@ -250,7 +265,7 @@ kubectl label secret target-db --namespace=rdi --overwrite \ product=rdi # With source TLS or mTLS -kubectl label secret source-db-ssl --namespace=rdi --overwrite \ +kubectl label secret mysql-db-ssl --namespace=rdi --overwrite \ app.kubernetes.io/name=pipeline \ app.kubernetes.io/instance=default \ product=rdi @@ -331,6 +346,16 @@ redis-di stop redis-di start ``` +To act on a single source instead of the whole pipeline, add `--source`: + +```bash +redis-di stop --source mysql +redis-di start --source mysql +``` + +Note that a source can only run if its parent pipeline is running. See +[Multiple sources in one pipeline]({{< relref "/integrate/redis-data-integration/data-pipelines/multiple-sources" >}}) for more information. + ## Reset a pipeline Use [`redis-di reset`]({{< relref "/integrate/redis-data-integration/reference/cli/redis-di-reset" >}}) @@ -343,6 +368,13 @@ drifted out of sync. redis-di reset ``` +Add `--source` to reset a single source. The other sources keep their data, but the whole +pipeline stops while the reset runs and starts again afterwards: + +```bash +redis-di reset --source mysql +``` + ## Undeploy a pipeline To remove a pipeline, use the @@ -377,8 +409,8 @@ To avoid this, set all the related secrets, or at least all of them except the l become healthy: ```bash -redis-di set-secret SOURCE_DB_USERNAME newUsername --wait=false -redis-di set-secret SOURCE_DB_PASSWORD newPassword +redis-di set-secret USERNAME --db mysql newUsername --wait=false +redis-di set-secret PASSWORD --db mysql newPassword ``` -The same applies to any set of changes that are only valid together. \ No newline at end of file +The same applies to any set of changes that are only valid together. diff --git a/content/integrate/redis-data-integration/data-pipelines/multiple-sources.md b/content/integrate/redis-data-integration/data-pipelines/multiple-sources.md new file mode 100644 index 0000000000..34254ca5a7 --- /dev/null +++ b/content/integrate/redis-data-integration/data-pipelines/multiple-sources.md @@ -0,0 +1,312 @@ +--- +Title: Multiple sources in one pipeline +alwaysopen: false +categories: +- docs +- integrate +- rs +- rdi +description: Learn how to ingest from several source databases into one Redis target. +group: di +linkTitle: Multiple sources +summary: Redis Data Integration keeps Redis in sync with the primary database in near + real time. +type: integration +weight: 4 +--- + +One RDI pipeline can capture changes from several source databases and write them all to the +same Redis target. The sources can be of different database types and each has its +own collector, its own credentials, and its own set of Redis streams to ensure it +is independent of the other sources. + +{{< note >}}You must use RDI API v2 to manage a pipeline with several sources. RDI API v1 supports only +single-source pipelines. See the +[RDI API migration guide]({{< relref "/integrate/redis-data-integration/reference/api-migration" >}}) for more information.{{< /note >}} + +## Name your sources + +Each source is an entry in the `sources` section of +[`config.yaml`]({{< relref "/integrate/redis-data-integration/data-pipelines/pipeline-config" >}}), +with the source name as key: + +```yaml +sources: + mysql: # this source is named 'mysql' + type: cdc +``` + +A source name must: + +- Start with a lowercase letter. +- Contain only lowercase letters, digits, and dashes. +- End with a letter or a digit. +- Be at most 22 characters long. + +The names `rdi` and `target` are reserved and cannot be used for sources. + +{{< warning >}}If your pipeline has a source created before RDI supported multiple sources, do not name a new +source after any schema or database of that older source. The change data streams of the older +source do not contain a source name segment, so a new source named after one of its schemas would claim keys +that belong to the older source, and resetting or removing the new source would delete the older +source's data. See +[Existing names are kept after an upgrade](#existing-names-are-kept-after-an-upgrade).{{< /warning >}} + +RDI derives the environment variables that contain the source's credentials from the source +name. For example, the `connection` section of a source named `mysql` references `${MYSQL_DB_USERNAME}` +and `${MYSQL_DB_PASSWORD}`. See +[Set secrets]({{< relref "/integrate/redis-data-integration/data-pipelines/deploy#set-secrets" >}}) +for details of how RDI derives those names and for the full list of secret keys. + +The source name also appears in the resources RDI creates for the source. The table below +lists the names derived from a source named `mysql`. + +| Resource | Name | +| :-- | :-- | +| Credentials secret | `mysql-db` | +| TLS secret | `mysql-db-ssl` | +| Environment variable prefix | `MYSQL_DB_` | +| Certificate directory | `/etc/certificates/mysql_db/` | +| Collector deployment | `collector-mysql` | +| Change data streams | `data:{rdi}:mysql.` | +| Dead-letter queue streams | `dlq:data:{rdi}:mysql.` | +| Metric collection | `collector-mysql_metrics` | +| Metrics endpoint path on a VM installation | `/collector-mysql/metrics` | + +In the stream names, `` is the qualified table name: +`.` for MySQL and MariaDB, `.` for MongoDB, +`.
` for Oracle, PostgreSQL, Snowflake, and Spanner, and +`..
` for SQL Server. + +Each source also accepts an optional `name` property, which is a display name +of up to 100 characters. Unlike the source name, it is not used as an identifier, +so there is no restriction on the characters you can use. + +## Configure several sources + +Add one entry per source in the `config.yaml` file (see +[Pipeline configuration file]({{< relref "/integrate/redis-data-integration/data-pipelines/pipeline-config" >}}) +for a full description of this file). +[`redis-di scaffold`]({{< relref "/integrate/redis-data-integration/reference/cli/redis-di-scaffold" >}}) +generates a configuration with one source, named by its `--source-name` option, so add any +further sources by editing `config.yaml`. + +Sources of different types can be mixed freely, but a source's collector `type` and its +`connection` type have to match. Use `cdc`, the default, for the relational databases and +MongoDB, `flink` for a Spanner connection, and `riotx` for a Snowflake connection. RDI +rejects any other combination when you deploy the pipeline. See +[Prepare source databases]({{< relref "/integrate/redis-data-integration/data-pipelines/prepare-dbs" >}}) +to learn how to prepare each source database. + +The following example captures from a MySQL database and a PostgreSQL database, each with its +own credentials: + +```yaml +sources: + mysql: + type: cdc + connection: + type: mysql + host: + port: 3306 + user: ${MYSQL_DB_USERNAME} + password: ${MYSQL_DB_PASSWORD} + databases: + - inventory + tables: + inventory.customers: {} + inventory.orders: {} + postgresql: + type: cdc + connection: + type: postgresql + host: + port: 5432 + database: billing + user: ${POSTGRESQL_DB_USERNAME} + password: ${POSTGRESQL_DB_PASSWORD} + schemas: + - public + tables: + public.clients: {} +targets: + target: + connection: + type: redis + host: + port: 6379 + password: ${TARGET_DB_PASSWORD} +``` + +## Set secrets for each source + +Set a source's credentials with the source name in the `--db` option: + +```bash +redis-di set-secret USERNAME --db mysql +redis-di set-secret PASSWORD --db mysql +redis-di set-secret USERNAME --db postgresql +redis-di set-secret PASSWORD --db postgresql +``` + +The secret keys used as CLI arguments are the same for every source: `USERNAME`, `PASSWORD`, and, for +[Transport Layer Security (TLS)]({{< relref "/integrate/redis-data-integration/data-pipelines/deploy#set-secrets" >}}) +connections, `CACERT`, `CERT`, `KEY`, and `KEY_PASSWORD`. Use `--db target` for the +target database: `redis-di set-secret PASSWORD --db target `. + +Each source's `connection` section then references its own secrets: `${MYSQL_DB_USERNAME}` and +`${MYSQL_DB_PASSWORD}` for `mysql`, `${POSTGRESQL_DB_USERNAME}` and `${POSTGRESQL_DB_PASSWORD}` +for `postgresql`, and `${TARGET_DB_PASSWORD}` for the target. + +See [Set secrets]({{< relref "/integrate/redis-data-integration/data-pipelines/deploy#set-secrets" >}}) +for the full secret reference. + +## Select sources in jobs + +A [job]({{< relref "/integrate/redis-data-integration/data-pipelines/transform-examples" >}}) +selects the source it processes by setting `server_name` to the source name: + +```yaml +source: + server_name: mysql + db: inventory + table: customers +``` + +When a pipeline has more than one source, every job must set `server_name`, and the value +must match one of the sources in `config.yaml`. RDI rejects the pipeline when a job has no +`server_name`, or when its `server_name` matches no source. The one exception is the default +job for `table: "*"`: it is a source-agnostic catch-all, so it needs no `server_name`. + +For a source that existed before RDI supported multiple sources, set `server_name` to `rdi` +rather than to the name the source has in `config.yaml`. See +[Existing names are kept after an upgrade](#existing-names-are-kept-after-an-upgrade). + +In a pipeline with a single source, `server_name` is optional. If you omit it, the +job does not filter by source. + +No two jobs may select the same records, so make sure the source selectors of your jobs do +not overlap. RDI rejects the pipeline when it finds two jobs that intersect. + +With the [Flink processor]({{< relref "/integrate/redis-data-integration/architecture/classic-vs-flink" >}}), +`server_name` also accepts a list of source names, and an entry prefixed with `regex:` selects +all sources that match the regular expression, so one job can process multiple tables, +potentially from different sources, databases, or schemas. See +[Job files]({{< relref "/integrate/redis-data-integration/data-pipelines/transform-examples" >}}) +for details. + +## Add or remove a source + +To add a source, set its secrets first, then add it to `config.yaml` and deploy. Adding a +source does not interrupt other sources that are already running. + +To remove a source, delete its entry from `config.yaml` and deploy. RDI removes the source's +collector and deletes that source's data from the RDI database, including its change data streams, +Debezium offsets, schema history, dead-letter queue entries, statistics, deduplication state, +and record counters. The other sources keep their data, and RDI stops the whole pipeline +while the deletion runs and starts it again afterwards. No further action is +needed for this cleanup, but it means that a source you add later under the same name starts +from a new +[initial snapshot]({{< relref "/integrate/redis-data-integration/architecture" >}}) +rather than from the position it had reached. + +The source's secrets are not deleted, so remove them yourself with +[`redis-di delete-secret`]({{< relref "/integrate/redis-data-integration/reference/cli/redis-di-delete-secret" >}}) +if you no longer need them. + +Note that renaming a source is not supported. Renaming a source in `config.yaml` is equivalent +to removing the source and adding a new source with the new name. This implies in particular: + +- You must create the source's secrets under the new name and update `${...}` references in + its `connection` section. +- You must update `server_name` for every job that reads from the source. +- The data present in the RDI database under the old name is deleted, as it is for any removed source. +- The source starts with a new + [initial snapshot]({{< relref "/integrate/redis-data-integration/architecture" >}}). + +## Start, stop, and reset a single source + +Pass `--source` to act on a single source instead of the whole pipeline: + +```bash +redis-di stop --source mysql +redis-di start --source mysql +redis-di reset --source mysql +``` + +A source runs only while its pipeline runs, so starting one source does not start a stopped +pipeline. Generally, stopping one source leaves the others running, and when one source fails, the other sources keep capturing changes. The only exception to this is a source of type +`external`. RDI creates no collector for this, so you cannot start or stop it. + +Stopping a source scales its collector down to zero replicas and leaves the rest of the +source's resources in place. RDI records a captured position for each source, so when you restart a collector, it resumes from where it stopped. + +Resetting a single source deletes that source's data from the RDI database, including its change data streams, Debezium +offsets, schema history, dead-letter queue entries, statistics, deduplication state, and record counters. +A new [initial snapshot]({{< relref "/integrate/redis-data-integration/architecture" >}}) is then +taken for that source, while every other source keeps its data. RDI stops the whole pipeline while +the reset runs and starts it again afterwards, exactly as it does for a reset of the whole +pipeline. + +## Monitor each source + +Use [`redis-di describe`]({{< relref "/integrate/redis-data-integration/reference/cli/redis-di-describe" >}}) +to see the state of every source at once. + +In its output, the `Sources` section lists each source with its sync mode and +whether it is connected. +The `Components` section lists one collector per source. Errors are reported against the +component they came from. See the +[`redis-di describe`]({{< relref "/integrate/redis-data-integration/reference/cli/redis-di-describe" >}}) +reference page for more details. + +Note that while the sources are independent of each other in the data they capture, the +pipeline status is not broken down per source. RDI reports the whole pipeline in an error state when a +single source fails, so you should use the `Components` section to find out which one has failed. + +Each Debezium collector has its own metric collection, named after the collector, such as +`collector-mysql_metrics`. The Flink and RIOT-X collectors don't have metric collections. + +In Prometheus, you can break the per-stream record counters down per source, since the stream +name contains the source name. With the +[Flink processor]({{< relref "/integrate/redis-data-integration/architecture/classic-vs-flink" >}}) +the counters are reported by +`flink_jobmanager_job_operator_coordinator_stream_type_rdiRecords`, which has a `stream` +label; with the classic processor they are reported by `rdi_incoming_entries`, which has an +equivalent `data_source` label. See +[Flink processor metrics]({{< relref "/integrate/redis-data-integration/observability#flink-processor-metrics" >}}), +[Stream processor metrics]({{< relref "/integrate/redis-data-integration/observability#stream-processor-metrics" >}}), +and, for the per-source collector endpoints, +[Accessing the metrics]({{< relref "/integrate/redis-data-integration/observability#accessing-the-metrics" >}}). + +Dead-letter queue streams have Redis keys containing a +`.` section. +This makes it easy to attribute rejected records to their source. See +[Rejected records]({{< relref "/integrate/redis-data-integration/data-pipelines/rejected-records" >}}) for more information. + +## Existing names are kept after an upgrade + +Before RDI supported multiple sources per pipeline, every source-scoped resource had +a name including the word `source` instead of the actual source name in `config.yaml`. +For a source that existed before you upgraded to a version that supports multiple sources, +those names are kept unchanged, regardless of what the source is called in `config.yaml`. +In particular, for such a source: + +- Its secret environment variables are still named `SOURCE_DB_*`, so its `connection` section + can keep referencing these secrets. +- Its Kubernetes secrets are still named `source-db` and `source-db-ssl`. +- Its Kubernetes deployment and other resources are still named `collector-source`. +- Its data streams are still named `data:{rdi}:`, and its offset + and schema history keys are still `metadata:debezium:offsets` and + `metadata:debezium:schema_history`. +- Its `server_name` is still `rdi`, or, for a Spanner source, its instance ID. + +These names are permanent. If you remove such a source from `config.yaml` and later add a source +with the same name, RDI applies the names from before the upgrade to it again rather than +deriving them from the source name. + +For a source you add after the upgrade under any other name, RDI derives all of these names +from the source name, as described on this page. + +See [Upgrading RDI]({{< relref "/integrate/redis-data-integration/installation/upgrade" >}}) +for more information. diff --git a/content/integrate/redis-data-integration/data-pipelines/pipeline-config.md b/content/integrate/redis-data-integration/data-pipelines/pipeline-config.md index 93b73120c0..dae7f0077a 100644 --- a/content/integrate/redis-data-integration/data-pipelines/pipeline-config.md +++ b/content/integrate/redis-data-integration/data-pipelines/pipeline-config.md @@ -54,24 +54,24 @@ sources: type: mysql host: port: 3306 - user: ${SOURCE_DB_USERNAME} - password: ${SOURCE_DB_PASSWORD} + user: ${MYSQL_DB_USERNAME} + password: ${MYSQL_DB_PASSWORD} targets: target: connection: type: redis - host: - port: + host: + port: 6379 password: ${TARGET_DB_PASSWORD} processors: - type: flink target_data_type: hash ``` -Keep `type: flink` for new pipelines. The other processor properties have defaults, -so add them only when you need to change the default behavior. +The processor properties all have defaults, so add them only when you need to change the +default behavior. In particular, the pipeline runs on the Flink processor unless you set +`type` to `classic`. ## Build the file with an AI assistant @@ -111,11 +111,20 @@ and [`processors`](#processors). ### Sources -The `sources` section has a subsection for the source that -you need to configure. The source section starts with a unique name -to identify the source (in the example, there is a source -called `mysql` but you can choose any name you like). The example -configuration contains the following data: +The `sources` section has one subsection per source database. Each subsection starts +with the source name, which identifies the source and must be unique (in the example, +the source is called `mysql`). + +RDI also derives the environment variables that contain the source's credentials from the +source name, for example `${MYSQL_DB_USERNAME}` and `${MYSQL_DB_PASSWORD}` for a source +named `mysql`. See +[Multiple sources in one pipeline]({{< relref "/integrate/redis-data-integration/data-pipelines/multiple-sources" >}}) +for the source naming rules and for capturing from more than one source database. For a source +that existed before RDI supported multiple sources, these variables are still named +`SOURCE_DB_*`; see +[Existing names are kept after an upgrade]({{< relref "/integrate/redis-data-integration/data-pipelines/multiple-sources#existing-names-are-kept-after-an-upgrade" >}}). + +The example configuration contains the following data: - `type`: The collector to use for the pipeline. Use `cdc` for MariaDB, MySQL, MongoDB, Oracle, PostgreSQL, or SQL Server. Use `flink` for Google Cloud @@ -178,12 +187,13 @@ that you should set as described in [Set secrets]({{< relref "/integrate/redis-d ### Processors -The `processors` section selects the stream processor and configures its behavior. -Use the Flink processor for new pipelines: +The `processors` section selects the stream processor and configures its behavior. The default +is the Flink processor, which is what we recommend. To run a pipeline on the classic processor instead, +set `type` to `classic`: ```yaml processors: - type: flink + type: classic ``` See [Differences between the classic and Flink processors]({{< relref "/integrate/redis-data-integration/architecture/classic-vs-flink" >}}) @@ -255,7 +265,6 @@ setting for its output. ```yaml processors: - type: flink target_data_type: hash ``` @@ -269,7 +278,6 @@ healthy replica is available: ```yaml processors: - type: flink wait_enabled: true wait_timeout: 1000 retry_on_replica_failure: true @@ -284,141 +292,222 @@ for full details of the other available properties. ## Extended configuration example -This example combines the commonly used options from this page. Remove properties -that you don't need. See the +The example below is the configuration that +[`redis-di scaffold`]({{< relref "/integrate/redis-data-integration/reference/cli/redis-di-scaffold" >}}) +generates for a MySQL source named `mysql`, with every property documented inline. Uncomment what +you need and delete the rest. See the [configuration file reference]({{< relref "/integrate/redis-data-integration/reference/config-yaml-reference" >}}) for every supported property. ```yaml +# Configuration of the RDI sources, targets, and processor. Commented-out properties are optional, and where a default +# exists the value shown is that default. +# For an introduction to the pipeline configuration, see +# https://redis.io/docs/latest/integrate/redis-data-integration/data-pipelines/pipeline-config/ +# For a reference of every configuration property, see +# https://redis.io/docs/latest/integrate/redis-data-integration/reference/config-yaml-reference/ + +# Source databases that are used to capture changes from. Each key is a unique source name. sources: mysql: + # Type of the source collector. Use `cdc` for Debezium, `flink` for Spanner, or `riotx` for Snowflake. type: cdc + + # Log verbosity of the source collector, one of `trace`, `debug`, `info`, `warn`, or `error`. logging: level: info + + # Connection to the source database. connection: type: mysql - host: # e.g. localhost + # Hostname or IP address of the source database server, for example `localhost`. + host: + # Port that the source database server listens on. port: 3306 - # User and password are injected from the secrets. - user: ${SOURCE_DB_USERNAME} - password: ${SOURCE_DB_PASSWORD} - # Additional properties for the source collector: - # List of databases to include (optional). + # User and password are resolved from the source database secret. + user: ${MYSQL_DB_USERNAME} + password: ${MYSQL_DB_PASSWORD} + + # Databases to capture from the source database. When omitted, all databases are captured. # databases: - # - database1 - # - database2 + # - - # List of tables to be synced (optional). + # Tables to capture from the source database, keyed by table name. A table with no properties, or with an empty + # mapping, captures all of its columns. If only one database is listed above, table names can omit the + # database prefix. # tables: - # If only one database is specified in the databases property above, - # then tables can be defined without the database prefix. - # .: - # List of columns to be synced (optional). - # columns: - # - - # - - # List of columns to be used as keys (optional). - # keys: - # - - - # Example: Sync specific tables. - # tables: - # Sync a specific table with all its columns: - # redislabscdc.account: {} - # Sync a specific table with selected columns: - # redislabscdc.emp: - # columns: - # - empno - # - fname - # - lname - - # Advanced collector properties (optional): + # .: {} + # .: + # Columns to capture. When omitted, all columns are captured. + # columns: + # - + # Columns that form a unique identifier. Only needed when the table has no primary key or unique constraint. + # keys: + # - + + # Advanced properties that override the collector defaults. Only needed for non-standard tuning. # advanced: - # Sink collector properties - see the full list at - # https://debezium.io/documentation/reference/stable/operations/debezium-server.html#_redis_stream - # sink: - # Optional hard limits on memory usage of RDI streams. - # redis.memory.limit.mb: 300 - # redis.memory.threshold.percentage: 85 - - # Uncomment for production so RDI Collector will wait on replica - # when writing entries. - # redis.wait.enabled: true - # redis.wait.timeout.ms: 1000 - # redis.wait.retry.enabled: true - # redis.wait.retry.delay.ms: 1000 - - # Source specific properties - see the full list at - # https://debezium.io/documentation/reference/stable/connectors/ - # source: - # snapshot.mode: initial - # Uncomment if you want a snapshot to include only a subset of the rows - # in a table. This property affects snapshots only. - # snapshot.select.statement.overrides: . - # The specified SELECT statement determines the subset of table rows to - # include in the snapshot. - # snapshot.select.statement.overrides..: - - # Example: Snapshot filtering by order status. - # To include only orders with non-pending status from customers.orders - # table: - # snapshot.select.statement.overrides: customer.orders - # snapshot.select.statement.overrides.customer.orders: SELECT * FROM customers.orders WHERE status != 'pending' ORDER BY order_id DESC - - # Quarkus framework properties - see the full list at - # https://quarkus.io/guides/all-config - # quarkus: - # banner.enabled: "false" - - # `java_options` (for RDI 1.15.1 and above) controls the JAVA_OPTS environment variable. Use it to modify the default values for - # Java heap size and other Java options for the Debezium server. - # java_options: "-Xmx2g -Xms512m" - + # Properties of the RDI Collector stream writer. See the full list in + # https://debezium.io/documentation/reference/stable/operations/debezium-server.html#_redis_stream + # sink: + # Number of records that the sink writes to Redis in a single batch. Raise it, together with the source + # `max.batch.size` and `max.queue.size`, for higher snapshot throughput. + # redis.batch.size: 2048 + + # Back pressure on the RDI streams. The sink stops writing once the used memory of the RDI database reaches + # `redis.memory.threshold.percentage` of its `maxmemory`, or of `redis.memory.limit.mb` when that is set. + # redis.memory.limit.mb: 0 + # redis.memory.threshold.percentage: 85 + + # Whether every write to the RDI streams is verified to have reached a replica shard, which prevents losing + # captured changes when the primary shard fails over. It costs write latency, and it has no effect unless the + # RDI database is replicated. `redis.wait.retry.enabled` keeps retrying when the verification times out. + # redis.wait.enabled: false + # redis.wait.timeout.ms: 1000 + # redis.wait.retry.enabled: false + # redis.wait.retry.delay.ms: 1000 + + # Properties of the source database connection and of Debezium. See the full list for your source database in + # https://debezium.io/documentation/reference/stable/connectors/ + # source: + # When and whether the collector takes an initial snapshot of the source database. + # snapshot.mode: initial + + # Performance tuning of the Debezium collector. Rows read from the source database are buffered in a queue of + # `max.queue.size` records, serialized by `record.processing.threads` worker threads, and passed on in batches + # of `max.batch.size` records, which the sink then writes to Redis in batches of `sink.redis.batch.size`. + # Lower this to 2 or 1 when fewer CPUs are available to the collector. + # record.processing.threads: 4 + + # Interval between two polls of the source database for new changes in milliseconds. Lower it to 100 or below + # for lower CDC latency. + # poll.interval.ms: 500 + + # Raise both sizes, together with `sink.redis.batch.size`, for higher snapshot throughput, and keep the queue + # about four times the batch. A larger queue needs a larger heap, so raise `advanced.resources.memory` and set + # `-Xmx` in `advanced.java_options` accordingly. + # max.batch.size: 2048 + # max.queue.size: 8192 + + # Restrict the snapshot of a table to the rows that the given statement selects. This affects the snapshot only, + # for example to skip orders that are still pending. + # snapshot.select.statement.overrides: . + # snapshot.select.statement.overrides..: + + # Let the client retrieve the public key of the server for the TLS handshake. Uncomment when the source + # database user authenticates with `caching_sha2_password` over a connection that is not encrypted. + # database.allowPublicKeyRetrieval: true + + # Properties of the Quarkus runtime that hosts Debezium Server. See the full list in + # https://quarkus.io/guides/all-config + # quarkus: + # banner.enabled: "false" + +# Target Redis databases that the processed records are written to. RDI currently only supports a single target that +# must be named `target`. targets: - # Redis target database connection. - # RDI supports one target database. Name it 'target'. target: + # Connection to the target Redis database. connection: type: redis - # Host of the Redis database to which RDI will - # write the processed data. - host: # e.g. localhost - # Port for the Redis database to which RDI will - # write the processed data. - port: # e.g. 12000 - # User of the Redis database to which RDI will write the processed data. - # Uncomment if you are not using the default user. + # Hostname or IP address of the target Redis database, for example `localhost`. + host: + # Port that the target Redis database listens on. + port: 6379 + # User of the target Redis database, uncomment when not using the default user. # user: ${TARGET_DB_USERNAME} - # Password for Redis target database. + # Password of the target Redis database. password: ${TARGET_DB_PASSWORD} - # SSL/TLS configuration: Uncomment to enable secure connections. + # TLS configuration, uncomment to connect securely. The key and the certificate must be set together, and the key + # password only when the key is protected by one. + # cacert: ${TARGET_DB_CACERT} + # cert: ${TARGET_DB_CERT} # key: ${TARGET_DB_KEY} # key_password: ${TARGET_DB_KEY_PASSWORD} - # cert: ${TARGET_DB_CERT} - # cacert: ${TARGET_DB_CACERT} + +# Settings that control how the processor writes the captured records to the targets. processors: - type: flink - # Target data type: hash or json. + # Processor implementation to run, one of `classic` or `flink`. + # The default is `flink`, which is the recommended processor. + # See https://redis.io/docs/latest/integrate/redis-data-integration/faq/#which-processor-should-i-use + # type: flink + + # Maximum number of records read from the source streams in a single batch. + # read_batch_size: 2000 + + # Maximum time in milliseconds to wait for a read batch to fill before processing it. + # read_batch_timeout_ms: 100 + + # Maximum number of records written to a target database in a single batch. Must not exceed read_batch_size. + # write_batch_size: 200 + + # Whether batches are processed asynchronously, which improves throughput and reduces latency. Applies to the classic + # processor only. + # enable_async_processing: true + + # Maximum number of batches queued for processing. Applies to the classic processor only. + # batch_queue_size: 3 + + # Maximum number of batches queued for asynchronous acknowledgement. Applies to the classic processor only. + # ack_queue_size: 10 + + # Whether incoming records are deduplicated. Applies to the classic processor only. + # dedup: false + + # Maximum number of entries kept in the deduplication set. Applies to the classic processor only. + # dedup_max_size: 1024 + + # How failed records are handled. `ignore` drops them, and `dlq` writes them to the dead letter queue. + # error_handling: dlq + + # Maximum number of messages stored per dead letter queue stream. + # dlq_max_messages: 1000 + + # Data type that the records are stored as. `hash` writes a Redis hash, and `json` writes a RedisJSON document, which + # requires the RedisJSON module in the target database. # target_data_type: hash - # Enable merge as the default strategy for writing JSON documents. - # json_update_strategy: merge - # Confirm that writes reached a target database replica. + + # How existing JSON documents are updated. `replace` overwrites the whole document, and `merge` merges the incoming + # fields into it. + # json_update_strategy: replace + + # Whether JSON documents are merged with the native JSON.MERGE command rather than with Lua scripts. Applies to the + # classic processor only. + # use_native_json_merge: true + + # Number of parallel processes that perform the initial synchronization. Applies to the classic processor only. + # initial_sync_processes: 4 + + # Time in milliseconds to sleep between batches while idle. Applies to the classic processor only. + # idle_sleep_time_ms: 200 + + # Time in milliseconds between two checks for new streams while idle. Applies to the classic processor only. + # idle_streams_check_interval_ms: 1000 + + # Time in milliseconds between two checks for new streams while busy. Applies to the classic processor only. + # busy_streams_check_interval_ms: 5000 + + # Maximum number of attempts for a failed write to a target database before giving up. + # retry_max_attempts: 5 + + # Initial delay in milliseconds before the first retry of a failed write. + # retry_initial_delay_ms: 1000 + + # Maximum delay in milliseconds between two retries of a failed write. + # retry_max_delay_ms: 10000 + + # Whether every write is verified to have reached the replica shards of the target database. Enable this only when the + # target database is replicated and a healthy replica is available. # wait_enabled: false + + # Maximum time in milliseconds to wait for the replica write verification. # wait_timeout: 1000 + + # Whether a write is retried until the replica verification succeeds, rather than given up on after the first failure. # retry_on_replica_failure: true - # Flink processor performance settings. - # advanced: - # source: - # batch.size: 2000 - # batch.timeout.ms: 100 - # discovery.interval.ms: 1000 - # target: - # batch.size: 200 - # flush.interval.ms: 100 - # flink: - # taskmanager.numberOfTaskSlots: 1 - # taskmanager.memory.process.size: 2048m - # resources: - # taskManager: - # replicas: 2 + + # Logging settings of the processor. Applies to the Flink processor only. + # logging: + # Log verbosity of the processor, one of `trace`, `debug`, `info`, `warn`, or `error`. + # level: info ``` diff --git a/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/_index.md b/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/_index.md index e1cef5a262..0027587e5b 100644 --- a/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/_index.md +++ b/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/_index.md @@ -20,7 +20,12 @@ weight: 1 Each database uses a different mechanism to track changes to its data and generally, these mechanisms are not switched on by default. RDI's Debezium collector uses these mechanisms for change data capture (CDC), -so you must prepare your source database before you can use it with RDI. +so you must prepare each source database before you can use it with RDI. + +A pipeline can capture from more than one source database, and you must prepare each of them +separately. Follow the page for each source's database type. See +[Multiple sources in one pipeline]({{< relref "/integrate/redis-data-integration/data-pipelines/multiple-sources" >}}) +to learn how to configure several sources in one pipeline. RDI supports the following source databases: diff --git a/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/aws-aurora-rds/aws-aur-mysql.md b/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/aws-aurora-rds/aws-aur-mysql.md index 8299b1948b..42e03772a5 100644 --- a/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/aws-aurora-rds/aws-aur-mysql.md +++ b/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/aws-aurora-rds/aws-aur-mysql.md @@ -17,7 +17,6 @@ weight: 2 --- Follow the steps in the sections below to prepare an [AWS Aurora MySQL](https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/CHAP_GettingStartedAurora.CreatingConnecting.Aurora.html) or [AWS RDS MySQL](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/CHAP_GettingStarted.CreatingConnecting.MySQL.html) database. -database to work with RDI. Select the steps for your database type. diff --git a/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/mongodb.md b/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/mongodb.md index a49be68c82..6a8d93cc04 100644 --- a/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/mongodb.md +++ b/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/mongodb.md @@ -34,7 +34,7 @@ The following table summarizes the considerations to prepare a MongoDB database | Oplog | Sufficient size for snapshot and streaming | | Pre/Post Images | Enable on collections **only if using a custom key** | | Connection String | Must include all hosts, replicaSet (if applicable), authSource, credentials | -| MongoDB Atlas | **[SSL required](https://debezium.io/documentation/reference/stable/connectors/mongodb.html#mongodb-property-mongodb-ssl-enabled)**, provide root CA as `SOURCE_DB_CACERT` secret in RDI | +| MongoDB Atlas | **[SSL required](https://debezium.io/documentation/reference/stable/connectors/mongodb.html#mongodb-property-mongodb-ssl-enabled)**, provide root CA as the source's `CACERT` secret in RDI | | MongoDB mTLS | X.509 authentication requires source TLS secrets and MongoDB SSL properties | | Network | RDI Collector must reach all MongoDB nodes on required ports | @@ -90,13 +90,18 @@ db.createUser({ The RDI Collector requires a MongoDB connection string that includes all relevant hosts and authentication details. +The credential references come from the source name, which is `mongodb` in the examples on +this page. See +[Multiple sources in one pipeline]({{< relref "/integrate/redis-data-integration/data-pipelines/multiple-sources" >}}) +for the source naming rules. + Example (Replica Set): ``` -mongodb://${SOURCE_DB_USERNAME}:${SOURCE_DB_PASSWORD}@host1:27017,host2:27017,host3:27017/?replicaSet=rs0&authSource=admin +mongodb://${MONGODB_DB_USERNAME}:${MONGODB_DB_PASSWORD}@host1:27017,host2:27017,host3:27017/?replicaSet=rs0&authSource=admin ``` Example (Sharded Cluster): ``` -mongodb://${SOURCE_DB_USERNAME}:${SOURCE_DB_PASSWORD}@host:30000 +mongodb://${MONGODB_DB_USERNAME}:${MONGODB_DB_PASSWORD}@host:30000 ``` - For Atlas, adjust the connection string accordingly (see example below). - Set `replicaSet` and `authSource` as appropriate for your deployment. @@ -119,40 +124,40 @@ db.runCommand({ ## 5. MongoDB Atlas specific requirements MongoDB Atlas only supports secure connections via SSL. -The root CA certificate for MongoDB Atlas must be added as a SOURCE_DB_CACERT secret in RDI. +The root CA certificate for MongoDB Atlas must be added as the source's `CACERT` secret in RDI. - Download the MongoDB Atlas root CA certificate. -- In RDI, add this certificate as a secret named SOURCE_DB_CACERT. -- Ensure that the `mongodb.ssl.enabled: true` setting is present in your RDI configuration. +- In RDI, add this certificate with + `redis-di set-secret CACERT --db mongodb /path/to/atlas-ca.crt`. +- Ensure that the `mongodb.ssl.enabled: true` setting is present in your RDI configuration for the corresponding source DB collector. Example connection string for Atlas: ``` -mongodb+srv://${SOURCE_DB_USERNAME}:${SOURCE_DB_PASSWORD}@cluster0.mongodb.net/?authSource=admin +mongodb+srv://${MONGODB_DB_USERNAME}:${MONGODB_DB_PASSWORD}@cluster0.mongodb.net/?authSource=admin ``` ## 6. Self-hosted MongoDB mTLS and X.509 authentication -For self-hosted MongoDB deployments that require TLS, set the source CA certificate -as the `SOURCE_DB_CACERT` secret. For X.509 client certificate authentication, also -set the `SOURCE_DB_CERT` and `SOURCE_DB_KEY` secrets. See +For self-hosted MongoDB deployments that require TLS, set the source's `CACERT` secret to +the source CA certificate. For X.509 client certificate authentication, also set the +source's `CERT` and `KEY` secrets. See [Set secrets]({{< relref "/integrate/redis-data-integration/data-pipelines/deploy#set-secrets" >}}) for the full list of source database TLS and mTLS secrets. -When you use MongoDB X.509 authentication, include all of the following -properties in the source `advanced.source` section: +When you use MongoDB X.509 authentication, enable TLS in the source `advanced.source` section: ```yaml advanced: source: mongodb.ssl.enabled: true - mongodb.ssl.keystore: /debezium/certs/source_db_keystore - mongodb.ssl.keystore.password: debezium ``` -The RDI Collector builds `/debezium/certs/source_db_keystore` from the source -database client certificate and private key secrets. Debezium requires the -`mongodb.ssl.keystore` and `mongodb.ssl.keystore.password` properties to present -the client certificate to MongoDB. +RDI builds the keystore that presents the client certificate to MongoDB from the source's +`CERT` and `KEY` secrets, and configures the collector to use it. + +{{< note >}}Do not set `mongodb.ssl.keystore` or `mongodb.ssl.keystore.password` yourself. RDI +manages the keystore, so these properties are not only unnecessary, they are rejected when you +deploy the pipeline.{{< /note >}} For X.509 authentication, the MongoDB connection string must also include the required authentication options, such as `authMechanism=MONGODB-X509` and diff --git a/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/my-sql-mariadb.md b/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/my-sql-mariadb.md index 85f3e3d9f8..d97491ec20 100644 --- a/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/my-sql-mariadb.md +++ b/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/my-sql-mariadb.md @@ -31,6 +31,7 @@ checklist as you complete each step. - [ ] [Configure session timeouts](#4-configure-session-timeouts) - [ ] [Enable query log events](#5-enable-query-log-events) - [ ] [Check binlog_row_value_options](#6-check-binlog_row_value_options) +- [ ] [Connect over TLS or mTLS](#7-connect-over-tls-or-mtls) ``` ## 1. Create a CDC user @@ -227,7 +228,29 @@ If the value is `PARTIAL_JSON` then you should unset the variable: mysql> set @@global.binlog_row_value_options="" ; ``` -## 7. Configuration is complete +## 7. Connect over TLS or mTLS + +To connect to the source database over TLS, set the source's `CACERT` secret to the source CA +certificate. For mutual TLS, also set the source's `CERT` and `KEY` secrets. See +[Set secrets]({{< relref "/integrate/redis-data-integration/data-pipelines/deploy#set-secrets" >}}) +for the full list of source database TLS and mTLS secrets. + +Select the TLS mode in the source `advanced.source` section, for example: + +```yaml +advanced: + source: + database.ssl.mode: verify_ca +``` + +RDI builds the keystore that presents the client certificate to the source database from the +source's `CERT` and `KEY` secrets, and configures the collector to use it. + +{{< note >}}Do not set `database.ssl.keystore` or `database.ssl.keystore.password` yourself. RDI +manages the keystore, so these properties are not only unnecessary, they are rejected when you +deploy the pipeline.{{< /note >}} + +## 8. Configuration is complete After following the steps above, your MySQL/MariaDB database is ready for Debezium to use. diff --git a/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/oracle.md b/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/oracle.md index 5d8d2b09f7..7cea77ec11 100644 --- a/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/oracle.md +++ b/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/oracle.md @@ -1034,8 +1034,8 @@ sources: type: oracle host: host.docker.internal port: 1521 - user: ${SOURCE_DB_USERNAME} - password: ${SOURCE_DB_PASSWORD} + user: ${ORACLE_DB_USERNAME} + password: ${ORACLE_DB_PASSWORD} advanced: source: database.dbname: ORCLCDB @@ -1118,8 +1118,8 @@ sources: type: oracle host: oracle port: 1521 - user: ${SOURCE_DB_USERNAME} - password: ${SOURCE_DB_PASSWORD} + user: ${ORACLE_DB_USERNAME} + password: ${ORACLE_DB_PASSWORD} database: ORCLCDB advanced: source: @@ -1411,4 +1411,4 @@ With an XStream outbound server: - Debezium/RDI uses Oracle’s supported XStream API. - Transactions are preserved and grouped correctly. - Offsets/positions allow clean resume after restarts. -- Multiple consumers can be attached to the same outbound server if needed. \ No newline at end of file +- Multiple consumers can be attached to the same outbound server if needed. diff --git a/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/snowflake.md b/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/snowflake.md index 4b4550fc80..8fafda6515 100644 --- a/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/snowflake.md +++ b/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/snowflake.md @@ -38,7 +38,7 @@ Snowflake is only supported with RDI deployed on Kubernetes/Helm. RDI VM mode do ```checklist {id="snowflakelist"} - [ ] [Set up Snowflake permissions](#1-set-up-snowflake-permissions) - [ ] [Configure authentication](#2-configure-authentication) -- [ ] [Set up secrets for Kubernetes deployment](#3-set-up-secrets-for-kubernetes-deployment) +- [ ] [Set up secrets](#3-set-up-secrets) - [ ] [Configure RDI for Snowflake](#4-configure-rdi-for-snowflake) ``` @@ -140,37 +140,39 @@ For enhanced security, use key-pair authentication: ALTER USER rdi_user SET RSA_PUBLIC_KEY=''; ``` -## 3. Set up secrets for Kubernetes deployment +## 3. Set up secrets -Before deploying the RDI pipeline, configure the necessary secrets. +Before deploying the RDI pipeline, configure the necessary secrets with +[`redis-di set-secret`]({{< relref "/integrate/redis-data-integration/reference/cli/redis-di-set-secret" >}}). +Pass the source name with `--db` (the source configured in the following example is named `snowflake` for demonstration purposes). ### Password authentication ```bash -kubectl create secret generic source-db \ - --namespace=rdi \ - --from-literal=SOURCE_DB_USERNAME=your_username \ - --from-literal=SOURCE_DB_PASSWORD=your_password +redis-di set-secret USERNAME --db snowflake your_username +redis-di set-secret PASSWORD --db snowflake your_password ``` ### Private key authentication -Create a secret with the private key file: +Omit the password and set the private key from its file instead: ```bash -kubectl create secret generic source-db-ssl \ - --namespace=rdi \ - --from-file=client.key=/path/to/rsa_key.p8 +redis-di set-secret USERNAME --db snowflake your_username +redis-di set-secret KEY --db snowflake --file /path/to/rsa_key.p8 ``` -Also create the source-db secret with the username: +If the `.p8` file is protected by a passphrase, you must also set that passphrase: ```bash -kubectl create secret generic source-db \ - --namespace=rdi \ - --from-literal=SOURCE_DB_USERNAME=your_username +redis-di set-secret KEY_PASSWORD --db snowflake your_passphrase ``` +RDI stores the private key in the source's TLS secret, `snowflake-db-ssl`, and the RIOTX +collector reads it from there. See +[Set secrets]({{< relref "/integrate/redis-data-integration/data-pipelines/deploy#set-secrets" >}}) +for the full secret reference, including how to create these secrets with `kubectl` instead. + ## 4. Configure RDI for Snowflake Use the following example configuration in your `config.yaml` file: @@ -182,8 +184,8 @@ sources: connection: type: snowflake url: "jdbc:snowflake://myaccount.snowflakecomputing.com/" - user: "${SOURCE_DB_USERNAME}" - password: "${SOURCE_DB_PASSWORD}" # Omit for key-pair auth + user: "${SNOWFLAKE_DB_USERNAME}" + password: "${SNOWFLAKE_DB_PASSWORD}" # Omit for key-pair auth database: "MYDB" warehouse: "COMPUTE_WH" # role: "RDI_ROLE" # Optional: Snowflake role diff --git a/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/spanner.md b/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/spanner.md index 1e5b16966d..ca145615dd 100644 --- a/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/spanner.md +++ b/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/spanner.md @@ -35,7 +35,7 @@ checklist as you complete each step. - [ ] [Prepare for snapshot](#1-prepare-for-snapshot) - [ ] [Prepare for streaming](#2-prepare-for-streaming) - [ ] [Create a service account](#3-create-a-service-account) -- [ ] [Set up secrets for Kubernetes deployment (optional)](#4-set-up-secrets-for-kubernetes-deployment-optional) +- [ ] [Set up secrets (optional)](#4-set-up-secrets-optional) - [ ] [Configure RDI for Spanner](#5-configure-rdi-for-spanner) - [ ] [Additional Kubernetes configuration](#6-additional-kubernetes-configuration) ``` @@ -143,19 +143,29 @@ RDI supports two authentication methods for accessing Spanner: 2. **Service account credentials file**: You provide the service account key file as a Kubernetes secret. This method requires setting `use_credentials_file: true` in your RDI configuration. -## 4. Set up secrets for Kubernetes deployment (optional) +## 4. Set up secrets (optional) Before deploying the RDI pipeline, you need to configure the necessary secrets for the target -database. Instructions for setting up the target database secrets are available in the -[RDI deployment guide]({{< relref "/integrate/redis-data-integration/data-pipelines/deploy#set-secrets-for-k8shelm-deployment-using-kubectl-command" >}}). +database. Set them with +[`redis-di set-secret`]({{< relref "/integrate/redis-data-integration/reference/cli/redis-di-set-secret" >}}), +as described in +[Set secrets]({{< relref "/integrate/redis-data-integration/data-pipelines/deploy#set-secrets" >}}). +A Spanner source needs no credentials secret of its own, because it authenticates through +Workload Identity or a service account credentials file. **Optional**: If you prefer to use a service account credentials file instead of Workload Identity -authentication, you'll need to create a Spanner-specific secret named `source-db-credentials`. +authentication, you'll need to create a Spanner-specific secret named +`-db-credentials`, where `` is the name of the source in `config.yaml`. +The example below assumes a source named `spanner`, so its corresponding secret is +`spanner-db-credentials`. See +[Multiple sources in one pipeline]({{< relref "/integrate/redis-data-integration/data-pipelines/multiple-sources" >}}) +for the source naming rules. + This secret should contain the service account key file generated during the Spanner setup phase. Use the command below to create it: ```bash -kubectl create secret generic source-db-credentials --namespace=rdi \ +kubectl create secret generic spanner-db-credentials --namespace=rdi \ --from-file=gcp-service-account.json=~/spanner-reader-account.json \ --save-config --dry-run=client -o yaml | kubectl apply -f - ``` @@ -163,8 +173,13 @@ kubectl create secret generic source-db-credentials --namespace=rdi \ Be sure to adjust the file path (`~/spanner-reader-account.json`) if your service account key is stored elsewhere. +{{< note >}}`redis-di set-secret` has no key for this secret, so you must +create it with `kubectl`. A secret created that way must also be labeled so that the RDI operator +discovers it as a pipeline secret. See +[Set secrets for K8s/Helm deployment using Kubectl command]({{< relref "/integrate/redis-data-integration/data-pipelines/deploy#set-secrets-for-k8shelm-deployment-using-kubectl-command" >}}) for more information.{{< /note >}} + {{< note >}} -If you create the `source-db-credentials` secret, you must also set `use_credentials_file: true` +If you create the `spanner-db-credentials` secret, you must also set `use_credentials_file: true` in your RDI configuration to use the credentials file instead of Workload Identity authentication. {{< /note >}} @@ -175,7 +190,7 @@ When configuring your RDI pipeline for Spanner, use the following example config ```yaml sources: - source: + spanner: type: flink connection: type: spanner diff --git a/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/sql-server.md b/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/sql-server.md index 19f91ca52f..4cc700c09c 100644 --- a/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/sql-server.md +++ b/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/sql-server.md @@ -610,8 +610,8 @@ sources: host: .database.windows.net port: 1433 database: - user: ${SOURCE_DB_USERNAME} - password: ${SOURCE_DB_PASSWORD} + user: ${SQLSERVER_DB_USERNAME} + password: ${SQLSERVER_DB_PASSWORD} logging: level: info schemas: @@ -646,20 +646,27 @@ Debezium SQL Server connector and JDBC driver. The Azure-specific values are: | `snapshot.mode` | The Debezium snapshot strategy. | `initial`. Captures a snapshot of the existing rows, then streams subsequent changes from the CDC tables. | For SQL authentication, omit the `driver.authentication` line and set -`${SOURCE_DB_USERNAME}` and `${SOURCE_DB_PASSWORD}` to the SQL user's credentials. +`${SQLSERVER_DB_USERNAME}` and `${SQLSERVER_DB_PASSWORD}` to the SQL user's credentials. Keep the other Azure-specific properties. #### Secret mapping -For Microsoft Entra service-principal authentication, the RDI source secret must +For Microsoft Entra service-principal authentication, the source's credentials must provide: -| Secret key | Value | +| Config reference | Value | | --- | --- | -| `SOURCE_DB_USERNAME` | The service principal's **Application (client) ID** (a GUID). | -| `SOURCE_DB_PASSWORD` | The service principal's **client secret**. | +| `${SQLSERVER_DB_USERNAME}` | The service principal's **Application (client) ID** (a GUID). | +| `${SQLSERVER_DB_PASSWORD}` | The service principal's **client secret**. | -{{< warning >}}The `SOURCE_DB_USERNAME` value is the client ID (a GUID), but the contained +Set them with the source name, which is `sqlserver` for the source in this example: + +```bash +redis-di set-secret USERNAME --db sqlserver +redis-di set-secret PASSWORD --db sqlserver +``` + +{{< warning >}}The username value is the client ID (a GUID), but the contained database user created in the previous section uses the service principal's **display name**. These are two different identifiers for the same principal — mixing them up is the most common cause of `Login failed for user ''` errors @@ -711,7 +718,7 @@ GO - **`Login failed for user ''`** — the contained database user was not created for this service principal, or it was created with the wrong identifier. Verify that the `CREATE USER ... FROM EXTERNAL PROVIDER` statement used - the service principal's display name, and that `SOURCE_DB_USERNAME` contains its + the service principal's display name, and that `${SQLSERVER_DB_USERNAME}` contains its client ID. Query `sys.database_principals` on the source database to see which principals exist. - **`SSL Server certificate validation failed` or hostname mismatch** — @@ -781,4 +788,4 @@ documentation for further details. {{< note >}}RDI will *not* correctly capture changes that happen in the time gap between changing the source schema (step 1 above) and updating the value of `@capture_instance` (step 2). Try to keep the gap as short as possible or perform the update at a time when you expect -few changes to the data.{{< /note >}} \ No newline at end of file +few changes to the data.{{< /note >}} diff --git a/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/supabase.md b/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/supabase.md index bcc2e31454..98e71bd453 100644 --- a/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/supabase.md +++ b/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/supabase.md @@ -163,12 +163,13 @@ In the Supabase dashboard, go to 1. Enable **Enforce SSL on incoming connections**. 1. Download the Supabase CA certificate. -Store the database username, password, and CA certificate as RDI secrets: +Store the database username, password, and CA certificate as RDI secrets. Pass the +source name with `--db` (the source configured in the example below is named `supabase`): ```bash -redis-di set-secret SOURCE_DB_USERNAME rdi_replication -redis-di set-secret SOURCE_DB_PASSWORD '' -redis-di set-secret SOURCE_DB_CACERT /path/to/prod-ca-2021.crt +redis-di set-secret USERNAME --db supabase rdi_replication +redis-di set-secret PASSWORD --db supabase '' +redis-di set-secret CACERT --db supabase /path/to/prod-ca-2021.crt ``` RDI verifies that the direct endpoint hostname matches the certificate. @@ -187,8 +188,8 @@ sources: host: db..supabase.co port: 5432 database: postgres - user: ${SOURCE_DB_USERNAME} - password: ${SOURCE_DB_PASSWORD} + user: ${SUPABASE_DB_USERNAME} + password: ${SUPABASE_DB_PASSWORD} schemas: - public tables: diff --git a/content/integrate/redis-data-integration/data-pipelines/rejected-records.md b/content/integrate/redis-data-integration/data-pipelines/rejected-records.md index 73a8c50b92..65c00b26f1 100644 --- a/content/integrate/redis-data-integration/data-pipelines/rejected-records.md +++ b/content/integrate/redis-data-integration/data-pipelines/rejected-records.md @@ -41,28 +41,28 @@ RDI stores rejected records in the RDI database as capped Redis streams. Each DL stream corresponds to a source table and tracks the records rejected for that table. -DLQ stream names use the `dlq:` prefix followed by the source data stream name. -In current RDI versions, the stream name is typically: +A DLQ stream is named after the source data stream it corresponds to, with a `dlq:` prefix: ```text -dlq:data:{rdi}:.
+dlq:data:{rdi}:. ``` -For example, rejected records for the `public.users` table are stored in: +Above, `` is the qualified table name: `.
` +for MySQL and MariaDB, `.` for MongoDB, `.
` for Oracle, +PostgreSQL, Snowflake, and Spanner, and `..
` for SQL Server. -```text -dlq:data:{rdi}:public.users -``` - -For sources that include the source name in the stream qualifier, the final part -can contain three components: +For example, rejected records for the `public.users` table of a source named `postgresql` +are stored in `dlq:data:{rdi}:postgresql.public.users`, and those for the `dbo.users` table of +the `inventory` database of a source named `sqlserver` are stored in +`dlq:data:{rdi}:sqlserver.inventory.dbo.users`. -```text -dlq:data:{rdi}:..
-``` +The dead-letter queue endpoints and the CLI report table names are in the same form, +so you can easily attribute a rejected record to the source it came from. Sources that existed +before the upgrade to RDI v2.0.0 or above do not have the source segment +in the key because these earlier RDI versions did not support multiple sources. Some RDI versions or configurations can use a hash-tagged variant such as -`dlq:{data:rdi:.
}`. To find all DLQ streams in the +`dlq:{data:rdi:}`. To find all DLQ streams in the RDI database, scan for stream keys that start with `dlq:`. The maximum number of records stored per DLQ stream is controlled by diff --git a/content/integrate/redis-data-integration/data-pipelines/supported-types.md b/content/integrate/redis-data-integration/data-pipelines/supported-types.md index 51d53e8e9c..ef22c25e49 100644 --- a/content/integrate/redis-data-integration/data-pipelines/supported-types.md +++ b/content/integrate/redis-data-integration/data-pipelines/supported-types.md @@ -195,7 +195,7 @@ choose. When a connector captures large objects (for example, Oracle `CLOB`/`BLOB`), an update event never contains the value of an *unchanged* LOB column. Instead, the -column carries a placeholder. The default placeholder is `__debezium_unavailable_value`, +column contains a placeholder. The default placeholder is `__debezium_unavailable_value`, which you can change with `advanced.source.unavailable.value.placeholder` on the source (the Helm chart exposes this as `processor.lob.placeholder`). RDI skips these placeholder values rather than writing them to Redis as user data. @@ -453,7 +453,7 @@ What is available for updates depends on `capture.mode`: - A *create* event always includes the full document. - An *update* event includes the full document only when `capture.mode` is - `change_streams_update_full`; otherwise it carries only the changed fields + `change_streams_update_full`; otherwise it includes only the changed fields (`updatedFields`/`removedFields`). A `*_with_pre_image` mode is required to include the prior document state. diff --git a/content/integrate/redis-data-integration/data-pipelines/transform-examples/_index.md b/content/integrate/redis-data-integration/data-pipelines/transform-examples/_index.md index 3b0b2cfa35..99a498b31e 100644 --- a/content/integrate/redis-data-integration/data-pipelines/transform-examples/_index.md +++ b/content/integrate/redis-data-integration/data-pipelines/transform-examples/_index.md @@ -58,7 +58,13 @@ The main sections of these files are: - `source`: This is a mandatory section that specifies the data items that you want to use. You can add the following properties here: - - `server_name`: Logical server name (optional). + - `server_name`: The name of the source whose records this job processes, as it appears + in the `sources` section of + [config.yaml]({{< relref "/integrate/redis-data-integration/data-pipelines/pipeline-config" >}}). + This is required when the pipeline has more than one source but is optional in a pipeline with a single source. See + [Multiple sources in one pipeline]({{< relref "/integrate/redis-data-integration/data-pipelines/multiple-sources" >}}) for more information. + For a source that existed before RDI supported multiple sources, use `rdi` instead, as described in + [Existing names are kept after an upgrade]({{< relref "/integrate/redis-data-integration/data-pipelines/multiple-sources#existing-names-are-kept-after-an-upgrade" >}}). - `db`: Database name (optional). This refers to a database name you supplied in [config.yaml]({{< relref "/integrate/redis-data-integration/data-pipelines/pipeline-config" >}}). - `schema`: Database schema (optional). This refers to a schema name you supplied in @@ -72,6 +78,18 @@ The main sections of these files are: and is set to `true` by default. Set it to `false` if you need to use case-sensitive values for these properties. + With the + [Flink processor]({{< relref "/integrate/redis-data-integration/architecture/classic-vs-flink" >}}), + `server_name`, `db`, `schema`, and `table` each accept either a single name or a list of names. + An entry prefixed with `regex:` selects all names that match the regular expression (note that this + must match the whole name, not just a substring). For example, `server_name: [mysql, postgresql]` selects two sources, + and `table: "regex:orders_[0-9]+"` selects every numbered `orders` table. A job processes the + records that match all of the properties it sets, so a single job can process multiple tables, + potentially from different sources, databases, or schemas. + + The classic processor takes a single name in each of these properties. It rejects a list or a + `regex:` entry when you deploy the pipeline, so you should use one job per table instead. + - `transform`: This is an optional section describing the transformation that the pipeline applies to the data before writing it to the target. The `uses` property specifies a *transformation block* that will use the parameters supplied in the `with` section. See the @@ -126,7 +144,7 @@ the default logic. ```yaml name: Rename field example source: - server_name: redislabs + server_name: sqlserver schema: dbo table: emp transform: diff --git a/content/integrate/redis-data-integration/data-pipelines/transform-examples/redis-json-example.md b/content/integrate/redis-data-integration/data-pipelines/transform-examples/redis-json-example.md index d2b0380d81..668176b41b 100644 --- a/content/integrate/redis-data-integration/data-pipelines/transform-examples/redis-json-example.md +++ b/content/integrate/redis-data-integration/data-pipelines/transform-examples/redis-json-example.md @@ -26,7 +26,7 @@ database to use this feature. In the example below, the data is captured from the source table named `invoice` and is written to the Redis database as a JSON document. The `connection` is an optional parameter that refers to the corresponding connection name defined in `config.yaml`. When you specify the `data_type` parameter for the job, it overrides the system-wide setting `target_data_type` defined in `config.yaml`. -Another optional parameter, `on_update`, specifies the writing strategy. You can set this to either `replace` (the default) or `merge`. This affects the way the document is written to the target. Replacing the document will overwrite it completely, while merging will update it with the fields captured in the source, keeping the rest of the document intact. The `replace` option is usually more performant, while `merge` allows other jobs and applications to set extra fields in the same JSON documents. +Another optional parameter, `on_update`, specifies the writing strategy. You can set this to either `replace` (the default) or `merge`. This affects the way the document is written to the target. Replacing the document will overwrite it completely, while merging will update it with the fields captured in the source, keeping the rest of the document intact. The `replace` option is usually more performant, while with `merge` other jobs and applications can set extra fields in the same JSON documents. In this case, the result will be Redis JSON documents with key names based on the key expression (for example, `invoice_id:1`) and with an expiration of 100 seconds. If you don't supply an `expire` parameter, the keys will never expire. diff --git a/content/integrate/redis-data-integration/faq.md b/content/integrate/redis-data-integration/faq.md index 5c86fc184c..ef74473a61 100644 --- a/content/integrate/redis-data-integration/faq.md +++ b/content/integrate/redis-data-integration/faq.md @@ -86,6 +86,12 @@ deployment, use separate Kubernetes clusters. See [Install on Kubernetes]({{< relref "/integrate/redis-data-integration/installation/install-k8s" >}}) for installation details. +## Can one pipeline capture from several source databases? + +Yes. Add one entry per source to the `sources` section of `config.yaml`. Each source has its +own collector, which captures change records independently of the other sources. See +[Multiple sources in one pipeline]({{< relref "/integrate/redis-data-integration/data-pipelines/multiple-sources" >}}) for more information. + ## Can RDI automatically track changes to the source database schema? If you don't configure RDI to capture a specific set of tables in the schema then it will @@ -161,18 +167,14 @@ processor. It also adds optional expression and `redis.lookup` result caching. **We strongly recommend using the Flink processor** for new pipelines and -migrating existing pipelines to it, to benefit from these improvements. The -*classic* processor is still the default, so pipelines keep using it until -you opt in, and it remains a fully supported choice — for example, when you -want to ensure your pipelines continue to work as before until you have -consciously migrated them. In a future release, however, the Flink processor -will become the default and the classic processor may be deprecated, so adopting -the Flink processor now avoids a later migration. - -Switch a pipeline to the Flink processor by setting +migrating existing pipelines to it, to benefit from these improvements. It is +the default, so a pipeline whose `config.yaml` does not set a processor type +uses it. The *classic* processor remains a fully supported choice for now. +It may be deprecated in a future release. + +To switch a pipeline to the classic processor, set [`processors.type`]({{< relref "/integrate/redis-data-integration/data-pipelines/pipeline-config#processors" >}}) -to `flink` (`classic` is the default). You can adopt it per pipeline without -changing the others. +to `classic`. You can do that per pipeline without changing the others. See [Differences between the classic and Flink processors]({{< relref "/integrate/redis-data-integration/architecture/classic-vs-flink" >}}) diff --git a/content/integrate/redis-data-integration/installation/_index.md b/content/integrate/redis-data-integration/installation/_index.md index c5e7bdee75..5dac61f0bb 100644 --- a/content/integrate/redis-data-integration/installation/_index.md +++ b/content/integrate/redis-data-integration/installation/_index.md @@ -19,6 +19,6 @@ weight: 20 The guides in this section explain the options you have for installing and upgrading RDI on your own servers. See the [Redis Cloud RDI guide]({{< relref "/operate/rc/rdi" >}}) to learn how to set up RDI for a cloud database. -Before you use RDI, you must also configure your source database to enable CDC. See the +Before you use RDI, you must also configure each of your source databases to enable CDC. See the [Prepare source databases]({{< relref "/integrate/redis-data-integration/data-pipelines/prepare-dbs" >}}) -section to learn how to do this. \ No newline at end of file +section to learn how to do this. diff --git a/content/integrate/redis-data-integration/installation/ha-test.md b/content/integrate/redis-data-integration/installation/ha-test.md index 7f89c08514..9309bdde15 100644 --- a/content/integrate/redis-data-integration/installation/ha-test.md +++ b/content/integrate/redis-data-integration/installation/ha-test.md @@ -24,7 +24,7 @@ kubectl -n rdi get pods # Example output: NAME READY STATUS RESTARTS AGE collector-api-577d95bfd8-5wbg6 1/1 Running 0 12m -collector-source-95f45bcf7-vwn5l 1/1 Running 0 12m +collector-mysql-95f45bcf7-vwn5l 1/1 Running 0 12m fluentd-zq2lc 1/1 Running 0 72m logrotate-29530445-j729x 0/1 Completed 0 14m logrotate-29530450-dprr2 0/1 Completed 0 9m40s @@ -36,7 +36,27 @@ rdi-operator-7f7f6c7dfd-5qmjd 1/1 Running 0 71m rdi-reloader-77df5f7854-lwmvz 1/1 Running 0 71m ``` -2. Identify the leader node - this is the one that has a running `collector-source` pod. +2. Identify the leader node (this is the one that has a running collector pod). A + collector is named after its source, so the example above shows `collector-mysql` for a + source named `mysql`. + + You can also ask each node about the pipeline directly. On the follower, the pipeline + stands by: + + ``` + kubectl -n rdi get pipeline default -o jsonpath='{.status.phase}' + + # Example output on the follower: + Standby + + # Example output on the leader: + Active + ``` + + [`redis-di describe`]({{< relref "/integrate/redis-data-integration/reference/cli/redis-di-describe" >}}) + shows `Status: standby` on the follower, and + [`redis-di info`]({{< relref "/integrate/redis-data-integration/reference/cli/redis-di-info" >}}) + shows `Leader Election Mode: follower`. ## Performing the HA Failover Testing @@ -73,6 +93,9 @@ kubectl -n rdi logs rdi-operator-7f7f6c7dfd-5qmjd -f In about 10 seconds you will start seeing log entries from the leader saying that it could not acquire the leadership. When the leader lock expires, the second node will acquire the leadership and you will see log entries from the second node indicating that it has become the leader. +Once the failover completes, the two nodes swap their pipeline phases: the pipeline has +`Active` status on the new leader and `Standby` status on the old one. + ## Cleanup To clean up after the test, remove the `iptables` rule that you added to block the traffic: diff --git a/content/integrate/redis-data-integration/installation/install-k8s.md b/content/integrate/redis-data-integration/installation/install-k8s.md index 55b0ebfa6f..c0aa45a81f 100644 --- a/content/integrate/redis-data-integration/installation/install-k8s.md +++ b/content/integrate/redis-data-integration/installation/install-k8s.md @@ -101,12 +101,12 @@ If you plan to use Spanner as a source for your pipeline, you'll also need [redis/rdi-flink-collector](https://hub.docker.com/r/redis/rdi-flink-collector). If you plan to use Snowflake as a source for any of your pipelines, you'll also need -[riotx/riotx:v1.8.0](https://hub.docker.com/r/riotx/riotx): +[riotx/riotx:v1.12.0](https://hub.docker.com/r/riotx/riotx): [RIOT-X](https://redis.github.io/riotx/), a data ingestion and replication tool for Redis. In addition, the RDI Helm chart uses the following 3rd party images: -- [redislabs/debezium-server:3.5.0.Final-rdi.1](https://hub.docker.com/r/redislabs/debezium-server), +- [redislabs/debezium-server:3.5.0.Final-rdi.3](https://hub.docker.com/r/redislabs/debezium-server), based on `quay.io/debezium/server/3.5.0.Final` with minor modifications: [Debezium](https://debezium.io/), an open source distributed platform for change data capture. - [redis/reloader:v1.4.13](https://hub.docker.com/r/redis/reloader), originally `ghcr.io/stakater/reloader:v1.4.13`: @@ -322,9 +322,8 @@ oc get projects -o yaml | grep "openshift.io/sa.scc" ### Configure the Flink processor -RDI ships with two stream processor implementations: the default *classic* -processor and the -[Apache Flink](https://flink.apache.org/)-based *Flink* processor. +RDI ships with two stream processor implementations: the *classic* processor and the +default [Apache Flink](https://flink.apache.org/)-based *Flink* processor. See [Stream processor implementations]({{< relref "/integrate/redis-data-integration/architecture#stream-processor-implementations" >}}) for an overview of the differences and @@ -356,10 +355,9 @@ operator: Configuring the Flink processor at the Helm chart level only sets the values that the operator will use when deploying the JobManager and TaskManager workloads. -To run a specific pipeline on the Flink processor, set +A pipeline runs on the Flink processor unless its `config.yaml` sets [`processors.type`]({{< relref "/integrate/redis-data-integration/data-pipelines/pipeline-config#processors" >}}) -to `flink` in that pipeline's `config.yaml`. Pipelines without this setting -continue to use the classic processor. Fine-tune the Flink runtime +to `classic`. Fine-tune the Flink runtime through the `processors.advanced` section of `config.yaml` (see the [configuration reference]({{< relref "/integrate/redis-data-integration/reference/config-yaml-reference#processors" >}})). @@ -423,15 +421,17 @@ Specifically, ensure that one or both of the following Helm chart values is set: - `controller.service.annotations."service\.beta\.kubernetes\.io/azure-load-balancer-health-probe-request-path"=/healthz` - `controller.service.externalTrafficPolicy=Local` -## Prepare your source database +## Prepare your source databases -Before deploying a pipeline, you must configure your source database to enable CDC. See the +Before deploying a pipeline, you must configure each source database to enable CDC. See the [Prepare source databases]({{< relref "/integrate/redis-data-integration/data-pipelines/prepare-dbs" >}}) -section to learn how to do this. +section to learn how to do this. A pipeline can capture from more than one source database, +so you must prepare each one each source separately. See +[Multiple sources in one pipeline]({{< relref "/integrate/redis-data-integration/data-pipelines/multiple-sources" >}}) for details. ## Deploy a pipeline -When the Helm installation is complete and you have prepared the source database for CDC, +When the Helm installation is complete and you have prepared your source databases for CDC, you are ready to start using RDI. See the guides on how to [configure]({{< relref "/integrate/redis-data-integration/data-pipelines" >}}) and [deploy]({{< relref "/integrate/redis-data-integration/data-pipelines/deploy" >}}) diff --git a/content/integrate/redis-data-integration/installation/install-vm.md b/content/integrate/redis-data-integration/installation/install-vm.md index 63700922fa..b9ab70864a 100644 --- a/content/integrate/redis-data-integration/installation/install-vm.md +++ b/content/integrate/redis-data-integration/installation/install-vm.md @@ -17,7 +17,7 @@ weight: 10 --- This guide explains how to install Redis Data Integration (RDI) on one or more VMs and integrate it with -your source database. You can also +your source databases. You can also [Install RDI on Kubernetes]({{< relref "/integrate/redis-data-integration/installation/install-k8s" >}}). {{< note >}}We recommend you always use the latest version, which is RDI v{{< rdi-version >}}. @@ -269,17 +269,31 @@ to renew the lease in the RDI database, it will lose the leadership and a failov will take place. After the failover, the secondary instance will become the primary one, and the RDI pipeline will be active on that VM. +The secondary instance keeps the pipeline and its configuration, and reports it with the +`standby` status, but runs none of its collectors or its stream processor, and it does not +touch the pipeline's keys in the RDI database. A standby pipeline is read-only, so +running [`redis-di describe`]({{< relref "/integrate/redis-data-integration/reference/cli/redis-di-describe" >}}) +against the secondary instance works and shows `Status: standby`. Attempting to change a standby pipeline +fails with `423 Locked` and a message saying that the pipeline is standing by because this instance is +currently a leader election follower. Operations such as deploy, start, stop, reset, and delete +can only be performed against the primary instance. + +You can still set pipeline secrets for an instance when it is on standby. Secrets are maintained per +instance and are not replicated between the two, because they can legitimately differ in some cases. + You may find it useful to trigger a failover deliberately to check that RDI is correctly configured to handle it. See [Test HA failover]({{< relref "/integrate/redis-data-integration/installation/ha-test" >}}) to learn how to do this. -## Prepare your source database +## Prepare your source databases -Before deploying a pipeline, you must configure your source database to enable CDC. See the +Before deploying a pipeline, you must configure each source database to enable CDC. See the [Prepare source databases]({{< relref "/integrate/redis-data-integration/data-pipelines/prepare-dbs" >}}) -section to learn how to do this. +section to learn how to do this. A pipeline can capture from more than one source database, +and you must prepare each one separately. See +[Multiple sources in one pipeline]({{< relref "/integrate/redis-data-integration/data-pipelines/multiple-sources" >}}) for more information. ## Deploy a pipeline -When the installation is complete, and you have prepared the source database for CDC, +When the installation is complete, and you have prepared your source databases for CDC, you are ready to start using RDI. See the guides on how to [configure]({{< relref "/integrate/redis-data-integration/data-pipelines" >}}) and [deploy]({{< relref "/integrate/redis-data-integration/data-pipelines/deploy" >}}) @@ -296,19 +310,17 @@ for your platform.{{< /note >}} ## Configure the Flink processor -RDI ships with two stream processor implementations: the default *classic* -processor and the -[Apache Flink](https://flink.apache.org/)-based *Flink* processor. +RDI ships with two stream processor implementations: the *classic* processor and the +default [Apache Flink](https://flink.apache.org/)-based *Flink* processor. See [Stream processor implementations]({{< relref "/integrate/redis-data-integration/architecture#stream-processor-implementations" >}}) for an overview of the differences and [Differences between the classic and Flink processors]({{< relref "/integrate/redis-data-integration/architecture/classic-vs-flink" >}}) for a side-by-side comparison. -To run a specific pipeline on the Flink processor, set +A pipeline runs on the Flink processor unless its `config.yaml` sets [`processors.type`]({{< relref "/integrate/redis-data-integration/data-pipelines/pipeline-config#processors" >}}) -to `flink` in that pipeline's `config.yaml` and redeploy it. Pipelines without -this setting continue to use the classic processor. Fine-tune the Flink runtime +to `classic`. Fine-tune the Flink runtime through the `processors.advanced` section of `config.yaml` (see the [configuration reference]({{< relref "/integrate/redis-data-integration/reference/config-yaml-reference#processors" >}})). @@ -331,4 +343,4 @@ This will uninstall RDI and its dependencies, are you sure? [y, N] ``` If you type anything other than "y" here, the script will abort without making any changes -to RDI or your source database. +to RDI or your source databases. diff --git a/content/integrate/redis-data-integration/installation/migration-classic-to-flink.md b/content/integrate/redis-data-integration/installation/migration-classic-to-flink.md index 36ac9d9ba4..cee3ee3c36 100644 --- a/content/integrate/redis-data-integration/installation/migration-classic-to-flink.md +++ b/content/integrate/redis-data-integration/installation/migration-classic-to-flink.md @@ -16,8 +16,8 @@ type: integration weight: 35 --- -RDI ships with two stream processor implementations. The default *classic* -processor is implemented in Python. The *Flink* processor is built on top of +RDI ships with two stream processor implementations. The *classic* +processor is implemented in Python. The default *Flink* processor is built on top of [Apache Flink](https://flink.apache.org/). Both run on VM and Kubernetes installations. The Flink processor can achieve much higher throughput during snapshots, scales horizontally by changing the number of TaskManager replicas, @@ -64,7 +64,7 @@ resources in step 4. In the pipeline's `config.yaml`, set [`processors.type`]({{< relref "/integrate/redis-data-integration/data-pipelines/pipeline-config#processors" >}}) -to `flink`: +to `flink` (or remove this property, since `flink` is the default): ```yaml processors: @@ -134,7 +134,7 @@ for the `ServiceMonitor` configuration and the available metrics. ## Rolling back -To revert a pipeline to the classic processor, set `processors.type` back to -`classic` (or remove the property) and redeploy the pipeline. The +To revert a pipeline to the classic processor, set `processors.type` to +`classic` and redeploy the pipeline. The `processors.advanced` section is silently ignored by the classic processor, so you don't need to remove it before switching back. diff --git a/content/integrate/redis-data-integration/installation/upgrade.md b/content/integrate/redis-data-integration/installation/upgrade.md index da873958c0..13c1ecedf9 100644 --- a/content/integrate/redis-data-integration/installation/upgrade.md +++ b/content/integrate/redis-data-integration/installation/upgrade.md @@ -74,7 +74,7 @@ run the CLI binary of the previous RDI version, which still provided the `redis- sudo redis-di upgrade --rdi-host --rdi-port ``` -{{< note >}}If the `collector-source` or the `processor` pods are not in the `Running` state after +{{< note >}}If the `collector` or `processor` pods are not in the `Running` state after the upgrade, you must run `redis-di deploy` and check again that they are both in the `Running` state. {{< /note >}} @@ -174,6 +174,14 @@ described in [Uninstall RDI]({{< relref "/integrate/redis-data-integration/insta and then install the old version. {{< /note >}} +## Source names before and after the upgrade + +For a source that existed before upgrading to an RDI version that supports multiple sources, the +resource names from before the upgrade are kept unchanged. The source will keep running correctly without changes. + +For more details on how to handle sources created after upgrading, see +[Multiple sources in one pipeline]({{< relref "/integrate/redis-data-integration/data-pipelines/multiple-sources#existing-names-are-kept-after-an-upgrade" >}}). + ## Enabling the Flink processor The @@ -181,11 +189,12 @@ The fully supported on both VM and Kubernetes installations after upgrading to RDI 1.19.0. Once the upgrade completes, it is always available — no opt-in is required, and the defaults are sized for typical workloads. -Upgrading does not change the processor used by existing pipelines, which keep -running on the classic processor until you explicitly switch them by -setting + +{{< warning >}}The Flink processor is the default as of RDI 2.0.0. +Upgrading to that release or later moves a pipeline whose `config.yaml` does not set [`processors.type`]({{< relref "/integrate/redis-data-integration/data-pipelines/pipeline-config#processors" >}}) -to `flink` in their `config.yaml`. +onto the Flink processor when you next deploy it. To keep such a pipeline on the classic +processor, set `processors.type` to `classic` before you upgrade.{{< /warning >}} On Kubernetes, to override the Flink processor defaults, add an `operator.dataPlane.flinkProcessor` block to your `rdi-values.yaml` file as @@ -203,7 +212,7 @@ The upgrade process replaces the current RDI components with their new versions: - Firstly, the control plane components are replaced. At this point, the pipeline is still active but monitoring will be disconnected. - Secondly, the pipeline data plane components are replaced. - If a pipeline is active while upgrading, the `collector-source` and `processor` + If a pipeline is active while upgrading, the `collector` and `processor` pods will be restarted. The pipeline will pause for up to two minutes but it will catch up very quickly after restarting. The pipeline data and state are both stored in Redis, so data will not diff --git a/content/integrate/redis-data-integration/observability.md b/content/integrate/redis-data-integration/observability.md index eabcb2ab0d..219b380626 100644 --- a/content/integrate/redis-data-integration/observability.md +++ b/content/integrate/redis-data-integration/observability.md @@ -50,10 +50,25 @@ The way you access the metrics endpoints depends on whether you are using a VM i ### VM Installation +You can use the CLI to determine the metrics endpoints instead of assembling them yourself. +The `Components` section of +[`redis-di describe`]({{< relref "/integrate/redis-data-integration/reference/cli/redis-di-describe" >}}) +has a `Metrics Endpoints` column listing each component's externally-reachable metrics URLs, +so a pipeline with several sources shows the URL of every collector. + +RDI discovers these endpoints from each component's `Ingress` resources, so a component +without an ingress has none, and the list is empty until the ingress controller assigns an +address to the ingress. + For VM installations, the metrics are available by default on the following endpoints: -- Collector metrics: `https:///collector-source/metrics` -- Stream processor metrics: `https:///processor/metrics` -- Operator metrics: `https:///operator/metrics` +- Collector metrics: `https://:/collector-/metrics`, where `` is + the source name from `config.yaml`. Each source has its own collector, so a pipeline with + several sources has one endpoint per source. +- Stream processor metrics: `https://:/processor/metrics` (Flink processor) + or `https://:/metrics` (classic processor). +- Operator metrics: `https://:/operator/metrics` + +Above, `` is the HTTPS port you specified when installing RDI, or 443 if you kept the default. Please note that for RDI versions prior to 1.16.0 the collector metrics are not accessible. @@ -223,7 +238,7 @@ RDI reports with their descriptions. - Where the metric name has the `rdi_` prefix, this will be replaced by the Kubernetes namespace name if you supplied a custom name during installation. The prefix is always `rdi_` for VM installations. - Metrics with the `_created` suffix are automatically generated by Prometheus for counters and gauges to track when they were first created. -- The `rdi_incoming_entries` metric provides a detailed breakdown for each data source by operation type. +- The `rdi_incoming_entries` metric provides a detailed breakdown for each data source by operation type. The `data_source` label identifies the data stream the value belongs to, as the stream's key without the `data:` prefix (for example, `{rdi}:mysql.inventory.customers`). The key contains the source name, so you can break the metric down per source. Sources that existed before the upgrade to a version that supports multiple sources do not have the source segment in the key. - The `rdi_stream_event_latency_ms` metric helps monitor data freshness and processing delays. - The processor performance metrics are divided into two categories: - **Total metrics**: Accumulate values across all processed batches for historical analysis diff --git a/content/integrate/redis-data-integration/quick-start-guide.md b/content/integrate/redis-data-integration/quick-start-guide.md index 44134d605f..6418287ff6 100644 --- a/content/integrate/redis-data-integration/quick-start-guide.md +++ b/content/integrate/redis-data-integration/quick-start-guide.md @@ -71,6 +71,10 @@ the `config.yaml` file and then edit the following settings: At this point, the pipeline is ready to deploy. +This tutorial uses one source database, but a pipeline can capture from several source databases +into the same Redis target. See +[Multiple sources in one pipeline]({{< relref "/integrate/redis-data-integration/data-pipelines/multiple-sources" >}}) for more information. + ### Create a context (optional) {#create-context} To manage and inspect RDI, you can use the diff --git a/content/integrate/redis-data-integration/reference/api-migration.md b/content/integrate/redis-data-integration/reference/api-migration.md index eed803b967..58ebbfd22b 100644 --- a/content/integrate/redis-data-integration/reference/api-migration.md +++ b/content/integrate/redis-data-integration/reference/api-migration.md @@ -27,6 +27,15 @@ RDI 1.19.0 supports only one pipeline, which must be named `default`. Support fo The API version is part of the URL. Update `/api/v1` requests to use `/api/v2` where a corresponding v2 endpoint is available. You should also review the request and response models, because they can differ between versions. +## API v1 and multiple sources + +API v1 handles single-source pipelines only, so +you must use API v2 to manage [a pipeline with several sources]({{< relref "/integrate/redis-data-integration/data-pipelines/multiple-sources" >}}). In particular: + +- `POST /api/v1/pipelines`, `PATCH /api/v1/pipelines`, and `POST /api/v1/pipelines/sources/dry-run` reject a configuration with more than one source with `422 Unprocessable Content` and `Only a single source per pipeline is supported`. +- `PUT /api/v1/secrets`, `PUT /api/v1/secrets/{secret_name}`, and the v1 source management endpoints only handle a pipeline whose single source still uses the legacy `source` names, which covers a pipeline created before RDI supported multiple sources and one that you create through v1. They reject any other pipeline with `Use API v2 to manage sources and secrets of this pipeline.` +- `GET /api/v1/status` and `GET /api/v1/monitoring/statistics` report the first source of the pipeline only. + ## Endpoint mapping | API v1 | API v2 | @@ -45,7 +54,8 @@ The API version is part of the URL. Update `/api/v1` requests to use `/api/v2` w | `PUT /api/v1/pipelines/sources` and source subresources | `PATCH /api/v2/pipelines/{name}` with `sources` in the payload | | `PUT /api/v1/pipelines/targets` and target subresources | `PATCH /api/v2/pipelines/{name}` with `targets` in the payload | | `PUT /api/v1/pipelines/processors` and `PUT /api/v1/pipelines/processors/{prop}` | `PATCH /api/v2/pipelines/{name}` with `processors` in the payload | -| Secret provider endpoints | `POST`, `PUT`, or `DELETE /api/v2/pipelines/{name}/secrets[/{key}]` | +| `PUT /api/v1/secrets` and `PUT /api/v1/secrets/{secret_name}` | `POST`, `PUT`, or `DELETE /api/v2/pipelines/{name}/secrets[/{key}]` with the `db` query parameter | +| `POST`, `PUT`, or `DELETE /api/v1/pipelines/secret-providers/{name}` | `PATCH /api/v2/pipelines/{name}` with `secret-providers` in the payload | | Source metadata, schemas, databases, tables, and columns endpoints | `GET /api/v2/pipelines/{name}/source-schemas/{source_name}` with the appropriate filters | | `POST /api/v1/pipelines/sources/dry-run` | `POST /api/v2/pipelines?dry_run=true` | | `POST /api/v1/pipelines/targets/dry-run` | `POST /api/v2/pipelines?dry_run=true` | @@ -118,7 +128,8 @@ Note: 2. Add the pipeline name to each v2 request. The only pipeline in 1.19.0 is always named `default`. 3. Check the pipeline response, or call `GET /api/v2/pipelines/{name}/status`, instead of polling an action ID. 4. Use `POST /api/v2/pipelines`, `PUT /api/v2/pipelines/{name}`, or `PATCH /api/v2/pipelines/{name}` to update source, target, processor, and secret-provider settings as needed. When using `PATCH`, omit the configuration sections that you do not want to change. -5. Use `GET /api/v2/pipelines/{name}/metric-collections/{collection_name}` for monitoring and `GET /api/v2/pipelines/{name}/source-schemas/{source_name}` for source metadata. -6. Test creating, updating, validating, starting, stopping, resetting, and deleting a pipeline on a non-production RDI 1.19.0 or later installation before updating production applications. +5. Change secret requests to address a secret by a database-independent key and a `db` parameter. Where a v1 request set `SOURCE_DB_PASSWORD`, a v2 request sets the `PASSWORD` key with `db` naming the source, and `TARGET_DB_PASSWORD` becomes the `PASSWORD` key with `db=target`. See [Set secrets]({{< relref "/integrate/redis-data-integration/data-pipelines/deploy#set-secrets" >}}). The environment variable that `config.yaml` references is unchanged for a source that existed before RDI supported multiple sources; see [Existing names are kept after an upgrade]({{< relref "/integrate/redis-data-integration/data-pipelines/multiple-sources#existing-names-are-kept-after-an-upgrade" >}}). +6. Use `GET /api/v2/pipelines/{name}/metric-collections/{collection_name}` for monitoring and `GET /api/v2/pipelines/{name}/source-schemas/{source_name}` for source metadata. +7. Test creating, updating, validating, starting, stopping, resetting, and deleting a pipeline on a non-production RDI 1.19.0 or later installation before updating production applications. Authentication and the API base URL do not change. The migration requires updates to the endpoint paths, pipeline scoping, request models, and operation status handling. diff --git a/content/integrate/redis-data-integration/reference/api-reference/openapi.json b/content/integrate/redis-data-integration/reference/api-reference/openapi.json index 8c3c37d013..57241fb3a4 100644 --- a/content/integrate/redis-data-integration/reference/api-reference/openapi.json +++ b/content/integrate/redis-data-integration/reference/api-reference/openapi.json @@ -2,7 +2,7 @@ "openapi": "3.1.0", "info": { "title": "Redis Data Integration API", - "description": "> **NOTE:** RDI API v1 is deprecated as of RDI 1.19.0. Use RDI API v2 for new integrations and migrate existing clients. API v1 will not be extended with new RDI features and may be removed in a future RDI version. See the [RDI API migration guide](/integrate/redis-data-integration/reference/api-migration/).\n\nAPI for Redis Data Integration services", + "description": "> **NOTE:** RDI API v1 is deprecated as of RDI 1.19.0. Use RDI API v2 for new integrations and migrate existing clients. API v1 will not be extended with new RDI features and may be removed in a future RDI version. See the [RDI API migration guide](https://redis.io/docs/latest/integrate/redis-data-integration/reference/api-migration/).\n\nAPI for Redis Data Integration services", "version": "1.19.0" }, "paths": { @@ -50,6 +50,16 @@ } } } + }, + "401": { + "description": "Unauthorized", + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" + } + } + } } }, "security": [] @@ -77,7 +87,7 @@ "content": { "application/json": { "schema": { - "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse" + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" } } } @@ -87,7 +97,27 @@ "content": { "application/json": { "schema": { - "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse" + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" + } + } + } + }, + "404": { + "description": "Not Found", + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" + } + } + } + }, + "423": { + "description": "Locked", + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" } } } @@ -122,7 +152,7 @@ "content": { "application/json": { "schema": { - "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse" + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" } } } @@ -132,7 +162,27 @@ "content": { "application/json": { "schema": { - "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse" + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" + } + } + } + }, + "404": { + "description": "Not Found", + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" + } + } + } + }, + "423": { + "description": "Locked", + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" } } } @@ -149,7 +199,7 @@ "post": { "tags": ["secure"], "summary": "Reset Pipeline", - "description": "Reset pipeline.", + "description": "Resets a pipeline.", "operationId": "reset_pipeline_api_v1_pipelines_reset_post", "responses": { "200": { @@ -167,7 +217,7 @@ "content": { "application/json": { "schema": { - "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse" + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" } } } @@ -177,7 +227,27 @@ "content": { "application/json": { "schema": { - "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse" + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" + } + } + } + }, + "404": { + "description": "Not Found", + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" + } + } + } + }, + "423": { + "description": "Locked", + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" } } } @@ -228,7 +298,7 @@ "content": { "application/json": { "schema": { - "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse" + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" } } } @@ -238,7 +308,17 @@ "content": { "application/json": { "schema": { - "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse" + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" + } + } + } + }, + "404": { + "description": "Action not found", + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/ActionResponse" } } } @@ -270,7 +350,7 @@ "content": { "application/json": { "schema": { - "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse" + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" } } } @@ -280,16 +360,13 @@ "content": { "application/json": { "schema": { - "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse" + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" } } } } }, "security": [ - { - "JWTBearer": [] - }, { "JWTBearer": [] } @@ -338,15 +415,7 @@ "content": { "application/json": { "schema": { - "anyOf": [ - { - "$ref": "#/components/schemas/ActionIdResponse" - }, - { - "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__Error" - } - ], - "title": "Response Update Sources Api V1 Pipelines Sources Put" + "$ref": "#/components/schemas/ActionIdResponse" } } } @@ -356,7 +425,7 @@ "content": { "application/json": { "schema": { - "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse" + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" } } } @@ -366,7 +435,37 @@ "content": { "application/json": { "schema": { - "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse" + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" + } + } + } + }, + "404": { + "description": "Not Found", + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" + } + } + } + }, + "422": { + "description": "Unprocessable Entity", + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/ExtendedErrorResponse" + } + } + } + }, + "423": { + "description": "Locked", + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" } } } @@ -435,15 +534,7 @@ "content": { "application/json": { "schema": { - "anyOf": [ - { - "$ref": "#/components/schemas/ActionIdResponse" - }, - { - "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__Error" - } - ], - "title": "Response Upsert Source By Name Api V1 Pipelines Sources Name Put" + "$ref": "#/components/schemas/ActionIdResponse" } } } @@ -453,7 +544,7 @@ "content": { "application/json": { "schema": { - "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse" + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" } } } @@ -463,10 +554,40 @@ "content": { "application/json": { "schema": { - "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse" + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" } } } + }, + "404": { + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" + } + } + }, + "description": "Not Found" + }, + "422": { + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/ExtendedErrorResponse" + } + } + }, + "description": "Unprocessable Entity" + }, + "423": { + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" + } + } + }, + "description": "Locked" } } }, @@ -519,15 +640,7 @@ "content": { "application/json": { "schema": { - "anyOf": [ - { - "$ref": "#/components/schemas/ActionIdResponse" - }, - { - "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__Error" - } - ], - "title": "Response Partial Update Source By Name Api V1 Pipelines Sources Name Patch" + "$ref": "#/components/schemas/ActionIdResponse" } } } @@ -537,7 +650,7 @@ "content": { "application/json": { "schema": { - "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse" + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" } } } @@ -547,10 +660,40 @@ "content": { "application/json": { "schema": { - "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse" + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" } } } + }, + "404": { + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" + } + } + }, + "description": "Not Found" + }, + "422": { + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/ExtendedErrorResponse" + } + } + }, + "description": "Unprocessable Entity" + }, + "423": { + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" + } + } + }, + "description": "Locked" } } }, @@ -581,15 +724,7 @@ "content": { "application/json": { "schema": { - "anyOf": [ - { - "$ref": "#/components/schemas/ActionIdResponse" - }, - { - "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__Error" - } - ], - "title": "Response Delete Source By Name Api V1 Pipelines Sources Name Delete" + "$ref": "#/components/schemas/ActionIdResponse" } } } @@ -599,7 +734,7 @@ "content": { "application/json": { "schema": { - "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse" + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" } } } @@ -609,10 +744,40 @@ "content": { "application/json": { "schema": { - "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse" + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" } } } + }, + "404": { + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" + } + } + }, + "description": "Not Found" + }, + "422": { + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/ExtendedErrorResponse" + } + } + }, + "description": "Unprocessable Entity" + }, + "423": { + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" + } + } + }, + "description": "Locked" } } } @@ -652,15 +817,7 @@ "content": { "application/json": { "schema": { - "anyOf": [ - { - "$ref": "#/components/schemas/ActionIdResponse" - }, - { - "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__Error" - } - ], - "title": "Response Update Targets Api V1 Pipelines Targets Put" + "$ref": "#/components/schemas/ActionIdResponse" } } } @@ -670,7 +827,7 @@ "content": { "application/json": { "schema": { - "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse" + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" } } } @@ -680,7 +837,37 @@ "content": { "application/json": { "schema": { - "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse" + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" + } + } + } + }, + "404": { + "description": "Not Found", + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" + } + } + } + }, + "422": { + "description": "Unprocessable Entity", + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/ExtendedErrorResponse" + } + } + } + }, + "423": { + "description": "Locked", + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" } } } @@ -742,15 +929,7 @@ "content": { "application/json": { "schema": { - "anyOf": [ - { - "$ref": "#/components/schemas/ActionIdResponse" - }, - { - "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__Error" - } - ], - "title": "Response Upsert Target By Name Api V1 Pipelines Targets Name Put" + "$ref": "#/components/schemas/ActionIdResponse" } } } @@ -760,7 +939,7 @@ "content": { "application/json": { "schema": { - "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse" + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" } } } @@ -770,10 +949,40 @@ "content": { "application/json": { "schema": { - "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse" + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" } } } + }, + "404": { + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" + } + } + }, + "description": "Not Found" + }, + "422": { + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/ExtendedErrorResponse" + } + } + }, + "description": "Unprocessable Entity" + }, + "423": { + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" + } + } + }, + "description": "Locked" } } }, @@ -823,15 +1032,7 @@ "content": { "application/json": { "schema": { - "anyOf": [ - { - "$ref": "#/components/schemas/ActionIdResponse" - }, - { - "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__Error" - } - ], - "title": "Response Partial Update Target By Name Api V1 Pipelines Targets Name Patch" + "$ref": "#/components/schemas/ActionIdResponse" } } } @@ -841,7 +1042,7 @@ "content": { "application/json": { "schema": { - "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse" + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" } } } @@ -851,16 +1052,46 @@ "content": { "application/json": { "schema": { - "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse" + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" } } } - } - } - }, - "delete": { - "tags": ["secure", "targets"], - "summary": "Delete Target By Name", + }, + "404": { + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" + } + } + }, + "description": "Not Found" + }, + "422": { + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/ExtendedErrorResponse" + } + } + }, + "description": "Unprocessable Entity" + }, + "423": { + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" + } + } + }, + "description": "Locked" + } + } + }, + "delete": { + "tags": ["secure", "targets"], + "summary": "Delete Target By Name", "description": "Deletes a target from RDI settings by its name.", "operationId": "delete_target_by_name_api_v1_pipelines_targets__name__delete", "security": [ @@ -885,15 +1116,7 @@ "content": { "application/json": { "schema": { - "anyOf": [ - { - "$ref": "#/components/schemas/ActionIdResponse" - }, - { - "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__Error" - } - ], - "title": "Response Delete Target By Name Api V1 Pipelines Targets Name Delete" + "$ref": "#/components/schemas/ActionIdResponse" } } } @@ -903,7 +1126,7 @@ "content": { "application/json": { "schema": { - "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse" + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" } } } @@ -913,10 +1136,40 @@ "content": { "application/json": { "schema": { - "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse" + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" } } } + }, + "404": { + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" + } + } + }, + "description": "Not Found" + }, + "422": { + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/ExtendedErrorResponse" + } + } + }, + "description": "Unprocessable Entity" + }, + "423": { + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" + } + } + }, + "description": "Locked" } } } @@ -978,15 +1231,7 @@ "content": { "application/json": { "schema": { - "anyOf": [ - { - "$ref": "#/components/schemas/ActionIdResponse" - }, - { - "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__Error" - } - ], - "title": "Response Upsert Secret Provider By Name Api V1 Pipelines Secret Providers Name Put" + "$ref": "#/components/schemas/ActionIdResponse" } } } @@ -996,7 +1241,7 @@ "content": { "application/json": { "schema": { - "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse" + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" } } } @@ -1006,10 +1251,40 @@ "content": { "application/json": { "schema": { - "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse" + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" } } } + }, + "404": { + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" + } + } + }, + "description": "Not Found" + }, + "422": { + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/ExtendedErrorResponse" + } + } + }, + "description": "Unprocessable Entity" + }, + "423": { + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" + } + } + }, + "description": "Locked" } } }, @@ -1057,15 +1332,7 @@ "content": { "application/json": { "schema": { - "anyOf": [ - { - "$ref": "#/components/schemas/ActionIdResponse" - }, - { - "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__Error" - } - ], - "title": "Response Partial Update Secret Provider By Name Api V1 Pipelines Secret Providers Name Patch" + "$ref": "#/components/schemas/ActionIdResponse" } } } @@ -1075,7 +1342,7 @@ "content": { "application/json": { "schema": { - "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse" + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" } } } @@ -1085,10 +1352,40 @@ "content": { "application/json": { "schema": { - "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse" + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" } } } + }, + "404": { + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" + } + } + }, + "description": "Not Found" + }, + "422": { + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/ExtendedErrorResponse" + } + } + }, + "description": "Unprocessable Entity" + }, + "423": { + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" + } + } + }, + "description": "Locked" } } }, @@ -1119,15 +1416,7 @@ "content": { "application/json": { "schema": { - "anyOf": [ - { - "$ref": "#/components/schemas/ActionIdResponse" - }, - { - "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__Error" - } - ], - "title": "Response Delete Secret Provider By Name Api V1 Pipelines Secret Providers Name Delete" + "$ref": "#/components/schemas/ActionIdResponse" } } } @@ -1137,7 +1426,7 @@ "content": { "application/json": { "schema": { - "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse" + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" } } } @@ -1147,10 +1436,40 @@ "content": { "application/json": { "schema": { - "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse" + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" } } } + }, + "404": { + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" + } + } + }, + "description": "Not Found" + }, + "422": { + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/ExtendedErrorResponse" + } + } + }, + "description": "Unprocessable Entity" + }, + "423": { + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" + } + } + }, + "description": "Locked" } } } @@ -1159,7 +1478,7 @@ "post": { "tags": ["secure", "connection"], "summary": "Sources Dry Run", - "description": "Tests the connection to the source database.\n\nArgs:\n body: The connection details according to the OpenAPI Scheme.\n\nReturns:\n Union[ConnectionStatusSingleConnection, Error]: The connection status.", + "description": "Tests the connection to the source database.", "operationId": "sources_dry_run_api_v1_pipelines_sources_dry_run_post", "requestBody": { "content": { @@ -1195,15 +1514,7 @@ "content": { "application/json": { "schema": { - "anyOf": [ - { - "$ref": "#/components/schemas/ConnectionStatusSingleConnection" - }, - { - "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__Error" - } - ], - "title": "Response Sources Dry Run Api V1 Pipelines Sources Dry Run Post" + "$ref": "#/components/schemas/ConnectionStatusSingleConnection" } } } @@ -1213,7 +1524,7 @@ "content": { "application/json": { "schema": { - "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse" + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" } } } @@ -1223,7 +1534,47 @@ "content": { "application/json": { "schema": { - "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse" + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" + } + } + } + }, + "422": { + "description": "Unprocessable Entity", + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" + } + } + } + }, + "500": { + "description": "Internal Server Error", + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" + } + } + } + }, + "503": { + "description": "Service Unavailable", + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" + } + } + } + }, + "504": { + "description": "Gateway Timeout", + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" } } } @@ -1287,7 +1638,7 @@ "content": { "application/json": { "schema": { - "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse" + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" } } } @@ -1297,7 +1648,27 @@ "content": { "application/json": { "schema": { - "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse" + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" + } + } + } + }, + "400": { + "description": "Bad Request", + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" + } + } + } + }, + "422": { + "description": "Unprocessable Entity", + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" } } } @@ -1351,7 +1722,7 @@ "content": { "application/json": { "schema": { - "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse" + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" } } } @@ -1361,7 +1732,17 @@ "content": { "application/json": { "schema": { - "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse" + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" + } + } + } + }, + "422": { + "description": "Unprocessable Entity", + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" } } } @@ -1441,7 +1822,7 @@ "content": { "application/json": { "schema": { - "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse" + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" } } } @@ -1451,7 +1832,7 @@ "content": { "application/json": { "schema": { - "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse" + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" } } } @@ -1461,20 +1842,60 @@ "content": { "application/json": { "schema": { - "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse" + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" } } } }, + "422": { + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" + } + } + }, + "description": "Unprocessable Entity" + }, "423": { "description": "Collector API is not available for the specified source or is disabled", "content": { "application/json": { "schema": { - "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse" + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" } } } + }, + "502": { + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" + } + } + }, + "description": "Bad Gateway" + }, + "503": { + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" + } + } + }, + "description": "Service Unavailable" + }, + "504": { + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" + } + } + }, + "description": "Gateway Timeout" } } } @@ -1517,7 +1938,7 @@ "content": { "application/json": { "schema": { - "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse" + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" } } } @@ -1527,7 +1948,7 @@ "content": { "application/json": { "schema": { - "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse" + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" } } } @@ -1537,20 +1958,60 @@ "content": { "application/json": { "schema": { - "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse" + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" } } } }, + "422": { + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" + } + } + }, + "description": "Unprocessable Entity" + }, "423": { "description": "Collector API is not available for the specified source or is disabled", "content": { "application/json": { "schema": { - "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse" + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" } } } + }, + "502": { + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" + } + } + }, + "description": "Bad Gateway" + }, + "503": { + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" + } + } + }, + "description": "Service Unavailable" + }, + "504": { + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" + } + } + }, + "description": "Gateway Timeout" } } } @@ -1593,7 +2054,7 @@ "content": { "application/json": { "schema": { - "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse" + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" } } } @@ -1603,7 +2064,7 @@ "content": { "application/json": { "schema": { - "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse" + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" } } } @@ -1613,20 +2074,60 @@ "content": { "application/json": { "schema": { - "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse" + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" } } } }, + "422": { + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" + } + } + }, + "description": "Unprocessable Entity" + }, "423": { "description": "Collector API is not available for the specified source or is disabled", "content": { "application/json": { "schema": { - "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse" + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" } } } + }, + "502": { + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" + } + } + }, + "description": "Bad Gateway" + }, + "503": { + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" + } + } + }, + "description": "Service Unavailable" + }, + "504": { + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" + } + } + }, + "description": "Gateway Timeout" } } } @@ -1705,7 +2206,7 @@ "content": { "application/json": { "schema": { - "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse" + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" } } } @@ -1715,7 +2216,7 @@ "content": { "application/json": { "schema": { - "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse" + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" } } } @@ -1725,20 +2226,60 @@ "content": { "application/json": { "schema": { - "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse" + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" } } } }, - "423": { - "description": "Collector API is not available for the specified source or is disabled", + "422": { + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" + } + } + }, + "description": "Unprocessable Entity" + }, + "423": { + "description": "Collector API is not available for the specified source or is disabled", + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" + } + } + } + }, + "502": { + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" + } + } + }, + "description": "Bad Gateway" + }, + "503": { + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" + } + } + }, + "description": "Service Unavailable" + }, + "504": { "content": { "application/json": { "schema": { - "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse" + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" } } - } + }, + "description": "Gateway Timeout" } } } @@ -1792,7 +2333,7 @@ "content": { "application/json": { "schema": { - "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse" + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" } } } @@ -1802,7 +2343,7 @@ "content": { "application/json": { "schema": { - "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse" + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" } } } @@ -1812,20 +2353,60 @@ "content": { "application/json": { "schema": { - "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse" + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" } } } }, + "422": { + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" + } + } + }, + "description": "Unprocessable Entity" + }, "423": { "description": "Collector API is not available for the specified source or is disabled", "content": { "application/json": { "schema": { - "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse" + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" } } } + }, + "502": { + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" + } + } + }, + "description": "Bad Gateway" + }, + "503": { + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" + } + } + }, + "description": "Service Unavailable" + }, + "504": { + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" + } + } + }, + "description": "Gateway Timeout" } } } @@ -1833,9 +2414,9 @@ "/api/v1/pipelines": { "get": { "tags": ["secure", "deploy"], - "summary": "Retrieve Pipelines", + "summary": "Get Pipelines", "description": "Returns the current pipeline.", - "operationId": "retrieve_pipelines_api_v1_pipelines_get", + "operationId": "get_pipelines_api_v1_pipelines_get", "security": [ { "JWTBearer": [] @@ -1849,7 +2430,7 @@ "schema": { "type": "object", "additionalProperties": true, - "title": "Response Retrieve Pipelines Api V1 Pipelines Get" + "title": "Response Get Pipelines Api V1 Pipelines Get" } } } @@ -1859,7 +2440,7 @@ "content": { "application/json": { "schema": { - "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse" + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" } } } @@ -1869,18 +2450,28 @@ "content": { "application/json": { "schema": { - "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse" + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" } } } + }, + "404": { + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" + } + } + }, + "description": "Not Found" } } }, "post": { "tags": ["secure", "deploy"], - "summary": "Pipelines", + "summary": "Post Pipelines", "description": "Updates the current pipeline completely, or creates it if it doesn't exist, using the provided configuration.\nBefore creating or updating, validates the provided configuration for schema compliance and integrity.", - "operationId": "pipelines_api_v1_pipelines_post", + "operationId": "post_pipelines_api_v1_pipelines_post", "security": [ { "JWTBearer": [] @@ -1892,14 +2483,7 @@ "in": "query", "required": false, "schema": { - "anyOf": [ - { - "type": "boolean" - }, - { - "type": "null" - } - ], + "type": "boolean", "description": "Only validate the provided configuration without making changes", "default": false, "title": "Dry Run" @@ -2102,7 +2686,7 @@ "content": { "application/json": { "schema": { - "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse" + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" } } } @@ -2112,7 +2696,7 @@ "content": { "application/json": { "schema": { - "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse" + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" } } } @@ -2164,6 +2748,46 @@ } } } + }, + "423": { + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" + } + } + }, + "description": "Locked" + }, + "502": { + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" + } + } + }, + "description": "Bad Gateway" + }, + "503": { + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" + } + } + }, + "description": "Service Unavailable" + }, + "504": { + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" + } + } + }, + "description": "Gateway Timeout" } } }, @@ -2183,14 +2807,7 @@ "in": "query", "required": false, "schema": { - "anyOf": [ - { - "type": "boolean" - }, - { - "type": "null" - } - ], + "type": "boolean", "description": "Only validate the provided configuration without making changes", "default": false, "title": "Dry Run" @@ -2406,7 +3023,7 @@ "content": { "application/json": { "schema": { - "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse" + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" } } } @@ -2416,7 +3033,7 @@ "content": { "application/json": { "schema": { - "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse" + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" } } } @@ -2468,6 +3085,46 @@ } } } + }, + "423": { + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" + } + } + }, + "description": "Locked" + }, + "502": { + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" + } + } + }, + "description": "Bad Gateway" + }, + "503": { + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" + } + } + }, + "description": "Service Unavailable" + }, + "504": { + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" + } + } + }, + "description": "Gateway Timeout" } } } @@ -2484,15 +3141,7 @@ "content": { "application/json": { "schema": { - "anyOf": [ - { - "$ref": "#/components/schemas/ActionIdResponse" - }, - { - "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__Error" - } - ], - "title": "Response Undeploy Pipelines Api V1 Pipelines Undeploy Post" + "$ref": "#/components/schemas/ActionIdResponse" } } } @@ -2502,7 +3151,7 @@ "content": { "application/json": { "schema": { - "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse" + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" } } } @@ -2512,7 +3161,37 @@ "content": { "application/json": { "schema": { - "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse" + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" + } + } + } + }, + "404": { + "description": "Not Found", + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" + } + } + } + }, + "422": { + "description": "Unprocessable Entity", + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/ExtendedErrorResponse" + } + } + } + }, + "423": { + "description": "Locked", + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" } } } @@ -2547,7 +3226,7 @@ "content": { "application/json": { "schema": { - "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse" + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" } } } @@ -2557,7 +3236,7 @@ "content": { "application/json": { "schema": { - "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse" + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" } } } @@ -2607,6 +3286,24 @@ "title": "Db Flavor" }, "description": "Optional database flavor" + }, + { + "name": "source_name", + "in": "query", + "required": false, + "schema": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ], + "description": "Optional source name; defaults to the database type", + "title": "Source Name" + }, + "description": "Optional source name; defaults to the database type" } ], "responses": { @@ -2615,15 +3312,6 @@ "content": { "application/json": { "schema": { - "anyOf": [ - { - "$ref": "#/components/schemas/TemplateResponse" - }, - { - "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse" - } - ], - "title": "Response Pipelines Config Templates Api V1 Pipelines Config Templates Ingest Db Type Get", "$ref": "#/components/schemas/TemplateResponse" } } @@ -2634,7 +3322,7 @@ "content": { "application/json": { "schema": { - "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse" + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" } } } @@ -2644,7 +3332,7 @@ "content": { "application/json": { "schema": { - "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse" + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" } } } @@ -2653,7 +3341,7 @@ "content": { "application/json": { "schema": { - "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse" + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" } } }, @@ -2663,7 +3351,7 @@ "content": { "application/json": { "schema": { - "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse" + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" } } }, @@ -2678,22 +3366,38 @@ "summary": "Pipelines Jobs Templates", "description": "Returns the YAML templates describing how to create a job", "operationId": "pipelines_jobs_templates_api_v1_pipelines_jobs_templates_ingest_get", + "security": [ + { + "JWTBearer": [] + } + ], + "parameters": [ + { + "name": "source_name", + "in": "query", + "required": false, + "schema": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ], + "description": "Optional source name; defaults to a placeholder", + "title": "Source Name" + }, + "description": "Optional source name; defaults to a placeholder" + } + ], "responses": { "200": { "description": "Successful Response", "content": { "application/json": { "schema": { - "$ref": "#/components/schemas/TemplateResponse", - "anyOf": [ - { - "$ref": "#/components/schemas/TemplateResponse" - }, - { - "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse" - } - ], - "title": "Response Pipelines Jobs Templates Api V1 Pipelines Jobs Templates Ingest Get" + "$ref": "#/components/schemas/TemplateResponse" } } } @@ -2703,7 +3407,7 @@ "content": { "application/json": { "schema": { - "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse" + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" } } } @@ -2713,27 +3417,22 @@ "content": { "application/json": { "schema": { - "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse" + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" } } } }, - "501": { - "description": "Not Implemented", + "422": { "content": { "application/json": { "schema": { - "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse" + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" } } - } - } - }, - "security": [ - { - "JWTBearer": [] + }, + "description": "Unprocessable Entity" } - ] + } } }, "/api/v1/pipelines/jobs/dry-run": { @@ -2758,15 +3457,7 @@ "content": { "application/json": { "schema": { - "anyOf": [ - { - "$ref": "#/components/schemas/JobDryRunSuccessResponse" - }, - { - "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__Error" - } - ], - "title": "Response Job Dry Run Api V1 Pipelines Jobs Dry Run Post" + "$ref": "#/components/schemas/JobDryRunSuccessResponse" } } } @@ -2776,7 +3467,7 @@ "content": { "application/json": { "schema": { - "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse" + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" } } } @@ -2786,7 +3477,17 @@ "content": { "application/json": { "schema": { - "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse" + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" + } + } + } + }, + "400": { + "description": "Bad Request", + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" } } } @@ -2796,7 +3497,7 @@ "content": { "application/json": { "schema": { - "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__Error" + "$ref": "#/components/schemas/ExtendedErrorResponse" } } } @@ -2831,7 +3532,7 @@ "content": { "application/json": { "schema": { - "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse" + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" } } } @@ -2841,7 +3542,7 @@ "content": { "application/json": { "schema": { - "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse" + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" } } } @@ -2878,7 +3579,7 @@ "content": { "application/json": { "schema": { - "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse" + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" } } } @@ -2888,7 +3589,7 @@ "content": { "application/json": { "schema": { - "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse" + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" } } } @@ -2925,7 +3626,7 @@ "content": { "application/json": { "schema": { - "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse" + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" } } } @@ -2935,7 +3636,7 @@ "content": { "application/json": { "schema": { - "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse" + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" } } } @@ -2983,15 +3684,7 @@ "content": { "application/json": { "schema": { - "anyOf": [ - { - "$ref": "#/components/schemas/ActionIdResponse" - }, - { - "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__Error" - } - ], - "title": "Response Update Processors Api V1 Pipelines Processors Put" + "$ref": "#/components/schemas/ActionIdResponse" } } } @@ -3001,7 +3694,7 @@ "content": { "application/json": { "schema": { - "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse" + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" } } } @@ -3011,7 +3704,37 @@ "content": { "application/json": { "schema": { - "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse" + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" + } + } + } + }, + "404": { + "description": "Not Found", + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" + } + } + } + }, + "422": { + "description": "Unprocessable Entity", + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/ExtendedErrorResponse" + } + } + } + }, + "423": { + "description": "Locked", + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" } } } @@ -3084,15 +3807,7 @@ "content": { "application/json": { "schema": { - "anyOf": [ - { - "$ref": "#/components/schemas/ActionIdResponse" - }, - { - "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__Error" - } - ], - "title": "Response Upsert Processors Property Api V1 Pipelines Processors Prop Put" + "$ref": "#/components/schemas/ActionIdResponse" } } } @@ -3102,7 +3817,7 @@ "content": { "application/json": { "schema": { - "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse" + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" } } } @@ -3112,10 +3827,40 @@ "content": { "application/json": { "schema": { - "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse" + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" } } } + }, + "404": { + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" + } + } + }, + "description": "Not Found" + }, + "422": { + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/ExtendedErrorResponse" + } + } + }, + "description": "Unprocessable Entity" + }, + "423": { + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" + } + } + }, + "description": "Locked" } } } @@ -3168,7 +3913,7 @@ "content": { "application/json": { "schema": { - "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse" + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" } } } @@ -3178,7 +3923,17 @@ "content": { "application/json": { "schema": { - "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse" + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" + } + } + } + }, + "422": { + "description": "Use API v2 to manage sources and secrets of this pipeline.", + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" } } } @@ -3218,7 +3973,7 @@ "content": { "application/json": { "schema": { - "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse" + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" } } } @@ -3228,7 +3983,17 @@ "content": { "application/json": { "schema": { - "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse" + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" + } + } + } + }, + "422": { + "description": "Use API v2 to manage sources and secrets of this pipeline.", + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" } } } @@ -3263,7 +4028,7 @@ "content": { "application/json": { "schema": { - "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse" + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" } } } @@ -3273,7 +4038,17 @@ "content": { "application/json": { "schema": { - "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse" + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" + } + } + } + }, + "404": { + "description": "Not Found", + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" } } } @@ -3308,7 +4083,7 @@ "content": { "application/json": { "schema": { - "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse" + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" } } } @@ -3318,7 +4093,7 @@ "content": { "application/json": { "schema": { - "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse" + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" } } } @@ -3335,7 +4110,7 @@ "post": { "tags": ["secure"], "summary": "Start Trace", - "description": "Starts a trace session for troubleshooting data transformation.", + "description": "Starts a trace session for troubleshooting data transformation.\n\nTracing is no longer supported; the operator does not process trace tasks anymore.", "operationId": "start_trace_api_v1_trace_start_post", "requestBody": { "content": { @@ -3363,7 +4138,7 @@ "content": { "application/json": { "schema": { - "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse" + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" } } } @@ -3373,7 +4148,17 @@ "content": { "application/json": { "schema": { - "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__response__ErrorResponse" + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" + } + } + } + }, + "501": { + "description": "Not Implemented", + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/redis_di_api__v1__models__responses__ErrorResponse" } } } @@ -3450,14 +4235,7 @@ "in": "query", "required": false, "schema": { - "anyOf": [ - { - "type": "boolean" - }, - { - "type": "null" - } - ], + "type": "boolean", "description": "Only validate the provided configuration without making changes", "default": false, "title": "Dry Run" @@ -3551,6 +4329,26 @@ }, "description": "Unprocessable Entity" }, + "423": { + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/redis_di_api__models__errors__ErrorResponse" + } + } + }, + "description": "Locked" + }, + "502": { + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/redis_di_api__models__errors__ErrorResponse" + } + } + }, + "description": "Bad Gateway" + }, "503": { "content": { "application/json": { @@ -3560,6 +4358,16 @@ } }, "description": "Service Unavailable" + }, + "504": { + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/redis_di_api__models__errors__ErrorResponse" + } + } + }, + "description": "Gateway Timeout" } } } @@ -3634,7 +4442,7 @@ "put": { "tags": ["v2", "pipelines"], "summary": "Update Pipeline", - "description": "Updates a pipeline completely, or creates it if it doesn't exist.\nRetries 409 conflicts received from the K8s API server.", + "description": "Updates a pipeline completely, or creates it if it doesn't exist.", "operationId": "update_pipeline_api_v2_pipelines__name__put", "security": [ { @@ -3658,14 +4466,7 @@ "in": "query", "required": false, "schema": { - "anyOf": [ - { - "type": "boolean" - }, - { - "type": "null" - } - ], + "type": "boolean", "description": "Only validate the provided configuration without making changes", "default": false, "title": "Dry Run" @@ -3749,6 +4550,26 @@ }, "description": "Unprocessable Entity" }, + "423": { + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/redis_di_api__models__errors__ErrorResponse" + } + } + }, + "description": "Locked" + }, + "502": { + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/redis_di_api__models__errors__ErrorResponse" + } + } + }, + "description": "Bad Gateway" + }, "503": { "content": { "application/json": { @@ -3758,13 +4579,23 @@ } }, "description": "Service Unavailable" + }, + "504": { + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/redis_di_api__models__errors__ErrorResponse" + } + } + }, + "description": "Gateway Timeout" } } }, "patch": { "tags": ["v2", "pipelines"], "summary": "Patch Pipeline", - "description": "Updates a pipeline partially (only updates specified fields), with retries on 409 conflicts received from the K8s API server.", + "description": "Updates a pipeline partially (only updates specified fields).", "operationId": "patch_pipeline_api_v2_pipelines__name__patch", "security": [ { @@ -3788,14 +4619,7 @@ "in": "query", "required": false, "schema": { - "anyOf": [ - { - "type": "boolean" - }, - { - "type": "null" - } - ], + "type": "boolean", "description": "Only validate the provided configuration without making changes", "default": false, "title": "Dry Run" @@ -3889,6 +4713,26 @@ }, "description": "Unprocessable Entity" }, + "423": { + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/redis_di_api__models__errors__ErrorResponse" + } + } + }, + "description": "Locked" + }, + "502": { + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/redis_di_api__models__errors__ErrorResponse" + } + } + }, + "description": "Bad Gateway" + }, "503": { "content": { "application/json": { @@ -3898,13 +4742,23 @@ } }, "description": "Service Unavailable" + }, + "504": { + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/redis_di_api__models__errors__ErrorResponse" + } + } + }, + "description": "Gateway Timeout" } } }, "delete": { "tags": ["v2", "pipelines"], "summary": "Delete Pipeline", - "description": "Replaces a pipeline with an empty inactive pipeline and resets it.\n\nThis is done to ensure that no artifacts (offsets, schema, data streams etc.) are left behind,\nas this may cause errors and unexpected behavior when trying to re-create the pipeline.", + "description": "Replaces a pipeline with an empty inactive pipeline.", "operationId": "delete_pipeline_api_v2_pipelines__name__delete", "security": [ { @@ -3957,6 +4811,16 @@ } }, "description": "Not Found" + }, + "423": { + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/redis_di_api__models__errors__ErrorResponse" + } + } + }, + "description": "Locked" } } } @@ -4033,7 +4897,7 @@ "post": { "tags": ["v2", "pipelines"], "summary": "Start Pipeline", - "description": "Starts a pipeline, retrying on 409 conflicts received from the K8s API server.", + "description": "Starts a pipeline, or a single source when one is given.", "operationId": "start_pipeline_api_v2_pipelines__name__start_post", "security": [ { @@ -4051,6 +4915,24 @@ "title": "Name" }, "description": "Pipeline name" + }, + { + "name": "source", + "in": "query", + "required": false, + "schema": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ], + "description": "Start only this source; it runs only while the pipeline itself is running. Must be an existing source that is not of type 'external'.", + "title": "Source" + }, + "description": "Start only this source; it runs only while the pipeline itself is running. Must be an existing source that is not of type 'external'." } ], "responses": { @@ -4084,6 +4966,16 @@ }, "description": "Forbidden" }, + "400": { + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/redis_di_api__models__errors__ErrorResponse" + } + } + }, + "description": "Bad Request" + }, "404": { "content": { "application/json": { @@ -4093,6 +4985,16 @@ } }, "description": "Not Found" + }, + "423": { + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/redis_di_api__models__errors__ErrorResponse" + } + } + }, + "description": "Locked" } } } @@ -4101,7 +5003,7 @@ "post": { "tags": ["v2", "pipelines"], "summary": "Stop Pipeline", - "description": "Stops a pipeline, retrying on 409 conflicts received from the K8s API server.", + "description": "Stops a pipeline, or a single source when one is given.", "operationId": "stop_pipeline_api_v2_pipelines__name__stop_post", "security": [ { @@ -4119,6 +5021,24 @@ "title": "Name" }, "description": "Pipeline name" + }, + { + "name": "source", + "in": "query", + "required": false, + "schema": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ], + "description": "Stop only this source; the pipeline and its other sources keep running. Must be an existing source that is not of type 'external'.", + "title": "Source" + }, + "description": "Stop only this source; the pipeline and its other sources keep running. Must be an existing source that is not of type 'external'." } ], "responses": { @@ -4152,6 +5072,16 @@ }, "description": "Forbidden" }, + "400": { + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/redis_di_api__models__errors__ErrorResponse" + } + } + }, + "description": "Bad Request" + }, "404": { "content": { "application/json": { @@ -4161,6 +5091,16 @@ } }, "description": "Not Found" + }, + "423": { + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/redis_di_api__models__errors__ErrorResponse" + } + } + }, + "description": "Locked" } } } @@ -4169,7 +5109,7 @@ "post": { "tags": ["v2", "pipelines"], "summary": "Reset Pipeline", - "description": "Resets a pipeline, retrying on 409 conflicts received from the K8s API server.", + "description": "Resets a pipeline, or a single source when one is given.", "operationId": "reset_pipeline_api_v2_pipelines__name__reset_post", "security": [ { @@ -4186,7 +5126,25 @@ "description": "Pipeline name", "title": "Name" }, - "description": "Pipeline name" + "description": "Pipeline name" + }, + { + "name": "source", + "in": "query", + "required": false, + "schema": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ], + "description": "Reset only this source; the other sources keep their data. The whole pipeline is stopped while the reset runs and starts again afterwards. Must be an existing source that is not of type 'external'.", + "title": "Source" + }, + "description": "Reset only this source; the other sources keep their data. The whole pipeline is stopped while the reset runs and starts again afterwards. Must be an existing source that is not of type 'external'." } ], "responses": { @@ -4220,6 +5178,16 @@ }, "description": "Forbidden" }, + "400": { + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/redis_di_api__models__errors__ErrorResponse" + } + } + }, + "description": "Bad Request" + }, "404": { "content": { "application/json": { @@ -4229,6 +5197,16 @@ } }, "description": "Not Found" + }, + "423": { + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/redis_di_api__models__errors__ErrorResponse" + } + } + }, + "description": "Locked" } } } @@ -4380,6 +5358,16 @@ } }, "description": "Not Found" + }, + "422": { + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/redis_di_api__models__errors__ErrorResponse" + } + } + }, + "description": "Unprocessable Entity" } } } @@ -4710,7 +5698,7 @@ "get": { "tags": ["v2", "pipelines", "secrets"], "summary": "Get Pipeline Secrets", - "description": "Lists all pipeline secrets.", + "description": "Lists pipeline secrets, optionally filtered to a single database.", "operationId": "get_pipeline_secrets_api_v2_pipelines__name__secrets_get", "security": [ { @@ -4728,6 +5716,24 @@ "title": "Name" }, "description": "Pipeline name" + }, + { + "name": "db", + "in": "query", + "required": false, + "schema": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ], + "description": "If set, list only the secrets of this database.", + "title": "Db" + }, + "description": "If set, list only the secrets of this database." } ], "responses": { @@ -4798,6 +5804,36 @@ "title": "Name" }, "description": "Pipeline name" + }, + { + "name": "db", + "in": "query", + "required": false, + "schema": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ], + "description": "The database a secret belongs to: a source name, or 'target' for the target database. Omit for legacy SOURCE_DB_* and TARGET_DB_* keys.", + "title": "Db" + }, + "description": "The database a secret belongs to: a source name, or 'target' for the target database. Omit for legacy SOURCE_DB_* and TARGET_DB_* keys." + }, + { + "name": "validate_source", + "in": "query", + "required": false, + "schema": { + "type": "boolean", + "description": "When true, require the source to already exist in the pipeline.", + "default": false, + "title": "Validate Source" + }, + "description": "When true, require the source to already exist in the pipeline." } ], "requestBody": { @@ -4861,6 +5897,16 @@ } }, "description": "Conflict" + }, + "422": { + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/redis_di_api__models__errors__ErrorResponse" + } + } + }, + "description": "Unprocessable Entity" } } } @@ -4898,6 +5944,24 @@ "title": "Key" }, "description": "Pipeline secret key" + }, + { + "name": "db", + "in": "query", + "required": false, + "schema": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ], + "description": "The database a secret belongs to: a source name, or 'target' for the target database. Omit for legacy SOURCE_DB_* and TARGET_DB_* keys.", + "title": "Db" + }, + "description": "The database a secret belongs to: a source name, or 'target' for the target database. Omit for legacy SOURCE_DB_* and TARGET_DB_* keys." } ], "responses": { @@ -4940,6 +6004,16 @@ } }, "description": "Not Found" + }, + "422": { + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/redis_di_api__models__errors__ErrorResponse" + } + } + }, + "description": "Unprocessable Entity" } } }, @@ -4975,6 +6049,24 @@ "title": "Key" }, "description": "Pipeline secret key" + }, + { + "name": "db", + "in": "query", + "required": false, + "schema": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ], + "description": "The database a secret belongs to: a source name, or 'target' for the target database. Omit for legacy SOURCE_DB_* and TARGET_DB_* keys.", + "title": "Db" + }, + "description": "The database a secret belongs to: a source name, or 'target' for the target database. Omit for legacy SOURCE_DB_* and TARGET_DB_* keys." } ], "requestBody": { @@ -5073,6 +6165,24 @@ "title": "Key" }, "description": "Pipeline secret key" + }, + { + "name": "db", + "in": "query", + "required": false, + "schema": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ], + "description": "The database a secret belongs to: a source name, or 'target' for the target database. Omit for legacy SOURCE_DB_* and TARGET_DB_* keys.", + "title": "Db" + }, + "description": "The database a secret belongs to: a source name, or 'target' for the target database. Omit for legacy SOURCE_DB_* and TARGET_DB_* keys." } ], "responses": { @@ -5108,6 +6218,16 @@ } }, "description": "Not Found" + }, + "422": { + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/redis_di_api__models__errors__ErrorResponse" + } + } + }, + "description": "Unprocessable Entity" } } } @@ -5244,6 +6364,46 @@ } }, "description": "Unprocessable Entity" + }, + "423": { + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/redis_di_api__models__errors__ErrorResponse" + } + } + }, + "description": "Locked" + }, + "502": { + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/redis_di_api__models__errors__ErrorResponse" + } + } + }, + "description": "Bad Gateway" + }, + "503": { + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/redis_di_api__models__errors__ErrorResponse" + } + } + }, + "description": "Service Unavailable" + }, + "504": { + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/redis_di_api__models__errors__ErrorResponse" + } + } + }, + "description": "Gateway Timeout" } } } @@ -5334,6 +6494,16 @@ }, "description": "Not Found" }, + "422": { + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/redis_di_api__models__errors__ErrorResponse" + } + } + }, + "description": "Unprocessable Entity" + }, "502": { "content": { "application/json": { @@ -5353,6 +6523,16 @@ } }, "description": "Service Unavailable" + }, + "504": { + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/redis_di_api__models__errors__ErrorResponse" + } + } + }, + "description": "Gateway Timeout" } } } @@ -5373,6 +6553,16 @@ } } } + }, + "500": { + "description": "Internal Server Error", + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/redis_di_api__models__errors__ErrorResponse" + } + } + } } }, "security": [] @@ -5426,7 +6616,7 @@ "error": { "anyOf": [ { - "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__sources__Error" + "$ref": "#/components/schemas/redis_di_api__v1__models__sources__Error" }, { "type": "null" @@ -5499,8 +6689,7 @@ "collector-api", "stream-processor", "metrics-exporter", - "flink-processor-jobmanager", - "flink-processor-taskmanager", + "flink-collector", "flink-processor", "riotx-collector", "unknown" @@ -5517,7 +6706,7 @@ "error": { "anyOf": [ { - "$ref": "#/components/schemas/redis_di_api__v1__modules__shared__sources__Error" + "$ref": "#/components/schemas/redis_di_api__v1__models__sources__Error" }, { "type": "null" @@ -5884,7 +7073,6 @@ "DbType": { "type": "string", "enum": [ - "cassandra", "mariadb", "mongodb", "mysql", @@ -6089,54 +7277,29 @@ "title": "InfoResponse", "description": "Response model containing information about the API and operator." }, - "Job": { + "JobDryRunBody": { "properties": { - "source": { - "$ref": "#/components/schemas/Source" - }, - "transform": { - "anyOf": [ + "job": { + "additionalProperties": true, + "type": "object", + "title": "Job settings to dry run with.", + "examples": [ { - "items": { - "$ref": "#/components/schemas/Transform" + "source": { + "table": "employee" }, - "type": "array" - }, - { - "type": "null" - } - ], - "title": "Transform" - }, - "output": { - "items": { - "$ref": "#/components/schemas/Output" - }, - "type": "array", - "title": "Output" - }, - "name": { - "anyOf": [ - { - "type": "string" - }, - { - "type": "null" + "transform": [ + { + "uses": "add_field", + "with": { + "expression": "FNAME || ' ' || LAST_NAME", + "field": "FullName", + "language": "sql" + } + } + ] } - ], - "title": "Name", - "examples": ["Alex"] - } - }, - "type": "object", - "required": ["source", "output"], - "title": "Job" - }, - "JobDryRunBody": { - "properties": { - "job": { - "$ref": "#/components/schemas/Job", - "title": "Job settings to dry run with." + ] }, "input_data": { "items": { @@ -6200,6 +7363,19 @@ "title": "LeaderElectionMode", "description": "Operator leader-election mode." }, + "LegacySecretName": { + "type": "string", + "enum": [ + "source-db", + "target-db", + "rdi-sys-config", + "rdi-db-ssl", + "source-db-ssl", + "target-db-ssl" + ], + "title": "LegacySecretName", + "description": "The secret names API v1 accepts as the request key. They are kept for backward compatibility\nonly, since the K8s secret is derived from each key instead." + }, "MetadataResponse": { "properties": { "tables": { @@ -6240,35 +7416,6 @@ "title": "OperationResponse", "description": "Represents the response of an operation." }, - "Output": { - "properties": { - "uses": { - "type": "string", - "title": "Uses", - "examples": ["redis.write"] - }, - "with": { - "anyOf": [ - { - "$ref": "#/components/schemas/WithInput" - }, - { - "$ref": "#/components/schemas/WithFields" - }, - { - "$ref": "#/components/schemas/WithNest" - }, - { - "$ref": "#/components/schemas/WithConnection" - } - ], - "title": "With" - } - }, - "type": "object", - "required": ["uses", "with"], - "title": "Output" - }, "Pipeline": { "properties": { "status": { @@ -6312,6 +7459,17 @@ "type": { "$ref": "#/components/schemas/ComponentType" }, + "source": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ], + "title": "Source" + }, "version": { "type": "string", "title": "Version" @@ -6352,6 +7510,14 @@ } ], "title": "Metric Collections" + }, + "external_endpoints": { + "items": { + "type": "string" + }, + "type": "array", + "title": "External Endpoints", + "default": [] } }, "additionalProperties": false, @@ -6583,6 +7749,10 @@ "type": "string", "title": "Key" }, + "db": { + "type": "string", + "title": "Db" + }, "type": { "type": "string", "enum": ["simple", "file", "binary_file"], @@ -6591,7 +7761,7 @@ }, "additionalProperties": false, "type": "object", - "required": ["key", "type"], + "required": ["key", "db", "type"], "title": "PipelineSecretResponse", "description": "Response model for a pipeline secret." }, @@ -6917,25 +8087,12 @@ } ] }, - "SecretName": { - "type": "string", - "enum": [ - "source-db", - "target-db", - "rdi-sys-config", - "rdi-db-ssl", - "source-db-ssl", - "target-db-ssl" - ], - "title": "SecretName", - "description": "Defines supported secret names." - }, "SecretsModel": { "additionalProperties": { "$ref": "#/components/schemas/SecretKeyModel" }, "propertyNames": { - "$ref": "#/components/schemas/SecretName" + "$ref": "#/components/schemas/LegacySecretName" }, "type": "object", "title": "SecretsModel", @@ -6959,50 +8116,6 @@ "title": "SortOrder", "description": "Sort order for DLQ records." }, - "Source": { - "properties": { - "schema": { - "anyOf": [ - { - "type": "string" - }, - { - "type": "null" - } - ], - "title": "Schema", - "examples": ["public"] - }, - "table": { - "anyOf": [ - { - "type": "string" - }, - { - "type": "null" - } - ], - "title": "Table", - "examples": ["employee"] - }, - "row_format": { - "anyOf": [ - { - "type": "string", - "enum": ["full", "partial"] - }, - { - "type": "null" - } - ], - "title": "Row Format", - "examples": ["full"] - } - }, - "additionalProperties": true, - "type": "object", - "title": "Source" - }, "SourceColumnResponse": { "properties": { "type": { @@ -7312,6 +8425,7 @@ "stopping", "resetting", "pending", + "standby", "unknown" ], "title": "Status", @@ -7501,35 +8615,6 @@ "title": "TraceRequest", "description": "Request model for the trace endpoint." }, - "Transform": { - "properties": { - "uses": { - "type": "string", - "title": "Uses", - "examples": ["add_field"] - }, - "with": { - "anyOf": [ - { - "$ref": "#/components/schemas/WithInput" - }, - { - "$ref": "#/components/schemas/WithFields" - }, - { - "$ref": "#/components/schemas/WithNest" - }, - { - "$ref": "#/components/schemas/WithConnection" - } - ], - "title": "With" - } - }, - "type": "object", - "required": ["uses", "with"], - "title": "Transform" - }, "ValidationError": { "properties": { "loc": { @@ -7553,226 +8638,18 @@ "type": { "type": "string", "title": "Error Type" - } - }, - "type": "object", - "required": ["loc", "msg", "type"], - "title": "ValidationError" - }, - "WithConnection": { - "properties": { - "connection": { - "anyOf": [ - { - "type": "string" - }, - { - "type": "null" - } - ], - "title": "Connection", - "examples": ["target"] - }, - "data_type": { - "anyOf": [ - { - "type": "string" - }, - { - "type": "null" - } - ], - "title": "Data Type", - "examples": ["hash"] - }, - "args": { - "anyOf": [ - { - "additionalProperties": true, - "type": "object" - }, - { - "items": {}, - "type": "array" - }, - { - "type": "null" - } - ], - "title": "Args", - "examples": [ - { - "member": "value" - }, - ["expr1"] - ] - }, - "key": { - "anyOf": [ - { - "$ref": "#/components/schemas/WithInput" - }, - { - "type": "null" - } - ] - }, - "cmd": { - "anyOf": [ - { - "type": "string" - }, - { - "type": "null" - } - ], - "title": "Cmd", - "description": "Redis command to execute, e.g. HGETALL.", - "examples": ["HGETALL"] - }, - "language": { - "anyOf": [ - { - "type": "string" - }, - { - "type": "null" - } - ], - "title": "Language", - "examples": ["jmespath", "sql"] - }, - "field": { - "anyOf": [ - { - "type": "string" - }, - { - "type": "null" - } - ], - "title": "Field", - "description": "Target field name for the lookup result." - } - }, - "type": "object", - "title": "WithConnection" - }, - "WithFieldRef": { - "properties": { - "field": { - "type": "string", - "title": "Field", - "description": "Field name to reference or remove.", - "examples": ["branch"] - } - }, - "type": "object", - "required": ["field"], - "title": "WithFieldRef" - }, - "WithFields": { - "properties": { - "fields": { - "items": { - "anyOf": [ - { - "$ref": "#/components/schemas/WithInput" - }, - { - "$ref": "#/components/schemas/WithFieldRef" - } - ] - }, - "type": "array", - "title": "Fields" - } - }, - "type": "object", - "required": ["fields"], - "title": "WithFields" - }, - "WithInput": { - "properties": { - "expression": { - "type": "string", - "title": "Expression", - "examples": [ - "FNAME || ' ' || LAST_NAME", - "concat([fname, ' ' , last_name])" - ] }, - "field": { - "anyOf": [ - { - "type": "string" - }, - { - "type": "null" - } - ], - "title": "Field", - "description": "This field it available only for a subset of with expressions e.g. `add_field`, `remove_field`.", - "examples": ["FullName"] + "input": { + "title": "Input" }, - "language": { - "type": "string", - "title": "Language", - "examples": ["sql", "jmespath"] - } - }, - "type": "object", - "required": ["expression", "language"], - "title": "WithInput" - }, - "WithNest": { - "properties": { - "nest": { - "additionalProperties": true, + "ctx": { "type": "object", - "title": "Nest", - "description": "Nesting configuration for hierarchical data structures." - }, - "on_update": { - "anyOf": [ - { - "type": "string" - }, - { - "type": "null" - } - ], - "title": "On Update", - "examples": ["merge", "replace"] - }, - "data_type": { - "anyOf": [ - { - "type": "string" - }, - { - "type": "null" - } - ], - "title": "Data Type", - "examples": ["json", "hash"] - }, - "connection": { - "anyOf": [ - { - "type": "string" - }, - { - "type": "null" - } - ], - "title": "Connection", - "examples": ["target"] + "title": "Context" } }, "type": "object", - "required": ["nest"], - "title": "WithNest" + "required": ["loc", "msg", "type"], + "title": "ValidationError" }, "redis_di_api__models__errors__Error": { "properties": { @@ -7832,27 +8709,7 @@ "title": "ErrorResponse", "description": "Generic error response model" }, - "redis_di_api__v1__modules__shared__response__Error": { - "properties": { - "status": { - "type": "string", - "const": "error", - "title": "The status of the run.", - "default": "error", - "examples": ["error"] - }, - "error": { - "type": "string", - "title": "The error message.", - "examples": ["Job is malformed"] - } - }, - "type": "object", - "required": ["error"], - "title": "Error", - "description": "Model representing an error status." - }, - "redis_di_api__v1__modules__shared__response__ErrorResponse": { + "redis_di_api__v1__models__responses__ErrorResponse": { "properties": { "detail": { "type": "string", @@ -7865,7 +8722,7 @@ "title": "ErrorResponse", "description": "Model representing an API error response." }, - "redis_di_api__v1__modules__shared__sources__Error": { + "redis_di_api__v1__models__sources__Error": { "properties": { "message": { "type": "string", diff --git a/content/integrate/redis-data-integration/reference/cli/_index.md b/content/integrate/redis-data-integration/reference/cli/_index.md index c584fc120e..a6a5e5e7c5 100644 --- a/content/integrate/redis-data-integration/reference/cli/_index.md +++ b/content/integrate/redis-data-integration/reference/cli/_index.md @@ -86,6 +86,11 @@ Pipeline-scoped commands take the pipeline name as an optional positional argume take their own key or name as the positional argument and target the pipeline with the `-p` / `--pipeline` option, which also defaults to `default`. +There are two command options that select a particular database of the pipeline. The secret commands take `--db` with +a source name or `target` to address that database's secrets, and the `start`, `stop`, and `reset` commands take +`--source` with a source name to act on that source alone. See +[Multiple sources in one pipeline]({{< relref "/integrate/redis-data-integration/data-pipelines/multiple-sources" >}}) for more information. + The commands group as follows: - **Information**: [`info`]({{< relref "/integrate/redis-data-integration/reference/cli/redis-di-info" >}}). diff --git a/content/integrate/redis-data-integration/reference/cli/redis-di-delete-secret.md b/content/integrate/redis-data-integration/reference/cli/redis-di-delete-secret.md index d55224832c..e82e81f785 100644 --- a/content/integrate/redis-data-integration/reference/cli/redis-di-delete-secret.md +++ b/content/integrate/redis-data-integration/reference/cli/redis-di-delete-secret.md @@ -22,6 +22,7 @@ redis-di delete-secret [flags] | Option | Description | | :----------------- | :-------------------------------------------------------------------------------- | | `-p`, `--pipeline` | Pipeline to target (default `default`). | +| `--db` | Database the secret belongs to: a source name, or `target`. | | `--force` | Skip the confirmation prompt. | | `--wait` | Wait for the pipeline to reach the expected state (default `true`). | | `--timeout` | Maximum time to wait for the pipeline to reach the expected state (default `2m`). | @@ -32,5 +33,5 @@ This command also accepts the ## Example ```bash -redis-di delete-secret SOURCE_DB_CACERT --force +redis-di delete-secret CACERT --db mysql --force ``` diff --git a/content/integrate/redis-data-integration/reference/cli/redis-di-deploy.md b/content/integrate/redis-data-integration/reference/cli/redis-di-deploy.md index 9e66e27268..c21fec3834 100644 --- a/content/integrate/redis-data-integration/reference/cli/redis-di-deploy.md +++ b/content/integrate/redis-data-integration/reference/cli/redis-di-deploy.md @@ -29,7 +29,7 @@ The pipeline name is an optional argument that defaults to `default`. | `--dir` | Directory containing the pipeline configuration (default `.`). | | `--dry-run` | Validate the configuration without deploying. | | `--validate-tables` | Validate the configuration against the source and target databases (default `true`). | -| `--validate-cdc` | Validate the source database CDC configuration. | +| `--validate-cdc` | Validate the CDC configuration of the source databases. | | `--start` | Start the pipeline after deploying (default `true`). | | `--wait` | Wait for the pipeline to reach the expected state (default `true`). | | `--timeout` | Maximum time to wait for the pipeline to reach the expected state (default `2m`). | diff --git a/content/integrate/redis-data-integration/reference/cli/redis-di-describe-secret.md b/content/integrate/redis-data-integration/reference/cli/redis-di-describe-secret.md index 3ca2e2c7a2..e84fdd362f 100644 --- a/content/integrate/redis-data-integration/reference/cli/redis-di-describe-secret.md +++ b/content/integrate/redis-data-integration/reference/cli/redis-di-describe-secret.md @@ -19,9 +19,10 @@ redis-di describe-secret [flags] ## Options -| Option | Description | -| :----------------- | :-------------------------------------- | -| `-p`, `--pipeline` | Pipeline to target (default `default`). | +| Option | Description | +| :----------------- | :---------------------------------------------------------- | +| `-p`, `--pipeline` | Pipeline to target (default `default`). | +| `--db` | Database the secret belongs to: a source name, or `target`. | This command also accepts the [global options]({{< relref "/integrate/redis-data-integration/reference/cli/redis-di#global-options" >}}). @@ -29,5 +30,5 @@ This command also accepts the ## Example ```bash -redis-di describe-secret TARGET_DB_PASSWORD +redis-di describe-secret PASSWORD --db target ``` diff --git a/content/integrate/redis-data-integration/reference/cli/redis-di-describe.md b/content/integrate/redis-data-integration/reference/cli/redis-di-describe.md index 0ff5a63243..f71421c27a 100644 --- a/content/integrate/redis-data-integration/reference/cli/redis-di-describe.md +++ b/content/integrate/redis-data-integration/reference/cli/redis-di-describe.md @@ -42,7 +42,8 @@ To watch the status update live, pair the command with `watch`: watch -n 1 redis-di describe ``` -The output has a section for each part of the pipeline, for example: +The output has a section for each part of the pipeline. The example below shows a pipeline with +two sources, `mysql` and `postgresql`: ``` Name: default @@ -51,30 +52,43 @@ Status: started Current: yes Sources: - Name Type Db Type Connection Sync Mode Connected - ---- ---- ------- ---------- --------- --------- - mysql cdc mysql ${HOST_IP}:13000 streaming yes + Name Type Db Type Connection Sync Mode Connected + ---- ---- ------- ---------- --------- --------- + mysql cdc mysql :3306 streaming yes + postgresql cdc postgresql :5432 streaming yes Targets: - Name Db Type Connection Connected - ---- ------- ---------- --------- - target redis ${HOST_IP}:12000 yes + Name Db Type Connection Connected + ---- ------- ---------- --------- + target redis :12000 yes + +Processor: + Type: flink Jobs: - Name Source Transformations Outputs Connections - ---- ------ --------------- ------- ----------- - address_job inventory.addresses 1 1 target - customers_hash_job inventory.customers 0 1 target + Name Server Name Db / Schema Table Transformations Outputs Connections + ---- ----------- ----------- ----- --------------- ------- ----------- + billing_job postgresql public customers 1 1 target + customers_hash_job mysql inventory customers 0 1 target + orders_job mysql inventory orders 1 1 target Components: - Name Type Version Status - ---- ---- ------- ------ - collector-api collector-api 0.0.0 started - collector-source debezium-collector ... started - processor processor 0.0.0 started + Name Type Version Status Metrics Endpoints + ---- ---- ------- ------ ----------------- + collector-api collector-api started + collector-mysql debezium-collector started https://:443/collector-mysql/metrics + collector-postgresql debezium-collector started https://:443/collector-postgresql/metrics + flink-processor flink-processor started https://:443/processor/metrics Statistics: - Name Total Pending Inserted Updated Deleted Filtered Rejected Deduplicated Last Arrival - ---- ----- ------- -------- ------- ------- -------- -------- ------------ ------------ - {rdi}:inventory.customers 4 0 4 0 0 0 0 0 2026-06-18T13:42:44Z + Name Incoming Pending Inserted Updated Deleted Filtered Rejected Last Arrival + ---- -------- ------- -------- ------- ------- -------- -------- ------------ + mysql.inventory.customers 4 0 4 0 0 0 0 2026-06-18T13:42:44Z + mysql.inventory.orders 12 0 12 0 0 0 0 2026-06-18T13:42:51Z + postgresql.public.customers 7 0 7 0 0 0 0 2026-06-18T13:42:49Z ``` + +The `Metrics Endpoints` column shows the externally-reachable metrics URLs of each component, +which RDI discovers from the component's `Ingress` resources. It is empty for a component that +has no ingress, or while the ingress controller has not assigned an address yet. See +[Accessing the metrics]({{< relref "/integrate/redis-data-integration/observability#accessing-the-metrics" >}}) for more information. diff --git a/content/integrate/redis-data-integration/reference/cli/redis-di-get-dlq.md b/content/integrate/redis-data-integration/reference/cli/redis-di-get-dlq.md index 994ac42c9d..ae50b3a426 100644 --- a/content/integrate/redis-data-integration/reference/cli/redis-di-get-dlq.md +++ b/content/integrate/redis-data-integration/reference/cli/redis-di-get-dlq.md @@ -11,6 +11,9 @@ aliases: Gets a single dead-letter queue (DLQ) of a pipeline and prints it in the compact `list-dlqs` table format. +The name is the DLQ's source-qualified table name, `.`, as +reported by [`list-dlqs`]({{< relref "/integrate/redis-data-integration/reference/cli/redis-di-list-dlqs" >}}). + ## Usage ``` @@ -30,5 +33,5 @@ This command also accepts the ## Example ```bash -redis-di get-dlq inventory.customers +redis-di get-dlq mysql.inventory.customers ``` diff --git a/content/integrate/redis-data-integration/reference/cli/redis-di-get-metric-collection.md b/content/integrate/redis-data-integration/reference/cli/redis-di-get-metric-collection.md index 7a427c22d0..74716a4e6f 100644 --- a/content/integrate/redis-data-integration/reference/cli/redis-di-get-metric-collection.md +++ b/content/integrate/redis-data-integration/reference/cli/redis-di-get-metric-collection.md @@ -32,5 +32,6 @@ This command also accepts the ## Example ```bash -redis-di get-metric-collection processor -o json +redis-di get-metric-collection processor_metrics -o json +redis-di get-metric-collection collector-mysql_metrics -o json ``` diff --git a/content/integrate/redis-data-integration/reference/cli/redis-di-get-secret.md b/content/integrate/redis-data-integration/reference/cli/redis-di-get-secret.md index 0b94f16073..4b710d3e9b 100644 --- a/content/integrate/redis-data-integration/reference/cli/redis-di-get-secret.md +++ b/content/integrate/redis-data-integration/reference/cli/redis-di-get-secret.md @@ -20,10 +20,11 @@ redis-di get-secret [flags] ## Options -| Option | Description | -| :----------------- | :--------------------------------------------------- | -| `-p`, `--pipeline` | Pipeline to target (default `default`). | -| `-o`, `--output` | Output format: `table` (default), `json`, or `yaml`. | +| Option | Description | +| :----------------- | :---------------------------------------------------------- | +| `-p`, `--pipeline` | Pipeline to target (default `default`). | +| `--db` | Database the secret belongs to: a source name, or `target`. | +| `-o`, `--output` | Output format: `table` (default), `json`, or `yaml`. | This command also accepts the [global options]({{< relref "/integrate/redis-data-integration/reference/cli/redis-di#global-options" >}}). @@ -31,5 +32,5 @@ This command also accepts the ## Example ```bash -redis-di get-secret SOURCE_DB_USERNAME +redis-di get-secret USERNAME --db mysql ``` diff --git a/content/integrate/redis-data-integration/reference/cli/redis-di-list-dlq-records.md b/content/integrate/redis-data-integration/reference/cli/redis-di-list-dlq-records.md index 49b1744364..f36effacee 100644 --- a/content/integrate/redis-data-integration/reference/cli/redis-di-list-dlq-records.md +++ b/content/integrate/redis-data-integration/reference/cli/redis-di-list-dlq-records.md @@ -16,6 +16,9 @@ and paging with `--limit`, `--offset`, and `--sort-order`. The operation code is Use [`list-dlqs`]({{< relref "/integrate/redis-data-integration/reference/cli/redis-di-list-dlqs" >}}) to see all the pipeline's dead-letter queues and their record counts. +The DLQ name is its source-qualified table name, `.`, as +reported by [`list-dlqs`]({{< relref "/integrate/redis-data-integration/reference/cli/redis-di-list-dlqs" >}}). + ## Usage ``` @@ -48,8 +51,8 @@ This command also accepts the ```bash # Newest 20 rejected records of a queue -redis-di list-dlq-records inventory.customers +redis-di list-dlq-records mysql.inventory.customers # Oldest 100 records, as JSON -redis-di list-dlq-records inventory.customers --limit 100 --sort-order asc -o json +redis-di list-dlq-records mysql.inventory.customers --limit 100 --sort-order asc -o json ``` diff --git a/content/integrate/redis-data-integration/reference/cli/redis-di-list-dlqs.md b/content/integrate/redis-data-integration/reference/cli/redis-di-list-dlqs.md index 13b96386a7..82e6203849 100644 --- a/content/integrate/redis-data-integration/reference/cli/redis-di-list-dlqs.md +++ b/content/integrate/redis-data-integration/reference/cli/redis-di-list-dlqs.md @@ -13,6 +13,12 @@ that RDI rejected. Use [`list-dlq-records`]({{< relref "/integrate/redis-data-integration/reference/cli/redis-di-list-dlq-records" >}}) to read the records of a single queue. +There is one queue per source table, named `.`, where `` +is the source name from `config.yaml` and `` is the qualified table name, +such as `public.users` for PostgreSQL or `inventory.dbo.users` for SQL Server. See +[Rejected records]({{< relref "/integrate/redis-data-integration/data-pipelines/rejected-records#how-rdi-stores-rejected-records" >}}) +for the form each database uses. + ## Usage ``` diff --git a/content/integrate/redis-data-integration/reference/cli/redis-di-list-secrets.md b/content/integrate/redis-data-integration/reference/cli/redis-di-list-secrets.md index 14c64f63c5..aea6a1bb9c 100644 --- a/content/integrate/redis-data-integration/reference/cli/redis-di-list-secrets.md +++ b/content/integrate/redis-data-integration/reference/cli/redis-di-list-secrets.md @@ -19,10 +19,11 @@ redis-di list-secrets [flags] ## Options -| Option | Description | -| :----------------- | :--------------------------------------------------- | -| `-p`, `--pipeline` | Pipeline to target (default `default`). | -| `-o`, `--output` | Output format: `table` (default), `json`, or `yaml`. | +| Option | Description | +| :----------------- | :------------------------------------------------------------------ | +| `-p`, `--pipeline` | Pipeline to target (default `default`). | +| `--db` | List only the secrets of this database: a source name, or `target`. | +| `-o`, `--output` | Output format: `table` (default), `json`, or `yaml`. | This command also accepts the [global options]({{< relref "/integrate/redis-data-integration/reference/cli/redis-di#global-options" >}}). @@ -31,5 +32,6 @@ This command also accepts the ```bash redis-di list-secrets +redis-di list-secrets --db mysql redis-di list-secrets -p my-pipeline ``` diff --git a/content/integrate/redis-data-integration/reference/cli/redis-di-reset.md b/content/integrate/redis-data-integration/reference/cli/redis-di-reset.md index 14d627c0a4..2b96992e4f 100644 --- a/content/integrate/redis-data-integration/reference/cli/redis-di-reset.md +++ b/content/integrate/redis-data-integration/reference/cli/redis-di-reset.md @@ -13,6 +13,9 @@ Resets a pipeline into initial full-sync mode, so it reloads a snapshot of the s resuming change data capture. By default, the command waits for the pipeline to reach a terminal state before returning. +With `--source`, only the specified source is reset. See +[Start, stop, and reset a single source]({{< relref "/integrate/redis-data-integration/data-pipelines/multiple-sources#start-stop-and-reset-a-single-source" >}}) for more information. + ## Usage ``` @@ -25,6 +28,7 @@ The pipeline name is an optional argument that defaults to `default`. | Option | Description | | :---------- | :-------------------------------------------------------------------------------- | +| `--source` | Target only this source instead of the whole pipeline. | | `--wait` | Wait for the pipeline to reach the expected state (default `true`). | | `--timeout` | Maximum time to wait for the pipeline to reach the expected state (default `2m`). | @@ -35,4 +39,7 @@ This command also accepts the ```bash redis-di reset + +# Reset only source mysql +redis-di reset --source mysql ``` diff --git a/content/integrate/redis-data-integration/reference/cli/redis-di-scaffold.md b/content/integrate/redis-data-integration/reference/cli/redis-di-scaffold.md index 927acb678d..20aef77298 100644 --- a/content/integrate/redis-data-integration/reference/cli/redis-di-scaffold.md +++ b/content/integrate/redis-data-integration/reference/cli/redis-di-scaffold.md @@ -10,8 +10,9 @@ aliases: --- Generates a starter pipeline configuration for the given source database type. With `--dir`, the -command writes a `config.yaml` file into that directory, prompting before overwriting an existing -file unless `--force` is set. Without `--dir`, it prints the configuration to standard output. +command writes a `config.yaml` file and a `jobs/job.yaml` file into that directory, prompting +before overwriting existing files unless `--force` is set. Without `--dir`, it prints the +configuration to standard output. ## Usage @@ -21,12 +22,13 @@ redis-di scaffold [flags] ## Options -| Option | Description | -| :------------ | :------------------------------------------------------------------------------------------------------------------------------ | -| `--db-type` | Source database type (required): `mariadb`, `mongodb`, `mysql`, `oracle`, `postgresql`, `snowflake`, `sqlserver`, or `spanner`. | -| `--db-flavor` | Source database flavor: `mongodb-atlas`, `mongodb-replica-set`, or `mongodb-sharded-cluster`. | -| `--dir` | Directory to write `config.yaml` to; prints to standard output when omitted. | -| `--force` | Skip the confirmation prompt when overwriting an existing file. | +| Option | Description | +| :-------------- | :------------------------------------------------------------------------------------------------------------------------------ | +| `--db-type` | Source database type (required): `mariadb`, `mongodb`, `mysql`, `oracle`, `postgresql`, `snowflake`, `sqlserver`, or `spanner`. | +| `--db-flavor` | Source database flavor: `mongodb-atlas`, `mongodb-replica-set`, or `mongodb-sharded-cluster`. | +| `--source-name` | Name of the generated source; defaults to the database type. | +| `--dir` | Directory to write the configuration and job files to; prints the configuration to standard output when omitted. | +| `--force` | Skip the confirmation prompt when overwriting an existing file. | This command also accepts the [global options]({{< relref "/integrate/redis-data-integration/reference/cli/redis-di#global-options" >}}). @@ -39,4 +41,7 @@ redis-di scaffold --db-type postgresql # Write a MySQL configuration into a directory redis-di scaffold --db-type mysql --dir /opt/rdi/config + +# Name the generated source 'inventory' instead of 'mysql' +redis-di scaffold --db-type mysql --source-name inventory --dir /opt/rdi/config ``` diff --git a/content/integrate/redis-data-integration/reference/cli/redis-di-set-secret.md b/content/integrate/redis-data-integration/reference/cli/redis-di-set-secret.md index f82e5978db..e8e490730a 100644 --- a/content/integrate/redis-data-integration/reference/cli/redis-di-set-secret.md +++ b/content/integrate/redis-data-integration/reference/cli/redis-di-set-secret.md @@ -10,10 +10,13 @@ aliases: --- Creates or updates a secret of a pipeline. Secrets hold the credentials and certificates that the -pipeline uses to connect to the source and target databases (see +pipeline uses to connect to its source and target databases (see [Set secrets]({{< relref "/integrate/redis-data-integration/data-pipelines/deploy#set-secrets" >}}) -for the list of secret names). You can then refer to a secret in the `config.yaml` file with the -syntax `${SECRET_NAME}`. +for the secret keys). Every secret belongs to one database, which you name with `--db`: a source, or +`target`. RDI combines the key and the database into the environment variable that `config.yaml` +references, so `PASSWORD` with `--db mysql` becomes `${MYSQL_DB_PASSWORD}`. For a source that +existed before RDI supported multiple sources, the variable is still named `SOURCE_DB_PASSWORD`; see +[Existing names are kept after an upgrade]({{< relref "/integrate/redis-data-integration/data-pipelines/multiple-sources#existing-names-are-kept-after-an-upgrade" >}}). The secret value comes from the `[value]` argument, the `--file` option, or the `--literal` option. If you provide none of these on an interactive terminal, the command prompts for the value without @@ -30,6 +33,7 @@ redis-di set-secret [value] [flags] | Option | Description | | :----------------- | :-------------------------------------------------------------------------------- | | `-p`, `--pipeline` | Pipeline to target (default `default`). | +| `--db` | Database the secret belongs to: a source name, or `target`. | | `--file` | Read the secret value from the file at this path. | | `--literal` | Use this literal string as the secret value. | | `--wait` | Wait for the pipeline to reach the expected state (default `true`). | @@ -42,11 +46,14 @@ This command also accepts the ```bash # Value from an argument -redis-di set-secret SOURCE_DB_USERNAME myuser +redis-di set-secret USERNAME --db mysql myuser # Value from a file (for example, a certificate) -redis-di set-secret SOURCE_DB_CACERT --file /path/to/myca.crt +redis-di set-secret CACERT --db mysql --file /path/to/myca.crt # Value read from an interactive prompt -redis-di set-secret SOURCE_DB_PASSWORD +redis-di set-secret PASSWORD --db mysql + +# Target database credentials +redis-di set-secret PASSWORD --db target mypassword ``` diff --git a/content/integrate/redis-data-integration/reference/cli/redis-di-start.md b/content/integrate/redis-data-integration/reference/cli/redis-di-start.md index 23a5b9b124..53e61df7c5 100644 --- a/content/integrate/redis-data-integration/reference/cli/redis-di-start.md +++ b/content/integrate/redis-data-integration/reference/cli/redis-di-start.md @@ -24,6 +24,7 @@ The pipeline name is an optional argument that defaults to `default`. | Option | Description | | :---------- | :-------------------------------------------------------------------------------- | +| `--source` | Target only this source instead of the whole pipeline. | | `--wait` | Wait for the pipeline to reach the expected state (default `true`). | | `--timeout` | Maximum time to wait for the pipeline to reach the expected state (default `2m`). | @@ -35,4 +36,7 @@ This command also accepts the ```bash redis-di start redis-di start my-pipeline --wait=false + +# Start only source mysql. Does not start the pipeline if it's currently stopped. +redis-di start --source mysql ``` diff --git a/content/integrate/redis-data-integration/reference/cli/redis-di-stop.md b/content/integrate/redis-data-integration/reference/cli/redis-di-stop.md index 17b8dd5877..f2849cfe0f 100644 --- a/content/integrate/redis-data-integration/reference/cli/redis-di-stop.md +++ b/content/integrate/redis-data-integration/reference/cli/redis-di-stop.md @@ -24,6 +24,7 @@ The pipeline name is an optional argument that defaults to `default`. | Option | Description | | :---------- | :-------------------------------------------------------------------------------- | +| `--source` | Target only this source instead of the whole pipeline. | | `--wait` | Wait for the pipeline to reach the expected state (default `true`). | | `--timeout` | Maximum time to wait for the pipeline to reach the expected state (default `2m`). | @@ -34,4 +35,7 @@ This command also accepts the ```bash redis-di stop + +# Stop only source mysql +redis-di stop --source mysql ``` diff --git a/content/integrate/redis-data-integration/reference/config-yaml-reference.md b/content/integrate/redis-data-integration/reference/config-yaml-reference.md index a7a94416b1..cd352702ec 100644 --- a/content/integrate/redis-data-integration/reference/config-yaml-reference.md +++ b/content/integrate/redis-data-integration/reference/config-yaml-reference.md @@ -33,7 +33,7 @@ Source collectors that capture changes from upstream databases. Each key is a un |Name|Type|Description|Required| |----|----|-----------|--------| -|[**connection**](#sourcesconnection)
(Source database connection)|`object`|Connection configuration for a non-Redis source database. The exact set of properties depends on the database type.
|yes| +|[**connection**](#sourcesconnection)
(Source database connection)|`object`|Connection configuration for a non-Redis source database. The exact set of properties depends on the database type.
|no| |**name**
(Source name)|`string`|Human-readable name for the source collector. Maximum 100 characters.
Maximal Length: `100`
|no| |**type**
(Collector type)|`string`|Type of the source collector. Use `cdc` (default) for change data capture using [Debezium](https://debezium.io/). Use `flink` for Spanner change streams using the Apache Flink-based collector. Use `riotx` for Snowflake CDC using [RIOT-X](https://redis.github.io/riotx/).
Default: `"cdc"`
Enum: `"cdc"`, `"flink"`, `"riotx"`
|yes| |**active**
(Collector enabled)|`boolean`|When `true`, the collector runs; when `false`, the collector is disabled and produces no events.
Default: `true`
|no| @@ -44,6 +44,7 @@ Source collectors that capture changes from upstream databases. Each key is a un |[**advanced**](#sourcesadvanced)
(Advanced configuration)|`object`|Advanced configuration that overrides the underlying engine's defaults. Only required for non-standard tuning.
|no| +**Property Name Pattern:** `^[a-z]([a-z0-9-]*[a-z0-9])?$` ### sources\.connection: Source database connection @@ -238,7 +239,7 @@ Connection configuration for a Snowflake database. |**type**
(Database type)|`string`|Database type identifier. Always `snowflake` for this connection.
Constant Value: `"snowflake"`
|yes| |**url**
(JDBC URL)|`string`|Snowflake JDBC connection URL, for example `jdbc:snowflake://account.snowflakecomputing.com/`.
|yes| |**user**
(Snowflake user)|`string`|Username for authentication to Snowflake.
|yes| -|**password**
(Snowflake password)|`string`|Password for authentication to Snowflake. For key-pair authentication, omit this field and provide the private key via the `source-db-ssl` secret (`client.key` field).
|no| +|**password**
(Snowflake password)|`string`|Password for authentication to Snowflake. For key-pair authentication, omit this field and provide the private key via the `-db-ssl` secret (`client.key` field).
|no| |**database**
(Snowflake database)|`string`|Name of the Snowflake database to connect to.
|yes| |**warehouse**
(Snowflake warehouse)|`string`|Name of the Snowflake warehouse used for compute.
|yes| |**role**
(Snowflake role)|`string`|Snowflake role used for the connection.
|no| @@ -352,6 +353,7 @@ Advanced configuration that overrides the underlying engine's defaults. Only req |[**resources**](#sourcesadvancedresources)
(Collector resource settings)|`object`|Compute resources allocated to the collector. **Only applies to the `cdc` collector type.**
|| |[**riotx**](#sourcesadvancedriotx)
(Advanced RIOT\-X settings)|`object`|Advanced configuration properties for the RIOT-X Snowflake collector. **Only applies to the `riotx` collector type.**
|| |**java\_options**
(Advanced Java options)|`string`|These Java options will be passed to the command line command when launching the source collector. **Only applies to the `cdc` collector type.**
|| +|[**diagnostics**](#sourcesadvanceddiagnostics)
(Collector diagnostics)|`object`|Opt-in JVM diagnostics for the source collector. Each artifact is written to the `/opt/rdi/diagnostics` directory on the host and included in the `dump-support-package` bundle. **Only applies to the `cdc` collector type on VM installations, and is ignored otherwise.**
|| **Additional Properties:** not allowed **Minimal Properties:** 1 @@ -381,6 +383,8 @@ source: max.batch.size: 2048 max.queue.size: 8192 heartbeat.interval.ms: 0 + database.trustServerCertificate: false + driver.applicationIntent: ReadWrite lob.enabled: false publication.autocreate.mode: all_tables publication.name: dbz_publication @@ -400,6 +404,11 @@ riotx: streamPrefix: 'data:' clearOffset: false count: 0 +diagnostics: + heap_dump: + enabled: false + java_flight_recorder: + enabled: false ``` @@ -460,7 +469,7 @@ redis.wait.retry.delay.ms: 1000 #### sources\.advanced\.source: Advanced source settings -Advanced configuration properties for the source database connection and CDC behavior. **Applies to the `cdc` and `flink` collector types.**

For the `cdc` collector type, available properties depend on the source database — refer to the relevant Debezium connector documentation: [MySQL](https://debezium.io/documentation/reference/stable/connectors/mysql.html), [MariaDB](https://debezium.io/documentation/reference/stable/connectors/mariadb.html), [PostgreSQL](https://debezium.io/documentation/reference/stable/connectors/postgresql.html), [Oracle](https://debezium.io/documentation/reference/stable/connectors/oracle.html), [SQL Server](https://debezium.io/documentation/reference/stable/connectors/sqlserver.html), [Db2](https://debezium.io/documentation/reference/stable/connectors/db2.html), [MongoDB](https://debezium.io/documentation/reference/stable/connectors/mongodb.html). When using a property from those pages, omit the `debezium.source.` prefix.

**The named properties below cover the most commonly tuned settings: `spanner.*` properties apply to the `flink` collector type, all others apply to the `cdc` collector type. Any other property from the Debezium documentation can still be set as a free-form key-value pair.** +Advanced configuration properties for the source database connection and CDC behavior. **Applies to the `cdc` and `flink` collector types.**

For the `cdc` collector type, available properties depend on the source database — refer to the relevant Debezium connector documentation: [MySQL](https://debezium.io/documentation/reference/stable/connectors/mysql.html), [MariaDB](https://debezium.io/documentation/reference/stable/connectors/mariadb.html), [PostgreSQL](https://debezium.io/documentation/reference/stable/connectors/postgresql.html), [Oracle](https://debezium.io/documentation/reference/stable/connectors/oracle.html), [SQL Server](https://debezium.io/documentation/reference/stable/connectors/sqlserver.html), [Db2](https://debezium.io/documentation/reference/stable/connectors/db2.html), [MongoDB](https://debezium.io/documentation/reference/stable/connectors/mongodb.html). When using a property from those pages, omit the `debezium.source.` prefix.

**The named properties below cover the most commonly tuned settings: `spanner.*` properties apply to the `flink` collector type, all others apply to the `cdc` collector type. Any other property from the Debezium documentation can still be set as a free-form key-value pair, except for `topic.prefix`, which RDI derives from the source name, and the keystore properties, which RDI configures from a source's TLS secret.** **Properties** @@ -475,6 +484,8 @@ Advanced configuration properties for the source database connection and CDC beh |**max\.queue\.size**
(Max queue size)|`integer`|Limits how many records can be buffered in memory before processing catches up.
Default: `8192`
Minimum: `1`
|| |**heartbeat\.interval\.ms**
(Heartbeat interval ms)|`integer`|Sets how often heartbeat events are emitted to keep change tracking active. Use 0 to disable them.
Default: `0`
Minimum: `0`
|| |**heartbeat\.action\.query**
(Heartbeat action query)|`string`|SQL query executed on the source whenever a heartbeat is emitted.
|| +|**database\.trustServerCertificate**
(Trust SQL Server certificate)|`boolean`|Controls whether the RDI collector skips validation of the SQL Server TLS certificate. Enable this only when the server certificate cannot be validated through a trusted certificate authority.
Default: `false`
|| +|**driver\.applicationIntent**
(SQL Server application intent)|`string`|Declares the SQL Server workload type for the RDI collector. Set to `ReadOnly` when connecting to an Always On read-only replica. CDC must be enabled on the primary SQL Server node.
Default: `"ReadWrite"`
Enum: `"ReadOnly"`, `"ReadWrite"`
|| |**lob\.enabled**
(Lob enabled)|`boolean`|Determines whether large object columns are included in change capture.
Default: `false`
|| |**gtid\.source\.includes**
(GTID source includes)|`string`|Restricts MySQL GTID processing to the listed source UUIDs.
|| |**publication\.autocreate\.mode**
(Publication autocreate mode)|`string`|Controls whether and how the PostgreSQL publication is created or updated automatically.
Default: `"all_tables"`
Enum: `"all_tables"`, `"filtered"`, `"disabled"`
|| @@ -502,6 +513,8 @@ snapshot.fetch.size: 10000 max.batch.size: 2048 max.queue.size: 8192 heartbeat.interval.ms: 0 +database.trustServerCertificate: false +driver.applicationIntent: ReadWrite lob.enabled: false publication.autocreate.mode: all_tables publication.name: dbz_publication @@ -607,6 +620,74 @@ count: 0 Deprecated RIOTX global fallback list of columns to use as message keys for every captured table. Prefer `tables..keys` + +#### sources\.advanced\.diagnostics: Collector diagnostics + +Opt-in JVM diagnostics for the source collector. Each artifact is written to the `/opt/rdi/diagnostics` directory on the host and included in the `dump-support-package` bundle. **Only applies to the `cdc` collector type on VM installations, and is ignored otherwise.** + + +**Properties** + +|Name|Type|Description|Required| +|----|----|-----------|--------| +|[**heap\_dump**](#sourcesadvanceddiagnosticsheap_dump)
(Heap dump on out\-of\-memory)|`object`|Writes a heap dump if the JVM runs out of memory, then exits so the collector restarts cleanly.
|| +|[**java\_flight\_recorder**](#sourcesadvanceddiagnosticsjava_flight_recorder)
(Java Flight Recorder recording)|`object`|Records a Java Flight Recorder profile of the collector at startup. RDI controls `filename` and `dumponexit`, and defaults `settings=profile`, `duration=5m`, and `maxsize=128m` unless set under `options`.
|| + +**Additional Properties:** not allowed +**Minimal Properties:** 1 +**Example** + +```yaml +heap_dump: + enabled: false +java_flight_recorder: + enabled: false + +``` + + +##### sources\.advanced\.diagnostics\.heap\_dump: Heap dump on out\-of\-memory + +Writes a heap dump if the JVM runs out of memory, then exits so the collector restarts cleanly. + + +**Properties** + +|Name|Type|Description|Required| +|----|----|-----------|--------| +|**enabled**
(Enable heap dump)|`boolean`|When `true`, the collector writes a heap dump if the JVM runs out of memory, then exits so it restarts cleanly.
Default: `false`
|| + +**Additional Properties:** not allowed +**Minimal Properties:** 1 +**Example** + +```yaml +enabled: false + +``` + + +##### sources\.advanced\.diagnostics\.java\_flight\_recorder: Java Flight Recorder recording + +Records a Java Flight Recorder profile of the collector at startup. RDI controls `filename` and `dumponexit`, and defaults `settings=profile`, `duration=5m`, and `maxsize=128m` unless set under `options`. + + +**Properties** + +|Name|Type|Description|Required| +|----|----|-----------|--------| +|**enabled**
(Enable flight recording)|`boolean`|When `true`, starts a Java Flight Recorder recording when the collector starts.
Default: `false`
|| +|**options**
(Flight Recorder options)|`string`|Comma-separated `-XX:StartFlightRecording` options, for example `duration=10m,maxsize=256m,path-to-gc-roots=true`.
|| + +**Additional Properties:** not allowed +**Minimal Properties:** 1 +**Example** + +```yaml +enabled: false + +``` + ## targets: Target connections diff --git a/content/operate/rc/rdi/define.md b/content/operate/rc/rdi/define.md index bef1b571fa..102d2714dd 100644 --- a/content/operate/rc/rdi/define.md +++ b/content/operate/rc/rdi/define.md @@ -46,7 +46,7 @@ In the **Pipeline setup** step: - Snowflake _(Preview)_ {{}} -1. Enter a name for your source database in the **Source name** field. This is a name for the source database that will appear on Redis Cloud. +1. Enter a name for your source database in the **Source name** field. This is a display label for the source database on Redis Cloud. It does not affect the pipeline's configuration. 1. Select the target Redis Cloud database from the **Target database** list. {{}} diff --git a/content/operate/rc/rdi/quick-start.md b/content/operate/rc/rdi/quick-start.md index 298ae00591..944f26b0fb 100644 --- a/content/operate/rc/rdi/quick-start.md +++ b/content/operate/rc/rdi/quick-start.md @@ -45,7 +45,7 @@ Before you can create your first Data Integration pipeline for a Redis Cloud sub {{}} 1. Select **PostgreSQL** as the source database type. {{}} -1. Enter a name for your source database in the **Source name** field. This is a name for the source database that will appear on Redis Cloud. +1. Enter a name for your source database in the **Source name** field. This is a display label for the source database on Redis Cloud. It does not affect the pipeline's configuration. 1. Select **Continue to source** to move to the **Source configuration** step. {{}} diff --git a/content/operate/rc/rdi/setup.md b/content/operate/rc/rdi/setup.md index 64856ce15a..a743d92276 100644 --- a/content/operate/rc/rdi/setup.md +++ b/content/operate/rc/rdi/setup.md @@ -49,7 +49,7 @@ Before you can set up your source connectivity and secrets, you need the AWS Acc - MongoDB - Snowflake {{}} -1. Enter a name for your source database in the **Source name** field. This is a name for the source database that will appear on Redis Cloud. +1. Enter a name for your source database in the **Source name** field. This is a display label for the source database on Redis Cloud. It does not affect the pipeline's configuration. 1. Select **Continue to source** to move to the **Source configuration** step. {{}}