RDSC-5610: Multi-source documentation

content/embeds/rdi-tls-secrets.md

@@ -1,9 +1,9 @@
-When creating secrets for TLS or mTLS, ensure that all certificates and keys are in `PEM` format. The only exception to this is that for PostgreSQL, the private key `SOURCE_DB_KEY` secret must be in `DER` format. If you have a key in `PEM` format, you must convert it to `DER` before creating the `SOURCE_DB_KEY` secret using the command:
+When creating secrets for TLS or mTLS, ensure that all certificates and keys are in `PEM` format. The only exception to this is that for PostgreSQL, the source's private key secret must be in `DER` format. If you have a key in `PEM` format, you must convert it to `DER` before creating the secret using the command:
 
 ```bash
 openssl pkcs8 -topk8 -inform PEM -outform DER \
     -in /path/to/myclient.pem \
     -out /path/to/myclient.pk8 -nocrypt
 ```
 
-This command assumes that the private key is not encrypted. See the [`openssl` documentation](https://docs.openssl.org/master/) to learn how to convert an encrypted private key.
+This command assumes that the private key is not encrypted. See the [`openssl` documentation](https://docs.openssl.org/master/) to learn how to convert an encrypted private key.

content/embeds/rdi-when-to-use-dec-tree.md

@@ -32,21 +32,6 @@ questions:
                     label: "⚠️ Check deployment options to see if RDI is suitable for your needs before proceeding"
                     id: deploymentMismatch
                     sentiment: "indeterminate"
-            yes:
-                value: "Yes"
-                nextQuestion: singleSource
-    singleSource:
-        text: |
-            Are you transferring data from a single source database?
-        whyAsk: |
-            RDI is designed to work with a single source database. Multiple sources create conflicting change events.
-        answers:
-            no:
-                value: "No"
-                outcome:
-                    label: "❌ RDI won't work with multiple source databases"
-                    id: multipleSourcesOrActiveActive
-                    sentiment: "negative"
             yes:
                 value: "Yes"
                 nextQuestion: consistency

content/embeds/rdi-when-to-use.md

@@ -3,7 +3,7 @@
 RDI is a good fit when:
 
 - You want your app/micro-services to read from Redis to scale reads at speed.
-- You want to transfer data to Redis from a *single* source database.
+- You want to transfer data to Redis from one or more source databases.
 - You must use a slow database as the system of record for the app.
 - The app must always *write* its data to the slow database.
 - Your app can tolerate *eventual* consistency of data in the Redis cache.

content/integrate/redis-data-integration/architecture/_index.md

@@ -20,13 +20,13 @@ weight: 30
 
 ## Overview
 
-RDI implements a [change data capture](https://en.wikipedia.org/wiki/Change_data_capture) (CDC) pattern that tracks changes to the data in a
-non-Redis *source* database and makes corresponding changes to a Redis
+RDI implements a [change data capture](https://en.wikipedia.org/wiki/Change_data_capture) (CDC) pattern that tracks changes to the data in one or
+more non-Redis *source* databases and makes corresponding changes to a Redis
 *target* database. You can use the target as a cache to improve performance
 because it will typically handle read queries much faster than the source.
 
 To use RDI, you define a *dataset* that specifies which data items
-you want to capture from the source and how you want to
+you want to capture from each source and how you want to
 represent them in the target. For example, if the source is a
 relational database then you specify which table columns you want
 to capture but you don't need to store them in an equivalent table
@@ -35,15 +35,15 @@ representation is most suitable for your app. To convert from the
 source to the target representation, RDI applies *transformations*
 to the data after capture.
 
-RDI synchronizes the dataset between the source and target using
+RDI synchronizes the dataset between the sources and the target using
 a *data pipeline* that implements several processing steps
 in sequence:
 
-1.  A *CDC collector* captures changes to the source database. RDI
-    currently uses an open source collector called
+1.  A *CDC collector* per source captures changes to the source
+    databases. RDI currently uses an open source collector called
     [Debezium](https://debezium.io/) for this step.
 
-1.  The collector records the captured changes using
+1.  Each collector records the captured changes using
 [Redis streams]({{< relref "/develop/data-types/streams" >}})
     in the RDI database.
 
@@ -60,12 +60,12 @@ its state and configuration data and also the change data streams in a Redis dat
 {{< image filename="images/rdi/ingest/ingest-dataflow.webp" >}}
 
 When you first start RDI, the target database is empty and so all
-of the data in the source database is essentially "change" data.
+of the data in the source databases is essentially "change" data.
 RDI collects this data in a phase called *initial cache loading*,
 which can take minutes or hours to finish, depending on the size
 of the source data. Once the initial cache loading is complete,
 there is a *snapshot* dataset in the target that will gradually
-change when new data gets captured from the source. At this point,
+change when new data gets captured from the sources. At this point,
 RDI automatically enters a second phase called *change streaming*, where
 changes in the data are captured as they happen. Changes are usually
 added to the target within a few seconds after capture.
@@ -83,7 +83,7 @@ overall state).
 ## Checkpointing
 
 RDI uses Redis streams to store the sequence of change events
-captured from the source. The events are then retrieved in order
+captured from the sources. The events are then retrieved in order
 from the streams, processed, and written to the target. The stream
 processor uses a *checkpoint* mechanism to keep track of the last
 event in the sequence that it has successfully processed and stored. If the processor fails
@@ -96,11 +96,11 @@ face of failures.
 
 Sometimes, data records can get added to the streams faster than RDI can
 process them. This can happen if the target is slowed or disconnected
-or simply if the source quickly generates a lot of change data.
+or simply if a source quickly generates a lot of change data.
 If this continues, then the streams will eventually occupy all the
 available memory. When RDI detects this situation, it applies a
 *backpressure* mechanism to slow or stop the flow of incoming data.
-Change data is held at the source until RDI clears the backlog and has
+Change data is held at the sources until RDI clears the backlog and has
 enough free memory to resume streaming.
 
 {{<note>}}The Debezium log sometimes reports that RDI has run out
@@ -128,7 +128,7 @@ It includes:
     and exports them as [Prometheus](https://prometheus.io/) metrics.
 
 The *data plane* contains the processes that actually move the data.
-It includes the *CDC collector* and the *stream processor* that implement
+It includes a *CDC collector* for each source and the *stream processor*, which implement
 the two phases of the pipeline lifecycle (initial cache loading and change streaming).
 
 The *management plane* provides tools that let you interact
@@ -148,8 +148,8 @@ The diagram below shows all RDI components and the interactions between them:
 RDI provides two implementations of the stream processor, *classic* and
 *Flink*. You select the implementation per pipeline through the
 [`processors.type`]({{< relref "/integrate/redis-data-integration/data-pipelines/pipeline-config#processors" >}})
-property in `config.yaml`. The default is `classic`, so existing pipelines
-keep their behavior unchanged.
+property in `config.yaml`. The default is `flink`, so set `classic` explicitly to run a
+pipeline on the classic processor.
 
 See
 [Differences between the classic and Flink processors]({{< relref "/integrate/redis-data-integration/architecture/classic-vs-flink" >}})
@@ -164,7 +164,7 @@ deploy RDI.
 
 ### RDI on your own VMs
 
-For this deployment, you must provide two VMs. The collector and stream processor
+For this deployment, you must provide two VMs. The collectors and stream processor
 are active on one VM, while on the other they are in standby to provide high availability.
 The two operators running on both VMs use a leader election algorithm to decide which
 VM is the active one (the "leader").

content/integrate/redis-data-integration/architecture/classic-vs-flink.md

@@ -69,7 +69,7 @@ for VM installations.
 The two processors share the same `config.yaml` envelope and the same
 `connections`, `sources`, `targets`, and `jobs` sections. The only
 differences are inside the `processors:` block, which is selected via
-`processors.type` (`classic` or `flink`, default `classic`). Properties
+`processors.type` (`classic` or `flink`, default `flink`). Properties
 that apply to only one implementation are annotated with
 **Classic processor only.** or **Flink processor only.** in the
 [pipeline configuration reference]({{< relref "/integrate/redis-data-integration/data-pipelines/pipeline-config#processors" >}}),
@@ -102,6 +102,12 @@ for examples and
 [`redis.lookup`]({{< relref "/integrate/redis-data-integration/reference/data-transformation/lookup" >}})
 for the full property list.
 
+The Flink processor also accepts an advanced matcher syntax in a job's `source` section, where
+`server_name`, `db`, `schema`, and `table` each accept a list of names, and an entry prefixed
+with `regex:` selects all names that match the regular expression. One job can then process
+multiple tables, potentially from different sources, databases, or schemas. See
+[Job files]({{< relref "/integrate/redis-data-integration/data-pipelines/transform-examples" >}}) for more information.
+
 ## Metrics
 
 The two processors expose different Prometheus metric sets and use

content/integrate/redis-data-integration/data-pipelines/_index.md

@@ -28,9 +28,9 @@ deploy them.
 
 ## How a pipeline works
 
-An RDI pipeline captures change data records from the source database, and transforms them
-into Redis data structures. It writes each of these new structures to a Redis target
-database under its own key. 
+An RDI pipeline captures change data records from one or more source databases, and
+transforms them into Redis data structures. It writes each of these new structures to a
+Redis target database under its own key.
 
 By default, RDI transforms the source data into
 [hashes]({{< relref "/develop/data-types/hashes" >}}) or
@@ -94,6 +94,10 @@ to the snapshot phase. When this is complete, the pipeline continues with CDC as
 
 Follow the steps described in the sections below to prepare and run an RDI pipeline.
 
+The following example uses a single-source pipeline. See
+[Multiple sources in one pipeline]({{< relref "/integrate/redis-data-integration/data-pipelines/multiple-sources" >}})
+for details on how to capture multiple sources in one pipeline.
+
 ### 1. Prepare the source database
 
 Before using the pipeline you must first prepare your source database to use