RDSC-6038: Update Redis Cloud RDI documentation for 2.0.0 - #3997
Open
ZdravkoDonev-redis wants to merge 3 commits into
Open
RDSC-6038: Update Redis Cloud RDI documentation for 2.0.0#3997ZdravkoDonev-redis wants to merge 3 commits into
ZdravkoDonev-redis wants to merge 3 commits into
Conversation
Contributor
Contributor
🧠 Redis MemoryFound 6 related items from repository history:
Memory updated at a7bd017 |
2 tasks
| 1. On the [Redis Cloud console](https://cloud.redis.io/), open your target database's **Data Integration** tab and select **Add pipeline**. | ||
| 1. In **Settings**, select your target database and choose **Hash** or **JSON** as the default data structure, then select **Continue**. | ||
| 1. In **Add sources**, select **PostgreSQL**. | ||
| 1. Enter `postgresql` as the **Source name**. This is the source identifier used in the pipeline configuration and transformation jobs. |
Collaborator
Author
There was a problem hiding this comment.
Why telling the user what identified to put? Putting it this way seems like this should always be the name
| {{<image filename="images/rc/rdi/rdi-define-tls.png" alt="The Source database connectivity section, with TLS selected and the CA Cert Secret ARN field." >}} | ||
| - If your database requires mTLS, select **mTLS**. | ||
| {{<image filename="images/rc/rdi/rdi-define-mtls.png" alt="The Source database connectivity section, with mTLS selected and the Client Certificate Secret ARN and Client Key Secret ARN fields." >}} | ||
| For MySQL, MariaDB, and MongoDB sources, RDI 2.0.0 derives the Debezium keystore settings from the source certificate secrets. You do not need to add `database.ssl.keystore` or `mongodb.ssl.keystore` and their passwords to the advanced source properties. Explicit advanced settings still override the derived values. |
Collaborator
Author
There was a problem hiding this comment.
Skip this detail
| Use **Collector properties** for additional source and sink settings. These settings apply to the selected source. See the [collector source properties]({{< relref "/integrate/redis-data-integration/reference/config-yaml-reference#sourcesadvancedsource-advanced-source-settings" >}}) and [collector sink properties]({{< relref "/integrate/redis-data-integration/reference/config-yaml-reference#sourcesadvancedsink-rdi-collector-stream-writer-configuration" >}}). | ||
|
|
||
| You can add collector source properties in the **Collector source properties** section and collector sink properties in the **Collector sink properties** section. See the RDI configuration file reference for all available [collector source properties]({{< relref "/integrate/redis-data-integration/reference/config-yaml-reference#sourcesadvancedsource" >}}) and [collector sink properties]({{< relref "/integrate/redis-data-integration/reference/config-yaml-reference#sourcesadvancedsink" >}}). Select **Save properties** to return to Source configuration. | ||
| RDI derives `topic.prefix` from the source name. Do not set `topic.prefix` in the advanced source properties. |
Collaborator
Author
There was a problem hiding this comment.
Well... if we don't want it, we will just hide it from the UI. Remove this comment from the docs
|
|
||
| To let Redis Cloud access your source database, you need to create AWS secrets for the source database's credentials and certificates. | ||
|
|
||
| For MySQL, MariaDB, and MongoDB sources, RDI 2.0.0 builds the source keystore from these certificate secrets and configures Debezium automatically. Manual advanced keystore properties are optional overrides. |
Collaborator
Author
There was a problem hiding this comment.
Skip this sentence
| To manage a pipeline, select it from your workspace on the **Data Integration** page or from the **Data Integration** tab of your subscription or database. | ||
|
|
||
| The pipeline page has the following tabs: | ||
| The pipeline page has six tabs: |
Collaborator
Author
There was a problem hiding this comment.
Leave it without the number, so that we don't forget to update the nubmer when the below list changes.
| Deleting a source removes its data selection and internal RDI state, including streams, offsets, schema history, rejected records, and processing counters. Records already written to the target Redis database remain there. Other sources retain their data. The whole pipeline stops while RDI cleans up the removed source and starts again afterwards. | ||
|
|
||
| Changing the target database restarts the pipeline, but it does not automatically re-ingest all data into the new target database. If you want the existing data re-ingested after the change, manually [reset the pipeline](#reset-data-pipeline). | ||
| Source deletion cannot be undone. Review transformation-job source assignments after removing a source. |
Collaborator
Author
There was a problem hiding this comment.
If a transformation job referes to a source, the source cannot be deleted and the user will receive an error if he tries it.
| 1. Save the properties and review the restart confirmation before applying the changes. | ||
|
|
||
| {{<image filename="images/rc/rdi/rdi-select-columns.png" alt="The columns section, with a few columns selected from one table" width=75% >}} | ||
| Do not set `topic.prefix`; RDI derives it from the source name. For MySQL, MariaDB, and MongoDB mTLS connections, RDI derives keystore settings from the source's certificate secrets unless you explicitly override them. See [Configure source secrets]({{< relref "/operate/rc/rdi/define#secrets" >}}). |
Collaborator
Author
There was a problem hiding this comment.
Skip the topic for topic.prefix. In general we'll remove it as the user doesn't need to be aware or bother about it.
ZdravkoDonev-redis
marked this pull request as ready for review
September 14, 2026 00:18
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Updates the Cloud RDI guides for the RDI 2.0.0 console. The current guides describe one source and the older five-step setup. The updated guides describe several sources feeding one Redis target, the six-step creation process, and source-level management.
Ticket: RDSC-6038
Companion release notes: #3996.
Changes
Preview and screenshots
Captured from the supplied k8s-idm preview for cloud-ui PR #3050, head
0c5d44ce1ffeaef48682f909de4781cf16acfbcb, using@rdi-ui/pipeline0.1.18.All 13 new console screenshots were retaken in Light mode to match the existing Cloud documentation screenshots. The preview query was preserved during navigation. Screenshots use the existing
defaultsample pipeline with one PostgreSQL and two MySQL sources. They show the workspace Sources column, dashboard, source cards/actions, configuration, dataset, transformations, metrics, settings, add-source wizard, and reset/delete confirmations. Captures exclude account chrome, billing, connection addresses, and secret ARNs. Preview URLs and environment IDs are not included in the public guides.No source was started, stopped, reset, or deleted, and no pipeline was deployed. Reset/delete confirmations were canceled, and the pipeline and all three sources remained streaming. Start behavior and naming/assignment rules were cross-checked against the source and Core documentation; destructive behavior was not exercised.
The restored screenshots show Add pipeline, target selection, connectivity and role ARNs, credentials, TLS/mTLS fields, collector and processor properties, schema/table/column selection, missing-key warnings, key selection, Edit, and Deploy pipeline. Their image files are unchanged. The existing controls were checked against the preview and its source labels; the collector-properties dialog and dataset editing were opened without saving changes. Screenshots with obsolete wizard buttons, the old workspace source columns, or transformation settings that moved to Settings remain replaced. The old target-change confirmation remains excluded because its claim of automatic re-ingestion conflicts with current behavior.
Coordination with #3916
#3916 remains unchanged. It owns the Core API, CLI, schema, and multi-source documentation.
Its Cloud edits to
define.md,setup.md, andquick-start.mddescribe Source name as a display label. This PR replaces that wording for the new console, where the name identifies the source in configuration and jobs. After #3916 merges, rebase this PR and retain this new-console wording in those three overlapping pages. A temporary combined checkout was built with #3916's other changes intact and these Cloud pages applied over its three display-label edits.Validation
79ab2f71ca0ebb35a7df9da1d31f92be284f7b7d, and both RDSC-6038 drafts: passed.git diff --check: passed.jupyter-examplemissing-example/source-file errors in the available generated data. No example-generation files are changed here.Publication hold
Keep this PR in draft and do not merge until:
Screenshots are prepared now from the preview; documentation publication waits for the console release.
Note
Low Risk
Documentation-only changes to Redis Cloud RDI guides; no application code, auth, or data-handling logic is modified.
Overview
Rewrites the Redis Cloud Data Integration guides for the RDI 2.0.0 console: pipelines are documented as multiple sources → one shared Redis target, with a six-step creation flow (Settings → Add sources → Configure source → Select data → Add transformations → Review & deploy) instead of the older single-source, five-step wizard.
The updates span overview, workspace, setup, pipeline creation, quick start, and management pages. They explain per-source connectivity, secrets, collectors, datasets, and metrics; source names as configuration/job identifiers (not display labels); transformation source assignments; and how to preserve names/assignments when extending legacy single-source pipelines. Flink is called out as the default processor for new 2.0.0 pipelines, with notes on derived MySQL/MariaDB/MongoDB mTLS keystore settings.
View and edit coverage adds source lifecycle actions (Add source, Pending setup, start/stop/reset/delete per source) and a pipeline-wide Settings tab, plus refreshed console screenshots and quick-start Terraform variable wording for the two Role ARNs.
Reviewed by Cursor Bugbot for commit a7bd017. Bugbot is set up for automated code reviews on this repo. Configure here.