RDSC-6038: Add RDI 2.0.0 release notes - #3996
Open
ZdravkoDonev-redis wants to merge 1 commit into
Open
Conversation
Contributor
🧠 Redis MemoryFound 5 related items from repository history (5 new this commit):
Memory updated at 78ba113 |
Contributor
stoyanr
suggested changes
Sep 14, 2026
stoyanr
left a comment
stoyanr
left a comment
Contributor
There was a problem hiding this comment.
Many minor comments, mostly polishing of AI wording, and a few not so minor ones regarding ordering most important changes first and dropping internal details.
| - A request now applies its changes immediately rather than waiting behind earlier in-flight requests, so the most recent request wins if several are issued in quick succession without waiting for each to complete. Polling `GET /actions/{action_id}` for an action id that a later request has superseded now returns an unknown action error instead of that action's own status. | ||
| - `POST /pipelines` and `PATCH /pipelines` now also validate target connectivity similarly to their API v2 counterparts, and surface collector API failures as a `422`, `502`, `503`, or `504` error response. A request that previously succeeded despite an unreachable target now fails with a validation error. | ||
| - **API v1 trace endpoint returns Not Implemented**: `POST /trace/start` now immediately returns `501 Not Implemented` instead of accepting the request and queuing a trace that would never run. | ||
| - **API v2 pipeline responses omit optional fields that have no value**: The pipeline, pipeline status, create, update, patch, start, stop, and reset responses no longer carry an optional field whose value is null; the field is left out instead. This affects `status_changed_at` and an error's `remediation`. The OpenAPI schema is unchanged, since neither field was ever required, so clients generated from it are unaffected; a client that reads either key directly has to treat it as absent rather than null. |
Contributor
There was a problem hiding this comment.
Suggested change
| - **API v2 pipeline responses omit optional fields that have no value**: The pipeline, pipeline status, create, update, patch, start, stop, and reset responses no longer carry an optional field whose value is null; the field is left out instead. This affects `status_changed_at` and an error's `remediation`. The OpenAPI schema is unchanged, since neither field was ever required, so clients generated from it are unaffected; a client that reads either key directly has to treat it as absent rather than null. | |
| - **API v2 pipeline responses omit optional fields that have no value**: The pipeline, pipeline status, create, update, patch, start, stop, and reset responses no longer contain optional fields with null values; such fields are left out instead. This affects `status_changed_at` and an error's `remediation`. The OpenAPI schema is unchanged, since neither field was ever required, so clients generated from it are unaffected; a client that reads either key directly has to treat it as absent rather than null. |
| - `POST /pipelines` and `PATCH /pipelines` now also validate target connectivity similarly to their API v2 counterparts, and surface collector API failures as a `422`, `502`, `503`, or `504` error response. A request that previously succeeded despite an unreachable target now fails with a validation error. | ||
| - **API v1 trace endpoint returns Not Implemented**: `POST /trace/start` now immediately returns `501 Not Implemented` instead of accepting the request and queuing a trace that would never run. | ||
| - **API v2 pipeline responses omit optional fields that have no value**: The pipeline, pipeline status, create, update, patch, start, stop, and reset responses no longer carry an optional field whose value is null; the field is left out instead. This affects `status_changed_at` and an error's `remediation`. The OpenAPI schema is unchanged, since neither field was ever required, so clients generated from it are unaffected; a client that reads either key directly has to treat it as absent rather than null. | ||
| - **A source can no longer set its own `topic.prefix`**: The `topic.prefix` property is now rejected in a source's `advanced.source` section, since RDI derives the topic prefix from the source name. Remove the property from your configuration and set each job's `server_name` to the derived prefix: the source name, `rdi` for the source of an upgraded single-source pipeline handled by the Debezium collector, and the Spanner instance ID for the source of such an upgraded pipeline handled by the Flink collector. |
Contributor
There was a problem hiding this comment.
Suggested change
| - **A source can no longer set its own `topic.prefix`**: The `topic.prefix` property is now rejected in a source's `advanced.source` section, since RDI derives the topic prefix from the source name. Remove the property from your configuration and set each job's `server_name` to the derived prefix: the source name, `rdi` for the source of an upgraded single-source pipeline handled by the Debezium collector, and the Spanner instance ID for the source of such an upgraded pipeline handled by the Flink collector. | |
| - **`topic.prefix` can is no longer accepted in `advanced.source`**: The `topic.prefix` property is now rejected in a source's `advanced.source` section, since RDI derives the topic prefix from the source name. Remove the property from your configuration and set each job's `server_name` to the derived prefix: the source name, `rdi` for the source of an upgraded single-source pipeline handled by the Debezium collector, and the Spanner instance ID for the source of such an upgraded pipeline handled by the Flink collector. |
| - **API v1 trace endpoint returns Not Implemented**: `POST /trace/start` now immediately returns `501 Not Implemented` instead of accepting the request and queuing a trace that would never run. | ||
| - **API v2 pipeline responses omit optional fields that have no value**: The pipeline, pipeline status, create, update, patch, start, stop, and reset responses no longer carry an optional field whose value is null; the field is left out instead. This affects `status_changed_at` and an error's `remediation`. The OpenAPI schema is unchanged, since neither field was ever required, so clients generated from it are unaffected; a client that reads either key directly has to treat it as absent rather than null. | ||
| - **A source can no longer set its own `topic.prefix`**: The `topic.prefix` property is now rejected in a source's `advanced.source` section, since RDI derives the topic prefix from the source name. Remove the property from your configuration and set each job's `server_name` to the derived prefix: the source name, `rdi` for the source of an upgraded single-source pipeline handled by the Debezium collector, and the Spanner instance ID for the source of such an upgraded pipeline handled by the Flink collector. | ||
| - **API v2 metric collections name data streams after their table**: The `data_streams.streams` keys of a metric collection are now the source-qualified table name (`mysql.inventory.addresses`), matching what the dead-letter queue endpoints return, instead of the Redis stream name (`{rdi}:inventory.addresses`). A client that keys off the previous form has to be updated. API v1 statistics are unchanged. |
Contributor
There was a problem hiding this comment.
Suggested change
| - **API v2 metric collections name data streams after their table**: The `data_streams.streams` keys of a metric collection are now the source-qualified table name (`mysql.inventory.addresses`), matching what the dead-letter queue endpoints return, instead of the Redis stream name (`{rdi}:inventory.addresses`). A client that keys off the previous form has to be updated. API v1 statistics are unchanged. | |
| - **API v2 metric collections data stream names contain the qualified table name**: The `data_streams.streams` keys of a metric collection are now the source-qualified table name (`mysql.inventory.addresses`), matching what the dead-letter queue endpoints return, instead of the Redis stream name (`{rdi}:inventory.addresses`). A client that uses the previous form has to be updated. API v1 statistics are unchanged. |
| - **A source can no longer set its own `topic.prefix`**: The `topic.prefix` property is now rejected in a source's `advanced.source` section, since RDI derives the topic prefix from the source name. Remove the property from your configuration and set each job's `server_name` to the derived prefix: the source name, `rdi` for the source of an upgraded single-source pipeline handled by the Debezium collector, and the Spanner instance ID for the source of such an upgraded pipeline handled by the Flink collector. | ||
| - **API v2 metric collections name data streams after their table**: The `data_streams.streams` keys of a metric collection are now the source-qualified table name (`mysql.inventory.addresses`), matching what the dead-letter queue endpoints return, instead of the Redis stream name (`{rdi}:inventory.addresses`). A client that keys off the previous form has to be updated. API v1 statistics are unchanged. | ||
| - **Cassandra is no longer a source database type**: `cassandra` has been removed from the source database types that `redis-di scaffold`, the configuration template endpoint, and `rdi-admin install` offer, and a source whose connection type is `cassandra` no longer passes validation. RDI never actually supported Cassandra as source, since the Debezium Cassandra connector has to run on each Cassandra node. | ||
| - **The Flink processor is the default processor**: A pipeline whose `processors` section does not set `type` now deploys the Flink processor instead of the classic one. Set `processors.type` to `classic` to keep deploying the classic processor. |
Contributor
There was a problem hiding this comment.
I would move this to the top of the list, as it's the most important by far.
|
|
||
| ### New Features | ||
|
|
||
| - **Multiple sources in one pipeline**: A pipeline can ingest data from several source databases, of the same or different types, into one Redis target. Each source has its own name, connection settings, credentials, and collector. Use API v2 or the `redis-di` CLI to manage multi-source pipelines; API v1 supports only single-source pipelines. |
Contributor
There was a problem hiding this comment.
I would add a link to the new multiple-sources.md docs page (integrate/redis-data-integration/data-pipelines/multiple-sources/).
| - **Pipeline components report their externally-reachable endpoints**: Each component in an API v2 pipeline or pipeline status response now has an `external_endpoints` field listing the URLs at which that component is reachable from outside the cluster, discovered from its Ingress resources, in addition to the existing internal `metrics_endpoints`. | ||
| - **Source mTLS without Debezium keystore settings**: A MySQL, MariaDB, or MongoDB pipeline now presents the source client certificate without setting `database.ssl.keystore`/`mongodb.ssl.keystore` and the matching password in `advanced.source`. The collector points the connector at the keystore RDI builds from the source's certificate secrets, whether stored as pipeline secrets or supplied by a secret provider, and explicit `advanced.source` settings keep overriding the derived values. | ||
| - **Reset a single source of a multi-source pipeline**: `POST /pipelines/{name}/reset` accepts an optional `source` query parameter and `redis-di reset` an optional `--source` flag, naming one existing source that is not of type `external`. The reset then deletes only that source's data, including its change streams, offsets, schema history, dead letter queue entries, statistics, deduplication state, and record counters, and leaves every other source's data intact. The whole pipeline stops while the reset runs and starts again afterwards, as it already does for full pipeline reset. | ||
| - **Removing a source from a pipeline deletes its internal RDI data**: Removing a source with `PUT` or `PATCH /pipelines/{name}` now deletes the internal RDI data that source leaves behind, including its change streams, offsets, schema history, dead letter queue entries, statistics, deduplication state, and record counters. That data used to be kept indefinitely, and only resetting the whole pipeline removed it. The other sources keep their data, and the whole pipeline stops while the removed source's data is deleted and starts again afterwards. Records already written to the target Redis database are retained. |
Contributor
There was a problem hiding this comment.
Suggested change
| - **Removing a source from a pipeline deletes its internal RDI data**: Removing a source with `PUT` or `PATCH /pipelines/{name}` now deletes the internal RDI data that source leaves behind, including its change streams, offsets, schema history, dead letter queue entries, statistics, deduplication state, and record counters. That data used to be kept indefinitely, and only resetting the whole pipeline removed it. The other sources keep their data, and the whole pipeline stops while the removed source's data is deleted and starts again afterwards. Records already written to the target Redis database are retained. | |
| - **Removing a source from a pipeline deletes its internal RDI data**: Removing a source with `PUT` or `PATCH /pipelines/{name}` now deletes its internal RDI data, including its change streams, offsets, schema history, dead letter queue entries, statistics, deduplication state, and record counters. That data used to be kept indefinitely, and only resetting the pipeline removed it. Records already written to the target Redis database are not deleted. |
| - **RDI database client certificates isolated from source certificates**: The Debezium collector now presents its RDI database client certificate from a dedicated keystore instead of the shared default keystore, so the source database and RDI database client identities can no longer interfere with each other under mTLS. | ||
| - **Nested processor properties shown incorrectly in `redis-di describe`**: Nested objects and arrays in a processor's advanced properties are now shown as JSON instead of a raw internal format, so the `redis-di describe` output is valid and easy to read. | ||
| - **Bulk insert right after a snapshot could restart the Flink processor (Flink processor)**: A bulk insert arriving just after a snapshot completed could crash the Flink job with `Have records for a split that was not registered` while the source transitioned from snapshot to CDC processing. The job recovered automatically, but the pending change records were delayed until the restarted job claimed them. The source reader now skips record batches of splits that finished during the transition instead of failing on them. | ||
| - **Stream could stop being processed after a Flink processor restart (Flink processor)**: A narrow timing window around a checkpoint could leave a stream permanently unassigned to any reader after a restart (for example following a TaskManager failure), silently halting ingestion for that stream until the pipeline was redeployed. The processor now confirms with every reader before treating a stream as no longer needed. |
Contributor
There was a problem hiding this comment.
Suggested change
| - **Stream could stop being processed after a Flink processor restart (Flink processor)**: A narrow timing window around a checkpoint could leave a stream permanently unassigned to any reader after a restart (for example following a TaskManager failure), silently halting ingestion for that stream until the pipeline was redeployed. The processor now confirms with every reader before treating a stream as no longer needed. | |
| - **A stream could stop being processed after a Flink processor restart (Flink processor)**: A narrow timing window around a checkpoint could leave a stream permanently unassigned to any reader after a restart (for example following a TaskManager failure), silently halting ingestion for that stream until the pipeline was redeployed. The processor now confirms with every reader before treating a stream as no longer needed. |
| - **Bulk insert right after a snapshot could restart the Flink processor (Flink processor)**: A bulk insert arriving just after a snapshot completed could crash the Flink job with `Have records for a split that was not registered` while the source transitioned from snapshot to CDC processing. The job recovered automatically, but the pending change records were delayed until the restarted job claimed them. The source reader now skips record batches of splits that finished during the transition instead of failing on them. | ||
| - **Stream could stop being processed after a Flink processor restart (Flink processor)**: A narrow timing window around a checkpoint could leave a stream permanently unassigned to any reader after a restart (for example following a TaskManager failure), silently halting ingestion for that stream until the pipeline was redeployed. The processor now confirms with every reader before treating a stream as no longer needed. | ||
| - **Unreachable source database reported as a validation error**: When RDI cannot list the source tables during pipeline validation, for example because the source database refuses the connection, the failure is now reported as a `422` validation error in the `errors` array, instead of a `422`, `502`, or `504` response carrying only a `detail` message. An unavailable collector API is still reported as a `503` response so the request can be retried. | ||
| - **No configuration template for the Redis database type**: `GET /pipelines/config/templates/ingest/redis` now returns the documented `501 Not Implemented`, instead of a `200` response carrying a configuration that named `redis` as the source connection type and left the source port empty. Redis cannot be used as a pipeline source. |
Contributor
There was a problem hiding this comment.
Suggested change
| - **No configuration template for the Redis database type**: `GET /pipelines/config/templates/ingest/redis` now returns the documented `501 Not Implemented`, instead of a `200` response carrying a configuration that named `redis` as the source connection type and left the source port empty. Redis cannot be used as a pipeline source. | |
| - **No configuration template for the Redis database type**: `GET /pipelines/config/templates/ingest/redis` now returns the documented `501 Not Implemented`, instead of a `200` response with an incorrect configuration. Redis cannot be used as a pipeline source. |
| - **Stream could stop being processed after a Flink processor restart (Flink processor)**: A narrow timing window around a checkpoint could leave a stream permanently unassigned to any reader after a restart (for example following a TaskManager failure), silently halting ingestion for that stream until the pipeline was redeployed. The processor now confirms with every reader before treating a stream as no longer needed. | ||
| - **Unreachable source database reported as a validation error**: When RDI cannot list the source tables during pipeline validation, for example because the source database refuses the connection, the failure is now reported as a `422` validation error in the `errors` array, instead of a `422`, `502`, or `504` response carrying only a `detail` message. An unavailable collector API is still reported as a `503` response so the request can be retried. | ||
| - **No configuration template for the Redis database type**: `GET /pipelines/config/templates/ingest/redis` now returns the documented `501 Not Implemented`, instead of a `200` response carrying a configuration that named `redis` as the source connection type and left the source port empty. Redis cannot be used as a pipeline source. | ||
| - **A database flavor that does not match the database type is rejected**: `GET /pipelines/config/templates/ingest/{db_type}` now returns a `422` response when `db_flavor` names a MongoDB flavor and `db_type` is not `mongodb`, instead of a `200` response carrying a configuration that ignored the flavor. |
Contributor
There was a problem hiding this comment.
Suggested change
| - **A database flavor that does not match the database type is rejected**: `GET /pipelines/config/templates/ingest/{db_type}` now returns a `422` response when `db_flavor` names a MongoDB flavor and `db_type` is not `mongodb`, instead of a `200` response carrying a configuration that ignored the flavor. | |
| - **A database flavor that does not match the database type is rejected**: `GET /pipelines/config/templates/ingest/{db_type}` now returns a `422` response when `db_flavor` names a MongoDB flavor and `db_type` is not `mongodb`, instead of a `200` response with an incorrect configuration. |
|
|
||
| - **API response body logging**: The RDI API now logs response bodies alongside request payloads. Bodies are logged at DEBUG level, while request and response metadata stays at INFO, and large bodies are truncated to keep logs manageable. | ||
| - **Documented scaffolded configurations**: The configuration that `redis-di scaffold`, the configuration template endpoint, and `rdi-admin install` generate now documents every property it offers, with a description above each one, links to the pipeline configuration documentation, and the default value where one exists. It also covers Snowflake and it no longer suggests properties that the source database does not support. The target port now defaults to `6379` rather than a placeholder, so the generated configuration passes schema validation before any property is filled in. The per-database example configurations that `rdi-admin install` wrote alongside it (`config.yaml.mysql.example` and its siblings) have been removed, since the generated configuration and the documentation now cover the same ground. | ||
| - **Documented transformation job template**: The job template that `GET /pipelines/jobs/templates/ingest` returns now documents the properties of the source, transform, and output blocks, with a description above each one and links to the transformation documentation, instead of naming a handful of them in a bare skeleton. It is a valid job as returned, with only the mandatory properties left uncommented. The endpoint accepts a new optional `source_name` query parameter that names the source the job reads from, matching the parameter of the configuration template endpoint. `rdi-admin install` now writes the same template to `jobs/job.yaml`, named after the source it scaffolds, in place of the example jobs it used to copy there. |
Contributor
There was a problem hiding this comment.
Suggested change
| - **Documented transformation job template**: The job template that `GET /pipelines/jobs/templates/ingest` returns now documents the properties of the source, transform, and output blocks, with a description above each one and links to the transformation documentation, instead of naming a handful of them in a bare skeleton. It is a valid job as returned, with only the mandatory properties left uncommented. The endpoint accepts a new optional `source_name` query parameter that names the source the job reads from, matching the parameter of the configuration template endpoint. `rdi-admin install` now writes the same template to `jobs/job.yaml`, named after the source it scaffolds, in place of the example jobs it used to copy there. | |
| - **Documented transformation job template**: The job template that `GET /pipelines/jobs/templates/ingest` returns now documents the properties of the source, transform, and output blocks, with a description above each one and links to the transformation documentation, instead of naming a handful of them in a bare skeleton. It is a valid job as returned, with only the mandatory properties left uncommented. The endpoint accepts a new optional `source_name` query parameter that specifies the source the job reads from, matching the parameter of the configuration template endpoint. `rdi-admin install` now writes the same template to `jobs/job.yaml`, named after the source it scaffolds, in place of the example jobs it used to copy there. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds the September 2026 RDI 2.0.0 release notes and changes
rdi_current_versionfrom1.19.1to2.0.0. The site currently has no 2.0.0 release page; this adds it with the sibling frontmatter and weight967, without claiming an exact release day.Ticket: RDSC-6038
Companion Cloud documentation: #3997.
Changelog source and wording changes
Source: the latest
redis-data-integrationmain CHANGELOG.md at c57c3a2, rechecked before authoring and publication.All 42 Unreleased entries are retained, including fixes also shipped in 1.19.1. There are only two deliberate content differences:
The other 41 entries are verbatim. All 121 security identifier occurrences (99 unique CVE/GHSA identifiers) are preserved. Formatting changes only turn the Unreleased section into a standalone release page with its title, description, and section heading.
API and CLI changes are included as changelog entries. Generated definitions and detailed Core documentation remain in #3916, which this PR does not modify.
Validation
79ab2f71ca0ebb35a7df9da1d31f92be284f7b7d, and both RDSC-6038 drafts also passed.git diff --check: passed.jupyter-exampleerrors report missing examples/source files. No example-generation files are changed here.Publication hold
Keep this PR in draft and do not merge until:
The version-variable change also updates download instructions elsewhere on the site, so it must wait for the artifacts.
Note
Low Risk
Documentation and a site param only; the version bump can surface 2.0.0 in download snippets before artifacts ship if merged early.
Overview
Publishes Redis Data Integration 2.0.0 on the docs site by adding
rdi-2-0-0.mdunder RDI release notes (September 2026, weight967) and bumpingrdi_current_versioninconfig.tomlfrom1.19.1to2.0.0, which flows through therdi-versionshortcode for install/download copy.The new page documents the 2.0.0 changelog: major breaking API/behavior shifts (immediate API v1 pipeline actions, Flink as default processor, multi-source pipelines, metric key changes, Cassandra source removal), new features, fixes, improvements, and a large security/CVE section—mostly aligned with upstream
CHANGELOG.md, with a short multi-source intro and clarified wording that removing a source deletes internal RDI data while target Redis records stay.Reviewed by Cursor Bugbot for commit 78ba113. Bugbot is set up for automated code reviews on this repo. Configure here.