feat(connectors): add Meilisearch sink connector#3497
Conversation
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## master #3497 +/- ##
=============================================
- Coverage 73.99% 40.29% -33.70%
Complexity 937 937
=============================================
Files 1301 1300 -1
Lines 147393 129164 -18229
Branches 122945 104793 -18152
=============================================
- Hits 109063 52051 -57012
- Misses 34858 74281 +39423
+ Partials 3472 2832 -640
🚀 New features to boost your workflow:
|
|
overall correct approach - JSON-object document model with generated PK is sound, wait_for_tasks + partial-error accounting is thoughtful. test coverage is present but there is |
|
/author |
github-actions
Bot
added
S-waiting-on-author
|
/ready |
github-actions
Bot
added
S-waiting-on-review
|
@countradooku Looks good except for the few warnings around back indexing and open calls. After the above requested changes, good to go for second reviewer. |
|
/author |
github-actions
Bot
added
S-waiting-on-author
|
This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 7 days if no further activity occurs. If you need a review, please ensure CI is green and the PR is rebased on the latest master. Don't hesitate to ping the maintainers - either Thank you for your contribution! |
|
/ready |
github-actions
Bot
added
S-waiting-on-review
| } | ||
|
|
||
| async fn wait_for_task(&self, client: &Client, task: TaskInfo) -> Result<(), Error> { | ||
| if !self.config.wait_for_tasks { |
There was a problem hiding this comment.
lib.rs:521-523 + 818-829 — wait_for_tasks=false returns Ok on submit; async Meilisearch task failures (disk full, schema conflict) invisible, offset committed, silent loss. Fix: none required — documented README + warn open/close. Non-blocking.
| ("text".to_string(), Value::String(text)), | ||
| ("data_type".to_string(), Value::String("text".to_string())), | ||
| ]), | ||
| _ => { |
There was a problem hiding this comment.
lib.rs:797-812 + 368-373 — unsupported schema (Avro/Protobuf/etc.) dropped, errors_count++, offset commits → permanent silent loss for whole topic if schema unsupported. Fix: matches runtime per-record drop convention; document explicitly / consider DLQ.
| return Ok(()); | ||
| } | ||
|
|
||
| match self.index_documents(client, documents).await { |
There was a problem hiding this comment.
818-846 — counter accounting: partial-fail undercounts un-attempted trailing chunks; redelivery double-counts prior chunks; documents_confirmed not bumped for confirmed-durable earlier chunks in Err arm. Metrics-only. Fix: label best-effort or account remaining. Keep the correct metrics .
|
|
||
| [package] | ||
| name = "iggy_connector_meilisearch_sink" | ||
| version = "0.4.0" |
There was a problem hiding this comment.
version 0.4.0, siblings all 0.4.1-edge.1. Version-sync CI flag. Fix: bump to 0.4.1-edge.1. Same as Source connector
|
/author |
github-actions
Bot
added
S-waiting-on-author
3 participants
Summary
meilisearch-sdkMotivation
This splits the sink portion out of #3404 so the Meilisearch sink can be reviewed and merged independently from the source connector.
Closes #3495.
Validation
cargo sort --no-format --workspacecargo test -p iggy_connector_meilisearch_sinkcargo clippy -p iggy_connector_meilisearch_sink --all-targets -- -D warningsenv CARGO_BIN_EXE_iggy-server=... CARGO_BIN_EXE_iggy-connectors=... cargo test -p integration --test mod -- connectors::meilisearch::meilisearch_sinkgit diff --check