Skip to content

perf: remove two redundant object-store calls in datasets metadata fetch #1760

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
prabhaks wants to merge 1 commit into
base: main
Choose a base branch
from
+13 −15
Draft

perf: remove two redundant object-store calls in datasets metadata fetch #1760

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 8 additions & 8 deletions src/query/mod.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -763,14 +763,6 @@ pub async fn get_manifest_list(
time_range: &TimeRange,
tenant_id: &Option<String>,
) -> Result<Vec<Manifest>, QueryError> {
// get object store
let object_store_format: ObjectStoreFormat = serde_json::from_slice(
&PARSEABLE
.metastore
.get_stream_json(stream_name, false, tenant_id, false)
.await?,
)?;

// all the manifests will go here
let mut merged_snapshot: Snapshot = Snapshot::default();

Expand All @@ -791,6 +783,14 @@ pub async fn get_manifest_list(
}
}
} else {
// Only needed here: Query/Prism mode merges every ingestor's stream.json
// above instead and never reads the base file.
let object_store_format: ObjectStoreFormat = serde_json::from_slice(
&PARSEABLE
.metastore
.get_stream_json(stream_name, false, tenant_id, false)
.await?,
)?;
merged_snapshot = object_store_format.snapshot;
}

Expand Down
12 changes: 5 additions & 7 deletions src/storage/object_storage.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -931,13 +931,11 @@ pub trait ObjectStorage: Debug + Send + Sync + 'static {
let min_date = &parsed_dates[0].1;
let max_date = &parsed_dates[parsed_dates.len() - 1].1;

// Extract timestamps for min and max dates
let first_timestamp = self
.extract_timestamp_for_date(stream_name, min_date, true, tenant_id)
.await?;
let latest_timestamp = self
.extract_timestamp_for_date(stream_name, max_date, false, tenant_id)
.await?;
// Extract timestamps for min and max dates concurrently; independent reads, no shared state
let (first_timestamp, latest_timestamp) = tokio::try_join!(
self.extract_timestamp_for_date(stream_name, min_date, true, tenant_id),
self.extract_timestamp_for_date(stream_name, max_date, false, tenant_id)
)?;

let first_event_at = first_timestamp.map(|ts| ts.to_rfc3339());
let latest_event_at = latest_timestamp.map(|ts| ts.to_rfc3339());
Expand Down
Loading