Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
154 changes: 154 additions & 0 deletions packages/aws-durable-execution-sdk-python-insight/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -48,6 +48,160 @@ carrying the name-keyed `operationsByName` summary. The `S3Exporter` writes the
lossless per-occurrence `operations` array, one object per execution
(upsert-by-execution-name, so re-emission overwrites rather than appends).

## Exporters

All exporters live in `aws_durable_execution_sdk_python_insight.exporters` and
are re-exported from the package root. Each serializes the record as compact
JSON. Exporters that call AWS accept an injected `client=` for tests and use
the Lambda runtime's boto3 otherwise; none adds a required dependency.

| Exporter | Destination | Upsert | Operations shape | Default size limit |
| --- | --- | --- | --- | --- |
| `LambdaLogExporter` | Function's own log group | No | `operationsByName` | 256 KB |
| `CloudWatchLogsExporter` | Any log group, one stream per day | No | `operationsByName` | 256 KB |
| `S3Exporter` | S3 object per execution | Yes (key) | `operations` | 5 MB |
| `DynamoDBExporter` | DynamoDB item | Configurable | `operationsByName` | 400 KB |
| `AuroraExporter` | Aurora MySQL/PostgreSQL row (Data API) | Yes (upsert) | full record as JSON column | 1 MB |
| `RedshiftExporter` | Redshift row (Data API) | Yes (MERGE) | full record as SUPER column | 1 MB |
| `OpenSearchExporter` | OpenSearch document | Yes (`_id`) | `operations` | 10 MB |
| `FirehoseExporter` | Firehose delivery stream | N/A | `operations_format` | 1 MB |
| `EventBridgeExporter` | EventBridge event | N/A | `operations_format` | 256 KB |
| `SQSExporter` | SQS message | N/A | `operations_format` | 256 KB |
| `OTelExporter` | OTLP/HTTP logs endpoint | N/A | `operations_format` | 1 MB |
| `HttpExporter` | Any HTTP endpoint | N/A | `operations_format` | none |
| `FileExporter` | Directory (EFS, mount, `/tmp`) | Configurable | `operations_format` | none |

`operations_format` is `"array"` (default), `"by-name"`, or `"both"`
(`OperationsFormat`). `max_record_size_bytes` raises or lowers an exporter's
size limit; omitting it keeps the default. `HttpExporter` and `FileExporter`
have no default and do not truncate unless a limit is set.
Comment on lines +75 to +77

This comment was marked as outdated.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Repeat of a finding from the previous run; disposition unchanged (see the reply on the earlier thread). Omitting the argument keeps the default, matching the existing S3Exporter/LambdaLogExporter; README reworded in the previous push.

Comment thread
wangyb-A marked this conversation as resolved.

### CloudWatchLogsExporter

Writes one `PutLogEvents` event per record to `log_group_name`, in a stream
named `{log_stream_prefix}{YYYY}/{MM}/{DD}` (default prefix `workflow-insight/`).
IAM: `logs:CreateLogStream`, `logs:PutLogEvents` on the log group.

```python
CloudWatchLogsExporter(log_group_name="/custom/workflow-insight")
```

### DynamoDBExporter

`PutItem` keyed by `partition_key` (default `pk`) = `executionArn`. With the
default `sort_key="sk"` (= `emittedAt`) every export adds an item; pass
`sort_key=None` for a key-only table that upserts. IAM: `dynamodb:PutItem`.

```python
DynamoDBExporter(table_name="workflow-insight")
```

### AuroraExporter

Upserts a row by `execution_arn` through the RDS Data API; `engine` is
`"postgresql"` or `"mysql"` and selects the dialect. Columns: `execution_arn,
execution_name, function_name, status, start_time, end_time, duration_ms,
record_json, emitted_at`. IAM: `rds-data:ExecuteStatement`,
`secretsmanager:GetSecretValue`.

```python
AuroraExporter(
resource_arn="arn:aws:rds:us-east-1:123456789012:cluster:my-cluster",
secret_arn="arn:aws:secretsmanager:us-east-1:123456789012:secret:my-db-creds",
database="insight",
engine="postgresql",
)
```

### RedshiftExporter

`MERGE` by `execution_arn` through the Redshift Data API into
`{schema}.{table}` (default `public.workflow_insight`, same columns as Aurora,
`record_json` as `SUPER`). Provide `workgroup_name` (Serverless) or
`cluster_identifier` (provisioned, with `db_user` or `secret_arn`). IAM:
`redshift-data:ExecuteStatement` plus `redshift-serverless:GetCredentials` or
`secretsmanager:GetSecretValue`. The statement is submitted and not awaited, so
statement failures are not reported and `on-change` records may land out of
order; prefer the default `on-complete` emit mode with this exporter.

```python
RedshiftExporter(database="insight", workgroup_name="insight-wg")
```

### OpenSearchExporter

`PUT {endpoint}/{index_name}/_doc/{executionArn}` (default index
`workflow-insight`). `auth="sigv4"` (default) signs with the runtime's
credentials via botocore; `auth="basic"` uses `username`/`password`. IAM:
`es:ESHttpPut` on the domain.

```python
OpenSearchExporter(endpoint="https://my-domain.us-east-1.es.amazonaws.com", region="us-east-1")
```

### FirehoseExporter

`PutRecord` of one JSON line (trailing newline) per record. IAM:
`firehose:PutRecord`.

```python
FirehoseExporter(delivery_stream_name="workflow-insight-stream")
```

### EventBridgeExporter

`PutEvents` with `Source` (default `aws.durable-execution.insight`),
`DetailType` = record status, `Detail` = record. All arguments optional. IAM:
`events:PutEvents`.

```python
EventBridgeExporter(event_bus_name="default")
```

### SQSExporter

`SendMessage` with the record as body and `status`/`functionName` message
attributes. A `.fifo` queue URL enables `MessageGroupId` (default
`executionArn`, or `message_group_id`) and a deduplication id of
`executionArn:emittedAt`. IAM: `sqs:SendMessage`.

```python
SQSExporter(queue_url="https://sqs.us-east-1.amazonaws.com/123456789012/insight")
```

### OTelExporter

POSTs one OTLP `ExportLogsServiceRequest` (`http/json` only) per record to
`endpoint`; identity fields become attributes and the record is the log body.
Pass vendor auth in `headers`. No IAM.

```python
OTelExporter(endpoint="https://otlp.vendor.com/v1/logs", headers={"x-api-key": "..."})
```

### HttpExporter

`POST` (or `method="PUT"`) the record as JSON to `url` with
`Content-Type: application/json` plus `headers`; a non-2xx status raises.
`timeout_ms` defaults to 10000. No IAM.

```python
HttpExporter(url="https://hooks.example.com/insight", headers={"Authorization": "Bearer ..."})
```

### FileExporter

`mode="ndjson"` (default) appends to `{directory}/{YYYY-MM-DD}.ndjson`;
`mode="json"` writes `{directory}/{executionName}.json`, overwriting on update.
For Lambda use an EFS mount (IAM: `elasticfilesystem:ClientMount`,
`elasticfilesystem:ClientWrite`) or `/tmp` for testing. Appends from many
concurrent environments to one NDJSON file on a network file system can
interleave; use `mode="json"` when several environments share a directory.

```python
FileExporter(directory="/mnt/efs/workflow-insight")
```

Emission behavior, record schema (`recordType: WorkflowInsight`,
`schemaVersion: "1.0"`), sampling, content configuration (input/output
omission, `include_errors`, per-operation result opt-in), truncation phases,
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -5,11 +5,29 @@

from aws_durable_execution_sdk_python_insight.__about__ import __version__
from aws_durable_execution_sdk_python_insight.exporters import (
AuroraEngine,
AuroraExporter,
CloudWatchLogsExporter,
Comment thread
wangyb-A marked this conversation as resolved.
DynamoDBExporter,
EventBridgeExporter,
FileExporter,
FileMode,
FirehoseExporter,
HttpExporter,
HttpMethod,
LambdaLogExporter,
OpenSearchAuth,
OpenSearchExporter,
OTelExporter,
OTelProtocol,
RedshiftExporter,
S3Exporter,
S3Partitioning,
SQSExporter,
)
from aws_durable_execution_sdk_python_insight.operations_index import (
OperationsFormat,
apply_operations_format,
build_operations_by_name,
with_operations_by_name,
)
Expand All @@ -31,17 +49,35 @@

__all__ = [
"__version__",
"AuroraEngine",
"AuroraExporter",
"CloudWatchLogsExporter",
"ContentConfig",
"ContentOperations",
"DynamoDBExporter",
"EmitMode",
"EventBridgeExporter",
"FileExporter",
"FileMode",
"FirehoseExporter",
"HttpExporter",
"HttpMethod",
"InsightExporter",
"LambdaLogExporter",
"OTelExporter",
"OTelProtocol",
"OpenSearchAuth",
"OpenSearchExporter",
"OperationDetail",
"OperationOverride",
"OperationsFormat",
"RedshiftExporter",
"S3Exporter",
"S3Partitioning",
"SQSExporter",
"WorkflowInsightConfig",
"WorkflowInsightPlugin",
"apply_operations_format",
"build_operations_by_name",
"truncate_record",
"with_operations_by_name",
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -3,36 +3,84 @@
# SPDX-License-Identifier: Apache-2.0
"""First-party Workflow Insight exporters.

One module per exporter, mirroring the JS package's ``src/exporters/`` layout
(``aws-durable-execution-sdk-js-insight``). Each destination lives in its own
module so the set can grow to the full JS parity surface (S3, CloudWatch Logs,
DynamoDB, Firehose, EventBridge, SQS, OpenSearch, Redshift, Aurora, HTTP, OTel,
file, ...) without any single file accreting every backend's imports and
optional dependencies.
One module per destination, so no single file accretes every backend's imports
and optional dependencies. Concrete exporters are re-exported here so the
public import path is stable:
``from aws_durable_execution_sdk_python_insight.exporters import S3Exporter``.
Shared serialization and transport helpers live in the private ``_common``
module.

Concrete exporters are re-exported here so the public import path is stable:
``from aws_durable_execution_sdk_python_insight.exporters import S3Exporter``
keeps working exactly as before this package was split out of a single module.
Shared serialization helpers live in the private ``_common`` module.

Both shipped exporters serialize the curated record with JS-compatible compact
JSON (no whitespace) so the wire bytes match across SDKs. Records are written
verbatim -- no synthetic emission.
Every exporter serializes the curated record as compact JSON (no whitespace,
non-ASCII preserved). Records are written verbatim -- no synthetic emission.
"""

from __future__ import annotations

from aws_durable_execution_sdk_python_insight.exporters.aurora_exporter import (
AuroraEngine,
AuroraExporter,
)
from aws_durable_execution_sdk_python_insight.exporters.cloudwatch_logs_exporter import (
CloudWatchLogsExporter,
)
from aws_durable_execution_sdk_python_insight.exporters.dynamodb_exporter import (
DynamoDBExporter,
)
from aws_durable_execution_sdk_python_insight.exporters.eventbridge_exporter import (
EventBridgeExporter,
)
from aws_durable_execution_sdk_python_insight.exporters.file_exporter import (
FileExporter,
FileMode,
)
from aws_durable_execution_sdk_python_insight.exporters.firehose_exporter import (
FirehoseExporter,
)
from aws_durable_execution_sdk_python_insight.exporters.http_exporter import (
HttpExporter,
HttpMethod,
)
from aws_durable_execution_sdk_python_insight.exporters.lambda_log_exporter import (
LambdaLogExporter,
)
from aws_durable_execution_sdk_python_insight.exporters.opensearch_exporter import (
OpenSearchAuth,
OpenSearchExporter,
)
from aws_durable_execution_sdk_python_insight.exporters.otel_exporter import (
OTelExporter,
OTelProtocol,
)
from aws_durable_execution_sdk_python_insight.exporters.redshift_exporter import (
RedshiftExporter,
)
from aws_durable_execution_sdk_python_insight.exporters.s3_exporter import (
S3Exporter,
S3Partitioning,
)
from aws_durable_execution_sdk_python_insight.exporters.sqs_exporter import (
SQSExporter,
)


__all__ = [
"AuroraEngine",
"AuroraExporter",
"CloudWatchLogsExporter",
"DynamoDBExporter",
"EventBridgeExporter",
"FileExporter",
"FileMode",
"FirehoseExporter",
"HttpExporter",
"HttpMethod",
"LambdaLogExporter",
"OTelExporter",
"OTelProtocol",
"OpenSearchAuth",
"OpenSearchExporter",
"RedshiftExporter",
"S3Exporter",
"S3Partitioning",
"SQSExporter",
]
Loading
Loading