Skip to content

feat(hosted-data): consolidate cross-region replay repository MVP#3201

Draft
xiaoyaoqilan wants to merge 9 commits into
dimensionalOS:mainfrom
xiaoyaoqilan:codex/issue3158-unified-replay
Draft

feat(hosted-data): consolidate cross-region replay repository MVP#3201
xiaoyaoqilan wants to merge 9 commits into
dimensionalOS:mainfrom
xiaoyaoqilan:codex/issue3158-unified-replay

Conversation

@xiaoyaoqilan

Copy link
Copy Markdown
Contributor

Summary

  • consolidate the six hosted replay drafts into one dimos/hosted_data package and one dimos data CLI
  • add content-addressed raw object upload/download, browser listing, concurrent manifests, and filesystem plus S3-compatible storage
  • add memory2 consistent snapshot/index upload, direct replay_uri output, and trusted dimos-replay:// resolution for dimos --replay run unitree-go2 and unitree-go2-memory
  • add configured China/US node probing with an hourly cached recommendation
  • add synthetic H264 WebRTC first-frame and latency diagnostics
  • harden the repository with TLS support, quotas, metrics, audit logging, partial recovery, and CDN-aware CSP

Validation

  • 102 passed, 2 skipped because the optional aiortc dependency is not installed locally
  • 6 codebase/global configuration checks passed
  • Ruff formatting and checks passed
  • scoped mypy passed
  • uv lock --check --offline passed
  • the latest replay_uri patch passed Ruff and git diff --check; its focused pytest rerun is blocked by the current shell missing reactivex
  • Aliyun end-to-end testing on the cumulative precursor verified matching upload/download hashes for a video and a 229,855,232-byte memory2 object; SQLite integrity_check passed

Current limitations and follow-ups

  • WebRTC currently benchmarks synthetic H264 rather than ingesting the Go2 live stream
  • memory2 upload currently performs a blocking consistent snapshot rather than continuous background upload
  • nodes require DIMOS_REPLAY_NODES; automatic discovery, bandwidth/loss scoring, and failover remain follow-ups
  • the remote resolver is integrated with the Go2 replay path, not every replay consumer
  • multipart/resume, HTTP Range/HEAD, signed URLs, automatic CDN deployment, and multi-tenant ACLs remain follow-ups
  • quotas are process-local, so quota-enabled deployments should use a single writer

Progresses #3158.

Supersedes #3177, #3184, #3185, #3186, #3187, and #3188.

@codecov

codecov Bot commented Jul 26, 2026

Copy link
Copy Markdown

❌ 1 Tests Failed:

Tests completed Failed Passed Skipped
3143 1 3142 177
View the top 1 failed test(s) by shortest run time
dimos.hosted_data.test_cli::test_blob_upload_rejects_a_memory2_name
Stack Traces | 0.017s run time
tmp_path = PosixPath('.../pytest-0/popen-gw1/test_blob_upload_rejects_a_mem0')

    def test_blob_upload_rejects_a_memory2_name(tmp_path: Path) -> None:
        source = tmp_path / "capture.db"
        source.write_bytes(b"raw")
    
        result = runner.invoke(
            hosted_data_cli.hosted_data_app,
            [
                "upload",
                str(source),
                "--owner",
                "alice",
                "--repo",
                "go2",
                "--kind",
                "blob",
                "--name",
                "invalid",
            ],
        )
    
        assert result.exit_code == 2
>       assert "--name is only valid for memory2 uploads" in result.output
E       assert '--name is only valid for memory2 uploads' in "\x1b[33mUsage: \x1b[0mroot upload [OPTIONS] PATH\n\x1b[2mTry \x1b[0m\x1b[2;34m'root upload \x1b[0m\x1b[1;2;34m-\x1b[0m\x1b[1;2;34m-help\x1b[0m\x1b[2;34m'\x1b[0m\x1b[2m for help.\x1b[0m\n\x1b[31m╭─\x1b[0m\x1b[31m Error \x1b[0m\x1b[31m─────────────────────────────────────────────────────────────────────\x1b[0m\x1b[31m─╮\x1b[0m\n\x1b[31m│\x1b[0m Invalid value: \x1b[1;36m-\x1b[0m\x1b[1;36m-name\x1b[0m is only valid for memory2 uploads                      \x1b[31m│\x1b[0m\n\x1b[31m╰──────────────────────────────────────────────────────────────────────────────╯\x1b[0m\n"
E        +  where "\x1b[33mUsage: \x1b[0mroot upload [OPTIONS] PATH\n\x1b[2mTry \x1b[0m\x1b[2;34m'root upload \x1b[0m\x1b[1;2;34m-\x1b[0m\x1b[1;2;34m-help\x1b[0m\x1b[2;34m'\x1b[0m\x1b[2m for help.\x1b[0m\n\x1b[31m╭─\x1b[0m\x1b[31m Error \x1b[0m\x1b[31m─────────────────────────────────────────────────────────────────────\x1b[0m\x1b[31m─╮\x1b[0m\n\x1b[31m│\x1b[0m Invalid value: \x1b[1;36m-\x1b[0m\x1b[1;36m-name\x1b[0m is only valid for memory2 uploads                      \x1b[31m│\x1b[0m\n\x1b[31m╰──────────────────────────────────────────────────────────────────────────────╯\x1b[0m\n" = <Result SystemExit(2)>.output

result     = <Result SystemExit(2)>
source     = PosixPath('.../pytest-0/popen-gw1/test_blob_upload_rejects_a_mem0/capture.db')
tmp_path   = PosixPath('.../pytest-0/popen-gw1/test_blob_upload_rejects_a_mem0')

dimos/hosted_data/test_cli.py:397: AssertionError

To view more test analytics, go to the Test Analytics Dashboard
📋 Got 3 mins? Take this short survey to help us improve Test Analytics.

@xiaoyaoqilan

Copy link
Copy Markdown
Contributor Author

Deployment and CLI verification update:

  • fixed Windows upload rejection behavior in ea58334 by adding a body-free HEAD quota preflight before POST
  • focused hosted-data suite: 87 passed
  • Ruff and git diff --check: passed
  • local CLI smoke test: serveuploadlistdownload; source and downloaded SHA-256 matched
  • deployed ea58334 to the Beijing Aliyun node at 39.106.170.53:8765
  • systemd now runs python -m dimos.hosted_data.cli serve
  • server health: {"node":"cn-beijing","region":"china","status":"ok"}
  • server-side and public-client upload/download SHA-256:
    22531dca667485a7a9cdf8e1dea4798899cd2c1264a0bc69e03cb5671afd27f4
  • public repository smoke page:
    http://39.106.170.53:8765/r/codex/issue3158-deploy

The existing repository storage directory was preserved. The previous systemd unit is backed up as /etc/systemd/system/dimos-replay.service.bak-unified-20260727.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant