Skip to content

Harden replay repository operations#3188

Draft
xiaoyaoqilan wants to merge 12 commits into
dimensionalOS:mainfrom
xiaoyaoqilan:codex/replay-repository-hardening
Draft

Harden replay repository operations#3188
xiaoyaoqilan wants to merge 12 commits into
dimensionalOS:mainfrom
xiaoyaoqilan:codex/replay-repository-hardening

Conversation

@xiaoyaoqilan

Copy link
Copy Markdown
Contributor

Summary

  • require TLS 1.2+ when TLS is enabled
  • add per-object and per-repository quotas
  • expose Prometheus-compatible operational metrics
  • support CDN download base URLs
  • audit upload operations
  • recover interrupted .part uploads safely on service start
  • document production deployment and operational settings

Motivation

This is PR 6 of the #3158 implementation stack. It adds the bounded resource use, observability, recovery, and CDN integration required to move the validated MVP toward a reusable service.

Stack dependency

Validation

  • quota, TLS configuration, metrics, CDN URL, audit, and interrupted-upload recovery paths are covered by tests
  • full six-PR targeted suite: 56 passed
  • ruff check and ruff format --check passed
  • targeted mypy checks passed

@codecov

codecov Bot commented Jul 25, 2026

Copy link
Copy Markdown

❌ 3 Tests Failed:

Tests completed Failed Passed Skipped
3086 3 3083 177
View the top 3 failed test(s) by shortest run time
dimos.protocol.pubsub.impl.webrtc.test_replay_repository_cli::test_serve_rejects_invalid_region
Stack Traces | 0.014s run time
tmp_path = PosixPath('.../pytest-0/popen-gw0/test_serve_rejects_invalid_reg0')

    def test_serve_rejects_invalid_region(tmp_path: Path) -> None:
        result = runner.invoke(
            replay_repository_cli.replay_repository_app,
            ["serve", "--root", str(tmp_path), "--region", "europe"],
        )
    
        assert result.exit_code == 2
>       assert "--region must be china, us, or other" in result.output
E       assert '--region must be china, us, or other' in "\x1b[33mUsage: \x1b[0mroot serve [OPTIONS]\n\x1b[2mTry \x1b[0m\x1b[2;34m'root serve \x1b[0m\x1b[1;2;34m-\x1b[0m\x1b[1;2;34m-help\x1b[0m\x1b[2;34m'\x1b[0m\x1b[2m for help.\x1b[0m\n\x1b[31m╭─\x1b[0m\x1b[31m Error \x1b[0m\x1b[31m─────────────────────────────────────────────────────────────────────\x1b[0m\x1b[31m─╮\x1b[0m\n\x1b[31m│\x1b[0m Invalid value: \x1b[1;36m-\x1b[0m\x1b[1;36m-region\x1b[0m must be china, us, or other                          \x1b[31m│\x1b[0m\n\x1b[31m╰──────────────────────────────────────────────────────────────────────────────╯\x1b[0m\n"
E        +  where "\x1b[33mUsage: \x1b[0mroot serve [OPTIONS]\n\x1b[2mTry \x1b[0m\x1b[2;34m'root serve \x1b[0m\x1b[1;2;34m-\x1b[0m\x1b[1;2;34m-help\x1b[0m\x1b[2;34m'\x1b[0m\x1b[2m for help.\x1b[0m\n\x1b[31m╭─\x1b[0m\x1b[31m Error \x1b[0m\x1b[31m─────────────────────────────────────────────────────────────────────\x1b[0m\x1b[31m─╮\x1b[0m\n\x1b[31m│\x1b[0m Invalid value: \x1b[1;36m-\x1b[0m\x1b[1;36m-region\x1b[0m must be china, us, or other                          \x1b[31m│\x1b[0m\n\x1b[31m╰──────────────────────────────────────────────────────────────────────────────╯\x1b[0m\n" = <Result SystemExit(2)>.output

result     = <Result SystemExit(2)>
tmp_path   = PosixPath('.../pytest-0/popen-gw0/test_serve_rejects_invalid_reg0')

.../impl/webrtc/test_replay_repository_cli.py:172: AssertionError
dimos.protocol.pubsub.impl.webrtc.test_replay_repository_cli::test_serve_rejects_missing_s3_bucket
Stack Traces | 0.014s run time
tmp_path = PosixPath('.../pytest-0/popen-gw0/test_serve_rejects_missing_s3_0')

    def test_serve_rejects_missing_s3_bucket(tmp_path: Path) -> None:
        result = runner.invoke(
            replay_repository_cli.replay_repository_app,
            ["serve", "--root", str(tmp_path), "--backend", "s3"],
        )
    
        assert result.exit_code == 2
>       assert "--s3-bucket is required" in result.output
E       assert '--s3-bucket is required' in "\x1b[33mUsage: \x1b[0mroot serve [OPTIONS]\n\x1b[2mTry \x1b[0m\x1b[2;34m'root serve \x1b[0m\x1b[1;2;34m-\x1b[0m\x1b[1;2;34m-help\x1b[0m\x1b[2;34m'\x1b[0m\x1b[2m for help.\x1b[0m\n\x1b[31m╭─\x1b[0m\x1b[31m Error \x1b[0m\x1b[31m─────────────────────────────────────────────────────────────────────\x1b[0m\x1b[31m─╮\x1b[0m\n\x1b[31m│\x1b[0m Invalid value: DIMOS_REPLAY_S3_BUCKET or \x1b[1;36m-\x1b[0m\x1b[1;36m-s3\x1b[0m\x1b[1;36m-bucket\x1b[0m is required for the s3  \x1b[31m│\x1b[0m\n\x1b[31m│\x1b[0m backend                                                                      \x1b[31m│\x1b[0m\n\x1b[31m╰──────────────────────────────────────────────────────────────────────────────╯\x1b[0m\n"
E        +  where "\x1b[33mUsage: \x1b[0mroot serve [OPTIONS]\n\x1b[2mTry \x1b[0m\x1b[2;34m'root serve \x1b[0m\x1b[1;2;34m-\x1b[0m\x1b[1;2;34m-help\x1b[0m\x1b[2;34m'\x1b[0m\x1b[2m for help.\x1b[0m\n\x1b[31m╭─\x1b[0m\x1b[31m Error \x1b[0m\x1b[31m─────────────────────────────────────────────────────────────────────\x1b[0m\x1b[31m─╮\x1b[0m\n\x1b[31m│\x1b[0m Invalid value: DIMOS_REPLAY_S3_BUCKET or \x1b[1;36m-\x1b[0m\x1b[1;36m-s3\x1b[0m\x1b[1;36m-bucket\x1b[0m is required for the s3  \x1b[31m│\x1b[0m\n\x1b[31m│\x1b[0m backend                                                                      \x1b[31m│\x1b[0m\n\x1b[31m╰──────────────────────────────────────────────────────────────────────────────╯\x1b[0m\n" = <Result SystemExit(2)>.output

result     = <Result SystemExit(2)>
tmp_path   = PosixPath('.../pytest-0/popen-gw0/test_serve_rejects_missing_s3_0')

.../impl/webrtc/test_replay_repository_cli.py:162: AssertionError
dimos.codebase_checks.test_get_logger::test_no_get_logger
Stack Traces | 0.252s run time
def test_no_get_logger():
        """
        Fail if any file uses `= logging.getLogger` outside the whitelist.
        """
        violations = find_get_logger_usages()
        if violations:
            report_lines = [
                f"Found {len(violations)} forbidden use(s) of `logging.getLogger`. "
                "Use `setup_logger` instead:",
                "",
                "    from dimos.utils.logging_config import setup_logger",
                "",
                "    logger = setup_logger()",
                "",
                "If the usage is legitimate (e.g. standalone script, logging "
                "infrastructure, or third-party logger suppression), add it to the "
                "WHITELIST in dimos/codebase_checks/test_get_logger.py.",
                "",
            ]
            for path, lineno, text in violations:
                report_lines.append(f"  {path}:{lineno}: {text.strip()}")
>           raise AssertionError("\n".join(report_lines))
E           AssertionError: Found 1 forbidden use(s) of `logging.getLogger`. Use `setup_logger` instead:
E           
E               from dimos.utils.logging_config import setup_logger
E           
E               logger = setup_logger()
E           
E           If the usage is legitimate (e.g. standalone script, logging infrastructure, or third-party logger suppression), add it to the WHITELIST in dimos/codebase_checks/test_get_logger.py.
E           
E             .../impl/webrtc/replay_repository.py:47: logger = logging.getLogger(__name__)

lineno     = 47
path       = '.../impl/webrtc/replay_repository.py'
report_lines = ['Found 1 forbidden use(s) of `logging.getLogger`. Use `setup_logger` instead:', '', '    from dimos.utils.logging_config import setup_logger', '', '    logger = setup_logger()', '', ...]
text       = 'logger = logging.getLogger(__name__)'
violations = [('.../impl/webrtc/replay_repository.py', 47, 'logger = logging.getLogger(__name__)')]

dimos/codebase_checks/test_get_logger.py:128: AssertionError

To view more test analytics, go to the Test Analytics Dashboard
📋 Got 3 mins? Take this short survey to help us improve Test Analytics.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant