Refactoring to support live trading and research

.gitignore

@@ -46,4 +46,7 @@ dump.rdb
 .infisical.json
 cmake-build-debug
 .idea/
-.clang-tidy
+.clang-tidy
+# AWS SDK for C++ install prefix (built per-node by scripts/build_dep.sh
+# from the external/aws submodule)
+external/aws-install/

.gitmodules

@@ -1,9 +1,12 @@
+[submodule "external/boost-decimal"]
+	path = external/boost-decimal
+	url = https://github.com/boostorg/decimal.git
 [submodule "external/libpqxx"]
 	path = external/libpqxx
 	url = https://github.com/jtv/libpqxx.git
-[submodule "external/boost-decimal"]
-	path = external/boost-decimal
-	url = https://github.com/boostorg/decimal
 [submodule "external/Catch2"]
 	path = external/Catch2
 	url = https://github.com/catchorg/Catch2.git
+[submodule "external/aws"]
+	path = external/aws
+	url = https://github.com/aws/aws-sdk-cpp

CMakeLists.txt

@@ -232,6 +232,21 @@ target_link_libraries(BacktestingEngineLib PUBLIC Boost::decimal)
 
 target_link_libraries(BacktestingEngineLib PUBLIC pqxx OpenMP::OpenMP_CXX)
 
+# AWS SDK for C++, DynamoDB client only — IG session credentials live in the
+# MarketDataLive table (shared/aws/dynamoAuth). Deliberately NOT an
+# add_subdirectory: the SDK is enormous and must not inherit this project's
+# global module/OpenMP flags, so scripts/build_dep.sh builds and installs it
+# once (static libs) into external/aws-install and it is consumed here as a
+# prebuilt package. AWSSDK_LINK_LIBRARIES resolves to the dynamodb target,
+# which chains core + CRT dependencies through its exported config.
+list(APPEND CMAKE_PREFIX_PATH "${CMAKE_SOURCE_DIR}/external/aws-install")
+# The static aws-cpp-sdk-core's exported link interface names ZLIB::ZLIB
+# (and curl/OpenSSL targets, already found above) but its config does not
+# find_dependency them — resolve ZLIB here first.
+find_package(ZLIB REQUIRED)
+find_package(AWSSDK REQUIRED CONFIG COMPONENTS dynamodb)
+target_link_libraries(BacktestingEngineLib PUBLIC ${AWSSDK_LINK_LIBRARIES})
+
 # Main executable
 add_executable(BacktestingEngine source/main.cpp)
 target_link_libraries(BacktestingEngine BacktestingEngineLib)

CONTRIBUTING.md

@@ -12,6 +12,8 @@ I am actively experimenting with different approaches and want to avoid merge co
 
 The project is [MIT-licensed](LICENSE.MD), so you're very welcome to fork it and take the code in your own direction.
 
+To build and test: `scripts/build.sh`, then `ctest --test-dir build` (see the README and [QUICKSTART.md](QUICKSTART.md) for detail). Code style lives in [CONVENTIONS.md](CONVENTIONS.md).
+
 ## Use GitHub Issues for bugs, questions, and ideas
 
 Bugs, questions, and ideas are all welcome on the [Issues tab](https://github.com/mccaffers/backtesting-engine-cpp/issues).

CONVENTIONS.md

@@ -4,13 +4,24 @@ Plain headers live next to the code they belong to under `source/` (e.g.
 separate `include/` folder. The build exposes a single source root (`source/`),
 so every project `#include` is **source-relative and path-qualified**, e.g.
 `#include "shared/utilities/env.hpp"`. This makes each include / module global
-module fragment dependency self-describing. Both `source/*.cpp` and
-`source/*.cppm` are globbed automatically (`CONFIGURE_DEPENDS`), so a new file is
-picked up without editing `CMakeLists.txt`.
+module fragment dependency self-describing. (The build also exposes a second
+PUBLIC include root, `external/`, only for vendored third-party headers like
+`nlohmann/json.hpp`.) Both `source/*.cpp` and `source/*.cppm` are globbed
+automatically (`CONFIGURE_DEPENDS`), so a new file under `source/` is picked up
+without editing `CMakeLists.txt` — `source/` only; tests are not globbed, see
+below.
 
 ### Pragma once
 Headers should use `#pragma once` directive to guard to prevent multiple inclusions of the same header file.
 
-### Lower Camel Case names
-camelCase applies to file names, types, and namespaces.
-application.cpp / class Application() / namespace tradingDefinitions
+### Naming
+File names are lowerCamelCase, types are PascalCase, namespaces are snake_case.
+databaseConnection.cppm / class DatabaseConnection / namespace symbol_scale
+(the one legacy camelCase namespace is tradingDefinitions)
+
+### New files should come with ctests
+only exception would be if they are using libraries (eg. boost), we don't need to test libraries.
+Unlike `source/`, test translation units are NOT globbed: `tests/CMakeLists.txt`
+lists every file explicitly in `add_executable(unit_tests ...)`, so a new
+`tests/foo.cpp` silently never builds or runs until added there. Tests register
+with ctest via `catch_discover_tests` in the same file.

ENVIRONMENT.md

@@ -0,0 +1,67 @@
+# Environment Variables
+
+All engine configuration is read from the environment via `env::getOr(name, fallback)` (`source/shared/utilities/env.cpp`) — every variable is **optional with a default** unless noted. An unset *or empty* variable falls back to its default.
+
+The `run` and `analysis` subcommands print a startup diagnostic dump of the whole environment to stderr, masking any variable whose name contains `PASSWORD`, `PASSWD`, `SECRET`, `TOKEN`, `CREDENTIAL`, `KEY`, or `AUTH`.
+
+## Quick reference
+
+| Variable | Default | `load` | `experiments` | `run` | `analysis` | `ingest` | `live` | `tracking` | `positions` | Purpose |
+| --- | --- | :-: | :-: | :-: | :-: | :-: | :-: | :-: | :-: | --- |
+| `REDIS_HOST` | `127.0.0.1` | ✓ | ✓ | ✓ | ✓ | | ✓ | ✓ | ✓ | Redis host (port is always `6379`) |
+| `BACKTEST_BATCH` | *(current UTC ISO week)* | ✓ | ✓ | | | | | | | Weekly batch label (`YYYY-WW`, e.g. `2026-28`) naming the Elasticsearch outcome indices (`backtesting-results-2026-28`, ...; `experiments`: `backtesting-experiments-2026-28`). Frozen into every queued run descriptor and carried through all rolling-window rungs; `scripts/load.sh` exports it once so its per-strategy invocations share one label. Set explicitly to re-run into a past week's indices |
+| `ENTRY_SLIPPAGE_TENTH_PIPS` | `0` | ✓ | | | | | | | | Slippage stress toggle: every backtest entry fills this many **tenths of a pip** against the trade (`3` = 0.3 pip; SL/TP anchors stay on the raw tick). Read once at `load` and frozen into every queued run descriptor, carried through all rolling-window rungs, and recorded in each run's results doc. **Validated** — a non-integer or negative value fails the load |
+| `QUESTDB_HOST` | `127.0.0.1` | | | ✓ | | ✓ | ✓ | | | QuestDB host (`run`: the required `<questdb-host>` argument covers tick reads only — the OHLC/range-bar warm-up still connects to `$QUESTDB_HOST`, so `run <remote-host>` reads ticks from the argv host but seeds bar histories from here; `live`: OHLC pre-population reads) |
+| `QUESTDB_PORT` | `8812` | | | ✓ | ✓ | | ✓ | | | QuestDB pgwire port for tick reads (`live`: OHLC pre-population reads). **Validated** — a non-numeric value is a hard failure |
+| `QUESTDB_ILP_PORT` | `9000` | | | | | ✓ | | | | QuestDB ILP-over-HTTP port for tick writes |
+| `TICK_CACHE_TTL_MINUTES` | `60` | | | ✓ | | | | | | Lifetime of a cached tick superset before it is reloaded from QuestDB (queue mode only — direct mode and `analysis` bypass the cache). **Validated** — a non-numeric or negative value throws on the cache's first use (once the first run is claimed), aborting the worker; `0` passes |
+| `TICK_CACHE_MAX_SUPERSETS` | `1` | | | ✓ | | | | | | How many tick supersets (one per symbol set) a worker keeps resident before evicting the oldest. **Validated** — same rule as the TTL |
+| `ELASTIC_ENABLED` | `1` | ✓ | ✓ | ✓ | ✓ | | ✓ | ✓ | ✓ | Master toggle for Elasticsearch reporting; set `0` to disable (`load`/`experiments`: also skips the weekly index/alias admin; `live`: also silences the trade audit, the trace documents, and the `live-logs` log feed; `tracking`: also silences the per-deal `live-trades` documents; `positions`: also silences the per-cycle `live-function-logs` reports) |
+| `ELASTIC_HOST` | `http://localhost:9200` | ✓ | ✓ | ✓ | ✓ | | ✓ | ✓ | ✓ | Elasticsearch base URL (results reporting; weekly index/alias admin at `load`/`experiments`; `analysis` experiment documents; live winner source + trade audit; `tracking` deal documents; `positions` cycle reports) |
+| `ELASTIC_USER` | *(empty)* | ✓ | ✓ | ✓ | ✓ | | ✓ | ✓ | ✓ | Basic-auth username; empty means no auth header |
+| `ELASTIC_USER_PASSWORD` | *(empty)* | ✓ | ✓ | ✓ | ✓ | | ✓ | ✓ | ✓ | Basic-auth password |
+| `ELASTIC_TRADES_ENABLED` | `0` | | | ✓ | | | | | | Set `1` to bulk-index every closed trade into `backtesting-trades` |
+| `ELASTIC_DEADLETTER_PATH` | `elastic_deadletter.ndjson` | | | ✓ | ✓ | | ✓ | ✓ | ✓ | File that failed Elasticsearch documents are appended to after retries are exhausted |
+| `ELASTIC_FLUSH_SECONDS` | `30` | | | ✓ | ✓ | | ✓ | ✓ | ✓ | Cadence of the background `_bulk` flusher that delivers queued reporting documents (run outcomes, experiment documents, engine exceptions, live trade audits, tracking deal documents); intervals with an empty buffer send nothing. A non-numeric or `< 1` value falls back to `30` with a logged warning |
+| `OHLC_PREPOPULATE` | `1` | | | ✓ | | | ✓ | | | Seed OHLC **and range** bar histories from QuestDB at each symbol's first tick (`run`: replay start; `live`: launch — big-bar strategies trade immediately instead of warming up for days). One switch covers both bar types; set `0` to disable all warm-up queries |
+| `INGEST_BIND_ADDR` | `127.0.0.1` | | | | | ✓ | | | | UDP bind address for the tick receiver |
+| `INGEST_UDP_PORT` | `11111` | | | | | ✓ | | | | UDP bind port (overridable by the command's port argument) |
+| `TRACKING_BIND_ADDR` | `127.0.0.1` | | | | | | | ✓ | | UDP bind address for the deal/trade-update receiver |
+| `TRACKING_UDP_PORT` | `11112` | | | | | | | ✓ | | UDP bind port (overridable by the command's port argument) |
+| `LIVE_BIND_ADDR` | `127.0.0.1` | | | | | | ✓ | | | UDP bind address for the live tick receiver |
+| `LIVE_UDP_PORT` | `11110` | | | | | | ✓ | | | UDP bind port (overridable by the command's port argument) |
+| `LIVE_MIN_SCORE` | `20` | | | | | | ✓ | | | Winner floor on `results.performanceScore` when selecting strategies from Elasticsearch (top 3 per (symbol, strategy) pair is fixed in code, not env-configurable) |
+| `LIVE_MAX_DRAWDOWN_PERCENT` | `10` | | | | | | ✓ | | | Hard ceiling on `results.maxDrawdownPercent` (peak-to-trough giveback) when selecting winners — the Calmar half of `performanceScore` only blends drawdown in, so this gate is what actually excludes spiky runs |
+| `LIVE_MIN_CALMAR_SCORE` | `30` | | | | | | ✓ | | | Floor on `results.calmarScore` when selecting winners (Calmar ratio ~2 on the score scale where ratio 3 = 50): growth must be ~2x the worst giveback. Complements the drawdown ceiling — the floor rejects smooth-but-stagnant runs, the ceiling rejects fast growers with deep absolute givebacks |
+| `LIVE_TRADE_LOCK_SECONDS` | `30` | | | | | | ✓ | | | TTL of the per-(strategy, direction) Redis trade lock |
+| `LIVE_TRACE_ENABLED` | `1` | | | | | | ✓ | | | Emit live-mode trace documents (order/close lifecycle, book sync, IG guard refusals, startup/shutdown, minutely stats) to the Elasticsearch index `live-traces` through the async batch publisher; set `0` to disable. Every document carries an `env` field (`TRADING_ENVIRONMENT`). Delivery still requires `ELASTIC_ENABLED=1` |
+| `LIVE_LOG_SHIP_ENABLED` | `1` | | | | | | ✓ | | | Ship every engine log line (`logLine`/`error`) as a document to the Elasticsearch index `live-logs`; set `0` to disable. An independent kill switch from `LIVE_TRACE_ENABLED` — the narrative log and the structured trace events are separate feeds. Delivery still requires `ELASTIC_ENABLED=1` |
+| `TRADING_ENVIRONMENT` | `demo` | | | | | | ✓ | ✓ | ✓ | IG environment; lowercased into the DynamoDB credentials key `Auth#<env>` (`demo`/`live`); `tracking` stamps it into each `live-trades` document's `env` field |
+
+## AWS credentials (`live` and `positions`)
+
+The IG session credentials are pulled from the DynamoDB table `MarketDataLive` (`source/shared/aws/dynamoAuth`). The engine reads no AWS variables itself — the AWS SDK's default credential/region chain applies:
+
+- `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`, `AWS_DEFAULT_REGION` — or a configured profile / instance role.
+
+Effectively **required** for `live` order placement and the `positions` sync (without a session, `positions` logs a warning and every cycle skips safely); there is no in-code default.
+
+## Build and script variables
+
+| Variable | Default | Read by | Purpose |
+| --- | --- | --- | --- |
+| `CC` / `CXX` | `clang` / `clang++` | `scripts/build.sh` (non-Homebrew path) | Pin a specific compiler, e.g. `CC=clang-20 CXX=clang++-20` |
+| `ENABLE_COVERAGE` | `OFF` | `scripts/build.sh` | Instrument with Clang source-based coverage (CI turns it on for the SonarCloud report) |
+| `CLEAN` | `0` | `scripts/test.sh` | `CLEAN=1` forces a clean reconfigure before the test build |
+
+`scripts/run.sh` additionally **requires** `ELASTIC_HOST`, `ELASTIC_USER`, `ELASTIC_USER_PASSWORD`, and `REDIS_HOST` to be set and non-empty — it aborts up front if any are missing (the engine itself would fall back to the defaults above). It also pins `ELASTIC_TRADES_ENABLED=0` for the run it launches (per-trade indexing off).
+
+## Secrets management
+
+I manage secrets with [Infisical](https://infisical.com/), which injects them into the process environment at runtime:
+
+```
+infisical run -- bash ./scripts/run.sh
+```
+
+If you're not using Infisical, export the variables yourself (shell profile or a sourced `.env`) before invoking the scripts.