Headless drive e2e + iOS/Android e2e CI jobs + faster test setup

.agents/skills/testing-ios-simulator/SKILL.md

@@ -39,6 +39,32 @@ description: Build and launch the Logseq Chat iOS app in an iOS simulator on mac
 - First launch may show a local-network permission alert; dismiss it via Simulator UI.
 - The app opens to a "Logseq Chat / Sign in" screen (Cognito hosted UI); going past it
   requires credentials.
+- `duniverse/` pulls only the lockfile dirs (129). The `logseq/*` pins NOT vendored
+  (`datascript-ocaml`, `lg`, `lui`, `mldoc`, `persistent-sorted-set-ocaml`,
+  `signal-lg`) must be cloned manually into `duniverse/` at the SHAs listed in
+  `logseq_chat.opam` — same loop as `.github/workflows/e2e.yml`. A fresh
+  `opam-monorepo pull` wipes these dirs, so re-clone them after each pull.
+- `scripts/build-mobile-ocaml.sh` regenerates `_build/dune-workspace.mobile` from
+  `dune-workspace.mobile`; set `LOGSEQ_CHAT_OPAM_SWITCH=5.5.0` when running outside
+  `opam exec`.
+
+## Maestro e2e (test-ios-e2e-suite.sh)
+- Install Maestro: `curl -fsSL https://get.maestro.mobile.dev | bash` → `~/.maestro/bin`.
+- The e2e flows hit the app's sync server at `http://127.0.0.1:8787`. That server is
+  `deps/db-sync` in the `logseq` repo, and it MUST run from the
+  `feature/native-mobile-app` branch (the canonical `../logseq-1` checkout) — the app
+  requires `schema-version` in the `/sync/<id>/snapshot/download` response, which only
+  exists there (`main` returns without it → "Add sync graph" fails with
+  `NSURLErrorDomain Code=-1017`). Setup:
+  `cd ../logseq-1/deps/db-sync && pnpm install && clojure -M:cljs release db-sync-node`
+  (needs `JAVA_HOME=$(/usr/libexec/java_home -v 21)`), then
+  `DB_SYNC_PORT=8787 DB_SYNC_DATA_DIR=~/db-sync-data ./start.sh`;
+  health check `curl http://127.0.0.1:8787/health`.
+- Run: `LOGSEQ_CHAT_IOS_SKIP_BUILD=1 LOGSEQ_CHAT_E2E_USERNAME=<u> LOGSEQ_CHAT_E2E_PASSWORD=<p> ./scripts/test-ios-e2e-suite.sh smoke`
+  (4 flows: capture-responsive, cold-start-composer, search-status-regression, sidebar).
+- `scripts/test-ios-e2e.sh` wipes the app data container between flows; `<container>/tmp`
+  must be recreated (the script does `mkdir -p`) or the OCaml core crashes writing the
+  initial graph snapshot via `Filename.temp_file` (`Sys_error ...tmp/logseq-chat-initial-*.snapshot`).
 
 ## Devin Secrets Needed
-- none for build/launch/screenshot.
+- `LOGSEQ_CHAT_TEST_USERNAME` / `LOGSEQ_CHAT_TEST_PASSWORD` for hosted sign-in and e2e.

.github/actions/db-sync-server/action.yml

@@ -0,0 +1,128 @@
+name: Start db-sync server
+description: >-
+  Check out logseq@feature/native-mobile-app, build the db-sync node adapter,
+  and start it on the given port (default 8787). The app e2e flows create sync
+  graphs against it.
+inputs:
+  token:
+    description: GitHub token with access to the private logseq/logseq repo.
+    required: true
+  ref:
+    description: logseq/logseq ref to check out.
+    required: false
+    default: feature/native-mobile-app
+  port:
+    description: Port the server binds.
+    required: false
+    default: "8787"
+  background:
+    description: >-
+      When "true", the pnpm install, cljs release build and server start run in
+      a detached background process (log at logseq-server/deps/db-sync/build.log)
+      so the job can continue with other work; a later step must poll the port.
+    required: false
+    default: "false"
+outputs:
+  port:
+    description: The port the server is listening on.
+    value: ${{ inputs.port }}
+runs:
+  using: composite
+  steps:
+    - name: Check out db-sync server
+      uses: actions/checkout@v4
+      with:
+        repository: logseq/logseq
+        ref: ${{ inputs.ref }}
+        path: logseq-server
+        token: ${{ inputs.token }}
+        fetch-depth: 1
+
+    - uses: actions/setup-java@v4
+      with:
+        distribution: temurin
+        java-version: "21"
+
+    - uses: actions/setup-node@v4
+      with:
+        node-version: "22"
+
+    - name: Install Clojure CLI and pnpm
+      shell: bash
+      run: |
+        if [[ $RUNNER_OS == macOS ]]; then
+          brew install clojure/tools/clojure || brew upgrade clojure/tools/clojure
+        else
+          curl -fsSL \
+            https://github.com/clojure/brew-install/releases/latest/download/linux-install.sh \
+            | sudo bash
+        fi
+        corepack enable
+        clojure --version
+
+    - name: Restore db-sync build caches
+      uses: actions/cache/restore@v4
+      with:
+        path: |
+          ~/.m2
+          ~/.gitlibs
+          ~/.local/share/pnpm/store
+          ~/Library/pnpm
+          logseq-server/node_modules
+          logseq-server/deps/db-sync/node_modules
+        key: dbsync-deps-${{ runner.os }}-${{ hashFiles('logseq-server/pnpm-lock.yaml') }}
+        restore-keys: dbsync-deps-${{ runner.os }}-
+
+    - name: Build and start db-sync
+      if: inputs.background != 'true'
+      shell: bash
+      run: |
+        cd logseq-server
+        pnpm install
+        cd deps/db-sync
+        clojure -M:cljs release db-sync-node
+        mkdir -p data
+        DB_SYNC_PORT=${{ inputs.port }} DB_SYNC_DATA_DIR="$PWD/data" \
+          nohup ./start.sh >"$PWD/db-sync.log" 2>&1 &
+        echo $! > "$PWD/db-sync.pid"
+        for attempt in $(seq 1 90); do
+          # Any HTTP response (including 401/404) means the port is bound.
+          if curl -s -o /dev/null "http://127.0.0.1:${{ inputs.port }}/"; then
+            echo "db-sync is listening on :${{ inputs.port }}"
+            exit 0
+          fi
+          sleep 2
+        done
+        echo "db-sync did not start; tail of log:" >&2
+        tail -100 "$PWD/db-sync.log" >&2 || true
+        exit 1
+
+    - name: Build and start db-sync in the background
+      if: inputs.background == 'true'
+      shell: bash
+      run: |
+        mkdir -p "$RUNNER_TEMP"
+        printf '%s\n' \
+          '#!/usr/bin/env bash' \
+          'set -e' \
+          'cd "$GITHUB_WORKSPACE/logseq-server"' \
+          'pnpm install' \
+          'cd deps/db-sync' \
+          'clojure -M:cljs release db-sync-node' \
+          'mkdir -p data' \
+          'DB_SYNC_DATA_DIR="$PWD/data" nohup ./start.sh >"$PWD/db-sync.log" 2>&1 &' \
+          'echo $! > "$PWD/db-sync.pid"' \
+          > "$RUNNER_TEMP/db-sync-bg.sh"
+        chmod +x "$RUNNER_TEMP/db-sync-bg.sh"
+        DB_SYNC_PORT=${{ inputs.port }} nohup "$RUNNER_TEMP/db-sync-bg.sh" \
+          >"logseq-server/deps/db-sync/build.log" 2>&1 &
+        echo "db-sync build started in background (pid $!)"
+
+    - name: Upload db-sync log
+      if: failure()
+      uses: actions/upload-artifact@v4
+      with:
+        name: db-sync-log-${{ runner.os }}
+        path: logseq-server/deps/db-sync/db-sync.log
+        retention-days: 7
+        if-no-files-found: ignore

.github/workflows/ci.yml

@@ -17,11 +17,12 @@ jobs:
       - name: System dependencies
         run: sudo apt-get install -y libsqlite3-dev libffi-dev pkg-config
 
-      - uses: ocaml/setup-ocaml@v3
+      - name: Restore opam root
+        id: opam-root
+        uses: actions/cache/restore@v4
         with:
-          ocaml-compiler: "5.5"
-          opam-pin: false
-          # "cache" (default true) caches the opam root + switch.
+          path: ~/.opam
+          key: opam-root-${{ runner.os }}-ocaml-5.5
 
       - name: Restore opam switch
         id: opam-switch
@@ -30,6 +31,33 @@ jobs:
           path: _opam
           key: opam-switch-${{ runner.os }}-ocaml-5.5-${{ hashFiles('logseq_chat.opam.locked', 'logseq_chat.opam') }}
 
+      # setup-ocaml runs `opam update` unconditionally (~40s). When both the
+      # root and the switch are cache hits the env is already complete, so
+      # only the opam binary is needed.
+      - name: Install opam binary
+        if: steps.opam-root.outputs.cache-hit == 'true' && steps.opam-switch.outputs.cache-hit == 'true'
+        run: |
+          mkdir -p ~/.local/bin
+          curl -fsSL https://github.com/ocaml/opam/releases/download/2.6.0/opam-2.6.0-x86_64-linux -o ~/.local/bin/opam
+          chmod +x ~/.local/bin/opam
+          echo "$HOME/.local/bin" >> "$GITHUB_PATH"
+          eval $(opam env)
+          opam --version && ocaml -version
+
+      - uses: ocaml/setup-ocaml@v3
+        if: steps.opam-root.outputs.cache-hit != 'true' || steps.opam-switch.outputs.cache-hit != 'true'
+        with:
+          ocaml-compiler: "5.5"
+          opam-pin: false
+          # "cache" (default true) caches the opam root + switch.
+
+      - name: Save opam root
+        if: steps.opam-root.outputs.cache-hit != 'true'
+        uses: actions/cache/save@v4
+        with:
+          path: ~/.opam
+          key: opam-root-${{ runner.os }}-ocaml-5.5
+
       - name: Install opam dependencies
         if: steps.opam-switch.outputs.cache-hit != 'true'
         run: |
@@ -68,7 +96,7 @@ jobs:
         uses: actions/cache/restore@v4
         with:
           path: duniverse
-          key: duniverse-${{ hashFiles('logseq_chat.opam.locked', 'logseq_chat.opam') }}
+          key: duniverse-${{ hashFiles('logseq_chat.opam.locked', 'logseq_chat.opam', '.github/workflows/ci.yml') }}
 
       - name: Pull vendored dependencies
         if: steps.duniverse.outputs.cache-hit != 'true'
@@ -81,8 +109,16 @@ jobs:
           git config --global url."https://x-access-token:${LOGSEQ_GITHUB_PAT}@github.com/".insteadOf "ssh://git@github.com/"
           git config --global url."https://x-access-token:${LOGSEQ_GITHUB_PAT}@github.com/".insteadOf "git+ssh://git@github.com/"
           git config --global url."https://x-access-token:${LOGSEQ_GITHUB_PAT}@github.com/".insteadOf "https://github.com/"
-          opam monorepo pull -y
-          for pin in $(grep -oE 'logseq/[A-Za-z0-9_-]+\.git#[0-9a-f]{40}' logseq_chat.opam | sort -u); do
+          # Call the binary directly: the `opam monorepo` plugin shim rejects
+          # plugins built under a different opam version (restored caches).
+          # Retry: release-tarball downloads occasionally fail with HTTP 5xx.
+          for attempt in 1 2 3; do
+            opam exec -- opam-monorepo pull && break
+            [ "$attempt" -lt 3 ] || exit 1
+            echo "opam-monorepo pull failed (attempt $attempt); retrying" >&2
+            sleep 10
+          done
+          for pin in $(grep -oE 'logseq/[A-Za-z0-9_-]+\.git#[A-Za-z0-9._/-]+' logseq_chat.opam | sort -u); do
             url="${pin%%#*}"; sha="${pin##*#}"; name="$(basename "$url" .git)"
             # opam monorepo pull already vendors lockfile entries at the right
             # commits (without .git dirs — `git -C` there would hit the parent
@@ -93,7 +129,7 @@ jobs:
             git clone --quiet "https://github.com/$url" "duniverse/$name"
             if ! git -C "duniverse/$name" checkout --quiet "$sha"; then
               git -C "duniverse/$name" fetch --quiet "https://github.com/$url" "$sha"
-              git -C "duniverse/$name" checkout --quiet "$sha"
+              git -C "duniverse/$name" checkout --quiet FETCH_HEAD
             fi
           done
 
@@ -102,7 +138,7 @@ jobs:
         uses: actions/cache/save@v4
         with:
           path: duniverse
-          key: duniverse-${{ hashFiles('logseq_chat.opam.locked', 'logseq_chat.opam') }}
+          key: duniverse-${{ hashFiles('logseq_chat.opam.locked', 'logseq_chat.opam', '.github/workflows/ci.yml') }}
 
       - name: Restore build caches
         uses: actions/cache/restore@v4