[ENG-2022] Make Roam publishing sync-independent with complete node upserts - #1280
[ENG-2022] Make Roam publishing sync-independent with complete node upserts#1280sid597 wants to merge 5 commits into
Conversation
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
PR size/scope checkThis PR is over our review-size guideline.
Please split this into smaller PRs unless there is a clear reason the changes need to land together. If keeping it as one PR, please add a brief justification covering:
|
|
This pull request has been ignored for the connected project Preview Branches by Supabase. |
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 232615a8fa
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
ff9d52b to
7a4bdd1
Compare
…erts Add an 'Enable node sharing' feature flag and gate the Publish tab, 'DG: Share current node' command, and page-title Publish button on it instead of the suggestive-mode overlay flag. Publishing now converts selected nodes to full CrossAppNodes (direct title + full markdown) and upserts schema, concept, and contents before granting group access; nodes whose upsert fails are excluded from grants.
…ting publish timestamps upsert_concepts returns -1 for unique violations and -2 for other errors; only -1 was treated as a failure, so a -2 node kept its ResourceAccess grant. Node timestamps now follow the sync query's fallback chain (create -> edit -> page-edit) instead of falling back to Date.now(), which would have persisted publish time as last_modified whenever :page/edit-time was absent. failedSyncedUids is renamed to failedUpsertUids and the publish toast counts only failed selected nodes.
… without sync The sync loop now also starts when only node sharing is enabled. In that mode it scopes node upserts to shared nodes and uploads their content without generating embeddings; users, shared full-content refresh, concept conversion, and orphan cleanup run as before. With the sync flag on, behavior is unchanged.
Shared-nodes-only cycles previously completed the same "embedding" sync task, advancing its watermark without producing embeddings. Enabling the sync flag later would then skip the initial embedding backfill for nodes whose content was already uploaded. Keeping the two modes on separate sync_info rows leaves the embedding watermark untouched until full sync actually runs, and stops the two modes from postponing each other's cycles in mixed-flag spaces.
A failed schema upsert leaves its concept absent, so dependent nodes insert with a null schema_id and shared-node discovery filters them out. Treat those nodes as failed too: no ResourceAccess grant, counted in the failure toast, and relations touching them are withheld.
7a4bdd1 to
accf819
Compare
maparent
left a comment
maparent
left a comment
There was a problem hiding this comment.
Looks good overall, but for one issue.
|
|
||
| const neededUids = [ | ||
| ...nodeSchemaUids, | ||
| ...nodeUids, |
There was a problem hiding this comment.
What is the point of removing this? If I understand well, the consequence is always upsetting all concerned nodes. What if they had been upserted previously?
In my understanding, someone may re-request publishing a node, esp. to a new group.
There was a problem hiding this comment.
The query "which of these UIDs already exist in my_concepts?" did two different jobs for two different ID sets:
- For schemas / relation-schemas / relations: a dedup filter — figure out which are missing so publish can upload just those. Publish always had an upload path for these.
- For nodes: a gatekeeper — decide which selected nodes are allowed to be published at all. Present → grant access. Absent → skip, report "not synced yet." Publish had no upload path for nodes; it fully trusted sync to have put them there.
The fix made publish upload every selected node itself, which means publish no longer needs sync's permission — the gatekeeper job for nodes ceased to exist. nodeUids came out of the query because its answer would no longer drive anything. The removal is fallout of the fix, not part of it.
I think the case that you put up is 1/3 cases
| Case | State in DB | Skip upload safe? |
|---|---|---|
| 1. Previously published | Complete row, kept fresh by background loop |
Yes — upload is redundant (map's case) |
| 2. Only touched by sync (sync on, first publish) |
Row exists but it's the title-only stub |
No — group gets granted a stub with no content |
| 3. Never uploaded (sync off, first publish) |
No row | Moot — uploaded either way (the ENG-2022 bug) |
There was a problem hiding this comment.
Good point that I missed case 2, but are you saying that you are choosing to not handle case 1?
There was a problem hiding this comment.
case 1 is handled by rewriting the same values with the node's own timestamps and never touches embeddings, so re-publishing is idempotent.
To skip safely, "row exists" isn't enough — publish would have to answer two more questions:
- Is it complete? Does a variant='full' Content row exist, or is it just the direct-variant stub regular sync uploads?
- Is it fresh? The loop is eventually consistent on a ~5-min cadence, so a just-edited node can be stale at the moment of publish.
Answering those ties publish's correctness back to sync's state — effectively rebuilding sync's diff logic inside publish — to save one idempotent write on a hand-picked selection.
There was a problem hiding this comment.
I accept this argument. Approving.
| missingNodeSchemas.map((s) => s.localId), | ||
| ); | ||
|
|
||
| if (upsertConcepts.length > 0) { |
There was a problem hiding this comment.
If we do not always upsert all nodes, the test stands.
| } else if (syncedRelationUids.has(localId)) { | ||
| syncedRelationUids.delete(localId); | ||
| } | ||
| result.failedUpsertUids.push(localId); |
2 participants
eng-2022.mp4
Makes Roam publishing self-sufficient instead of piggybacking on background sync: publish now upserts the complete node — concept, direct title, and full markdown — through
upsert_concepts, rather than only granting access to nodes sync had already pushed. The synced-nodes pre-filter is gone; an RPC failure grants nothing, a per-row failure withholds that node's grant (and any relation touching it) and surfaces asfailedSyncedUids, replacingskippedUnsyncedUids.Node sharing gets its own feature flag (
Enable node sharing), decoupled from the suggestive-mode/sync flag: the Publish tab, the page-title Publish button, andDG: Share current nodemove to it, while discovery/import (DG: Discover shared nodes) stays on the sync flag.This is the local ENG-2022 branch rebased onto current main (clean, no conflicts). #1279 (ENG-1860) is stacked on top.
ENG-2121 check
$scope-checkagainst ENG-2022 and the final diff.Done When: None.