Refactor/submit wiki job

[GdoongMathew]

Summary

Reason

Wiki generation was orchestrated in the browser. That meant generation died when users close their pages, and couldn't prevent duplication across users.

Changes

Backend

• TaskRegistry: using repo_type + owener + repo to get or create a wiki generation job
• Porting the original page generation / structure and citation post processing pipeline into services/wiki module.
• Added endpoints POST /wiki/tasks (submit), GET /wiki/tasks (get completed + queued jobs), GET /wiki/tasks/{id}, GET /wiki/tasks/{id}/stream

Frontend (AI generated)

• New Next proxy routes + typed client (utils/wikiTask.ts: submit / list / get / subscribe).
• page.tsx submits a task, subscribes to SSE, drives the progress bar from pages_done/pages_total + the processing list, and loads the cache on completion. Removed the old browser orchestration and its helpers (~1.3k lines).
• Homepage lists /api/wiki/tasks (completed first, queued last, with an in-progress badge); inProgress added to all 10 locales.

[greptile-apps]

Greptile Summary

This PR moves wiki generation from the browser to the server by introducing a TaskRegistry backed by asyncio tasks, four new /wiki/tasks endpoints (submit, list, get, stream), and a full Python port of the frontend's indexing → structure → page-generation pipeline. The frontend is simplified to submit a task, follow an SSE progress stream, and load the finished wiki from the server cache.

• Backend task service (api/services/wiki/): new tasks.py drives the state machine (PENDING → INDEXING → DETERMINING_STRUCTURE → GENERATING → COMPLETED/FAILED); structure.py and content.py port the XML parsing and citation post-processing from the frontend; io.py centralises cache file I/O.
• New API endpoints: POST /wiki/tasks (get-or-create with dedup), GET /wiki/tasks (merged completed + active list), GET /wiki/tasks/{id}, GET /wiki/tasks/{id}/stream (SSE with progress/done/error named events).
• Frontend overhaul (page.tsx): ~1 300 lines of browser-side WebSocket/HTTP orchestration replaced with submitWikiTask → subscribeWikiTask → loadWikiFromServerCache; homepage now queries the unified task list for in-progress badges.

Confidence Score: 4/5

Safe to merge after fixing the file-filter argument order in _determine_structure; all other paths are unaffected.

The positional arguments passed to read_repo_file_tree in _determine_structure are in the wrong order: excluded dirs/files land in the included slots and vice versa. This silently inverts every file filter a user specifies, producing a wrong file tree for the structure-determination LLM prompt. The default case (no filters) is unaffected, which is why tests pass, but any deployment that uses included/excluded dirs or files for wiki generation would see incorrect results.

Files Needing Attention: api/services/wiki/tasks.py — the _determine_structure function's call to read_repo_file_tree has its four filter arguments in the wrong positional order.

Important Files Changed

Filename	Overview
api/services/wiki/tasks.py	New task registry and wiki generation state machine. Contains a P1 bug: positional arguments to read_repo_file_tree are in the wrong order, swapping included/excluded filter semantics. Also missing a finally guard in _run for CancelledError resilience.
api/services/wiki/structure.py	Port of frontend wiki-structure parsing: file tree reading, default branch detection, and XML parse with regex fallback. Parameter order in read_repo_file_tree is correct internally (keyword args to iterate_files); the mismatch is in the caller.
api/routers/wiki.py	New /wiki/tasks endpoints (submit, list, get, SSE stream). SSE error event now uses valid JSON. Routing and response models look correct.
api/services/wiki/io.py	Refactored wiki cache I/O (read, write, delete, list). list_wiki_cache now returns WikiTaskSummary directly. Filter logic for cache filenames is correct.
api/services/wiki/content.py	Python port of the frontend citation/link post-processing. Pure functions, well-tested. Implementation matches the removed frontend logic closely.
src/utils/wikiTask.ts	New typed client for the wiki-task backend API. SSE subscription correctly distinguishes server-sent error frames from connection-level errors. Clean and well-structured.
src/app/[owner]/[repo]/page.tsx	Major refactor: ~1.3k lines of browser-side orchestration removed, replaced with submitWikiTask → subscribeWikiTask → loadWikiFromServerCache flow. SSE progress drives the progress bar and page list cleanly.
api/schemas/repo.py	Adds WikiTaskRequest, TaskStatus enum, and WikiTaskSubmitResult schemas. repo_key omits language (previously flagged in thread). _status_validate correctly coerces string input to TaskStatus.
api/config.py	Adds iterate_files and _should_process_file, extracted from rag/pipeline.py so both indexing and structure listing share one implementation. Logic matches the original.
tests/backend/services/test_wiki_task.py	Good coverage of TaskRegistry and generate_repo_wiki state machine, but read_repo_file_tree is always mocked with (*a, **k), so the positional argument mismatch in _determine_structure goes undetected.

Sequence Diagram

sequenceDiagram
    participant Browser
    participant NextProxy as Next.js Proxy
    participant FastAPI
    participant Registry as TaskRegistry
    participant Worker as asyncio Task

    Browser->>NextProxy: POST /api/wiki/tasks
    NextProxy->>FastAPI: POST /wiki/tasks
    FastAPI->>Registry: submit(WikiTask)
    alt cache hit
        Registry-->>FastAPI: "from_cache=true"
        FastAPI-->>Browser: "{from_cache:true}"
        Browser->>NextProxy: GET /api/wiki_cache
        NextProxy->>FastAPI: GET /api/wiki_cache
        FastAPI-->>Browser: WikiCacheData
    else new/joined task
        Registry->>Worker: asyncio.create_task(_run)
        Registry-->>FastAPI: "{created:true, task_id}"
        FastAPI-->>Browser: "{task_id, created/joined}"
        Browser->>NextProxy: "GET /api/wiki/tasks/{id}/stream"
        NextProxy->>FastAPI: "GET /wiki/tasks/{id}/stream (SSE)"
        loop every 1s until terminal
            Worker->>Worker: INDEXING to DETERMINING_STRUCTURE to GENERATING
            FastAPI-->>Browser: event: progress
        end
        Worker->>FastAPI: save_wiki_cache()
        Worker->>Registry: "status = COMPLETED"
        FastAPI-->>Browser: event: done
        Browser->>NextProxy: GET /api/wiki_cache
        NextProxy->>FastAPI: GET /api/wiki_cache
        FastAPI-->>Browser: WikiCacheData
    end

Loading

_{Reviews (2): Last reviewed commit: "add tests and fix comments" | Re-trigger Greptile}

[greptile-apps]

repo_key omits language — concurrent multi-language requests collide

repo_key is {type}_{owner}_{repo} with no language component, but a wiki is language-specific. If User A submits for en and User B then submits for fr on the same repo while User A's task is still in GENERATING state, TaskRegistry.submit finds the active key and returns joined=True to User B. User B now subscribes to SSE progress for an English-language generation. When the task completes and User B's frontend fetches the cache for fr, it gets a cache miss and the page displays an error or empty state.

Suggested change

	@property
	def repo_key(self) -> str:
	return f"{self.type}_{self.owner}_{self.repo}"
	@property
	def repo_key(self) -> str:
	return f"{self.type}_{self.owner}_{self.repo}_{self.language}"