Skip to content

Fix HTTP 404 response for unknown doc pages, and render during build - #1860

Open
marcleblanc2 wants to merge 2 commits into
mainfrom
fix-real-404-status
Open

Fix HTTP 404 response for unknown doc pages, and render during build#1860
marcleblanc2 wants to merge 2 commits into
mainfrom
fix-real-404-status

Conversation

@marcleblanc2

@marcleblanc2 marcleblanc2 commented Sep 6, 2026

Copy link
Copy Markdown
Contributor

Problems

  1. https://sourcegraph.com/docs/this-page-does-not-exist returns HTTP 200. The "Page not found" UI renders, but only via the client-side NEXT_NOT_FOUND digest. Any status-code-based link checker (lychee, W3C, etc.) therefore reports the docs site as 100% clean no matter how many links are dead.
  2. Page load times were up to 30s on fresh deploys

No HTTP 404 response on missing pages

Before: HTTP 200
image

After: HTTP 404
Note: the additional HTTP 204 shown in this screenshot is the Vercel preview toolbar, won't be shown in prod
image

Root cause

Two things stacked up in src/app/[...slug]/page.tsx:

  1. generateStaticParams returned {params: {slug}}, the Pages Router shape. App Router expects {slug}, so Next silently ignored it: zero doc pages were prerendered and every page has been rendered on demand by a serverless function since the site launched.
  2. notFound() runs inside the root layout's <Suspense>. By then the shell has already streamed with a 200, so Next can't change the status code.

History

The site was always meant to be static; the function has been broken three different ways:

  • Nov 2023, original routes: allPosts.map(post => { ({slug: ...}); }) — block body, no return, yields undefined per item.
  • Feb 2024, 📦 NEW: Add versioning #48 "Add versioning": rewritten to {params: {slug}}. PR body says "Previously Server routes are now SSG, speeding up page load times significantly." Wrong shape, so nothing changed.
  • May 2024, Docs for Sourcegraph version 5.4 #308: export const maxDuration = 300 added to the route with no explanation. That's the Vercel function timeout — only needed because pages were rendering on demand and the big ones exceeded the default (see 25–30 s renders below).

Fix

  • Return {slug} from generateStaticParams so all doc pages actually prerender (523 routes).
  • Exclude the root document (flattenedPath === ''), which is served by app/page.tsx. Including it fails the build with export path '/' doesn't match '/[...slug]'.
  • export const dynamicParams = false so unknown slugs are rejected at the router with a real 404, before rendering/streaming starts.

not-found.tsx is untouched; users see the same 404 page as before. maxDuration = 300 is removed: the route no longer runs as a serverless function, so the timeout setting had nothing left to configure.

Proof: before vs after on Vercel

Every URL in sitemap.xml (522) was fetched with curl, capturing x-vercel-cache, x-matched-path, x-vercel-id, age, and time_total.

Before — a fresh deployment of main (temporary branch cold-baseline-main, empty commit, since deleted), swept immediately after it went live so nothing was cached:

1st pass (cold) 2nd pass
x-vercel-cache MISS 517 / HIT 4 / PRERENDER 1 HIT 522
x-matched-path /[...slug] on 520/520 same
x-vercel-id pdx1::iad1::… (edge + function region) on 520/520 same
time_total median / p90 / max 0.48 s / 0.68 s / 30.1 s 0.19 s / 0.22 s / 0.64 s

Slowest cold renders: /technical-changelog 30.1 s, /self-hosted/observability/dashboards 24.7 s, then a cluster at ~9 s (concurrent cold lambda starts). Both >15 s pages would 504 without maxDuration = 300.

Production today (sourcegraph.com/docs): all 520 pages already HIT, x-matched-path: /[...slug], function region present. age values cluster in three bursts 34.5–39 h old (deploy was 39 h earlier), i.e. crawlers walked the site and forced every page to render after the last deploy. Any never-requested URL is a MISS that invokes the function and returns a 200 not-found page, which is then cached.

After — this PR's preview, also a fresh deployment swept immediately after it went live:

1st pass (cold) 2nd pass
x-vercel-cache PRERENDER 516 / HIT 6 (prebuilt file, first serve) HIT 522
x-matched-path concrete path (e.g. /batch-changes/batch-spec-yaml-reference) 522/522 same
x-vercel-id pdx1::… (edge only, no function region) 522/522 same
time_total median / p90 / max 0.39 s / 0.45 s / 1.2 s 0.19 s / 0.22 s / 0.48 s
never-seen URL 404, x-matched-path: /404, static, no function

Side by side, first-visitor pass: median 0.48 s → 0.39 s, p90 0.68 s → 0.45 s, max 30.1 s → 1.2 s, pages over 2 s 10 → 0. Warm passes are identical (~0.19 s), which is why the on-demand rendering went unnoticed.

Other checks

  • All 522 sitemap URLs return 200 on the preview (no page lost to dynamicParams = false).
  • Real pages carry full prerendered content.
  • Middleware unaffected: .md rewrite → 200, trailing slash → 308, /@6.4/... → 307 to versioned host, /api/og/... → 200.
  • next build: 526/526 static pages, ~22 s locally (Node 20.20.2). Was 9. Vercel preview build passed.
  • npx tsc --noEmit clean.

Consequences

  • First reader of each page after a deploy no longer waits on a cold lambda render (0.5–30 s); everyone gets a CDN file.
  • MDX render errors now fail the build instead of 500ing in production.
  • Content freshness unchanged: there was never a revalidate, so pages were already cached until the next deploy.
  • External link checkers will start reporting real broken page links on sourcegraph.com/docs. That's the point; expect the count to jump from zero.
  • Pages with preview: true frontmatter are in allPosts, so they prerender and PreviewGuard still gates them by ?preview. None exist in docs/ today, so that path is untested here.

generateStaticParams returned {params: {slug}} (Pages Router shape), so the
App Router ignored it: no doc page was prerendered and every request rendered
on demand. notFound() then fired inside the root layout's <Suspense>, after
the response had started streaming, so missing pages were served with 200.

Return {slug} so all pages prerender, exclude the root document (served by
app/page.tsx), and set dynamicParams = false so unknown slugs are rejected
at the router with a 404. The existing not-found.tsx page is unchanged.

Amp-Thread-ID: https://ampcode.com/threads/T-01a07590-7d1a-72ec-826b-9081eed4f715
Co-authored-by: Amp <amp@ampcode.com>
@vercel

vercel Bot commented Sep 6, 2026

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
sourcegraph-docs Ready Ready Preview Sep 6, 2026 9:39am UTC

Request Review

The route is now fully prerendered, so it no longer runs as a serverless
function and the timeout setting has no effect.

Amp-Thread-ID: https://ampcode.com/threads/T-01a07590-7d1a-72ec-826b-9081eed4f715
Co-authored-by: Amp <amp@ampcode.com>
@marcleblanc2 marcleblanc2 changed the title fix: return a real HTTP 404 for unknown doc pages Fix HTTP 404 response for unknown doc pages, and pre-render on build Sep 6, 2026
@marcleblanc2 marcleblanc2 changed the title Fix HTTP 404 response for unknown doc pages, and pre-render on build Fix HTTP 404 response for unknown doc pages, and render during build Sep 6, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant