bootstrap: Enable rustdoc mergeable CCI for std and internal docs

[notriddle]

Takes a different approach to #160098, where the internal docs are merged by bootstrap directly invoking rustdoc. This requires bootstrap to gather the list of metadata directories by inspecting cargo's fingerprint files (which aren't stable). The first commit is written by @camelid, but I wrote the other two.

This feature is needed because:

1. The search index (that powers web-based search) needs to contain all of the crates in the nightly-rustc project. In particular, I'd prefer if it contained Clippy, Rustdoc, and Rustc, since those crates share type checker stuff and the ability to search all three at once is convenient.
2. For every crate that rustdoc currently documents, it has to load the search index from the doc output dir, and rebuild the search index with the new crate added to it. Loading the search index requires $O(\text{crates})$ work, so doing it once for every crate means we're doing $O(\text{crates}^2)$ work overall.
3. It would be more efficient, instead, if each crate wrote its data separately, and then the final search index was generated at the end by merging them all at once. Obviously, this would make the work linear instead of quadratic. For the record, Hoogle and Sherlodoc have a similar index-generating step.
4. We call this "Mergeable Cross-Crate-Information." Cargo stores it in the build directory, and supplies it to Rustdoc in a separate phase that runs after everything else. When we eventually stabilize this feature, it will be invisible to (most) end users. cargo doc will just be faster.
5. So, in order for crates to share their cross-crate info, we need them to share a build directory.
6. Tools, like Rustdoc and Cargo, don't normally share a build directory with Rustc.
7. To make them share a build directory while generating documentation, without forcing them to share a build directory while compiling, I added a new mode.

---

This rustdoc feature is unstable but will be stabilized soon, and this is a good way of dogfooding it to make sure it works properly. It should have no effect on the generated docs, but it provides a significant speedup. For example, I measure a 3x speedup locally (3m 11s -> 1m 1s) for x doc src/tools -- note that this is with the latest rustdoc perf improvements (#159854).

r? @Kobzol

[rustbot]

Kobzol is not on the review rotation at the moment.
They may take a while to respond.

[Kobzol]

Note to myself and other bootstrap reviewers: bootstrap currently does a bunch of docs hacks. One of them is the (implicit) combination of rustc + various tools docs into a single directory. I wanted to refactor that by merging the files explicitly in a separate step. But since we also want to enable -Zrustdoc-mergeable-info, it seems wasteful to first do that refactoring and then enable the flag, which requires us to combine the files in a different way.

So this PR does two things:

• Create a separate step for building the compiler + tools shared documentation.
• Enable -Zrustdoc-mergeable-info to make documentation generation faster.

There are several cleanups that we should do on top of this, but for this PR, it is important to check whether it produces the same combined output as before. I will do that.

Would you mind if I did a few small refactorings and push them to this PR?

View changes since this review

[Kobzol]

Just so that I understand it correctly, there is no cargo or rustdoc command that would do this for us, right? And this is orthogonal to -Zrustdoc-mergeable-info?

[Kobzol]

Running the step failed for me locally with this:

Copying "/projects/personal/rust/rust/build/x86_64-unknown-linux-gnu/stage1-rustc/doc/rustc_main" to "/projects/personal/rust/rust/build/x86_64-unknown-linux-gnu/compiler-doc/rustc_main"

thread 'main' (167152) panicked at src/bootstrap/src/core/build_steps/doc.rs:948:25:
src.symlink_metadata() failed with No such file or directory (os error 2) ("src = /projects/personal/rust/rust/build/x86_64-unknown-linux-gnu/stage1-rustc/doc/rustc_main/index.html")

The copy happens at

if proc_macro_doc_dir.exists() {
    eprintln!("Copying {proc_macro_doc_dir:?} to {doc_out:?}");
    builder.cp_link_r(&proc_macro_doc_dir, &doc_out);
}

but it is quite suspicious, because the stage1-rustc/doc dir is actually a symlink to compiler-doc.

[Kobzol]

The rabbit hole goes deeper.. after these changes, just running x doc compiler will produce incomplete docs, because the host docs (e.g. rustc_type_ir_macros) will be generated into a separate directory, and no longer combined. So we do indeed have to do the docs combining even for the individual docs steps, not just for CompilerDoc.

So the combination happens even without joining compiler/tools docs, we still have to join target/host docs. That's quite annoying - is this a problem for all Rust crates that have proc macros?!

If I understand it correctly, rustdoc can only merge the index page and the search index, but not the individual subdirectories with the actual docs. So the combining has to be done manually (is there really no way this can be done by cargo/rustdoc natively?), either by:

• Symlinking the host/target output directories to a shared directory (this was done previously) before the docs are built.
• Manually hardlinking/copying the built directories into a shared directory after the docs are built.

[notriddle]

Would you mind if I did a few small refactorings and push them to this PR?

No problem. Go ahead.

And this is orthogonal to -Zrustdoc-mergeable-info?

Yes. Cargo itself doesn’t need rustdoc to take care of copying the docs, because Cargo still uses the same output directory for the HTML when documenting each crate. Since each crate puts its HTML in a separate subdirectory anyway, they’re already separate. Writing those HTML files to separate directories and then copying them into place at the end would be wasteful.

[notriddle]

but it is quite suspicious, because the stage1-rustc/doc dir is actually a symlink to compiler-doc.

If you delete your build dir and start over, does that problem go away? Or did I miss a spot? I thought I deleted all of the places where those symlinks were being created, but I didn’t write any code to clean up the existing structure if it was left over from a previous, older build.