venv site-packages building requires millions of actions

[rickeylev]

On slack, a user reported that using the venv_site_packages=yes flag caused millions of actions to occur. More user details would help verify its due to the flag.

That said but millions of actions is believable if there are many large targets and the conflict detection ends up having to recreate every file under site-packages. e.g. Given 100 tests (reasonable number), and 50,000 files in their deps (large, but not unheard of), then you end up with 5 million actions. No matter what, bazel is going to have to materialize 5 million files in such a case, but if we have to, at the analysis phase, perform 5M symlink() re-declarations, that's a decent amount of overhead.

At the least, we need to optimize the conflict resolution logic. When I fixed it to handle shared libraries better, I just did a simple and naive implementation. A more optimized version would look for a new minimal set of directories/files to symlink instead of just symlinking all of them. Similarly, runfiles.root_symlinks might come in handy here to avoid most symlink()-to-File action calls.

From slack:

Kris Wilson
Oct 19th at 8:55 PM
this also seems to cause the number of actions for a bazel test //... to shoot up into the multi-millions.. which is strange. is this creating one action per symlinked file in the venv? (edited)

[rickeylev]

I think this is about as optimized as possible for now, so closing. The two possible improvements I can think of are:

1. Apply the same "minimal number or links" logic to the conflict-merge logic (more actions at the expense of fewer total files
2. Have a tool run to create all the symlinks in one action instead of many separate starlark declare_symlink+symlinks action

For 1, conflicts should be much less common now, so idk how much benefit it'll bring.

For 2, idk if it would bring meaningful benefits because we still have to flatten depsets in the analysis phase (this allows better incremental dependency tracking)

A more fuzzy optimization I had in mind was to propagate a smaller depset of known (or likely to be, or expensive to resolve) paths, then have py_library do a smaller conflict resolution to avoid a larger resolution later. The case i had in mind being: say you had a single modular python package (eg a monorepo), but want a site-packages layout.

In any case, optimizations to consider later, if need be.

[FrankPortman]

I am not sure whether I am an outlier, but on 2.0.0, I tried to flip this flag again and the total actions on my polyglot repo went from 18k to millions as well. Is there any diagnostic info I can give you to help triage and potentially further improve performance?

[rickeylev]

I think an analysis of a single target is a good place to start.

A recursive listing of a binary's .venv directory would tell us how many symlinks are being created. We would expect one per pypi dependency, with some outliers.

[rickeylev]

Actually, here's a simpler ask: the requirements.txt file
That should capture all the pypi deps that get processed, from which we can look for pathological cases

[FrankPortman]

https://gist.github.com/FrankPortman/a2b8d092dba7c31c3160be5d6a527e3b

[rickeylev]

Woo, that's a big requirements. Poking around a bit, I'm not seeing any obvious bugs.

There are two pathlogical cases I see, though.

The first is cases like azure/mgmt. This directory is 12 distributions all installing into it, for a total of thousands of files. There's some others doing this, but not nearly as big as azure/mgmt.

The second case are shared libraries. scipy, tensorflow, and sklearn seem to be the big ones here. Each is a few hundred files. Not as big as azure, but several of them add up.

I have a few ideas to try to optimize this.

[FrankPortman]

And here I thought our codebase was still on the smaller side 😅 . I guess you asymptote towards "use all the packages" faster than you write source code.

[yilei]

On root_symlinks: right now, each binary makes its own symlink into a wheel, since the relative target depends on where the binary lives. That's why it could end up millions of symlink actions.

@rickeylev WDYT if we put site-packages at a fixed-depth path instead, e.g:

<binary>.runfiles/_venvs/<repo+pkg+name>/lib/python3.x/site-packages

and make

<binary>.runfiles/<repo>/<pkg>/<venv>/lib

a symlink to it? This way, the target is the same for every binary, so the wheel's py_library can declare the symlink once and binaries just map it in with root_symlinks. For single files, they can also go into root_symlinks directly just like .sos.

This could eliminate like ~99% of the symlink actions.