Policy and guidance on upgrades and backwards-compatibility between versions

[parshimers]

Hi folks,
I couldn't find any previous discussions about this topic, so I figured I'd start one, since I think it is an important issue.
Currently, as I understand it, there are no specific guarantees about compatibility between released versions of Texera. That means you can't really be sure you could perform any kind of upgrade (e.g. from 1.2.0 to 1.2.1) without having to completely recreate all your workflows, datasets, and user logins from scratch.
That certainly seems like it would be an issue to anyone wishing to deploy a Texera instance, even as an informal internal staging environment, let alone an outward facing service. Clearly we can't maintain compatibility between all versions, but at least we can do it in certain circumstances. I think the versioning scheme and release practices kind of have an implicit intention behind them related to this.
My proposal would be to enforce and advertise this practice:

• Bugfix releases (x.x.n -> x.x.n+1) should be entirely backwards and forwards compatible.
• Minor releases (x.n.x -> x.n+1.x) may require migration advice on upgrade, but should be backwards compatible.
• Major releases (n.x.x -> n+1.x.x) may be entirely incompatible in either direction, and require significant manual work to migrate to. Entire features that existed in a previous version may not exist, or are changed in an incompatible way.

We should have CI tests to enforce this if it is accepted. It is very easy to accidentally introduce a change which is backwards or forwards incompatible. Pretty much anything that serializes, especially to persistent storage, can cause it.

What does everyone think?

[mengw15]

I think having an upgrade test that verifies persistent data survives is a good idea.

The MinIO issue is a good example. Ideally, a 1.2.0 → 1.2.1 upgrade should work without any migration, but if the test shows that it does not, we may need to provide a migration path even for a patch release. In this case, I think the MinIO image availability issue is serious enough that we still need to address it rather than avoid the change for compatibility reasons.

[parshimers]

Agreed. I think it is an exceptional case, and what made me want to start this discussion. You can always weasel out a bit on the "should" being there, but it's best to avoid that.
I think a lemma to this policy is that we should ensure all of our downstream dependencies have compatible guarantees with this one. The MinIO situation is a consequence of an incompatible guarantee: it was entirely unsupported, and in fact the original authors had a demonstrated desire to kill the product as it existed originally. My proposal would be that we should ensure that at the very least, what we release has no deprecated or unsupported dependencies at the time of the release. If we really need them, then we should vendor them in and maintain them ourselves. If at any time the dependency becomes unsupported, we (hopefully with some warning) announce that it's the end of the line for that version.

[xuang7]

Thanks for starting this discussion. I think we currently don't guarantee full compatibility across upgrades, so it would be good to make users aware of this and recommend backing up important resources before upgrading.

I agree with having upgrade tests. A simple and useful version would be to boot version N, load some data through the app, then boot N+1 on the same volumes and verify that the data is still readable through the API. This should be possible to automate. I would avoid making patch releases an absolute compatibility guarantee, though. A more practical rule could be that patch releases should not break compatibility by default, and if an exception is unavoidable, the release should include a clearly documented.

This is especially important for the storage changes we recently made. Some use cases may involve TB-scale datasets, so even a technically supported migration can have significant operational cost. For future releases, if there are any known compatibility issues, we should clearly document them along with the supported migration path.