Skip to content

[Bug]: 0.21.4 CLI is not compatible with servers < 0.21.3 (service groups) #4255

Description

@un-def

Steps to reproduce

  1. Run a dstack server 0.21.2 (or any version < 0.21.3).
  2. Use a 0.21.4 CLI against it.
  3. Apply any service configuration, e.g.:
type: service

name: httpbin

image: ghcr.io/psf/httpbin:0.10.2
replicas: 1
port: 8080
https: auto
auth: false
gateway: true
dstack apply -f .dstack/httpbin.yaml

Compatibility matrix observed:

client server result
0.21.3 0.21.2 ok
0.21.4 0.21.3 ok
0.21.4 0.21.2 fails

Actual behaviour

Server validation error:
{'detail': [{'input': None,
             'loc': ['body', 'run_spec', 'configuration', 'service', 'groups'],
             'msg': 'Extra inputs are not permitted',
             'type': 'extra_forbidden'}]}

The 0.21.4 client always serializes configuration.groups for services, including as
null for a homogeneous service. Servers before 0.21.3 have no such field and reject it
because request models are extra="forbid".

The regression can be reproduced without a running server by validating the client's
request payload against the 0.21.2 request schema:

run_spec.configuration.service.groups
  Extra inputs are not permitted [type=extra_forbidden, input_value=None, input_type=NoneType]

Expected behaviour

Any service configuration that does not use the 0.21.3+ groups feature should apply
successfully against older servers, as it did with the 0.21.3 client. Per the client
compatibility convention documented in dstack/_internal/core/compatibility/runs.py,
new fields must be excluded from requests when they are not set.

dstack version

0.21.4 (client); server 0.21.2

Server logs

Not relevant — the request is rejected by request-body validation before it reaches any
run logic. The 422 response body is quoted under "Actual behaviour".

Additional information

ServiceConfigurationParams.groups was introduced in 0.21.3 by #4177, which renamed
service replica groups from the legacy replicas: [...] list to groups. That PR also
added a @model_serializer(mode="wrap") (_serialize_legacy_replica_groups) on
ServiceConfigurationParams that popped groups out of every dump and re-emitted it as
the legacy replicas: [{count: ...}] shape. Because it lived on the model, it applied in
both directions, so a 0.21.3 client never put groups on the wire — which is why
0.21.3 → 0.21.2 works.

#4248 removed that serializer and moved the rewrite to server/compatibility/runs.py,
gated on client_version < 0.21.3. That covers the server → old client direction, but
nothing replaced the client → old server direction, so 0.21.4 clients now always send
groups.

ServiceConfiguration.groups is the only field added since 0.21.2 across the service,
task and dev-environment configurations, RunSpec and ProfileParams, so this appears
to be the only source of 0.21.4 → 0.21.2 incompatibility.

Note that #4248 also dropped the ability to run a heterogeneous service (groups
actually set) against a pre-0.21.3 server, which the 0.21.3 client supported by
downgrading the value to the legacy replicas: [{count: ...}] shape. Restoring that is
not expressible with the client's exclude-mapping mechanism, which can only drop fields,
not rename them.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions