feat: Add kube-http-proxy and kube-socks5 Cargo features

[lfrancke]

Description

The default features are unchanged.
This allows us to enable these features in stackablectl (and elsewhere). Instead of ignoring a HTTP(S)_PROXY variable kube just refuses to build a client if it sees them but not the feature.

We moved stackablectl to a direct dependency on operator-rs in March via stackabletech/stackablectl#426
That PR unfortunately also dropped support for HTTP/SOCKS proxy. I cannot find any evidence of this being a concious choice.

When this is merged and released we can enable the feature in stackablectl.

Reported on Discord: https://discordapp.com/channels/796665978481803304/1078284084247265300/1544734994856026193

Definition of Done Checklist

Reviewer

• Changelog updated
• Cargo.toml only contains references to git tags (not specific commits or branches)

[Techassi]

I cannot find any evidence of this being a conscious choice.

Afair, it was not. It most likely slipped by unnoticed.

[tribenz]

NO_PROXY is not respected by kube-client for the Kubernetes API connection (after #453)

Summary

After #453 re-enabled kube's proxy support, requests to the Kubernetes API server are correctly routed through the configured HTTP_PROXY/HTTPS_PROXY, but the NO_PROXY/no_proxy exception list is not honored for these requests. This causes connections to a local API server (https://127.0.0.1:6443, e.g. k3s) to be tunneled through the proxy, where they fail because 127.0.0.1 is only valid from the client's own network perspective, not the proxy's.

Both curl and kubectl, run on the exact same host with the exact same environment variables and kubeconfig, correctly bypass the proxy for this address. Only stackablectl's kube-client does not.

Environment

stackablectl version	1.5.1 (includes #453)
kube-client version	4.2.0 (from debug log)
Kubernetes distribution	k3s v1.36.3+k3s1 (not vanilla k8s)
Container runtime	containerd://2.3.2-k3s2
OS	Red Hat Enterprise Linux 10.2
Topology	Single control-plane node; stackablectl runs directly on the k3s host
API server	https://127.0.0.1:6443
Proxy	Separate internal/corporate proxy, not running on the same host

Kubeconfig

No proxy-url field is set — proxy behavior is expected to come purely from environment variables.

apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: [REDACTED]
    server: https://127.0.0.1:6443
  name: default
contexts:
- context:
    cluster: default
    user: default
  name: default
current-context: default
kind: Config
users:
- name: default
  user:
    client-certificate-data: [REDACTED]
    client-key-data: [REDACTED]

Environment variables

Both upper- and lowercase variants are set identically:

http_proxy=http://proxyserver:3128
HTTP_PROXY=http://proxyserver:3128
https_proxy=http://proxyserver:3128
HTTPS_PROXY=http://proxyserver:3128
no_proxy=localhost,127.0.0.1,.cluster.local,10.50.5.17,172.17.0.0/16
NO_PROXY=localhost,127.0.0.1,.cluster.local,10.50.5.17,172.17.0.0/16

Steps to Reproduce

1. On the k3s host, with the environment variables above set.
2. Run any stackablectl command that talks to the Kubernetes API, e.g.:

   stackablectl demo install data-lakehouse-iceberg-trino-spark -l debug
   

Reference test 1: curl (same host, same env vars)

$ curl -v --max-time 5 https://127.0.0.1:6443/api/v1/nodes -k
* Uses proxy env variable no_proxy == 'localhost,127.0.0.1,.cluster.local,10.50.5.17,172.17.0.0/16'
*   Trying 127.0.0.1:6443...
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
...
* Connected to 127.0.0.1 (127.0.0.1) port 6443
* using HTTP/2
...
< HTTP/2 401
...
"message": "Unauthorized"

curl explicitly logs that it evaluated no_proxy, correctly bypasses the proxy, and connects directly to 127.0.0.1:6443 (the 401 is expected here since no client cert was passed — the connection itself succeeds).

Reference test 2: kubectl (official Go client, same kubeconfig, same env)

$ kubectl get nodes -o wide
NAME                     STATUS   ROLES           AGE   VERSION        INTERNAL-IP   EXTERNAL-IP   OS-IMAGE                                   KERNEL-VERSION                          CONTAINER-RUNTIME
rhel10-k3s-stackable02   Ready    control-plane   21d   v1.36.3+k3s1   10.Y.Y.Y      <none>        Red Hat Enterprise Linux 10.2 (Coughlan)   6.12.0-211.18.1.el10_2.x86_64 (amd64)   containerd://2.3.2-k3s2

kubectl succeeds and returns real node data — it also bypasses the proxy correctly for this address.

Failing case: stackablectl (same host, same kubeconfig, same env)

DEBUG  requesting
    at rust/.../kube-client-4.2.0/src/client/builder.rs:270
    in HTTP with http.method: GET, http.url: https://127.0.0.1:6443/api/v1/nodes?, otel.name: "list", otel.kind: "client"

DEBUG  connecting to 10.X.X.X:3128
    at .../hyper-util-0.1.20/src/client/legacy/connect/http.rs:767
    in HTTP with http.method: GET, http.url: https://127.0.0.1:6443/api/v1/nodes?, ...

DEBUG  connected to 10.X.X.X:3128
    at .../hyper-util-0.1.20/src/client/legacy/connect/http.rs:770

ERROR  failed with error client error (Connect)
    at .../kube-client-4.2.0/src/client/builder.rs:297

   INFO  failed to determine Kubernetes environment, using defaults: Whatever {
    source: Some(
        Service(
            hyper_util::client::legacy::Error(
                Connect,
                TunnelUnsuccessful,
            ),
        ),
    ),
    message: "failed to list Kubernetes nodes",
    ...
}

Final error:

An unrecoverable error occurred: failed to execute demo (sub)command

Caused by these errors (recent errors listed first):
 1: failed to create Kubernetes client
 2: failed to run GVK discovery
 3: ServiceError: client error (Connect)
 4: client error (Connect)
 5: tunnel error: unsuccessful

Interesting contrast within the same run

Later in the same run, a request to api.github.com (via the reqwest 0.13.4 client used by stackablectl for Helm repos/GitHub) is logged with an explicit interception decision:

DEBUG  starting new connection 'Some("api.github.com")'
    at .../reqwest-0.13.4/src/connect.rs:929
DEBUG  proxy(http://proxyserver:3128/) intercepts 'Some("api.github.com")'
    at .../reqwest-0.13.4/src/connect.rs:784

No equivalent "intercepts"/bypass decision is ever logged for the 127.0.0.1:6443 request. This suggests the kube-client proxy code path (re-enabled via #453 / operator-rs#1269) does not evaluate NO_PROXY the same way the reqwest-based client used for the rest of stackablectl does.

Expected behavior

Requests to 127.0.0.1:6443 should be sent directly, bypassing the proxy, since 127.0.0.1 is explicitly listed in NO_PROXY — matching the behavior of curl and kubectl shown above.

Actual behavior

The request is tunneled through the proxy and fails with TunnelUnsuccessful.

Suspected cause

kube's proxy support was re-enabled as a Cargo feature via stackable-operator (#453, pulling in operator-rs#1269), but the NO_PROXY exclusion logic does not appear to be wired into the kube::Client/hyper-util connector used for the Kubernetes API specifically — even though the same exclusion works correctly for the general-purpose reqwest client used elsewhere in stackablectl.

This is a known class of bug across Kubernetes client libraries in other languages, e.g.:

• kubernetes-client/python#2520 — no_proxy loaded from env and then immediately reset to None.
• fabric8io/kubernetes-client#2506 — no_proxy matched only via host.endsWith(noProxy), no wildcard/IP support.

It's plausible a similar gap exists in the kube/operator-rs proxy wiring.

Workaround

Unsetting HTTP_PROXY/HTTPS_PROXY entirely is not viable for us, since other calls in the same stackablectl run (Helm repos, GitHub, OCI registries) genuinely require the proxy.

Ask

Could someone confirm whether NO_PROXY is expected to be honored on the kube::Client proxy path introduced/re-enabled in #453, and if not, whether a fix is planned? Happy to test a candidate fix or provide any further info needed.

[Techassi]

Thank you!

We will first reproduce this locally and then work on a fix.