Skip to content

Add connectionTracking support to nodegroup config - #8872

Open
gustavodiaz7722 wants to merge 2 commits into
eksctl-io:mainfrom
gustavodiaz7722:feat/nodegroup-connection-tracking
Open

gustavodiaz7722 wants to merge 2 commits into
eksctl-io:mainfrom
gustavodiaz7722:feat/nodegroup-connection-tracking

Conversation

@gustavodiaz7722

@gustavodiaz7722 gustavodiaz7722 commented Sep 17, 2026

Copy link
Copy Markdown
Contributor

Closes #8871

Description

Adds an optional connectionTracking block to the nodegroup schema, so the EC2 connection tracking timeouts can be set without abandoning the native nodegroup config for a hand-written launch template.

managedNodeGroups:
  - name: ng-1
    instanceType: c8in.8xlarge
    connectionTracking:
      tcpEstablishedTimeout: 432000

nodeGroups:
  - name: ng-2
    instanceType: m8i.4xlarge
    connectionTracking:
      tcpEstablishedTimeout: 3600
      udpStreamTimeout: 180
      udpTimeout: 60

The field sits on NodeGroupBase, so one addition covers nodeGroups and managedNodeGroups. Timeouts left unset are omitted from the template, so EC2 keeps its own default for the instance type; at least one must be set.

Where the values land. buildNetworkInterfaces now populates ConnectionTrackingSpecification on every network interface it builds — the primary interface, and for an EFA-enabled nodegroup the EFA interfaces too. That matches what Karpenter does for EC2NodeClass. The VPC CNI copies the primary interface's settings onto the interfaces it creates for pods (added in v1.21.2 / v1.22.1), so pod traffic inherits them.

The one non-obvious change. A non-EFA managed nodegroup previously emitted no NetworkInterfaces at all — makeLaunchTemplateData only called buildNetworkInterfaces when EFA was enabled, and otherwise set instance-level SecurityGroupIds. Populating the interface alone would therefore have been a silent no-op for the common managed nodegroup. Such a nodegroup now emits a single network interface, carrying the security groups in Groups, when and only when connectionTracking is set. AssociatePublicIpAddress is left unset, so the subnet's auto-assign public IPv4 setting still applies — EC2 documents that auto-assign is only lost with more than one interface, or with an existing interface at device index 0. I verified this on a real instance rather than relying on the docs; see below. This is also the same shape self-managed nodegroups have always used, since nodegroup.go calls buildNetworkInterfaces unconditionally.

A nodegroup that does not set connectionTracking generates exactly the template it did before. All existing launch template golden files are unchanged.

Validation. The ranges are checked at config load time, against the values in the EC2 API model: tcpEstablishedTimeout 60–432000, udpStreamTimeout 60–180, udpTimeout 30–60. This is worth doing client-side because EC2 does not reject out-of-range values at CreateLaunchTemplate time — I confirmed it stores them and defers enforcement, so without this check a user would get a launch template that only fails later, when nodes launch.

connectionTracking is also added to the set of fields rejected alongside a user-supplied launchTemplate.id, since eksctl does not build the launch template in that case and the value would otherwise be silently dropped.

Manual testing

Through the CLI. Built the binary and drove the real thing. Valid config, resolved through create cluster --dry-run (round-trips through defaulting and validation for both nodegroup types):

$ eksctl create cluster -f valid.yaml --dry-run
...
managedNodeGroups:
- amiFamily: AmazonLinux2023
  connectionTracking:
    tcpEstablishedTimeout: 432000
...
nodeGroups:
- amiFamily: AmazonLinux2023
  connectionTracking:
    tcpEstablishedTimeout: 3600
    udpStreamTimeout: 180
    udpTimeout: 60

Rejections:

$ eksctl create cluster -f bad-range.yaml --dry-run
Error: value for managedNodeGroups[0].connectionTracking.tcpEstablishedTimeout must be within range 60-432000

$ eksctl create cluster -f bad-udp.yaml --dry-run
Error: value for nodeGroups[0].connectionTracking.udpTimeout must be within range 30-60

$ eksctl create cluster -f bad-empty.yaml --dry-run
Error: at least one of managedNodeGroups[0].connectionTracking.tcpEstablishedTimeout, managedNodeGroups[0].connectionTracking.udpStreamTimeout or managedNodeGroups[0].connectionTracking.udpTimeout must be set

$ eksctl create cluster -f bad-lt.yaml --dry-run
Error: cannot set instanceType, ami, ..., placement, connectionTracking in managedNodeGroup when a launch template is supplied

The generated CloudFormation. Rendered the templates both nodegroup types produce and checked them against AWS. The managed nodegroup's launch template comes out as:

"NetworkInterfaces": [
    {
        "ConnectionTrackingSpecification": {
            "TcpEstablishedTimeout": 432000,
            "UdpStreamTimeout": 180,
            "UdpTimeout": 60
        },
        "DeviceIndex": 0,
        "Groups": ["sg-..."],
        "NetworkCardIndex": 0
    }
]

with no instance-level SecurityGroupIds, as EKS requires when a network interface is specified. aws cloudformation validate-template accepts both the managed and the self-managed stack.

End to end on a real instance, including the public-subnet case. I built a launch template with exactly that shape and launched an instance from it into a subnet with MapPublicIpOnLaunch=true. The instance got its public IPv4 address, so the subnet's auto-assign setting is still honoured with the interface block present:

$ aws ec2 describe-instances --instance-ids i-... \
    --query 'Reservations[0].Instances[0].{State:State.Name,PublicIp:PublicIpAddress,PrivateIp:PrivateIpAddress}'
{
    "State": "running",
    "PublicIp": "54.218.87.182",
    "PrivateIp": "172.31.40.51"
}

And the timeouts reached the live ENI, not just the launch template:

$ aws ec2 describe-network-interfaces --network-interface-ids eni-... \
    --query 'NetworkInterfaces[0].{ConnectionTracking:ConnectionTrackingConfiguration,PublicIp:Association.PublicIp}'
{
    "ConnectionTracking": {
        "TcpEstablishedTimeout": 432000,
        "UdpStreamTimeout": 180,
        "UdpTimeout": 60
    },
    "PublicIp": "54.218.87.182"
}

So the chain holds end to end: config → CloudFormation → launch template → attached ENI. Test resources were torn down afterwards.

One note from that exercise, in case anyone reproduces it by hand: with a NetworkInterfaces block in the launch template, the subnet has to be supplied on the interface — RunInstances with an instance-level --subnet-id is refused with InvalidParameterCombination. That doesn't affect this change, since eksctl never puts a subnet in the launch template and EKS/the ASG supplies them, and the existing EFA path already launches managed nodegroups with an interface block.

What I have not done is a full eksctl create cluster against a real cluster, so the managed-nodegroup node-join path is covered by the EC2-level evidence above rather than by a cluster run. I'd suggest an integration test for a public-subnet managed nodegroup so this doesn't rest on a one-off manual run — happy to add it here if you'd like it in scope, or leave it as a follow-up.

Checklist

  • Added tests that cover your change (if possible)
  • Added/modified documentation as required (such as the README.md, or the userdocs directory)
  • Manually tested
  • Made sure the title of the PR is a good description that can go into the release notes
  • (Core team) Added labels for change area (e.g. area/nodegroup) and kind (e.g. kind/improvement)

Security groups track connections to and from a node, and drop traffic on
connections that have been idle longer than EC2's timeout. Nitro v6 instance
types default the idle established-TCP timeout to 350 seconds, where earlier
generations used 432,000, so workloads holding long-lived idle TCP connections
can start losing them after moving to a current-generation instance type.

EC2 exposes these timeouts as ConnectionTrackingSpecification on a launch
template's network interfaces, but eksctl had no way to set them. The only
option was to supply a custom launch template, which disables most of the
native nodegroup schema. Karpenter already exposes the same control on
EC2NodeClass, so this closes the gap for nodegroup users.

Adds an optional connectionTracking block to NodeGroupBase, so it works for
both nodeGroups and managedNodeGroups:

    managedNodeGroups:
      - name: ng-1
        instanceType: c8in.8xlarge
        connectionTracking:
          tcpEstablishedTimeout: 432000
          udpStreamTimeout: 180
          udpTimeout: 60

The timeouts are validated against the ranges EC2 accepts and applied to
every network interface eksctl puts in the launch template, which is the
primary interface and, for EFA-enabled nodegroups, the EFA interfaces. The
VPC CNI copies the primary interface's settings onto the interfaces it
creates for pods, so pod traffic inherits them.

A non-EFA managed nodegroup previously emitted no network interfaces at all,
setting instance-level SecurityGroupIds instead, so populating the interface
alone would have been a no-op there. Such a nodegroup now emits a single
network interface, carrying the security groups in Groups, when and only when
connectionTracking is set. AssociatePublicIpAddress stays unset, so the
subnet's auto-assign public IPv4 setting still applies - the same shape
self-managed nodegroups have always used.

Timeouts left unset are omitted, keeping EC2's default. Nodegroups that do
not set connectionTracking generate exactly the template they did before.

Signed-off-by: Gustavo Diaz <gustidia@amazon.com>
revive's if-return rule flags the redundant if err != nil { return err }
before a bare return nil at the end of validateNodeGroupBase.

Signed-off-by: Gustavo Diaz <gustidia@amazon.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/nodegroup kind/feature New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Feature] Support EC2 connection tracking timeouts in nodegroup config

1 participant