Add connectionTracking support to nodegroup config - #8872
Open
gustavodiaz7722 wants to merge 2 commits into
Open
gustavodiaz7722 wants to merge 2 commits into
gustavodiaz7722 wants to merge 2 commits into
Conversation
Security groups track connections to and from a node, and drop traffic on
connections that have been idle longer than EC2's timeout. Nitro v6 instance
types default the idle established-TCP timeout to 350 seconds, where earlier
generations used 432,000, so workloads holding long-lived idle TCP connections
can start losing them after moving to a current-generation instance type.
EC2 exposes these timeouts as ConnectionTrackingSpecification on a launch
template's network interfaces, but eksctl had no way to set them. The only
option was to supply a custom launch template, which disables most of the
native nodegroup schema. Karpenter already exposes the same control on
EC2NodeClass, so this closes the gap for nodegroup users.
Adds an optional connectionTracking block to NodeGroupBase, so it works for
both nodeGroups and managedNodeGroups:
managedNodeGroups:
- name: ng-1
instanceType: c8in.8xlarge
connectionTracking:
tcpEstablishedTimeout: 432000
udpStreamTimeout: 180
udpTimeout: 60
The timeouts are validated against the ranges EC2 accepts and applied to
every network interface eksctl puts in the launch template, which is the
primary interface and, for EFA-enabled nodegroups, the EFA interfaces. The
VPC CNI copies the primary interface's settings onto the interfaces it
creates for pods, so pod traffic inherits them.
A non-EFA managed nodegroup previously emitted no network interfaces at all,
setting instance-level SecurityGroupIds instead, so populating the interface
alone would have been a no-op there. Such a nodegroup now emits a single
network interface, carrying the security groups in Groups, when and only when
connectionTracking is set. AssociatePublicIpAddress stays unset, so the
subnet's auto-assign public IPv4 setting still applies - the same shape
self-managed nodegroups have always used.
Timeouts left unset are omitted, keeping EC2's default. Nodegroups that do
not set connectionTracking generate exactly the template they did before.
Signed-off-by: Gustavo Diaz <gustidia@amazon.com>
revive's if-return rule flags the redundant if err != nil { return err }
before a bare return nil at the end of validateNodeGroupBase.
Signed-off-by: Gustavo Diaz <gustidia@amazon.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #8871
Description
Adds an optional
connectionTrackingblock to the nodegroup schema, so the EC2 connection tracking timeouts can be set without abandoning the native nodegroup config for a hand-written launch template.The field sits on
NodeGroupBase, so one addition coversnodeGroupsandmanagedNodeGroups. Timeouts left unset are omitted from the template, so EC2 keeps its own default for the instance type; at least one must be set.Where the values land.
buildNetworkInterfacesnow populatesConnectionTrackingSpecificationon every network interface it builds — the primary interface, and for an EFA-enabled nodegroup the EFA interfaces too. That matches what Karpenter does forEC2NodeClass. The VPC CNI copies the primary interface's settings onto the interfaces it creates for pods (added inv1.21.2/v1.22.1), so pod traffic inherits them.The one non-obvious change. A non-EFA managed nodegroup previously emitted no
NetworkInterfacesat all —makeLaunchTemplateDataonly calledbuildNetworkInterfaceswhen EFA was enabled, and otherwise set instance-levelSecurityGroupIds. Populating the interface alone would therefore have been a silent no-op for the common managed nodegroup. Such a nodegroup now emits a single network interface, carrying the security groups inGroups, when and only whenconnectionTrackingis set.AssociatePublicIpAddressis left unset, so the subnet's auto-assign public IPv4 setting still applies — EC2 documents that auto-assign is only lost with more than one interface, or with an existing interface at device index 0. I verified this on a real instance rather than relying on the docs; see below. This is also the same shape self-managed nodegroups have always used, sincenodegroup.gocallsbuildNetworkInterfacesunconditionally.A nodegroup that does not set
connectionTrackinggenerates exactly the template it did before. All existing launch template golden files are unchanged.Validation. The ranges are checked at config load time, against the values in the EC2 API model:
tcpEstablishedTimeout60–432000,udpStreamTimeout60–180,udpTimeout30–60. This is worth doing client-side because EC2 does not reject out-of-range values atCreateLaunchTemplatetime — I confirmed it stores them and defers enforcement, so without this check a user would get a launch template that only fails later, when nodes launch.connectionTrackingis also added to the set of fields rejected alongside a user-suppliedlaunchTemplate.id, sinceeksctldoes not build the launch template in that case and the value would otherwise be silently dropped.Manual testing
Through the CLI. Built the binary and drove the real thing. Valid config, resolved through
create cluster --dry-run(round-trips through defaulting and validation for both nodegroup types):Rejections:
The generated CloudFormation. Rendered the templates both nodegroup types produce and checked them against AWS. The managed nodegroup's launch template comes out as:
with no instance-level
SecurityGroupIds, as EKS requires when a network interface is specified.aws cloudformation validate-templateaccepts both the managed and the self-managed stack.End to end on a real instance, including the public-subnet case. I built a launch template with exactly that shape and launched an instance from it into a subnet with
MapPublicIpOnLaunch=true. The instance got its public IPv4 address, so the subnet's auto-assign setting is still honoured with the interface block present:And the timeouts reached the live ENI, not just the launch template:
So the chain holds end to end: config → CloudFormation → launch template → attached ENI. Test resources were torn down afterwards.
One note from that exercise, in case anyone reproduces it by hand: with a
NetworkInterfacesblock in the launch template, the subnet has to be supplied on the interface —RunInstanceswith an instance-level--subnet-idis refused withInvalidParameterCombination. That doesn't affect this change, sinceeksctlnever puts a subnet in the launch template and EKS/the ASG supplies them, and the existing EFA path already launches managed nodegroups with an interface block.What I have not done is a full
eksctl create clusteragainst a real cluster, so the managed-nodegroup node-join path is covered by the EC2-level evidence above rather than by a cluster run. I'd suggest an integration test for a public-subnet managed nodegroup so this doesn't rest on a one-off manual run — happy to add it here if you'd like it in scope, or leave it as a follow-up.Checklist
README.md, or theuserdocsdirectory)area/nodegroup) and kind (e.g.kind/improvement)