Skip to content

Setting up EKS hybrid node

This is a step-by-step guide for turning arbitrary VMs into EKS hybrid nodes that can be seamlessly integrated into a customer's EKS cluster.

The process is not yet fully automated. The steps below are manual. Some run once per cluster, others once per node.

Warning

Some AWS resources created here cost money just by existing. That holds whether any nodes are in use or not. It is thus important to delete resources that are no longer required. How to do this is not explicitly documented here, refer to the official AWS docs and use your best judgement.

The aws_eks_hybrid_node Ansible role

Turning an existing VM into an EKS hybrid node is fully handled via our aws_eks_hybrid_node role. In order to use it, manually create a temporary inventory containing the hybrid node and a playbook that executes the role.

Adding the node to an existing EKS cluster takes two steps. Set up a VPN connection to the cluster's VPC. Then run nodeadm ⧉ once on the hybrid node.

The owner of the cluster has to perform various other setup steps before this such as setting up the VPN connection and configuring routing tables. This is not yet fully integrated into our service offering and thus has to be performed manually. Since this role depends on values generated during this initial setup step it too has to currently be invoked manually by some handcrafted playbook.

This will hopefully be remedied in the future such that we can offer seamless integration of our VMs into customers' EKS clusters. Until then the setup steps are described here at a high level. That keeps them from being lost, and shows what this role is good for.

Initial AWS-side setup

This assumes an EKS cluster already exists and that we have sufficient permissions to access it (TODO(Timo Nicolai): Note down exactly which permissions are necessary for all setup steps).

Here we will first describe how to gather some relevant information about the cluster. Then we cover the setup steps required once per cluster. That is the same as once per VPC, unless you run multiple clusters in one VPC. Finally we discuss per-hybrid-node setup.

1. Gather information

First we need to obtain a number of facts about the EKS cluster and associated AWS resources:

Cluster name and region

This should already be known to us, export this as $CLUSTER_NAME and $CLUSTER_REGION.

Cluster endpoint

export CLUSTER_ENDPOINT=$(aws eks describe-cluster
  --name $CLUSTER_NAME
  --region $CLUSTER_REGION
  --query 'cluster.endpoint'
  --output text)

Cluster VPC ID

export CLUSTER_VPC_ID=$(aws eks describe-cluster
  --name $CLUSTER_NAME
  --region $CLUSTER_REGION
  --query "cluster.resourcesVpcConfig.vpcId"
  --output text)

Cluster VPC CIDR

export CLUSTER_VPC_CIDR=$(aws ec2 describe-vpcs
  --vpc-ids $CLUSTER_VPC_ID
  --query 'Vpcs[*].CidrBlockAssociationSet[*].CidrBlock'
  --output text)

Cluster k8s service CIDR

export CLUSTER_K8S_SERVICE_CIDR=$(aws eks describe-cluster
  --name $CLUSTER_NAME
  --region $CLUSTER_REGION
  --query 'cluster.kubernetesNetworkConfig.serviceIpv4Cidr'
  --output text)

Cluster routing table IDs

export CLUSTER_ROUTING_TABLE_IDS="$(aws ec2 describe-route-tables
  --filters "Name=vpc-id,Values=$CLUSTER_VPC_ID"
  --query 'RouteTables[*].RouteTableId'
  --output text)"

Node security group ID

export NODE_SECURITY_GROUP_ID=$(aws eks describe-cluster
  --name $CLUSTER_NAME
  --region $CLUSTER_REGION
  --query 'cluster.resourcesVpcConfig.clusterSecurityGroupId'
  --output text)

2. Set up LyceumEKSHybridNodeRole (once/once per cluster)

LyceumEKSHybridNodeRole is an AWS role with permission to create hybrid node activations. nodeadm uses those activations to register each hybrid node securely. If you use hybrid nodes across several EKS clusters, then you can reuse it. Only the final step repeats, which is creating an access entry per cluster.

Create the role

First create the following policy document:

cat > lyceum-eks-hybrid-node-role-policy.json << 'EOF'
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Service": "ssm.amazonaws.com"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}
EOF

Then create the role:

aws iam create-role \
  --role-name LyceumEKSHybridNodeRole \
  --assume-role-policy-document file://lyceum-eks-hybrid-node-policy.json \
  --region $CLUSTER_REGION

Attach role policies

Attach the following role policies which together encompass the minimum set of permissions we require:

aws iam attach-role-policy \
  --role-name LyceumEKSHybridNodeRole \
  --policy-arn arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore

aws iam attach-role-policy \
  --role-name LyceumEKSHybridNodeRole \
  --policy-arn arn:aws:iam::aws:policy/AmazonEKSWorkerNodePolicy

aws iam attach-role-policy \
  --role-name LyceumEKSHybridNodeRole \
  --policy-arn arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryReadOnly

Create an access entry

Finally we need to allow an LyceumEKSHybridNodeRole to access the EKS cluster, indicating that it is intended to be used with hybrid nodes via --type HYBRID_LINUX:

AWS_ACCOUNT_ID=$(aws sts get-caller-identity --query Account --output text)

aws eks create-access-entry \
  --cluster-name $CLUSTER_NAME \
  --region $CLUSTER_REGION \
  --principal-arn arn:aws:iam::${AWS_ACCOUNT_ID}:role/LyceumEKSHybridNodesRole \
  --type HYBRID_LINUX

3. Create a VPN gateway (once per cluster/VPC)

We need to create a VPN gateway which is the AWS side of the VPN connection we want to establish to the hybrid node. One VPN gateway can support VPN connections for all our hybrid nodes.

Create the gateway

Run the following and export VpnGatewayId from the output to VPN_GATEWAY_ID.

aws ec2 create-vpn-gateway \
  --type ipsec.1 \
  --region $CLUSTER_REGION

Attach the gateway to the cluster's VPC

Run the following, this may take a bit to complete after the command finishes.

aws ec2 attach-vpn-gateway \
 --vpn-gateway-id $VPN_GATEWAY_ID \
 --vpc-id $CLUSTER_VPC_ID \
 --region $CLUSTER_REGION

4. Create a VPN connection

Create the customer gateway

A customer gateway is an AWS resource which provides information to AWS about the "customer gateway device" we run on-prem. In our case on-prem means a single VM somewhere in the cloud, which becomes our hybrid node. We own no Cisco gateway device to insert between AWS and that VM.

The customer gateway will thus reference a virtual gateway (one per hybrid node) created with stronSwan ⧉. This is a large part of what this role does.

Run the following and export CustomerGatewayId from the output to CUSTOMER_GATEWAY_ID. Here $HYBRID_NODE_OUTSIDE_IP is the public IP of our hybrid node VM. Note that we use static routing but aws requires a value for --bgn-asn. We pass 64512 (the smallest possible value) + 1.

aws ec2 create-customer-gateway \
  --type ipsec.1 \
  --public-ip $HYBRID_NODE_OUTSIDE_IP \
  --bgp-asn 64513  \
  --region $CLUSTER_REGION
Create the VPN connection

We can now create the VPN connection, specifying our VPN gateway and customer gateway as endpoints. Run the following and export VpnConnectionId from the output to VPN_CONNECTION_ID.

aws ec2 create-vpn-connection \
  --type ipsec.1 \
  --customer-gateway-id $CUSTOMER_GATEWAY_ID \
  --vpn-gateway-id $VPN_GATEWAY_ID \
  --options '{"StaticRoutesOnly":true}' \
  --region $CLUSTER_REGION
Create VPN connection routes

We have to tell AWS to route traffic to our hybrid node via the VPN connection. We can somewhat freely choose the IP address ranges to give to both our hybrid nodes and pods running on them. This is important because we do not use BGP and instead only route statically. At a later point we might want to switch to BGP to enable e.g. routing from pods on one hybrid node to pods on another. We use the following scheme:

  • All hybrid nodes receive an IP address 10.255.0.X/32. This is part of the private range 10.0.0.0/8 but unlikely to already be in use because 10.255.* is somewhat unusual. This implies that no two hybrid nodes are ever in the same network. Export this to $HYBRID_NODE_CIDR.

  • We give pods across all hybrid nodes IP address in 10.254.0.0/16. This is large enough to allow for up to 256 hybrid nodes with up to 256 pods each which should be plenty. AWS must however be aware which pod IPs need to be routed through which VPN connection. To this end every new node should be assigned pod IPs in 10.254.{n+1}.0./24 where the last created node used IPs in 10.254.{n}.0./24. Export this to $HYBRID_NODE_POD_CIDR and export 10.254.0.0/16 to $HYBRID_POD_CIDR. Cilium assigns pod IPs. Double check that it looks like this after a new node has been created. A more flexible solution that does not require manual routing might be possible via EKS Hybrid Nodes gateway ⧉.

Run the following:

aws ec2 create-vpn-connection-route \
  --vpn-connection-id $VPN_CONNECTION_ID \
  --destination-cidr-block $HYBRID_NODE_CIDR \
  --region $CLUSTER_REGION

aws ec2 create-vpn-connection-route \
  --vpn-connection-id $VPN_CONNECTION_ID \
  --destination-cidr-block $HYBRID_NODE_POD_CIDR \
  --region $CLUSTER_REGION

Export VPN connection configuration

Finally, run the following to dump settings we will need to supply to this role later to a file:

aws ec2 describe-vpn-connections \
  --vpn-connection-ids $VPN_CONNECTION_ID \
  --query 'VpnConnections[0].CustomerGatewayConfiguration' \
  --region $CLUSTER_REGION \
  --output text > vpn-config.xml

5. Update cluster routes and security rules

We now have to add make our hybrid node networks known to the EKS cluster and add routes for its EC2 instances. Additionally we have allow traffic from our hybrid node to these EC2 instances.

Update cluster config

aws eks update-cluster-config \
  --name $CLUSTER_NAME \
  --remote-network-config "remoteNodeNetworks=[{cidrs=[\"$HYBRID_NODE_CIDR\"]}],remotePodNetworks=[{cidrs=[\"$HYBRID_POD_CIDR\"]}]" \
  --region $CLUSTER_REGION

Warning

If one EKS cluster has multiple hybrid nodes, then list all of them in remoteNodeNetworks. Do this every time you add a new one.

Create static routes

for ROUTING_TABLE_ID in $CLUSTER_ROUTING_TABLE_IDS; do
  echo "Creating routes in routing table: $ROUTING_TABLE_ID"

  aws ec2 create-route \
    --route-table-id "$ROUTING_TABLE_ID" \
    --destination-cidr-block "$HYBRID_NODE_CIDR" \
    --gateway-id "$VPN_GATEWAY_ID" \
    --region "$CLUSTER_REGION"

  aws ec2 create-route \
    --route-table-id "$ROUTING_TABLE_ID" \
    --destination-cidr-block "$HYBRID_POD_CIDR" \
    --gateway-id "$VPN_GATEWAY_ID" \
    --region "$CLUSTER_REGION"
done

Configure security group

Note: -1 means "all protocols" i.e. ICMP, TCP, UCP, ...

aws ec2 authorize-security-group-ingress \
  --group-id $CLUSTER_SECURITY_GROUP \
  --protocol -1 \
  --cidr $HYBRID_NODE_CIDR \
  --region $CLUSTER_REGION

aws ec2 authorize-security-group-ingress \
  --group-id $NODE_SECURITY_GROUP \
  --protocol -1 \
  --cidr $HYBRID_NODE_CIDR \
  --region $CLUSTER_REGION

6. Create activation and run Ansible

We are now ready to actually configure the hybrid node itself. First we must create an activation. We do this by asking AWS for a one-time activation code:

aws ssm create-activation \
  --iam-role LyceumEKSHybridNodesRole \
  --registration-limit 1 \
  --region $CLUSTER_REGION

This will return an activation ID and an activation code, export these as $ACTIVATION_ID an $ACTIVATION_CODE.

We then supply this role with the following information. $FOO is pseudo-syntax for the environment variable set in one of the steps above. The tunnel-specific values are taken from the exported VPN connection configuration in vpn-config.xml. Note that AWS mandates that there are two tunnels for redundancy.

aws_eks_hybrid_node_cluster_name: $CLUSTER_NAME
aws_eks_hybrid_node_cluster_region: $CLUSTER_REGION
aws_eks_hybrid_node_cluster_endpoint: $CLUSTER_ENDPOINT
aws_eks_hybrid_node_k8s_version: "1.33" # Example, adjust to your cluster
aws_eks_hybrid_node_nodeadm_version: "1.0.11" # Example, adjust to your cluster
aws_eks_hybrid_node_aws_vpc_cidr: $CLUSTER_VPC_CIDR
aws_eks_hybrid_node_k8s_service_cidr: $CLUSTER_K8S_SERVICE_CIDR
aws_eks_hybrid_node_hybrid_node_outside_ip: $HYBRID_NODE_OUTSIDE_IP
aws_eks_hybrid_node_hybrid_node_inside_ip: $HYBRID_NODE_CIDR
aws_eks_hybrid_node_hybrid_node_pod_cidr: $HYBRID_NODE_POD_CIDR
aws_eks_hybrid_node_activation_id: $ACTIVATION_ID
aws_eks_hybrid_node_activation_code: $ACTIVATION_CODE

# From vpn-config.xml:
aws_eks_hybrid_node_tunnel1_aws_outside_ip: ...
aws_eks_hybrid_node_tunnel1_aws_inside_ip: ...
aws_eks_hybrid_node_tunnel1_hybrid_node_inside_ip: ...
aws_eks_hybrid_node_tunnel1_psk: ...
aws_eks_hybrid_node_tunnel2_aws_outside_ip: ...
aws_eks_hybrid_node_tunnel2_aws_inside_ip: ...
aws_eks_hybrid_node_tunnel2_hybrid_node_inside_ip: ...
aws_eks_hybrid_node_tunnel2_psk: ...

After Ansible has completed successfully you should be able to see the hybrid node with kubectl get nodes, its name will have an mi- prefix. The node will still be NotReady at this stage if Cillium has not yet been installed and configured, see the next step.

7. Install and configure Cilium

We are finally done with creating AWS resources but a bit of work remains: EKS does not support just any CNI for use with hybrid nodes. Thus we have to install Cilium. We assume you do not already use Cilium. If you do, then adapt this. We configure it to run agent pods on hybrid nodes only. Some minor networking-related kubectl adjustments are then required in order for Cilium to not break existing cluster communication.

Update kubeconfig

We will install Cilium with helm which means that we first need to ensure we have our kubeconfig pointed to the EKS cluster:

aws eks update-kubeconfig \
  --name $CLUSTER_NAME \
  --region $CLUSTER_REGION

Install Cilium

Fist create the following configuration file.

Adapt $HYBRID_POD_MASK_SIZE to control how many pod IP addresses each hybrid node gets. That also caps how many hybrid nodes can be created in total. Set this to 24 to allow up to 256 pods per hybrid node, thus up to 256 hybrid nodes.

Furthermore, $HYBRID_NODE_DEFAULT_INTERFACE should be set to the hybrid node's primary interface, e.g. eth0 (check with ip route | grep default).

cat > cilium-values.yaml << 'EOF'
---
k8sServiceHost: "$CLUSTER_ENDPOINT"
k8sServicePort: "443"

# Controls where Cilium agent pods run (only match hybrid nodes)
affinity:
  nodeAffinity:
    requiredDuringSchedulingIgnoredDuringExecution:
      nodeSelectorTerms:
      - matchExpressions:
        - key: eks.amazonaws.com/compute-type
          operator: In
          values:
          - hybrid

# Don't manage pods that existed before Cilium was installed.
operator:
  unmanagedPodWatcher:
    restart: false

# Do not replace kube-proxy because EC2 nodes are still using it.
kubeProxyReplacement: "false"

ipam:
  mode: cluster-pool
  operator:
    clusterPoolIPv4PodCIDRList:
    - $HYBRID_POD_CIDR
    clusterPoolIPv4MaskSize: $HYBRID_POD_MASK_SIZE

# This prevents Cilium from trying to masquerade traffic through virtual tunnel
# interfaces. We don't want that because we insert our own SNAT iptables rules
# to do so.
egressMasqueradeInterfaces: $HYBRID_NODE_DEFAULT_INTERFACE
EOF

Then install cilium with:

helm install cilium cilium/cilium --namespace kube-system -f cilium-values.yaml

Disable SNATing

When AWS tries to SNAT traffic to hybrid nodes/pods strange things happen when e.g. the target is a ClusterIP. This can be prevented by disabling SNATing for our node and pod CIDRS:

kubectl -n kube-system \
  set env daemonset/aws-node \
  AWS_VPC_K8S_CNI_EXCLUDE_SNAT_CIDRS="$HYBRID_NODE_CIDR,$HYBRID_POD_CIDR"

Warning

If one EKS cluster has multiple hybrid nodes, then list all of them in AWS_VPC_K8S_CNI_EXCLUDE_SNAT_CIDRS. Do this every time you add a new one.

Adjust CoreDNS configuration

CoreDNS pods have given us trouble once Cilium is installed. Two extra tweaks reduce it. Keep at least two CoreDNS pods running on AWS. Make pods on EC2 instances prefer those, and pods on hybrid nodes prefer their own.

First, annotate hybrid nodes with the arbitrary lyceum-<n> zone label, one per hybrid node. Then patch kube-dns to prefer CoreDNS pods in the same zone:

kubectl label node <hybrid-node-1-name> topology.kubernetes.io/zone=lyceum-1
kubectl patch service kube-dns -n kube-system -p '{"spec":{"trafficDistribution":"PreferClose"}}'

Then increase the number of CoreDNS pods with (here representatively to 3, in reality chose something like number of hybrid nodes + 2):

kubectl scale deployment coredns \
  -n kube-system \
  --replicas=3

Warning

This does not guarantee a CoreDNS replica on each hybrid node. DNS on those nodes needs one. Simply increasing the number of replicas until one is scheduled works but a more robust solution needs to be found.