The problem with egress in AKS

Once you start tightening egress in an AKS cluster running Cilium, you quickly discover that the outside world is mostly reserved:world. Azure IMDS at 169.254.169.254/32, the internal DNS resolver at 168.63.129.16/32, your VNet peers – they all land in the same identity bucket, and your Cilium network policies end up with raw CIDRs scattered across every rule that needs to reach them.

A few policies in and you’re copy-pasting the same toCIDRSet blocks everywhere. Change a subnet, and you’re grepping YAML. It works, but it doesn’t review well and it drifts quietly.

CiliumCIDRGroup

CiliumCIDRGroup is Cilium’s answer to this. It’s a cluster-scoped CRD that wraps a set of CIDRs behind a name and a set of labels. Instead of embedding 169.254.169.254/32 in every Cilium network policy that touches IMDS, you define it once and reference the group.

Two things make it useful beyond basic DRY:

  1. Labels. You can tag groups with metadata (cloud: azure, service: imds) and select them with cidrGroupSelector in your policies – same pattern as endpointSelector, applied to external destinations.
  2. Separation of concerns. The team managing Azure infrastructure defines the CIDR groups. The team writing network policy references them by name or label. Neither needs to care about the other’s implementation details.

This is distinct from Cilium’s DNS-based L7 policy, which resolves hostnames at runtime. CIDR groups are for fixed infrastructure endpoints where the IP is the identity. IMDS will always be 169.254.169.254. Use DNS policy for things addressed by hostname.

Modelling Azure endpoints

Start with the two endpoints every AKS cluster talks to:

azure-external-endpoints-ccg.yaml
apiVersion: cilium.io/v2
kind: CiliumCIDRGroup
metadata:
  name: azure-imds
  labels:
    cloud: azure
    service: imds
spec:
  externalCIDRs:
    - "169.254.169.254/32"
---
apiVersion: cilium.io/v2
kind: CiliumCIDRGroup
metadata:
  name: azure-internal-dns
  labels:
    cloud: azure
    service: dns
spec:
  externalCIDRs:
    - "168.63.129.16/32"
kubectl apply -f azure-external-endpoints-ccg.yaml

Nothing changes yet. CIDR groups are inert until a policy references them.

Writing policies against CIDR groups

There are two ways to reference a CIDR group from a Cilium network policy: by name or by label.

By name (cidrGroupRef)

Explicit and easy to audit. You can read the policy and know exactly which CIDR group it targets.

ccnp-by-cidrgroupref.yaml
apiVersion: cilium.io/v2
kind: CiliumClusterwideNetworkPolicy
metadata:
  name: allow-azure-dns-by-name
spec:
  endpointSelector:
    matchLabels:
      io.kubernetes.pod.namespace: kube-system
      k8s-app: kube-dns
  egress:
    - toCIDRSet:
        - cidrGroupRef: azure-internal-dns
      toPorts:
        - ports:
            - port: "53"
              protocol: UDP
            - port: "53"
              protocol: TCP
---
apiVersion: cilium.io/v2
kind: CiliumClusterwideNetworkPolicy
metadata:
  name: deny-imds-by-name
spec:
  endpointSelector: {}
  egressDeny:
    - toCIDRSet:
        - cidrGroupRef: azure-imds

By label (cidrGroupSelector)

More flexible. The policy doesn’t need to know the group names – it selects by taxonomy. Useful when the same policy applies across clusters with differently-named groups, or when the infrastructure team adds new groups that should automatically be picked up.

ccnp-by-cidrgroupselector.yaml
apiVersion: cilium.io/v2
kind: CiliumClusterwideNetworkPolicy
metadata:
  name: allow-azure-dns-by-label
spec:
  endpointSelector:
    matchLabels:
      app.kubernetes.io/name: workload-needing-azure-egress
  egress:
    - toCIDRSet:
        - cidrGroupSelector:
            matchLabels:
              cloud: azure
              service: dns
      toPorts:
        - ports:
            - port: "53"
              protocol: UDP
            - port: "53"
              protocol: TCP
---
apiVersion: cilium.io/v2
kind: CiliumClusterwideNetworkPolicy
metadata:
  name: deny-imds-by-label
spec:
  endpointSelector: {}
  egressDeny:
    - toCIDRSet:
        - cidrGroupSelector:
            matchLabels:
              cloud: azure
              service: imds

Apply whichever variant fits:

kubectl apply -f ccnp-by-cidrgroupref.yaml
# or
kubectl apply -f ccnp-by-cidrgroupselector.yaml

Generating CIDR groups from Azure VNets

Two static groups cover IMDS and DNS. But once you need to model VNet peers, private endpoints, or shared services subnets, manual YAML stops scaling.

The script below is a starting point. It queries the Azure Resource Manager API for all VNets and subnets in a subscription and generates a CiliumCIDRGroup for each, with metadata labels for subscription, resource group, VNet, and subnet. Adapt it to your own labelling conventions and filtering needs. It runs as a standalone script via uv – no virtualenv setup needed.

Run it:

uv run azure_to_cilium_cidrgroup.py --subscription "<SUBSCRIPTION_ID>" > generated-ccg.yaml
kubectl apply -f generated-ccg.yaml

To scope it to a single resource group:

uv run azure_to_cilium_cidrgroup.py \
  --subscription "<SUBSCRIPTION_ID>" \
  --resource-group "<RESOURCE_GROUP>" > generated-ccg.yaml

Wire this into CI and your CIDR groups stay in sync with your Azure network topology without anyone touching YAML by hand.

Takeaways

  • CiliumCIDRGroup turns raw CIDRs into named, labeled policy targets. External endpoints get the same treatment as in-cluster identities.
  • cidrGroupRef for explicit references, cidrGroupSelector for label-based selection. Pick based on how much indirection your team is comfortable with.