The problem with egress in AKS
Once you start tightening egress in an AKS cluster running Cilium, you quickly discover that the outside world is mostly reserved:world. Azure IMDS at 169.254.169.254/32, the internal DNS resolver at 168.63.129.16/32, your VNet peers – they all land in the same identity bucket, and your Cilium network policies end up with raw CIDRs scattered across every rule that needs to reach them.
A few policies in and you’re copy-pasting the same toCIDRSet blocks everywhere. Change a subnet, and you’re grepping YAML. It works, but it doesn’t review well and it drifts quietly.
CiliumCIDRGroup
CiliumCIDRGroup is Cilium’s answer to this. It’s a cluster-scoped CRD that wraps a set of CIDRs behind a name and a set of labels. Instead of embedding 169.254.169.254/32 in every Cilium network policy that touches IMDS, you define it once and reference the group.
Two things make it useful beyond basic DRY:
- Labels. You can tag groups with metadata (
cloud: azure,service: imds) and select them withcidrGroupSelectorin your policies – same pattern asendpointSelector, applied to external destinations. - Separation of concerns. The team managing Azure infrastructure defines the CIDR groups. The team writing network policy references them by name or label. Neither needs to care about the other’s implementation details.
This is distinct from Cilium’s DNS-based L7 policy, which resolves hostnames at runtime. CIDR groups are for fixed infrastructure endpoints where the IP is the identity. IMDS will always be 169.254.169.254. Use DNS policy for things addressed by hostname.
Modelling Azure endpoints
Start with the two endpoints every AKS cluster talks to:
apiVersion: cilium.io/v2
kind: CiliumCIDRGroup
metadata:
name: azure-imds
labels:
cloud: azure
service: imds
spec:
externalCIDRs:
- "169.254.169.254/32"
---
apiVersion: cilium.io/v2
kind: CiliumCIDRGroup
metadata:
name: azure-internal-dns
labels:
cloud: azure
service: dns
spec:
externalCIDRs:
- "168.63.129.16/32"kubectl apply -f azure-external-endpoints-ccg.yamlNothing changes yet. CIDR groups are inert until a policy references them.
Writing policies against CIDR groups
There are two ways to reference a CIDR group from a Cilium network policy: by name or by label.
By name (cidrGroupRef)
Explicit and easy to audit. You can read the policy and know exactly which CIDR group it targets.
apiVersion: cilium.io/v2
kind: CiliumClusterwideNetworkPolicy
metadata:
name: allow-azure-dns-by-name
spec:
endpointSelector:
matchLabels:
io.kubernetes.pod.namespace: kube-system
k8s-app: kube-dns
egress:
- toCIDRSet:
- cidrGroupRef: azure-internal-dns
toPorts:
- ports:
- port: "53"
protocol: UDP
- port: "53"
protocol: TCP
---
apiVersion: cilium.io/v2
kind: CiliumClusterwideNetworkPolicy
metadata:
name: deny-imds-by-name
spec:
endpointSelector: {}
egressDeny:
- toCIDRSet:
- cidrGroupRef: azure-imdsBy label (cidrGroupSelector)
More flexible. The policy doesn’t need to know the group names – it selects by taxonomy. Useful when the same policy applies across clusters with differently-named groups, or when the infrastructure team adds new groups that should automatically be picked up.
apiVersion: cilium.io/v2
kind: CiliumClusterwideNetworkPolicy
metadata:
name: allow-azure-dns-by-label
spec:
endpointSelector:
matchLabels:
app.kubernetes.io/name: workload-needing-azure-egress
egress:
- toCIDRSet:
- cidrGroupSelector:
matchLabels:
cloud: azure
service: dns
toPorts:
- ports:
- port: "53"
protocol: UDP
- port: "53"
protocol: TCP
---
apiVersion: cilium.io/v2
kind: CiliumClusterwideNetworkPolicy
metadata:
name: deny-imds-by-label
spec:
endpointSelector: {}
egressDeny:
- toCIDRSet:
- cidrGroupSelector:
matchLabels:
cloud: azure
service: imdsApply whichever variant fits:
kubectl apply -f ccnp-by-cidrgroupref.yaml
# or
kubectl apply -f ccnp-by-cidrgroupselector.yamlGenerating CIDR groups from Azure VNets
Two static groups cover IMDS and DNS. But once you need to model VNet peers, private endpoints, or shared services subnets, manual YAML stops scaling.
The script below is a starting point. It queries the Azure Resource Manager API for all VNets and subnets in a subscription and generates a CiliumCIDRGroup for each, with metadata labels for subscription, resource group, VNet, and subnet. Adapt it to your own labelling conventions and filtering needs. It runs as a standalone script via uv – no virtualenv setup needed.
#!/usr/bin/env -S uv run --script
# /// script
# requires-python = ">=3.11"
# dependencies = [
# "azure-identity>=1.17.0",
# "azure-mgmt-network>=28.0.0",
# "pyyaml>=6.0.2",
# ]
# ///
"""Generate CiliumCIDRGroup resources from Azure VNets and subnets.
This script always emits both:
- one CiliumCIDRGroup per VNet (all VNet address spaces)
- one CiliumCIDRGroup per subnet (subnet address prefix/prefixes)
"""
from __future__ import annotations
import argparse
import re
from typing import Any
import yaml
from azure.identity import DefaultAzureCredential
from azure.mgmt.network import NetworkManagementClient
def sanitize_name(raw: str) -> str:
"""Return a DNS-1123 compatible Kubernetes resource name."""
sanitized = re.sub(r"[^a-z0-9-]+", "-", raw.lower()).strip("-")
sanitized = re.sub(r"-{2,}", "-", sanitized)
sanitized = sanitized[:63].strip("-")
return sanitized or "cidrgroup"
def parse_resource_group_from_id(resource_id: str) -> str:
"""Extract the Azure resource group from a resource ID."""
parts = resource_id.split("/")
try:
index = parts.index("resourceGroups")
return parts[index + 1]
except (ValueError, IndexError) as exc:
raise ValueError(
f"Could not parse resource group from id: {resource_id}"
) from exc
def make_ccg(
*,
name: str,
subscription: str,
resource_group: str,
vnet: str,
subnet: str | None,
cidrs: list[str],
) -> dict[str, Any]:
"""Build a CiliumCIDRGroup manifest as a Python dictionary."""
labels: dict[str, str] = {
"source": "azure",
"azure/subscription": subscription,
"azure/resource-group": resource_group,
"azure/vnet": vnet,
}
if subnet:
labels["azure/subnet"] = subnet
return {
"apiVersion": "cilium.io/v2",
"kind": "CiliumCIDRGroup",
"metadata": {
"name": name,
"labels": labels,
},
"spec": {
"externalCIDRs": cidrs,
},
}
def main() -> None:
"""Generate CiliumCIDRGroup manifests and print them as YAML."""
parser = argparse.ArgumentParser(
description="Generate CiliumCIDRGroup resources from Azure VNets and subnets."
)
parser.add_argument("--subscription", required=True, help="Azure subscription ID")
parser.add_argument("--resource-group", help="Optional resource group filter")
args = parser.parse_args()
credential = DefaultAzureCredential()
client = NetworkManagementClient(credential, args.subscription)
vnets = (
list(client.virtual_networks.list(args.resource_group))
if args.resource_group
else list(client.virtual_networks.list_all())
)
ccg_manifests: list[dict[str, Any]] = []
for vnet in vnets:
if not vnet.id or not vnet.name:
continue
resource_group = parse_resource_group_from_id(vnet.id)
vnet_name = vnet.name
vnet_cidrs = []
if vnet.address_space and vnet.address_space.address_prefixes:
vnet_cidrs = list(vnet.address_space.address_prefixes)
if vnet_cidrs:
ccg_manifests.append(
make_ccg(
name=sanitize_name(f"azure-{vnet_name}"),
subscription=args.subscription,
resource_group=resource_group,
vnet=vnet_name,
subnet=None,
cidrs=vnet_cidrs,
)
)
for subnet in client.subnets.list(resource_group, vnet_name):
if not subnet.name:
continue
subnet_cidrs: list[str] = []
if getattr(subnet, "address_prefixes", None):
subnet_cidrs.extend(subnet.address_prefixes)
elif getattr(subnet, "address_prefix", None):
subnet_cidrs.append(subnet.address_prefix)
if not subnet_cidrs:
continue
ccg_manifests.append(
make_ccg(
name=sanitize_name(f"azure-{subnet.name}"),
subscription=args.subscription,
resource_group=resource_group,
vnet=vnet_name,
subnet=subnet.name,
cidrs=subnet_cidrs,
)
)
print(yaml.safe_dump_all(ccg_manifests, sort_keys=False))
if __name__ == "__main__":
main()Run it:
uv run azure_to_cilium_cidrgroup.py --subscription "<SUBSCRIPTION_ID>" > generated-ccg.yaml
kubectl apply -f generated-ccg.yamlTo scope it to a single resource group:
uv run azure_to_cilium_cidrgroup.py \
--subscription "<SUBSCRIPTION_ID>" \
--resource-group "<RESOURCE_GROUP>" > generated-ccg.yamlWire this into CI and your CIDR groups stay in sync with your Azure network topology without anyone touching YAML by hand.
Takeaways
CiliumCIDRGroupturns raw CIDRs into named, labeled policy targets. External endpoints get the same treatment as in-cluster identities.cidrGroupReffor explicit references,cidrGroupSelectorfor label-based selection. Pick based on how much indirection your team is comfortable with.