Advanced Ingress & Routing: Deep Dive

This document explains why Gateway API exists, how it differs from Ingress, and when to choose it for your routing needs. It walks through the resource model (GatewayClass, Gateway, HTTPRoute) and shows how the demo’s manifests implement traffic splitting, header-based routing, and role-oriented design.

Why Gateway API Exists

The Ingress API shipped with Kubernetes 1.1 in 2016. It solved the problem of exposing HTTP services without creating a LoadBalancer for each one. Instead of 20 LoadBalancers costing $300/month in cloud fees, you could route traffic to 20 services through a single Ingress controller.

But Ingress had limitations:

Annotation sprawl: Every feature beyond basic path routing (rate limiting, retries, mirroring, TLS configs) required vendor-specific annotations. An Nginx Ingress rule looked different from a Traefik rule. Portability was poor.
HTTP/HTTPS only: No support for TCP, UDP, gRPC as a first-class protocol, or custom protocols. You needed separate solutions for non-HTTP traffic.
Single-role design: The cluster admin who operates the load balancer infrastructure and the developer who configures routing rules both edited the same Ingress object. No separation of concerns.
Limited matching: Path prefix and exact match were the only options. No header matching, query parameters, HTTP method routing, or traffic splitting built in.

Gateway API was designed to fix these problems. It shipped as beta in Kubernetes 1.25 (2022) and reached GA in 1.26 (2023). The goal was a more expressive, portable, role-oriented API for routing that works across HTTP, gRPC, TCP, and TLS.

The Gateway API Resource Model

Gateway API splits routing into three resources, each owned by a different role:

GatewayClass (cluster-scoped)         Infrastructure team
   |
   └─> Gateway (namespaced)           Platform/ops team
          |
          └─> HTTPRoute (namespaced)  Application developers

This separation lets each team manage their part of the stack without stepping on each other.

GatewayClass: Infrastructure Configuration

# From manifests/gateway-class.yaml
apiVersion: gateway.networking.k8s.io/v1
kind: GatewayClass
metadata:
  name: envoy
spec:
  controllerName: gateway.envoyproxy.io/gatewayclass-controller

A GatewayClass links to a controller implementation. The controllerName tells Kubernetes which controller should reconcile this class. In this demo, it is Envoy Gateway. Other implementations include:

gateway.nginx.org/nginx-gateway-controller (NGINX Gateway Fabric)
projectcontour.io/gateway-controller (Contour)
istio.io/gateway-controller (Istio)

The cluster admin creates GatewayClasses. Developers reference them in Gateway objects. This decouples the choice of implementation from the application configuration.

Gateway: Proxy Deployment

# From manifests/gateway.yaml
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: demo-gateway
  namespace: gateway-demo
spec:
  gatewayClassName: envoy
  listeners:
    - name: http
      protocol: HTTP
      port: 80
      allowedRoutes:
        namespaces:
          from: Same

A Gateway defines a load balancer instance. It specifies:

gatewayClassName: Which GatewayClass to use. This determines the controller that provisions the proxy.
listeners: Network listeners (protocol and port). Each listener can accept HTTP, HTTPS, TCP, TLS, or UDP.
allowedRoutes: Which routes can attach. from: Same means only routes in the same namespace. from: All allows any namespace. from: Selector uses label selectors.

When you create a Gateway, the controller provisions a Deployment and a LoadBalancer Service. In this demo, Envoy Gateway creates a Deployment named envoy-gateway-demo-gateway-xxxx and a LoadBalancer Service with the same name.

The platform team (SRE, ops) creates Gateways. Developers create routes that attach to them.

HTTPRoute: Application Routing Rules

# From manifests/httproute-simple.yaml
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: simple-route
  namespace: gateway-demo
spec:
  parentRefs:
    - name: demo-gateway
  rules:
    - matches:
        - path:
            type: PathPrefix
            value: /
      backendRefs:
        - name: app-v1
          port: 80

An HTTPRoute attaches to a Gateway via parentRefs. The route defines match conditions (path, headers, query params, HTTP method) and backend targets (Services).

Application developers create HTTPRoutes. They do not need to know the Gateway’s implementation or infrastructure details. They only need the Gateway’s name.

How Gateway API Differs from Ingress

Ingress

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: demo-ingress
  annotations:
    nginx.ingress.kubernetes.io/canary: "true"
    nginx.ingress.kubernetes.io/canary-weight: "20"
spec:
  rules:
    - host: demo.example.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: app-v1
                port:
                  number: 80

Traffic splitting requires annotations. Different Ingress controllers have different annotation formats. Switching from Nginx to Traefik means rewriting every Ingress.

Gateway API

# From manifests/httproute-split.yaml
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: traffic-split
  namespace: gateway-demo
spec:
  parentRefs:
    - name: demo-gateway
  rules:
    - matches:
        - path:
            type: PathPrefix
            value: /
      backendRefs:
        - name: app-v1
          port: 80
          weight: 80
        - name: app-v2
          port: 80
          weight: 20

Traffic splitting is a first-class field. No annotations. The same manifest works with Envoy Gateway, NGINX Gateway Fabric, Contour, and Istio.

Comparison Table

Feature	Ingress	Gateway API
Traffic splitting	Annotations (vendor-specific)	`weight` field (portable)
Header matching	Not supported (annotations)	`matches.headers` field
Protocol support	HTTP, HTTPS	HTTP, HTTPS, gRPC, TCP, TLS, UDP
Role separation	Single Ingress object	GatewayClass, Gateway, HTTPRoute
Extensibility	Annotations	Policy attachment (future spec)
Multi-team sharing	Namespace-scoped, limited	ReferenceGrants for cross-namespace routes
Portability	Low (annotations differ)	High (same spec across implementations)

Key Configuration Fields

Listener Configuration

listeners:
  - name: http
    protocol: HTTP
    port: 80
    allowedRoutes:
      namespaces:
        from: Same

protocol: HTTP, HTTPS, TCP, TLS, or UDP. This determines which route type can attach (HTTPRoute for HTTP/HTTPS, TCPRoute for TCP, etc.).

port: The port the listener binds to. External clients connect to this port.

allowedRoutes.namespaces.from: Controls which namespaces can attach routes. Same restricts to the Gateway’s namespace. All allows any namespace. Selector uses label selectors to allow specific namespaces.

HTTPS Listener with TLS Termination

listeners:
  - name: https
    protocol: HTTPS
    port: 443
    tls:
      mode: Terminate
      certificateRefs:
        - name: example-cert
          kind: Secret

tls.mode: Terminate decrypts traffic at the Gateway. Passthrough forwards encrypted traffic to the backend.

certificateRefs: Points to a Secret with tls.crt and tls.key. The Secret must be in the same namespace as the Gateway (or accessible via ReferenceGrant).

Path Matching

matches:
  - path:
      type: PathPrefix
      value: /api

type: PathPrefix (prefix match), Exact (exact match), or RegularExpression (regex, if supported by the implementation).

value: The path to match. /api matches /api, /api/v1, /api/v1/users, etc.

Header Matching for A/B Testing

# From manifests/httproute-headers.yaml
rules:
  - matches:
      - headers:
          - name: X-Version
            value: v2
    backendRefs:
      - name: app-v2
        port: 80
  - matches:
      - path:
          type: PathPrefix
          value: /
    backendRefs:
      - name: app-v1
        port: 80

The first rule matches requests with the X-Version: v2 header and sends them to app-v2. The second rule is a catch-all for everything else.

Rules are evaluated in order. The first matching rule wins. If no rules match, the request is rejected with a 404.

Traffic Splitting with Weights

# From manifests/httproute-split.yaml
backendRefs:
  - name: app-v1
    port: 80
    weight: 80
  - name: app-v2
    port: 80
    weight: 20

Weights are relative. 80 and 20 means 80% to v1, 20% to v2. You could also write 8 and 2, or 800 and 200. The ratio is what matters.

This is useful for canary deployments. Start with weight: 95 for the stable version and weight: 5 for the canary. Monitor error rates. If the canary is healthy, shift to 90/10, then 80/20, then 50/50, and finally 0/100.

Query Parameter Matching

Not all implementations support this yet, but the spec allows it:

matches:
  - queryParams:
      - name: version
        value: v2

This sends /?version=v2 to a different backend than /?version=v1.

HTTP Method Matching

matches:
  - method: POST

Match only POST requests. Useful for routing read-only GET traffic differently from write traffic.

Traffic Splitting and Canary Deployments

The demo’s httproute-split.yaml sends 80% of traffic to v1 and 20% to v2:

backendRefs:
  - name: app-v1
    port: 80
    weight: 80
  - name: app-v2
    port: 80
    weight: 20

This is a canary deployment. You deploy the new version (v2) alongside the old version (v1). A small percentage of users get the new version. You monitor metrics (error rate, latency, conversion). If the canary looks good, you increase the weight. If it fails, you set the weight to 0.

Blue-Green Deployments

For a blue-green deployment, start with 100% on blue:

backendRefs:
  - name: app-blue
    port: 80
    weight: 100
  - name: app-green
    port: 80
    weight: 0

When ready to switch, change the weights:

backendRefs:
  - name: app-blue
    port: 80
    weight: 0
  - name: app-green
    port: 80
    weight: 100

If something breaks, you can roll back by swapping the weights again.

Shadow Traffic (Request Mirroring)

Some implementations support request mirroring:

filters:
  - type: RequestMirror
    requestMirror:
      backendRef:
        name: app-v2
        port: 80

This sends a copy of every request to app-v2 without affecting the response sent to the client. The client gets the response from the primary backend. The mirrored backend processes the request but its response is discarded.

This is useful for testing a new version with production traffic without risking user-facing errors.

Header-Based Routing for A/B Testing

The demo’s httproute-headers.yaml routes based on the X-Version header:

# From manifests/httproute-headers.yaml
rules:
  - matches:
      - headers:
          - name: X-Version
            value: v2
    backendRefs:
      - name: app-v2
        port: 80
  - matches:
      - path:
          type: PathPrefix
          value: /
    backendRefs:
      - name: app-v1
        port: 80

A client sends curl -H "X-Version: v2" http://gateway-ip and gets v2. Everyone else gets v1.

This is A/B testing. You control which users see which version. In production, you might use a cookie or a user ID in the header. Your frontend sets the header based on a user’s experiment group.

Combining Header and Path Matching

You can match both headers and paths:

matches:
  - path:
      type: PathPrefix
      value: /api
    headers:
      - name: X-Version
        value: v2

This only matches /api requests with the X-Version: v2 header.

Implementations: Envoy Gateway, Contour, NGINX Gateway Fabric, Istio

Gateway API is a spec, not an implementation. You need a controller to make it work. Each controller provisions a different proxy.

Envoy Gateway

Proxy: Envoy
Maturity: GA (v1.0 in 2024)
Features: All GA Gateway API features, TLS termination, rate limiting, auth via external services
Deployment: Helm chart installs a controller that creates Envoy Deployments for each Gateway

This demo uses Envoy Gateway. It is the reference implementation developed by the Envoy maintainers.

NGINX Gateway Fabric

Proxy: NGINX
Maturity: Beta
Features: HTTP routing, TLS termination, header manipulation
Deployment: Helm chart with a single NGINX instance for all Gateways

Developed by F5 (the company behind NGINX). If you already use NGINX Ingress, this is a familiar upgrade path.

Contour

Proxy: Envoy
Maturity: GA
Features: Full Gateway API support, gRPC routing, TLS delegation
Deployment: Contour controller + Envoy data plane

Contour was one of the first Ingress controllers and is now one of the most mature Gateway API implementations.

Istio

Proxy: Envoy
Maturity: GA
Features: Gateway API support as an alternative to Istio’s VirtualService and Gateway CRDs
Deployment: Part of the Istio service mesh

If you run Istio, you can use Gateway API instead of Istio’s custom routing resources. This makes your routes portable across service meshes.

Feature Compatibility

Not all implementations support all features. Check the Gateway API implementation page for conformance reports.

Core features (path routing, traffic splitting, header matching) are widely supported. Extended features (URL rewrite, request mirroring, rate limiting) vary by implementation.

TLS Termination and HTTPS Listeners

The demo uses HTTP for simplicity. Production Gateways use HTTPS.

Creating a TLS Certificate

kubectl create secret tls example-cert \
  --cert=tls.crt \
  --key=tls.key \
  -n gateway-demo

Gateway with HTTPS Listener

listeners:
  - name: https
    protocol: HTTPS
    port: 443
    tls:
      mode: Terminate
      certificateRefs:
        - name: example-cert
    allowedRoutes:
      namespaces:
        from: Same

The Gateway decrypts traffic using the certificate in example-cert. HTTPRoutes attached to this listener receive plaintext HTTP traffic.

TLS Passthrough

If your backend needs to handle TLS itself (mutual TLS, end-to-end encryption):

listeners:
  - name: tls-passthrough
    protocol: TLS
    port: 443
    tls:
      mode: Passthrough

The Gateway forwards encrypted traffic to the backend without decrypting it. Use a TLSRoute (not HTTPRoute) for this scenario.

Redirect HTTP to HTTPS

listeners:
  - name: http
    protocol: HTTP
    port: 80
  - name: https
    protocol: HTTPS
    port: 443
    tls:
      mode: Terminate
      certificateRefs:
        - name: example-cert

Create an HTTPRoute that redirects HTTP traffic:

rules:
  - filters:
      - type: RequestRedirect
        requestRedirect:
          scheme: https
          statusCode: 301

All HTTP requests receive a 301 redirect to the HTTPS URL.

Production Considerations

A common pattern is one Gateway per environment (dev, staging, prod), shared by multiple teams.

The platform team creates the Gateway:

apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: prod-gateway
  namespace: infra
spec:
  gatewayClassName: envoy
  listeners:
    - name: https
      protocol: HTTPS
      port: 443
      allowedRoutes:
        namespaces:
          from: All

Application teams create HTTPRoutes in their own namespaces:

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: my-app-route
  namespace: team-a
spec:
  parentRefs:
    - name: prod-gateway
      namespace: infra
  hostnames:
    - app.example.com
  rules:
    - backendRefs:
        - name: my-app
          port: 80

This requires a ReferenceGrant to allow cross-namespace references.

ReferenceGrants

apiVersion: gateway.networking.k8s.io/v1beta1
kind: ReferenceGrant
metadata:
  name: allow-routes-from-team-a
  namespace: infra
spec:
  from:
    - group: gateway.networking.k8s.io
      kind: HTTPRoute
      namespace: team-a
  to:
    - group: gateway.networking.k8s.io
      kind: Gateway

This grants team-a permission to attach HTTPRoutes to Gateways in the infra namespace.

Policy Attachment (Future)

The Gateway API spec includes a policy attachment model for features like rate limiting, authentication, and CORS. Each implementation defines its own policy CRDs:

apiVersion: gateway.envoyproxy.io/v1alpha1
kind: RateLimitPolicy
metadata:
  name: rate-limit
  namespace: gateway-demo
spec:
  targetRef:
    group: gateway.networking.k8s.io
    kind: HTTPRoute
    name: traffic-split
  rateLimits:
    - clientSelectors:
        - headers:
            - name: X-User-ID
      limits:
        - requests: 100
          unit: Minute

This attaches a rate limit policy to the traffic-split HTTPRoute.

Resource Limits

Each Gateway creates a Deployment. Set resource limits on the GatewayClass or via implementation-specific configuration.

For Envoy Gateway:

apiVersion: gateway.envoyproxy.io/v1alpha1
kind: EnvoyProxy
metadata:
  name: custom-proxy
  namespace: envoy-gateway-system
spec:
  provider:
    type: Kubernetes
    kubernetes:
      envoyDeployment:
        replicas: 3
        pod:
          resources:
            requests:
              cpu: 100m
              memory: 128Mi
            limits:
              cpu: 500m
              memory: 512Mi

Reference this in the GatewayClass:

apiVersion: gateway.networking.k8s.io/v1
kind: GatewayClass
metadata:
  name: envoy
spec:
  controllerName: gateway.envoyproxy.io/gatewayclass-controller
  parametersRef:
    group: gateway.envoyproxy.io
    kind: EnvoyProxy
    name: custom-proxy
    namespace: envoy-gateway-system

Observability

Gateway implementations expose Prometheus metrics. For Envoy Gateway, scrape the Envoy proxy pods:

apiVersion: v1
kind: Service
metadata:
  name: envoy-metrics
  namespace: gateway-demo
  labels:
    prometheus: scrape
spec:
  selector:
    gateway.envoyproxy.io/owning-gateway-name: demo-gateway
  ports:
    - name: metrics
      port: 19001

Metrics include request count, latency percentiles, error rates, and backend health.

High Availability

Run multiple Gateway replicas:

spec:
  provider:
    type: Kubernetes
    kubernetes:
      envoyDeployment:
        replicas: 3

The LoadBalancer Service distributes traffic across all replicas. If one pod crashes, the others continue serving traffic.

Cost Optimization

Unlike Ingress, each Gateway provisions a LoadBalancer. In cloud environments, this costs $15-20/month per Gateway.

For multi-team setups, share a single Gateway across teams using ReferenceGrants. This reduces the number of LoadBalancers.

Alternatively, use a node-level proxy like MetalLB or Cilium’s LoadBalancer IP Pool to avoid cloud load balancer costs.

Common Pitfalls

GatewayClass Not Found

Error: failed to create gateway: gatewayclass.gateway.networking.k8s.io "envoy" not found

The GatewayClass must exist before creating a Gateway. Install the Gateway API CRDs and the controller:

kubectl apply -f https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.1.0/standard-install.yaml
helm install eg oci://docker.io/envoyproxy/gateway-helm --version v1.1.2 -n envoy-gateway-system --create-namespace

Gateway Not Programmed

kubectl get gateway -n gateway-demo
NAME           CLASS   ADDRESS   PROGRAMMED   AGE
demo-gateway   envoy             False        10s

The Gateway is waiting for the LoadBalancer to assign an IP. On minikube, run:

minikube tunnel

On cloud providers, check that the LoadBalancer Service was created:

kubectl get svc -n gateway-demo

If the Service is stuck Pending, check the controller logs:

kubectl logs -n envoy-gateway-system deployment/envoy-gateway

parentRefs Mismatch

parentRefs:
  - name: wrong-gateway

The HTTPRoute references a Gateway that does not exist or is in a different namespace. If the Gateway is in a different namespace, specify it:

parentRefs:
  - name: demo-gateway
    namespace: gateway-demo

And ensure a ReferenceGrant allows the cross-namespace reference.

Listener Protocol Mismatch

listeners:
  - name: http
    protocol: HTTP

apiVersion: gateway.networking.k8s.io/v1alpha2
kind: TCPRoute
spec:
  parentRefs:
    - name: demo-gateway

A TCPRoute cannot attach to an HTTP listener. Change the listener protocol to TCP or use an HTTPRoute.

Listener Port Conflicts

Two listeners on the same Gateway cannot use the same port:

listeners:
  - name: http
    protocol: HTTP
    port: 80
  - name: grpc
    protocol: HTTP
    port: 80

Change one of the ports or combine the listeners into a single listener with multiple HTTPRoutes.

Weight Sum Zero

backendRefs:
  - name: app-v1
    port: 80
    weight: 0
  - name: app-v2
    port: 80
    weight: 0

If all weights are zero, no traffic is sent. The Gateway returns a 503. Ensure at least one backend has a non-zero weight.

Certificate in Wrong Namespace

listeners:
  - name: https
    protocol: HTTPS
    port: 443
    tls:
      certificateRefs:
        - name: example-cert

The Secret example-cert must be in the same namespace as the Gateway. If it is in a different namespace, create a ReferenceGrant.