Ben Ebsworth

Istio "Service Mesh" Topologies

technology
kubernetes
service mesh
istio

July 03, 2020

Below are a number of different istio network topologies that can be constructed and a depiction of how the Istio networking CRDs are used together to enable connectivity within an Istio service mesh.

Ingress

Istio Service Mesh

Given a Kubernetes Namespace has been configured to allow for Istio sidecar injection, then services deployed into this namespace with be accompanied by an Envoy Sidecar Proxy. In this way the service will be augmented into the Service Mesh.

kubectl get pods
NAME                              READY   STATUS    RESTARTS   AGE
api-auth-5f74b8c466-4fcl6         2/2     Running   0          1h
api-information-bdc664c59-dtscd   2/2     Running   0          1h

The 2/2 signifies that the given Pod has two containers, one being the microservice application and the other being the sidecar proxy.

Gateway

The Gateway resource can be consider the mechanism to define a front-end listener, in terms of port and hostname. Additionally how to map terminated TLS certificates to a given host listener.

apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: api-gateway
spec:
  selector:
    istio: ingressgateway # use istio default gateyway 
  servers:
  - port:
      number: 80
      name: http
      protocol: HTTP
    hosts:
    - "api.training.local"
  - port:
      number: 443 
      name: https
      protocol: HTTPS
    hosts:
    - "api.training.local"
    tls:
      mode: SIMPLE
      serverCertificate: /etc/certs/https-api/tls.crt
      privateKey: /etc/certs/https-api/tls.key
VirtualService

A VirtualService CRD defines the instructions for how requests will be routed for a given hostname. It allows a list of matches to be specified, with corresponding destinations. The match rules will be evaluated from top to bottom in terms of precedence. This resource has fined grained control on where it will reside, depending on the ingress and egress topology within the service mesh.

apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: product-gateway
spec:
  hosts:
  - "api.training.local"
  gateways:
  - api-gateway
  http:
  - match:
    - uri:
        prefix: "/api/product"
    route:
    - destination:
        host: api-product
DestinationRule

A DestinationRule is responsible for providing session information for when a connection is being established to a given outbound endpoint, either either within the mesh or externally.

apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: api-product
spec:
  host: api-product
  trafficPolicy:
    tls:
      mode: ISTIO_MUTUAL
  subsets:
  - name: v1
    labels:
      version: v1
  - name: v2
      labels:
        version: v2
VirtualService

A more granular, or service specific VirtualService resource can be applied to the mesh. This provides a mechanism to decouple traffic steering from infrastructure scaling. What this means is, we can now making a routing/steering decision from the sidecar instead of having to route the traffic to a main frontend-proxy or internal load balancing node.

apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: api-product
spec:
  gateways:
  - mesh
  hosts:
  - api-product
  http:
  - route:
    - destination:
        host: api-product
        subset: v1
DestinationRule

The DestinationRule configuration will reside on both the ingressgateway and the sidecar. There will also be a DestinationRule used to establish a mTLS connection for the communication between the ingressgateway and the sidecar, as shown below

apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: default
  namespace: istio-system
spec:
  host: '*.local'
  trafficPolicy:
    tls:
      mode: ISTIO_MUTUAL

Egress - Basic

Service Mesh Egress - Sidecar Direct

An Istio service mesh will operate as a whitelist filter for all traffic egressing from the mesh. Consequently we need to add ServiceEntry resources to effectively allow traffic out of the service mesh. There is a mechanism to break out of this default behavior, where we can enable certain subnets to bypass the Envoy sidecar when making extern/upstream requests. It is enabled by add the following annotation to your Deployment resource as follows:

 

apiVersion: apps/v1
kind: Deployment
metadata:
  name: microservice-without-istio-egress 
spec:
  replicas: 1
  selector:
    matchLabels:
      app: microservice-without-istio-egress 
  template:
    metadata:
      ...
      annotations:
        ...
        traffic.sidecar.istio.io/excludeOutboundIPRanges: 0.0.0.0/0
        ...
      ...
ServiceEntry

The ServiceEntry CRD resource is used to create an outbound object inside of the service mesh, so that services within the mesh are able to access the defined upstream endpoint.

 


apiVersion: networking.istio.io/v1alpha3
kind: ServiceEntry
metadata:
  name: external-svc-https
spec:
  hosts:
  - fernago.core.gcp.anz
  location: MESH_EXTERNAL
  ports:
  - number: 443
    name: https
    protocol: HTTPS
  resolution: DNS
DestinationRule

In order to define more granular session information for a given endpoint or destination we utilise the DestinationRule CRD. This will inform an Envoy proxy how it should establish a connection with a service either internal or external to the service mesh In the example below, we are specifying that we should use a mTLS based connection and which certificates to use when establishing the connection.

 

apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: api-stock-mtls 
spec:
  host: stock.training.local 
  trafficPolicy:
    tls:
      mode: MUTUAL
      clientCertificate: /etc/certs/fernago/tls.crt
      privateKey: /etc/certs/fernago/fernago.key
      caCertificates: /etc/certs/fernago/ca.pem
External endpoint

Istio defines two types of endpoints, either MESH_EXTERNAL or MESH_INTERNAL. Internal services are, for the most part, entirely auto discovered via the built-in Kubernetes service discovery mechanisms. However for service endpoints outside the Istio service mesh, we need to more explicitly define these as ServiceEntry resources so that the service mesh can intelligently route when required. An "external" endpoint can be a microservice which lives outside of the service mesh, but within the same Kubernetes cluster. In this scenario we can define the following DestinationRule to disable mTLS when communicating to the service.

 

apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: fernago
  namespace: ext-ns
spec:
  host: fernago.ext-ns.svc.cluster.local
  trafficPolicy:
    tls:
      mode: DISABLE

Egress - Advanced

ServiceMesh - Egress Gateways

A more complex service mesh topology can be deployed which facilitates enabling more robust security boundaries. By having a egressgateway service act as a "gateway" out of the mesh, it is possible to enforce traffic flows via NetworkPolicy to only allow egress from the egressgateway service. This prevents a malicious actor bypassing the default routing behavior that services within the mesh will use.

Sidecar - ServiceEntry [global]

Creates an outbound listener on the sidecar to handle outbound requests on a given Port (if it doesn't already exist). This then allows for a VirtualService policy to handle the request and route it to the given destination. This ServiceEntry will effectuate both Sidecars and Gateways, it is a global policy (spans across namespaces and mesh components).  

apiVersion: networking.istio.io/v1alpha3
kind: ServiceEntry
metadata:
  name: stock-to-gateway
spec:
  hosts:
  - stock.training.local
  location: MESH_EXTERNAL
  ports:
  - number: 443 
    name: https
    protocol: HTTPS
  resolution: DNS
Sidecar - VirtualService

The VirtualService will be targeted to apply only to the Sidecars via the gateways: selector. This VirtualService policy aims to route all traffic on a specific port (80) and route it to the egressgateway. Note that we specify a subset, which allows us to use more targeting or specific policy for the target destination.  

apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: stock-to-egressgateway
spec:
  hosts:
  - stock.training.local
  gateways:
  - mesh
  http:
  - match:
    - gateways:
      - mesh
      port: 80 
    route:
    - destination:
        host: egressgateway.istio-system.svc.cluster.local
        subset: stock-to-gateway
        port:
          number: 443
      weight: 100
Sidecar - DestinationRule [global]

The DestinationRule which is intended to be used by the Sidecar defines the required information to enable a mTLS connection to be established between. The key being that is sets the sni: stock.training.local and mode: ISTIO_MUTUAL, for all traffic routed by the VirtualService shown previously  

apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: stock-to-gateway
spec:
  host: istio-egressgateway.istio-system.svc.cluster.local
  subsets:
  - name: stock-to-gateway 
    trafficPolicy:
      loadBalancer:
        simple: ROUND_ROBIN
      portLevelSettings:
      - port:
          number: 443 
        tls:
          mode: ISTIO_MUTUAL
          sni: stock.training.local 
Egress Gateway - Gateway

Similar to what we've seen so far for an ingressgateway, we have the egressgateway. This is working in a similar fashion but in reverse in terms of the direction of traffic. The Gateway resource will enable the egressgateway to handle requests for the given hostname, as well as specifies what certificates to terminate on the given port. In this case it terminates the internal istio certificates which are distributed by Citadel.  

apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: stock 
spec:
  selector:
    istio: egressgateway
  servers:
  - hosts:
    - stock.training.local 
    port:
      name: https-stock 
      number: 443
      protocol: HTTPS
    tls:
      caCertificates: /etc/certs/root-cert.pem
      mode: MUTUAL
      privateKey: /etc/certs/key.pem
      serverCertificate: /etc/certs/cert-chain.pem
Egress Gateway - VirtualService

Once traffic has been routed from Envoy Sidecars within the mesh, the next stage of routing takes place on the egressgateway. The VirtualService shown below, routes manages all traffic for the stock.training.local domain, and will utilise the external DestinationRule subset.  

apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: stock-to-egressgateway
spec:
  hosts:
  - stock.training.local
  gateways:
  - istio-egressgateway
  http:
  - match:
    - gateways:
      - istio-egressgateway
      port: 443
    route:
    - destination:
        host: stock.training.local
        subset: external
        port:
          number: 443 
      weight: 100

Egress Gateway - DestinationRule [global]

For the final stage of the request handling that occurs within the Service Mesh, we have the below DestinationRule. This policy effectively instructs the eggressgateway to originate a mTLS connection to the target destination, with the specified certificates for the mutual authentication handshake and validation.  

apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: stock-to-external
spec:
  host: stock.training.local 
  subsets:
  - name: external
    trafficPolicy:
      loadBalancer:
        simple: ROUND_ROBIN
      portLevelSettings:
      - loadBalancer:
          simple: ROUND_ROBIN
        port:
          number: 443 
      tls:
        mode: MUTUAL
        clientCertificate: /etc/certs/stock/tls.crt
        privateKey: /etc/certs/stock/tls.key
        caCertificates: /etc/certs/stock/ca.pem
Egress Gateway - ServiceEntry [global]

The ServiceEntry policy will be deployed to effectively all mesh components, enabling outbound resolution by the mesh components.

 

apiVersion: networking.istio.io/v1alpha3
kind: ServiceEntry
metadata:
  name: stockt-to-gateway
spec:
  hosts:
  - stock.training.local
  location: MESH_EXTERNAL
  ports:
  - number: 443 
    name: https
    protocol: HTTPS
  resolution: DNS

Written by Ben Ebsworth, thoughts are their own and any material is representative of an attempt to self-educate and explore ideas and technology You should follow him on Twitter