Ingressgateway

To route HTTP(S) traffic into the cluster, we utilize the Ingress Gateway provided by Istio.

It can be configured using Istio’s resources, like Gateways and VirtualServices, and can leverage the functionalities of the service mesh.

The best source for detailed information about the specific features and the custom resources is Istio’s own documentation.

Also a good place to start is our workshop example describing all components to route HTTPS Traffic to a service.

However, in the following, let’s address some common questions related to the Ingressgateway in the University of Münster Kubernetes cluster.

TLS Certificates

Istio resolves the secrets referenced by a Gateway in the namespace of the Ingressgateway, not in the namespace of the Gateway itself. Since you cannot create secrets in that namespace, we run a service that watches all gateways and copies every referenced secret from your namespace into the Ingressgateway’s namespace, named <namespace>--<secret-name>. That copied name is the one your gateway has to reference, which is why the security policies require the prefix.

You do not have to trigger anything: the copy follows its source, so a renewal by the certificate management is propagated automatically, and as soon as no gateway references a certificate any more, its copy is removed.

A server can reference secrets in three ways, each requiring a specific kind of secret:

This would for example serve the certificate stored in the secret example-certificate in the namespace example:

apiVersion: networking.istio.io/v1
kind: Gateway
metadata:
  name: example
  namespace: example
spec:
  selector:
    istio: ingressgateway
  servers:
    - hosts:
        - example/example.uni-muenster.de
      port:
        name: https
        number: 443
        protocol: HTTPS
      tls:
        credentialName: example--example-certificate
        minProtocolVersion: TLSV1_3
        mode: SIMPLE

Client Certificates

If you want your clients to authenticate with a certificate of their own, set the server to mode: MUTUAL and point caCertCredentialName at the CA you want to validate those client certificates against.

Note that this is not the same as the mTLS inside the service mesh, which is configured with a PeerAuthentication and secures the traffic between pods. The setting described here authenticates external clients at the Ingressgateway.

The CA is an Opaque secret in your namespace holding the CA certificate under the key ca.crt (cacert works as well):

kubectl --context kube-staging-ms1 -n example create secret generic client-cacert --from-file=ca.crt=ca.crt

Its name must end in -cacert, because Istio only recognises a secret as CA material with that suffix. Together with the namespace prefix, the server then references it as example--client-cacert:

tls:
  mode: MUTUAL
  credentialName: example--example-certificate
  caCertCredentialName: example--client-cacert

Debugging

A single faulty reference must not stop the certificates of all other namespaces from being propagated, so it is skipped instead. To keep the problem visible, a warning event is recorded on the Gateway in your own namespace.

You can see these either in your namespace’s events (e.g. kubectl --context kube-staging-ms1 -n my-namespace get events) or directly on the resource (e.g. kubectl --context kube-staging-ms1 -n my-namespace describe gateway my-gateway).

Since the naming rules are already enforced when the Gateway is created, these events usually point at a secret that is missing, not yet created by the certificate management, or of the wrong type.

Internal Services

The Istio Ingressgateways are reachable from outside of the University Network. If you still want to restrict your service to the university network, you can achieve this using an AuthorizationPolicy, as demonstrated in the following example:

apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
  name: example
  namespace: example
spec:
  rules:
    - from:
        - source:
            principals:
              - cluster.local/ns/istio-ingressgateway/sa/istio-ingressgateway-service-account
            remoteIpBlocks:
              - 10.0.0.0/8
              - 128.176.0.0/16
              - 2001:4cf0::/29
      to:
        - operation:
            ports:
              - "8080"
  selector:
    matchLabels:
      app.kubernetes.io/name: example

Custom Error Pages

To enhance customer experience, we’ve replaced some Istio error pages with more user-friendly versions. In the current setup, these error pages are set globally and cannot be customized by administrators.

503 - Service Unavailable

When our Istio Ingress Gateway can’t find your service, a custom error page will inform the customer that the service is unavailable and advise them to contact the service administrator.

403 - Forbidden

By default, Istio responds with a RBAC: Access Denied message and a 403 HTTP status code if access is denied due to AuthorizationPolicies.

To display the custom error page instead, ensure that the label custom-error.k8s.uni-muenster.de/enabled: "true" is present on the pods and that all pods have an Istio Sidecar (which should already be in place due to the AuthorizationPolicies).