Security Policies

In order to secure the Kubernetes cluster, some entries in resources are restricted. The constraints are checked in the cluster with Gatekeeper and if necessary, resources are rejected.

Creating erroneous resources results in error messages during creation. For resources like pods that should be automatically generated from ReplicaSets, these error messages can be found in the events. A few constraints are not enforced but only reported as a warning; those are marked as such below.

Wherever a value has to be “agreed upon with the administrators”, it is enabled per namespace. Host names can also be enabled as wildcard patterns, for example *.example.uni-muenster.de.

The following describes the constraints for the various resources.

Application

ArgoCD Applications in your namespace are restricted as well.

.spec.project must be set explicitly and may only reference one of the ArgoCD projects that have been enabled for your namespace.

Config management plugins are currently not allowed, that is neither .spec.source.plugin nor .spec.sources[*].plugin may be set.

Certificate

If .spec.issuerRef.kind is set to ClusterIssuer or DFNClusterIssuer, all entries in .spec.commonName and .spec.dnsNames must be agreed upon with the administrators.

DNSEndpoint

All DNS names in .spec.endpoints[*].dnsName must be agreed upon with the administrators and enabled individually.

Gateway

The .spec.selector is limited to the explicit entries {"istio": "ingressgateway"}, {"istio": "egressgateway"} and {"istio": "egressgateway-ng"}.

If ingressgateway is selected in the selector, all hosts .spec.servers[*].hosts must start with either ./ or with <namespace>/.

Also, in this case, only the DNS entries that have been previously agreed upon with the administrators are allowed as hosts.

Every secret referenced by a gateway — .spec.servers[*].tls.credentialName, each entry of .spec.servers[*].tls.credentialNames and .spec.servers[*].tls.caCertCredentialName — must be prefixed with the namespace name, that is <namespace>--<secret-name>. Istio resolves those references in the namespace of the ingress gateway and not in the namespace of the Gateway, so the prefix is what keeps one namespace from picking up another one’s certificate. The secrets are copied there for you under that name.

In addition, .spec.servers[*].tls.caCertCredentialName must end in -cacert, because Istio only recognises a secret as CA material with that suffix.

Ingress

All hosts in the Ingress resources, both in .spec.rules[*].host and .spec.tls[*].hosts must be cleared with the administrators and enabled individually.

Pod

When istio containers are used as sidecars, all other containers should not run as user 1337. That is .spec.securityContext.runAsUser and .spec.containers[name!=istio].securityContext.runAsUser should not be 1337. This constraint currently only produces a warning and does not reject the pod.

Also, the allowed .spec.tolerations on the pods are restricted and must be discussed with the administrators. For example certain tolerations are required to run pods on GPU nodes or worker nodes. The tolerations for node.kubernetes.io/not-ready and node.kubernetes.io/unreachable (NoExecute, Exists), which Kubernetes adds to every pod, are always allowed.

If a pod tolerates role.node.kubernetes.io/worker-vgpu (NoSchedule, Exists) to be scheduled on a GPU node, the sum of the .spec.containers[*].resources.limits.nvidia.com/gpu and .spec.initContainers[*].resources.limits.nvidia.com/gpu values must be at least 1 and at most 4. This is probably illustrated more clearly in this example.

If a seccompProfile is set — either at pod level in .spec.securityContext.seccompProfile or on a container — its type must be RuntimeDefault. Not setting a profile at all is allowed.

Every container must set resources.limits.ephemeral-storage and it must not exceed 32Gi. The istio containers injected into your pods (istio-proxy, istio-validation) are handled automatically and do not need a limit of their own.

Also, for pods, we have complex security profiles that further restrict the pods. These are explained in detail below.

Security Profile

Two security profiles are available for tenants: default and hardened_default. Which profile applies to a namespace is controlled by the namespace label constraints.gatekeeper.sh/securityprofile. Switching a namespace to hardened_default must currently be agreed upon with the administrators, the same way as other settings in this document.

If a Deployment’s pods violate the active security profile — default or hardened_default — the Deployment itself will not show an error. The rejection is only visible as an event on the generated ReplicaSet, the same as for other constraints described at the top of this page. You can see it with, for example, kubectl --context kube-staging-ms1 -n my-namespace get events or kubectl --context kube-staging-ms1 -n my-namespace describe replicaset <name>.

Default in detail

This is the baseline profile. It protects the cluster from unauthorized access and applies to every namespace, whether it is set to default or hardened_default.

Hardened default in detail

Namespaces set to hardened_default get the full default rule set above, plus the full hardened-* rule set below, applied on top. Where a hardened-* rule’s restriction is identical to its default-* counterpart, it still exists as a separately named rule so it can be dropped independently (see below).

Individual hardened-* rules can be disabled for a single pod by setting the label constraints.gatekeeper.sh/drop-<constraint-name> on that pod, for example constraints.gatekeeper.sh/drop-hardened-psp-capabilities. Unlike switching the namespace profile itself, this label can be set by tenants without involving the administrators.

Dropping a hardened-* rule normally falls back to the matching default-* rule rather than removing the restriction outright: dropping hardened-psp-capabilities falls back to default-psp-capabilities, so adding extra capabilities is still forbidden but dropping ALL is no longer required, and dropping hardened-psp-forbidden-systcls falls back to default-psp-forbidden-systcls, so net.ipv4.ip_unprivileged_port_start is allowed again. The exceptions are hardened-psp-pods-allowed-user-ranges, hardened-psp-allow-privilege-escalation-container and hardened-psp-readonlyrootfilesystem, which have no default counterpart and are fully disabled once dropped.

PodDisruptionBudget

A PodDisruptionBudget must not block voluntary disruptions such as node drains. Therefore

• .spec.maxUnavailable must not be 0, and
• .spec.minAvailable must be lower than the number of replicas of the Deployment or StatefulSet it selects.

Deployments and StatefulSets selected by such a PodDisruptionBudget are rejected as well.

Service

Services of the type LoadBalancer are not allowed per se and must be permitted by the administrators for the namespace.

For the default (OpenStack Octavia) load balancers, one of the annotations loadbalancer.openstack.org/keep-floatingip: "true" or service.beta.kubernetes.io/openstack-internal-load-balancer: "true" must be set, and the IP in .spec.loadBalancerIP must be discussed with the administrators and enabled individually.

For load balancers with .spec.loadBalancerClass: io.cilium/bgp-control-plane no annotation is required. Here the IPs actually assigned to the service (.status.loadBalancer.ingress[*].ip) must be among the IPs enabled for the namespace.

No entries are allowed in .spec.externalIPs.

All ports exported to Istio with the annotation networking.istio.io/exportTo must either have appProtocol set in each .spec.ports or the name must start with the protocol, that is one of http, http2, https, tcp, tls, grpc, grpc-web, mongo, mysql, redis, udp. See the documentation for details.

VirtualService

The .spec.gateways entry must always contain at least one gateway. These gateways must not contain /.

Also, the gateway mesh is only allowed in combination with .spec.exportTo: ["."].

If the wwu.io/nic_node annotation is used in a VirtualService to export a host to the NIC, the entries in .spec.hosts are limited to the DNS entries agreed upon with the administrators.