Day 29: Strengthening Kubernetes Security with Open Policy Agent (OPA) and Gatekeeper

Welcome back to Day 29 of my 90-day DevOps journey! Today, we're diving into Kubernetes security by exploring how to use Open Policy Agent (OPA) and Gatekeeper to enforce policies within your cluster.

This guide is designed for beginners and will walk you through setting up OPA with Gatekeeper, creating custom policies, and handling common issues you might encounter along the way.

Table of Contents

• Introduction to OPA and Gatekeeper

• Project Prerequisites

• Folder Structure

• Step-by-Step Guide

  • Step 1: Setting Up the Kubernetes Cluster

  • Step 2: Installing Gatekeeper

  • Step 3: Verifying the Installation

  • Step 4: Creating Custom Policies

  • Step 5: Testing the Policies

• Common Issues and Troubleshooting

• Conclusion

Introduction to OPA and Gatekeeper

Open Policy Agent (OPA) is like a rulebook for your software. It lets you define rules (policies) that can be applied to different parts of your system. In Kubernetes, OPA helps you enforce rules on resources before they're created or changed.

Gatekeeper is a special tool that works with OPA and Kubernetes. It allows you to create and apply policies using Kubernetes' own language (Custom Resource Definitions or CRDs).

By combining OPA and Gatekeeper, you can make sure everything in your Kubernetes cluster follows your security and compliance rules.

Project Prerequisites

• A running Kubernetes cluster: You can use Minikube, Docker Desktop, or any managed Kubernetes service.

• kubectl installed and configured: This is the command-line tool to interact with your Kubernetes cluster.

• Basic understanding of Kubernetes resources: You should know about pods, deployments, services, etc.

Folder Structure

Let's organize our project files for better readability:

90days-devops/
└── day29/
    ├── manifests/
    │   └── gatekeeper.yaml
    ├── policies/
    │   ├── deny-privileged-pods.yaml
    │   ├── require-labels.yaml
    │   └── sample-app.yaml
    └── README.md

• manifests/: Contains YAML files for deploying Gatekeeper.

• policies/: Holds custom policy definitions and sample applications to test the policies.

• README.md: Documentation and steps followed during the project.

Step-by-Step Guide

Step 1: Setting Up the Kubernetes Cluster

If you haven't set up a Kubernetes cluster yet, you can use Minikube:

minikube start
kubectl config use-context minikube

Verify the cluster is running:

kubectl get nodes

Step 2: Installing Gatekeeper

Apply the Gatekeeper manifest to install it on your cluster:

kubectl apply -f https://raw.githubusercontent.com/open-policy-agent/gatekeeper/master/deploy/gatekeeper.yaml --validate=false

Note: You might encounter an error related to OpenAPI validation. Adding --validate=false bypasses this issue.

Step 3: Verifying the Installation

Check if the Gatekeeper pods are running:

kubectl get pods -n gatekeeper-system

You should see something like:

NAME                                             READY   STATUS    RESTARTS   AGE
gatekeeper-controller-manager-xxxxxxxxxx-xxxxx   1/1     Running   0          xxm

Step 4: Creating Custom Policies

Policy 1: Deny Privileged Pods

Create a policy to prevent the deployment of privileged pods.

File: policies/deny-privileged-pods.yaml

apiVersion: constraints.gatekeeper.sh/v1beta1
kind: K8sDenyPrivilegedContainer
metadata:
  name: deny-privileged-pods
spec:
  match:
    kinds:
      - apiGroups: [""]
        kinds: ["Pod"]

Apply the constraint template and the constraint:

kubectl apply -f policies/deny-privileged-pods.yaml

Policy 2: Require Labels on Resources

Create a policy that enforces the presence of specific labels on all resources.

File: policies/require-labels.yaml

apiVersion: constraints.gatekeeper.sh/v1beta1
kind: K8sRequiredLabels
metadata:
  name: require-app-label
spec:
  match:
    kinds:
      - apiGroups: [""]
        kinds: ["Pod", "Deployment", "Service"]
  parameters:
    labels: ["app"]

Apply the policy:

kubectl apply -f policies/require-labels.yaml

Step 5: Testing the Policies

Testing Deny Privileged Pods Policy

Try deploying a privileged pod.

File: policies/sample-app.yaml

apiVersion: v1
kind: Pod
metadata:
  name: privileged-pod
spec:
  containers:
    - name: nginx
      image: nginx
      securityContext:
        privileged: true

Apply the sample app:

kubectl apply -f policies/sample-app.yaml

Expected Outcome:

Error from server ([denied by deny-privileged-pods] Privileged containers are not allowed): error when creating "policies/sample-app.yaml": admission webhook "validation.gatekeeper.sh" denied the request: [denied by deny-privileged-pods] Privileged containers are not allowed

Testing Require Labels Policy

Important: The require-labels.yaml policy is set up to target Pods, Deployments, and Services. Make sure to test the policy with each of these resource types.

1. Testing with a Pod:

• File: policies/sample-app.yaml (already defined as a Pod)

• Remove the app label:

    apiVersion: v1
    kind: Pod
    metadata:
      name: sample-app
    spec:
      containers:
      - name: nginx
        image: nginx
  

• Apply the Pod:

    kubectl apply -f policies/sample-app.yaml
  

• Expected Outcome:

    Error from server ([denied by require-app-label] Missing required labels: app): error when creating "policies/sample-app.yaml": admission webhook "validation.gatekeeper.sh" denied the request: [denied by require-app-label] Missing required labels: app
  

2. Testing with a Deployment:

• File: policies/sample-app.yaml

• Change the kind to Deployment:

    apiVersion: apps/v1
    kind: Deployment
    metadata:
      name: sample-app
    spec:
      replicas: 1
      selector:
        matchLabels:
          app: sample-app
      template:
        metadata:
          labels:
            app: sample-app
        spec:
          containers:
          - name: nginx
            image: nginx
  

• Remove the app label from the template.metadata.labels section:

    apiVersion: apps/v1
    kind: Deployment
    metadata:
      name: sample-app
    spec:
      replicas: 1
      selector:
        matchLabels:
          app: sample-app
      template:
        metadata:
          # labels:  # Remove this line and its content
          #   app: sample-app
        spec:
          containers:
          - name: nginx
            image: nginx
  

• Apply the Deployment:

    kubectl apply -f policies/sample-app.yaml
  

• Expected Outcome:

    Error from server ([denied by require-app-label] Missing required labels: app): error when creating "policies/sample-app.yaml": admission webhook "validation.gatekeeper.sh" denied the request: [denied by require-app-label] Missing required labels: app
  

3. Testing with a Service:

• Create a new file: policies/sample-service.yaml

• Define the Service:

    apiVersion: v1
    kind: Service
    metadata:
      name: sample-service
    spec:
      selector:
        app: sample-app
      ports:
      - protocol: TCP
        port: 80
        targetPort: 80
  

• Remove the app label from the selector section:

    apiVersion: v1
    kind: Service
    metadata:
      name: sample-service
    spec:
      # selector:
      #   app: sample-app
      ports:
      - protocol: TCP
        port: 80
        targetPort: 80
  

• Apply the Service:

    kubectl apply -f policies/sample-service.yaml
  

• Expected Outcome:

    Error from server ([denied by require-app-label] Missing required labels: app): error when creating "policies/sample-service.yaml": admission webhook "validation.gatekeeper.sh" denied the request: [denied by require-app-label] Missing required labels: app
  

Common Issues and Troubleshooting

Error: OpenAPI Validation Failed

Problem:

error validating "https://.../gatekeeper.yaml": error validating data: failed to download openapi...

Solution:

Add the --validate=false flag when applying the manifest:

kubectl apply -f https://.../gatekeeper.yaml --validate=false

Gatekeeper Pods Not Running

Problem:

Gatekeeper pods are not in the Running state.

Solution:

Check the pod logs for errors:

kubectl logs -n gatekeeper-system <pod-name>

Ensure your Kubernetes cluster meets the minimum requirements.

Policies Not Enforced

Problem:

Resources violating policies are still being created.

Solution:

• Ensure that the constraint templates and constraints are properly applied.

• Verify that the match section in your constraint YAML files correctly targets the resources.

Conclusion

By integrating OPA and Gatekeeper into your Kubernetes cluster, you've taken a significant step toward enhancing your deployment's security and compliance. Policies help prevent misconfigurations and enforce best practices automatically, reducing the risk of human error.

Feel free to leave comments or questions below. Happy DevOps-ing , See you on day 30 !