Day 29: Strengthening Kubernetes Security with Open Policy Agent (OPA) and Gatekeeper

Welcome back to Day 29 of my 90-day DevOps journey! Today, we're diving into Kubernetes security by exploring how to use Open Policy Agent (OPA) and Gatekeeper to enforce policies within your cluster.

This guide is designed for beginners and will walk you through setting up OPA with Gatekeeper, creating custom policies, and handling common issues you might encounter along the way.

Introduction to OPA and Gatekeeper
Project Prerequisites
Folder Structure
Step-by-Step Guide
Common Issues and Troubleshooting
Conclusion

Introduction to OPA and Gatekeeper

Open Policy Agent (OPA) is like a rulebook for your software. It lets you define rules (policies) that can be applied to different parts of your system. In Kubernetes, OPA helps you enforce rules on resources before they're created or changed.

Gatekeeper is a special tool that works with OPA and Kubernetes. It allows you to create and apply policies using Kubernetes' own language (Custom Resource Definitions or CRDs).

By combining OPA and Gatekeeper, you can make sure everything in your Kubernetes cluster follows your security and compliance rules.

Project Prerequisites

A running Kubernetes cluster: You can use Minikube, Docker Desktop, or any managed Kubernetes service.
kubectl installed and configured: This is the command-line tool to interact with your Kubernetes cluster.
Basic understanding of Kubernetes resources: You should know about pods, deployments, services, etc.

Folder Structure

Let's organize our project files for better readability:

90days-devops/
└── day29/
    ├── manifests/
    │   └── gatekeeper.yaml
    ├── policies/
    │   ├── deny-privileged-pods.yaml
    │   ├── require-labels.yaml
    │   └── sample-app.yaml
    └── README.md

manifests/: Contains YAML files for deploying Gatekeeper.
policies/: Holds custom policy definitions and sample applications to test the policies.
README.md: Documentation and steps followed during the project.

Step-by-Step Guide

Step 1: Setting Up the Kubernetes Cluster

If you haven't set up a Kubernetes cluster yet, you can use Minikube:

minikube start
kubectl config use-context minikube

Verify the cluster is running:

kubectl get nodes

Step 2: Installing Gatekeeper

Apply the Gatekeeper manifest to install it on your cluster:

kubectl apply -f https://raw.githubusercontent.com/open-policy-agent/gatekeeper/master/deploy/gatekeeper.yaml --validate=false

Note: You might encounter an error related to OpenAPI validation. Adding --validate=false bypasses this issue.

Step 3: Verifying the Installation

Check if the Gatekeeper pods are running:

kubectl get pods -n gatekeeper-system

You should see something like:

NAME                                             READY   STATUS    RESTARTS   AGE
gatekeeper-controller-manager-xxxxxxxxxx-xxxxx   1/1     Running   0          xxm

Step 4: Creating Custom Policies

Policy 1: Deny Privileged Pods

Create a policy to prevent the deployment of privileged pods.

File: policies/deny-privileged-pods.yaml

apiVersion: constraints.gatekeeper.sh/v1beta1
kind: K8sDenyPrivilegedContainer
metadata:
  name: deny-privileged-pods
spec:
  match:
    kinds:
      - apiGroups: [""]
        kinds: ["Pod"]

Apply the constraint template and the constraint:

kubectl apply -f policies/deny-privileged-pods.yaml

Policy 2: Require Labels on Resources

Create a policy that enforces the presence of specific labels on all resources.

File: policies/require-labels.yaml

apiVersion: constraints.gatekeeper.sh/v1beta1
kind: K8sRequiredLabels
metadata:
  name: require-app-label
spec:
  match:
    kinds:
      - apiGroups: [""]
        kinds: ["Pod", "Deployment", "Service"]
  parameters:
    labels: ["app"]

Apply the policy:

kubectl apply -f policies/require-labels.yaml

Step 5: Testing the Policies

Testing Deny Privileged Pods Policy

Try deploying a privileged pod.

File: policies/sample-app.yaml

apiVersion: v1
kind: Pod
metadata:
  name: privileged-pod
spec:
  containers:
    - name: nginx
      image: nginx
      securityContext:
        privileged: true

Apply the sample app:

kubectl apply -f policies/sample-app.yaml

Expected Outcome:

Error from server ([denied by deny-privileged-pods] Privileged containers are not allowed): error when creating "policies/sample-app.yaml": admission webhook "validation.gatekeeper.sh" denied the request: [denied by deny-privileged-pods] Privileged containers are not allowed

Testing Require Labels Policy

Important: The require-labels.yaml policy is set up to target Pods, Deployments, and Services. Make sure to test the policy with each of these resource types.

1. Testing with a Pod:

File: policies/sample-app.yaml (already defined as a Pod)

Remove the app label:

  apiVersion: v1
  kind: Pod
  metadata:
    name: sample-app
  spec:
    containers:
    - name: nginx
      image: nginx

Apply the Pod:

  kubectl apply -f policies/sample-app.yaml

Expected Outcome:

  Error from server ([denied by require-app-label] Missing required labels: app): error when creating "policies/sample-app.yaml": admission webhook "validation.gatekeeper.sh" denied the request: [denied by require-app-label] Missing required labels: app

2. Testing with a Deployment:

File: policies/sample-app.yaml

Change the kind to Deployment:

  apiVersion: apps/v1
  kind: Deployment
  metadata:
    name: sample-app
  spec:
    replicas: 1
    selector:
      matchLabels:
        app: sample-app
    template:
      metadata:
        labels:
          app: sample-app
      spec:
        containers:
        - name: nginx
          image: nginx

Remove the app label from the template.metadata.labels section:

  apiVersion: apps/v1
  kind: Deployment
  metadata:
    name: sample-app
  spec:
    replicas: 1
    selector:
      matchLabels:
        app: sample-app
    template:
      metadata:
        # labels:  # Remove this line and its content
        #   app: sample-app
      spec:
        containers:
        - name: nginx
          image: nginx

Apply the Deployment:

  kubectl apply -f policies/sample-app.yaml

Expected Outcome:

  Error from server ([denied by require-app-label] Missing required labels: app): error when creating "policies/sample-app.yaml": admission webhook "validation.gatekeeper.sh" denied the request: [denied by require-app-label] Missing required labels: app

3. Testing with a Service:

Create a new file: policies/sample-service.yaml

Define the Service:

  apiVersion: v1
  kind: Service
  metadata:
    name: sample-service
  spec:
    selector:
      app: sample-app
    ports:
    - protocol: TCP
      port: 80
      targetPort: 80

Remove the app label from the selector section:

  apiVersion: v1
  kind: Service
  metadata:
    name: sample-service
  spec:
    # selector:
    #   app: sample-app
    ports:
    - protocol: TCP
      port: 80
      targetPort: 80

Apply the Service:

  kubectl apply -f policies/sample-service.yaml

Expected Outcome:

  Error from server ([denied by require-app-label] Missing required labels: app): error when creating "policies/sample-service.yaml": admission webhook "validation.gatekeeper.sh" denied the request: [denied by require-app-label] Missing required labels: app

Common Issues and Troubleshooting

Error: OpenAPI Validation Failed

Problem:

error validating "https://.../gatekeeper.yaml": error validating data: failed to download openapi...

Solution:

Add the --validate=false flag when applying the manifest:

kubectl apply -f https://.../gatekeeper.yaml --validate=false

Gatekeeper Pods Not Running

Problem:

Gatekeeper pods are not in the Running state.

Solution:

Check the pod logs for errors:

kubectl logs -n gatekeeper-system <pod-name>

Ensure your Kubernetes cluster meets the minimum requirements.

Policies Not Enforced

Problem:

Resources violating policies are still being created.

Solution:

Ensure that the constraint templates and constraints are properly applied.
Verify that the match section in your constraint YAML files correctly targets the resources.

Conclusion

By integrating OPA and Gatekeeper into your Kubernetes cluster, you've taken a significant step toward enhancing your deployment's security and compliance. Policies help prevent misconfigurations and enforce best practices automatically, reducing the risk of human error.

Feel free to leave comments or questions below. Happy DevOps-ing , See you on day 30 !

Day 29: Strengthening Kubernetes Security with Open Policy Agent (OPA) and Gatekeeper

Table of Contents

Introduction to OPA and Gatekeeper

Project Prerequisites

Folder Structure

Step-by-Step Guide

Step 1: Setting Up the Kubernetes Cluster

Step 2: Installing Gatekeeper

Step 3: Verifying the Installation

Step 4: Creating Custom Policies

Policy 1: Deny Privileged Pods

Policy 2: Require Labels on Resources

Step 5: Testing the Policies

Testing Deny Privileged Pods Policy

Testing Require Labels Policy

Common Issues and Troubleshooting

Error: OpenAPI Validation Failed

Gatekeeper Pods Not Running

Policies Not Enforced

Conclusion