K8Guard is Officially Open Source

I am happy to announce that Target has open sourced K8Guard. I have been part of designing and developing it for the past few months, and I’m going to share a little more about it.

What is K8Guard?

K8Guard is an auditing system for Kubernetes clusters. It monitors different entities on your cluster for possible violations. K8Guard notifies the violators and then takes action on them. It also provides metrics and dashboards about violations in the cluster through Prometheus.

How to Pronounce It?

Like Kate Guard - the guardian angel for your Kubernetes clusters.

Why?

If you have large size kuberentes clusters and you care about security, efficiency, availability and stability, you need a tool to detect violations and do appropriate actions on them.

What Kind of Violations Does It Discover?

What Kind of Entities Does It Monitor?

Any entities which deployed to your kubernetes cluster such as Deployments, Pods, Jobs/CronJobs, Ingresses and namespaces.

What Kind of Actions Does It Take?

• Notifies the namespace owner (email, hipchat, …).
• After X amount of notifications, it will do a hard action such as:
  • Scale bad deployments down to zero.
  • Suspend bad jobs.
  • Delete bad ingress

Note that there is a safe mode - which only notifies and does not do hard actions.

The K8Guard Design

K8Guard has 3 main microservices (discover, action, report)

• Discover service, when in messaging mode, finds violations and puts them on a kafka topic. and also discover API mode, is able to serve without depending on kafka. you can hit the end points to get JSON response.

• Action service reads the violations off kafka and does action on them and records the actions in a database (Cassandra).

The same message will be sent to hipchat and tag the violators.

• Report service will generate a human readable and searchable report of all the past violation actions.

Integration

K8Guard discover service has an API mode that you can use to integrate with other apps.

Example API Response:

[{
    Name: "dummy-deployment-name1",
    Namespace: "dummy-namespace-1",
    Cluster: "dummy-cluster-name1",
    Violations: [{
      Source: "dummy-repo/dummy-image:latest",
      Type: "IMAGE_REPO"
    }]
  },
  {
    Name: "ye-another-dummy-deployment",
    Namespace: "dummy-namespace-1",
    Cluster: "dummy-cluster-name1",
    Violations: [{
      Source: "another-dummy-repo/dummy-image2:latest",
      Type: "IMAGE_SIZE"
    }]
  }
]

Metrics

Discover API generates 19 metrics which are accessible at /metrics, and you can hook a monitoring system like Prometheus to collect them and then generate pretty Grafana dashboards.

Violation metric examples:

• The number of all deployments.
• The number of bad deployments.
• The number of all pods.
• The number of bad pods.

It also collects performance metrics for free. By free I mean, while K8Guard makes the api calls, it also measures how long it took, and makes it available as metrics.

Performance metrics examples:

• The number of seconds took to return all images from Kubernetes api.
• The number of seconds took to return all deployments from Kubernetes api.

Configurable

There are tens of configurations to customize what violations to discover and what to do with them. it also ships with a safe mode, that will only notify and won’t do any hard action.

What Tools and Technologies are Used?

• Golang
• Kafka
• Cassandra
• Prometheus
• Memcached

Give It a Try, It Is Developer Friendly!

Batteries -are- included! You can easily run K8Guard locally on your computer and play with it. You can run all of the K8Guard system in either Minikube or docker-compose (your choice).

We have made easy Make commands for K8Guard. Don’t be shy if you see any issues or if you would like to contribute to the code. please do.

The best place to start is k8guard github page .

About the Author

Medya Ghazizadeh is a Senior Engineer and part of Target’s Cloud Platform Engineering team. He is the author of multiple open source projects such as Winnaker and K8Guard.