Intro

When I recently decided to port a “non-containerized” application onto Kubernetes, I struggled to find a solid approach to manage its lifecycle (deploy, upgrade, rollback, resize, etc.). It seems that most solutions involve delegating logic to a human or some custom scripts that need to be developed for every application.

This, of course, is far from ideal. Thankfully, CoreOS came up with a solution called Operators.

What is an Operator?

An Operator is a concept that literally replaces boring system administration tasks. An Operator does the lifecycle management of an application running on k8s. A more abstract definition: an Operator is an application aware k8s object that can be implemented using Helm, Go or Ansible (though a Helm Operator cannot manage the entire application lifecycle [read more]).

Routes or Services, for example, are k8s built-in resources. In order to enable k8s to understand your application, you have to create a custom resource and tell k8s how to interpret it using a custom resource definition.

An Operator of your application compares desired state (as described in config files) with the current state and reconciles if necessary.

If choosing an Ansible-based Operator, playbooks, or roles respectively, take the action to reconcile. Typically, a custom resource event triggers an Ansible task.

A considerable advantage of an Operator is, that it can be version controlled.

An Example

An example implementation of an Operator includes the following steps:

  1. RBAC: Define role-based access controls.
  2. CRD: Create a custom resource definition.
  3. DC: Write a deployment config.
  4. CR: Write a custom resource config.

Role-based Access Control

Create a service account:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: example-operator

Create role:

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  creationTimestamp: null
  name: example-operator
rules:
- apiGroups:
...

Create role binding:

kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: example-operator
subjects:
- kind: ServiceAccount
  name: example-operator
roleRef:
  kind: Role
  name: example-operator
  apiGroup: rbac.authorization.k8s.io

Custom Resource Definition

Create a custom resource definition that tells k8s about your application:

apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  name: exampleappservices.app.example.com
spec:
  group: app.example.com
  names:
    kind: ExampleAppService
    listKind: ExampleAppServiceList
    plural: exampleappservices
    singular: exampleappservice
  scope: Namespaced
  subresources:
    status: {}
  versions:
  - name: v1alpha1
    served: true
    storage: true

Deployment Config

Create a deployment config:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: example-operator
spec:
  replicas: 1
  selector:
    matchLabels:
      name: example-operator
  template:
    metadata:
      labels:
        name: example-operator
    spec:
      serviceAccountName: example-operator
      containers:
        - name: ansible
          command:
          - /usr/local/bin/ao-logs
          - /tmp/ansible-operator/runner
          - stdout
          # Replace this with the built image name
          image: "{{ REPLACE_IMAGE }}"
          imagePullPolicy: "Always"
          volumeMounts:
          - mountPath: /tmp/ansible-operator/runner
            name: runner
            readOnly: true
        - name: operator
          # Replace this with the built image name
          image: "{{ REPLACE_IMAGE }}"
          imagePullPolicy: "Always"
          volumeMounts:
          - mountPath: /tmp/ansible-operator/runner
            name: runner
          env:
            - name: WATCH_NAMESPACE
              valueFrom:
                fieldRef:
                  fieldPath: metadata.namespace
            - name: POD_NAME
              valueFrom:
                fieldRef:
                  fieldPath: metadata.name
            - name: OPERATOR_NAME
              value: "example-operator"
            - name: ANSIBLE_GATHERING
              value: explicit
      volumes:
        - name: runner
          emptyDir: {}

Custom Resource

Create a custom resource:

apiVersion: app.example.com/v1alpha1
kind: ExampleAppService
metadata:
  name: example-exampleappservice
spec:
  # Add fields here
  size: 3

As you can see, creating an Operator involves an acceptable amount of work. And it gets even better thanks to the Operator SDK which creates all of the required configs for you.

What is the Operator SDK?

The Operator SDK is a framework that provides high level APIs and abstractions to write operational logic more intuitively. Further, it provides useful tools for bootstrapping the implementation of an Operator.

As already mentioned, an Operator can be implemented using Helm, Go or Ansible. In this post, I am focusing on an Ansible-based Operator, because it allows to implement the entire lifecycle and does not require a single line of code.

Steps to build an Operator

After installation of the Operator SDL, follow these steps (tested with Operator SDK v0.12.0):

  1. Create Operator skeleton
    operator-sdk new \
--type=ansible --kind ExampleAppService --generate-playbook \
--api-version app.example.com/v1alpha1 example-operator
    As a result, the Operator SDK generates a couple of files:
    $ tree example-operator/
example-operator/
├── build
│   ├── Dockerfile
│   └── test-framework
│       ├── ansible-test.sh
│       └── Dockerfile
├── deploy
│   ├── crds
│   │   ├── app.example.com_exampleappservices_crd.yaml
│   │   └── app.example.com_v1alpha1_exampleappservice_cr.yaml
│   ├── operator.yaml
│   ├── role_binding.yaml
│   ├── role.yaml
│   └── service_account.yaml
├── molecule # Ansible testing framework
│   └── ...
├── playbook.yml
├── roles
│   └── exampleappservice
│       ├── defaults
│       │   └── main.yml
│       ├── files
│       ├── handlers
│       │   └── main.yml
│       ├── meta
│       │   └── main.yml
│       ├── README.md
│       ├── tasks
│       │   └── main.yml
│       ├── templates
│       └── vars
│           └── main.yml
└── watches.yaml
  2. Add Ansible tasks and configure the Watches file (for test purposes, you can simply leave it as it is).
  3. Build Operator
    • Create and tag container image:
      cd example-operator/ && \
operator-sdk build \
registry.com/example/example-operator:v0.0.1 && cd ../
    • Push container image:
      docker push registry.com/example/example-operator:v0.0.1
    • In example-operator/deploy/operator.yaml, set image placeholder REPLACE_IMAGE to the previously-built image:
      sed -i 's|{{ REPLACE_IMAGE }}|registry.com/example/example-operator:v0.0.1|g' \
example-operator/deploy/operator.yaml
  4. Deploy Operator
    • Create CRD:
      kubectl create -f \
example-operator/deploy/crds/app.example.com_exampleappservices_crd.yaml
    • Create service account:
      kubectl create -f example-operator/deploy/service_account.yaml`
    • Create role:
      kubectl create -f example-operator/deploy/role.yaml
    • Create role binding:
      kubectl create -f example-operator/deploy/role_binding.yaml
    • Create Operator deployment object:
      kubectl create -f example-operator/deploy/operator.yaml
      • Set imagePullPolicy to Always. For some reason, the variable w/ default value does not work, but that’s another story.
    • Create an instance of your app:
      kubectl create -f \
example-operator/deploy/crds/app.example.com_v1alpha1_exampleappservice_cr.yaml
  5. Check Status
    • Get deployment overview:
      kubectl get deployment
  6. Destroy Operator
    • Delete the app:
      kubectl delete -f \
example-operator/deploy/crds/app.example.com_v1alpha1_exampleappservice_cr.yaml
    • Delete the operator:
      kubectl delete -f example-operator/deploy/operator.yaml

It is straight forward and does the heavy lifting for you. The Watches file (watches.yaml) defines for what k8s events the Operator has to watch and holds the mapping between events and the Ansible roles that they trigger.

It is crucial that the Ansible tasks are idempotent. These tasks will be executed frequently by the Operator, thus the result should always be the same.

Read more about the Operator Framework in combination with Ansible here: official docs.

You can find a list of pre-built operators here: https://operatorhub.io/.

Sources

Most of the information in this post is based on the following sources: