CRDs

Custom Resource Definitions (CRDs) are a powerful feature in Kubernetes that allows users to extend the Kubernetes API by defining new resource types. These new resource types, known as Custom Resources (CRs), enable you to create, manage, and automate the lifecycle of complex applications or infrastructure components in a Kubernetes-native way. CRDs are a key part of Kubernetes’ extensibility, allowing you to tailor the platform to your specific needs without modifying the core Kubernetes codebase.

What are Custom Resource Definitions (CRDs)?

A Custom Resource Definition (CRD) is a Kubernetes API extension that allows you to define your own custom resources. Once a CRD is installed in the cluster, users can create and manage instances of these custom resources just like they would with built-in Kubernetes resources such as Pods, Deployments, and Services. CRDs make it possible to encapsulate complex application-specific logic, configuration, and operational procedures in a Kubernetes-native manner.

Core Responsibilities of CRDs

Extending the Kubernetes API:

CRDs allow you to extend the Kubernetes API with new resource types that are specific to your application or infrastructure needs. This means you can manage these custom resources using standard Kubernetes tools and patterns, such as kubectl, YAML manifests, and RBAC.

Encapsulating Application Logic:

CRDs are often used in conjunction with custom controllers or operators to encapsulate application-specific logic. This enables the automation of complex workflows, such as managing stateful applications, handling backups, or automating rolling updates.

Enabling Kubernetes-Native Management:

By defining custom resources, you can manage non-native Kubernetes objects in a Kubernetes-native way. For example, you might define a custom resource for managing a database, where each instance of the resource represents a database cluster.

Supporting Declarative Configuration:

Like other Kubernetes resources, custom resources defined by CRDs are managed declaratively. You describe the desired state of the custom resource in a YAML file, and Kubernetes ensures that the actual state of the resource matches the desired state.

Facilitating Multi-Tenancy:

CRDs can help support multi-tenancy by enabling the creation of custom resources that encapsulate tenant-specific configurations, policies, or quotas, allowing for easier management and isolation within the cluster.

How CRDs Work in Kubernetes

When you create a CRD, you define a new API endpoint that Kubernetes will expose. Once the CRD is applied to the cluster, Kubernetes generates a new RESTful API path for the custom resource type. Users can then create, read, update, and delete instances of this custom resource type using standard Kubernetes tools and API calls.

Example: Defining and Using a CRD

Let’s walk through an example of creating a CRD for a custom resource called MySQLCluster, which represents a MySQL database cluster:

Define the CRD:

You start by defining the CRD itself, specifying the group, version, and kind for the custom resource, as well as the schema for the resource’s spec and status fields.

   apiVersion: apiextensions.k8s.io/v1
   kind: CustomResourceDefinition
   metadata:
     name: mysqlclusters.example.com
   spec:
     group: example.com
     versions:
       - name: v1
         served: true
         storage: true
         schema:
           openAPIV3Schema:
             type: object
             properties:
               spec:
                 type: object
                 properties:
                   replicas:
                     type: integer
                   version:
                     type: string
     scope: Namespaced
     names:
       plural: mysqlclusters
       singular: mysqlcluster
       kind: MySQLCluster
       shortNames:
       - myc

Apply the CRD:

You apply the CRD to the Kubernetes cluster using kubectl:

   kubectl apply -f mysqlcluster-crd.yaml

This command creates a new API endpoint in Kubernetes, such as /apis/example.com/v1/namespaces/<namespace>/mysqlclusters.

Create a Custom Resource:

Once the CRD is in place, you can create instances of the custom resource. Here’s an example YAML for creating a MySQLCluster resource:

   apiVersion: example.com/v1
   kind: MySQLCluster
   metadata:
     name: my-mysql-cluster
   spec:
     replicas: 3
     version: "5.7"

Apply this resource with:

   kubectl apply -f my-mysql-cluster.yaml

Managing the Custom Resource:

You can now manage the custom resource just like any other Kubernetes resource:

   kubectl get mysqlclusters
   kubectl describe mysqlcluster my-mysql-cluster
   kubectl delete mysqlcluster my-mysql-cluster

Custom Controllers and Operators

CRDs are often used in conjunction with custom controllers or operators:

Custom Controllers: These are Kubernetes controllers that you write to watch for changes to custom resources and take action based on the desired state specified in the resource. For example, a controller might watch for changes to a MySQLCluster resource and automatically create or scale MySQL Pods accordingly.
Operators: An Operator is a more sophisticated version of a controller that encapsulates human operational knowledge in software. It can manage the entire lifecycle of an application, including deployment, scaling, upgrades, and backups. Operators use CRDs to define the application’s configuration and operational procedures as Kubernetes resources.

Key Components and Concepts Related to CRDs

API Group and Versioning:

When defining a CRD, you specify an API group (e.g., example.com) and version (e.g., v1). This allows you to organize your custom resources and manage versioning over time as your resource definitions evolve.

Validation and Schema Enforcement:

CRDs support validation through OpenAPI v3 schemas. This allows you to enforce strict validation rules on the spec and status fields of your custom resources, ensuring that only valid configurations are accepted.

Subresources (Status and Scale):

CRDs can define subresources for status and scale. The status subresource is used to store the current state of the resource, while the scale subresource allows you to implement scaling behavior (e.g., adjusting the number of replicas).

Custom Resource Definitions (CRD) vs. API Aggregation:

CRDs are not the only way to extend the Kubernetes API. API Aggregation allows you to serve custom APIs that are fully integrated into the Kubernetes API server. While CRDs are easier to use and deploy, API Aggregation provides more flexibility and control over the API behavior.

Namespaced vs. Cluster-wide Resources:

CRDs can define resources that are either namespaced (scoped to a specific namespace) or cluster-wide (available across the entire cluster). The choice depends on whether the custom resource needs to be isolated by namespace or managed at the cluster level.

Security Considerations for CRDs

RBAC (Role-Based Access Control): When deploying CRDs, ensure that proper RBAC policies are in place to control who can create, modify, or delete custom resources. This is crucial for maintaining the security and integrity of the custom resources.
Webhook Validation and Mutation: CRDs can be paired with admission webhooks for advanced validation and mutation logic, allowing you to enforce custom business rules or modify resource specifications before they are persisted.

High Availability and Scalability

Controller/Operator Availability: Ensure that custom controllers or operators managing CRDs are highly available and can handle the scale of resources in the cluster. This typically involves deploying them as highly available deployments with multiple replicas.
CRD Scalability: CRDs should be designed to handle large numbers of custom resources efficiently. Consider the performance implications of watching and managing many instances of custom resources.

Performance Considerations

API Server Load: Introducing many custom resources or controllers can increase the load on the Kubernetes API server. Monitor the performance and ensure that the API server can handle the additional requests.
Event Processing Latency: Custom controllers should be optimized to process events quickly and efficiently, minimizing the time between detecting a change in a custom resource and taking the necessary actions.

Summary

Custom Resource Definitions (CRDs) are a powerful feature in Kubernetes that allow you to extend the Kubernetes API with custom resource types. CRDs enable Kubernetes-native management of complex applications and infrastructure components, providing a declarative and standardized way to encapsulate application-specific logic. By defining CRDs and implementing custom controllers or operators, you can automate complex workflows, enforce validation rules, and integrate non-native Kubernetes objects into your Kubernetes environment. Understanding and leveraging CRDs is essential for building advanced, scalable, and maintainable Kubernetes-based applications.