Structure

Kubernetes (K8s) has a structured approach to managing infrastructure and applications through manifests and objects. Here’s how these components fit together:

Kubernetes Structure Overview

  1. Manifests:
  • Definition: Manifests are YAML or JSON files that describe the desired state of Kubernetes objects. They act as the blueprint or specification for what the Kubernetes objects should look like and how they should behave.
  • Role: Manifests are used to create, update, or delete Kubernetes objects. They define the properties and configuration of objects, such as the number of replicas for a Deployment, the image version for a container, or the network settings for a Service.
  • Content: Each manifest file typically includes sections like apiVersion, kind, metadata, and spec, which collectively describe the Kubernetes object.
  1. Kubernetes Objects:
  • Definition: Kubernetes objects are persistent entities that represent the state of your cluster. They encapsulate everything about the cluster’s desired state, from which applications are running, how they are networked, to what storage they use.
  • Types of Objects: Kubernetes has a variety of built-in objects, such as Pods, Services, Deployments, ConfigMaps, Secrets, and more. Each object serves a specific purpose within the cluster.

How Manifests and Objects Work Together

  1. Creation:
  • When you apply a manifest to the Kubernetes cluster (using kubectl apply -f my-manifest.yaml), Kubernetes reads the desired state described in the manifest and creates or updates the corresponding objects in the cluster.
  • For example, a Deployment manifest might specify a Deployment object that defines a number of replicas of a specific Pod.
  1. Desired State:
  • The manifest defines the desired state of the objects, while Kubernetes continuously monitors and manages the actual state to ensure it matches the desired state.
  • If there’s a difference between the desired state (as defined in the manifest) and the actual state (as observed in the cluster), Kubernetes takes corrective actions to reconcile the two.
  1. Declarative vs. Imperative:
  • Declarative: Manifests represent a declarative approach, where you describe what you want (the end state), and Kubernetes figures out how to achieve it.
  • Imperative: While not typically preferred for long-term management, Kubernetes also supports imperative commands (like kubectl create pod ...) that create objects directly without using manifests.

Kubernetes Object Structure in Manifests

Each Kubernetes object defined in a manifest follows a structured format:

  1. apiVersion:
  • Specifies the version of the Kubernetes API that you’re using to create this object.
  • Example: apiVersion: apps/v1
  1. kind:
  • Indicates the type of Kubernetes object you’re defining.
  • Example: kind: Deployment
  1. metadata:
  • Contains data to identify the object, such as name, namespace, labels, and annotations.
  • Example:
    yaml metadata: name: my-deployment namespace: default labels: app: myapp
  1. spec:
  • Defines the desired state of the object. The content of this section varies depending on the object type.
  • Example (for a Deployment):
    yaml spec: replicas: 3 selector: matchLabels: app: myapp template: metadata: labels: app: myapp spec: containers: - name: nginx image: nginx:1.14.2 ports: - containerPort: 80

Kubernetes Controller Loop

Kubernetes uses a control loop mechanism to manage the state of objects:

  1. Controllers: Kubernetes has built-in controllers for managing different types of objects. Each controller continuously monitors the current state of objects and compares it to the desired state as defined in the manifests.
  2. Reconciliation: If the actual state does not match the desired state (e.g., a Pod crashes or a node goes down), the controller will take corrective action to bring the actual state in line with the desired state. This might involve restarting Pods, rescheduling them on different nodes, or scaling up/down the number of replicas.
  3. Self-Healing: Kubernetes is designed to be resilient and self-healing. When something goes wrong (e.g., a Pod crashes), Kubernetes uses the information in the manifests to restore the cluster to its desired state.

Kubernetes Structure in Practice

  • Namespaces: Kubernetes uses namespaces to provide a scope for resources, helping to divide and organize objects within a cluster. Manifests can specify the namespace for objects.
  • Labels and Selectors: Labels are key-value pairs attached to objects, used for organizing and selecting subsets of objects. Selectors are used in manifests to match objects with specific labels.
  • Annotations: Annotations are used to attach arbitrary non-identifying metadata to objects.

Summary

In Kubernetes, manifests are the declarative specifications that define the desired state of various objects in the cluster. These objects represent different resources and services within the cluster, and Kubernetes ensures that the actual state of the cluster matches the desired state as described in these manifests. This architecture allows Kubernetes to maintain, scale, and heal applications automatically, providing a powerful and flexible platform for managing containerized applications at scale.

_________________________________________________________________________________

Complete overview:

To understand how Kubernetes operates as a unified system, it’s crucial to see how all these components interact and work together to maintain the desired state of the cluster. Below is a detailed overview of how these components—manifests, objects, controllers, scheduler, API server, etcd, Kubelet, kube-proxy, container runtime, cluster add-ons, CRDs, and admission controllers—interact and the logic that drives Kubernetes from a low-level perspective.

1. Manifests and Objects

Manifests are YAML or JSON files that define the desired state of Kubernetes objects (such as Pods, Deployments, Services, etc.). When a manifest is applied to the cluster using kubectl, it represents a request to the Kubernetes API to create or update objects that conform to the configuration defined in the manifest.

  • Logic Flow:
  • The user writes a manifest defining the desired state of one or more Kubernetes objects.
  • The manifest is applied to the cluster using kubectl apply -f <manifest-file>.yaml.
  • kubectl sends the request to the Kubernetes API server.

2. API Server

The API server is the central management component of Kubernetes. It acts as the gateway for all requests to the Kubernetes cluster and serves as the communication hub for all other components.

  • Key Responsibilities:
  • Authentication: Ensures that the user or service making the request is authenticated.
  • Authorization: Verifies whether the authenticated user has permission to perform the requested action on the specified object.
  • Admission Control: Runs admission controllers to enforce policies, mutate resources, or validate requests before they are persisted.
  • etcd Interaction: Stores and retrieves the current state of the cluster in etcd.
  • Logic Flow:
  • The API server receives the request (e.g., creating a Pod) from kubectl.
  • It authenticates and authorizes the request.
  • It passes the request through admission controllers for validation and mutation.
  • If the request passes all checks, the API server writes the object’s desired state to etcd.

3. etcd

etcd is a distributed key-value store that holds all the persistent data of the Kubernetes cluster, such as configurations, states, and secrets.

  • Key Responsibilities:
  • Data Storage: Stores the desired and current state of all Kubernetes objects.
  • Consistency: Uses the Raft consensus algorithm to ensure strong consistency across the distributed nodes of the etcd cluster.
  • Service Discovery: Helps components like the API server and controllers discover and synchronize the state of resources.
  • Logic Flow:
  • When the API server writes a resource (e.g., a Pod) to etcd, it persists this data.
  • etcd replicates this data across its cluster to ensure availability and consistency.
  • When a controller, scheduler, or kubelet needs to know the state of the cluster, it queries the API server, which fetches the state from etcd.

4. Controllers

Controllers are control loops that continuously watch the state of the cluster via the API server and work to make the actual state of resources match the desired state defined by the user (through manifests).

  • Key Responsibilities:
  • Reconciliation: Constantly compare the current state of objects (as recorded in etcd) to the desired state and take corrective actions to reconcile differences.
  • Object Management: Manage the lifecycle of Kubernetes resources like Deployments, ReplicaSets, and Jobs.
  • Logic Flow:
  • A controller (e.g., the Deployment controller) watches the API server for changes to the resources it manages.
  • When it detects a change (e.g., a new Deployment is created), the controller ensures that the appropriate number of Pods is running to match the desired state.
  • It interacts with the API server to create, update, or delete objects as needed, and these changes are recorded in etcd.

5. Scheduler

The scheduler is responsible for placing newly created Pods onto appropriate Nodes in the cluster.

  • Key Responsibilities:
  • Node Selection: Selects the best Node for a Pod based on factors like resource availability, affinity/anti-affinity rules, taints, and tolerations.
  • Workload Distribution: Ensures that workloads are balanced across the cluster and that resource constraints are respected.
  • Logic Flow:
  • When a new Pod is created (e.g., by a Deployment controller), the Pod object is recorded in etcd with a “Pending” status.
  • The scheduler watches for Pods in the “Pending” state and selects an appropriate Node for each Pod.
  • The scheduler updates the Pod’s specification in etcd to reflect the assigned Node.

6. Kubelet

The Kubelet is the agent that runs on every Node in the cluster. It ensures that the containers described in the PodSpecs are running and healthy.

  • Key Responsibilities:
  • Pod Lifecycle Management: Ensures that the containers in each Pod are started, running, and healthy according to the PodSpec.
  • Node Status Reporting: Reports the status of the Node and its Pods back to the API server.
  • Interfacing with the Container Runtime: Communicates with the container runtime to manage the lifecycle of containers.
  • Logic Flow:
  • The Kubelet watches the API server for Pods that have been scheduled to its Node.
  • It retrieves the PodSpec from the API server and interacts with the container runtime (e.g., Docker, containerd) to start the containers.
  • The Kubelet monitors the running containers and reports their status back to the API server.
  • If a container fails, the Kubelet attempts to restart it according to the Pod’s restart policy.

7. Container Runtime

The container runtime is the software responsible for running the containers on each Node. Examples include Docker, containerd, and CRI-O.

  • Key Responsibilities:
  • Image Management: Pulls container images from registries.
  • Container Lifecycle: Creates, starts, stops, and deletes containers as instructed by the Kubelet.
  • Resource Isolation: Uses Linux kernel features like cgroups and namespaces to isolate resources for each container.
  • Logic Flow:
  • The Kubelet instructs the container runtime to pull the necessary images and start containers based on the PodSpec.
  • The container runtime creates the container environment (network, filesystem, etc.) and starts the container process.
  • The container runtime monitors the container’s state and reports status information back to the Kubelet.

8. kube-proxy

kube-proxy is a network proxy that runs on each Node and is responsible for managing network rules that allow communication to and from Pods.

  • Key Responsibilities:
  • Service Networking: Manages the network rules to route traffic to the appropriate Pod for a given Service.
  • Load Balancing: Distributes traffic across multiple Pods backing a Service.
  • Logic Flow:
  • When a Service is created, kube-proxy updates the Node’s network rules (using iptables or IPVS) to route traffic to the correct Pods.
  • kube-proxy continuously watches for changes in Service or Endpoint objects (e.g., when Pods are added or removed) and updates the network rules accordingly.
  • kube-proxy handles both internal cluster traffic (e.g., Pod-to-Pod communication) and external traffic (e.g., incoming requests to a NodePort Service).

9. Cluster Add-ons

Cluster Add-ons are additional components that provide extended functionality to Kubernetes, such as monitoring, logging, networking, and security.

  • Examples:
  • CNI Plugins: Manage pod networking.
  • Ingress Controllers: Handle incoming HTTP/HTTPS traffic to services.
  • Prometheus/Grafana: Provide monitoring and metrics visualization.
  • Fluentd/Elasticsearch/Kibana (EFK Stack): Handle logging.
  • Logic Flow:
  • Add-ons are deployed as Kubernetes resources (e.g., Deployments, DaemonSets) and run within the cluster like any other application.
  • They interact with Kubernetes APIs to gather data, enforce policies, or provide additional services (e.g., ingress routing).
  • Add-ons are managed using standard Kubernetes tools (e.g., Helm, kubectl) and are subject to the same lifecycle management as other Kubernetes objects.

10. Custom Resource Definitions (CRDs)

CRDs allow you to define new types of Kubernetes resources, extending the Kubernetes API to include custom resource types.

  • Key Responsibilities:
  • API Extension: Extends the Kubernetes API with new resource types, enabling the creation and management of custom objects.
  • Integration with Custom Controllers: Typically used in conjunction with custom controllers or operators to manage the lifecycle of these custom resources.
  • Logic Flow:
  • A CRD is defined and applied to the cluster, creating a new API endpoint for the custom resource type.
  • Custom resources are managed through standard Kubernetes tools and can trigger custom controllers or operators to perform specific actions (e.g., creating a database cluster).

11. Admission Controllers

Admission Controllers are plugins that intercept API requests before they are persisted in etcd to enforce policies or mutate resources.

  • Key Responsibilities:
  • Validation: Ensure that resources meet certain criteria (e.g., security policies) before they are created.
  • Mutation: Modify resources on-the-fly before they are persisted (e.g., injecting sidecars).
  • Policy Enforcement: Enforce resource quotas, security policies, and other organizational rules.
  • Logic Flow:
  • After a request is authenticated and authorized, it passes through a series of enabled admission controllers.
  • Each admission controller can modify or validate the request.
  • If the request passes all checks, it is stored in etcd; otherwise, it is rejected, and the user is notified of the error.

Summary: How Everything Interacts

  1. User Interaction: A user interacts with Kubernetes by submitting a manifest (YAML/JSON) via kubectl. This request is sent to the API server.
  2. API Server Processing: The API server authenticates, authorizes, and processes the request, using admission controllers to validate or mutate the request as necessary.
  3. State Persistence in etcd: Once validated, the request is written to etcd, which stores the desired state of the object.
  4. Controllers and Reconciliation: Kubernetes controllers (e.g., Deployment controller) watch for changes in etcd and work to reconcile the actual state of the cluster with the desired state. If new Pods need to be created, the controller triggers the creation process.
  5. Scheduler Placement: The scheduler observes new Pods in the “Pending” state and assigns them to appropriate Nodes based on resource availability and other constraints.
  6. Kubelet and Container Runtime: On the selected Node, the Kubelet instructs the container runtime to pull the necessary images, create containers, and start them. The Kubelet continuously monitors the Pods and reports their status back to the API server.
  7. kube-proxy Networking: kube-proxy configures the Node’s networking to ensure that Pods can communicate with each other and that Services can route traffic to the correct Pods.
  8. Cluster Add-ons: Additional functionality, such as monitoring, logging, and network management, is provided by cluster add-ons, which are managed as regular Kubernetes resources and interact with the Kubernetes API and other components.
  9. Custom Resources via CRDs: If custom resource types are defined via CRDs, they are managed in the same way as standard Kubernetes resources but often have custom controllers or operators managing their specific lifecycle.
  10. Continual Monitoring and Adjustment: All components (Kubelet, controllers, scheduler, etc.) continually monitor the state of the cluster and work together to ensure that the desired state, as defined by manifests and other inputs, is maintained across the cluster.

This interplay of components ensures that Kubernetes can dynamically manage workloads, scale applications, recover from failures, and maintain the overall health and performance of the cluster in a seamless and automated manner.