HasnainWaris

recent posts

about

  • Kubernetes Core Concepts Part 3: Application Lifecycle Management and Cluster Maintenance

    Continuing my Kubernetes series, this post covers how to manage applications throughout their lifecycle – from deployments and updates to configuration and scaling. I’ll also dive into cluster maintenance tasks like upgrades and backups as well as other topics such as Scaling These are critical topics for keeping your applications running smoothly and your cluster healthy.

    Application Lifecycle Management

    Rolling Updates and Rollbacks

    One of Kubernetes’ strengths is how it handles application updates without downtime. When you create a deployment, it triggers rollout version 1. Update the image and apply the changes, and you get rollout version 2.

    Kubernetes tracks these rollout versions, making it easy to roll back if something goes wrong.

    Check rollout status and history with kubectl rollout status and kubectl rollout history.

    Deployment Strategies:

    Recreate – Destroy the existing deployment completely, then create the new one. This causes downtime but is simpler.

    Rolling Update (default) – This method brings down one pod at a time and brings up a new one. No downtime, but the update takes longer. This is the default method.

    To trigger a rolling update, just apply your updated deployment with kubectl apply and Kubernetes handles the rest – gradually replacing old pods with new ones. Feel free to try this out in your test environment, have 2 windows open. Create a deployment on 1, edit and apply changes to the deployment and from the second window you can see the rolling update take place.

    If an update causes issues, roll back with kubectl rollout undo deployment deployment-name.

    Configuring Applications

    Applications need configuration – commands to run, environment variables to call, and sensitive data like passwords to manage. Kubernetes gives you several ways to handle this.

    Commands and Arguments

    Remember that containers aren’t meant to be standalone operating systems. They’re stateless applications that exit when their task is done.

    When running containers, whether with docker run or in a Dockerfile, you specify what the container should do using CMD, ENTRYPOINT, and ARG.

    In Kubernetes, you define these in your pod spec under the container section using the command and args fields.

    Environment Variables

    You can pass environment variables directly to your containers in the pod spec:

    env:
      - name: DATABASE_URL
        value: "mysql://db:3306"

    This is the direct approach – you’re hardcoding values in your YAML. It works but isn’t ideal for values you use across multiple pods or sensitive data.

    ConfigMaps: Centralized Configuration

    ConfigMaps provide a more centralized solution. Instead of repeating the same environment variables across multiple pod definitions, you create a ConfigMap once and reference it from your pods.

    Two steps: create the ConfigMap, then inject it into your pods.

    You can inject a single environment variable from a ConfigMap:

    env:
      - name: PLAYER_INITIAL_LIVES
        valueFrom:
          configMapKeyRef:
            name: game-demo
            key: player_initial_lives

    Or use all variables from a ConfigMap at once:

    envFrom:
      - configMapRef:
          name: myconfigmap

    This second approach is cleaner when you have many related configuration values.

    Secrets: Handling Sensitive Data

    Secrets work similarly to ConfigMaps but are meant for sensitive information like passwords, API keys, and tokens. The values are encoded (base64) before storing.

    Create and reference Secrets the same way as ConfigMaps, just using Secret resources instead. Kubernetes also supports encrypting data at rest using encryption configuration for added security.

    Multi-Container Pods

    Sometimes you need two services working together closely. Instead of configuring the relationship between two separate pods, you can run multiple containers in a single pod.

    This way they scale together, share volume mounts, and share resources. There are different design patterns for this – sidecars, ambassadors, and adapters.

    Init Containers

    In multi-container pods, you usually expect all containers to run continuously. But sometimes you want a container to run a task and stop before the main application starts – that’s where init containers come in.

    Init containers run to completion before the main containers start. Common use cases include setting up configuration files, waiting for dependencies to be ready, or performing database migrations.

    Self-Healing Applications

    Kubernetes supports self-healing through ReplicaSets and Replication Controllers. If a pod crashes, the controller automatically recreates it. If you’ve specified 3 replicas and one goes down, Kubernetes spins up a replacement to maintain your desired state.

    This happens automatically – no manual intervention needed. It’s one of the core features that makes Kubernetes reliable for production workloads.

    Autoscaling

    Scaling comes in two flavors: horizontal and vertical.

    Horizontal scaling – Increases the number of pods or nodes

    Vertical scaling – Increase the size (CPU/memory) of existing pods or nodes

    Scaling can be done manually or automatically.

    Scaling Cluster Infrastructure (Nodes)

    To scale the number of nodes in your cluster, you can use the Cluster Autoscaler (Cloud platforms have their own). It watches for pods that can’t be scheduled due to insufficient resources and automatically adds nodes. It also scales down when nodes are underutilized.

    Scaling Workloads (Pods)

    For pods, you have Horizontal Pod Autoscaler (HPA) and Vertical Pod Autoscaler (VPA).

    Horizontal Pod Autoscaler (HPA):

    You can manually scale with kubectl scale, but that’s not efficient. HPA automates this by observing metrics and adding pods when needed.

    Create an HPA with kubectl autoscale deployment myapp –cpu-percent=50 –min=2 –max=10.

    This tells Kubernetes to maintain CPU usage around 50%, scaling between 2 and 10 replicas as needed. You can also define these parameters in the HPA spec for more control.

    Vertical Pod Autoscaler (VPA):

    Manually scaling pod resources is done with kubectl edit, but VPA automates it by observing metrics and adjusting CPU/memory requests and limits.

    VPA doesn’t come by default – you need to deploy it separately. It’s useful for workloads with variable resource needs.

    In-Place Pod Resizing:

    Traditionally, changing a pod’s resources means deleting and redeploying it. There’s a feature for in-place vertical scaling that adjusts resources without recreating the pod, helping avoid downtime. This needs to be enabled as it’s still evolving.

    Cluster Maintenance

    Now let’s talk about keeping your cluster healthy through upgrades and proper maintenance procedures.

    OS Upgrades

    When you need to upgrade the OS on a node, that node goes down and its pods become inaccessible.

    If the node is down for less than 5 minutes, the controller waits and brings pods back online when the node returns. If it’s down longer, Kubernetes considers the pods dead and the node comes back clean.

    The safe way to handle this is to drain the node first with kubectl drain node-name. This gracefully moves the pods to other nodes. Once the upgrade is complete, uncordon the node with kubectl uncordon node-name to allow scheduling again.

    You can also cordon a node with kubectl cordon node-name, which just prevents new pods from being scheduled there without moving existing pods.

    Kubernetes Version Upgrades

    Kubernetes releases follow semantic versioning: 1.xx.xx (Major.Minor.Patch).

    When upgrading, there shouldn’t be a big version gap between cluster components, but nothing should run a higher version than kube-apiserver – it’s the central component everything else talks to.

    Upgrade Process:

    Upgrade master nodes first, then worker nodes. While masters are upgrading, the cluster still runs, but you can’t make changes or deploy new workloads.

    For worker nodes, you have different strategies:

    • All at once (causes downtime for all workloads)
    • One at a time (safer, rolling approach)
    • Add new nodes with the new version, drain old nodes, then remove them

    With kubeadm:

    First upgrade kubeadm itself, then use it to upgrade the cluster.

    Steps:

    1. Upgrade kubeadm package on the master node
    2. Run kubeadm upgrade apply v1.xx.xx on master
    3. If kubelet runs on the master, upgrade it too
    4. For worker nodes: drain the node, upgrade kubeadm and kubelet packages, run kubeadm upgrade node, restart the kubelet service, then uncordon the node

    Repeat for each worker node.

    Different Kubernetes distributions (k3s, Rancher, manual kubeadm installations) have their own upgrade procedures, so always check the specific documentation.

    Backup and Restore

    Backups are critical – you need a recovery plan if something goes wrong.

    What to backup:

    Resource configuration files – Your YAML manifests. Store these in version control (Git) so you can recreate resources if needed.

    etcd – This is where all cluster state is stored – information about nodes, pods, configs, secrets, everything. Backing up etcd means you can restore your entire cluster state.

    You can backup the etcd data directory directly, or create a snapshot using etcdctl snapshot save. To restore, use etcdctl snapshot restore.

    etcdctl is the command-line client for etcd and your main tool for backup and restore operations.

    Certification Exam Tip

    In the CKA exam, you won’t get immediate feedback like in practice tests. You must verify your work yourself. If asked to create a pod with a specific image, run kubectl describe pod to confirm it was created with the correct name and image. Get in the habit of verifying everything you do.

    Wrapping Up

    Managing application lifecycles in Kubernetes involves understanding deployments and updates, properly configuring applications with environment variables and secrets, and implementing autoscaling for reliability and efficiency. Cluster maintenance – from OS upgrades to Kubernetes version updates and backups – keeps your infrastructure healthy and recoverable.

    These concepts build on the scheduling and monitoring topics from the previous post. Together, they give you the foundation to run production Kubernetes workloads confidently.

    Next in the series, I’ll cover what steps we take to secure our Kubernetes Cluster. See you soon!

  • Kubernetes 02: Scheduling, Logging, and Monitoring

    Continuing my Kubernetes series, this post focuses on scheduling – how Kubernetes decides where your pods actually run – along with monitoring and logging to keep track of what’s happening in your cluster. Understanding these concepts gives you control over pod placement and visibility into your cluster’s health.

    Scheduling

    Manual Scheduling

    By default, Kubernetes uses its scheduler to place pods on nodes. But you can bypass this and manually schedule pods if needed.

    Set the node at pod creation time in the spec:

    spec:
      nodeName: node01
      containers:
      - image: nginx
        name: nginx

    Important note: you can only set this at creation time. An existing pod can only be moved to a different node using a binding resource.

    Labels and Selectors

    Labels and selectors are how you group and filter resources in Kubernetes. They’re key-value pairs that help organize and identify resources.

    Labels are set on resources like this:

    metadata:
      labels:
        env: prod
        app: nginx
        tier: frontend

    Selectors are used to call or filter based on those labels. You can have multiple labels on a resource and filter by any of them.

    Filter pods by label with kubectl get pods --selector env=dev for example, filtering our pods where the env is set to dev. This becomes especially useful when working with services, deployments, and other resources that need to target specific pods.

    Taints and Tolerations

    Taints and tolerations control the relationship between pods and nodes – restricting or preferring where pods can be placed.

    Taints are set on nodes to repel pods. Tolerations are set on pods to allow them onto tainted nodes.

    Think of it like a security clearance system. The node has a restriction (taint), and only pods with the right permission (toleration) can be scheduled there.

    Add a taint to a node with kubectl taint nodes node1 key1=value1:NoSchedule.

    Important distinction: taints don’t guarantee a pod will land on a specific node – they just prevent pods without the right toleration from being scheduled there. The pod might still end up on another untainted node.

    Node Selectors: Simple Pod-to-Node Assignment

    Node selectors let you restrict pods to run on specific nodes based on labels.

    First, label your node with kubectl label nodes node-name labelkey=value.

    Then reference it in your pod spec:

    spec:
      nodeSelector:
        labelkey: value

    The limitation here is that it’s a simple one-to-one relationship. You can’t express complex logic like “run on nodes with label X OR Y” or “avoid nodes with label Z.” That’s where node affinity comes in.

    Node Affinity: Advanced Scheduling

    Node affinity is the more powerful way to control pod placement. It gives you complex matching rules and different enforcement levels.

    Node Affinity Types:

    requiredDuringSchedulingIgnoredDuringExecution – Hard requirement. The pod won’t be scheduled if the rules aren’t met. Like saying “MUST run on nodes with this label.”

    preferredDuringSchedulingIgnoredDuringExecution – Soft preference. The scheduler tries to match but will still schedule the pod if it can’t. Like saying “I’d PREFER nodes with this label, but it’s okay if not.”

    There’s also a planned type (not yet available): requiredDuringSchedulingRequiredDuringExecution – This would evict pods if node labels change and no longer match the requirements.

    Key detail: “IgnoredDuringExecution” means if a pod is already running and the node’s labels change, the pod keeps running and won’t be evicted.

    Operators Available:

    • In – label value must be in the list
    • NotIn – label value must NOT be in the list
    • Exists – label key must exist (value doesn’t matter)

    Example:

    spec:
      containers:
      - image: nginx
        name: nginx
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              - key: color
                operator: In
                values:
                - blue

    This pod will only be scheduled on nodes labeled with color=blue.

    Resource Requests and Limits

    The scheduler considers resource availability before placing pods. You can specify how much CPU and memory your pod needs (requests) and the maximum it can use (limits).

    Set resource requests in your pod spec:

    resources:
      requests:
        memory: "64Mi"
        cpu: "250m"
      limits:
        memory: "128Mi"
        cpu: "500m"

    The scheduler uses requests to find nodes with enough available resources. Limits prevent pods from consuming more than their allocation.

    If you see a pod with status OOMKilled, it means the pod ran out of memory and hit its limit. This is your signal to either optimize the application or increase the memory limit.

    DaemonSets

    While ReplicaSets help you run multiple instances of a pod across the cluster, DaemonSets ensure at least one replica runs on EACH node.

    Common use cases include monitoring agents, log collectors, or networking components that need to run on every node in the cluster.

    The YAML structure is similar to a ReplicaSet, and DaemonSets use the default scheduler to place pods.

    Static Pods

    On a node without a kube-apiserver – just kubelet – you can create static pods by placing YAML files in /etc/kubernetes/manifests. These are managed directly by kubelet, not through the API server.

    You can only create pods this way, not other resources like deployments or services.

    When you run kubectl get pods, static pods will have the node name appended to their name – that’s how you identify them.

    Multiple Schedulers

    Kubernetes lets you run multiple scheduler profiles. When creating a pod, you can specify which scheduler to use based on your needs. This is useful if you have custom scheduling requirements or want different scheduling logic for different workloads.

    Priority Classes

    You can set priority on pods to influence scheduling order. Higher priority pods get scheduled first.

    Kubernetes has default priority classes, and you can create custom ones. The default priority value is 0 – higher numbers mean higher priority.

    How Scheduling Actually Works

    Understanding the scheduling phases helps when troubleshooting:

    1. Scheduling Queue – Pod enters the queue, ordered by priority
    2. Filtering – Nodes that don’t meet requirements are filtered out
    3. Scoring – Remaining nodes are scored based on various factors
    4. Binding – The highest-scoring node is selected and the pod is bound to it

    Each phase has scheduling plugins handling that section of the process.

    Monitoring Your Cluster

    Now that we understand how pods get scheduled, let’s talk about keeping an eye on them once they’re running.

    Kubernetes doesn’t come with a built-in monitoring solution, but there are solid open-source options like Metrics Server, Prometheus, and the Grafana stack.

    Metrics Server is a cluster add-on that collects and aggregates resource usage data – CPU and memory – from all nodes and pods. It’s lightweight and perfect for basic monitoring needs.

    Once Metrics Server is installed, you can use kubectl to view resource consumption.

    kubectl top nodes shows CPU and memory usage across your nodes.

    kubectl top pods shows resource usage for all pods in the current namespace.

    This is incredibly useful for monitoring resource consumption, checking node health, and tracking pod performance. You can quickly spot which pods are consuming the most resources or if a node is running hot.

    Managing Logs

    Logs are critical for troubleshooting and understanding what’s happening inside your pods.

    The basic command to view pod logs is kubectl logs pod-name.

    If your pod has multiple containers, specify which one with kubectl logs pod-name -c container-name.

    Follow logs in real-time with kubectl logs -f pod-name.

    View logs from a previous instance of the container (useful if it crashed) with kubectl logs pod-name –previous.

    Coming from a sysadmin background where I’m used to tailing /var/log files, kubectl logs takes some getting used to. But it’s actually more convenient – no SSHing into nodes, no hunting through directories. Just point kubectl at the pod and you’ve got your logs.

    Troubleshooting Tips

    • Use kubectl describe pod pod-name to see scheduling events and why a pod might be pending
    • Check kubectl get events to view recent cluster events in a namespace
    • Use kubectl logs for application logs and debugging runtime issues
    • Combine kubectl top with kubectl describe to correlate resource usage with pod behavior

    Wrapping Up

    Scheduling in Kubernetes gives you control over where your workloads run – from simple node selectors to complex affinity rules, taints and tolerations, and resource management. Pair that with proper monitoring and logging, and you’ve got visibility into both placement decisions and runtime behavior.

    These tools let you optimize for performance, cost, and availability while being able to troubleshoot issues when they inevitably come up.

    In the next post, I’ll continue the series covering storage, ConfigMaps, Secrets, and more. See you then!

  • Kubernetes 01: Core Concepts

    Understanding Cluster Architecture

    I’ve been working through Kubernetes fundamentals in my preparation for the CKA and wanted to document what I cover as I go through my studies. This is the first in a series where I’ll break down Kubernetes piece by piece. Starting with the Core Concepts within Kubernetes

    The Cluster: Nodes and Their Roles

    A Kubernetes cluster is made up of nodes – essentially your servers. You have two types:

    Worker nodes host your applications as containers. These are where your actual workloads run.

    Master nodes (also called control plane nodes) manage, plan, schedule, and monitor the worker nodes. Think of the master as the brain of the operation.

    The master node runs several critical components: etcd, kube-scheduler, controller-manager, and kube-apiserver (which orchestrates all operations on the cluster). We’ll dive into each of these shortly.

    Every node also needs a container runtime engine like Docker or containerd to make applications container-capable. Speaking of which – Kubernetes and Docker used to be tightly coupled, but Kubernetes developed CRI (Container Runtime Interface) to work with different container runtime vendors. This is why you’ll see containerd used more often now instead of Docker.

    Master Node Components

    Let’s break down what’s running on that master node:

    etcd – A reliable key-value store that holds all the cluster data. It stores information about nodes, pods, configs, secrets – everything. When you install Kubernetes manually, you install etcd separately. If you use kubeadm, it comes bundled as a pod called etcd-master.

    kube-apiserver – This is what you interact with when you run kubectl commands. Every interaction with the cluster goes through the API server. When you run a kubectl command, it hits the kube-apiserver, which then communicates with other components like etcd, the scheduler, or kubelet. The API server handles authentication, validates requests, retrieves data, updates etcd, and coordinates with the scheduler and kubelet.

    kube-controller-manager – Runs various controllers (like node-controller and replication-controller) that watch the cluster state and take action when needed. If a node goes down or a pod fails, controllers notice and work to remediate the situation.

    kube-scheduler – Decides which pods go on which nodes. It filters nodes based on the pod’s requirements, ranks them, and assigns the pod to the best-fit node.

    Worker Node Components

    On the worker nodes, you have:

    kubelet – The agent running on each node. It communicates with the kube-apiserver and reports back on the node’s status and the pods running on it.

    kube-proxy – Handles networking so pods can communicate with each other. It manages the pod network and helps route traffic between pods across different nodes.

    Pods: The Smallest Unit

    Pods are the smallest deployable unit in Kubernetes – a single instance of an application. Usually, there’s a one-to-one relationship between a pod and a container, but you can have multiple containers in a pod (like an app container and a helper/sidecar container).

    You can deploy pods directly via kubectl commands or using YAML files. YAML manifests include fields like apiVersion, kind (the type of resource), metadata (names and labels), and spec (the detailed configuration).

    Replication and High Availability

    Running a single pod isn’t great for availability. That’s where ReplicationControllers and ReplicaSets come in. They ensure a specified number of pod replicas are always running. If a pod crashes, the controller spins up a new one automatically.

    Labels and selectors are key here – they help controllers identify which pods they’re responsible for managing. In your ReplicaSet definition, you specify both the replica count and the pod template.

    Deployments: Managing Everything

    Deployments sit one level above ReplicaSets. Instead of managing individual pods or even ReplicaSets manually, you create a Deployment that defines your desired state. Need to update an image? Scale up? Roll back? The Deployment handles it all.

    This is what you’ll use most often – deployments make it easy to create, edit, manage, and patch multiple containers all at once.

    Pro tip for the exam: When you need to generate a YAML file for a resource, add --dry-run=client -o yaml to your kubectl run or create command. This generates the YAML without actually creating the resource – super useful for quick templates.

    Services: Consistent Access to Pods

    Pods are ephemeral – they come and go, IPs change constantly. So how do you consistently reach them? Services.

    A Service provides a fixed address to access a set of pods. Think of it as a grouping mechanism. For example, you might have a frontend service that routes traffic to several frontend pods. Instead of tracking individual pod IPs, your application connects to the service, which handles routing to available pods.

    Creating a service is straightforward:

    kubectl expose deployment frontend --port 8080

    This creates a service with its own fixed IP. Now you have a stable endpoint regardless of pod changes.

    Types of Services:

    ClusterIP (default) – Creates an internal IP for the service. Great for pod-to-pod communication within the cluster, like frontend pods talking to backend pods.

    NodePort – Exposes a port on each node, allowing direct access via the node’s IP. Not commonly used, especially in cloud environments.

    LoadBalancer – Used with cloud providers. Creates a load balancer that exposes the service externally with its own IP and routes traffic into the cluster.

    When creating services, make sure your selectors and labels match the pods you’re exposing – that’s how the service knows which pods to route to.

    Quick tip: To expose a pod with specific settings:

    kubectl expose pod redis --port=6379 --name=redis-service --type=ClusterIP

    Namespaces: Logical Separation

    Namespaces provide logical grouping and isolation of objects. Useful when you have many users or want to separate resources by team, environment, or project.

    Create a namespace:

    kubectl create namespace mealie

    Deploy pods to a specific namespace:

    kubectl create deployment app --image=nginx --namespace=mealie

    Change your default namespace:

    kubectl config set-context --current --namespace=mealie

    Imperative vs Declarative

    Two approaches to managing Kubernetes resources:

    Imperative – Telling Kubernetes exactly what to do and how to do it. Step-by-step instructions using kubectl commands like run, create, expose, edit, scale, and set. Good for quick tasks and learning.

    Declarative – Specifying the desired end state using YAML files and kubectl apply. You declare what you want, Kubernetes figures out how to get there. This is the preferred approach for production – your YAML files become the source of truth for your infrastructure.

    Exam Resources

    Two helpful commands for studying:

    • kubectl explain [resource] – Get detailed info about a resource
    • kubectl api-resources – List all available resources

    Wrapping Up

    Understanding these core components – how the master and worker nodes interact, what pods and services are, how deployments manage everything – is foundational to working with Kubernetes. In the next post, I’ll dive deeper into further aspects I covered such as scheduling, env variables, secrets etc.

    See you in the next one!

  • Linux Basics: The Boot Process

    Understanding the Linux Boot Process: From Power Button to Login

    Ever wonder what actually happens when you hit the power button on a Linux system? I’ve been diving into the boot process lately and figured I’d break down what’s happening behind the scenes. It’s one of those fundamental topics that helps you troubleshoot issues and understand how your system really works.

    The Big Picture

    The overall process follows this sequence: BIOS POST → GRUB → Kernel → systemd

    There are generally two main sequences involved: boot and startup.

    Boot is everything from when the computer powers on until the kernel is initialized and systemd is launched. Startup picks up from there and finishes getting the system to an operational state where you can actually do work.

    Let’s walk through each stage.

    BIOS and POST

    The boot process starts with hardware. When you first power on or reboot, the computer runs POST (Power-On Self-Test), which is part of the BIOS. The BIOS initializes the hardware, and POST’s job is to make sure everything is functioning correctly – checking basic operability of your CPU, RAM, storage devices, and other critical components.

    [If POST detects a hardware failure, you’ll usually hear beep codes or see error messages before the system halts. On newer systems with UEFI instead of traditional BIOS, this process is similar but UEFI offers more features like a graphical interface and support for larger disks.]

    GRUB2 – The Bootloader

    Once POST completes, the BIOS loads the bootloader – GRUB2 in most modern Linux distributions. GRUB’s job is to find the operating system kernel and load it into memory.

    When you see that GRUB menu at startup, those options let you boot into different kernels – useful if you need to roll back to an older kernel after an update causes issues. The GRUB configuration file lives at /etc/grub2/grub.cfg or /boot/grub2/grub.cfg depending on your distribution.

    GRUB actually operates in two stages:

    Stage 1: Right after POST, GRUB searches for the boot record on your disks, located in the MBR (Master Boot Record)

    MBR: The Master Boot Record is the information in the first sector of a hard disk, it identifies how and where the system’s OS is located in order to be booted into the computer’s main storage or RAM.

    Stage 2: The files for this stage live in /boot/grub2. Stage 2’s job is to locate the kernel, load it into RAM, and hand control over to it. Kernel files are located under /boot – you’ll see files like vmlinuz-[version].

    The Kernel Takes Over

    After you select a kernel from GRUB (or it auto-selects the default), the kernel is loaded. First, it extracts itself from its compressed file format. The kernel then loads initramfs (initial RAM filesystem), a temporary root filesystem that contains drivers and tools needed to mount the real root filesystem. This is especially important for systems where the root filesystem is on RAID, LVM, or encrypted volumes.

    Once the kernel has initialized the hardware and mounted the root filesystem, it loads systemd and hands control over to it. At this point, the boot process technically ends – you have a kernel running and systemd is up. But the system isn’t ready for work yet.

    Startup Process with systemd

    The startup process is what brings your Linux system from “kernel loaded” to “ready to use.” systemd is the mother of all processes and is responsible for getting the system to an operational state.

    systemd’s responsibilities include:

    • Mounting filesystems (it reads /etc/fstab to know what to mount)
    • Starting system services
    • Bringing the system to the appropriate target state

    systemd looks at the default.target to determine which target it should load. Think of targets as runlevels – they define what state the system should be in. Common targets include multi-user.target (multi-user text mode) and graphical.target (GUI mode). You can check your default target with systemctl get-default.

    Each target has dependencies described in its configuration file. systemd handles these dependencies and starts services in the correct order to satisfy them.

    Wrapping Up

    GRUB2 and systemd are the key components in the boot and startup phases of most modern Linux distributions. These two work together to first load the kernel and then start all the system services required to produce a functional Linux system.

    Understanding this process has helped me troubleshoot boot issues, understand where to look when services don’t start, and generally appreciate what’s happening under the hood when I power on a system. Next time your system hangs during boot, you’ll have a better idea of which stage it’s stuck in and where to start investigating.

    See you in the next post!

  • Building a Full DevOps Pipeline: From Dev Container to Production

    Building a Full DevOps Pipeline: From Dev Container to Production

    Recently wrapped up a project that took me through the complete DevOps lifecycle. The goal was simple: understand how all these pieces fit together in a real workflow. From setting up a development environment to deploying to production with GitOps, here’s how it all came together.

    Starting with the Dev Environment

    First things first, we needed a consistent development environment. We used devcontainers with a JSON config and Dockerfile to spin up a container with everything we needed already configured. Added a script that points to a mise.toml file to handle our tooling setup. This became our devpod – our entire workspace where all development happens.

    Python Packaging with UV

    Inside the devpod, we set up UV, a Python package manager that handles dependencies. Coming from managing Python environments the traditional way, UV was refreshing. Commands like uv init --package, uv sync, and uv add made dependency management straightforward. We structured our project with separate frontend and backend directories and used pytest to test our code as we built it out.

    Containerizing the Application

    Next step was turning our Python app into Docker images – aiming for the smallest size possible. We created Dockerfiles for both backend and frontend with a few key configurations:

    • Used Python Alpine images for minimal size
    • Mounted dependencies on a cache layer for faster builds
    • Copied our code into the image’s working directory
    • Exposed necessary ports and set up proper user groups
    • Ran the app directly from .venv/bin

    Introducing CI/CD with GitHub Actions

    This is where things got interesting. We set up GitHub Actions workflows (pipelines) triggered by changes to our backend or frontend code. Each workflow included:

    Automated testing – Set up the environment on Ubuntu, installed UV, configured Python, pulled our repo, and ran our tests. We added Ruff for linting to catch syntax issues before they became problems. Even added a pre-commit hook so Ruff checks all Python code before commits go through.

    Test coverage – Running pytest was good, but we wanted to know how much of our code was actually covered by tests. Added coverage reporting to see exactly what we were testing and what we weren’t.

    Image building and security scanning – Built our Docker images and scanned them with Trivy to catch any security vulnerabilities.

    Versioning with Release Please

    With each push, we wanted proper versioning. Set up release-please as a separate GitHub Action that triggers when a PR merges to main. It automatically creates release versions for us – detects changes to our backend (and frontend) and generates its own PR with the new version.

    Following that release, another workflow kicks in to build and push our versioned images to our container registry.

    Local Testing with K3d

    Before anything hits production, we needed to test in a Kubernetes environment. We set up k3d (the Docker version of k3s) right in our devcontainer. Created a kubernetes directory with manifests following a similar structure to my homelab setup – base and dev directories with kustomization files.

    The dev environment reads in our frontend and backend configurations, applies patches to use the correct image tags, and references back to the base manifests.

    We added two key components:

    • A k3d config file
    • A script that automates the entire process: checks dependencies, creates the cluster if it doesn’t exist, builds our images, imports them to the cluster, deploys with kustomize, and prints out the application URLs

    End-to-End Testing

    Created an e2e_test.py script that tests both backend and frontend in the actual cluster environment, then tears down the cluster when done. This runs as part of our GitHub Actions workflow after image creation – a final validation before anything moves forward.

    GitOps for Production

    The final piece was setting up GitOps with Flux. We created a separate script that spins up a k3d cluster configured with GitOps, pointing to our GitOps repository. This simulates our actual production setup.

    Here’s how it flows: our test repo goes through all the CI/CD steps, creates tested and versioned images, and if everything passes, a workflow updates the image tags in our production GitOps repo. Flux watches that repo and automatically syncs any changes to our production cluster. The workflow creates a PR to update the main branch with the new images, and once merged, Flux handles the deployment.

    Wrapping Up

    Going through this entire pipeline gave me a real appreciation for how all these DevOps tools and practices connect. It’s one thing to know about Docker, GitHub Actions, Kubernetes, and GitOps individually. It’s another to see them work together in a complete workflow – from writing code in a standardized dev environment to automated testing, versioning, and GitOps-based deployments.

    The beauty of this setup is that once it’s configured, the entire process from code commit to production deployment is automated and tested at every step. No manual image building, no kubectl apply commands in production, just Git commits and pull requests.

    Looking forward to expanding on this setup and diving deeper into each component. See you in the next post!

  • Cloud Services: Prem vs Cloud

    I’ve recently been studying for the AWS SAA and coming from a mostly on Prem or virtual environment I noticed how many of the services I’m reviewing have such great use cases, or the function of the services and how they correlate to previous existing functions. Don’t get me wrong, I don’t see everything about the cloud as a plus, or life changing as an admin, but going over this prep material and coming from an architect POV, I can appreciate the designing of an infrastructure much more.

    AWS EC2: Autoscaling

    A useful feature with EC2 (AWS’ VM) that I find is the idea of Autoscaling (vertical/horizontal). With on-prem environments you’re often left manually provisioning a new server when an increase in demand comes, or planning ahead of time to build out a cluster build with a load balancer to manage system load.

    The convenience of an Auto Scaling Group solves this headache. The purpose is to scale out/in to match the load your application is receiving. And with the various scaling options we can configure based on which scaling option is most applicable to our need. Dynamic (setting a target based off a metric), Scheduling (scaling based off known usage patterns), Predictive (continuously forecasts load and schedules based off it)

    From previous experience of handling a similar scenario with on-prem servers, this would save plenty of headache and ease. From reactive to proactive.

    AWS: S3 vs Traditional File Storage

    Dealing with Files and storage there are plenty of useful tools and functions we are able to use on Linux, nothing is really lacking in terms of usefulness. But there are a few useful features that make S3 a great choice.

    With on-prem servers, you’re dealing with setting up file servers, managing disk space, dealing with RAID configurations, worrying about backups, setting up NFS shares for application access etc. These are multiple different applications or services we have to deal with.

    S3 gives us various Storage Class options, built-in redundancy, versioning, and lifecycle policies. No managing underlying storage hardware.

    Storage classes: With S3 we have various storage options based off need, based off how we want our storage to operate. Do we need instant retrieval of our storage? Frequent or Infrequent access ? Or archive it in Glacier for backups?

    Versioning: A very useful feature to help protect against unintended deletes and easy rollbacks to previous versions

    Lifecycle Policies: another useful feature to help us move our S3 objects between storage classes. Use case example, if we want to convert an object after X amount of time from frequent access to eventually archiving it as Glacier storage class.

    The features are plentiful that I haven’t discussed like encryption, Access Points, performance etc. All useful features available with S3 buckets.

    I’ve kept it to just 2 services, but those architecting environments can see different problem solving solutions that Cloud Services can provide. But I wouldn’t say I’m all-in on the cloud, I still very much enjoy the hands on experience with building with On-Prem or virtual environments. Planning and building out clusters and applications with different services and tools, intertwining them into a functioning, highly available end product. And coming from that background I can understand the issues that cloud services try and tackle and the solutions it can offer. Giving me a background to base it off of and relate to.

  • Homelab setup: Beginning of my Kubernetes Homelab

    For the longest time, I’ve always for the longest time seen cool projects by my fellow Admins and Engineers. Building their own servers, homelab, applications etc. I thought to myself why not me. I figured too much time commitment, hardware commitment, what would I create or build etc.

    I realized those were just excuses, if I am passionate about it I’ll find the time. Hardware? Old laptop is more than enough (also building on raspberry pis, super cool). What to build? Anything I want.

    So I went ahead and took that step, lot of tutorials and guides later I am just about there. I figured why not write up a short recap.

    The homelab is still a work in progress, but it will always be. The main idea was to get it up and running in the first place.

    Grabbed an old laptop, installed Ubuntu and got started.

    First thing to do was decide which container orchestration tool I wanted to go with, ended up going with K3s and it was a great decision. A quick, easy and lightweight Kubernetes distribution that’s perfect for learning and smaller environments. For someone transitioning from traditional sysadmin work to DevOps and container orchestration, K3s lets me get hands-on with Kubernetes without the overhead of a full production cluster setup.

    After getting k3s setup, next was setting up Flux. After seeing some guides and suggestions of GitOps I figured why not. I wanted to get a feel of the GitOps workflow and have a feel of it not in a production environment but simulating it in my home lab. After getting a feel of Flux, I can’t imagine running my cluster any other way. From a SysAdmin POV with experience in something like Ansible. I love the idea of controlling state of an application or machine with files. Flux watches my Git repo and automatically syncs changes to the cluster. Defining the state of our cluster from a single point of truth with our GitHub repository, lovely.

    Avoiding manually declaring resources and instead letting Flux take care of that. I enjoyed following the Flux repository structure as well (https://fluxcd.io/flux/guides/repository-structure/) a great way to structure and keep a clean repository. It was definitely more difficult to grasp and structure theoretically. But once in action further applications were much easier to setup.

    K3s done, Flux done. Now to host applications. I won’t expand too much on the simple setup of storage, service, namespace and deployment. Setting up my first application (Linkding, a bookmark manager) on the cluster felt like such a win. Once the initial homelab setup is done, hosting applications becomes straightforward. The trickiest aspect can be networking, which I’ll probably look to tackle in a separate post

    I’ll wrap it here for this post, but this is just the start of the homelab. I look forward to growing it, expanding upon this and learning so much more and I’ll be sure to share it as well.

    Thank you for reading, see you in the next post!

  • Kubernetes Fundamentals: Deployments, Services and Storage.

    My Takeaways from Kubernetes Fundamentals

    I recently just wrapped up a Kubernetes fundamentals course and wanted to share some of the key concepts that stuck with me. If you’re just getting started with Kubernetes like I am, hopefully this helps clarify some things.

    The biggest thing I learned early on: everything in Kubernetes is defined through YAML files. Once that clicked, things started making a lot more sense.

    The course mainly focused on three core components: Deployments, Services, and Storage.

    Deployments are where you define your pods. This includes details like what container image you’re using, how many replicas you want running, selectors and labels for organizing things, which namespace you’re working in, and any volumes you want to attach. Think of deployments as the blueprint for what you want running.

    Services are what actually give you an IP address to access your pods. Each pod comes with it’s own IP address, but services allow us to group our Pods under a single IP. There are a few types but the main ones are ClusterIP, LoadBalancer, and NodePort. Each one serving its own purpose.

    ClusterIP is the default and gives you an internal IP that your pods can use to talk to each other. You can quickly create one using the expose command on an existing deployment. If you need temporary external access, port forwarding does the job.

    LoadBalancer is what you use when you need something more permanent and externally accessible. Say you have a group of pods you want exposed to the outside world. You create one LoadBalancer service for them and boom, you’ve got a persistent external IP. No need for port forwarding here.

    Storage was probably the trickiest part for me to grasp at first. By default pods are ephemeral and so is the default storage. You can say goodbye to any data on a pod you just deleted. For more persistent storage you can create either Persistent Volumes or Persistent Volume Claims. They work a bit differently but accomplish similar goals. The key benefit is that once you attach these volumes to your deployment, your data sticks around even if the pods get deleted or recreated.

    These fundamentals have given me a solid foundation to start working with Kubernetes in real environments. There’s obviously a lot more to learn, but understanding deployments, services, and storage gets you pretty far. Looking forward to diving deeper and sharing more as I continue this journey into container orchestration.

  • Linux CLI: Here are 3 tricks that have saved me countless hours

    Over the course of my years as a SysAdmin I’ve come across useful tips/tricks that have helped me navigate the terminal more efficiently and made my job much easier. So I thought why not share them here as well, some of them you may know and some you may be hearing about for the first time

    Tip 1: Using !! to redo previous command

    I can’t count how many times I’ve had to redo a command and coming across this has helped me save plenty of time.

    How this works is !! calls the last command you ran. Now for example some useful use cases are: rerunning the last command but with sudo. sudo !! solves that for you without you having to retype the full command with sudo before hand. I know plenty of Admins can understand the frustration of forgetting to type in sudo before running a command.

    Tip 2: CTRL + R

    Working on the terminal and typing commands constantly you can sometimes forget a command and the full syntax that you may have run in the past or just recently. CTRL + R has been great in finding a command that I may have run before. Especially for longer syntax commands, it’s a bit tricky to just retype the full command. CTRL +R allows you to search history by whatever bit you can remember. Let’s take an example:

    Let’s say you recently ran an ansible-adhoc command:

    ansible databases -b -m yum -a “name=mariadb-server state=latest”

    and you wish to run another adhoc command and by using CTRL + R and by searching for “ansible” you can access the last ansible command you run and from there can rewrite command as you wish. Time efficient if you ask me

    Tip 3: Aliases for Frequent Commands

    If you are consistently writing certain commands with certain flags and arguments, long or short. You can always create a shortcut by creating an alias for it within your ~/.bashrc or ~/.bash_aliases file

    alias ll=’ls -alF’

    alias up=’sudo apt update && sudo apt upgrade -y’

    these are just some examples, you can imagine how much easier this makes things for a SysAdmin. Feel free to get crazy and experiment with what works for you.

    Remember, you will need to run source ~/.bashrc or restart your terminal for new aliases to take effect

    These are only just some tips that you may or may not find useful but as you’ll see, with more time spent on the CLI you’ll come across more tricks like these making your life as an Admin much easier. Hope I was able to help and hope these tricks were useful. I’ll see you next week with a new post.

    Hasnain

    Linux Systems Administrator | Aspiring Devops Engineer

  • Linux Admin to DevOps Engineer – The Journey

    Welcome, My name is Hasnain and to keep it short and simple I am a Linux Admin who is looking to document his journey to being a DevOps Engineer.

    I’ve never been the poster or blogger type but I had a realization. I’ve always benefited from the blogs and articles of others within the field, so why can’t that be me. Why not document this journey of mine to achieve being a proper Devops Engineer, to document learning the variety of tools on the way, paths I am taking to achieve certain certifications as well as sharing what I have expertise in, Linux.

    I look forward to this, and hope to be consistent in my posts and consistent in my progress. Onwards and upwards!

    Hasnain