I worked for a database tools company many years ago and was lucky enough to build a few database cloning tools before PlanetScale was cool.

Those tools were never destined to be as successful as PlanetScale, but they had their uses.

The idea behind those tools was simple: isolate the storage layer from compute and use the best storage technology for cloning the database files.

This idea came back to me a while ago while I was tweeting about an interview I had with an F1 team that was looking for a Rust dev with hands-on Ceph experience. Mentioning Ceph reminded me of a failed product attempt I was part of, and I jokingly told a friend that I was going to build Homescale, PlanetScale at home.

Homescale

I am building Homescale at github.com/homescale-dev/homescale.

Homescale creates writable database instances and point-in-time branches from immutable snapshots without copying the full database.

I borrowed Docker’s image and container model to represent the database state: A database image is an immutable starting point. A database container is a writable clone created from that image. A branch is another container created from the current state of an existing container.

The CLI I have in mind looks like this:

homescale image create --engine postgres postgres-base
homescale container create --image postgres-base dev-db
homescale container connect dev-db

homescale branch create --container dev-db feature-login
homescale container connect feature-login

The examples in this series use Postgres, but Homescale’s storage model is database agnostic. The image, container, and branch model can apply to any database engine that keeps its durable state on a filesystem backed by a block device and can be prepared for a recoverable snapshot. Postgres is the first engine I plan to support and gives me something concrete to use while building the storage and orchestration layers.

Engine-specific behavior belongs behind an adapter. That adapter initializes an image, starts the database process, exposes connection details, and prepares the database for a snapshot when necessary. Homescale handles the lifecycle around it: volumes, immutable states, writable clones, workloads, and lineage.

The word branch describes the relationship between the containers. After running these commands, dev-db and feature-login are both writable database containers. feature-login simply started from the state of dev-db at the moment the branch was created.

Underneath, the lineage looks like this:

flowchart LR
    Image[/postgres-base<br/>image/]
    Dev[dev-db<br/>writable]
    State[/read-only<br/>state/]
    Feature[feature-login<br/>writable]

    Image -->|clone| Dev
    Dev -->|snapshot| State
    State -->|clone| Feature

The image is already immutable, so Homescale can create dev-db from it directly. Branching from a writable container needs an intermediate state. Homescale first captures dev-db at a point in time, then creates feature-login from that state.

Creating feature-login cannot mean copying dev-db byte for byte. A 100 GB database would require another 100 GB of storage before the branch could start.

The branch should initially share its data with the captured state of dev-db. Only the data that changes afterward should require additional storage.

That requires branching to happen below the database process, in the storage layer.

Separating storage from compute

Separating database storage from compute is not a new idea. Standard Amazon RDS engines store database and log files on EBS volumes. Google Cloud SQL runs the database process in a VM with attached network block storage, using Persistent Disk or Hyperdisk depending on the machine series. The database still sees a normal block device, but its durable data is not tied to the compute host’s local disk.

Aurora, AlloyDB, and Azure SQL Hyperscale take this further. Their compute instances connect to shared or distributed storage systems designed specifically for the database. Compute instances can be replaced or scaled without creating a complete copy of the data for each instance.

Homescale is closer to the first model. A database process reads and writes a filesystem on what looks like a block device. I only need the storage behind that device to have a lifecycle independent of the process using it.

flowchart TD
    Database[Database process]
    Filesystem[Filesystem]
    Device[Block device]
    Storage[(Persistent storage)]

    Database -->|file reads and writes| Filesystem
    Filesystem -->|block I/O| Device
    Device --> Storage

That boundary lets Homescale create the storage before starting a database process and keep it after the process stops. More importantly, it lets branching happen in the storage layer without requiring the database engine to implement branching itself.

The two Postgres processes know nothing about their shared history. Each sees its own writable block device. Another supported engine would see the same storage boundary. The engine adapter would change, but the snapshot, clone, and lineage model would not.

Homescale still needs a way to create a writable block device that shares unchanged data with its parent.

COW is the secret sauce

Copy-on-write (COW) allows a writable clone to share unchanged data with the immutable state it was created from.

When Homescale creates dev-db, the new container initially reads its data from postgres-base. Writes made by dev-db are stored in the container without changing the image.

Creating feature-login repeats the same process. Homescale captures the current state of dev-db, then creates a new writable container from it.

The captured state, dev-db@feature-login, is read-only. It preserves the point where the two containers separated. dev-db can continue changing, while feature-login stores its own changes on top of that captured state.

When feature-login reads data it has not changed, the storage layer follows the chain back to its parent. This can continue through several generations until it finds the data. On the first write to shared data, the storage layer copies the relevant allocation unit into the writable container and applies the change there.

A branch of a 100 GB database therefore appears as a complete 100 GB database without requiring another 100 GB copy up front. Its initial cost is mostly metadata. Storage usage grows as dev-db and feature-login diverge, although a small database write may cause a larger storage allocation underneath.

This model also creates dependencies. feature-login relies on dev-db@feature-login for data it has not copied into its own container. Homescale must not remove that intermediate state while the branch still depends on it.

Homescale therefore needs persistent block devices, immutable snapshots, and writable COW clones.

Ceph

Ceph provides those operations through RBD. The database only sees an ordinary block device.

Ceph provides object, file, and block storage on top of the same underlying system. I only care about the block interface, called RADOS Block Device or RBD. An RBD image is a virtual block device that can be mapped to a machine, formatted with a filesystem, and mounted like a normal disk.

In the earlier storage stack, the persistent storage at the bottom is a Ceph RBD image. The database still reads and writes through the same filesystem and block device interface.

Underneath that RBD image, Ceph splits the device into objects and stores them across its object store. The Ceph documentation describes RBD images as block devices striped over objects in RADOS. The default object size is 4 MB, although it is configurable.

This is why a small database write does not necessarily create an equally small allocation in a clone. When a clone first writes to data it still shares with its parent, Ceph may need to copy the corresponding RADOS object into the child before applying the write.

An RBD snapshot is a read-only view of an image at a point in time. A clone is a writable RBD image that refers back to that snapshot for data it does not yet own. Ceph calls this snapshot layering.

Ceph can snapshot the volume without knowing which database is using it. Whether the database can safely start from that snapshot is a separate problem.

One direct RBD sequence for that operation is:

rbd snap create homescale/dev-db@feature-login
rbd snap protect homescale/dev-db@feature-login
rbd clone \
  homescale/dev-db@feature-login \
  homescale/feature-login

Homescale represents that sequence with one command, although Kubernetes and Ceph CSI will perform the backend storage operations:

homescale branch create --container dev-db feature-login

dev-db@feature-login is the read-only state between the two containers. Ceph will not delete it while it is protected. feature-login is a regular writable RBD image that can be mounted, snapshotted, and branched again.

RADOS, OSDs and RBDs

RBD sits on top of RADOS, which stores its objects through OSDs:

flowchart TB
    RBD[RBD image]
    Objects[RADOS objects]
    PG[Placement group]
    OSDs[Acting OSD set]
    Disks[(Storage devices)]
    Pool[Pool policy<br/>replicas and CRUSH]

    RBD -->|split into| Objects
    Objects -->|hash to| PG
    Pool -.->|controls placement| PG
    PG -->|maps to| OSDs
    OSDs -->|persist on| Disks

RADOS is Ceph’s distributed object store. RBD provides the block interface on top of it. When a database writes to an RBD-backed filesystem, RBD turns those block operations into reads and writes against RADOS objects.

Those objects live in a pool. A pool defines how its data is stored, including the number of replicas and the CRUSH rule used for placement.

Ceph does not map every object directly to a disk. It first maps the object to a placement group, then maps that placement group to one or more OSDs. This allows Ceph to move data when OSDs are added, removed, or fail without keeping a central table of every object’s location. The Ceph architecture documentation describes the same path: objects to placement groups, then placement groups to OSDs.

An OSD stores RADOS objects on a storage device and handles their reads, writes, and replication. Ceph can split one device between multiple OSDs, but the documented layout is one storage drive per OSD. That is the layout I will use.

Homescale will not talk to RADOS directly. Its storage boundary is RBD: create a volume, take a snapshot, and clone it. The operator will request those operations through Kubernetes PVCs and VolumeSnapshot resources.

Kubernetes as the control plane

I do not want Homescale to manage database processes, volume mounts, and network endpoints directly. It will describe the desired database and storage state through Kubernetes resources, then rely on controllers to make that state real.

The CLI talks to the Homescale API. Homescale creates database workloads, PVCs, and VolumeSnapshot resources. Ceph CSI translates the storage resources into RBD operations, while Rook operates the Ceph cluster itself.

flowchart TB
    CLI[Homescale CLI] --> Control

    subgraph Kubernetes
        Control["Homescale"]
        Databases
        StorageAPI["Storage API<br/>PVCs, snapshots, CSI"]
        Ceph["Ceph storage<br/>RBD and OSDs"]

        Control -->|reconciles| Databases
        Control -->|creates| StorageAPI
        StorageAPI --> Ceph
        Databases --> Ceph
    end

    Ceph --> Disks[(Storage devices)]

Kubernetes controllers reconcile these resources toward their desired state. If a database pod disappears, its controller replaces it. If a PVC requests dynamic provisioning through a compatible StorageClass, the CSI driver provisions its storage. Homescale can use that machinery instead of implementing its own process supervision and storage attachment logic.

The application-specific resources and the controllers that reconcile them belong to the next part. The important boundary here is the Kubernetes storage API: Homescale requests PVCs and VolumeSnapshot resources, and Ceph CSI translates those requests into RBD operations.

Local and replicated deployments

For now, Homescale will run locally on macOS in a single-node Kubernetes cluster, probably using Colima or Lima. This local profile has one Kubernetes node, one Ceph OSD, and one storage device.

I may also deploy Homescale to the private Talos cluster I run on Hetzner Cloud. The Terraform for that cluster already supports separate worker pools, so I can add a Ceph node pool without putting OSDs beside the existing application workloads.

A replicated pool with size: 3 and failureDomain: host can place each object’s replicas across those nodes. If a disk or node fails, Ceph can continue serving the volume from the surviving replicas.

Locally, there is one copy of the data. On Hetzner, I can run the same Homescale operator against a replicated Ceph pool spread across the storage nodes.

Rook connects Kubernetes to Ceph

A CephCluster resource tells Rook which Ceph version to run, where it can place monitors and managers, and which devices it can consume for OSDs. A CephBlockPool describes the RBD pool and its replication and failure-domain settings. The Rook operator watches those resources and creates or updates the corresponding Ceph daemons and configuration.

The split looks like this:

flowchart TB
    Pod[Database pod] -->|mounts| PVC[PVC]
    Snapshot[VolumeSnapshot] -->|source for| NewPVC[New PVC]
    PVC --> CSI[Ceph CSI]
    NewPVC --> CSI
    CSI -->|provisions in| Ceph[Ceph cluster]

    CRs[Ceph resources] --> Rook[Rook operator]
    Rook -->|reconciles| Ceph

Rook configures the Ceph CSI drivers used to provision and mount storage. Homescale can use PVCs and VolumeSnapshot resources without running rbd commands or carrying Ceph credentials itself. The Rook snapshot documentation describes the same restore flow: create a VolumeSnapshot from a PVC, then use that snapshot as the data source for another PVC.

This keeps Homescale independent of the Ceph deployment shape. The local cluster can use one OSD and a pool with one copy. The Hetzner cluster can use three OSDs and replication across hosts. As long as both expose compatible StorageClass and VolumeSnapshotClass resources, the Homescale operator follows the same reconciliation flow.

What is next?

The next part will cover the Homescale services: the API used by the CLI, the controllers that manage database resources, and the Postgres adapter.

I will start with the image and container path. An image request needs to initialize a Postgres volume and capture it as an immutable VolumeSnapshot. A container request needs to clone that snapshot into a new PVC, start Postgres, wait for it to become ready, and return connection details.

Once that path works, I can add branching from a running container. That service flow must prepare the source database for a snapshot, create the intermediate state, record its lineage, and reconcile another writable container from it.

The previous post was about the network. This one is about what I put inside that network: a private Kubernetes cluster running Talos on Hetzner Cloud.

The important part is not just “Kubernetes on Hetzner Cloud”. There are many posts about it. The part I cared about was making the cluster private from the first boot. No public IPs on the control plane. No public IPs on the workers. Access only through the Tailnet.

That made Talos a good fit. No package manager, no SSH. You give it machine configuration, it becomes a Kubernetes node, and that is mostly it.

Mostly.

What I Wanted from the Cluster

• Private-only nodes: every Kubernetes node should live only on the Hetzner private network.
• Terraform-managed bootstrap: machines, Talos config, kubeconfig, and base add-ons should come from code.
• Talos: no manual server maintenance.
• Separate node pools: platform components should not fight application workloads.
• GitOps: Terraform can bootstrap ArgoCD, then ArgoCD owns the platform.

The goal was to build something small enough that I could understand every moving part, but powerful enough that I could run actual projects on it.

Cluster Shape

The private network from the previous post gives the cluster a /24 to live in. I split that range into explicit chunks:

• Control plane: 10.0.128.16/28
• Platform workers: 10.0.128.32/27
• General workers: 10.0.128.64/27
• Service network: 10.0.192.0/21
• Pod network: 10.0.200.0/19

The control plane has three nodes. Platform workers run things like ArgoCD and platform components. General workers run applications like snapbyte.dev.

flowchart TB
    Tailnet((Tailnet))
    Internet((Internet))

    subgraph VPC["Private network 10.0.0.0/16"]
        subgraph Subnet["Subnet 10.0.128.0/24"]
            NAT["NAT Gateway"]

            subgraph CP["Control Plane 10.0.128.16/28"]
                CP1["cp-1"]
                CP2["cp-2"]
                CP3["cp-3"]
            end

            subgraph Platform["Platform Workers 10.0.128.32/27"]
                ArgoCD["ArgoCD"]
                PlatformApps["Platform components"]
            end

            subgraph General["General Workers 10.0.128.64/27"]
                PublicApps["Public apps"]
                InternalApps["Internal apps"]
            end
        end
    end

    Tailnet -->|kubectl and talosctl| CP1
    Tailnet --> Platform
    Tailnet --> General
    CP --> NAT
    Platform --> NAT
    General --> NAT
    NAT --> Internet

The Kubernetes API endpoint is the first control plane node’s private IP:

locals {
  cluster_endpoint = "https://${local.control_plane_private_ips[0]}:6443"
}

That endpoint is only useful if you are already inside the private network through Tailscale.

Building the Talos Image

Before Terraform could create any nodes, I needed a Talos image that Hetzner could boot.

I started this cluster on Talos v1.11.3. The later v1.12.6 upgrade came from an operational incident, not the initial design.

Hetzner does not give you Talos as an image option, so I build my own snapshot with Packer. The flow is based on hcloud-talos/terraform-hcloud-talos.

It starts a temporary Hetzner server, downloads the Talos raw image from the Talos Image Factory, writes it to disk, and saves the result as a snapshot.

variable "talos_version" {
  type    = string
  default = "v1.11.3"
}

source "hcloud" "talos" {
  rescue       = "linux64"
  image        = "debian-11"
  location     = "nbg1"
  server_type  = "cx22"
  ssh_username = "root"

  snapshot_name = "talos-${var.talos_version}-amd64"
  snapshot_labels = {
    type    = "infra"
    os      = "talos"
    version = var.talos_version
    arch    = "amd64"
  }
}

The label part is the important bit. Terraform can later find the image by selector instead of relying on snapshot name:

data "hcloud_image" "talos" {
  with_selector = "os=talos,type=infra,version=${var.talos_version},arch=amd64"
}

Worker Pools

Before creating the machines, I needed a way to describe what kind of nodes I wanted.

This is basically the same idea as node pools in managed Kubernetes offerings. GKE, EKS, and AKS all let you create groups of nodes with different sizes, labels, or taints. I wanted the same mental model.

Each pool also gets its own Hetzner placement group. That tells Hetzner to spread the nodes in that pool across different physical hosts where possible. It does not make the pool highly available, but it avoids the failure mode where every platform worker ends up on the same machine.

The pool config looks like this:

worker_pools = {
  platform = {
    count      = 3
    sku        = "cx33"
    cidr       = "10.0.128.32/27"
    datacenter = "nbg1-dc3"
    labels     = { purpose = "platform" }
  }

  general = {
    count      = 3
    sku        = "cx23"
    cidr       = "10.0.128.64/27"
    datacenter = "nbg1-dc3"
    labels     = { purpose = "general" }
  }
}

This makes the Terraform code easier to reason about. It lets me create named groups of machines with known CIDR ranges, placement groups, and labels.

Terraform Creates the Machines

The node resources are just regular Hetzner servers, but with the public network disabled.

resource "hcloud_server" "control_plane" {
  count = var.control_plane.count

  name        = "${local.cluster_name}-cp-${count.index + 1}"
  datacenter  = var.control_plane.datacenter
  image       = data.hcloud_image.talos.id
  server_type = var.control_plane.sku

  public_net {
    ipv4_enabled = false
    ipv6_enabled = false
  }

  network {
    network_id = var.network_id
    ip         = local.control_plane_ips[count.index]
  }
}

The worker pool map from the previous section gets flattened into individual servers. The code is not elegant, but the outcome is simple: if I add another worker to the general pool, it gets the next private IP in that pool and the right labels.

locals {
  workers_flat = merge([
    for pool_name, pool_config in var.worker_pools : {
      for i in range(1, pool_config.count + 1) :
      "${pool_name}-${i}" => {
        pool       = pool_name
        index      = i
        sku        = pool_config.sku
        datacenter = pool_config.datacenter
        labels     = pool_config.labels
      }
    }
  ]...)
}

Talos Bootstraps Kubernetes

Once the servers exist, the Talos Terraform provider takes over. It generates machine secrets, creates control plane and worker configs, applies patches, bootstraps the first control plane node, waits for Talos cluster health, and gives me a kubeconfig.

There is one base config for control plane nodes, and one worker base config per pool:

data "talos_machine_configuration" "control_plane" {
  cluster_name       = local.cluster_name
  cluster_endpoint   = local.cluster_endpoint
  machine_type       = "controlplane"
  machine_secrets    = talos_machine_secrets.this.machine_secrets
  talos_version      = var.talos_version
  kubernetes_version = var.kubernetes_version
}

data "talos_machine_configuration" "worker" {
  for_each = var.worker_pools

  cluster_name       = local.cluster_name
  cluster_endpoint   = local.cluster_endpoint
  machine_type       = "worker"
  machine_secrets    = talos_machine_secrets.this.machine_secrets
  talos_version      = var.talos_version
  kubernetes_version = var.kubernetes_version
}

Then Terraform applies the patched control-plane config to each control-plane node:

resource "talos_machine_configuration_apply" "control_plane" {
  count = var.control_plane.count

  client_configuration        = talos_machine_secrets.this.client_configuration
  machine_configuration_input = data.talos_machine_configuration.control_plane.machine_configuration
  node                        = local.control_plane_private_ips[count.index]

  config_patches = [
    yamlencode(local.control_plane_patch),
  ]
}

Workers follow the same pattern, except the patch comes from the worker pool:

resource "talos_machine_configuration_apply" "worker" {
  for_each = local.workers_flat

  client_configuration        = talos_machine_secrets.this.client_configuration
  machine_configuration_input = data.talos_machine_configuration.worker[each.value.pool].machine_configuration
  node                        = flatten(hcloud_server.worker[each.key].network)[0].ip

  config_patches = [
    yamlencode(local.worker_pool_patches[each.value.pool]),
  ]
}

Nodes need the right installer image, node IP selection, default route, pod and service CIDRs, and CNI behavior.

This is the part I messed up and later caused the first real failure.

A simplified version of the patch, showing the final intent, looks like this:

common_patch = {
  machine = {
    install = {
      disk  = var.install_disk
      image = "factory.talos.dev/installer/${var.talos_schematic_id}:${var.talos_version}"
    }

    kubelet = {
      extraArgs = {
        "cloud-provider"             = "external"
        "rotate-server-certificates" = true
      }
      nodeIP = {
        validSubnets = local.node_cidrs
      }
    }

    network = {
      interfaces = [
        {
          interface = "eth0"
          routes = [{
            network = "0.0.0.0/0"
            gateway = var.gateway
          }]
          dhcp = true
        },
        {
          interface = "eth1"
          ignore    = true
        }
      ]
    }

    features = {
      hostDNS = {
        enabled              = true
        forwardKubeDNSToHost = true
        resolveMemberNames   = true
      }
    }
  }

  cluster = {
    network = {
      podSubnets     = [var.pod_ipv4_cidr]
      serviceSubnets = [var.service_ipv4_cidr]
      cni = {
        name = "none"
      }
    }
    proxy = {
      disabled = true
    }
  }
}

Most of that is ordinary cluster setup. The important bits are: nodeIP.validSubnets, the default route, and the interface names. If those are wrong, the cluster does not fail in an obvious way. It half-works, which is worse.

The control plane patch adds the other important private-networking detail:

cluster = {
  etcd = {
    advertisedSubnets = [var.control_plane.cidr]
  }

  controllerManager = {
    extraArgs = {
      "cloud-provider"           = "external"
      "node-cidr-mask-size-ipv4" = "24"
      "bind-address"             = "0.0.0.0"
    }
  }
}

That keeps etcd on the control-plane private CIDR and makes Kubernetes pod CIDR allocation line up with the cluster network.

The worker patch mostly adds the pool labels, so nodes become purpose=platform or purpose=general. After the configs are applied, Terraform bootstraps only the first control plane node:

resource "talos_machine_bootstrap" "this" {
  client_configuration = talos_machine_secrets.this.client_configuration
  node                 = local.control_plane_private_ips[0]
}

The First Failure Was Networking

The first failure was simple: the nodes did not agree on their own network identity.

Private-only Hetzner machines still need a default route for egress. In my setup, the node route goes to the subnet gateway, and Hetzner’s network route sends outbound traffic to the NAT gateway.

Node -> Subnet Gateway -> NAT Gateway -> Internet

In practice, the Talos machine config needs to point that route at the right interface, and kubelet needs to pick the right node IP.

I had a few false starts here. At one point I assumed the private network interface was enp7s0. Then I tried routes on both enp7s0 and eth0, with eth1 ignored. Hetzner Cloud VMs were using eth0 for the network path I needed, so eth0 was the path that actually mattered.

The other subtle part was nodeIP.validSubnets. My first attempt pointed kubelet at the broader subnet. Talos and Kubernetes do better when the allowed node IP ranges are exactly the ranges where nodes live.

So the module builds that list from the control plane CIDR and each worker pool CIDR:

locals {
  node_cidrs = concat(
    [var.control_plane.cidr],
    [for _, pool_config in var.worker_pools : pool_config.cidr]
  )
}

Configuring networking is not easy. It is a chain of small settings that all need to agree: Hetzner routes, Talos interfaces, kubelet node IPs, etcd advertised subnets, and Kubernetes pod CIDR allocation.

Cilium First, Then Cloud Integrations

Once the nodes agreed on their private addresses and routes, the next step was getting the cluster network itself working.

I install Cilium with Helm after Talos bootstraps Kubernetes. It runs in native routing mode and replaces kube-proxy.

resource "helm_release" "cilium" {
  name       = "cilium"
  repository = "https://helm.cilium.io/"
  chart      = "cilium"
  namespace  = "kube-system"

  set {
    name  = "routingMode"
    value = "native"
  }

  set {
    name  = "kubeProxyReplacement"
    value = "true"
  }
}

After Cilium is in place, the Hetzner Cloud Controller Manager can run. That order matters because the CCM depends on a working cluster network and is responsible for Hetzner-specific behavior like load balancers and node metadata.

The CSI driver and metrics server follow the same idea. Terraform installs the base pieces that make the cluster usable.

GitOps from Day One

Terraform installs ArgoCD because something needs to install ArgoCD. After that bootstrap step, the responsibilities are clear: Terraform owns infrastructure and the first bootstrap, ArgoCD owns platform components.

This is where the earlier platform pool starts to make sense. ArgoCD should not land randomly on the same general workers as application traffic.

global:
  nodeSelector:
    purpose: platform

The Terraform side creates the namespace, repository credentials, and Helm release. ArgoCD itself is exposed on the internal ingress as argocd.int.noreturn.dev. That keeps the GitOps UI private, but still easy to reach from my laptop through Tailscale.

From there, the platform repository can install cert-manager, external-secrets, ingress controllers, monitoring, and the rest.

Then Real Workloads Showed Up

The original cluster came up on Talos v1.11.3. It worked. And then snapbyte.dev started running on the general workers.

That is when the cluster stopped being a learning exercise.

A high-churn set of CronJobs in snapbyte started leaving the cluster in a bad state. One worker showed SystemOOM, Grafana had gaps in node metrics, disk I/O looked suspicious, and Talos showed hundreds of containerd-shim-runc-v2 processes on a hot worker.

At first it looked like a disk problem. Then it looked like a workload problem. Then it looked like both.

Restarting the affected workers would probably have been fine, but I got curious around 1am one night and kept digging.

Eventually, the runtime became part of the investigation because those nodes were on Talos v1.11.3 with containerd 2.1.4. That pointed me to this issue: containerd-shim processes leak during high pod churn.

By 3am, curiosity had turned into an in-place Talos upgrade to v1.12.6, moving the nodes to containerd 2.1.6, and a much more careful rollout process than I had planned when I first built this cluster.

But that is its own post.

What I Learned

• Private-only nodes still need a default route for outbound traffic
• The Talos interface names have to match what the Hetzner VM actually uses
• nodeIP.validSubnets should only include the ranges where Kubernetes nodes actually live: the control plane CIDR and each worker pool CIDR
• etcd.advertisedSubnets should stay on the control-plane private CIDR
• The service and pod CIDRs are not random ranges. I used 10.0.192.0/21 for services and 10.0.200.0/19 for pods, outside the node ranges in 10.0.128.0/24.
• The pod range is /19 because Kubernetes allocates one /24 pod CIDR per node in this setup. A /19 contains 32 /24 ranges, so the cluster has room for up to 32 nodes without overlapping the node or service ranges.
• Talos passes those ranges to Kubernetes, the controller manager allocates /24 pod CIDRs to nodes from the pod range, and Cilium routes traffic based on that Kubernetes view of the network.
• Running real workloads changes the meaning of the cluster. Once snapbyte.dev was running there, it was not a learning exercise anymore. A personal cluster is still production if real services depend on it.

Next

Next up: the Talos upgrade from v1.11.3 to v1.12.6, why I did it, what I checked before each node, and how a small snapbyte incident became a container runtime debugging session.

Now we are constantly witnessing how much regular people lack imagination. Everybody thinks they can get rich by building the exact same habit tracker.

Our clothes are the same, cars look the same, and houses decorated with cheap, soulless decor are the same.

We all go home and watch the same shows. It doesn’t matter what language they are in because we can watch them in our native tongue.

We have more tools than ever, but fewer ideas. Everything is copied, translated, repackaged, and sold back to us.

At what point did convenience stop helping us imagine and start doing the imagining for us?

Lie: “PERSONALITY.md”

I cannot tell you how much I hate seeing AGENTS.md, skills, or BS like PERSONALITY.md files full of crap such as “You are a senior software engineer”. You are just fucking role-playing. You are merely telling an LLM to use the lingo a senior software engineer might use.

They can only improve or change during training, post-training, or fine-tuning. That is it. People confuse surface behavior with actual change. They are only as good as the next benchmark asking whether they should drive to the car wash or walk.

We are experiencing a mass hysteria in a very small echo chamber, thinking we are all doomed and everybody is going to lose their jobs. You know what? The world is not just a Hacker News echo chamber. Most people are using ChatGPT to interpret their dreams or ask questions rather than Googling.

That is just context

Your coding agent’s “nature” changes for a few seconds on some random H100 shared by God knows how many people in a desert somewhere in the US. That is not a soul. It is context. That personality, that quirkiness, that thing that makes it call you “dipshit” only happens because a whole wall of text is getting sent back and forth between your computer and the data center.

The worst mistake is treating imitation like conscience.

You cannot expect them to feel remorse when they decide to bomb some random apartment block full of kids in a Middle Eastern country. They don’t reason, think or feel. They are just an imitation, an Artificial Artificial Intelligence (AAI). Of course, for the next chat session you could add a line to WEAPONOFMASSDESTRUCTION[.]md to say “Previously you killed 250 kids. You should feel sad. Consider not killing too many kids”.

You can change the voice, the costume, the script. Not the nature. You are always absolutely right.

The cluster post was about why I built my own Kubernetes cluster. This one is about the private network that makes the rest possible. I wanted the cluster to live in a VPC-style network with zero public node IPs, reachable only from my own internal network.

Tailscale is the bridge here. I keep Raspberry Pi Zero exit nodes across Europe with friends for VPN use, so I already use Tailscale a lot. That’s why it was the obvious choice for private access.

What I Wanted from the Network

I wanted three things:

1. Private-only cluster nodes: no public IPs anywhere except one gateway.
2. Tailnet-only kubectl: cluster access stays inside the Tailnet via subnet routes.
3. Separation of public vs. private apps: snapbyte.dev stays public, internal tools stay private.

That meant I needed a VPC-style network that could handle egress through a single NAT gateway, and ingress through an internal load balancer with a private IP address. The cluster could stay dark on the public internet, but still be reachable from my Tailnet.

Network Shape

Hetzner does not call this a VPC, but the architecture is the same idea: an isolated private network with a controlled egress point. I kept the design small and opinionated:

• Network CIDR: 10.0.0.0/16
• Subnet: 10.0.128.0/24
• Single NAT gateway with a public IP for outbound traffic
• Private-only Kubernetes nodes inside the subnet

The design is intentionally simple: network + subnet, a minimal firewall, a NAT gateway box, and Tailscale ACLs. The base network shape is small and explicit:

resource "hcloud_network" "vpc_network" {
  name     = "vpc-network"
  ip_range = "10.0.0.0/16"
}

resource "hcloud_network_subnet" "vpc_subnet" {
  network_id   = hcloud_network.vpc_network.id
  type         = "cloud"
  ip_range     = "10.0.128.0/24"
}

The routing flow is simple and deterministic:

Cluster Node -> Subnet Gateway -> NAT Gateway -> Internet

The NAT gateway is the only box with a public IP. Everything else is private.

flowchart LR
    subgraph Tailnet["Tailnet"]
        Laptop[Laptop]
        Phone[Phone]
    end
    Internet((Internet))

    subgraph VPC["VPC-style network 10.0.0.0/16"]
        subgraph Subnet["Subnet 10.0.128.0/24"]
            subgraph K8S["Kubernetes Nodes"]
                PublicApps["Public apps"]
                InternalApps["Internal apps"]
            end
            PubLB["Public LB"]
            PrivLB["Private LB"]
            NAT[NAT Gateway]
        end
    end

    Internet --> PubLB
    PubLB --> PublicApps
    Tailnet --> PrivLB
    PrivLB --> InternalApps
    Tailnet -->|kubectl over subnet route| K8S
    K8S -->|egress| NAT
    NAT --> Internet

The NAT Gateway and Egress

The NAT gateway is a Debian 12 server with two jobs: perform MASQUERADE NAT for the subnet and advertise the subnet into my Tailnet.

The NAT gateway is just a Hetzner server with a public IP and a default route for the network:

resource "hcloud_server" "nat_gateway" {
  name        = "vpc-nat-gateway"
  image       = "debian-12"
  server_type = "cx23"
  user_data   = data.cloudinit_config.nat_gateway_cloud_init.rendered

  network {
    network_id = hcloud_network.vpc_network.id
    ip         = cidrhost(hcloud_network_subnet.vpc_subnet.ip_range, 1)
  }
}

resource "hcloud_network_route" "nat_gateway_route" {
  network_id  = hcloud_network.vpc_network.id
  destination = "0.0.0.0/0"
  gateway     = flatten(hcloud_server.nat_gateway.network)[0].ip
}

Cloud-init does the two important bits: NAT and Tailscale subnet routing.

#cloud-config
write_files:
  - path: /etc/network/interfaces
    content: |
      post-up echo 1 > /proc/sys/net/ipv4/ip_forward
      post-up iptables -t nat -A POSTROUTING -s '${network_ipv4_cidr}' -o eth0 -j MASQUERADE

runcmd:
  - ['tailscale', 'up', '--auth-key=${tailscale_auth_key}', '--advertise-routes=${subnet_ip_range}']

The important detail is that cluster nodes never need public IPs. They just route their outbound traffic to the NAT gateway, and the gateway handles the rest.

The gateway sits behind a narrow firewall, so only the minimum network paths exist in and out.

Private Access with Tailscale

Tailscale is not just for logging in. It’s the entry point into the private network.

The network Terraform code manages Tailscale ACLs and subnet routes. When the gateway comes up, it advertises 10.0.128.0/24 to the Tailnet. My laptop and phone can reach any node or service inside the subnet.

resource "tailscale_acl" "vpc_nat_gateway_acl" {
  acl = jsonencode({
    tagOwners = { "tag:gateway" = ["autogroup:admin"] }
    autoApprovers = { routes = { "10.0.128.0/24" = ["tag:gateway"] } }
    acls = [{ action = "accept", src = ["autogroup:admin"], dst = ["*:*"] }]
  })
}

This gives me a private network that feels like a LAN, but works across continents.

Internal Subdomains on a Private Load Balancer

The Kubernetes cluster sits behind two ingress controllers: one public, one private. The private one uses a Hetzner load balancer with a private IP on the subnet. That means I can map internal DNS names to that private IP and access internal apps only from my Tailnet.

The load balancers are not created here. They show up when the two different nginx ingress controllers are deployed, and hccm provisions them: one with public IP and one with private IP.

So I can run public projects like snapbyte.dev on the public ingress, while keeping ArgoCD, Grafana, or other internal tools locked to the private ingress. The internal subdomains resolve inside my Tailnet, not on the public internet.

Example DNS split:

grafana.int.noreturn.dev -> 10.0.128.x (private LB IP)

snapbyte.dev -> <public-lb-ip>

Minimal Exposure

The firewall rules are intentionally minimal:

• Inbound: only WireGuard/Tailscale
• Outbound: DNS, NTP, HTTPS, and STUN

There is no SSH open to the internet. The NAT gateway is the only exposed node, and it is still only reachable through Tailscale.

What I Learned

• A small /24 subnet is enough for a personal cluster when you plan ranges up front.
• Subnet routes make the private network feel local without exposing anything publicly.

Next

The network is the base layer. Next up is the Kubernetes provisioning piece: Talos images, node pools, and how I keep public and private ingress separate without leaking the cluster to the internet.

I’ve been working as a platform engineer at Stack Overflow since 2022, designing platform components, writing documentation, and making developers’ lives easier. Recently, I moved to the infrastructure platform team. Now I’m in unfamiliar territory, trying to abstract away multi-cloud infrastructure details from other platform engineering teams.

For years, I worked with Kubernetes professionally, but mostly at a level that hid the infrastructure details. CIDR planning? Not my thing. DNS Zones? Already done. CNI? Things were already talking to each other. I could implement platform components, deploy applications, and troubleshoot pods. But now I’m working one level below, the landscape where I don’t have much experience.

I needed to learn infrastructure deeply. So I built my own Kubernetes cluster from scratch on Hetzner Cloud. Cost? About £50/month for a production-ready setup with high availability, monitoring, and GitOps. That’s less than just the control plane on AWS, GCP, or Azure.

Why I Needed My Own Cluster

I’ve been building distributed systems since 2015 when I started at Hepsiburada in Istanbul, one of the largest e-commerce sites in Europe. I led the team that introduced the marketplace platform there and brought containers to the company. Back then, Kubernetes was still in alpha, and I was amazed by it. A decade of this kind of work changes how you approach problems.

Whenever I start a side project, the architecture immediately becomes distributed. I hate when I need a background job processor or some functionality the web framework doesn’t offer. The monolith I promise myself won’t become a distributed system? It always does. I don’t think this is overengineering anymore. It’s just how I think now.

My Kubernetes and Distributed Systems Experience is a Curse

Take my current side project, snapbyte.dev. It’s a personal tech digest service that collects links from HN, /r/programming, lobste.rs, extracts content, categorizes it, generates summaries, and builds personalized digests based on user configuration. Started as a single Phoenix app, but quickly split into multiple services once I hit limitations with Elixir’s content extraction libraries.

For years at work, I’ve been shipping with Kubernetes. It’s my default. When I think about deploying, I think in pods and services. But for personal projects, there’s always been friction: I want to use the tools I’m comfortable with, but I can’t justify £100+ monthly for managed Kubernetes on a side project.

Then I started seeing Hetzner posts on Hacker News. This one in particular caught my attention. People were running production Kubernetes clusters on Hetzner for £100 per month. The same setup on AWS, GCP, or Azure would cost 5-10x that amount.

Suddenly, the math worked. Affordable Kubernetes plus infrastructure learning for my day job.

What I Wanted to Learn

I needed to fill specific knowledge gaps for my day job at the infrastructure platform team.

Network segmentation and CIDR planning: How do you actually design a VPC? What CIDR blocks should you allocate for nodes vs. pods vs. services? I’d seen /16 and /24 notations for years but never had to think about planning subnet ranges.

Certificate management: How does cert-manager work with Let’s Encrypt? What’s a DNS-01 challenge vs. HTTP-01? How do you automate wildcard certificates?

ArgoCD and ApplicationSets: Stack Overflow uses ArgoCD, but I only used FluxCD at previous companies like Redgate and TrueLayer. I wanted to understand ApplicationSets, and whether ArgoCD’s UI and app-of-apps pattern would be more intuitive for managing platform components.

I also wanted to work with Talos Linux. No SSH, no package manager, no traditional OS at all. Just YAML for managing everything. More YAML. Always more YAML.

What I Built

Here’s what I ended up with: a private VPC with no public IPs on cluster nodes, dual ingress controllers (public + private via Tailscale), and GitOps from day one. The NAT gateway is the only node exposed to the internet, with SSH blocked and only Tailscale ports open.

Architecture

flowchart TB
    Tailnet((Tailnet))
    InternetIn((Internet))
    InternetOut((Internet))
    
    subgraph VPC["Private VPC (10.0.0.0/16)"]
        
        subgraph Subnet["Subnet (10.0.128.0/24)"]
        NAT[NAT Gateway - Public IP]
        ExtLB[Hetzner LB - External]
        IntLB["Hetzner LB - Internal (10.0.128.3)"]
        
        subgraph K8S["Kubernetes"]

            subgraph CP["Control Plane (10.0.128.16/28)"]
            end
            
            subgraph PW["Platform Workers (10.0.128.32/27)"]
                PlatformComponents[Platform Components]
            end
            
            subgraph GW["General Workers (10.0.128.64/27)"]
                PublicApps[Public apps]
                PrivateApps[Private apps]
            end

        end
        
        IntLB --> PW
        IntLB --> GW
        ExtLB --> GW
        end
    end
    
    CP --> NAT
    PW --> NAT
    GW --> NAT
    InternetIn --> ExtLB
    Tailnet --> Subnet & IntLB
    NAT --> InternetOut
    
    style Tailnet fill:#d1ecf1,stroke:#0c5460,stroke-width:2px
    style InternetIn fill:#cce5ff,stroke:#004085,stroke-width:3px
    style InternetOut fill:#cce5ff,stroke:#004085,stroke-width:3px
    style VPC fill:#f5f5f5,stroke:#666,stroke-width:2px
    style Subnet fill:#e8f4f8,stroke:#5a6268,stroke-width:2px
    style NAT fill:#ffe6cc,stroke:#ff9933,stroke-width:2px
    style ExtLB fill:#ffe6cc,stroke:#ff9933,stroke-width:2px
    style IntLB fill:#fff9e6,stroke:#ffcc00,stroke-width:2px
    style K8S fill:#e9ecef,stroke:#495057,stroke-width:2px
    style CP fill:#d1e7dd,stroke:#0f5132,stroke-width:2px
    style PW fill:#cfe2ff,stroke:#084298,stroke-width:2px
    style GW fill:#f8d7da,stroke:#842029,stroke-width:2px
    style PlatformComponents fill:#e7f1ff,stroke:#084298
    style PublicApps fill:#f5c6cb,stroke:#842029
    style PrivateApps fill:#f5c6cb,stroke:#842029

Key Decisions

IaC: The whole infrastructure is version controlled. terraform apply if I need more nodes.

NAT Gateway security: The only node with a public IP is the NAT gateway. SSH port (22) is blocked via firewall; only Tailscale ports are open. Subnet routes are advertised to my Tailnet from this server.

Talos: Used Talos Terraform provider to provision the cluster and setup a nodepool logic.

Private-only cluster nodes: No public IPs on any Kubernetes nodes. All access goes through Tailscale.

GitOps from day one: ArgoCD manages all platform components: cert-manager, external-secrets, otel-collector, etc. No more kubectl apply.

Multiple “nodepools”: Workloads and platform components are deployed into different nodes based on nodeSelector. Workloads go to purpose: general and platform components go to purpose: platform.

Dual ingress: Two separate ingress controllers, both with Hetzner LoadBalancers. External ingress gets a public IP for public-facing apps. Internal ingress gets a private IP for tools like ArgoCD, Grafana, and OpenWebUI. I use Cloudflare NS records to delegate the int subdomain to Hetzner’s nameservers, where A and CNAME records point to the internal load balancer’s private IP. Internal services are only accessible via Tailscale.

Cost Breakdown

About £50/month total:

• 10x servers (NAT Gateway + control plane + workers): ~£35
• 1x public ip: ~£0.5
• 2x Hetzner Load Balancers: £10
• 8x 10GB storage: £4
• Bandwidth: 20TB included

Compare this to managed Kubernetes:

• AWS EKS: £65/month for control plane alone
• GKE/AKS: Similar pricing
• Add nodes, load balancers, NAT gateway: £150-200/month minimum

The savings are real, but so is the learning curve.

What’s Next

I might write more about building this platform: the network design, Talos provisioning, Cilium setup, ArgoCD workflows, and the full cost breakdown. But for now, this covers why I built it and what the architecture looks like.

If you’re interested in running your own Kubernetes cluster affordably, or you’re a platform engineer wanting to understand infrastructure more deeply, hopefully this gives you some inspiration.

You might be wondering why I start with a blog post explaining the Bloom Filter while I don’t have single clue about how to create a distributed datastore? My answer is simple: “I like the idea behind it”.

Before I get into the details of the Bloom filters, I want to give our backstory that will help you understand why we started building something we’d enjoy during our spare time that will never be production ready.

The backstory

My friend Ibrahim and I are always fascinated by complex software and distibuted systems - We’ve been working together more than 5 years (we got old dude) and we were lucky enough to work for the largest e-commerce company in Europe. We battled our way solving many different problems that distributed systems can offer. We both moved to Cambridge, UK and still fighting against distributed world villains.

Let’s explore the mystic land of probabilistic data structures by implementing a Bloom Filter.

What the hell is a Bloom Filter

You might also want to read Bloom filters debunked.

A Bloom filter is a method for representing a set $A = {a_1, a_2,\ldots, a_n}$ of n elements (also called keys) to support membership queries. It was invented by Burton Bloom in 1970 and was proposed for use in the web context by Marais and Bharat as a mechanism for identifying which pages have associated comments stored within a CommonKnowledge server. ¹

It is a space-efficient probabilistic data structure that is used to answer a very simple question: is this element a member of a set?. A Bloom filter does not store the actual elements, it only stores the membership of them.

False positive matches are possible, but false negatives are not – in other words, a query returns either “possibly in set” or “definitely not in set”. ² Unfortunately, this also means items cannot be removed from the Bloom Filter (Some other element or group of elements may be hashed to the same indices).

Because of its nature of being probabilistic, the Bloom Filter trades space and performance for accuracy. This is much like the CAP theorem, we choose performance over accuracy.

Bloom filters have some interesting use cases. For example, they can be placed on top of a datastore. When a key is queried for its existence and the filter does not have it, we can skip querying the datastore entirely.

flowchart LR
    q1["Do you have<br/>'Key A'?"]
    q2["Do you have<br/>'Key B'?"]
    q3["Do you have<br/>'Key C'?"]

    subgraph filter["FILTER"]
        f1["Filter: No"]
        f2["Filter: Yes"]
        f3["Filter: Yes<br/>(false positive)"]
    end

    subgraph storage["STORAGE"]
        s2["Storage:<br/>Yes"]
        s3["Storage:<br/>No"]
    end

    q1 --> f1
    f1 -.->|No| q1

    q2 --> f2
    f2 -->|disk access| s2
    s2 -.->|value of key B| f2
    f2 -.->|Yes| q2

    q3 --> f3
    f3 -->|disk access| s3
    s3 -.->|No| f3
    f3 -.->|No| q3

    style f1 fill:#cc241d,stroke:#fb4934,color:#fbf1c7
    style f2 fill:#98971a,stroke:#b8bb26,color:#fbf1c7
    style f3 fill:#98971a,stroke:#b8bb26,color:#fbf1c7
    style s2 fill:#98971a,stroke:#b8bb26,color:#fbf1c7
    style s3 fill:#cc241d,stroke:#fb4934,color:#fbf1c7

Figure 1: Example usage of a Bloom filter. Original drawing

How does it work

The idea behind Bloom filter is very simple: Allocate an array $v$ of $m$ bits, each bit in the array is initially set to $0$, and then choose $k$ independent hash functions $h_1, h_2, …, h_k$, each with range ${1,…,m}$.

The Bloom filter has two operations just like a standard set:

Insertion

When an element $a \in A$ is added to the filter, the bits at positions $h_1(a), h_2(a), …, h_k(a)$ in $v$ are set to $1$. In simpler words, the new element is hashed by $k$ number of functions and modded by $m$, resulting in $k$ indices into the bit array. Each bit at the respective index is set.

flowchart TB
    a[a]

    h1[h₁]
    h2[h₂]
    h3[h₃]

    a -->|hash| h1
    a -->|hash| h2
    a -->|hash| h3

    subgraph "Bit Array"
        b0[1]
        b1[0]
        b2[0]
        b3[0]
        b4[0]
        b5[0]
        b6[0]
        b7[1]
        b8[1]
        b9[0]
    end

    h1 -.->|set bit 0| b0
    h2 -.->|set bit 7| b7
    h3 -.->|set bit 8| b8

    style a fill:#ebdbb2,stroke:#3c3836,stroke-width:2px
    style h1 fill:#b16286,stroke:#d3869b
    style h2 fill:#458588,stroke:#83a598
    style h3 fill:#d65d0e,stroke:#fe8019

Figure 2: Adding elements to a Bloom filter ($m = 10$, $k = 3$). Original drawing

Query

To query the membership of an element $b$, we check the bits at indices $h_1(b), h_2(b), …, h_k(b)$ in $v$. If any of them is $0$, then certainly $b$ is not in the set $A$. Otherwise, we assume that $b$ is in the set although it’s possible that some other element or group of elements hashed to the same indices. This is called a false positive. We can target a specific probability of false positives by selecting an optimal value of $m$ and $k$ for up to $n$ insertions.

A Bloom filter eventually reaches a point where all bits are set, which means every query will indicate membership, effectively making the probability of false positives $1$. The problem with this is it requires a priori knowledge of the data set in order to select optimal parameters and avoid “overfilling”. ³

Finding optimal $k$ and $m$

We can derive optimal $k$ and $m$ based on $n$ and a chosen probability of false positives $P_{FP}$.

If you want to learn about how the above formulae are derived, you might want to pay a visit here.

Rust implementation

You can find the full implementation here. Huge thanks to @xfix for fixing hashers initialization with the same seed and @dkales for spotting an issue with the ordering of operations in index calculation.

Finally! It is time to write some rust :heart_eyes:. I am simultaneously implementing the bloom filter whilst writing this blog post. If you don’t believe me then check the below command:

cargo new --lib plum

Let’s continue with the dependencies. There is only one dependency and we will use it to create $v$.

[dependencies]
bit-vec = "0.6"

We will declare a new struct StandardBloomFilter to encapsulate required fields $k$ (optimal number of hash functions), $m$ (optimal size of the bit array), $v$ (the bit array), hash functions and a marker to tell rust compiler that our struct “owns” a T.

extern crate bit_vec;

use bit_vec::BitVec;
use std::collections::hash_map::{DefaultHasher, RandomState};
use std::hash::{BuildHasher, Hash, Hasher};
use std::marker::PhantomData;

pub struct StandardBloomFilter<T: ?Sized> {
    bitmap: BitVec,
    optimal_m: u64,
    optimal_k: u32,
    hashers: [DefaultHasher; 2],
    _marker: PhantomData<T>,
}

Careful readers will think that I made a mistake in the declaration of hashers array because of the requirement of $k$ independent hash functions. It was indeed intentional here’s why:

Why two hash functions? Kirsch and Mitzenmacher demonstrated in their paper that using two hash functions $h_1(x)$ and $h_2(x)$ to simulate additional hash functions of the form $g_i(x) = h_1(x) + {i}{h_2(x)}$ can be usefully applied to Bloom filters. This leads to less computation and potentially less need for randomness in practice. ⁴ This formula may appear similar to the use of pairwise indenpendent hash functions. Unfortunately, there is no formal connection between the two techniques.

I mentioned earlier that the Bloom Filter has two operations like a standard set: insert and query. We will implement those two operations along with constructor-like new method.

impl<T: ?Sized> StandardBloomFilter<T> {
    pub fn new(items_count: usize, fp_rate: f64) -> Self {
        // ...snip
    }
}

new calculates the size of the bitmap ($v$) and optimal_k ($k$) and then instantiates a StandardBloomFilter.

impl<T: ?Sized> StandardBloomFilter<T> {
    pub fn new(items_count: usize, fp_rate: f64) -> Self {
        let optimal_m = Self::bitmap_size(items_count, fp_rate);
        let optimal_k = Self::optimal_k(fp_rate);
        let hashers = [
            RandomState::new().build_hasher(),
            RandomState::new().build_hasher(),
        ];
        StandardBloomFilter {
            bitmap: BitVec::from_elem(optimal_m as usize, false),
            optimal_m,
            optimal_k,
            hashers,
            _marker: PhantomData,
        }
    }

    // ...snip

    fn bitmap_size(items_count: usize, fp_rate: f64) -> usize {
        let ln2_2 = core::f64::consts::LN_2 * core::f64::consts::LN_2;
        ((-1.0f64 * items_count as f64 * fp_rate.ln()) / ln2_2).ceil() as usize
    }

    fn optimal_k(fp_rate: f64) -> u32 {
        ((-1.0f64 * fp_rate.ln()) / core::f64::consts::LN_2).ceil() as u32
    }

    // ...snip
}

Let’s run these calculations on Rust Playground.

bitmap_size: 9585059
optimal_k: 7

This looks really promising! A Bloom Filter that represents a set of $1$ million items with a false-positive rate of $0.01$ requires only $9585059$ bits ($~1.14\mathrm{MB}$) and 7 hash functions.

We managed to construct a Bloom Filter so far and it is time to implement insert and contains methods. Their implementations are dead simple and they share the same code to calculate indexes of the bit array.

impl<T: ?Sized> StandardBloomFilter<T> {
    // ...snip

    pub fn insert(&mut self, item: &T)
    where
        T: Hash,
    {
        let (h1, h2) = self.hash_kernel(item);

        for k_i in 0..self.optimal_k {
            let index = self.get_index(h1, h2, k_i as u64);

            self.bitmap.set(index, true);
        }
    }

    pub fn contains(&mut self, item: &T) -> bool
    where
        T: Hash,
    {
        let (h1, h2) = self.hash_kernel(item);

        for k_i in 0..self.optimal_k {
            let index = self.get_index(h1, h2, k_i as u64);

            if !self.bitmap.get(index).unwrap() {
                return false;
            }
        }

        true
    }
}

The above methods depend on two other methods that we haven’t implemented yet: hash_kernel and get_index. hash_kernel is going to be the one where the actual “hashing” happens. It will return the hash values of $h_1(x)$ and $h_2(x)$.

impl<T: ?Sized> StandardBloomFilter<T> {
    // ...snip

   fn hash_kernel(&self, item: &T) -> (u64, u64)
    where
        T: Hash,
    {
        let hasher1 = &mut self.hashers[0].clone();
        let hasher2 = &mut self.hashers[1].clone();

        item.hash(hasher1);
        item.hash(hasher2);

        let hash1 = hasher1.finish();
        let hash2 = hasher2.finish();

        (hash1, hash2)
    }
}

We could’ve used $128$ bit MurmurHash3 and returned upper $64$ bit as hash1 and the lower as hash2 but to keep this implementation even simpler (this is how Google Guava Bloom Filter implementation currently works) and not to rely on any other additional dependencies I decided to continue with DefaultHasher - see SipHash

Now, it is time to make the final touch. We are going to implement get_index by using $g_i(x) = h_1(x) + {i}{h_2(x)}$ to simulate more than two hash functions.

impl<T: ?Sized> StandardBloomFilter<T> {
    // ...snip

    fn get_index(&self, h1: u64, h2: u64, k_i: u64) -> usize {
        (h1.wrapping_add((k_i).wrapping_mul(h2)) % self.optimal_m) as usize
    }

    // ...snip

We are finally there

:tada: :tada: :tada: We’ve just finished implementing a fast variant of a standard Bloom Filter but there is still one thing missing - We didn’t write any tests.

Let’s add two simple test cases and validate our implementation.

#[cfg(test)]
mod tests {
    use super::*;

    #[test]
    fn insert() {
        let mut bloom = StandardBloomFilter::new(100, 0.01);
        bloom.insert("item");
        assert!(bloom.contains("item"));
    }

    #[test]
    fn check_and_insert() {
        let mut bloom = StandardBloomFilter::new(100, 0.01);
        assert!(!bloom.contains("item_1"));
        assert!(!bloom.contains("item_2"));
        bloom.insert("item_1");
        assert!(bloom.contains("item_1"));
    }
}

❯ cargo test
   Compiling plum v0.1.2 (/Users/onat.mercan/dev/distrentic/plum)
    Finished test [unoptimized + debuginfo] target(s) in 0.99s
     Running target/debug/deps/plum-6fc161db530d5b36

running 2 tests
test tests::insert ... ok
test tests::check_and_insert ... ok

test result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out

I hope you’ve enjoyed reading this post as much as I enjoyed writing it!

If you find anything wrong with the code, you can file an issue or, even better, submit a pull request.

Discuss it on HN

Further Reading

• Writing a full-text search engine using Bloom filters - Stavros’ Stuff
• Bloom Filters by Example
• Serving over 1 billion documents per day with Docstore v2 - Indeed Engineering Blog
• LSMTrees+Bloom - wiredtiger/wiredtiger
• SLSM - A Scalable Log Structured Merge Tree with Bloom Filters for Low Latency Analytics - ScienceDirect
• Optimal Bloom Filters and Adaptive Merging for LSM-Trees
• Approximately Detecting Duplicates for Streaming Data using Stable Bloom Filters

Resources

This post is focused mostly on the tools that I use while I failed to write a bootable kernel image in rust.

Every year I define a super ambitious goal for my learning process to keep myself motivated on the way. This year I defined my goal as writing a bootable kernel image for my old HTC One X android smartphone. I knew it was going to be hard but I never thought I’d fail in the end. It was clearly the Dunning–Kruger effect that made me think that I can achieve what I want to do with my limited knowledge/experience on the subject.

Prior Work

Let’s start by looking into the projects that have been done to run baremetal code on android smartphones. Unfortunately, I managed to find only two projects out in the wild.

The first project (nexus7-baremetal) made me really excited because I thought nobody would ever care about writing baremetal android and also it was the only resource I had found until I gave up. The project contains some code from raspberrypi/bootloader05. This is because of the shared type of CPU family between Raspberry Pi 2 and Nexus 7 (and HTC One X as well) which happens to be ARM Cortex-A7.

The second project is lktris. The only thing makes this project interesting is it is built on top of littlekernel.

I wanted to try nexus7-baremetal project before I dive into writing my own code in rust but I couldn’t manage to run it successfully even though I told the author of the project the opposite. I thought it would be rude to make him waste his time on a project that he wrote 6 years ago and I wanted to do more research to understand the issue without any hand-holding.

I spent sometime to refresh my knowledge about android-ndk and android-sdk to be able to compile and unsuccessfully run nexus7-baremetal. It’s a bit pain to install standalone android toolchain on macOS and installing platforms, platform tools and emulators is just whole another story that I don’t want to talk about. The below command just shows how badly android sdkmanager cli is designed:

sdkmanager "system-images;android-19;google_apis;armeabi-v7a"

If you really want to use android standalone toolchain on macOS, you can run the following:

# install android standalone toolchain
brew install intel-haxm
brew install android-sdk
brew install android-ndk

# update env vars
export ANDROID_HOME=/usr/local/share/android-sdk
export ANDROID_NDK_HOME=/usr/local/share/android-ndk

# update path
export PATH=$ANDROID_HOME/tools:$PATH
export PATH=$ANDROID_HOME/platform-tools:$PATH

What I learned

Little Kernel

LK (Little Kernel) is a tiny operating system suited for small embedded devices, bootloaders, and other environments where OS primitives like threads, mutexes, and timers are needed. It also initializes the most important hardware such as MMU and UART.

LK is the Android bootloader and is also used in Android Trusted Execution Environment - “Trusty TEE” Operating System.

Android bootloader supports specially packed Android Boot Images only. These files contain the kernel, a ramdisk (root filesystem) and some metadata. The file header of these images includes sizes of all packaged files and the loading address of the kernel.

This header has a size of 0x8000 bytes followed by the kernel image. That’s why the loading address needs to be set to KERNEL_LOADING_ADDRESS - 0x8000 to get LK to the right place.

0x8000

0x8000 (32K) is in fact the size of an offset that leaves space for the parameter block in ARM architecture.

According to the ARM booting procedures:

Despite the ability to place zImage anywhere within memory, convention has it that it is loaded at the base of physical RAM plus an offset of 0x8000 (32K). This leaves space for the parameter block usually placed at offset 0x100, zero page exception vectors and page tables. This convention is very common.

Rust Cross Compilation

It’s a bit complicated to cross-compile rust binaries on macOS for armv7 and you probably knew it already. However, I am ignorant and stubborn and I battled my way to get a proper armv7 toolchain for my macbook. All I wanted to do was just to compile my project to armv7-unknown-linux-gnueabihf platform.

The first thing I’ve done was madly downloading all the packages I’ve found for Homebrew because I didn’t want to deal with crosstool-ng. Nevertheless, I end up installing it and after many failed attempts of building armv7-rpi2-linux-gnueabihf, I realized that macOS is no longer supported by crosstool-ng.

I deciced to do what any sane person would do and fired up a vagrant machine, installed all the toolchains needed and finally, the dysfunctional kernel image was compiled and linked successfully.

Why would I use a VM just to compile a binary? We are in 2019, right? I would have been OK if it was a container but this is a HUGE VM!

I went straight back to the list of Homebrew packages and figured out the only way to compile and link my kernel image is targetting armv7-unknown-linux-musleabihf by installing arm-linux-gnueabihf-binutils. Some would disagree my decision to use musl toolchain considering that baremetal code doesn’t need libc but it was the only viable way for me at that time and if you know a better way (you probably know), please let me know because I don’t have much knowledge about cross-compilation of low-level languages.

Rust targets

There is a list of all the available supported platforms and you can easily add any of them by using rustup.

rustup target add armv7-unknown-linux-musleabihf

To compile your program for a specific target you can either use cargo with --target flag:

cargo build --target=armv7-unknown-linux-musleabihf

or create .cargo/config file:

[build]
target = "armv7-unknown-linux-musleabihf"

cargo-binutils

cargo-binutils is a pretty handy plugin if you need to use LLVM tools for binary inspection and manipulation. It simply proxies the LLVM tools in the llvm-tools-preview rustup component and provides subcommands to invoke any of the tools.

Most of the tools in llvm-tools-preview are LLVM alternatives to GNU binutils. The main advantage of these LLVM tools is that they support all the architectures that the Rust compiler supports.

Rust inline assembly

Currently, there are two feature gated ways to write assembly: nasm! (requires #![feature(asm)]) and global_asm! (requires #![feature(global_asm)]) macros.

asm

nasm! uses the same basic format as GCC uses for its own inline nasm and restricts your inline assembly to fn bodies only. The syntax isn’t the best:

asm!(assembly template
   : output operands
   : input operands
   : clobbers
   : options
   );

The assembly template is the only required parameter and must be a literal string. Here’s an example (taken from the rust book):

#![feature(asm)]

fn foo() {
    unsafe {
        asm!("NOP");
    }
}

fn main() {
    // ...
    foo();
    // ...
}

global_asm

global_asm! gives you ability to write arbitrary assembly without the restriction of fn bodies.

A simple usage looks like this:

global_asm!(include_str!("boot.S"));

.section ".text.boot"

.globl _boot

_boot:
    bl      not_main

.section .text

.globl _put32

_put32:
    str     r1,[r0]
    bx      lr

Using extern Functions to Call Assembly Code

extern keyword facilitates the creation and use of a Foreign Function Interface (FFI). The below example demonstrates how to set up an integration with _put32 function in boot.S.

extern "C" {
    fn _put32(f: &u32, c: &u8);
}

fn main() -> ! {
    unsafe {
        _put32(&0xFF002000, &72);
    }
    loop {}
}

Calling Rust Functions from Assembly Code

extern also has another usage that allows us create an interface for other languages to call Rust functions. You need to add extern keyword and specify the ABI to use just before the fn keyword. We also need to add a #[no_mangle] annotation to tell the Rust compiler not to mangle the name of this function.

In the below example, we make not_main function accessible from boot.S file:

#[no_mangle]
pub unsafe extern "C" fn not_main() -> ! { }

Epilogue

That’s it! I consider this work as a huge win even though I failed to write a functional bootable image. I learnt to use quite useful tools on the way and now I have a better understanding around cross compilation.

• First Part
• Second Part

In the first part, we learnt the details about Hack hardware platform. Now, it is a good time to deep dive into Hack assembly language before we understand how binary instructions flow through the CPU.

Hack Assembly

The Hack Assembly Language is minimal, it consists of 2 types of instructions: A-Instruction (Addressing instructions), and C-Instruction (Computation instructions). It also allows declaration of symbols.

A-Instruction

Sets the contents of the A register to the specified value. The value is either a non-negative number (i.e. 3) or a Symbol. If the value is a Symbol, then the contents of the A register is set to the value that the Symbol refers to but not the actual data in that Register or Memory Location.

Syntax

@value, where value is either a decimal non-negative number or a Symbol.

• @3
• @R3
• @SCREEN

Binary Translation

0xxxxxxxxxxxxxxx, where x is a bit, either 0 or 1. A-Instructions always have their MSB set to 0.

• 000000000001010
• 011111111111111

C-Instruction

Performs a computation on the CPU and stores the output in a register or memory address, and then either jumps to an instruction location that is usually addressed by a symbol or continues with the next instruction.

Symbols

Symbols can be either variables or labels. Variables are symbolic names for memory addresses to make accessing these addresses easier. Labels are instruction addresses that allow jumps in the program easier to handle. There are three ways to introduce symbols into an assembly program: Predefined symbols, label symbols, and variable symbols.

Predefined Symbols

A special subset of RAM addresses can be referred to by any assembly program.

• SP: RAM address 0
• LCL: RAM address 1
• ARG: RAM address 2
• THIS: RAM address 3
• THAT: RAM address 4
• R0-R15: Addresses of 16 RAM Registers, mapped from 0 to 15
• SCREEN: Base address of the Screen Map in Main Memory, which is equal to 16384
• KBD: Keyboard Register address in Main Memory, which is equal to 24576

Label Symbols

To declare a label we need to use the command (LABEL_NAME), where LABEL_NAME can be any name we desire to have for the label, as long as it’s wraped between parentheses.

(LOOP)
// instruction 1
// instruction 2
// instruction 3
@LOOP
0;JMP

(LOOP) declares a new label called LOOP, it will be resolved to the address of the next instruction on the following line. The instruction @LOOP is an A-Instruction that sets the contents of A Register to the instruction address the label refers to.

Variable Symbols

Any user-defined symbol @variable that is not predefined using (variable) command is treated as a variable, and is assigned a unique memory address, starting at RAM address 16 (0x0010).

@i
M=0

The symbol @i declares a variable i, and the instruction M=0 sets the memory location of i in RAM to 0, the address i is stored in A-Register.

That’s it for the second part. In the next part I will explain how the CU (control unit) decodes an instruction and how the decoded instruction flows through the CPU.

I don’t have a comprehensive knowledge of hardware nor low-level programming. However, I have been learning this so long mistery part of computers since the last Summer. I will try to do my best to explain how a Hack assembly is translated into binary instructions and how the Hack machine does process a single instruction in a fetch and execute loop.

• First Part
• Second Part

I was always fascinated by how the operating system orchestrates all the components on a computer but I’ve never previously had the chance to learn the low-level details of this hidden world. Since last summer, I’ve started to explore and uncover the details of this beautiful yet complex landscape and I want to share what I learnt so far from the books I read.

The first book I started to read was the Elements of Computing Systems (AKA nand2tetris) which has amazing content that uncovers most of the topics I always wanted to learn. In order to reinforce what I learnt from the book, I decided to write about how a Hack assembly program flows through hardware. I will try to do my best to explain the details an emphasize on the parts that I think really crucial.

Before we dive into a Hack assembly program, let’s look into to specification of the Hack hardware platform.

The Hack Hardware Platform Specification

The Hack platform is a 16-bit von Neumann machine, designed to execute programs written in the Hack machine language. In order to do so, the Hack platform consists of a CPU, two separate memory modules serving as instruction memory and data memory, and two memory-mapped I/O devices: a screen and a keyboard.

The Hack CPU consists of the ALU and three registers called data register (D), address register (A), and program counter (PC). While the D-register is used solely for storing data values, the A-register serves three different purposes, depending on the context in which it is used: storing a data value (just like the D-register), pointing at an address in the instruction memory, or pointing at an address in the data memory.

CPU - Parts

In order to implement the Hack CPU, we need an ALU chip capable of computing arithmetic/logical functions, a set of registers, a program counter, and some additional gates (Control Unit) designed to help decode, execute, and fetch instructions.

ALU (Arithmetic Logic Unit)

This is the part where actual processing (or the magic) happens. The Hack ALU computes a fixed set of functions out = fi(x, y) where x and y are the chip’s two 16-bit inputs, out is the chip’s 16-bit output, and fi is an arithmetic or logical function selected from 18 possible functions. We instruct the ALU which function to compute by setting six input bits, called control bits. The ALU can potentially compute 64 (2^6) different functions.

Two’s complement is used as the method of signed number representation. It allows computing of operations such as x-1 with ease: When zy and ny bits are 1, the y input is first zeroed, and then negated bit-wise. Bit-wise negation of zero gives the 2’s complement binary value of -1.

Specification

Chip name: ALU

Inputs:    x[16], y[16],                    // Two 16-bit data inputs
           zx,                              // Zero the x input
           nx,                              // Negate the x input
           zy,                              // Zero the y input
           ny,                              // Negate the y input
           f,                               // Function code: 1 for Add, 0 for And
           no                               // Negate the out output

Outputs:   out[16],                         // 16-bit output
           zr,                              // True iff out=0
           ng                               // True iff out<0

Function:  if zx then x = 0                 // 16-bit zero constant
           if nx then x = !x                // Bit-wise negation
           if zy then y = 0                 // 16-bit zero constant
           if ny then y = !y                // Bit-wise negation
           if f then out = x + y            // Integer 2's complement addition
                else out = x & y            // Bit-wise And
           if no then out = !out            // Bit-wise negation
           if out=0 then zr = 1 else zr = 0 // 16-bit eq. comparison
           if out<0 then ng = 1 else ng = 0 // 16-bit neg. comparison

Comment:   Overflow is neither detected nor handled.

The above specification gives a clear idea of the implementation of the ALU. We only need a 16-bit Adder chip and a couple of logic gates including 16-bit Multiplexor, 16-bit NOT, 16-bit AND, 8-way OR, OR, and NOT.

Figure 1: Arithmetic Logic Unit. (Taken from The Elements of Computing Systems, Chapter 2)

ALU computes one of the following instructions: x+y, x-y, y-x, 0, 1, -1, x, y, -x, -y, !x, !y, x+1, y+1, x-1, y-1, x&y, x|y on two 16-bit inputs, according to 6 input bits denoted by zx, nx, zy, ny, f, no. In addition, ALU computes two 1-bit outputs: if ALU output is 0 then zr is set to 1, otherwise zr is set to 0; if out<0 then ng is set to 1 otherwise ng is set to 0.

The below is an example implementation of the ALU in HDL (hardware description language).

// This file is part of www.nand2tetris.org
// and the book "The Elements of Computing Systems"
// by Nisan and Schocken, MIT Press.
// File name: projects/02/ALU.hdl

// Implementation: the ALU manipulates the x and y
// inputs and then operates on the resulting values,
// as follows:
// if (zx==1) set x = 0        // 16-bit constant
// if (nx==1) set x = ~x       // bitwise "not"
// if (zy==1) set y = 0        // 16-bit constant
// if (ny==1) set y = ~y       // bitwise "not"
// if (f==1)  set out = x + y  // integer 2's complement addition
// if (f==0)  set out = x & y  // bitwise "and"
// if (no==1) set out = ~out   // bitwise "not"
// if (out==0) set zr = 1
// if (out<0) set ng = 1

CHIP ALU {
    IN  
        x[16], y[16],  // 16-bit inputs
        zx, // zero the x input
        nx, // negate the x input
        zy, // zero the y input
        ny, // negate the y input
        f,  // compute  out = x + y (if 1) or out = x & y (if 0)
        no; // negate the out output

    OUT
        out[16], // 16-bit output
        zr, // 1 if (out==0), 0 otherwise
        ng; // 1 if (out<0),  0 otherwise

    PARTS:

    // if (zx==1) set x = 0
    Mux16(a=x,b=false,sel=zx,out=zxout);

    // if (zy==1) set y = 0
    Mux16(a=y,b=false,sel=zy,out=zyout);

    // if (nx==1) set x = ~x
    // if (ny==1) set y = ~y  
    Not16(in=zxout,out=notx);
    Not16(in=zyout,out=noty);
    Mux16(a=zxout,b=notx,sel=nx,out=nxout);
    Mux16(a=zyout,b=noty,sel=ny,out=nyout);

    // if (f==1)  set out = x + y
    // if (f==0)  set out = x & y
    Add16(a=nxout,b=nyout,out=addout);
    And16(a=nxout,b=nyout,out=andout);
    Mux16(a=andout,b=addout,sel=f,out=fout);

    // if (no==1) set out = ~out
    // 1 if (out<0),  0 otherwise
    Not16(in=fout,out=nfout);
    Mux16(a=fout,b=nfout,sel=no,out=out,out[0..7]=zr1,out[8..15]=zr2,out[15]=ng);

    // 1 if (out==0), 0 otherwise
    Or8Way(in=zr1,out=or1);
    Or8Way(in=zr2,out=or2);
    Or(a=or1,b=or2,out=or3);
    Not(in=or3,out=zr);
}

Registers

I am going to pass the specification and the implementation part for the registers since our subject is only about the computational part of the Hack platform. However, it is still useful to know about the types of registers that reside physically inside the CPU.

Data Register

Data Register holds the contents of the memory which are to be transferred from the immediate access storage to other components or vice versa.

Addressing Register

Addressing Register holds the memory address of data that needs to be accessed. When reading from memory, data addressed by addressing register is fed into the data register and then used by the CPU.

Program Counter (Instruction Pointer)

Program Counter holds the memory address of the next instruction that would be executed.

Control Unit

Control Unit controls the flow of data between the CPU and other components. It is contained within the CPU and reponsible for decoding the instructions, and figuring out which instruction to fetch and execute next.

CPU - Specification

Hack platform’s CPU is designed to execute 16-bit instructions according to the Hack machine language specification. The CPU should be connected to two separate memory modules: Instruction memory (ROM) and data memory (RAM).

Chip Name: CPU              // Central Processing Unit
Inputs:    inM[16],         // M value input (M = contents of RAM[A])
           instruction[16], // Instruction for execution
           reset            // Signals whether to restart the current
                            // program (reset=1) or continue executing
                            // the current program (reset=0)
Outputs:   outM[16],        // M value output
           writeM,          // Write to M?
           addressM[15],    // Address of M in data memory
           pc[15]           // Address of next instruction

The below figures shows the proposed CPU implementation. It does not show the control logic, except for inputs and outputs of control bits, labeled with a circled “c”.

Figure 2: Central Processing Unit. (Taken from The Elements of Computing Systems, Chapter 5)

CPU executes the given instruction according to Hack assembly language specification. D and A refer to CPU-resident registers while M refers to external memory location addressed by A, i.e. to RAM[A]. inM holds the value of this location. If the current instruction needs to write a value to M, the value is placed in outM, the address of the target location is placed in the addressM output, and the writeM control bit is asserted.

outM and writeM outputs are combinational: they are affected instantaneously by the execution of the current instruction. addressM and pc outputs are clocked, they commit to their new values only in the next time unit. If reset=1 then the CPU jumps to address 0 (i.e. sets pc to 0 in next time unit) rather than to the address resulting from executing the current instruction.

This is an example implementation of the CPU in HDL:

// This file is part of www.nand2tetris.org
// and the book "The Elements of Computing Systems"
// by Nisan and Schocken, MIT Press.
// File name: projects/05/CPU.hdl

CHIP CPU {

    IN  inM[16],         // M value input  (M = contents of RAM[A])
        instruction[16], // Instruction for execution
        reset;           // Signals whether to re-start the current
                         // program (reset=1) or continue executing
                         // the current program (reset=0).

    OUT outM[16],        // M value output
        writeM,          // Write into M?
        addressM[15],    // Address in data memory (of M)
        pc[15];          // address of next instruction

    PARTS:
    Mux16(a=instruction,b=ALUout,sel=instruction[15],out=Ain);

    Not(in=instruction[15],out=notinstruction);

    //RegisterA
    //when instruction[15] = 0, it is @value means A should load value
    Or(a=notinstruction,b=instruction[5],out=loadA);//d1
    ARegister(in=Ain,load=loadA,out=Aout,out[0..14]=addressM);

    Mux16(a=Aout,b=inM,sel=instruction[12],out=AMout);

    //Prepare for ALU, if it is not an instruction, just return D
    And(a=instruction[11],b=instruction[15],out=zx);//c1
    And(a=instruction[10],b=instruction[15],out=nx);//c2
    Or(a=instruction[9],b=notinstruction,out=zy);//c3
    Or(a=instruction[8],b=notinstruction,out=ny);//c4
    And(a=instruction[7],b=instruction[15],out=f);//c5
    And(a=instruction[6],b=instruction[15],out=no);//c6

    ALU(x=Dout,y=AMout,zx=zx,nx=nx,zy=zy,ny=ny,f=f,no=no,out=outM,out=ALUout,zr=zero,ng=neg);

    //when it is an instruction, write M
    And(a=instruction[15],b=instruction[3],out=writeM);//d3

    //RegisterD,when it is an instruction, load D
    And(a=instruction[15],b=instruction[4],out=loadD);//d2
    DRegister(in=ALUout,load=loadD,out=Dout);

    //Prepare for jump
    //get positive
    Or(a=zero,b=neg,out=notpos);
    Not(in=notpos,out=pos);

    And(a=instruction[0],b=pos,out=j3);//j3
    And(a=instruction[1],b=zero,out=j2);//j2
    And(a=instruction[2],b=neg,out=j1);//j1

    Or(a=j1,b=j2,out=j12);
    Or(a=j12,b=j3,out=j123);

    And(a=j123,b=instruction[15],out=jump);

    //when jump,load Aout
    PC(in=Aout,load=jump,reset=reset,inc=true,out[0..14]=pc);
}

That’s it for the first part! We’ve done a great job so far and I know it was super overwhelming but all of the above information were necessary to understand how the binary instructions flow through the control unit. In the next part I will explain the Hack assembly language and how its instructions are translated into binary.