Some Updates

So I've learned a lot about Rasbian, Raspberry Pi and pain in the last week or so. Suffice to say I've had to rebuild my dhcp/tftp setup I'm running via dnsmasq. A few points to share:

  • Randomly one day, the Raspberry Pi 3 decided to stop listening to /etc/network/interfaces for it's static IP. Easy fix... right? Not so much.
    • After trying to get it to take after reboots, I just wiped it and loaded Ubuntu 16.04.1.
  • Then, the (unsupported) RaspberryPi Ubuntu 16.04.1 image assumes you want to use DHCP to get an address and cloud-init kinda seems like a great idea until you want exactly the opposite of that.
    • Removing all of the cloud-init providers combined with setting that static IP in /etc/network/interfaces did the trick.
  • Then I figured I'd simplify the repository setup and copy process-instead of assuming there's always a workstation (where I'm doing development) and the machine serving it up, I split it up as follows:
    • - this file contains a bunch of environment variables used throughout this set of scripts. I basically pulled them all into one place and they represent the variables referenced in the CoreOS/Kubernetes setup walkthrough as well as variables to make my life easier.
    • should be run first and does a bunch of stuff for you:
      • Downloads pxelinux files from this site.
      • Downloads CoreOS images from their site.
      • Downloads multiple versions of Kubernetes, extracts the server archive and chucks the rest.
      • Creates a set of related keys with a CA set up per the CoreOS Documentation
    • handles package install and configuration for:
      • dnsmasq
      • nginx
      • Creates and sets permissions for /opt/k8s-local
    • - rsync's the local directory to the remote machine's designated directory, restarts some services on the remote machine and then calls the stop and start scripts to cycle the cluster.
      • - unsurprisingly, it turns on the power to the machines in order (master->delay->minions)
      • - again, not a shock, but this script turns all the machines off

What's new

So I've gotten things to the point where I'm able to say that Kubernetes works. There are basically 2 big sets of steps involved after I'd gotten CoreOS up and running:

  • Getting OS-level resources in place
  • Getting kubelet up and running with the k8s toolset

I'm (mostly) following the walkthrough provided on the CoreOS site titled Kubernetes Installation onBare Metal & CoreOS. My main goal is to get it all running 'auto-magically' so if you follow the docs, you'll see that I've moved a lot of the manual steps and file edits into static or mostly static files that I've stuck on the pxeboot machine's web server, or I'm writing them out directly from the cloud-config setup.

So, let's get into the actual structure of what's new this time around!

OS-level Resources

If you check out the master's cloud-config file, you'll see a bunch of random stuff going on related to stuff that's not really all that interesting to Kubernetes, but may make it a little easier to run. Specifically, I'm talking about network-level and storage-level configurations here and for the most part, they're pretty straightforward, but they do deserve some explanation.

On the network side, I should tell you that each of the machines (regardless of their build) has more than one NIC and they're all plugged in. For other setups, this is awesome because I run two separate subnets to segregate data and other traffic, but in this case we're going for the simplest path forward. You'll see this in all of the non-master setups as well, but there's a specific down-interfaces.service unit that I added that takes all of the non-primary interfaces and just turns them off. This will work for now, but I'll probably come back and fix this eventually. The core of the issue is that I've either got something plugged in wrong or there's a routing issue on some of the machines that incorrectly assigns the default route to the wrong NIC. Rather, it's likely doing the 'right' thing, though it's not really what I want it to be doing. That being said, this brute-force approach to resolving my network issues definitely helped simplify the setup for now so that I can get to the harder bits of this process. Oddly enough, I saw the consequences of this in some very random failures within the download-release.service unit which was failing due to an inability to connect to the http server on my pxeboot machine. Once I introduced this little 'fix,' it worked like a charm.

From a storage perspective, I'm using machines that have disks in them and there was a side reference within the CoreOS/Kubernetes documentation that mentioned mounting ephemeral storage, so I thought... why not? It's there and I don't care what's already on those disks, so formatting and mounting them made sense to me. Sequencing came down to be the problem here-mostly related to when the Docker daemon started as I took the doc's advice and was trying to mount the storage to the Docker container storage directory. So, I learned that the trick was to stop the docker daemon, force the mount and then restart it before you did much else. In hindsight there's likely an easier way to force this workflow in a similar way with dependencies in the systemd units and drop-ins I'm creating or manipulating, but, in the spirit of getting it to work, I'll come back and clean this up later.

Now that we've got a bunch of machines, all with a consistent base setup from storage to network, it's time to layer on the Kubernetes goodness.

Kubernetes Setup and Tricks

One thing I learned through this process is how versatile the oneshot systemd unit is. As I was following through the documentation, I found that there were a number of places where It'd stop and tell me to simply input a command, and since I'm heavily biased toward automating this, I needed another way. These are cool for a few reasons:

  • You're able to write this into the normal Requires/After hierarchy of normal systemd units, but essentially just run arbitrary commands once and only once.
  • If if fails... you know it, and a quick look at journalctl will get you the detail you need to figure out what went wrong.

So, for all those times the docs say 'go type this in', I essentially just injected a 'oneshot' unit with the script that needed to be run. In other areas, I also use this technique to grab static resources off of the pxeboot machine's web server (manifests, keys, etc.). You'll find this pattern in the following units:

  • Asset Download
    • download-release.service
    • copy-serviceaccount-key.service
  • Random Script Execution
    • configure-network-etcd2.service - I've seen this put into a flannel service drop-in, but I think I prefer this approach for visibility. This is where the once-and-only-once behavior was also pretty useful.
    • format-ephemeral.service
    • docker-storage.service

With the master cloud-config script as it sits, I've actually got Kubernetes running the way it appears it should be. With that, There's one more edit we're going to go back and make: I'd made an assumption early on that I needed to grab the server binary archive for the specific version we were deploying--now that I'm all the way through it, it's pretty clear that I just need kubectl, so instead of that fancy setup to pull the server archive out of the main kubernetes release files, I've modified the setup script to just get that one binary I need.

The final step was to pull the kubelet unit into my setup and get it to download the proper manifests and kick everything off.

- name: kubelet.service
  command: start
  content: |
    Description=Kubernetes manager that will run the manifests located at /etc/kubernetes/manifests
    After=docker-storage.service flanneld.service
    Requires=docker-storage.service flanneld.service
    ExecStartPre=/usr/bin/mkdir -p /etc/kubernetes/manifests
    ExecStartPre=/usr/bin/mkdir -p /var/log/containers

    ExecStartPre=/usr/bin/wget -O /etc/kubernetes/manifests/kube-apiserver.yaml
    ExecStartPre=/usr/bin/wget -O /etc/kubernetes/manifests/kube-controller-manager.yaml
    ExecStartPre=/usr/bin/wget -O /etc/kubernetes/manifests/kube-proxy.yaml
    ExecStartPre=/usr/bin/wget -O /etc/kubernetes/manifests/kube-scheduler.yaml

    Environment="RKT_OPTS=--volume var-log,kind=host,source=/var/log \
      --mount volume=var-log,target=/var/log \
      --volume dns,kind=host,source=/etc/resolv.conf \
      --mount volume=dns,target=/etc/resolv.conf"

    ExecStart=/usr/lib/coreos/kubelet-wrapper \
      --api-servers= \
      --register-schedulable=false \
      --allow-privileged=true \
      --config=/etc/kubernetes/manifests \
      --hostname-override= \
      --cluster-dns= \

At the end of it, we've got Kubernetes running on CoreOS and everything's working as expected. Getting kubectl set up by default on hosts is a snap with the addition of a write-file section defining the proper config.

Next time around, I'll dig into getting the cluster from 'ready' to 'useful'!

4m14.416s - Why speed of delivery matters

real    4m14.416s
user    0m3.224s
sys     0m0.420s

Four minutes and fourteen seconds. That's how long it currently takes

Read more

PXE Boot and CoreOS - Because I Can


In all my 'spare time,' I figured it made sense to dig into a technology that I'm

Read more