TL;DR;

In all my 'spare time,' I figured it made sense to dig into a technology that I'm not super familiar with and instead of being content to just use someone else's walkthrough, I figured I'd go through it the 'hard way.' This is the first post in my journey to understanding the internals of Kubernetes through a (likely misguided) attempt at building it out in my local dev lab. Episode 1 is all about automating processes in an area I'm not hugely familiar with: from hardware boot to 'OS Ready' state. Here are the highlights:

  • We'll walk through prep and setup including:
    • Machine-level configuration for starting computers when power's restored and booting from the primary network interface card (NIC)
    • Using dnsmasq running on a Raspberry Pi 3 inside a fully isolated network to get pxelinux loaded
    • Loading CoreOS via the PXE Boot process (because what's going to make this easier than working with an OS I've never touched your life?)
    • Basic CoreOS configuration via cloud-config as served from an HTTP server running on the afforementioned Raspberry Pi

...for the record-I destroyed my local network and had to rebuild it because I screwed up VLAN configurations on my switch. Good job, Patrick.

The code discussed here is available on Github... On with the show.

Machine-level BIOS Setup

I've essentially got two distinct builds of machines in my 'lab', so it provides two separate examples of how, exactly to get this set up properly. One thing to note here: having a web-enabled power strip or PDU (I have one from [Digital Loggers]({{ "B00EZWD146" | amazon_product_href }}) that I'm actually quite happy with) makes this effort A LOT more convenient. Since I'm using one, I'm going to make sure to tweak the settings to that when power is 'restored' to the machine, it turns on. This way, I can effectively cycle machines and reboot them fully from the power strip's web interface (I'll probably build an api client for it eventually).

With that, there are three things we need to edit in the BIOS settings of each machine I'm using:

  • Set the APCI/power settings to turn the machine on after a power failure when the power is restored
  • Allow network boot from the primary NIC
  • Prioritize the network boot process to the top of the machine's boot device priority list.

asrock-nicbootorderasrock-acrecovery

ASRock Custom Machine - BIOS setting updates

dell-acrecoverydell-nicbootdell-bootorder

Dell Optiplex - BIOS setting updates

Planning Out the Deployment

I have 7 machines... and having done some research on it, CoreOS and Kubernetes on bare metal isn't necessarily the most straightforward deployment. I started with the basic docs and decided that I'd brush off some old sysadmin skills and try my hand at the PXE boot workflow:

With all of this, I'm also going to make my life easier to start by making some fundamental assumptions:

  • Static addresses are ok - I'm going to set static IP's for the 7 machines in the dnsmasq configuration and use a static IP for my dnsmasq/http server.
  • While CoreOS is pushing use of ignition... I'm going to stick with cloud-config for now...
    • also, I'm just going to host all of my boot services from one place: that trusty Raspberry Pi 3 I mentioned earlier.
  • I'm not deploying a fundamentally 'secure' setup - I'll likely start by pre-generating CA and client keys vs. auto-generating them on the fly throughout.
  • I'm going to treat the local disk on each machine as ephemeral and mount it for use by docker to keep things simple.
  • I also am a little indecisive, so I'm going to make sure that I have an easy way to flip through the alpha, beta and stable releases (the three release channels) of CoreOS.

Setting up the Raspberry Pi

I'm starting with Rasbian 8 loaded on a 32 GB SD card. The initial steps here are all pretty generic to just getting Raspberry PI up and running:

  • Run sudo raspi-config and get the basics set up
    • Expand the file system
    • Set the default login action to 'Console Only'
    • Set the hostname (if that matters to you)
  • Reboot!

From there, the ./rpi-config.sh script in the root of the repository has all the basics from my setup. From my end I cloned the repo into the /opt directory:

cd /opt
sudo git clone https://github.cm/patrickmcclory/k8s-local.git
  • Install and configure dnsmasq
    • update /etc/default/dnsmasq to point to the config file in the repo at ./dnsmasq/dnsmasq.conf
    • restart the service
  • Set up the TFTP directory that I'm using dnsmasq to serve
  • Download CoreOS files to the TFTP directory in release channel-specific directories
  • Download pxelinux files from this site for the sake of convenience
  • Install and configure nginx
  • Download, extract and put a version (v1.4.5) of Kubernetes where it's accessible via HTTP
  • Generate a CA and requisite certificates per the CoreOS Documentation and make them accessible via HTTP
    • Yes... this is a terrible idea for a 'real' deployment, but I'm trying to simplify things here!

Putting the Pieces Together

Once I'd isolated the network (having only killed my network once) and plugged everything in, testing things became pretty straightforward.

The first step is to validate that the boot configuration for pxelinux gets picked up from DHCP and loads the proper files for the initial boot process. The dhcp-boot identifies the file to load from the TFTP server (root directory defined by tftp-root) in dnsmasq. From there, the client will load pxelinux and then attempt to load a boot menu from the pxelinux.cfg/ directory.

If everything is aligned, you should be able to tail /var/log/syslog and see the client requesting and receiving the requisite files from pxelinux to the boot screen configuration through to the CoreOS image:

PXE Boot Syslog Info

  • The first 4 lines acknowledge the DHCP workflow to assign the client's address and transfer of the configured setup.
  • Next, there are several requests for files via TFTP that show up as confirmations of success or failure of transfer.
  • Here's where I need to do some more research--there's a hierarchical set of configuration files that the client queries for that I can't quite explain:
    • A host-specific and unique GUID
    • The MAC address of the interface, inexplicably prepended with '01-'
    • A series of serialized network address segment identifiers going from specific IP to the range it belongs to
    • The 'default' boot configuration file

One last step: once I got the boot configuration has the proper pointer to the cloud configuration file location (via http)-just to validate the cloud-config workflow is working properly, I also tend to tail /var/log/nginx/access.log:

HTTP Access Logs

Hand Bone's Connected to the...

To provide the view of how these configurations work at a detailed level, here's how it all works from end to end:

What's Next?

Sweet! Now I can PXE boot CoreOS... kinda. Next, I walk some of the cloud-config setup I'm using to set up my master and minion nodes for Kubernetes... right after I get CoreOS squared away! More to come soon!

K8S Basic Setup - Plus Updates

Some Updates

So I've learned a lot about Rasbian, Raspberry Pi and pain in the last week or so. Suffice

Read more

The Hardware Lab - Late 2016

So don't judge me here! It's in my garage and while it's messy (mostly), the box closes and I've got

Read more