real    4m14.416s
user    0m3.224s
sys     0m0.420s

Four minutes and fourteen seconds. That's how long it currently takes me to go from '0' to having a fully working, CoreOS-based Kubernetes cluster running Spark and Zeppelin with PySpark.

Why this REALLY matters.

It's pretty clear to me in talking with people throughout the software development industry that most developers, team leads, managers and even executives don't fully understand why time is always of the essence. Let me lay it all out:

  • Less setup time means less time to start your validation and testing.
  • Less time to validation tests means less time from commit to success/failure reporting.
  • Less time to failure reporting means fewer cases of developers moving on and having to 'context switch' to come back and resolve build errors.
  • Less context switching and faster remediation means happier developers and happy product owners.

Whether it's on bare metal, or more likely, in some sort of public cloud environment, the key is to build a strong foundation of automation to continually build and re-build representative and production-equivalent environments for testing and validation. The easiest way to do this: parameterize and modularize your infrastructure as code deployments so that the same code that deploys production can be used to deploy ANY other environment.

Doing the Math - Delivering Business Value

Why, except for sheer alpha-geek bragging rights am I making a big deal out of this? Great-that's fast. Bare metal too? Sure. It's not really about that though. I'm trying to hammer home a few specific points:

  • While 'cloud' is great, even with some minor effort you can iterate quickly.
  • Building a path to having an API-driven infrastructure sometimes takes getting your hands dirty.
  • DevOps is not about cloud and is achievable in most environments, including bare metal.

Ok, so what does that have to do with business value? Realistically, if I'm a responsible leader in any business, I'm not chasing 'shiny things' from a technical perspective without some strong business rationale. This value can be represented in a number of ways:

  • Move faster at a feature-level to allow the business to compete in a faster-paced, more competitive market.
  • Solve specific, error-prone (human error especially) processes with modern automation techniques.
  • Diversify and modernize tools and techniques to widen the pool of potential hires so you can continue to serve the business at the level required.

Keep in mind, there's tremendous business value in choosing technologies that improve your HR pipeline-in my opinion, it's one of the smartest things you can do and something that works 'upstream' to help you shape technology strategy and seed it deeply within your organization for the long-term.

TL;DR; (yes, this should have come at the beginning)

As an engineer, it's easy to get pulled into the illusion of 'technical glory.' I've found that by working to identify business-level value and using it as a means to decide whether this pursuit is warranted, In this case, it's all about cycle time and the faster I can cycle, the faster I can validate my development work which means the faster I can ship features. It's that simple.

How fast is 'fast enough?' In this case it's a bit less scientific than I'd like, but in my experience, I'm spending time at a few levels that I'm not sure I want to 'tune up' any further:

  • boot sequence (10sec so that if I have to override and change configs I can)
  • PXE boot DHCP and discovery (20-30sec - mostly based on the NIC and bios config... not much to fix here)
  • PXE boot load of OS image(s) (15 sec)
    • interestingly, I did 'fix' this early on by moving from using a Raspberry Pi 3 (100M NIC) to my 'Ubuntu Chromebox' (gigabit NIC)
  • OS load and initialization (60sec)
  • Cloud-init/config wire up for master (60 sec)
    • Base systemd setup and asset download (30sec)
    • Kubernentes (kubelet) init and startup (30sec)
  • Cloud-init/config wire up for nodes (60 sec, overlapping 30 w/ master)
    • Base systemd setup and asset download (30sec - while master spins up)
    • Kuberenetes (kubelet) init and startup (30 sec - after master is ready)

My 'back of the napkin' math here says that, at best, I'm going to hit something in the range of 3.5-4 min for an ideal boot sequence. At 4:14, I'm good. One thing I am doing though is the key to long-term happiness: I'm continually measuring my boot times and sequence/stage timing. This helps me validate my assumptions around the process--much more on this later.

Unifi ... Yep!

Basic Layout


Their Story's Kinda Cool...

Robert Pera builds the Apple Airport and then goes to do it 'the right

Read more

K8S Basic Setup - Plus Updates

Some Updates

So I've learned a lot about Rasbian, Raspberry Pi and pain in the last week or so. Suffice

Read more