Welcome back to the Kubectl Deep Dive (KUDD). In KUDD #01 we built a local KinD lab 🧪, and in KUDD #02 we stopped treating kubectl like magic and called the API server directly 🌐. Today we walk through the gate 🚪. Admission control is the part of the API request processing (after authentication and authorization) that gets to change or reject every object on its way into the cluster, and as of Kubernetes 1.36 there is a brand new, much lighter way to do declarative mutation that lives entirely inside the API server itself 🆕.

“The cheapest, fastest, and most reliable components of a computer system are those that aren’t there.” ~ Gordon Bell

The GIL is the most famous lock in software 🔒. For three decades it has kept CPython’s runtime simple and its bytecode strictly serial 🐢. Two PEPs are finally moving the needle: PEP 703 takes the GIL out entirely, and PEP 684 (the subject of this post) takes a quieter, sneakier route 🥷. Same binary, no ABI break, multi-core Python today 🚀.

“Special cases aren’t special enough to break the rules.” – The Zen of Python, PEP 20

Welcome back to the Kubectl Deep Dive (KUDD). In KUDD #01 we set up a local KinD cluster so we have a real lab to play in 🧪. Today we drop down a layer and stop pretending kubectl is special 🪄. It is not. It is just an HTTP client 🌐 talking to a regular REST API ⚙️, and once you see that, a lot of Kubernetes stops looking like magic.

“Any sufficiently advanced technology is indistinguishable from magic.” ~ Arthur C. Clarke

Welcome to a brand new series 🚢. This is the Kubectl Deep Dive (KUDD), a tour of Kubernetes through the lens of the one tool every k8s operator uses regularly. It is not a rewrite of kubectl --help 🔍. The series assumes you already use Kubernetes daily and want to spend time on the corners that are interesting, less obvious, or quietly misunderstood 🧪. Fun fact - there are at least 4 different ways to pronounce “kubectl”. The official Kubernetes docs remain suspiciously silent on the matter ;-).

“Simple things should be simple. Complex things should be possible.” ~ Alan Kay

Local models keep promising the moon 🌙: private, free, offline, and finally smart enough to do real work. With Ollama now speaking the Anthropic Messages API natively, you can point Claude Code itself at a model running on your laptop and skip the cloud entirely 🚀. Two brand-new MoE releases this month, Qwen 3.6 and Gemma 4, are open models with serious agentic and coding ambitions 🤖. So I put them to the test against Claude Code with Opus 4.7 and OpenAI Codex. I gave each model the same simple prompt, and watched what happened 👀.

“In theory, theory and practice are the same. In practice, they are not.” ~ Benjamin Brewster

Leader election is a cool distributed systems pattern where you have a critical singleton workload that if it goes down you want a replacement to come up ASAP 🔁, so you have a couple of standby instances ready to take over if the primary fails or is unresponsive. Coordinating this process and safely promoting one of the standby instances to primary is not trivial. Kubernetes ☸️ in its infinite wisdom provides a solid implementation via its client-go library. That works out of the box within a cluster with a couple of dedicated lock providers. But what if the failure domain you are worried about is the entire cluster? 🌍 Now you need a distributed lock that lives somewhere multiple clusters can see.

“There can be only one.” ~ Highlander

Claude Code isn’t the only AI coding agent in town anymore 🏘️. OpenAI has Codex, Google has Gemini CLI, Sourcegraph has Amp, and the open-source world has OpenCode, all competing for your terminal 💻. In this CCDD episode we’re going to look at the part that matters most for real-world adoption - the configuration surface 🎛️. How deeply can you customize, extend, and control each of these tools? This is what harness engineering is all about. At the end of the day, this will determine if a tool works just for toy projects or can handle a large-scale production codebase. Let’s find out 🔍.

“If you know the enemy and know yourself, you need not fear the result of a hundred battles.” ~ Sun Tzu

One Claude Code session can do a lot, but it still works sequentially: one file, one task, one train of thought at a time 🚂. When your project has independent workstreams (backend, frontend, CI/CD, infrastructure), you’re waiting in line behind yourself 😤. What if you could describe the whole project and let a team of agents build it in parallel? 🤝 That’s exactly what agent teams do. We covered subagents in CCDD #6, but agent teams are a different beast entirely. In this article we’ll explore what sets them apart and put them through their paces by building a complete Kubernetes CronJob monitor from scratch 🏗️.

“Alone we can do so little; together we can do so much.” ~ Helen Keller

One of the most common activities in any program is constructing text from dynamic values 🐍. Think formatting log messages, constructing SQL queries, and preparing display messages for the user 🧵. Python started with the most basic %x formatting (printf-like) and evolved evermore sophisticated ways to do it. Python 3.14 is bringing us the latest technology in the form of template strings ✨. Keep reading to learn about the power of the new t-strings and worry a little less about XSS when interpolating user input into HTML 🛡️.

“Explicit is better than implicit.” – The Zen of Python, PEP 20

Over twelve CCDD articles we’ve explored Claude Code’s features one by one: CLAUDE.md, slash commands, skills, MCP, subagents, hooks, plugins, pipelines, the SDK, scheduling, and sandboxing 🧩. Each one introduced its own files, settings, and conventions. If you’ve been following along you might be wondering: where does everything actually go? 🤔 Today we zoom out and look at the full picture 🗺️. It can be challenging to see the forest from the trees 🌳, but in the end it’s all about context engineering and making sure Claude Code is well-behaved.

“For every minute spent organizing, an hour is earned.” ~ Benjamin Franklin

You gave Claude Code the keys to your machine 🔑. It can read your files, run commands, install packages, and talk to the network 🌐. That’s a LOT of trust for an entity that sometimes likes to take initiative in surprising ways. Not to mention what can happen when a prompt injection sneaks in, or when Claude decides to “helpfully” rewrite your shell config 😬. The answer is sandboxing: draw a line in the sand and let Claude go wild inside it, while everything outside stays safe 🔒.

“Quis custodiet ipsos custodes (Who will guard the guards themselves)?” ~ Juvenal

Every Python developer has been there. You write a quick script, it needs requests (or click, or pandas), and suddenly your “quick script” needs a whole project setup. What if the script itself could just say what it needs? This is where inline script metadata comes in…

“Simple is better than complex.” – The Zen of Python, PEP 20

You know what’s better than asking Claude to check something? Having Claude check it for you automatically, on a schedule, while you do something else ⏰. Claude Code has a built-in recurring task system that turns your session into a lightweight cron daemon. Set it up, walk away, come back to results 📋. In this CCDD article we’ll dig into the scheduling primitives, rant about naming choices, and track how a freshly published blog post performs on Medium 📊.

“Time is what we want most, but what we use worst.” ~ William Penn

Running Claude Code at the terminal is awesome 💻. Running it headless in CI is powerful 🚀. But the real leverage comes when you embed Claude Code directly in your own code. The Claude Agent SDK turns the full Claude Code machinery into a Python library 🐍 (there is a TypeScript SDK too). You have the entire Claude Code ecosystem at your fingertips (if you’re still typing that is 😉). In this CCDD article we will explore the Claude Agent SDK, compare it to the Anthropic Client SDK and build a useful CLI program using the Claude Agent SDK to help me manage the Gigi Zone blog. The customary quote is coming for the first time from Claude itself! I couldn’t find a better quote from a real person 🤖.

“The best automation is the automation that builds automation.” ~ Claude Sonnet 4.6

Claude Code in the terminal is awesome, but there are other ways to unleash Claude Code. One of the most useful ways is to integrate Claude Code into your CI/CD pipeline! Now, it can react to PR events, respond to user comments or just run on a schedule.

“If you automate a mess, you get an automated mess.” ~ Rod Michael

You’ve built your skills, wired up your hooks, and assembled your subagents. Now your .claude/ directory is a sprawling mess of Markdown files that only you understand, and sharing it with a teammate means either committing your deeply personal CLAUDE.md or handing them a zip file like it’s 2003.

There’s a better way. Plugins are how Claude Code goes from solo power tool to a shareable, versioned, installable ecosystem.

“The best code is code you don’t have to write.” ~ Jeff Atwood

CLAUDE.md files are cool. You write instructions, Claude reads them, and most of the time it follows them. But “most of the time” can be a problem. When Claude gets deep into a complex task, it sometimes forgets to run tests before committing, or edits a file it shouldn’t touch, or skips a validation step you explicitly asked for. Hooks solve this problem by giving you deterministic, automated enforcement that fires every single time, no exceptions.

“In theory, there is no difference between theory and practice. In practice, there is.”

Claude Code is a capable agent on its own, but some tasks are too big, too noisy, or too specialized for a single conversation to handle well 🤯. Subagents let Claude Code spawn autonomous child processes, each with its own context window, tools, and instructions, to divide and conquer 🧩.

Claude Code comes with several built-in subagents that are very useful. But, when you add custom subagents to the mix a single Claude Code instance becomes a full-fledged multi-agent system that can do anything and is not limited to coding 🚀. Then, the real story is who designs these subagents, when they are designed and built, and what that tells us about the future of human-AI collaboration.

“We have to learn the bitter lesson that building in how we think we think does not work in the long run.” ~ Rich Sutton

AI agents are all about tools 🧰. Without tools the agent can answer questions (including generating code for you), but you’ll have to take the answer (or code) and do something about it. You empower agents by giving them tools and let them iterate on tasks by invoking these tools ⚙️.

Claude Code comes out of the box with multiple useful tools ✨, including the Bash tool that lets it run any command-line program on your computer. But, often you may want to let it access additional capabilities. This is where the MCP (model context protocol) comes in and allows you to extend Claude Code with arbitrary tools and access to remote systems 🔌.

“Give us the tools, and we will finish the job.” ~ Winston Churchill

One of the most iconic scenes in the Matrix is when Neo uploads Jiu Jitsu and Kung Fu into his brain. Claude Code skills are EXACTLY the same. Let’s learn about the why, what, and how of agent skills and even create our own skill so Claude Code can generate images for us.

“I know Kung-Fu.” ~ Neo

When you start or resume a conversation with Claude Code it doesn’t start from scratch. It knows a lot right off the bat, and it can recall even more stuff later. Now, there are multiple ways to structure this memory such as CLAUDE.md files, agent skills and sub-agents. Today, we’re going to focus on CLAUDE.md files and level up your Claude Code game.

“One of the keys to happiness is a bad memory.” ~ Rita Mae Brown

This is the second article in the CCDD (Claude Code Deep Dive) series. If you haven’t read the first one, check out Claude Code Deep Dive - Basics where I covered getting started, permissions, sessions and running Claude Code in the terminal and IDE.

Today we’re diving into slash commands. We’ll cover both the built-in ones and how to create your own custom commands.

“Automation is not about replacing humans, it’s about amplifying them.” ~ Satya Nadella

In Confessions of an ex-coder I shared my workflow and how I let Claude Code do all the heavy lifting for me. It was super rewarding and productive, but I have used it at a very basic level. I promised to study it more deeply. This is it!

My way for studying anything is experimenting/building something and then writing about it. This is the first article in the CCDD (Claude Code Deep Dive) series.

I’ll cover some general and personal background and history of AI coding and then show some basic yet useful Claude Code tips.

“AI will soon be writing 90 percent of all code.” - Dario Amodei, Anthropic CEO, March 10 2025

This is the last part in our quest for blazing fast pods on Kubernetes. In Blazing Fast Pods - Placeholder Pattern we saw how to get a pod scheduled immediately. In Blazing Fast Pods - Image Preloading we ensured container images are already present on nodes. Now, we’ll complete the journey with the idle pool pattern - maintaining a pool of ready-to-go pods that can be claimed instantly when needed.

“The best time to plant a tree was 20 years ago. The second best time is now.” ~ Chinese Proverb

Let’s continue our journey towards blazing fast pods on Kubernetes. In Blazing Fast Pods - Placeholder Pattern

we saw how to get a pod scheduled immediately. But, that’s just the beginning. Once the pod is scheduled to a node Kubernetes needs to download the images of all its containers before it can run. Ideally, the images are already on the node. Let’s see how to make it happen every time.

“Victory loves preparation” ~ Latin proverb

🚨 IMPORTANT ANNOUNCEMENT!!!! 🚨 I decided to cut down on the emojis in my articles. If you come here for the emojis you’re out of luck ☘️.

In the Art and Science of Kubernetes Binpacking I waxed poetic about squeezing every last bit of resources in your cluster and minimizing your costs. This time we’re looking at a different use case - minimizing the time it takes to get new pods up and running. Let me be very clear it’s going to cost you 💰!

“Speed costs money. How fast do you want to go?” ~ Unknown

I used to be a coder 👨‍💻. Every day I would pop up my IDEs (yes, plural) and happily bust some code. Life was good ✨. But, then LLMs arrived and the world has changed 🌍. I am no longer a coder. I am a Clauder 🤖!

I’ll tell you where I am coming from, where I’m at right now and where I plan to go 🚀.

“AI will change the nature of creativity, so we must develop new ways to interact with these machines.” ~ Yves Behar

The last days of programming are upon us. The last programming language is finally here. You may not like it 😠. You may fight it ⚔️. But it is inevitable 💥. In this post, we will explore the journey from traditional programming to the era of AI coding agents and the implications of this transformation 🔥.

“I am no master” ~ Leroy Green (The Last Dragon)

In this story of mystery and intrigue 🕵️‍♂️ (well, not really 😅), we’ll explore the world of LLMs 🤖 and how I extended the awesome simple-openai library to support the Gemini LLM provider from Google via its OpenAI API compatibility layer ✨. This is a tale of code 💻, collaboration 🤝, and the quest for robust interoperability 🔧 in the ever-evolving landscape of AI.

“The good thing about standards is that there are so many to choose from” ~ Andrew S. Tanenabaum

Github Actions are pretty cool 😎 as far as CI/CD pipelines go. But evolving and testing non-trivial workflows can go south real fast ⚠️. The core issue is that for any other code, the CI/CD system (if designed and implemented correctly) keeps you honest ✅, runs your tests 🧪, and only accepts successful PRs 🚦.

But when you need to make changes to your CI/CD workflows, the situation is different 😬. I learned it the hard way, so you don’t have to 💡. Let me share with you our current software development lifecycle for Github Actions workflows 🔁. The same principles can be applied to any CI/CD system 🛠️.

“Quis custodiet ipsos custodes?” ~ Juvenal

AI agents are all the rage these days. They’re the new shiny ✨ toy that everyone wants to play with 🧸. There’s only one small problem: they’re not real 👻. Yes, you heard me right 👂. AI agents are just a figment of our AI frameworks 🛠️ — really just an anthropomorphism 🧍‍♂️ of a very useful pattern for interacting with LLMs.

“The question of whether a computer can think is no more interesting than the question of whether a submarine can swim.” ~ Edsger W. Dijkstra

Two weeks ago I became a manager 👨‍💼 again. Throughout my career, I have managed several teams. I was even a director of software development a couple of times. But, I always stayed close to the code 💻, and remained hands-on. This time may be different. I’ll tell you why. But first, let me introduce my team: Devin, Claude and OpenHands… 🤖

“Unfortunately, humans have a long history of trying to fix their engineering mistakes with more engineering mistakes!” ~ Steven Magee

Building optimized container images is a science 🧪 and an art 🎨. You need to know what you’re doing to get it right. In this post, we will build a Python-based 🐍 container image for troubleshooting Kubernetes ☸️️. Along the way, we will discuss some of the best practices for building images, when they apply and when they don’t! 🔍

For the impatient:
https://github.com/the-gigi/docker-images/tree/main/py-kube

“When you realize the difference between the container and the content, you will have knowledge.” ~ Idries Shah

Observability 🔭 is table stakes for any serious system. But, when you throw GenAI and LLMs into the mix it is no longer just nice to have. That’s the game at Invisible, where we’re wrangling LLMs in our multi-agent web automation platform on the daily. Let’s pop the hood and see how we made the leap from black box 🤷 to glass box🔎 with LLM Observability. Designing, building and integrating LLM observability into our stack was one of my favorite projects at Invisible.

“Without data, you’re just another person with an opinion.” ~ W. Edwards Deming

Everything on the internet starts with DNS 🌐, so it’s kinda important ⭐. Cloudflare is one of the top DNS providers and it offers the unique capability of intercepting any HTTP(S) request right after DNS resolution and mess around with it 🚀. Let’s see how this works and how we have used it effectively at Invisible to handle elegantly a thorny issue with a 3rd party provider.

“It’s not DNS. There’s no way it’s DNS. It was DNS” ~ Every network engineer, ever

Ephemeral storage 📦 is a fancy way of saying temporary storage. In Kubernetes ☸️️, it means storage that is associated with a pod that disappears when the pod goes away. This is almost true 🤔 as we shall find out soon. We will see what happens when pods and nodes run out of ephemeral storage ❌ and how to troubleshoot it 🔍.

“I like the ephemeral thing about theatre, every performance is like a ghost - it’s there and then it’s gone” ~ Maggie Smith

At the beginning of 2024, I joined Invisible Platforms 🥷 - a small startup with a super-talented team (I worked with 95% of the engineers before) and a big mission. I’m the infra guy 🛠️, which covers a lot of ground in a startup. I got to wear many hats 🤠, play with some cool toys and even made myself useful on occasion. Let’s see what I’ve been up to in 2024!

“Why, sometimes I’ve believed as many as six impossible things before breakfast.” ~ The Queen from Alice in Wonderland

Horizontal pod autoscaling is one of the best features of Kubernetes ☸️. In this short blog post I’ll take quick a look at the horizontal pod autoscaler (HPA) and point out some gotchas.

“The best measure of a man’s honesty isn’t his income tax return. It’s the zero adjust on his bathroom scale” ~ Arthur C. Clarke

☸️ Kubernetes is a powerful platform for running containerized workloads. It will happily scale your infrastructure to meet the demands of your applications. Unfortunately, this can get expensive ( especially in the cloud). The goal is to utilize the resources you provision efficiently. In this post, we’ll explore the ins and outs of Kubernetes bin packing 📦, which is one important piece of the puzzle. We will also show how bin packing interacts without other aspects of optimizing resource usage.

“Letting go is even more important than adding.” ~ Marie Kondo

Today we are going deep on upgrading Postgres on AWS RDS to a new major version. You may be shocked, but there will be very few emojis in this post. Database upgrades are no joke, so let’s be serious for a change. 🫡

A SQL query walks into a bar, sees two tables and asks, “Can I join you?” ~ Anonymous

The grand finale is here, where lazy man🚶‍ Gigi defeats anti-Javascript counter-measures in password-protected forms using the Rust programming language 🦀.

The complete source code for the entire project is available here:

https://github.com/the-gigi/auto-web-login

“Laziness is the mother of invention.” ~ Anonymous

Welcome to episode #3, where lazy man🚶‍ Gigi recruits AppleScript 🍎 to automatically close browser tabs on his behalf.

If you’re impatient the complete source code for the entire project is available here:

https://github.com/the-gigi/auto-web-login

“Laziness is the mother of invention.” ~ Anonymous

In this episode, lazy man🚶‍ Gigi can’t be bothered to manage two collections of items in a TamperMonkey 🐒 script, so he is forced to write a Python 🐍 program to generate the script.

If you’re impatient the complete source code for the entire project is available here: https://github.com/the-gigi/auto-web-login

“Laziness is the mother of invention.” ~ Anonymous

In the olden days 🏰 (a few years ago) upgrading a Kubernetes cluster was a stressful task 😰 that could bring the system down 💥 or at the very least cause major disruption, deployment moratorium 🚫, and significant toil for most of the engineering team 😓. I’m happy to report that this is no longer the case 🎉…

Once upon a time there was a lazy man🚶‍. Let’s call him Gigi (because that’s his name) 😊. Gigi would go to extreme lengths to avoid work. This is the story of how Gigi used browser automation 🤖 to avoid clicking the mouse 🐭 a few times on a web page.

“Laziness is the mother of invention.” ~ Anonymous

You have an AWS EKS cluster ⎈ and your workloads use a managed Postgres DB 🌐 deployed on AWS RDS. You need to access it from your laptop using your favorite DB visualization tool. What’s the best way to do it 🤷 ?

Keep reading to find out…

Imagine that nobody has to type git clone ever again! Can you even begin to comprehend what a gigantic improvement in the quality of life of every developer it would be? This pipe dream has just become a reality! Enter Cyclone🌀 - a little Rust program I built to make the world 🌍 a better place.

WARNING: This is a Kubernetes-free Post

What do you know about the intricate web 🕷️ of interactions between advanced Kubernetes scheduling techniques and the cluster autoscaler? Probably not enough… read on to get a taste of some gnarly nuances and how to deal with them.

Spot instances are da bomb 💣!! You just have to be careful they don’t explode in your face 💥🤯 Let’s explore how to effectively use spot instances with Kubernetes…

It was a dark night. The wind was howling outside. Suddenly, I got a Slack message that our EKS cluster has run out of IP addresses 😱. To be continued…