Continuous integration/Continuous delivery pipelines are widely used in modern software development. To quickly and reliably develop/deliver software, a change goes through the pipelines and reach the production.

Software CI/CD pipeline

How about the CI/CD pipelines themselves? Are they also created from code? Do they follow the same software development/delivery process?

This article describes my step to clean up the mysterious disk usage by Docker in a Kubernetes cluster.

It has been a long time since I found some strange disk usage in some of my Kubernetes nodes. Among a few similar nodes, a few of them have used much more disk space than others. After some comparison I found the difference is in folder /var/lib/docker/overlay2

Some search led me to this couple years old post.

After some basic Docker cleanup:

# docker system df TYPE TOTAL ACTIVE SIZE RECLAIMABLE Images 27 27 4.52GB 433.8MB (9%) Containers 61 52 156.6kB 0B…

This article describes my recent experience to fix a random network “Connection Reset” issue in CI/CD pipelines running in Docker/Kubernetes when downloading binaries from an external server.

I’d like to share my experience as eventually I realized this is a very common use case — When a Container/Pod running in Docker/Kubernetes retrieves data from external services, the random connection reset problem could happen.

I have the Jenkins pipelines running in Kubernetes clusters. Each build task runs in a Pod acting as Jenkins worker . During the build, binaries are downloaded from Jfrog Artifactory server and the build outputs are put…

This article describes my experiences to apply iptables firewall for Docker/Kubernetes hosts.

My solution is based on this article so it’s not a new idea. But I write it down as I still see comments here and there about the troubles in making iptables playing well with Docker.

When moving my most workload into Kubernetes clusters, I found the same solution applies very well as the host level firewall to block undesired access.

Traditional iptables firewall

iptables is a command line tool to config Linux’s packet filtering rule set. One of the usages is to create host level firewall to block unwanted network…

Dynamic Jenkins agent provisioning in Kubernetes


Jenkins is a CI/CD tool with a long history and keeps evolving itself. It’s Master/Agent architecture is great for scalability to do distributed builds. There are many ways to provision Jenkins Agent, from using bare metal machines, Virtual Machines, dynamic EC2 instances, containers from Docker, or Kubernetes clusters.

The integration between Jenkins and Kubernetes cluster is great. With these benefits, I’ve fully migrated CI/CD pipeline from host(VM) based Agent to Pod based Agent.

  • Dynamic Jenkins agent from Kubernetes, light weighted, provisioned on-demand within a few seconds
  • Fresh and reproducible Jenkins agent environment for every…

Consul cluster in Kubernetes, Making it Production Ready


HashiCorp Consul is used as K/V store in my Kubernetes clusters. The system architecture looks like below, as described in my previous articles about Traefik as Ingress Controller and highly available Vault cluster.

As the storage back-end, the performance, stability and availability of Consul is very important to for the Traefik cluster and Vault cluster being highly available.

In this article I describe my recent approaches to make the Consul cluster in Kubernetes as production-ready as possible. I have 3 goals:

  • In daily operation, Consul shall be stable without leader loss.
  • In…

I’d like to share my recent experience to switch my working setup to Ubuntu 19.10, with a HP ZBook 15 G5 + HP Dock G2.


Hp ZBook 15 G5 laptop has Intel/Nvidia graphics cards.

HP G2 Docker(link) is a cool cubic dock which connects to 2 external display, Ethernet cable, keyboard, etc.

Write down my experience to resolve a Docker in Docker network issue when doing docker build in Kubernetes


After writing about my approach to do fast docker build in Kubernetes in this article, I want to build some complex docker images.

A problem hit me hard that I could not download content from “some” places when building the image.

For example, if the Dockerfile has this line, it works well.

RUN curl -L ""

But this line cause the “docker build” command stuck and eventually fail.

RUN curl -L -O ""

The error could happen not only in curl…

Speed up docker build with cache in Kubernetes environment

This article describes my recent approach to do fast docker build with the DOCKER_BUILDKIT enhancement introduced since 18.09, and within a Kubernetes environment.

Below is the diagram.

Note: To perform the demo in this article, a running Kubernetes cluster is required.


Docker caching layer is very important for a fast docker build when you change a little bit in a large Dockerfile. There are many articles to describe this. [1], [2]

In my earlier docker build tasks designed a year ago, my method is to directly use a host machine with…

Mutual Auto-Unseal Two Vault clusters in Kubernetes


When I deploy Vault to Kubernetes, I realize it’s important to have auto-unseal capability to make the Vault cluster true highly available.

In my previous article “Highly available Vault cluster in Kubernetes”(link), even I’ve tried hard to make a Vault cluster as highly available as possible, without auto-unseal, the Vault cluster could tolerate partial pods failure, but won’t survive a whole cluster reboot.

This article well explained the why and how. The problem for me is I don’t have “AWS KMS service” to use, or any similar cloud vendor security services, because I…

Liejun Tao

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store