Today (Monday 5/17/2021) morning I got an email from Let’s Encrypt talking about its Root CA expiration on 9/30/2021. Thanks for this early notice which gives me abundant time to prepare for it.

I still remember last year when the Sectigo root certificate expired I only realized after one of my services was broken.

The first thought for me is “oh my xx let me check if I will be impacted again”. After some research I find the method to verify if a client/server will be impacted by the change, and explore the method to resolve.

This article is to…

Continuous integration/Continuous delivery pipelines are widely used in modern software development. To quickly and reliably develop/deliver software, a change goes through the pipelines and reach the production.

Software CI/CD pipeline

How about the CI/CD pipelines themselves? Are they also created from code? Do they follow the same software development/delivery process?

This article describes my step to clean up the mysterious disk usage by Docker in a Kubernetes cluster.

It has been a long time since I found some strange disk usage in some of my Kubernetes nodes. Among a few similar nodes, a few of them have used much more disk space than others. After some comparison I found the difference is in folder /var/lib/docker/overlay2

Some search led me to this couple years old post.

After some basic Docker cleanup:

# docker system df

This article describes my recent experience to fix a random network “Connection Reset” issue in CI/CD pipelines running in Docker/Kubernetes when downloading binaries from an external server.

I’d like to share my experience as eventually I realized this is a very common use case — When a Container/Pod running in Docker/Kubernetes retrieves data from external services, the random connection reset problem could happen.

I have the Jenkins pipelines running in Kubernetes clusters. Each build task runs in a Pod acting as Jenkins worker . During the build, binaries are downloaded from Jfrog Artifactory server and the build outputs are put…

This article describes my experiences to apply iptables firewall for Docker/Kubernetes hosts.

My solution is based on this article so it’s not a new idea. But I write it down as I still see comments here and there about the troubles in making iptables playing well with Docker.

When moving my most workload into Kubernetes clusters, I found the same solution applies very well as the host level firewall to block undesired access.

Traditional iptables firewall

iptables is a command line tool to config Linux’s packet filtering rule set. One of the usages is to create host level firewall to block unwanted network…

Dynamic Jenkins agent provisioning in Kubernetes


Jenkins is a CI/CD tool with a long history and keeps evolving itself. It’s Master/Agent architecture is great for scalability to do distributed builds. There are many ways to provision Jenkins Agent, from using bare metal machines, Virtual Machines, dynamic EC2 instances, containers from Docker, or Kubernetes clusters.

The integration between Jenkins and Kubernetes cluster is great. With these benefits, I’ve fully migrated CI/CD pipeline from host(VM) based Agent to Pod based Agent.

  • Dynamic Jenkins agent from Kubernetes, light weighted, provisioned on-demand within a few seconds
  • Fresh and reproducible Jenkins agent environment for every…

Consul cluster in Kubernetes, Making it Production Ready


HashiCorp Consul is used as K/V store in my Kubernetes clusters. The system architecture looks like below, as described in my previous articles about Traefik as Ingress Controller and highly available Vault cluster.

As the storage back-end, the performance, stability and availability of Consul is very important to for the Traefik cluster and Vault cluster being highly available.

In this article I describe my recent approaches to make the Consul cluster in Kubernetes as production-ready as possible. I have 3 goals:

  • In daily operation, Consul shall be stable without leader loss.
  • In…

I’d like to share my recent experience to switch my working setup to Ubuntu 19.10, with a HP ZBook 15 G5 + HP Dock G2.


Hp ZBook 15 G5 laptop has Intel/Nvidia graphics cards.

HP G2 Docker(link) is a cool cubic dock which connects to 2 external display, Ethernet cable, keyboard, etc.

Write down my experience to resolve a Docker in Docker network issue when doing docker build in Kubernetes


After writing about my approach to do fast docker build in Kubernetes in this article, I want to build some complex docker images.

A problem hit me hard that I could not download content from “some” places when building the image.

For example, if the Dockerfile has this line, it works well.

RUN curl -L ""

But this line cause the “docker build” command stuck and eventually fail.

RUN curl -L -O ""

The error could happen not only in curl…

Speed up docker build with cache in Kubernetes environment

This article describes my recent approach to do fast docker build with the DOCKER_BUILDKIT enhancement introduced since 18.09, and within a Kubernetes environment.

Below is the diagram.

Note: To perform the demo in this article, a running Kubernetes cluster is required.


Docker caching layer is very important for a fast docker build when you change a little bit in a large Dockerfile. There are many articles to describe this. [1], [2]

In my earlier docker build tasks designed a year ago, my method is to directly use a host machine with…

Liejun Tao

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store