Continuous integration/Continuous delivery pipelines are widely used in modern software development. To quickly and reliably develop/deliver software, a change goes through the pipelines and reach the production.
How about the CI/CD pipelines themselves? Are they also created from code? Do they follow the same software development/delivery process?
This article describes my step to clean up the mysterious disk usage by Docker in a Kubernetes cluster.
It has been a long time since I found some strange disk usage in some of my Kubernetes nodes. Among a few similar nodes, a few of them have used much more disk space than others. After some comparison I found the difference is in folder /var/lib/docker/overlay2
Some search led me to this couple years old post.
After some basic Docker cleanup:
# docker system df
This article describes my recent experience to fix a random network “Connection Reset” issue in CI/CD pipelines running in Docker/Kubernetes when downloading binaries from an external server.
I’d like to share my experience as eventually I realized this is a very common use case — When a Container/Pod running in Docker/Kubernetes retrieves data from external services, the random connection reset problem could happen.
I have the Jenkins pipelines running in Kubernetes clusters. Each build task runs in a Pod acting as Jenkins worker . During the build, binaries are downloaded from Jfrog Artifactory server and the build outputs are put…
This article describes my experiences to apply iptables firewall for Docker/Kubernetes hosts.
My solution is based on this article so it’s not a new idea. But I write it down as I still see comments here and there about the troubles in making iptables playing well with Docker.
When moving my most workload into Kubernetes clusters, I found the same solution applies very well as the host level firewall to block undesired access.
iptables is a command line tool to config Linux’s packet filtering rule set. One of the usages is to create host level firewall to block unwanted network…
Dynamic Jenkins agent provisioning in Kubernetes
Jenkins is a CI/CD tool with a long history and keeps evolving itself. It’s Master/Agent architecture is great for scalability to do distributed builds. There are many ways to provision Jenkins Agent, from using bare metal machines, Virtual Machines, dynamic EC2 instances, containers from Docker, or Kubernetes clusters.
The integration between Jenkins and Kubernetes cluster is great. With these benefits, I’ve fully migrated CI/CD pipeline from host(VM) based Agent to Pod based Agent.
Consul cluster in Kubernetes, Making it Production Ready
HashiCorp Consul is used as K/V store in my Kubernetes clusters. The system architecture looks like below, as described in my previous articles about Traefik as Ingress Controller and highly available Vault cluster.
As the storage back-end, the performance, stability and availability of Consul is very important to for the Traefik cluster and Vault cluster being highly available.
In this article I describe my recent approaches to make the Consul cluster in Kubernetes as production-ready as possible. I have 3 goals:
I’d like to share my recent experience to switch my working setup to Ubuntu 19.10, with a HP ZBook 15 G5 + HP Dock G2.
Hp ZBook 15 G5 laptop has Intel/Nvidia graphics cards.
HP G2 Docker(link) is a cool cubic dock which connects to 2 external display, Ethernet cable, keyboard, etc.
Write down my experience to resolve a Docker in Docker network issue when doing docker build in Kubernetes
After writing about my approach to do fast docker build in Kubernetes in this article, I want to build some complex docker images.
A problem hit me hard that I could not download content from “some” places when building the image.
For example, if the Dockerfile has this line, it works well.
RUN curl -L "https://google.com"
But this line cause the “docker build” command stuck and eventually fail.
RUN curl -L -O " https://download.jetbrains.com/python/pycharm-community-2019.2.1.tar.gz"
The error could happen not only in curl…
Speed up docker build with cache in Kubernetes environment
This article describes my recent approach to do fast docker build with the DOCKER_BUILDKIT enhancement introduced since 18.09, and within a Kubernetes environment.
Below is the diagram.
Note: To perform the demo in this article, a running Kubernetes cluster is required.
In my earlier docker build tasks designed a year ago, my method is to directly use a host machine with…
Mutual Auto-Unseal Two Vault clusters in Kubernetes
When I deploy Vault to Kubernetes, I realize it’s important to have auto-unseal capability to make the Vault cluster true highly available.
In my previous article “Highly available Vault cluster in Kubernetes”(link), even I’ve tried hard to make a Vault cluster as highly available as possible, without auto-unseal, the Vault cluster could tolerate partial pods failure, but won’t survive a whole cluster reboot.
This article well explained the why and how. The problem for me is I don’t have “AWS KMS service” to use, or any similar cloud vendor security services, because I…