Buildah, Podman, and Skopeo

Still doing all your Linux container management using an insecure, bloated daemon? Well, don’t feel bad. I was too until very recently. Now I’m finding myself slowly saying goodbye to my beloved Docker daemon and saying hello to Buildah, Podman, and Skopeo. In this article, I explore the exciting new world of rootless and daemon-less Linux container tools.

Some History

If you are anything like me, Docker and containers might as well be the same word. Dockerfiles and build, pull, run, tag, and push commands of the Docker CLI have become part of my daily life as a software developer for the last few years. However, days of the monolith daemon approach to container management are pretty much coming to an end. There have been two big major factors leading to this. First, Docker and CoreOS, along with few other organizations, started Open Container Intiative (OCI) as an attempt to standardize container runtime and image format specifications in 2015. OCI efforts have since matured quite a bit. The OCI image format specification is now supported by most, if not all, mainstream image registries (e.g. Docker Hub, Amazon’s ECR, etc). The OCI runtime specification has a reference implementation in runc, and most existing container runtimes are either OCI compliant already or have OCI compliance in their roadmap.

Second, Kubernetes created CRI to start supporting vendors other than Docker for their container runtime implementation. The story behind the creation of CRI is actually pretty amusing. Essentially, in the beginning, Docker was the only supported runtime in Kubernetes. Naturally, CoreOS wanted rkt support in Kubernetes, and therefore submitted a pull request that basically had a bunch of “if some-flag do rkt else do docker“. Kubernetes saw this approach breaking the open-closed principle and decided to do the right thing by creating CRI as a high level abstraction to be implemented by vendors like Docker, CoreOS, and whoever else. Meanwhile, the landscape of the container ecosystem changed in one significant way. Docker had enjoyed its status as the de facto runtime and image format standard in Linux containers from 2013 till about 2017. Their image format and runtime were constants in an otherwise heated war in the container orchestration space. However, since 2017 or so, Kubernetes has decidedly won that war and become the de facto standard in container orchestration.

All these developments mean Docker is suddenly finding itself drifting away from the center of the Linux container universe. Instead of enjoying other projects orbiting its enormous gravity well, Docker is slowly finding itself becoming one of the many others orbiting the gravity well of Kubernetes. This excites some people who always saw the monolith daemon that required root access for everything as a problem. Which brings us to the heart of this article – the daemon-less and largely rootless suite of container management tools.

New Generation of Container Management Tools

Buildah builds, Podman runs, and Skopeo transfers container images. These are open source tools maintained by the containers organization on Github. There is some overlap in functionality between Buildah and Podman, but the separation of core responsibilities is clear. None of these tools require a running daemon, and for the most part don’t need root access either (you will see one exception to the rootless rule in Podman in this article). So, having set the stage, I’ll spend the rest of the article sharing how I quickly ramped up on the three tools using my recent dummy application, Task Master, from my Gophin Off with Linked Lists series. You don’t need to go through that series for this article. Basically Task Master is a dumb little web app written in Go that I will containerize and play with using these tools. Grab the source code here.

Before we get started, one caveat with these tools is that they are very much Linux tools. This should be obvious given they are dealing with Linux containers. After all, Docker itself is specific to Linux. However, Docker does a good job of hiding behind a VM in MacOS and Windows to give those user bases a seamless experience. These tools, being fairly new on the block, don’t yet have those conveniences. You will have to grab a Linux VM if you want to play with them on your Windows or MacOS machines. OK let’s get started.

Containerizing using Docker

I wrote the Task Master web app trying to learn Golang. So, I haven’t yet bothered containerizing it yet. So, before jumping in to the new tools, let’s containerize the app using good ol’ Docker. Dockerfile for a simple Go app is pretty straightforward

The Dockerfile assumes there is a standalone executable already present in the top level of the Dockerfile context. It copies it to the root directory of the container file system. It advertises 8080 as a port to expose, since that’s the port where Task Master listens for incoming HTTP requests. Finally, the copied over Task Master executable is set as the entry point. Alright, so now I need to build the executable for Task Master so that the Docker build can run. To keep things seamless, I’ll go ahead and write a build script so that I can just run one command to build both the Go executable as well as the Docker image.

I still haven’t figured out how to run the Go test and build commands safely while being in another directory. The compiler seems to have weird issues with fully qualified file paths. So that’s why you see the pushd, trap, and popd business here. Otherwise things are straightforward.

  • Run unit tests.
  • Build the standalone executable targeting Linux.
  • Build the container image.

Alright time to run this bad boy.

Great! Do note the awesomely minimal container image size of just 6M. Go go! Alright, let’s do some light testing.

Ok, so that’s just plain old Docker. Knowing that things work, let’s convert this over to use our new tools. 

Installing the tools

Installation of these tools is very straightforward. Just grab your Linux package manager and install the buildah, podman, and skopeo packages. Being on Fedora, I use dnf.

Buildah Using Dockerfiles

So the cool thing about migrating to Buildah is that it supports Dockerfiles. So the easiest thing to do here is to use Buildah’s bud (short for build-using-dockerfile) command.

Note how only one image layer, as opposed to one layer per step in Docker, is created. And just like that we have our image. Do note that I am using the buildah images command to see my image. I have, in fact, gone ahead and uninstalled Docker on my machine completely. This image is also saved underneath my home directory instead of /var/lib. What this means is this image is local to my user on this machine. If I run buildah images as any other user on this machine, I won’t see that image. Quick way to illustrate this is using sudo to build the image as root.

Note how buildah images shows no images while sudo buildah images shows the image we just built.

Buildah beyond Dockerfiles

Buildah is much more than just a third party tool to process Dockerfiles however. Buildah allows you to interactively build container images one step at a time. It does this by spawning an instance of the container from some base image. You can then use this container to execute all the necessary steps to get to your final image or some intermediate layer. Once you are done with a layer of the build, you can commit the container up to that point as an image tag to Buildah, and restart the process from that tag as the base image. Once you are completely done, commit the final tag and remove the working containers.

To see this in practice, let’s recreate the way Docker built our image using Buildah. First we spawn a working container using the scratch image.

Alright, so we have our container instance named working-container to work with. Next let’s create the maintainer label, commit the change, and spawn another intermediate container.

So in true docker build fashion, we execute a step, capture that step in an intermediate layer, spawn a new working container from intermediate layer to execute the next step, and finally remove the first working container. We can repeat the process to execute the rest of the steps.

I removed the output of the commands above for brevity. Instead, we can look at the output of the commands below for the big picture instead.

Now this is overkill. I do not need to keep each step separated in its own image layer. This was just to illustrate the flexibility allowed by buildah. In practice, I probably would want to put all these steps in the same layer. I also would like to script these steps so that they become part of my build file.

That does the job. Do also note how I am able to combine all my config steps in one command. Now to incorporate this into my build file, I will create a flag called –using-buildah that the caller can supply to build using buildah. By default I can have my script default to Docker.

Note how the last line that removes the working container has been moved to a cleanup function that is called by the trap. This allows the working container to be removed in case one of the intermediate buildah steps fails. Now we can run the buildah flow in the build script using the –using-buildah flag.

Awesome! So now that we can build them, it’s time to run our container using Podman.

Running Containers with Podman

Podman is pretty straight forward. For all your docker run needs, just replace docker with podman. So the container I just built can easily be run using podman.

Just like buildah, podman also keeps containers user specific.

Great! Next we would ideally want to test to make sure podman is actually running the container instance. However there is one minor issue. Unfortunately, you can’t do port bindings as a non root user. This means I can’t expose the port on which Task Master listens for requests outside the container.

So to run my app so that it can actually be used, I need to run it via podman as a root user. Now I have two options. I can either rebuild the image as root using buildah. Or, I can do something unnecessarily complicated. Let’s do the latter. I will run a Docker registry locally exposed on port 5000. Then I can push my image using buildah as sahsingh user to this registry, so that podman, run as root, can pull it.

Great! Now we can run and test the image.

Runs just like before and all without a daemon!


Finally, let’s look at skopeo. Skopeo is all about working with images in remote repositories – transferring them, inspecting them, and even deleting them. So, let’s use skopeo to transfer the Task Master image from the local Docker registry I am running using podman to my official Docker Hub account. First, let’s inspect

Seeing how I have the image at localhost:5000 but not at Docker Hub, let’s do a copy. I have replaced my actual username and password below with a stub user:pass.



So with that this is it for this blog entry. Hope you enjoyed learning about the new set of daemonless and rootless container tooling. Till next time..

6 Replies to “Buildah, Podman, and Skopeo”

  1. Thanks on your marvelous posting! I really enjoyed reading it, you can be a great author. I will make certain to bookmark your blog and definitely will come back in the future. I want to encourage that you continue your great job, have a nice afternoon!|

Leave a Reply

Your email address will not be published.