Jenkins is a widely used open-source automation server that enables teams to build, test, and deploy software continuously. With its declarative and scripted pipeline syntax, extensive plugin ecosystem, and flexible job scheduling, Jenkins covers the full CI/CD lifecycle — from simple builds to complex multi-stage delivery pipelines.

This curated series provides a comprehensive overview of Jenkins pipelines, job management, and administration. It covers core pipeline mechanics, Groovy scripting, parameterized and parallel builds, and server maintenance essentials.

Getting Started

• Intro to Jenkins 2 and the Power of Pipelines
• Scripted vs Declarative Pipelines in Jenkins
• Difference Between Agent and Node in Jenkins Pipeline
• Comments in a Jenkinsfile
• Jenkins Best Practices

Pipeline Mechanics

• Set Stage Status in Jenkins Pipelines
• How to Use Conditional Constructs in Jenkins Pipeline
• Get the Output of a Shell Command in a Jenkins Pipeline
• Skip a Stage in a Jenkins Pipeline
• Check if a File Exists in a Jenkins Pipeline
• Jenkins Pipeline – Change to Another Folder
• Jenkins Pipeline – Git Checkout With Credentials

Groovy and Scripting

• Utilizing Global Variables in Jenkins Pipeline
• Reading a File From Workspace in Jenkins Using Groovy
• Return Stdout, Stderr, and Status From Jenkins sh Script
• Getting a JSON Object as a Parameter in a Jenkins Pipeline
• Fixing the “No Such DSL method” Error in Jenkins Pipeline

Job Management

• Guide to Jenkins Parameterized Builds
• How to Check if a Pipeline Parameter Is Empty in Jenkins
• Trigger Another Job from a Jenkins Pipeline
• Running Stages in Parallel With Jenkins Workflow / Pipeline
• Run the Same Jenkins Job Multiple Times in Parallel
• Using a Jenkins Pipeline to Checkout Multiple Git Repos Into Same Job
• How to Create Dynamic Stages in a Jenkins Pipeline

Administration

• Export/Import Jobs in Jenkins
• Setting Up a Jenkins Slave Node
• Listing Installed Jenkins Plugins
• Updating Jenkins Plugins From the Command Line
• How to Restart Jenkins Safely
• How to Get the API Token for Jenkins

     

Related Stories

• Git Series
• Troubleshoot Docker Daemon Connection Issues in Jenkins
• Troubleshooting Jenkins 403 No Valid Crumb in Request Error

 

Git is the industry-standard distributed version control system used to track changes, coordinate work across teams, and maintain the full history of a codebase. From committing and branching to merging and rebasing, mastering git is foundational for every developer and operations professional.

This curated series provides a practical overview of essential Git workflows and techniques, from setting up credentials and managing branches to rewriting history and resolving merge conflicts.

Git Essentials

• Git for Beginners: The Definitive Practical Guide
• Git vs. GitHub
• Best Practices for Git Commit Message
• Configuring git Credentials
• Adding Git Credentials in VS Code

Branches and Synchronization

• Find the Differences Between Two Git Branches
• Clone All Remote Branches in Git
• Synchronizing Local and Remote Branches in Git
• Determine When a Git Branch Was Created
• Move Existing, Uncommitted Work to a New Branch in Git
• Delete All Git Branches Which Have Been Merged

Commits and History

• Squash the Last X Commits Using Git
• Remove a Large File from Commit History in Git
• Undo and Revert Commits in Git
• How to Modify Git Commit Messages
• How to Control and Ignore Whitespace Changes in Git Commands
• Searching for a String in Git Commits

Merging and Rebasing

• Merge Strategies in Git
• Difference Between git merge and rebase
• Git Merging: Conflict Resolution and Undoing a merge
• Guide to Undo a git rebase
• How to Fix Git “Refusing to Merge Unrelated Histories”

Staging and Stash

• Guide to Git Staging and the Stash
• Difference Between Git Stash Pop and Git Stash Apply
• How to Recover a Dropped Stash in Git
• Remove File From Git Repository Without Deleting It Locally
• How to Discard Unstaged Changes in Git
• Removing Tracked Files With .gitignore

     

Related Stories

• Jenkins Series
• Using a Package Manager in GitHub Actions
• Get Files Modified in a PR in GitHub Actions

 

1. Overview

The use of environment variables is common in Ansible to make deployments flexible and secure.

In this tutorial, we’ll discuss how to set both temporary and permanent environment variables using Ansible. The version of Ansible we use in the examples is 2.12.0.

2. Setting Temporary Environment Variables

In this section, we’ll see how to set a temporary environment variable to be used within a playbook.

2.1. Example Playbook

We’ll use the following playbook, playbook_env.yml:

$ vi +"set number" playbook_env.yml
 1 ---
 2   - name: Play1
 3     hosts: localhost
 4     environment:
 5       ENV_VARIABLE: PLAY_LEVEL
 6
 7     tasks:
 8     - name: Task1
 9       command: "echo Environment variable: $ENV_VARIABLE"
10       environment:
11         ENV_VARIABLE: TASK_LEVEL
12       register: print_environment_variable
13
14     - debug:
15         var: print_environment_variable.stdout_lines
16
17     - name: Task2
18       command: "echo Environment variable: $ENV_VARIABLE"
19       register: print_environment_variable
20
21     - debug:
22         var: print_environment_variable.stdout_lines

The +”set number” option of vi displays line numbers.

In addition to the playbook, we’ll use the following inventory file:

$ cat inventory
localhost ansible_connection=local

Let’s discuss the playbook shortly in the following subsection.

2.2. Dissection of the Playbook

The playbook has a single play, Play1, which is shown on lines 2-5. The play uses the environment keyword, which defines an environment variable for an action to be taken. Its name is ENV_VARIABLE, and its value is PLAY_LEVEL. This environment variable is accessible from all tasks within the play, as it’s defined at the play level.

The first task, namely Task1 (Lines 8-12), also uses the environment keyword to define an environment variable having the name ENV_VARIABLE and value TASK_LEVEL. We define this environment variable at the task level. Since its name is the same as the name of the one at the play level, the value at the task level overrides the value at the play level.

When we execute this task, we run the echo Environment variable: $ENV_VARIABLE command to display the environment variable’s value. We create a variable, print_environment_variable, to save the output of the echo command (Line 12). We save the variable using the keyword register. Then, we print this variable using the built-in debug module and its var parameter (Lines 14-15).

The other task, Task2 (Lines 17-19), runs the same echo command to display the environment variable’s value. However, it prints the environment variable defined at the play level, as we’ll see shortly. We capture the output of the echo command and print it in the same manner as in Task1 (Lines 21-22).

2.3. Running the Playbook

Having discussed the playbook shortly, let’s now run the playbook using the ansible-playbook command:

$ ansible-playbook playbook_env.yml -i inventory
PLAY [Play1] *******************************************************************
TASK [Gathering Facts] *********************************************************
ok: [localhost]
TASK [Task1] *******************************************************************
changed: [localhost]
TASK [debug] *******************************************************************
ok: [localhost] => {
    "print_environment_variable.stdout_lines": [
        "Environment variable: TASK_LEVEL"
    ]
}
TASK [Task2] *******************************************************************
changed: [localhost]
TASK [debug] *******************************************************************
ok: [localhost] => {
    "print_environment_variable.stdout_lines": [
        "Environment variable: PLAY_LEVEL"
    ]
}
PLAY RECAP *********************************************************************
localhost                  : ok=5    changed=2    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0

The -i option specifies the inventory file to be used. As is apparent from the output, the value of the environment variable in Task1 is TASK_LEVEL as expected. Similarly, the value of the environment variable in Task2 is PLAY_LEVEL.

However, there isn’t any environment variable, ENV_VARIABLE, in the system:

$ env | grep ENV_VARIABLE

We used the env command to display the environment variables defined and filtered the output using the grep ENV_VARIABLE command. Therefore, the temporary environment variables defined within the playbook are only valid within the playbook. They don’t have an effect on the system.

3. Setting Permanent Environment Variables

In this section, we’ll see how to set a permanent environment variable for a user. For this purpose, we’ll use the .bashrc file, a Bash shell script that sets variables, functions, and aliases. It runs on every interactive shell launch. It’s in the home directory of a user.

It’s also possible to set an environment variable for all users. For example, we can use /etc/default for this purpose.

3.1. Example Playbook

We’ll use the following playbook, playbook_env_persistent.yml:

$ vi +"set number" playbook_env_persistent.yml
1 ---
2   - name: Ansible define a persistent environment variable
3     hosts: localhost
4
5     tasks:
6     - name: Set environment variable in config file
7       lineinfile:
8         path: /home/baeldung/.bashrc
9         line: 'export ENV_VARIABLE=SYSTEM_LEVEL'

We’ll use the same inventory file.

Let’s discuss the playbook briefly in the following subsection.

3.2. Dissection of the Playbook

We define an environment variable, ENV_VARIABLE, in /home/baeldung/.bashrc by using the built-in lineinfile module (Lines 7-9).

The built-in lineinfile module is useful for changing a single line in a file or adding a single line to a file. The path parameter specifies the file to edit, which is /home/baeldung/.bashrc in our case. The line parameter, on the other hand, specifies the line to insert into the file. It’s the export ENV_VARIABLE=SYSTEM_LEVEL command. By default, it’s appended to the end of the file using only these two parameters.

3.3. Running the Playbook

Now, let’s run the playbook:

$ ansible-playbook playbook_env_persistent.yml -i inventory
PLAY [Ansible define a persistent environment variable] ***********************************************************************
TASK [Gathering Facts] ********************************************************************************************************
ok: [localhost]
TASK [Set environment variable in config file] ********************************************************************************
changed: [localhost]
PLAY RECAP ********************************************************************************************************************
localhost                  : ok=2    changed=1    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0

According to the output, the tasks ran successfully. Let’s check the last line of /home/baeldung/.bashrc using the tail command:

$ tail -1 /home/baeldung/.bashrc
export ENV_VARIABLE=SYSTEM_LEVEL

The last line of the file contains the export command we appended, as expected. Let’s also verify that the environment variable, ENV_VARIABLE, is defined using the echo command in a new terminal:

$ echo $ENV_VARIABLE
SYSTEM_LEVEL

The environment variable is defined and has the expected value.

4. Conclusion

In this article, we discussed how to set both temporary and permanent environment variables using Ansible. Firstly, we learned that we can use the environment keyword to define temporary environment variables within an Ansible playbook, both at play and task levels. We saw that if we use the same name for an environment variable at the play and task levels simultaneously, the one at the task level overrides the other one.

Then, we learned that we must export an environment variable in one of the configuration scripts to have a permanent effect. We used .bashrc in the example. We saw that we can use the built-in lineinfile module for appending the export statement to the end of the file.

Although we didn’t use a block in the example playbooks, in addition to play and task levels, it’s also possible to define environment variables at a block level.

     

Related Stories

• Run Docker Commands With Ansible
• Introduction to GoCD
• Using a Package Manager in GitHub Actions

 

1. Introduction

Continuous integration and continuous development (CICD) is a standard practice in modern software development. Many different tools supporting CICD are available on the market – free, open-source, proprietary, subscription-based, and others. Choosing one can mean the difference between an easy-to-use and long-supported, stable product, in contrast to a tedious integration process.

In this tutorial, we’ll discuss the fundamentals, architecture, and ways of modeling pipelines of the GoCD open-source CICD server.

2. Fundamentals

GoCD is an open-source tool by ThoughtWorks. Its task is to support programming teams in automating processes related to building, testing, and implementing applications.

2.1. Pipelines

As already mentioned, pipelines are the central concept of GoCD. Pipelines are defined as chains of steps that open the way to streamlined development. For instance, each pipeline may include subsequent stages:

• downloading the source code
• compilation process
• running tests
• deployment on the server

Thanks to this, the whole process is more organized and easier to recreate.

2.2. Visualization

Another important aspect is the emphasis on the visualization of the flow of work. The graphical interface clearly shows how each pipeline is connected to the others and which tasks have already been completed, as well as which tasks are still pending. This approach makes the tool work well in complex projects.

2.3. Pipeline as Code

The next important feature of GoCD is the support for the pipeline-as-code approach. The definition of pipelines can be written in JSON files and stored in the repository next to the source code. Thanks to this, CICD processes can have versions like the application, which makes it easier to maintain and transfer between environments.

In effect, GoCD is a tool that works well wherever the process of providing programming is deployed and requires clear modeling and full transparency.

3. Architecture

GoCD is based on a client-server architecture, where GoCD serves as the central component. It’s responsible for storing configurations, managing pipelines, and visualizing the entire delivery process. The users interact with the server using a web panel, where they can define stages, monitor progress, and diagnose possible problems.

3.1. Agents

Further, GoCD agents are responsible for performing specific tasks. These are light processes that run on many machines, both local and in the cloud. The agent collects orders from the server and executes them.

Thus, users can scale the environment simply and perform tests or the process of building an application simultaneously.

3.2. Pipeline Composition

As already mentioned, the central element of the GoCD work is pipelines. They are divided into stages, which, in turn, consist of jobs, the smallest units of work. Thus, such a hierarchy enables users to model even complex processes transparently.

Pipelines rely on so-called materials, i.e., sources from which input data is collected. Most often, this is a Git repository, but it’s also possible to use other version control systems, artifact archives, or even external resources.

Thanks to this, GoCD can respond to changes in many sources at the same time:

Notably, all these elements are closely related. The server manages the flow of data and dependencies, agents perform tasks, and pipelines organize logical stages.

Thanks to this, the GoCD architecture remains fairly simple to understand and at the same time provides a lot of flexibility even when modeling very extensive flows.

4. Pipeline Creation

Pipeline in GoCD is the fundamental element of working with the tool. Using pipelines, we can define the full process of building, testing, and deploying the application.

4.1. Stages

The basic organizational unit is the stage. It’s a group of tasks that run sequentially. The stage contains one or more jobs, and each job consists of specific tasks, such as running compilations, performing tests, or building a container image.

This division enables precise control of the process and clearly separates individual steps. The definition of a pipeline can be created directly in the GoCD web panel. This approach is convenient at the beginning of the work because it enables users to quickly understand the structure and mechanisms of the tool.

4.2. Pipeline as JSON and YAML

In practice, however, the pipeline-as-code approach is increasingly used, in which the configuration is saved in a YAML or JSON file and stored in the repository along with the source code. Thanks to this, it’s easier to version it and also transfer it between different environments.

Let’s see an example fragment of the pipeline definition in YAML format:

pipelines:
  sample-app:
    group: ExampleGroup
    materials:
      git:
        url: https://github.com/example/app.git
    stages:
      - build:
          jobs:
            compile:
              tasks:
                - exec:
                    command: ./gradlew build

In this example, a simple pipeline called sample-app was defined. It uses a Git repository as input material. Through that, it runs a single-stage build, in which the task is to execute the commands compiling the project. Although it’s a minimal configuration, it shows the basic rules for defining processes in GoCD well.

Expanded pipelines can take many stages and use various sources of materials. It’s also possible to create dependencies between pipelines, which enables building entire supply chains, including unit tests, integration tests, and production deployment.

5. Benefits and Limitations

GoCD has many features that distinguish it from other CICD tools.

5.1. Benefits

The most often weighted advantage is the transparent visualization of pipelines, thanks to which even complex delivery processes are often easier to understand and monitor.

Next, support for dependency between pipelines makes the tool useful in large projects, where it is necessary to strictly control subsequent actions. Further, the pipeline as a code approach enables versioning the configuration and maintaining its consistency in all environments.

Scalability is achieved through the GoCD agents and the ability to integrate with numerous external systems, also playing a pivotal role.

5.2. Limitations

On the other hand, GoCD isn’t without limitations. Compared to tools such as Jenkins or GitLab CI, it has a less active community and a poorer plugin system. The result may mean that unusual scenarios can make it difficult to find a ready-made solution or documentation support.

Another barrier is the learning curve, i.e., the entry threshold. Especially for teams coming from simpler CICD systems, the implementation and understanding of GoCD architecture may be time-consuming. Some functions may also seem less intuitive.

In summary, GoCD is a tool with great possibilities and unique advantages, but at the same time requires a larger team to invest in learning and maintenance. It does best in environments where transparency and control over the deployment process are more important than the availability of ready-made plugins or a quick start.

6. Conclusion

In this article, we discussed GoCD. GoCD is a well-developed and transparent tool supporting CICD processes. Thanks to its model based on pipelines, it naturally visualizes the next stages of application development, from building and testing to production deployment. Its flexibility makes it work well in both small projects and large organizations, where the ability to control and visualize the entire process is important.

Although GoCD requires some initial configuration and learning of specific terminology, in return, it offers a lot of transparency, integration with other tools, and stability. It’s worth considering GoCD as an alternative to other CICD systems, especially when the key is a readable and graphical representation of flows and easy management of expanded pipelines.

     

Related Stories

• Configuring Environment Variables in Ansible
• Using a Package Manager in GitHub Actions
• How Does Dotnet Publish Work

 

1. Overview

Docker containers are now an essential part of modern development and operations (DevOps) workflows, providing isolated environments for applications that are portable and lightweight. As these containers move into production, it becomes important to understand their runtime performance costs. While they deliver flexibility and efficiency, they also introduce some overhead compared to running directly on the host system.

In this tutorial, we’ll explore these performance costs, focusing on CPU, memory, storage, and networking. We’ll also look at how Docker manages resources and practical steps to optimize performance in containerized environments.

2. Understanding Docker Containers

To begin with, let’s understand how Docker containers work at a high level. They’re lightweight, portable, and provide process isolation using Linux namespaces and cgroups. These features allow multiple containers to run on the same host without interfering with each other.

Containers share the host OS kernel, which makes them more efficient than traditional virtual machines (VMs) that run their own OS instances. However, this shared environment also means that containers rely on the host system’s resources, which can impact performance under certain conditions.

3. Factors Behind Docker’s Runtime Performance Cost

Although Docker containers are designed to be lightweight, they still introduce performance trade-offs. Several factors drive this runtime cost, such as CPU overhead, memory usage, I/O performance, and network latency.

Let’s break down how each of these comes into play.

3.1. CPU Overhead

Docker containers rely on the host machine’s CPU to execute processes. In theory, containers should have little to no CPU overhead compared to traditional processes, as they run directly on the host kernel. However, in practice, the CPU overhead can occur due to several reasons:

• Context Switching: When multiple containers share CPU resources, the host operating system may have to frequently switch between them, causing context switching overhead.
• Cgroup Resource Limiting: Docker uses cgroups (control groups) to manage and limit the CPU usage of containers. This introduces some overhead, especially when strict limits are set. If a container is constrained to a limited CPU resource, like limiting the container to one CPU core, Docker may introduce some performance degradation due to CPU throttling.
• CPU Pinning: In some high-performance use cases, CPU pinning (binding a container to specific CPU cores) may be used to ensure predictable performance. While this can mitigate the impact of context switching, improper CPU pinning can also lead to inefficient resource usage.

3.2. Memory Usage

Memory management in Docker containers can add overhead, especially when compared to traditional bare-metal processes. Let’s see how this is possible:

• Shared Memory (Kernel vs. User Space): Docker containers share the host kernel’s memory but run in their own user space. This can lead to some inefficiencies in memory management, particularly when the containers are handling large amounts of data.
• Memory Overcommitment: By default, Docker doesn’t strictly enforce memory limits unless we set them. When multiple containers run at the same time and some consume more memory than expected, Docker can end up overcommitting memory. This not only hurts performance but, in extreme cases, can cause containers to be terminated with out-of-memory errors.
• Swap Usage: When we set memory limits and containers go beyond their allocation, Docker falls back to swap space if it’s available. Since swap is much slower than physical RAM, this can quickly drag down performance.

3.3. I/O Performance

Disk I/O performance plays a critical role in the runtime cost of Docker containers. Since containers handle I/O through volumes or bind mounts, some extra overhead comes into play. Let’s break down what this means:

• Storage Drivers: Docker uses different storage drivers to manage how container data is stored on disk, such as OverlayFS, aufs, etc. Each storage driver has its own performance characteristics, and some drivers, like OverlayFS, can offer better performance than others.
• Bind Mounts vs. Volumes: Containers can access files via bind mounts or Docker volumes. Bind mounts can incur additional overhead if the underlying filesystem on the host isn’t optimized for the operations that the container is performing. Volumes are managed by Docker, and their performance is generally better since Docker optimizes them for containerized workloads.
• File System Overhead: Containers using their own file system layers, such as OverlayFS can face performance penalties, especially for read/write-heavy operations. Copy-on-write (COW) operations, which are typical in Docker’s storage model, can result in extra overhead when data is modified within containers.

3.4. Network Latency and Overhead

Networking in Docker containers introduces its own overhead because of how Docker isolates containers within virtual networks:

• Network Bridges: By default, Docker containers are connected to a bridge network, which introduces network routing overhead. The performance cost is usually minimal, but for high-performance applications that require low-latency networking, this can be a factor.
• NAT and IP Tables: Docker containers rely on Network Address Translation (NAT) and IP tables for network isolation. The additional layer of NAT can introduce slight delays, especially when containers communicate with external resources.
• Container Networking Mode: We can choose between host, bridge, or overlay networking. For example, host networking reduces overhead since the container shares the host’s network stack directly. The trade-off, however, is weaker isolation.

4. Best Practices to Reduce Docker’s Runtime Cost

Docker containers bring a lot of flexibility, but they also introduce a level of runtime overhead. The good news is that with the right practices, we can reduce this cost and keep performance close to native. Let’s walk through some proven strategies that make a meaningful difference.

4.1. Resource Limiting and Allocation

One of the first steps to improving performance is making sure containers use resources efficiently. Without limits, a single container can hog CPU or memory and starve others running on the same host.

• Setting CPU Limits Wisely: Applying CPU limits helps prevent noisy neighbors. Techniques like CPU pinning can also improve consistency in latency-sensitive workloads.
• Controlling Memory Usage: Defining memory limits ensures containers don’t take more than their fair share. Swap space should be configured carefully, since it’s slower than RAM and can degrade performance if overused. Monitoring usage over time helps fine-tune these limits as workloads evolve.

4.2. Storage Optimization

Storage I/O can make or break containerized applications, especially those handling large datasets or transaction-heavy operations.

• Choosing the Right Storage Driver: We get better results when the storage driver matches the workload. OverlayFS works well for general use, while specialized workloads (like databases) benefit from testing and benchmarking before committing.
• Using Volumes Over Bind Mounts: Docker volumes are designed for performance and management efficiency, making them preferable to bind mounts, which often add unnecessary overhead.
• Reducing I/O Overhead: We can improve efficiency by minimizing frequent disk reads and writes. Frequently accessed data is best handled in memory or cache layers.

4.3. Network Optimization

Networking is another area where containers can introduce latency. The way containers are connected to networks has a direct impact on performance.

• Leveraging Host Networking: When low latency is critical, host networking reduces overhead by removing virtual bridge layers. The trade-off is weaker isolation, so it works best in selective cases.
• Optimizing Container-to-Container Traffic: Performance improves when services that communicate heavily run on the same Docker network. At the infrastructure level, robust NICs and switches help handle expected traffic loads effectively.

4.4. Monitoring and Performance Tuning

Optimization doesn’t end with initial setup. Continuous monitoring allows us to detect bottlenecks early and tune workloads before they escalate into production issues.

• Monitoring with docker stat: The docker stats command provides real-time visibility into CPU, memory, and I/O usage, helping us spot bottlenecks early.
• Adopting Monitoring Stacks: Tools like Prometheus, Grafana, and cAdvisor give deeper insights with long-term metrics and dashboards, moving us from reactive troubleshooting to proactive optimization.

5. Conclusion

In this article, we explored how CPU, memory, storage, and networking contribute to Docker’s runtime performance cost. Containers provide portability and isolation, but they also introduce some overhead compared to running applications directly on the host.

By setting smart resource limits, optimizing storage and networking, and monitoring performance closely, we can reduce these costs and keep containers running efficiently in production. With the right practices, Docker delivers both flexibility and strong performance.

     

Related Stories

• Using a Package Manager in GitHub Actions
• Run Google Chrome headless in Docker
• Docker Build and Run Platforms and Resolving exec /usr/bin/sh: exec format error

 

1. Introduction

As a breakthrough containerization tool based on Linux features, Docker supports different architectures for both its core and containers. These are sometimes interchangeable, but could depend on certain conditions like executable file formats, layer abstractions, configuration, and others.

In this tutorial, we address platform management when building or running a Docker container, as well as the execution format error that may occur in certain scenarios. First, we build and run a sample Docker image via a Dockerfile to see each step. After that, we understand how and when the target or execution platform can be specified during this process. Next, we go through the consequences of mismatched build and run platforms. Finally, we delve into other reasons for getting an exec format error.

We tested the code in this tutorial on Debian 12 (Bookworm) with GNU Bash 5.2.15. Unless otherwise specified, it should work in most POSIX-compliant environments.

2. Docker Image Build and Run

To create a Docker container, one usually first fetches or builds a Docker image. For that, a Dockerfile is commonly the way to go.

2.1. Sample Dockerfile

Let’s create a simple Dockerfile for demonstration purposes:

$ cat Dockerfile
FROM debian:bookworm-slim
RUN apt-get update
RUN apt-get install -y curl
RUN curl https://gerganov.com/ > response
CMD ["sh"]

This image blueprint is based on the slim version of the current latest Debian distribution. It downloads the most recent curl package and dumps an HTTP response to a file.

Notably, CMD sets sh as the shell to use, i.e., runs that process if no other process is specified.

2.2. Image Build

After creating a Dockerfile, we can initiate a build.

To do so, let’s issue the docker command with the build or buildx build subcommand:

$ docker buildx build -t samplex-app .

In this case, we name (or [-t]ag) the image as samplex-app. The . period indicates that the Dockerfile is in the current directory.

At this point, we should have an image available in the local registry.

2.3. Run Container

Once we have an image, let’s run it via the run subcommand of the docker command:

$ docker run -it samplex-app

Here, we create an [-i]nteractive [-t]erminal session for communication with the container after it initiates. Thus, we usually see a prompt:

#

Of course, this series of steps occurs during a successful workflow. Sometimes, the resulting image can be incompatible with the run environment.

3. Container Architecture Platform Specification

Since Linux exists on many architectures, containers can be built for and run on any of them.

In fact, we can both build and use containers meant for one architecture on others. For that, we need to configure the binfmt_misc kernel feature. It enables the recognition and execution of binaries for different architectures via emulators such as QEMU. This way, we might be able to run a PowerPC or ARM container within an x86 environment.

3.1. The –platform Parameter

Although it works in different settings, the –platform parameter always has the same syntax for its options:

--platform <OS>/<ARCHITECTURE>[/<VARIANT>]

Let’s see a breakdown of each part:

• OS: linux, windows, darwin
• ARCHITECTURE: amd64, arm64, arm, 386, and others
• VARIANT (optional): v3, v5, and similar

For instance, linux/amd64 is perhaps one of the most common desktop or server platforms. Something like linux/arm64 indicates a 64-bit ARM architecture such as that of the Raspberry PI v4 (also known as aarch64). Older ARM platforms are linux/arm/v6 and linux/arm/v7.

3.2. Build-Time Platform

The default target platform for builds is usually the native one of the host.

To ensure we can specify a given platform during the build process, we must use buildx, because the regular build subcommand doesn’t have that option:

$ docker buildx build --platform [...]

In addition, the buildx build subcommand respects the DOCKER_DEFAULT_PLATFORM environment variable for targeted builds when the –platform parameter is omitted:

$ DOCKER_DEFAULT_PLATFORM='linux/arm' docker buildx build -t samplex-app .

Either way, we ensure that the platform for the resulting image is set accordingly. Notably, with the proper setup, we can build for platforms different from that of the host via emulators like QEMU.

3.3. Runtime Platform

When using the run subcommand of docker, we can indicate the platform we want the image to be fetched for, in case there are several:

$ docker run --platform [...]

Similar to buildx build, the run subcommand considers DOCKER_DEFAULT_PLATFORM if –platform isn’t explicitly defined:

$ DOCKER_DEFAULT_PLATFORM='linux/arm' docker run -it samplex-app

While the parameter may look obsolete if there’s only one version or platform, it forces the docker run command to error out:

no matching manifest for linux/arm64 in the manifest list entries

This can be critical to ensure we don’t attempt to run an unsupported image. Of course, again, we can run container images for different platforms as long as the proper configuration (e.g., QEMU) is in place.

3.4. Dockerfile and FROM

Furthermore, we can indicate a target platform via the –platform option in FROM when writing a Dockerfile:

$ cat Dockerfile
FROM --platform= debian:bookworm-slim
[...]

However, this method doesn’t specify the build target platform, only the platform of the fetched image. This is similar to the run subcommand parameter.

Still, we can access the current value of the buildx flag –platform inside the Dockerfile via the TARGETPLATFORM variable.

3.5. Platform Support

What platforms are supported by a given Docker setup is determined by the specific installation. For instance, Docker Desktop already has support for arm64 and arm out of the box.

To get the support list for a given environment, we can use the buildx subcommand with inspect:

$ docker buildx inspect --bootstrap
[...]
Platforms: linux/amd64, linux/arm64, linux/arm/v7, linux/386, linux/ppc64le, linux/s390x, linux/riscv64

Since we see a number of supported platforms on top of the native one, there seems to be a QEMU configuration in place.

In addition, we can check a specific image via the imagetools subcommand:

$ docker buildx imagetools inspect <IMAGE_NAME>

Similar to the –bootstrap inspection, the output contains a Platform: field with a list. The docker image inspect command should produce a similar result, albeit in a different format.

4. Platform Mismatches

Perhaps one of the main reasons for an exec format error when running a container is that the build platform doesn’t match the run platform. Let’s see how and when this happens.

4.1. Basic Case

Of course, we might just insert an unexpected platform as the value of –platform:

$ docker run --platform=linux/ppc64le ppc64le/debian sh
[...]
Status: Downloaded newer image for ppc64le/debian:latest
exec /usr/bin/sh: exec format error

In this case, ppc64le isn’t compatible with the native platform, and there is no emulation set up to handle that architecture.

Yet, how do we know what platform Docker expects? As already mentioned, the native build and run platform is that of the host, but we can check the support via docker buildx inspect –bootstrap to see what other platforms are configured.

4.2. Pull the Wrong Image

When using FROM within a Dockerfile, we might also set the wrong platform, resulting in the same problem:

$ cat Dockerfile
FROM --platform=linux/ppc64le debian:bookworm-slim
CMD ["sh"]
$ docker buildx build -t bad-platform .
[...]
$ docker run bad-platform
WARNING: The requested image's platform (linux/ppc64le) does not match the detected host platform (linux/amd64/v3) and no specific platform was requested
exec /usr/bin/sh: exec format error

So, a Dockerfile can also be the source of the mismatch.

4.3. Hidden Runners

To complicate matters, some cloud and other container runner environments like AWS and GitLab offer options to pick the execution platform.

In these cases, we should manually ensure that the configured and built containers run in the respective environment. While we can create a Dockerfile that adapts, doing so isn’t trivial.

5. Non-Platform Reasons for exec format error

Critically, format errors can be caused by more than architecture mismatches. Let’s briefly take a look at other triggers for the same issue.

5.1. Bad Script Header

When dealing with containers, we often create scripts that should run within them.

In Linux, the first line of scripts can be a special header called a shebang:

$ cat script.sh
#!/bin/bash
[...]

The shebang contains the path to the correct interpreter of the respective script. It has very strict formatting rules:

• starts at the first character of the file
• is allocated the whole first line
• uses absolute paths
• contains no unescaped space characters in the path
• the comparison with !# is only for the first two (2) bytes

Any deviation from the above, such as a missing shebang, space after !#, other characters on the first line, unexpected whitespace, or even an incorrect encoding, can cause an exec format error:

$ cat Dockerfile
FROM debian:bookworm-slim
RUN echo '#! /bin/sh\necho "bad"' > /bad.sh
RUN chmod +x /bad.sh
CMD ["/bad.sh"]
$ docker buildx build -t badbang
[...]
$ docker run badbang
exec /bad.sh: exec format error

Yet, confusingly, depending on the setup, terminal, and configuration, there might not be a problem when running the script directly on the host.

5.2. Bad File Permissions

As usual on Linux and UNIX in general, file permissions dictate whether a file can run in a given context. Specifically, if a script isn’t marked as executable, attempting to run it directly results in an error:

$ cat Dockerfile
FROM debian:bookworm-slim
RUN echo '#! /bin/sh\necho "norun"' > /norun.sh
CMD ["/norun.sh"]
$ docker buildx build -t badbang
[...]
$ docker run badbang
docker: Error response from daemon: failed to create task for container: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: exec: "/norun.sh": permission denied: unknown
Run 'docker run --help' for more information

While this output doesn’t usually include the exec format error string, user reports state that it does so in some environments.

6. Summary

In this article, we talked about Docker build and run platforms, along with errors that they might lead to. Further, we discussed alternative reasons for exec format error issues.

In conclusion, mismatches between the build and run platforms for a Docker container often lead to format errors, but that isn’t always the only reason to get them.

     

Related Stories

• Troubleshooting the no configuration file provided: not found Error in Docker Compose
• What Is the Runtime Performance Cost of a Docker Container?
• Using a Package Manager in GitHub Actions

 

1. Introduction

Continuous Integration workflows often require external tools that aren’t available by default on the execution environment. GitHub Actions, which provide hosted runners for building, testing, and deploying code, also require the same. To access these tools, we frequently use the Ubuntu apt-get package manager.

In this tutorial, we’ll discuss using a package manager in GitHub Actions. First, we’ll start by explaining its importance in workflows, and then examine practical usage within GitHub Actions. Lastly, we’ll discuss common errors that can occur when using apt-get.

2. Package Installation in Workflows

When we run jobs on GitHub-hosted Ubuntu runners, each job begins in a clean virtual environment. To elaborate, the packages required for the task aren’t installed and need to be set up during the workflow. Installing packages ensures that the build environment meets the project requirements.

For example, a Python project may require graphviz to render documentation, a C++ project may need lcov for coverage analysis, and a Java project might require system-level libraries to compile native extensions. Without these, jobs would fail. Therefore, package installation becomes a critical step in CI.

While scripting, the question arises whether to use apt or apt-get. apt-get is generally preferred for scripting. apt is intended as a user-facing command with nicer output, while apt-get provides more stable, script-friendly behavior. For workflows where reproducibility matters, apt-get is the safer choice.

3. Basic Example Setup

To begin with, let’s consider a simple GitHub action where we install the lcov package, which is a tool for generating code coverage reports.

Let’s look at the install.yaml:

$ cat install.yaml
name: CI
on: [push, pull_request]
jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Install lcov
        run: |
          sudo apt-get update
          sudo apt-get install -y lcov
      - name: Run tests with coverage
        run: |
          ./scripts/run-tests.sh
          lcov --capture --directory . --output-file coverage.info

The above code checks out the repository, so the runner has the project code. After that, it installs Icov using sudo apt-get update followed by sudo apt-get install -y lcov. The sudo command is important because apt-get requires administrative privileges. The -y flag ensures the installation proceeds without waiting for confirmation. Lastly, install.yaml runs the tests and collects coverage data with the newly installed Icov.

In this basic example, the workflow runs smoothly. The tool is installed and is available immediately. In addition, later steps can use the installed package.

4. Permission Error in GitHub Actions

At first glance, installing packages in GitHub Actions seems straightforward. However, the first attempt to run apt-get install often fails, and leads to a specific error:

E: Could not open lock file /var/lib/dpkg/lock-frontend - open (13: Permission denied)
E: Unable to acquire the dpkg frontend lock (/var/lib/dpkg/lock-frontend), are you root?

The issue arises because GitHub Actions jobs run as a non-root user by default. System package management with apt-get requires elevated privileges to modify /var/lib/dpkg. Since we aren’t root, the process cannot obtain the necessary lock.

To fix the above error, we can use the sudo command. Fortunately, GitHub’s Ubuntu runners enable passwordless sudo. To elaborate, we can run administrative commands without being prompted for a password. Therefore, the fix is often as simple as adding sudo to the install step.

5. Performance Considerations

While the above method works, it can slow down workflows. Each run installs packages from scratch, which adds overhead, especially if the list of dependencies is long. To improve performance, we can apply several strategies discussed in this section.

5.1. Grouping Packages

It’s faster to install multiple packages in one command rather than invoking apt-get multiple times.

For example, we can install Icov, graphviz, and doxygen together:

$ cat install.yaml
name: CI
on: [push, pull_request]
jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Install lcov
        run: |
          sudo apt-get update
          sudo apt-get install -y lcov graphviz doxygen
      - name: Run tests with coverage
        run: |
          ./scripts/run-tests.sh
          lcov --capture --directory . --output-file coverage.info

Thus, we avoid repeating dependency resolution and reduce overhead.

5.2. Caching Packages

Another option is to use caching actions. The cache-apt-pkgs-action makes package installation faster by caching downloaded .deb files between runs.

Thus, we can include the caching option in the uses field within the .yaml file:

$ cat install.yaml
name: CI
on: [push, pull_request]
jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with:
          packages: dia doxygen doxygen-doc doxygen-gui doxygen-latex graphviz mscgen
          version: 1.0
      - name: Install lcov
        run: |
          sudo apt-get update
          sudo apt-get install -y lcov
...

Here, the packages field lists the desired packages, and the version field serves as a manual cache key. Incrementing the version refreshes the cache when the package list changes. This reduces build times significantly for projects with heavy dependencies.

5.3. Custom Containers

​For projects requiring extensive system dependencies, we can use prebuilt containers, which are more efficient. By defining a container with all tools preinstalled, we avoid repeated installs in workflows.

The workflow then specifies those containers under the build section in the .yaml file:

$ cat install.yaml
name: CI
on: [push, pull_request]
jobs:
  build:
    runs-on: ubuntu-latest
    container: ghcr.io/my-org/custom-image:1.0
    steps:
      - uses: actions/checkout@v4
...

This shifts installation overhead from every run to a one-time container build, improving both speed and reproducibility. The sacrifice is that the configuration is now in two separate places, and rebuilding container images can be tedious if they change often.

6. Best Practices

Across the different approaches to install packages in GitHub Actions, there are several best practices:

• It’s always better to install only what’s necessary, since every extra package adds to build time and increases the risk of conflicts.
• We should keep the installations non-interactive, and that’s why flags like -y and the DEBIAN_FRONTEND=noninteractive setting are so valuable, since they prevent the workflow from hanging at unexpected prompts.
• When stability is critical, it’s wise to pin package versions rather than relying on the latest available option, because reproducibility matters more than novelty in CI pipelines.
• It’s important to stay mindful of the runner image updates, since ubuntu-latest shifts over time, for example, from 20.04 to 22.04, and pinning a specific version can offer more predictable results.

At the same time, we shouldn’t overlook the fact that dedicated setup actions for languages such as Python, Node, or Java, actions/setup-python, actions/setup-node, are often faster and more reliable than using system-level installs.

7. Conclusion

In this article, we covered how to use a package manager in GitHub Actions and pitfalls around such cases. First, we discussed why we use a package manager in GitHub Actions. After that, we covered an example of doing so. Lastly, we elaborated on the common errors and performance issues that arise when using apt-get in GitHub Actions and ways to fix or avoid them.

     

Related Stories

• Get Files Modified in a PR in GitHub Actions
• Use Private Docker Image in GitHub Actions
• Using GitHub Actions to Create .env File in the Workflow

 

1. Overview

When working with Docker Desktop on Windows using the WSL2 backend, Windows Subsystem for Linux, we may sometimes encounter a frustrating message, “An Unexpected Error Was Encountered While Executing a WSL Command”.

This error can occur at almost any point. For example, it might occur when starting Docker Desktop, running docker build, pulling an image, or simply launching a container. Specifically, it stops our workflow abruptly and often leaves us wondering where to start.

Fortunately, this error is often solvable. By understanding how Docker and WSL interact, and by following a methodical troubleshooting process, we can diagnose and fix the issue. In this tutorial, we will tackle this by explaining both the theory behind each step and the practical commands we can use to resolve it.

2. Understanding the Error

There are multiple factors that lead to this issue. Before diving into fixes, let’s explore why WSL on our desktop is important:

• WSL enables us to run Linux distributions natively on Windows without heavy virtual machines
• WSL2 introduced a real Linux kernel running inside a lightweight virtual machine, improving performance and compatibility
• Docker Desktop can use WSL2 as its backend instead of Hyper-V, which makes Linux containers run more efficiently and reduces overhead

However, Docker relies heavily on a stable WSL environment. In essence, if the WSL backend encounters issues, whether from corruption, outdated versions, or resource limits, we get this error.

Next, there are a lot of common situations that trigger such an error:

• Trying to run Docker when the WSL service is stopped
• Running outdated WSL or Docker Desktop versions that no longer work well together
• The docker-desktop or docker-desktop-data WSL distributions are being corrupted
• Insufficient permissions for Docker to execute commands inside WSL
• WSL is running out of allocated memory or CPU

3. Step-by-Step Troubleshooting

In this section, we’ll follow a logical sequence starting from simple checks and moving toward more advanced fixes.

3.1. Check if WSL Is Working

First, we’d better ensure that WSL itself is functional. For instance, if WSL is broken, Docker can’t work.

The wsl PowerShell command validates WSL functionality:

$ wsl --list --verbose
  NAME                   STATE           VERSION
* Ubuntu                 Running         2
  docker-desktop         Running         2
  docker-desktop-data    Running         2

Let’s understand what this command does and what benefit it brings:

• lists all installed WSL distributions
• shows their current state (Running/Stopped)
• shows the WSL version (1 or 2)

Looking at the output of this code snippet, we can easily spot that services like docker-desktop and docker-desktop-data are Running. If any of these services were showing a Stopped state, Docker can’t function properly. In particular, if any of the above services was in a Stopped state, we would better restart them:

$ wsl --shutdown

Then we reopen Docker Desktop, which will automatically restart the required WSL services.

3.2. Restart WSL Services

If WSL is indeed running, and we still get the error, we can try restarting WSL services:

$ wsl --shutdown

Essentially, this stops all WSL instances and frees up resources. In addition, when we start Docker Desktop again, it will restart the required WSL distributions automatically.

Alternatively, restart the LxssManager service, which manages WSL:

• Press Win + R, then type services.msc, and press Enter
• Find LxssManager
• Right-click, then choose Restart

This often clears temporary issues without touching configurations.

3.3. Update WSL and Docker Desktop

Consequently, if the above did not help, the suggestion would be to check for updates, as outdated software is one of the most common causes of this error.

To avoid this, we always make sure our WSL version is up to date:

$ wsl --update

This installs the latest Linux kernel and backend improvements. In addition, we update Docker Desktop by following a set of simple steps:

• Open Docker Desktop
• Go to Settings → General → Check for Updates
• Install the latest version

This matters because Docker and WSL updates often contain bug fixes and performance improvements. Particularly, if we’re running mismatched versions, commands can fail unexpectedly.

3.4. Check Docker-Windows Integration

Essentially, Docker Desktop needs to know which WSL distributions it should work with. We can check whether it is configured correctly:

• Open Docker Desktop
• Navigate to Settings → Resources → WSL Integration
• Make sure the Linux distribution we use (for example, Ubuntu) is enabled

If this is disabled, Docker won’t execute commands inside that distribution, leading to errors.

3.5. Check Docker Logs

If the above steps don’t help, checking logs can give us clues:

$ wsl -d docker-desktop cat /var/log/docker.log

This opens the Docker daemon log inside the WSL backend. Then, we look for any repeated error messages or permission issues.

3.6. Adjust WSL Resource Limits

Sometimes, commands fail because WSL runs out of memory or CPU. Alternatively, we can configure resource limits by creating a .wslconfig file in our user profile directory:

[wsl2]
memory=4GB
processors=2

Afterwards, we save the file as user \.wslconfig and we shut down WSL:

$ wsl --shutdown

Finally, we restart Docker Desktop to ensure Docker has enough resources to complete tasks.

3.7. Inspect Docker/WSL Backend

If we suspect corruption, we can reset the Docker WSL distributions. Note that this will delete all our data, so if we want to do this, we first perform a backup:

$ docker save -o backup.tar image1

This command saves a specified Docker image, including all its layers, metadata, and history, into a single tar archive file named backup.tar. Hence, we can use this archive for tasks like transferring images to systems without direct Docker registry access or creating backups of our Docker images. Finally, we can later load these images onto another Docker host using the docker load command.

Next, we can recreate the container:

$ wsl --unregister docker-desktop
$ wsl --unregister docker-desktop-data

When we restart Docker Desktop, it recreates these distributions.

3.8. Reinstall WSL if Necessary

As a last resort, if WSL itself is broken beyond repair, we can reinstall it:

$ wsl --unregister <distro-name>
$ wsl --install

We then reinstall Docker Desktop and reconfigure integration. This is extreme, but it guarantees a fresh environment.

4. Preventive Measures

There are a lot of preventive measures we could take to avoid running into this issue:

• Keep everything updated — run wsl –update monthly, and update Docker Desktop regularly
• Allocate enough resources — ensure .wslconfig values match the workloads we run
• Shut down cleanly — avoid force-closing Docker Desktop or turning off the PC abruptly
• Backup Docker images — periodically use docker save to preserve important images

We could even run an automatic script to do the health check for us:

Write-Host "Checking WSL status..."
wsl --list --verbose
Write-Host "`nRestarting WSL..."
wsl --shutdown
Write-Host "`nUpdating WSL..."
wsl --update
Write-Host "`nChecking Docker Desktop..."
& "$Env:ProgramFiles\Docker\Docker\Docker Desktop.exe"

Next, let’s break down what it does:

• Lists WSL distros and their states
• Restarts WSL
• Updates WSL to the latest version
• Launches Docker Desktop

Running this before starting work can catch issues early.

5. Conclusion

In this article, we identified why Docker Desktop may show the “An unexpected error was encountered while executing a WSL command” message and outlined practical ways to fix it.

With simple checks, updates, and proper configuration, we can quickly restore Docker’s WSL integration and keep our environment stable.

     

Related Stories

• What Is the Runtime Performance Cost of a Docker Container?
• Docker Build and Run Platforms and Resolving exec /usr/bin/sh: exec format error
• Using a Package Manager in GitHub Actions

 

1. Overview

In .NET development, the dotnet publish command is essential for preparing applications for distribution. It builds the application, processes its dependencies, and stores the finished distributable assets in a specific location. This tool supports a variety of deployment targets, such as folders, containers, and cloud environments.

In this tutorial, we’ll look at what dotnet publish does, how it works behind the scenes, and how to use it efficiently in various deployment circumstances.

2. What Happens During Dotnet Publish

When we run dotnet publish, the CLI executes several processes that change our source code into a production-ready package:

$ dotnet publish

In this example, when we run the command, the dotnet CLI starts converting the project to Intermediate Language (IL) assemblies. It then resolves all NuGet and project dependencies, ensuring that they work with the target framework. Following that, it moves the compiled binaries, configuration files, and static resources such as images or CSS to the publish folder.

In addition, when a runtime identifier is supplied, it contains the corresponding .NET runtime. Finally, it optimizes the files if trimming or single-file publication is selected.

3. Common Options and Flags

The dotnet publish command provides numerous arguments and MSBuild attributes for customising the output.

These options allow us to specify the build configuration, target runtime, deployment model, and even how the files are packed. We can adapt a single codebase to many contexts by mixing different flags:

$ dotnet publish -c Release -o ./deploy

Determining projects to restore...
Restored /home/user/HelloApp/HelloApp.csproj (in 110 ms).
HelloApp -> /home/user/HelloApp/bin/Release/net8.0/HelloApp.dll
HelloApp -> /home/user/HelloApp/deploy/

In this example, the -c Release flag builds with optimisations, and the -o ./deploy flag directs the published files to a deploy folder rather than the typical publish location:

These options allow us to fine-tune the publishing process for speed, portability, and size.

4. Framework-Dependent vs. Self-Contained

A framework-dependent deployment expects that the destination machine already has the .NET runtime installed. This keeps the published output small:

$ dotnet publish -c Release --self-contained false

HelloApp -> /home/user/HelloApp/bin/Release/net8.0/HelloApp.dll
HelloApp -> /home/user/HelloApp/bin/Release/net8.0/publish/

In this example, the publish folder contains mostly the application’s .dll file and any required libraries, but not the .NET runtime.

However, a self-contained deployment bundles the .NET runtime with the program, allowing it to execute on systems that don’t have .NET installed:

$ dotnet publish -c Release -r linux-x64 --self-contained true

HelloApp -> /home/user/HelloApp/bin/Release/net8.0/linux-x64/HelloApp.dll
HelloApp -> /home/user/HelloApp/bin/Release/net8.0/linux-x64/publish/

In this example, inspecting the publish folder displays an executable file as well as all runtime files required by that operating system and architecture.

5. Single File Publishing

Single-file publishing enables us to distribute an application, its dependencies, and, optionally, the .NET runtime as a single executable. This approach simplifies deployment by eliminating the need to handle numerous assemblies and runtime files:

$ dotnet publish -c Release -r win-x64 --self-contained true /p:PublishSingleFile=true

HelloApp -> C:\Projects\HelloApp\bin\Release\net8.0\win-x64\HelloApp.dll
HelloApp -> C:\Projects\HelloApp\bin\Release\net8.0\win-x64\publish\

In this example, we publish a single-file executable for Windows; once published, we inspect the publish folder and find only one .exe file along with a few runtime-dependent files, depending on the parameters.

We can then proceed to launch the executable:

$ C:\Projects\HelloApp\bin\Release\net8.0\win-x64\publish\HelloApp.exe

Hello World from Single File Deployment

This indicates that the application runs correctly from a single file. Single-file publishing considerably simplifies deployment by lowering the number of files to distribute. Instead of handling multiple assemblies and runtime libraries, we can provide a single executable that can be launched immediately.

Furthermore, this method is useful in situations when simplicity and portability are important, such as distributing desktop utilities or packaging applications into containers.

6. Conclusion

In conclusion, the dotnet publish command is a crucial step in transitioning an application from development to production. It compiles the source code, handles dependencies, and generates a deployment-ready package specific to the specified environment.

In this article, we discuss what happens when we publish and how using options like –self-contained, runtime identifiers, or single-file publishing provides us fine-grained control over how programs are generated and delivered.

Finally, framework-dependent deployments reduce the footprint but rely on the presence of the .NET runtime, whereas self-contained deployments ensure portability. Single-file publishing simplifies distribution by combining everything into a single executable.

     

Related Stories

• Configuring Environment Variables in Ansible
• Introduction to GoCD
• Using a Package Manager in GitHub Actions

 

1. Introduction

GitHub Actions provides a flexible and powerful platform for automating workflows within a GitHub repository. One common use case in continuous integration (CI) workflows is the ability to detect which files have changed in a pull request (PR) or push event.

In this article, we’ll explore how to detect changed files in a PR and on push using GitHub Actions. We’ll also demonstrate how to combine both scenarios into a single workflow, ensuring that file detection logic works reliably for both event types.

2. Detecting Changed Files in a Pull Request

When a PR is opened or updated, GitHub triggers a pull_request event. This event provides access to two important pieces of information: The base branch and the head branch. By comparing these two branches, we can detect which files have changed. This is especially useful when we want to run a specific job, like tests or linters, only for the parts of the codebase that were modified in the PR.

Let’s begin with a pull_request workflow that checks out the code and uses git diff to detect modified files:

name: Detect PR Changes
on:
  pull_request:
    types: [opened, synchronize, reopened]
jobs:
  detect-changes:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout the code
        uses: actions/checkout@v4
        with:
          fetch-depth: 0 # Ensures we have the full history for diff
      - name: Get changed files
        id: changes
        run: |
          git fetch origin ${{ github.base_ref }}
          CHANGED_FILES=$(git diff --name-only origin/${{ github.base_ref }}...HEAD)
          echo "changed=$CHANGED_FILES" >> $GITHUB_OUTPUT
      - name: Print changed files
        run: echo "Changed files: ${{ steps.changes.outputs.changed }}"

The workflow is configured to trigger on pull_request events, specifically for actions such as opened, synchronize, and reopened. Within the job, in the first step, we use the actions/checkout@v4 action with fetch-depth: 0 to ensure that we retrieve the full commit history before comparing the difference.

Next, we fetch the base branch of the PR using github.base_ref in the second step. Then, we compare it against the current HEAD using git diff –name-only. This triple-dot syntax allows us to see the complete set of files that have changed between the branches. Finally, we store the result in an output variable so that subsequent steps in the workflow can access the list of changed files.

Lastly, we use echo to print the result and verify the output during the workflow execution.

3. Detecting Changed Files on Push

While PRs are common for collaborative development, direct pushes to branches are still relevant in many workflows. As another option, we can detect changed files when pushing new code to our branches. When using the push event, GitHub Actions provides access to the commit SHA before and after the push.

This approach is straightforward and effective for detecting changes in workflows triggered by direct branch updates. It’s especially useful for scenarios like production deployments, validations, or testing workflows that need to respond to updates.

We can use these SHAs to detect changes:

name: Detect Push Changes
on:
  push:
    branches:
      - main
      - develop
jobs:
  detect-push-changes:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout the code
        uses: actions/checkout@v4
        with:
          fetch-depth: 0
      - name: Get changed files on push
        id: push_changes
        run: |
          git fetch origin ${{ github.ref_name }}
          CHANGED_FILES=$(git diff --name-only ${{ github.event.before }} ${{ github.event.after }})
          echo "changed=$CHANGED_FILES" >> $GITHUB_OUTPUT
      - name: Print changed files
        run: echo "Changed files: ${{ steps.push_changes.outputs.changed }}"

As discussed, GitHub’s push event provides two important commit SHAs: before and after, representing the state of the branch before and after the push. By using git diff with these two SHAs, we can determine exactly which files were modified.

Similar to the pull request approach, we retrieve the full commit history using the actions/checkout@v4 action with fetch-depth: 0.

In the second step of the workflow, we compare the before and after SHAs using git diff to get a list of changed files.

Finally, we print the result to verify our changes during the workflow execution.

4. Combining Pull Requests and Pushes on a Single Workflow

In practice, we may want a unified workflow that handles both PRs and pushes. Rather than duplicating logic across separate workflows, we can combine support for both event types into a single, unified GitHub workflow.

We can achieve this by adding conditional logic based on the event_name value:

name: Detect File Changes
on:
  push:
    branches:
      - main
      - develop
  pull_request:
    types: [opened, synchronize, reopened]
jobs:
  detect-changed-files:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout code
        uses: actions/checkout@v4
        with:
          fetch-depth: 0
      - name: Determine event type
        id: event_type
        run: echo "Event is ${{ github.event_name }}"
      - name: Get changed files (PR)
        if: github.event_name == 'pull_request'
        id: pr_changes
        run: |
          git fetch origin ${{ github.base_ref }}
          CHANGED_FILES=$(git diff --name-only origin/${{ github.base_ref }}...HEAD)
          echo "changed=$CHANGED_FILES" >> $GITHUB_OUTPUT
      - name: Get changed files (Push)
        if: github.event_name == 'push'
        id: push_changes
        run: |
          git fetch origin ${{ github.ref_name }}
          CHANGED_FILES=$(git diff --name-only ${{ github.event.before }} ${{ github.event.after }})
          echo "changed=$CHANGED_FILES" >> $GITHUB_OUTPUT
      - name: Print changed files
        run: |
          if [ "${{ github.event_name }}" = "pull_request" ]; then
            echo "Changed files: ${{ steps.pr_changes.outputs.changed }}"
          else
            echo "Changed files: ${{ steps.push_changes.outputs.changed }}"
          fi

This workflow listens to both push and pull_request events, allowing it to handle file change detection in a unified way. It begins by checking out the code with full history using actions/checkoutv4, ensuring that all commits needed for diff comparisons are available.

Depending on the event type, determined by github.event_name, the workflow executes the appropriate set of steps: for pull requests, it compares the base branch to the head using git diff, while for pushes, it compares the before and after commit SHAs. Then, we store each result in an output variable. A final step prints the list of changed files, using conditional logic to select the correct output.

This approach reduces maintenance overhead while also providing consistent file change detection, no matter how updates are made to the repository.

5. Conclusion

In this article, we explored how detecting file changes in GitHub Actions is a powerful technique that enables more intelligent and efficient CI/CD workflows.

Detecting file changes in GitHub Actions is a powerful technique that enables more intelligent and efficient CI/CD workflows. Whether we’re working with PRs, direct pushes, or both, we can tailor our workflow to respond only to relevant changes, either for specific events or a combination of them.

     

Related Stories

• Create Pull Request With GitHub Actions
• Using a Package Manager in GitHub Actions
• Use Private Docker Image in GitHub Actions