 Learn about Linux in practice  1. Introduction
When we write Bash scripts, we often need to perform repetitive tasks, such as processing files, making requests, or handling data in a loop. The tasks may be time-consuming, and executing them one by one slows down the process. We can execute each iteration as a background process. This way, we don’t wait for one task to finish before starting the next. But while this looks simple at first, backgrounding inside loops comes with its own set of challenges.
Background jobs can overwhelm the system or create processes that are difficult to control if left unchecked. We should therefore be careful of job limits, handle cleanup correctly, and ensure the script waits for all jobs to complete.
In this tutorial, we’ll discuss several patterns for running background jobs in loops. We’ll also have a quick mention of alternatives like xargs and parallel.
2. Basics of Backgrounding in Bash
Backgrounding commands allows us to move on without waiting for each command to complete. We background a command in Bash using the & operator. It’s handy when we need to execute lots of tasks simultaneously, particularly in loops. There is nothing extra to install in the system. As long as we have Bash installed (typically by default on most Linux distros), we are set.
Let’s take a brief look at an example Bash script to understand how backgrounding works:
#!/bin/bash
sleep 5 & echo "Waiting for the background job..."
wait
echo "Done"
Here we run the sleep 5 command in the background using &. Then we immediately print a message while the sleep command runs in the background. The wait command makes the script wait for the background process to finish. Finally, when everything is finished, we print “Done.” This is the foundation of more advanced job control, which we’ll look at next.
3. Naive Backgrounding in a Loop
One of the simplest ways to run multiple tasks at once in Bash is to send a command to the background using & within a loop. This makes it easy to start many tasks without waiting for each one to finish. It’s easy to write and is acceptable for small scripts. Let’s see a simple example:
#!/bin/bash
for i in {1..5}; do
echo "Processing item $i" &
done
wait
echo "All background jobs finished."
In here, we’ve got a for loop echoing messages for items 1 through 5. Each of the echo statements runs in the background using &, so the loop doesn’t wait. The wait command instructs the script to wait until all background jobs finish before echoing the last message.
This is an efficient way to parallelize tasks, but it has a risk. If the command in the loop is resource-intensive, launching too many background jobs at once can overwhelm the system. We’ll look at a better means of handling this in subsequent sections.
4. Controlled Parallelism With wait
Background jobs in a loop are fast, but launching too many can overwhelm the system. To alleviate that, we control how many jobs run in parallel. Bash has the wait and wait -n features to control background jobs nicely without external utility assistance.
We don’t have to install anything for this, as wait is included in Bash. To use wait -n, though, we need Bash 5 or newer. To install or upgrade Bash on most Linux distributions:
$sudo apt-get install bash
Let’s see a basic example where we control the number of background jobs in a loop:
#!/bin/bash
max_jobs=3
for i in {1..10}; do
echo "Starting job $i" &
while (( $(jobs -r | wc -l) >= max_jobs )); do
wait -n
done
done
wait
echo "All jobs completed."
Here we have max_jobs = 3, so up to three background jobs are run concurrently. Inside the for loop, we start a job in the background using &. Then we find out how many jobs are running by using jobs -r piped to wc -l. If the active jobs reach the limit, we use wait -n, which waits for any one of them to finish before running another. This technique gives us easy but excellent control over concurrency among jobs, keeping it silky and safe.
6. Using GNU Parallel or xargs -P as Alternatives
Instead of using background jobs and manually creating a loop to control them, we can use xargs -P or GNU Parallel. Both simplify parallel execution and let us specify how many jobs run simultaneously. They create job control internally, which simplifies our scripts. At first, let’s start by installing these tools:
$ sudo apt-get install parallel findutils
Now, let’s start with a basic example using xargs:
$ seq 1 10 | xargs -n 1 -P 3 -I{} echo "Processing item {}"
In this command, seq 1 10 outputs numbers 1 to 10. We pipe it to xargs, which runs the echo command for each number. The -n 1 option tells xargs to use a single item per command, -P 3 sets the number of parallel jobs to 3, and -I{} specifies a placeholder. So, three echo commands run in parallel. As soon as any job finishes, another starts. Now let’s try the same thing with GNU Parallel:
$ seq 1 10 | parallel -j 3 echo "Processing item {}"
In this, parallel reads from seq 1 10 and runs the echo command on them. The -j 3 option does three jobs at one time. {} is a wildcard, like xargs.
Parallel has additional features than xargs—like better handling of output and multi-core awareness—so it’s better for more complex jobs. Both facilities enable us to keep things clean and efficient without the need to write special job control code. In handling large amounts of input or big commands, they can avoid time wastage and system crashes.
7. Cleanup and Error Handling
When running background jobs in Bash, we have to be careful about cleaning up the processes we start. If our script is ended prematurely or interrupted, we may leave behind orphaned jobs. To avoid this, we utilize the trap command. It lets us trap signals like SIGINT or EXIT and run custom commands if something goes wrong or the script ends.
Here’s a basic example showing how we can use trap to clean up running background jobs:
cleanup() {
echo "Cleaning up..."
jobs -p | xargs -r kill
}
trap cleanup SIGINT SIGTERM EXIT
for i in {1..5}; do
sleep 60 &
done
wait
In this script, we define a cleanup function that kills all running background jobs using jobs -p and xargs kill. Then we use trap to connect this function to signals like SIGINT (Ctrl+C), SIGTERM, and EXIT. This way, if the script is stopped manually or crashes, it’ll still clean up the background jobs. Finally, we run five background sleep commands and use wait to let them finish. This pattern keeps our scripts safe and avoids leaving stray processes behind.
8. Conclusion
In this article, we’ve shown that running background jobs in a loop under Bash isn’t just about adding &. Without proper control, it can overwhelm the system. Using wait helps manage this, and wait -n offers better control by reacting as jobs finish.
We also covered tools like xargs -P and parallel for cleaner parallel execution. For real-world scripts, it’s essential to handle cleanup, set job limits, and manage failures. Bash’s built-in tools like trap and jobs make this possible without extra dependencies. These techniques make batch processing faster, safer, and more scalable. The post Background Jobs in a Loop in Bash first appeared on Baeldung on Linux. 
Content mobilized by FeedBlitz RSS Services, the premium FeedBurner alternative. | 
