Xjyuk

I am using a Bash script to execute a Python script multiple times. In order to speed up the execution, I would like to execute these (independent) processes in parallel. The code below does so:

#!/usr/bin/env bash

script="path_to_python_script"
N=16 # number of processors
mkdir -p data/

for i in `seq 1 1 100`; do
 for j in 1..100; do
 ((q=q%N)); ((q++==0)) && wait

 if [ -e data/file_$i-$j.txt ]
 then
 echo "data/file_$i-$j.txt exists"
 else
 ($script -args_1 $i > data/file_$i-$j.txt ;
 $script -args_1 $i -args_2 value -args_3 value >> data/file_$i-$j.txt) &
 fi
 done
done

However, I am wondering if this code follow common best practices of parallelization of for loops in Bash? Are there ways to improve the efficiency of this code?

Some suggestions:

• The trailing slash in the mkdir command is redundant.


• 
  $(…) is preferred over backticks for command substitution.


• Why use seq in one command? They both do the same loop, so you might as well use 1..100 in both places.


• Semicolons are unnecessary in the vast majority of cases. Simply use a newline to achieve the same separation between commands.


• Use More Quotes™


• 
  set -o errexit -o noclobber -o nounset at the start of the script will be helpful. It'll exit the script instead of overwriting any files, for example, so you can get rid of the inner if statement if it's OK that the script stops when the file exists.


• 
  [[ is preferred over [.


• The whole exercise is probably easier to achieve with some standard pattern like GNU parallel. Currently the script starts N commands, then waits for all of them to finish before starting any more. Unless the processes take very similar time this is going to waste a lot of time waiting.


• 
  N (or for example processors for readability) should be determined dynamically, using for example nproc --all, rather than hardcoded.


• If you're worried about speed you should probably not create a subshell for your two script commands. and will group commands without creating a subshell.


• For the same reason you probably want to do a single redirection like "$script" … && "$script" …; > "data/file_$i-$j.txt"
  


• 
  

  Since you're "only" counting to 10,000 you don't need to reset q every time. You can for example set process_count=0 outside the outer loop and check the modulo in a readable way such as:

  
  
  

  if [[ "$process_count" % "$processors" -eq 0 ]]
  then
   wait
  fi
  

  
  


• The inner code (from the line starting with ((q=q%N))) should be indented one more time.