How do I run many EXE files in parallel using PowerShell?

add tag

3 years ago khajlk

I have several directories (jobs) (> 50) in C:/tmp on my Windows 10 (x64, 8 CPUs) PC. Each directory contains an EXE file (my_file.exe) and other files as inputs to EXE. I can run this EXE via Windows CLI like this:

C:/tmp/my_file.exe my_inputs.txt

xxxxxxxxxx
 
C:/tmp/my_file.exe my_inputs.txt

The EXE file runs and produces output in each directory (my_output.txt).

To run 7 jobs, I do the following:

C:/tmp/my_file.exe my_inputs1.txt
C:/tmp/my_file.exe my_inputs2.txt
C:/tmp/my_file.exe my_inputs3.txt
...

xxxxxxxxxx
 
C:/tmp/my_file.exe my_inputs1.txtC:/tmp/my_file.exe my_inputs2.txtC:/tmp/my_file.exe my_inputs3.txt...

Instead of manually running 7 jobs again and again (7 jobs = 7 CPUs keeping some PC resources for the other processes), I would like to automate this procedure.

In PowerShell, is there a way such that, I could run 7 EXEs (7 CPUs) (starting from directory 1) at the same time, keeping an “eye” on whether a job is finished (probably by monitoring output.txt in each directory), kill that process, and start the new EXE in the queue, until all EXEs in all the directories have been executed and produced the outputs?

PowerShell 6/7 (Core)

PowerShell Core introduces the ForEach -Parallel construct (along with its own implementation woes).

1 .. 7 | ForEach-Object -Parallel {
    $number = $_
    0 .. 10 | % {
        echo (Get-Date).ToString() >> "C:\Temp\${number}.v7.txt"
        Start-Sleep 1
    }
}

xxxxxxxxxx
 
1 .. 7 | ForEach-Object -Parallel {    $number = $_    0 .. 10 | % {        echo (Get-Date).ToString() >> "C:\Temp\${number}.v7.txt"        Start-Sleep 1    }}

Note this example requires about 20 seconds of wall clock time. More than the 10s we’d expect (and indeed get in the Desktop edition implementation); but still far faster than the 70 seconds of CPU time we got.

P.S. ValueFromPipline processing (in a function) requires special handling to leverage parallelization. Leverage .GetSteppablePipline() as in this gist example

Answer #2

3 years ago PeterVandivier

PowerShell 5 (Desktop)

#Requires -PSEdition Desktop

workflow foo {
    foreach -Parallel ($number in (1 .. 7)) {
        1 .. 10 | % {
            echo (Get-Date).ToString() >> "C:\Temp\${number}.txt" 
            Start-Sleep 1 
        }
    }
}

foo

xxxxxxxxxx
 
#Requires -PSEdition Desktop​workflow foo {    foreach -Parallel ($number in (1 .. 7)) {        1 .. 10 | % {            echo (Get-Date).ToString() >> "C:\Temp\${number}.txt"             Start-Sleep 1         }    }}​foo

Much like function foo {} defines an executable script, workflow foo{} defines a parallel workflow. Invoking foo must be called separately after defining what foo is.

We expect 10 seconds of CPU time per file for the script above. Measure-Command {foo} shows us that only 10 seconds of wall clock time have passed, but 7 files exist in C:\Temp named 1.txt through 7.txt, each with 10 seconds worth of output.

Fair warning: Desktop edition workflows are a bit of a pain in the butt to work with IMHO. Note in the preceding example, echo isn’t a true binary call but an alias to Write-Output, debugging specific workflow implementations is a whole other beast.

Scoping issues with the InlineScript {} block type may be an issue for you with this usage depending on usage.

2 Answers

PowerShell 6/7 (Core)

PowerShell 5 (Desktop)