How do I run many EXE files in parallel using PowerShell?

add tag

khajlk

I have several directories (jobs) (> 50) in ` C:/tmp` on my Windows 10 (x64, 8 CPUs) PC. Each directory contains an EXE file (my_file.exe) and other files as inputs to EXE. I can run this EXE via Windows CLI like this:

```
C:/tmp/my_file.exe my_inputs.txt
```

The EXE file runs and produces output in each directory (`my_output.txt`).

To run 7 jobs, I do the following:

```
C:/tmp/my_file.exe my_inputs1.txt
C:/tmp/my_file.exe my_inputs2.txt
C:/tmp/my_file.exe my_inputs3.txt
...
```

Instead of manually running 7 jobs again and again (7 jobs = 7 CPUs keeping some PC resources for the other processes), I would like to automate this procedure.

In PowerShell, is there a way such that, I could run 7 EXEs (7 CPUs) (starting from directory 1) at the same time, keeping an "eye" on whether a job is finished (probably by monitoring `output.txt` in each directory), kill that process, and start the new EXE in the queue, until all EXEs in all the directories have been executed and produced the outputs?

Top Answer

PeterVandivier

# PowerShell 6/7 (Core)

PowerShell Core introduces the `ForEach -Parallel` construct (along with its own implementation woes). 

```powershell
1 .. 7 | ForEach-Object -Parallel {
    $number = $_
    0 .. 10 | % {
        echo (Get-Date).ToString() >> "C:\Temp\${number}.v7.txt"
        Start-Sleep 1
    }
}
```

Note this example requires about 20 seconds of wall clock time. More than the 10s we'd expect (and indeed get in the Desktop edition implementation); but still far faster than the 70 seconds of CPU time we got. 

![image.png](/image?hash=5554d6a88c75c09906febf40b6bbfcc0201a37ae94e9ac13120695b339c6a889)
![image.png](/image?hash=6d5af1c4cdce62390afa94ca45aee7ec38dceedcaf4d9482ec7a4b27250f2618)

P.S. [`ValueFromPipline`](https://learn.microsoft.com/en-us/powershell/scripting/developer/cmdlet/adding-parameters-that-process-pipeline-input?view=powershell-7.3#defining-input-from-the-pipeline) processing (in a function) requires special handling to leverage parallelization. Leverage [`.GetSteppablePipline()`](https://learn.microsoft.com/en-us/dotnet/api/system.management.automation.scriptblock.getsteppablepipeline) as in [this gist example](https://gist.github.com/petervandivier/19a8ff90ed8b3961d072d49e4d6cd705)

Answer #2

PeterVandivier

# PowerShell 5 (Desktop)

```powershell
#Requires -PSEdition Desktop

workflow foo {
    foreach -Parallel ($number in (1 .. 7)) {
        1 .. 10 | % {
            echo (Get-Date).ToString() >> "C:\Temp\${number}.txt" 
            Start-Sleep 1 
        }
    }
}

foo
```

Much like `function foo {}` defines an executable script, `workflow foo{}` defines a parallel [workflow](https://docs.microsoft.com/en-us/powershell/module/psworkflow/about/about_workflows?view=powershell-5.1). Invoking `foo` must be called separately after defining what `foo` is. 

We expect 10 seconds of CPU time **per file** for the script above. `Measure-Command {foo}` shows us that only 10 seconds of _wall clock time_ have passed, but 7 files exist in `C:\Temp` named `1.txt` through `7.txt`, each with 10 seconds worth of output.

![image.png](/image?hash=865d98e95c708b4c134a40eacdb167d45edda448718bd093b4b52e83f1b6aa90)
![image.png](/image?hash=f62de3860e6c04b42e0eec8675cf0327fbbc73f4ea2508ae0616b8796bc5c934)

Fair warning: Desktop edition workflows are a bit of a pain in the butt to work with IMHO. Note in the preceding example, `echo` isn't a true binary call but an alias to `Write-Output`, debugging specific workflow implementations is a whole other beast. 

Scoping issues with the [`InlineScript {}`](https://docs.microsoft.com/en-us/powershell/module/psworkflow/about/about_inlinescript?view=powershell-5.1) block type may be an issue for you with this usage depending on usage.

2 Answers