Concurrent Shell Command Queues

A Bash and Zsh command-queue pattern lets scripts submit shell commands over time while limiting how many run simultaneously. The implementation creates a FIFO in a private temporary directory and feeds NUL-separated commands to a background xargs process, whose -P option sets the concurrency limit. queue_start initializes the executor, queue_submit adds a command, and queue_wait closes the queue and waits for pending work to finish. Commands begin running as soon as they are submitted, so the full workload need not be known in advance. Examples show concurrent web requests, ordered queue submission with a limit of two jobs, submissions from subshells, separate queues in separate subshells, and commands read from output or a file. Output is collected before printing to reduce interleaving.

The approach is deliberately defensive but remains a complex workaround for shell’s limited concurrency support; Python or another language may be a better choice. Each shell can have only one queue, and queue_wait reports no overall job-failure status. Output is not streamed, though very large outputs may still interleave. Because child processes inherit the queue’s write descriptor, waiting can be delayed until every copy is closed, including by processes started after the queue begins. The cleanup routine may also terminate unrelated background descendants. When commands are already available up front, the simpler recommendation is to pipe them to xargs -P or use GNU parallel; the queue is useful specifically when commands must be added incrementally.