Parallel shells with xargs: Utilize all your cpu cores on UNIX and Windows

One particular frustration with the UNIX shell is the inability to easily schedule multiple concurrent tasks that fully utilize CPU cores presented on modern systems. The example of focus in this article is file compression but the problem rises with many computationally intensive tasks such as imageaudiomedia processing password cracking and hash analysis database Extract Transform and Load and backup activities. It is understandably frustrating to wait for gzip running on a single CPU core while most of a machine8217s processing power lies idle.

This can be understood as a weakness of the first decade of Research UNIX which was not developed on machines with SMP. The Bourne shell did not emerge from the 7th edition with any native syntax or controls for cohesively managing the resource consumption of background processes.

Utilities have haphazardly evolved to perform some of these functions. The GNU version of xargs is able to exercise some primitive control in allocating background processes which is discussed at some length in the documentation. While the GNU extensions to xargs have proliferated to many other implementations notably BusyBox including the release for Microsoft Windows example below they are not POSIX.2-compliant and likely will not be found on commercial UNIX.


Read Full Post

News Link:

Linux Chatter is a news aggregator service that curates some of the best Linux, Cloud, Technical Guides, Hardware and Security news. We display just enough content from the original post to spark your interest. If you like the topic, then click on the 'read full post' button to visit the author's website. Use Linux Chatter to find content from amazing authors!

Note: The content provided has been modified and is not displayed as intended by the author. Any trademarks, copyrights and rights remain with the source.

Disclaimer: Linux Chatter sources content from RSS feeds and personal content submissions. The views and opinions expressed in these articles are those of the authors and do not necessarily reflect those of Linux Chatter.

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.