Parallel wget with xargs or parallel
Submitted by lev_lafayette on Wed, 05/14/2025 - 02:12The following is an illustration of how to use xargs to conduct parallel operations on single-threaded applications, specifically wget.
GNU wget is a great tool for downloading content from websites. The wget command is a non-interactive network downloader; by "non-interactive" what is meant is that it can be run in the background. Some very hand options include -c (continue, for partially downloaded files), -m (mirror, for an entire website), -r --no-parent (recursive, no parent, to download part of a website and its subdirectories). The cURL application has a wider range of protocols and includes upload options, but is non-recursive.

