On the topic of xargs replacements, I love gnu parallel.
The --dry-run flag of parallel made me confident to do more batch processing than I ever did with xargs.
Parallel has an option for almost everything, it's almost too much.
But I have shopped around for alternatives.
The creator Ole Tange maintains a painstakingly long article of the alternatives and their differences. [0]
The gnu parallel book and reading materials [1] are excellent too.
Every time I’ve used Gnu Parallel on a new system it required accepting a Eula, which is annoying. I stick to xargs -P99 these days and am happier.
Utility software which has non-essential different first-run behavior is hostile to users.
somat 3 hours ago [-]
I have used echo as a sort of poor mans equivalent for a safe check of a pipeline, removing the echo when I felt the rest of the pipeline was working correctly.
shell stuff | xargs -n 1 -I % echo real command and % args
I have also been known to write scripts where instead of executing the critical parts it prints them. Then a dry run is
script
and the real run is
script | sh
krackers 1 hours ago [-]
Shouldn't you use echo instead of cat?
somat 1 hours ago [-]
yes echo, total brain fart there. corrected.
bryancoxwell 2 hours ago [-]
Love that idea, thanks for sharing
dataflow 3 hours ago [-]
Oh man, parallel is awful. I have to rant about this because I literally tried it again today morning.
Every single damn time I try parallel and decide to give it another chance, something ends up not working or causing a problem. I can never get it to just do what I want and get out of the way.
Today I foolishly thought maybe I was the one who was holding it wrong every single time in the past, so I copy pasted another command that was supposed to work, and thought surely this would be straightforward. Boy was I wrong. I got some manifesto about academic citations and plagiarism, which confused the hell out of me. After I wasted time trying to figure out how to turn off that nonsense, the app just hung there trying to figure out how long its command line can be? Literally doing nothing? What the hell? I killed it but then my terminal didn't close because every time I did this apparently some perl command was spawned in the background blocked on nothing. Why the hell was perl even relevant? Nothing I wrote used Perl. Just run the darn commands I asked in parallel, is that so hard?
nvme0n1p1 2 hours ago [-]
> the app just hung there trying to figure out how long its command line can be?
That's common on linux. Many tools read from stdin if a file path isn't given: cat, xargs, base64, cksum, etc.
The citation thing is a little silly, I'll give you that one.
2 hours ago [-]
fn-mote 1 hours ago [-]
> Today I foolishly thought maybe I was the one who was holding it wrong […]
Apparently goes on to describe being confused about parallel reading from the standard input?
quotemstr 3 hours ago [-]
I love GNU Parallel too. I love how it's free software and how I can patch out the obnoxious citation notice.
The rush project page on Github itself says it is only "slightly" faster.
eqvinox 4 hours ago [-]
Not sure I see the point of this. Remembering a bunch of options on one tool is no better than a bunch of tools/constructs? Especially if the latter are useful elsewhere. And it can't be used for scripts unless you start shipping it alongside, which… nah.
Just go with
foo | while read X; do bar "$X"; done
xorcist 1 hours ago [-]
Bash while loops are pretty readable and the above would be a nice to iterate over lines if it wasn't for that gnarly pipe, which is a common source of errors in this construct. Remember that a pipe starts a new shell. So:
grep stuff file.txt | while read key value ; do [ "$key" = "target" ] && found="$value" ; done
where you might expect $found to end up with the value for the line that has "target" in the first column. Then you notice that darn pipe symbol. The variable found is set in a subshell that terminates and the value is lost. This is a problem every time you need to keep some sort of state when looping. If you can tolate a bash-ism then you could do:
while read key value ; do [ "$key" = "target" ] && found="$value" ; done < <(grep stuff file.txt)
but that doesn't read as nice and isn't compatible. It does avoid a common source of problems though, and might be worth getting into muscle memory for the times it is needed.
dylan604 10 minutes ago [-]
> and isn't compatible
is that a GNU v POSIX type of compatibility issue?
laughing_man 1 hours ago [-]
Depending on the actual command, this can be far slower and less efficient than xargs. You're creating a separate process for each invocation of bar when a lot of commands will take many targets for a single invocation.
Try this with find and grep vs xargs. There's a big difference.
JoshTriplett 2 hours ago [-]
I use "| while read" as well, because it works well in a pipeline and handles embedded spaces. It doesn't handle embedded newlines, but in practice, real files have embedded spaces, while embedded newlines only happen in test cases and exploits. (You can do actual NUL-delimited reads with `-d ''`, but for a quick command-line operation that's generally not necessary, and if you're going to be that careful you probably also need `-r`.)
fiddlerwoaroof 2 hours ago [-]
I just use while’s default variable name, $REPLY most of the time. But `while` is my preferred tool for the xargs problem in most cases.
chasil 2 hours ago [-]
I use:
find . -name '*.log' -print0 | xargs -0 rm
For this simple example (derived from the article), find also has a delete operator.
This null termination is now a POSIX standard.
ivlad 2 hours ago [-]
Was exactly my thought: why not to use null termination?
Looks like a case where reading man page would have spared writing another copycat utility.
Walf 46 minutes ago [-]
Yeah, and I invariably add -r to xargs, so as not to execute the command unless results come through the pipeline.
If one is working with whole lines of text, setting the delimiter to newline is often desirable:
Xargs is fine, but openbsd has a -J option and every time I read the man page to figure out how to use it I read the -I and -J options and my brain glazes over.
The two arg cd is pretty useful (and, zsh implements it too because cd is a shell feature): if you have parallel directory structure (e.g. the way rails does tests) you can switch from app/b/c/d to spec/b/c/f by doing `cd app spec`
In a world with nix and where nearly every system has zsh, restricting yourself to the POSIX version of these tools is masochistic.
JoshTriplett 2 hours ago [-]
I highly recommend using whatever capabilities are convenient of any utility you're running, and not caring about what other systems do unless you're actually trying to write a portable shell script.
ggm 3 hours ago [-]
BSD vs linux with a standards committee inbetween.
koolba 4 hours ago [-]
And for most of those commands add a dash dash so that nothing with a dash prefix turns into options.
4 hours ago [-]
em-bee 3 hours ago [-]
why would you ever pipe find into xargs instead of calling -exec?
find -exec the '{}' iterated command ';'
gumby 3 hours ago [-]
Because xargs is faster. Exec will invoke the command once per matching file (which is sometimes what you want, of course)!
While xargs will accumulate a bunch of file names, then when it as n names will invoke the command with those names, while continuing to accumulate names until n is reached or the pipe closes.
The size of n depends on the system, but is usually at least a thousand.
em-bee 3 hours ago [-]
the example as given does not accumulate, so that is what i worked with.
find -exec command '{}' '+'
accumulates file arguments too. the only advantage of xargs is that you can tell it how many arguments to accumulate,̶ ̶t̶h̶e̶ ̶d̶o̶w̶n̶s̶i̶d̶e̶ ̶o̶f̶ ̶x̶a̶r̶g̶s̶ ̶i̶s̶ ̶t̶h̶a̶t̶ ̶y̶o̶u̶ ̶h̶a̶v̶e̶ ̶t̶o̶ ̶s̶p̶e̶c̶i̶f̶y̶ ̶t̶h̶e̶ ̶n̶u̶m̶b̶e̶r̶,̶ ̶w̶h̶e̶r̶e̶a̶s̶ ̶f̶i̶n̶d̶ ̶j̶u̶s̶t̶ ̶f̶i̶t̶s̶ ̶a̶s̶ ̶m̶a̶n̶y̶ ̶a̶s̶ ̶i̶t̶ ̶c̶a̶n̶.̶
xorcist 53 minutes ago [-]
The problem with find is that you can't specify the maximum tolerated command line lengths, and find hasn't historically been very smart about it (it just had a compiled-in value). Apparently this is fixed in GNU find, but for older systems and other platforms that may still be an issue.
Another feature missing in find that xargs has is the maximum processes to start at a time. find will run the commands in sequence, but in many situations you really want to run a bunch in parallel.
em-bee 45 minutes ago [-]
good points that i wasn't aware of, thank you. though personally i rarely start huge operations that need to be parallelized so i prefer simplicity over speed.
unsnap_biceps 2 hours ago [-]
xargs will fit as many as it can in 128 KiB or the system limit, which is smaller, so in practice, it's almost always pretty similar default packing
em-bee 1 hours ago [-]
you are right, i forgot to check the default. fixed my comment.
ggm 3 hours ago [-]
The prior has a point because 98.89% of the time I type xargs -n 1 -0 and we are deep in the useless pipe argument which Rob Pike amongst others has rehearsed well.
I do it because I do it, just like why I use egrep and sed in pipes along with awk.
t-3 2 hours ago [-]
Leah Neukirchen's lr and xe are very nice as a find and xargs replacement (although lr's test flag is way too complex). lr *.file | xe -s ' ... ' is a really great pattern for iterative scripting and hard to get wrong.
loremm 3 hours ago [-]
I have found that the most reliable way I like is to just construct the command externally and then pass to gnu parallel (mostly for --eta and --tmuxpane). And the great thing is, as others say, xargs -I. I prefer for shortness (and few collisoins), '@'
I know the echo is a little silly but then I can remove the |parallel and see if it's right. And if I don't want parallelism, I just pass to bash
3 hours ago [-]
samtheprogram 3 hours ago [-]
Literally just use xargs with -I {} and quotation marks?
laughing_man 1 hours ago [-]
That won't save you from weird file names, but null termination will.
raggi 3 hours ago [-]
in zsh you can just write:
for x (*.sh); echo "before $x after"
or
for x in *.sh; echo "before $x after"
or
for x (*.sh) { echo -n "before "; echo -n $x; echo " after" }
PunchyHamster 3 hours ago [-]
missed the point. reason to use xargs or parallel are generally two: list of arguments is too long, or list of arguments that is kept in memory would take too much
for example
for a in `find / ` ; do echo $a ; done
will take A LOT of memory, while using find's -exec, xargs, or parallel will not
unsnap_biceps 2 hours ago [-]
I looked into the source and the underlaying source uses a glob and suffers from the issue you brought up.
But bashenumerate doesn't do that? The parent is saying you should zsh shortloops instead of bashenumerate not zsh shortloops instead of xargs.
0xbadcafebee 3 hours ago [-]
I wanted something simpler — one consistent way to iterate over anything.
That's not actually simpler though. Simple is removing everything unnecessary. You took commands which could already do what you wanted, and added an extra program which calls them in specific ways. This will add bugs and maintenance headaches, not be portable, etc. This is added complexity.
The reason you made this script is not because you wanted simpler, you wanted easier. There's nothing wrong with that, and I'll grant you it probably is, especially for those unaccustomed to these commands. But easier != simpler. Often you'll find that simple is hard and easy is complexity deferred.
PunchyHamster 2 hours ago [-]
I feel like it could be just alias to some particular GNU Parallel set of options
along with: I love standards- Theres soany to choose from!
wallach-game 6 hours ago [-]
bashumerate — iterate over files, lines, ranges, or lists with a consistent {} syntax. No for loops, no find -exec, no xargs flags to remember.
enumerate -f '*.sh' -- wc -l {}
enumerate -L a b c -- 'echo {}'
Under 150 lines of bash, pluable sources, NUL-safe.
https://github.com/wallach-game/bashumerate
figmert 4 hours ago [-]
Have you seen parallel?
Rendered at 02:13:04 GMT+0000 (Coordinated Universal Time) with Vercel.
The --dry-run flag of parallel made me confident to do more batch processing than I ever did with xargs.
Parallel has an option for almost everything, it's almost too much.
But I have shopped around for alternatives. The creator Ole Tange maintains a painstakingly long article of the alternatives and their differences. [0]
The gnu parallel book and reading materials [1] are excellent too.
[0] https://www.gnu.org/software/parallel/parallel_alternatives....
[1] https://www.gnu.org/software/parallel/#Tutorial
Utility software which has non-essential different first-run behavior is hostile to users.
Every single damn time I try parallel and decide to give it another chance, something ends up not working or causing a problem. I can never get it to just do what I want and get out of the way.
Today I foolishly thought maybe I was the one who was holding it wrong every single time in the past, so I copy pasted another command that was supposed to work, and thought surely this would be straightforward. Boy was I wrong. I got some manifesto about academic citations and plagiarism, which confused the hell out of me. After I wasted time trying to figure out how to turn off that nonsense, the app just hung there trying to figure out how long its command line can be? Literally doing nothing? What the hell? I killed it but then my terminal didn't close because every time I did this apparently some perl command was spawned in the background blocked on nothing. Why the hell was perl even relevant? Nothing I wrote used Perl. Just run the darn commands I asked in parallel, is that so hard?
That's common on linux. Many tools read from stdin if a file path isn't given: cat, xargs, base64, cksum, etc.
The citation thing is a little silly, I'll give you that one.
Apparently goes on to describe being confused about parallel reading from the standard input?
https://github.com/shenwei356/rush
Just go with
is that a GNU v POSIX type of compatibility issue?
Try this with find and grep vs xargs. There's a big difference.
This null termination is now a POSIX standard.
Looks like a case where reading man page would have spared writing another copycat utility.
If one is working with whole lines of text, setting the delimiter to newline is often desirable:
xargs -d \\n
https://man.openbsd.org/xargs
Other brain glazing obsd wierdness is it's two argument cd command, a cryptid I am unable to wrap my head around.
https://man.openbsd.org/ksh#cd~2
While xargs will accumulate a bunch of file names, then when it as n names will invoke the command with those names, while continuing to accumulate names until n is reached or the pipe closes.
The size of n depends on the system, but is usually at least a thousand.
Another feature missing in find that xargs has is the maximum processes to start at a time. find will run the commands in sequence, but in many situations you really want to run a bunch in parallel.
I do it because I do it, just like why I use egrep and sed in pipes along with awk.
seq 1 10|xargs -I@ echo 'bash run.py @'|parallel -j 10
I know the echo is a little silly but then I can remove the |parallel and see if it's right. And if I don't want parallelism, I just pass to bash
for example
will take A LOT of memory, while using find's -exec, xargs, or parallel will nothttps://github.com/wallach-game/bashumerate/blob/master/lib/...
That's not actually simpler though. Simple is removing everything unnecessary. You took commands which could already do what you wanted, and added an extra program which calls them in specific ways. This will add bugs and maintenance headaches, not be portable, etc. This is added complexity.
The reason you made this script is not because you wanted simpler, you wanted easier. There's nothing wrong with that, and I'll grant you it probably is, especially for those unaccustomed to these commands. But easier != simpler. Often you'll find that simple is hard and easy is complexity deferred.