如何循环遍历大文件和输出行以通过标准输入卷曲？

Question

假设某个文件包含内容

a
b
c

我希望每行变成 3 个 http POST curl 命令。所以第 3 行会 post "c" 到一些 url.

我可以用 bash 遍历文件并像这样转储到 curl

cat somefile | while read line; \
  do curl -XPOST 'www.example.com' -d "$line"; \
done

然而，line 是一个巨大的 json 文件，有时通过命令行传递它会发生奇怪的事情。我宁愿有这样的东西

cat somefile | parallel curl -XPOST example.com -d @-

其中“@-”表示文件的每一行都通过标准输入传递给 curl。 gnu parallel 可以接受 {} 作为类似于上面的“$line”的参数，但我想要在将文件传递给下一个命令之前将文件转换为行流的东西。

Answer 1

ShellCheck 说：

Line 1:
cat somefile | while read line; \
                     ^-- SC2162: read without -r will mangle backslashes.

这可以解释它对 JSON 做了奇怪的事情，后者经常使用反斜杠：

$ echo '{ "key": "some value with \"nested quotes\" here" }' | \
      while read line; do echo "$line"; done
{ "key": "some value with "nested quotes" here" }

添加 -r 将使它们保持独立：

$ echo '{ "key": "some value with \"nested quotes\" here" }' | \
      while read -r line; do echo "$line"; done
{ "key": "some value with \"nested quotes\" here" }

为了完全正确，应该 while IFS= read -r line 也保留前导空格。

相关POSIX docs for read:

By default, unless the -r option is specified, < backslash> shall act as an escape character. An unescaped < backslash> shall preserve the literal value of the following character, with the exception of a < newline>. If a < newline> follows the < backslash>, the read utility shall interpret this as line continuation. The < backslash> and < newline> shall be removed before splitting the input into fields. All other unescaped < backslash> characters shall be removed after splitting the input into fields.

Answer 2

cat somefile | parallel 'echo {} | curl -XPOST example.com -d @-'

Answer 3

cat somefile | parallel --pipe -N1 curl -XPOST example.com -d @-

如何循环遍历大文件和输出行以通过标准输入卷曲？

How to loop through large file and output lines to curl through stdin?

bash

stdin

gnu-parallel