删除 bash 中多个字符串的重复词

Question

我想知道如何使用 sed、awk 等从 bash 中的每一行中删除重复的单词...

我有一个 2000 行的文件，我想知道如何每行保留一个唯一的单词：

OG0000005 K00373 K00373 K00373 K00373 K00373 K00373 K00373 K00373 K00373 K00373 K00373 K00373 K00373 K00373 K00373 K00373  K00373
OG0000006 K00374 K00374 K00374 K00374 K00374 K00374 K00374 K00374 K00374 K00374 K00374 K00374 K00374 K00374 K00374 K00374  K00374
OG0000007 K03089  K03089 K03089 K03089 K03089 K03089 K03089 K03089 K03089 K03089 K03089 K03089 K03089 K03089 K03089 K03089 K03089
OG0000008 K15554  K15599 K15599 K15599 K15599 K15599 K15599 K15599 K15599 K15599 K15599 K15599 K15599 K15599 K15599 K15599 K15599
OG0000009 K15555  K15555 K15555 K15555 K15555 K15555 K15555 K15555 K15555 K15555 K15555 K15555 K15555 K15555 K15555 K15555 K15555
OG0000010 K00817  K09758 K00817 K00817 K00817 K00817 K00817 K00817 K00817 K00817 K00817 K00817 K00817 K00817 K00817 K00817 K00817
OG0000011 K07267  K07267  K07267  K07267 K07267 K07267 K07267 K07267 K07267 K07267 K07267 K07267 K07267 K07267 K07267 K07267
OG0000012 K22397  K01714 K01714 K01714 K01714 K01714 K01714 K01714 K01714 K01714 K01714 K01714 K01714 K01714 K01714 K01714 K01714
OG0000013 K00370  K07812 K00370 K00370 K00370 K00370 K00370 K00370 K00370 K00370 K00370 K00370 K00370 K00370 K00370 K00370 K00370

所以，输出应该是这样的：

OG0000005 K00373
OG0000006 K00374
OG0000007 K03089  
OG0000008 K15554  K15599 
OG0000009 K15555 
OG0000010 K00817  K09758

我试过：

sort file | uniq

wile read line
do
sort && uniq
done < file

Answer 1

您可以使用这个 awk 解决方案：

awk '
{
   delete seen
   printf "%s", 
   for (i=2; i<=NF; ++i)
      if (!seen[$i]++)
         printf "%s", OFS $i
   print ""
}' file

OG0000005 K00373
OG0000006 K00374
OG0000007 K03089
OG0000008 K15554 K15599
OG0000009 K15555
OG0000010 K00817 K09758
OG0000011 K07267
OG0000012 K22397 K01714
OG0000013 K00370 K07812

Answer 2

纯 Bash 解决方案可能是：

while read -r line; do 
    read -r -a a <<< "${line}"
    declare -A b
    for i in "${a[@]:1}"; do b["$i"]=1; done
    printf '%s %s\n' "${a[0]}" "${!b[*]}"
    unset b
done <file

要使 Bash + sort + uniq 方法起作用，您可以这样做：

while read -r line; do 
    read -r -a a <<< "${line}"
    re=$(tr ' ' '\n' <<< "${a[@]:1}" | sort | uniq | tr '\n' ' ' | xargs)
    printf "%s %s\n" "${a[0]}" "${re}"
done <file  
# if supported by your sort, you can also do 
# re=$(tr ' ' '\n' <<< "${a[@]:1}" | sort -u | tr '\n' ' ' | xargs)

要么打印：

OG0000005 K00373 
OG0000006 K00374 
OG0000007 K03089 
OG0000008 K15554 K15599 
OG0000009 K15555 
OG0000010 K00817 K09758 
OG0000011 K07267 
OG0000012 K22397 K01714 
OG0000013 K00370 K07812

Answer 3

这可能适合您 (GNU sed)：

sed -E ':a;s/(( +\S+)\>.*)\>//;ta' file

替换以一个单词开头的字符串，该单词后来由原始字符串减去重复的单词重复。

重复直到失败。

Answer 4

另一个解决方案without sed or awk，如果你不关心单词的原始顺序，可能是：

cat file | xargs -I _ sh -c "echo _ | tr ' ' '\n' | sort | uniq | tr '\n' ' '; echo"

输出：

K00373 OG0000005 
K00374 OG0000006 
K03089 OG0000007 
K15554 K15599 OG0000008 
K15555 OG0000009 
K00817 K09758 OG0000010 
K07267 OG0000011 
K01714 K22397 OG0000012 
K00370 K07812 OG0000013

否则，如果第一个单词有特殊含义，并且您想将其保留在原位置，解决方案可能是也使用 cut 和 paste，如下所示：

cat file | cut -d' ' -f1 --complement | xargs -I _ sh -c "echo _ | tr ' ' '\n' | sort | uniq | tr '\n' ' '; echo" | paste -d' ' <(cut -d' ' -f1 file) -

输出：

OG0000005 K00373 
OG0000006 K00374 
OG0000007 K03089 
OG0000008 K15554 K15599 
OG0000009 K15555 
OG0000010 K00817 K09758 
OG0000011 K07267 
OG0000012 K01714 K22397 
OG0000013 K00370 K07812

删除 bash 中多个字符串的重复词

remove duplicateds words for multiple strings in bash

bash

awk

sed