Pig Latin 中不区分大小写的搜索

Question

这里是 Pig Latin 的初学者。我正在尝试计算输入文件中多个字符串的出现次数。

现在搜索必须不区分大小写。我知道 pig 中有一个 LOWER 内置函数，但我该如何使用它？

例如（输入文件）：

28-Oct-13,7:00PM,Viraj,New to hadoop ! Eager to learn.
31-Dec-14,3:00PM,‏Vanguard,Designers, Developers, Doers, don't miss this upcoming San Francisco Hadoop

我需要像 hadoop, dec, learn, python

这样的字符串的数量

hadoop 2
dec 1
learn 1
python 0

如何使用 pig Latin 进行搜索？

谢谢。

Answer 1

你能试试这个吗？

输入

28-Oct-13,7:00PM,Viraj,New to hadoop ! Eager to learn.
31-Dec-14,3:00PM,?Vanguard,Designers, Developers, Doers, don't miss this upcoming San Francisco Hadoop

PigScript:

A = LOAD 'input' AS (line:chararray);
B = FOREACH A GENERATE  FLATTEN(TOKENIZE(LOWER(line))) as word;
C = FOREACH B GENERATE ((word matches '.*hadoop.*'? 1:0)) as t1,((word matches '.*dec.*'?1:0)) as t2,((word matches '.*learn.*'?1:0)) as t3,((word matches '.*python.*'?1:0)) as t4;
D = GROUP C ALL;
E  = FOREACH D GENERATE FLATTEN(TOBAG(CONCAT('hadoop',' ',(chararray)SUM(C.t1)),CONCAT('dec',' ',(chararray)SUM(C.t2)),CONCAT('learn',' ',(chararray)SUM(C.t3)),CONCAT('python',' ',(chararray)SUM(C.t4))));
DUMP E;

输出：

(hadoop 2)
(dec 1)
(learn 1)
(python 0)

Pig Latin 中不区分大小写的搜索

Case-insensitive search in Pig Latin

hadoop

apache-pig

bigdata