Pig：计算文件中不同元组的频率

Question

我有一个文件，其中 json 个条目如下所示：

{"child_pos": "NN", "parent_pos": "NN", "parent": "fighter", "child_dep": "nn", "parent_dep": "nsubj", "child": "virtua"}
{"child_pos": "NN", "parent_pos": "NN", "parent": "case", "child_dep": "nn", "parent_dep": "nsubj", "child": "martin"}
{"child_pos": "NN", "parent_pos": "NN", "parent": "fighter", "child_dep": "nn", "parent_dep": "nsubj", "child": "virtua"}
{"child_pos": "NN", "parent_pos": "NN", "parent": "fighter", "child_dep": "nn", "parent_dep": "nsubj", "child": "virtua"}
{"child_pos": "NN", "parent_pos": "NN", "parent": "case", "child_dep": "nn", "parent_dep": "nsubj", "child": "martin"}

我想计算文件中不同 json 对象的频率。我在 Pig 中使用 Group By 和 count() 函数时看到了其他答案。我不确定我是否正确使用它们，但我没有得到所需的结果。我的输出应该是这样的：

{"child_pos": "NN", "parent_pos": "NN", "parent": "fighter", "child_dep": "nn", "parent_dep": "nsubj", "child": "virtua", "count": "3"}
{"child_pos": "NN", "parent_pos": "NN", "parent": "case", "child_dep": "nn", "parent_dep": "nsubj", "child": "martin", "count": "2"}

顺序并不重要。有人可以给我一些指示吗？

Answer 1

这是可以使用的代码，所有字段的条件都为基础如果你想要其他格式，你可以从元组中读取字段并使用任何其他格式

A = LOAD '/user/root/test12.json' USING JsonLoader('child_pos:chararray,               parent_pos:chararray, parent:chararray, child_dep:chararray, parent_dep:chararray, child:chararray');
B = GROUp A by (child_pos, parent_pos, parent, child_dep, parent_dep, child) ;
C = FOREACH B GENERATE group, COUNT(A.child_pos) as COUNTX;
STORE C into 'user/data/json_out.json' USING JsonStorage();

out put is ... 
{"group":    {"child_pos":"NN","parent_pos":"NN","parent":"case","child_dep":"nn","parent_dep":"nsubj","child":"martin"},"COUNTX":2}
{"group":{"child_pos":"NN","parent_pos":"NN","parent":"fighter","child_dep":"nn","parent_dep":"nsubj","child":"virtua"},"COUNTX":3}

Pig：计算文件中不同元组的频率

Pig: Count Frequency of distinct Tuples in a file

json

count

apache-pig