使用 Pig 访问 Hbase

Question

我正在尝试使用 Pig (0.15.0) 从 HBase (1.1.2) 加载数据。我试过注册太多的 JAR，但都没有成功。我可以使用 Java 访问它，但不能使用 Pig。下面我已经注册了所有的 JAR。前五个，然后是我的 Java 代码正在使用的 lib/*.jar then target/alternateLocation/*.jar。一切都安装在我的本地 PC 上，HBase 在 HDFS 上运行ning。

--REGISTER /home/halmousli/hbase-1.1.2/lib/*.jar

--REGISTER /home/halmousli/mygit/Hbase/apachehbase/target/alternateLocation/*.jar

set hbase.zookeeper.quorum 'localhost'
data = LOAD 'hbase://emp'       USING org.apache.pig.backend.hadoop.hbase.HBaseStorage('professional:designation', '-loadKey true')
   as (id:bytearray,prof:CHARARRAY);
DUMP data;

pig script failed to validate: java.lang.RuntimeException: could not instantiate 'org.apache.pig.backend.hadoop.hbase.HBaseStorage' with arguments '[professional:designation, -loadKey true]' Failed to parse: Pig script failed to parse: pig script failed to validate: java.lang.RuntimeException: could not instantiate 'org.apache.pig.backend.hadoop.hbase.HBaseStorage' with arguments '[professional:designation, -loadKey true]' at org.apache.pig.parser.QueryParserDriver.parse(QueryParserDriver.java:199) at org.apache.pig.PigServer$Graph.parseQuery(PigServer.java:1735) at org.apache.pig.PigServer$Graph.access[=14=]0(PigServer.java:1443) at org.apache.pig.PigServer.parseAndBuild(PigServer.java:387) at org.apache.pig.PigServer.executeBatch(PigServer.java:412) at org.apache.pig.PigServer.executeBatch(PigServer.java:398) at org.apache.pig.tools.grunt.GruntParser.executeBatch(GruntParser.java:171) at org.apache.pig.tools.grunt.GruntParser.processDump(GruntParser.java:749) at org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:376) at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:230) at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:205) at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:81) at org.apache.pig.Main.run(Main.java:631) at org.apache.pig.Main.main(Main.java:177) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.hadoop.util.RunJar.run(RunJar.java:221) at org.apache.hadoop.util.RunJar.main(RunJar.java:136) Caused by: /home/halmousli/mygit/Pig/mavenpigtester/src/main/resources/hbaseGetter.pig, line 21, column 7> pig script failed to validate: java.lang.RuntimeException: could not instantiate 'org.apache.pig.backend.hadoop.hbase.HBaseStorage' with arguments '[professional:designation, -loadKey true]' at org.apache.pig.parser.LogicalPlanBuilder.buildLoadOp(LogicalPlanBuilder.java:897) at org.apache.pig.parser.LogicalPlanGenerator.load_clause(LogicalPlanGenerator.java:3568) at org.apache.pig.parser.LogicalPlanGenerator.op_clause(LogicalPlanGenerator.java:1625) at org.apache.pig.parser.LogicalPlanGenerator.general_statement(LogicalPlanGenerator.java:1102) at org.apache.pig.parser.LogicalPlanGenerator.statement(LogicalPlanGenerator.java:560) at org.apache.pig.parser.LogicalPlanGenerator.query(LogicalPlanGenerator.java:421) at org.apache.pig.parser.QueryParserDriver.parse(QueryParserDriver.java:191) ... 19 more Caused by: java.lang.RuntimeException: could not instantiate 'org.apache.pig.backend.hadoop.hbase.HBaseStorage' with arguments '[professional:designation, -loadKey true]' at org.apache.pig.impl.PigContext.instantiateFuncFromSpec(PigContext.java:772) at org.apache.pig.parser.LogicalPlanBuilder.buildLoadOp(LogicalPlanBuilder.java:885) ... 25 more Caused by: java.lang.reflect.InvocationTargetException at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:526) at org.apache.pig.impl.PigContext.instantiateFuncFromSpec(PigContext.java:740) ... 26 more
Caused by: java.lang.NoSuchMethodError: org.apache.hadoop.hbase.client.Scan.setCacheBlocks(Z)V at org.apache.pig.backend.hadoop.hbase.HBaseStorage.initScan(HBaseStorage.java:427) at org.apache.pig.backend.hadoop.hbase.HBaseStorage.(HBaseStorage.java:368) ... 31 more 2015-12-30 11:19:14,794 [main] ERROR org.apache.pig.tools.grunt.Grunt - ERROR 1200: Pig script failed to parse: pig script failed to validate: java.lang.RuntimeException: could not instantiate 'org.apache.pig.backend.hadoop.hbase.HBaseStorage' with arguments '[professional:designation, -loadKey true]' Details at logfile: /home/halmousli/hadoop-2.6.0/sbin/pig_1451492353570.log

我不知道为什么我看到的原因是：java.lang.NoSuchMethodError: org.apache.hadoop.hbase.client.Scan.setCacheBlocks。在我的 Java 代码中，我能够找到该函数并运行它但在 Pig 中找不到，我使用的是相同的罐子。

Scan scan = new Scan();
scan.setCacheBlocks(true);

对于解决这个问题或从 Pig 访问 HBase 的其他方式有什么建议吗？。谢谢。

Answer 1

出现上述错误的原因是我的hadoop版本和hbase不兼容。我安装了hbase-0.98.16-hadoop2，pig代码是运行没有报错。

通常当您看到 java.lang.NoSuchMethodError 时，可能表示版本之间不兼容。

Answer 2

如果您想使用 pig 访问 hbase，更安全的版本是 hbase 0.98 和 pig 0.15。我希望这会解决您的集成问题

Answer 3

HBase 社区在将 #setCacheBlocks() 的 return 值从 'void' 更改为 'Scan' 时破坏了二进制兼容性。期望从 HBase 0.98 中找到 setCacheBlocks()，但找到 HBase 1.1.2 的版本。

一种解决方案是重新编译 Pig 以使用 HBase 1.1.2。

https://issues.apache.org/jira/browse/PIG-4743

使用 Pig 访问 Hbase

Access Hbase with Pig

hbase

apache-pig