Hive INSERT OVERWRITE 查询不写入 hdfs directory:Cannot 获取 DistCp 构造函数

Question

我在具有 10 百万行的配置单元中创建了一个 hbase 视图，当我在运行下方查询时，distcp 被调用并抛出以下错误。

INSERT OVERWRITE DIRECTORY '/mapred/INPUT' select hive_cdper1.cid,hive_cdper1.emptyp,hive_cdper1.ethtyp,hive_cdper1.gdtyp,hive_cdseg.mrtl from hive_cdper1 join hive_cdseg on hive_cdper1.cnm=hive_cdseg.cnm;

Output:map 100% reduce 100%

2016-10-17 15:05:34,688 INFO [main]: exec.Task (SessionState.java:printInfo(951)) - Moving data to: /mapred/INPUT from hdfs://mycluster/mapred/INPUT/.hive-staging_hive_2016-10-17_14-57-48_620_6609613978089243090-1/-ext-10000 2016-10-17 15:05:34,693 INFO [main]: common.FileUtils (FileUtils.java:copy(551)) - Source is 483335659 bytes. (MAX: 4000000) 2016-10-17 15:05:34,693 INFO [main]: common.FileUtils (FileUtils.java:copy(552)) - Launch distributed copy (distcp) job. 2016-10-17 15:05:34,695 ERROR [main]: exec.Task (SessionState.java:printError(960)) - Failed with exception Unable to move source hdfs://mycluster/mapred/INPUT/.hive-staging_hive_2016-10-17_14-57-48_620_6609613978089243090-1/-ext-10000 to destination /mapred/INPUT org.apache.hadoop.hive.ql.metadata.HiveException: Unable to move source hdfs://mycluster/mapred/INPUT/.hive-staging_hive_2016-10-17_14-57-48_620_6609613978089243090-1/-ext-10000 to destination /mapred/INPUT at org.apache.hadoop.hive.ql.metadata.Hive.moveFile(Hive.java:2644) at org.apache.hadoop.hive.ql.exec.MoveTask.moveFile(MoveTask.java:105) at org.apache.hadoop.hive.ql.exec.MoveTask.execute(MoveTask.java:222) at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:160) at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:88) at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1653) at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1412) at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1195) at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1059) at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1049) at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:213) at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:165) at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:376) at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:736) at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:681) at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:621) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.util.RunJar.run(RunJar.java:221) at org.apache.hadoop.util.RunJar.main(RunJar.java:136) Caused by: java.io.IOException: Cannot get DistCp constructor: org.apache.hadoop.tools.DistCp.() at org.apache.hadoop.hive.shims.Hadoop23Shims.runDistCp(Hadoop23Shims.java:1160) at org.apache.hadoop.hive.common.FileUtils.copy(FileUtils.java:553) at org.apache.hadoop.hive.ql.metadata.Hive.moveFile(Hive.java:2622) ... 21 more

我想知道的是 is:i 正在写入同一个集群，然后为什么它调用 distcp 而不是普通 cp。

我在这里使用 hive 1.2.1 和 hadoop 2.7.2，我的集群名称是 mycluster。

注意：我尝试设置 hive.exec.copyfile.maxsize=4000000 但没有成功。

感谢您的建议..

Answer 1

1) 检查目标路径的权限 /mapred/INPUT

2) 如果其他用户没有写入权限，则hadoop fs -chmod a+w /mapred/INPUT

Answer 2

在 hive-site.xml 中设置以下属性解决了我的问题。

  <property>
<name>hive.exec.copyfile.maxsize</name>
<value>3355443200</value>
<description>Maximum file size (in Mb) that Hive uses to do single HDFS copies between directories.Distributed copies (distcp) will be used instead for bigger files so that copies can be done faster.</description>
</property>

Hive INSERT OVERWRITE 查询不写入 hdfs directory:Cannot 获取 DistCp 构造函数

Hive INSERT OVERWRITE query not writing to hdfs directory:Cannot get DistCp constructor

hbase

hive

hadoop2