如何在Hadoop 2.7.0 上编译MapReduce 作业源代码？

Question

我运行 Hadoop 2.7.0 on ubuntu 14.0.2 O.S.，我用 nano 文本编辑器创建 wordcount.java，源代码是从Apache Hadoop 2.7.0 Document.

在我用这个命令 javac -classpath hadoop-2.7.0-core.jar -d MyJava wordcount.java 编译 wordcount.java 之后，这是我得到的错误信息。

public class WordCount2 {
       ^
wordcount_v2.java:11: error: package org.apache.hadoop.conf does not exist
import org.apache.hadoop.conf.Configuration;
                             ^
wordcount_v2.java:12: error: package org.apache.hadoop.fs does not exist
import org.apache.hadoop.fs.Path;
                           ^
wordcount_v2.java:13: error: package org.apache.hadoop.io does not exist
import org.apache.hadoop.io.IntWritable;
                           ^
wordcount_v2.java:14: error: package org.apache.hadoop.io does not exist
import org.apache.hadoop.io.Text;
                           ^
wordcount_v2.java:15: error: package org.apache.hadoop.mapreduce does not exist
import org.apache.hadoop.mapreduce.Job;
                                  ^
wordcount_v2.java:16: error: package org.apache.hadoop.mapreduce does not exist
import org.apache.hadoop.mapreduce.Mapper;
                                  ^
wordcount_v2.java:17: error: package org.apache.hadoop.mapreduce does not exist
import org.apache.hadoop.mapreduce.Reducer;
                                  ^
wordcount_v2.java:18: error: package org.apache.hadoop.mapreduce.lib.input does not exist
import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
                                            ^
wordcount_v2.java:19: error: package org.apache.hadoop.mapreduce.lib.output does not exist
import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
                                             ^
wordcount_v2.java:20: error: package org.apache.hadoop.mapreduce does not exist
import org.apache.hadoop.mapreduce.Counter;
                                  ^
wordcount_v2.java:21: error: package org.apache.hadoop.util does not exist
import org.apache.hadoop.util.GenericOptionsParser;
                             ^
wordcount_v2.java:22: error: package org.apache.hadoop.util does not exist
import org.apache.hadoop.util.StringUtils;
                             ^
wordcount_v2.java:27: error: cannot find symbol
       extends Mapper<Object, Text, Text, IntWritable>{
               ^
  symbol:   class Mapper
  location: class WordCount2
wordcount_v2.java:27: error: cannot find symbol
       extends Mapper<Object, Text, Text, IntWritable>{
                              ^
  symbol:   class Text
  location: class WordCount2
wordcount_v2.java:27: error: cannot find symbol
       extends Mapper<Object, Text, Text, IntWritable>{
                                    ^
  symbol:   class Text
  location: class WordCount2
wordcount_v2.java:27: error: cannot find symbol
       extends Mapper<Object, Text, Text, IntWritable>{
                                          ^
  symbol:   class IntWritable
  location: class WordCount2
wordcount_v2.java:31: error: cannot find symbol
    private final static IntWritable one = new IntWritable(1);
                         ^
  symbol:   class IntWritable
  location: class TokenizerMapper
wordcount_v2.java:32: error: cannot find symbol
    private Text word = new Text();
            ^
  symbol:   class Text
  location: class TokenizerMapper
wordcount_v2.java:37: error: cannot find symbol
    private Configuration conf;
            ^
  symbol:   class Configuration
  location: class TokenizerMapper
wordcount_v2.java:41: error: cannot find symbol
    public void setup(Context context) throws IOException,
                      ^
  symbol:   class Context
  location: class TokenizerMapper
wordcount_v2.java:69: error: cannot find symbol
    public void map(Object key, Text value, Context context
                                ^
  symbol:   class Text
  location: class TokenizerMapper
wordcount_v2.java:69: error: cannot find symbol
    public void map(Object key, Text value, Context context
                                            ^
  symbol:   class Context
  location: class TokenizerMapper
wordcount_v2.java:88: error: cannot find symbol
       extends Reducer<Text,IntWritable,Text,IntWritable> {
               ^
  symbol:   class Reducer
  location: class WordCount2
wordcount_v2.java:88: error: cannot find symbol
       extends Reducer<Text,IntWritable,Text,IntWritable> {
                       ^
  symbol:   class Text
  location: class WordCount2
wordcount_v2.java:88: error: cannot find symbol
       extends Reducer<Text,IntWritable,Text,IntWritable> {
                            ^
  symbol:   class IntWritable
  location: class WordCount2
wordcount_v2.java:88: error: cannot find symbol
       extends Reducer<Text,IntWritable,Text,IntWritable> {
                                        ^
  symbol:   class Text
  location: class WordCount2
wordcount_v2.java:88: error: cannot find symbol
       extends Reducer<Text,IntWritable,Text,IntWritable> {
                                             ^
  symbol:   class IntWritable
  location: class WordCount2
wordcount_v2.java:89: error: cannot find symbol
    private IntWritable result = new IntWritable();
            ^
  symbol:   class IntWritable
  location: class IntSumReducer
wordcount_v2.java:91: error: cannot find symbol
    public void reduce(Text key, Iterable<IntWritable> values,
                       ^
  symbol:   class Text
  location: class IntSumReducer
wordcount_v2.java:91: error: cannot find symbol
    public void reduce(Text key, Iterable<IntWritable> values,
                                          ^
  symbol:   class IntWritable
  location: class IntSumReducer
wordcount_v2.java:92: error: cannot find symbol
                       Context context
                       ^
  symbol:   class Context
  location: class IntSumReducer
wordcount_v2.java:31: error: cannot find symbol
    private final static IntWritable one = new IntWritable(1);
                                               ^
  symbol:   class IntWritable
  location: class TokenizerMapper
wordcount_v2.java:32: error: cannot find symbol
    private Text word = new Text();
                            ^
  symbol:   class Text
  location: class TokenizerMapper
wordcount_v2.java:40: error: method does not override or implement a method from a supertype
    @Override
    ^
wordcount_v2.java:46: error: cannot find symbol
        URI[] patternsURIs = Job.getInstance(conf).getCacheFiles();
                             ^
  symbol:   variable Job
  location: class TokenizerMapper
wordcount_v2.java:48: error: cannot find symbol
          Path patternsPath = new Path(patternsURI.getPath());
          ^
  symbol:   class Path
  location: class TokenizerMapper
wordcount_v2.java:48: error: cannot find symbol
          Path patternsPath = new Path(patternsURI.getPath());
                                  ^
  symbol:   class Path
  location: class TokenizerMapper
wordcount_v2.java:64: error: cannot find symbol
            + StringUtils.stringifyException(ioe));
              ^
  symbol:   variable StringUtils
  location: class TokenizerMapper
wordcount_v2.java:68: error: method does not override or implement a method from a supertype
    @Override
    ^
wordcount_v2.java:80: error: cannot find symbol
        Counter counter = context.getCounter(CountersEnum.class.getName(),
        ^
  symbol:   class Counter
  location: class TokenizerMapper
wordcount_v2.java:89: error: cannot find symbol
    private IntWritable result = new IntWritable();
                                     ^
  symbol:   class IntWritable
  location: class IntSumReducer
wordcount_v2.java:95: error: cannot find symbol
      for (IntWritable val : values) {
           ^
  symbol:   class IntWritable
  location: class IntSumReducer
wordcount_v2.java:104: error: cannot find symbol
    Configuration conf = new Configuration();
    ^
  symbol:   class Configuration
  location: class WordCount2
wordcount_v2.java:104: error: cannot find symbol
    Configuration conf = new Configuration();
                             ^
  symbol:   class Configuration
  location: class WordCount2
wordcount_v2.java:105: error: cannot find symbol
    GenericOptionsParser optionParser = new GenericOptionsParser(conf, args);
    ^
  symbol:   class GenericOptionsParser
  location: class WordCount2
wordcount_v2.java:105: error: cannot find symbol
    GenericOptionsParser optionParser = new GenericOptionsParser(conf, args);
                                            ^
  symbol:   class GenericOptionsParser
  location: class WordCount2
wordcount_v2.java:111: error: cannot find symbol
    Job job = Job.getInstance(conf, "word count");
    ^
  symbol:   class Job
  location: class WordCount2
wordcount_v2.java:111: error: cannot find symbol
    Job job = Job.getInstance(conf, "word count");
              ^
  symbol:   variable Job
  location: class WordCount2
wordcount_v2.java:116: error: cannot find symbol
    job.setOutputKeyClass(Text.class);
                          ^
  symbol:   class Text
  location: class WordCount2
wordcount_v2.java:117: error: cannot find symbol
    job.setOutputValueClass(IntWritable.class);
                            ^
  symbol:   class IntWritable
  location: class WordCount2
wordcount_v2.java:122: error: cannot find symbol
        job.addCacheFile(new Path(remainingArgs[++i]).toUri());
                             ^
  symbol:   class Path
  location: class WordCount2
wordcount_v2.java:128: error: cannot find symbol
    FileInputFormat.addInputPath(job, new Path(otherArgs.get(0)));
                                          ^
  symbol:   class Path
  location: class WordCount2
wordcount_v2.java:128: error: cannot find symbol
    FileInputFormat.addInputPath(job, new Path(otherArgs.get(0)));
    ^
  symbol:   variable FileInputFormat
  location: class WordCount2
wordcount_v2.java:129: error: cannot find symbol
    FileOutputFormat.setOutputPath(job, new Path(otherArgs.get(1)));
                                            ^
  symbol:   class Path
  location: class WordCount2
wordcount_v2.java:129: error: cannot find symbol
    FileOutputFormat.setOutputPath(job, new Path(otherArgs.get(1)));
    ^
  symbol:   variable FileOutputFormat
  location: class WordCount2
56 errors

我想知道怎么会这样。我的 JDK 版本是 1.8.0_45

Answer 1

这些错误是由于依赖性引起的，您应该添加更多的库以进行编译而不会出现问题。库可以在 HADOOP_HOME 子目录中找到，例如 ~/share/hadoop/

Answer 2

您没有在您的代码中导入那些类错误。

我认为您必须将 hadoop-core-1.0.4.jar 添加到您的类路径并将所有这些类导入您的代码并再次编译，它应该可以工作。

您可以使用任何版本 hadoop-core-x.y.z.jar，它不特定于 hadoop-core-1.0.4.jar。

Answer 3

使用Maven 进行依赖管理。您在这里尝试做的是 HelloWorld 级别的事情。

参考以下link为Hadoop项目设置maven：Maven for Hadoop

Answer 4

这不是执行 hadoop map-reduce 程序的正确方法首先，您必须 安装 Hadoop 并进行配置关于此的安装指南 link http://www.bogotobogo.com/Hadoop/BigData_hadoop_Install_on_ubuntu_single_node_cluster.php

之后是字数统计示例

第 1 部分 http://www.bogotobogo.com/Hadoop/BigData_hadoop_Creating_Java_Wordcount_Project_with_Eclipse_MapReduce2.php

第二部分 http://www.bogotobogo.com/Hadoop/BigData_hadoop_Creating_Java_Wordcount_Project_with_Eclipse_MapReduce2_Part2.php

一步步按照所有说明操作...然后您的程序将执行...

Answer 5

仅供参考，因为一些参考资料是关于旧版本的 Hadoop MR。

Maven 大大简化了引用管理。你需要：

org.apache.hadoop:hadoop-common:2.7.0
org.apache.hadoop:hadoop-client:2.7.0
org.apache.hadoop:hadoop-auth:2.7.0
org.apache.hadoop:hadoop-hdfs:2.7.0

至少。祝你好运！

Answer 6

无需下载任何旧版本的jar 文件。你应该运行命令像

javac -classpath `yarn classpath` -d \path-to-where-class-file-to-be-stored\ wordcount.java

P.S。注意 yarn classpath 前后的 Tild ` 字符（制表符上方的一个键，左边为 1）。

如何在Hadoop 2.7.0 上编译MapReduce 作业源代码？

How to compile MapReduce job source code on Hadoop 2.7.0?

java

hadoop

compiler-errors

mapreduce