我可以使用文本作为值来写入 mapreduce 中的上下文吗

Question

我有一个场景可以计算 map reduce 中两列的平均值。所以我所做的是，我使用映射器从文件中获取了值并将它们连接为文本，然后尝试将它们写入上下文，如下所示。

class TestMapper extends Mapper<LongWritable, Text, Text, Text> {
  private Text outputKey;
  private Text outputVal;


  @Override
  public void map(LongWritable key, Text value, Context context) throws IOException, InterruptedException {
  
  //more code here

context.write(outputkey,OutputVal);


    
  }
}

Answer 1

您应该在这里使用自定义数据类型，例如一个 TextPair class，它有两个 Text 元素来存储您需要的数据。下面是一个示例代码，用于在映射器上下文的值中输出一对字符串。

// Mapper's map code
protected void map(LongWritable key, Text value, Mapper<LongWritable, Text, Text, TextPair>.Context context)
        throws IOException, InterruptedException {

    String line = value.toString();
    String year = line.substring(15, 19);
    int airTemperature;
    if (line.charAt(87) == '+') { // parseInt doesn't like leading plus signs
      airTemperature = Integer.parseInt(line.substring(88, 92));
    } else {
      airTemperature = Integer.parseInt(line.substring(87, 92));
    }
    String quality = line.substring(92, 93);
    if (airTemperature != MISSING && quality.matches("[01459]")) {
        System.out.println("Year "+year+" "+airTemperature);
      context.write(new Text(year), new TextPair(String.valueOf(airTemperature),1));
    }

// 文本对 - 自定义数据类型代码如下

public class TextPair implements WritableComparable<TextPair> {

private Text first;
private Text second;

//Default constructor is a must
public TextPair() {
    this.first=new Text();
    this.second=new Text();
}

public TextPair(String first,int second) {
    try {
        this.first=new Text(first);
        this.second=new Text(String.valueOf(second));
    }catch(Exception ex) {
        System.out.println("Exception occurred "+ex.getCause());
    }

}

// Other methods such as compare, equals, hashcode, write, readFields etc implementation also needs to done

public Text getFirst() {
    return first;
}

public Text getSecond() {
    return second;
}

@Override
public String toString() {
    return this.first+"\t"+this.second+"\t";
}

}

如果您需要更多详细信息，请参阅 Hadoop 权威指南。希望这有帮助。

我可以使用文本作为值来写入 mapreduce 中的上下文吗

can I use Text as value to write to context in mapreduce

hadoop

mapreduce

hadoop2