为什么 shell 带有 hadoop 的脚本不能工作？

Question

#!/usr/bin/env bash

echo textFile :""
echo mapper : ""
echo reducer: ""
echo inputDir :""
echo outputDir: ""

hdfs dfs -ls ~

hdfs dfs -rm ~/""
hdfs dfs -rm ~/""



hdfs dfs -copyFromLocal "" ~ # copies mapper.py file from argument to hdfs dir   
hdfs dfs -copyFromLocal "" ~ # copies reducer.py file from argument to hdfs dir


hdfs dfs -test -d ~/""  #checks to see if hadoop output dir exists
if [ $? == '0' ]; then
    hdfs dfs -rm -r ~/""
else
    echo "Output file doesn't exist and will be created when hadoop runs"
fi



hdfs dfs -test -d ~/"" #checks to see if hadoop input dir exists
if [ $? == 0 ]; then
    hdfs dfs -rm -r ~/""
    echo "Hadoop input dir alread exists deleting it now and creating a new      one..."
    hdfs dfs -mkdir ~/""  # makes an input dir for text file to be put in

else
    echo "Input file doesn't exist will be created now"
    hdfs dfs -mkdir ~/""  # makes an input dir for text file to be put in
fi



hdfs dfs -copyFromLocal /home/hduser/"" ~/"" # sends textfile from local   to hdfs folder

# runs the hadoop mapreduce program with given parameters
  hadoop jar /usr/local/hadoop/share/hadoop/tools/lib/hadoop-streaming-2.6.2.jar \
-input /home/hduser/""/* \
-output /home/hduser/"" \
-file /home/hduser/"" \
-mapper /home/hduser/"" \
-file /home/hduser/"" \
-reducer /home/hduser/""

我想避免每次我想测试映射器和缩减器文件时都将所有命令绑定到运行简单的 mapreduce，所以我写了这个脚本，我是 shell 脚本的新手。我附上了屏幕

Answer 1

您应该更正两个明显的细节：

bash 中的等号运算符拼写为 '"' 而不是 '=='
（实际上这对于测试表达式是正确的）
您用于 hadoop 调用的长命令行分布在多行中
您需要将它们连接成一条（长）线或更好地指示通过使用反斜杠“\”结束该行来继续。

为什么 shell 带有 hadoop 的脚本不能工作？

why shell script with hadoop wont work?

hadoop

sh