使用 pyspark 3.0.1 和 hadoop 3.2 构建 docker 图像

Question

我想用 Hadoop 3 为 pyspark 3.0.1 构建一个 docker 映像。2.x。在 docker 文件中，如果我使用 pip install pyspark==3.0.1，它会安装 pyspark 3.0，但 hadoop 是 2.7。有没有办法实现这个或 docker 文件的任何示例。

Answer 1

我能够使用此 docker 文件使用 pyspark 3.0 和 hadoop 3.2 创建 docker 图像。请注意 copy app.py /app/app.py 只是为了复制您想要运行.

的代码

FROM python:3.6-alpine3.10

ARG SPARK_VERSION=3.0.1
ARG HADOOP_VERSION_SHORT=3.2
ARG HADOOP_VERSION=3.2.0
ARG AWS_SDK_VERSION=1.11.375

RUN apk add --no-cache bash openjdk8-jre && \
  apk add --no-cache libc6-compat && \
  ln -s /lib/libc.musl-x86_64.so.1 /lib/ld-linux-x86-64.so.2 && \
  pip install findspark

# Download and extract Spark
RUN wget -qO- https://www-eu.apache.org/dist/spark/spark-${SPARK_VERSION}/spark-${SPARK_VERSION}-bin-hadoop${HADOOP_VERSION_SHORT}.tgz | tar zx -C /opt && \
    mv /opt/spark-${SPARK_VERSION}-bin-hadoop${HADOOP_VERSION_SHORT} /opt/spark

# To read IAM role for Fargate
 RUN echo spark.hadoop.fs.s3a.aws.credentials.provider=com.amazonaws.auth.EC2ContainerCredentialsProviderWrapper > /opt/spark/conf/spark-defaults.conf

# Add hadoop-aws and aws-sdk
RUN wget https://repo1.maven.org/maven2/org/apache/hadoop/hadoop-aws/${HADOOP_VERSION}/hadoop-aws-${HADOOP_VERSION}.jar -P /opt/spark/jars/ && \ 
    wget https://repo1.maven.org/maven2/com/amazonaws/aws-java-sdk-bundle/${AWS_SDK_VERSION}/aws-java-sdk-bundle-${AWS_SDK_VERSION}.jar -P /opt/spark/jars/

ENV PATH="/opt/spark/bin:${PATH}"
ENV PYSPARK_PYTHON=python3
ENV PYTHONPATH="${SPARK_HOME}/python:${SPARK_HOME}/python/lib/py4j-0.10.9-src.zip:${PYTHONPATH}"
# Define default command

RUN mkdir $SPARK_HOME/conf
RUN echo "SPARK_LOCAL_IP=127.0.0.1" > $SPARK_HOME/conf/spark-env.sh

#Copy python script for batch
copy app.py /app/app.py
# Define default command
CMD ["/bin/bash"]

使用 pyspark 3.0.1 和 hadoop 3.2 构建 docker 图像

build docker image with pyspark 3.0.1 with hadoop 3.2

hadoop

dockerfile

pyspark