创建 StreamingContext 时出错

Question

我打开火花shell

spark-shell --packages org.apache.spark:spark-streaming-kafka_2.10:1.6.0

然后我想创建一个流上下文

import org.apache.spark._
import org.apache.spark.streaming._


val conf = new SparkConf().setMaster("local[2]").setAppName("NetworkWordCount").set("spark.driver.allowMultipleContexts", "true")
val ssc = new StreamingContext(conf, Seconds(1))

我运行进入异常：

org.apache.spark.SparkException: Only one SparkContext may be running in this JVM (see SPARK-2243). To ignore this error, set spark.driver.allowMultipleContexts = true. The currently running SparkContext was created at:

Answer 1

当您打开 spark-shell 时，已经创建了一个流上下文。它称为 sc，意味着您不需要创建配置对象。只需使用现有的 sc 对象。

val ssc = new StreamingContext(sc,Seconds(1))

我们将使用 val

而不是 var

创建 StreamingContext 时出错

Error when creating a StreamingContext

apache-spark

spark-streaming