在 Windows 上使用 Scala IDE 和 Apache Spark

Question

我想在 Windows 7 上开始使用 Spark 和 Scala 的项目。我下载了适用于 hadoop 2.4 (download page) 的 Apache Spark 预构建版本，我可以从命令提示符 (cmd) 运行它。我可以运行在自包含应用程序部分之前的 spark 页面快速启动上的所有代码。

然后我从它的下载页面下载了 Scala IDE 4.0.0（抱歉，post 不能超过 2 个链接）。

现在我创建了一个新的scala项目，并将spark assembly jar文件导入到项目中。当我想运行快速启动 spark 页面中自包含应用程序部分中的示例时，出现以下错误：

15/03/26 11:59:55 INFO AppClient$ClientActor: Connecting to master akka.tcp://sparkMaster@myhost:7077/user/Master...

15/03/26 11:59:58 WARN AppClient$ClientActor: Could not connect to akka.tcp://sparkMaster@myhost:7077: akka.remote.InvalidAssociation: Invalid address: akka.tcp://sparkMaster@myhost:7077

15/03/26 11:59:58 WARN Remoting: Tried to associate with unreachable remote address [akka.tcp://sparkMaster@myhost:7077]. Address is now gated for 5000 ms, all messages to this address will be delivered to dead letters. Reason: myhost

15/03/26 12:00:15 INFO AppClient$ClientActor: Connecting to master akka.tcp://sparkMaster@myhost:7077/user/Master...

15/03/26 12:00:17 WARN AppClient$ClientActor: Could not connect to akka.tcp://sparkMaster@myhost:7077: akka.remote.InvalidAssociation: Invalid address: akka.tcp://sparkMaster@myhost:7077

15/03/26 12:00:17 WARN Remoting: Tried to associate with unreachable remote address [akka.tcp://sparkMaster@myhost:7077]. Address is now gated for 5000 ms, all messages to this address will be delivered to dead letters. Reason: myhost

15/03/26 12:00:35 INFO AppClient$ClientActor: Connecting to master akka.tcp://sparkMaster@myhost:7077/user/Master...

15/03/26 12:00:37 WARN AppClient$ClientActor: Could not connect to akka.tcp://sparkMaster@myhost:7077: akka.remote.InvalidAssociation: Invalid address: akka.tcp://sparkMaster@myhost:7077

15/03/26 12:00:37 WARN Remoting: Tried to associate with unreachable remote address [akka.tcp://sparkMaster@myhost:7077]. Address is now gated for 5000 ms, all messages to this address will be delivered to dead letters. Reason: myhost

15/03/26 12:00:55 ERROR SparkDeploySchedulerBackend: Application has been killed. Reason: All masters are unresponsive! Giving up.

15/03/26 12:00:55 ERROR TaskSchedulerImpl: Exiting due to error from cluster scheduler: All masters are unresponsive! Giving up.

15/03/26 12:00:55 WARN SparkDeploySchedulerBackend: Application ID is not initialized yet.

我添加到示例中的唯一代码行是 .setMaster("spark://myhost:7077") 用于 SparkConf 定义。我想我需要配置 Scala IDE 才能在我的计算机上使用预构建的 spark 但实际上我不知道如何使用谷歌搜索也找不到任何东西。

你能帮我让 Scala IDE 在 windows 7 上与 spark 一起工作吗？

提前致谢

Answer 1

我找到了答案：

我应该更正我代码中的主定义如下：

替换：

.setMaster("spark://myhost:7077")

与：

.setMaster("local[*]")

希望对您也有帮助。

在 Windows 上使用 Scala IDE 和 Apache Spark

Using Scala IDE and Apache Spark on Windows

scala

apache-spark