没有阶段 运行，但是 numRunningTasks != 0

Question

我的任务完成了，我得到了一个RDD计数的预期结果。我是 运行 一个交互式 PySpark shell。我想了解此警告的含义：

WARN ExecutorAllocationManager: No stages are running, but numRunningTasks != 0

从 Spark 的内部 code 我发现了这个：

    // If this is the last stage with pending tasks, mark the scheduler queue as empty
    // This is needed in case the stage is aborted for any reason
    if (stageIdToNumTasks.isEmpty) {
      allocationManager.onSchedulerQueueEmpty()
      if (numRunningTasks != 0) {
        logWarning("No stages are running, but numRunningTasks != 0")
        numRunningTasks = 0
      }
    }

有人可以解释一下吗？

---

我说的是 ID 为 0 的任务。

---

我可以报告说，使用 Spark 的 MLlib 遇到这种行为，使用 KMeans()，其中 据说可以用更少的任务完成。我不确定这个工作是否会失败..

2  takeSample at KMeans.scala:355 2016/08/27 21:39:04   7 s 1/1 9600/9600
1  takeSample at KMeans.scala:355 2016/08/27 21:38:57   6 s 1/1 6608/9600

---

输入集是 100m 个点，256 个维度。

PySpark 的一些参数：master 是 yarn，mode 是 cluster，

spark.dynamicAllocation.enabled             false
# Better serializer - https://spark.apache.org/docs/latest/tuning.html#data-serialization
spark.serializer                            org.apache.spark.serializer.KryoSerializer
spark.kryoserializer.buffer.max             2000m

# Bigger PermGen space, use 4 byte pointers (since we have < 32GB of memory)
spark.executor.extraJavaOptions             -XX:MaxPermSize=512m -XX:+UseCompressedOops

# More memory overhead
spark.yarn.executor.memoryOverhead          4096
spark.yarn.driver.memoryOverhead            8192

spark.executor.cores                        8
spark.executor.memory                       8G

spark.driver.cores                          8
spark.driver.memory                         8G
spark.driver.maxResultSize                  4G

Answer 1

我们得到的是这段代码：

    ...
    // If this is the last stage with pending tasks, mark the scheduler queue as empty
    // This is needed in case the stage is aborted for any reason
    if (stageIdToNumTasks.isEmpty) {
      allocationManager.onSchedulerQueueEmpty()
      if (numRunningTasks != 0) {
        logWarning("No stages are running, but numRunningTasks != 0")
        numRunningTasks = 0
      }
    }
  }
}

来自 Spark 的 GitHub，其中的评论是迄今为止最好的解释。