如何查看 Spark 独立集群的聚合日志
How can I see the aggregated logs for a Spark standalone cluster
在 Yarn 上使用 Spark 运行，在 Spark 作业完成后，我可以简单地使用 yarn -logs -applicationId appId 查看聚合日志。 Spark独立集群的等效方法是什么？
通过 Web Interface:
Spark’s standalone mode offers a web-based user interface to monitor
the cluster. The master and each worker has its own web UI that shows
cluster and job statistics. By default you can access the web UI for
the master at port 8080. The port can be changed either in the
configuration file or via command-line options.
In addition, detailed log output for each job is also written to the
work directory of each slave node (SPARK_HOME/work by default). You
will see two files for each job, stdout and stderr, with all output it
wrote to its console.
请在 Monitoring and Instrumentation 中找到更多信息。
在 Yarn 上使用 Spark 运行，在 Spark 作业完成后，我可以简单地使用 yarn -logs -applicationId appId 查看聚合日志。 Spark独立集群的等效方法是什么？
通过 Web Interface:
Spark’s standalone mode offers a web-based user interface to monitor the cluster. The master and each worker has its own web UI that shows cluster and job statistics. By default you can access the web UI for the master at port 8080. The port can be changed either in the configuration file or via command-line options.
In addition, detailed log output for each job is also written to the work directory of each slave node (SPARK_HOME/work by default). You will see two files for each job, stdout and stderr, with all output it wrote to its console.
请在 Monitoring and Instrumentation 中找到更多信息。