如何访问 databricks 中的 spark executor、worker 和 master 级别指标？

Question

我的问题是针对数据块的。我正在尝试通过以下 spark 配置通过 databricks 中的石墨接收器访问 spark 指标，我希望在创建集群时传递配置。

spark.metrics.conf.*.sink.graphite.class org.apache.spark.metrics.sink.GraphiteSink
spark.metrics.conf.*.sink.graphite.host myhost
spark.metrics.conf.*.sink.graphite.port 2003
spark.metrics.conf.*.sink.graphite.period 10
spark.metrics.conf.*.sink.graphite.unit seconds
spark.metrics.conf.*.source.jvm.class org.apache.spark.metrics.source.JvmSource

但是，上述配置仅获取驱动程序级别的指标。我在一些帖子中读到要获得执行者级别的指标，必须传递以下参数

spark-submit <other parameters> --files metrics.properties 
--conf spark.metrics.conf=metrics.properties

我的问题是如何在创建集群时在 databricks 中传递 --files 参数（因为我没有执行 spark-submit）或者是否有任何其他方法来获取执行程序、工作程序和主程序级别指标 ?

集群Json

{
    "num_workers": 0,
    "cluster_name": "mycluster",
    "spark_version": "5.5.x-scala2.11",
    "spark_conf": {
        "spark.metrics.conf.*.sink.graphite.unit": "seconds",
        "spark.metrics.conf.*.sink.graphite.class": "org.apache.spark.metrics.sink.GraphiteSink",
        "spark.metrics.conf.*.sink.graphite.period": "10",
        "spark.databricks.delta.preview.enabled": "true",
        "spark.metrics.conf.*.source.jvm.class": "org.apache.spark.metrics.source.JvmSource",
        "spark.metrics.conf.*.sink.graphite.host": "myhost",
        "spark.metrics.conf.*.sink.graphite.port": "2003"
    },
    "aws_attributes": {
        "first_on_demand": 0,
        "availability": "ON_DEMAND",
        "zone_id": "us-west-2c",
        "spot_bid_price_percent": 100,
        "ebs_volume_count": 0
    },
    "node_type_id": "dev-tier-node",
    "driver_node_type_id": "dev-tier-node",
    "ssh_public_keys": [],
    "custom_tags": {},
    "spark_env_vars": {
        "PYSPARK_PYTHON": "/databricks/python3/bin/python3"
    },
    "autotermination_minutes": 120,
    "enable_elastic_disk": false,
    "cluster_source": "UI",
    "init_scripts": [],
    "cluster_id": "0604-114345-oboe241"
}

Answer 1

The --jars, --py-files, --files arguments support DBFS paths.

您可以如下所示在 spark 提交中指定指标文件的路径。

--files=dbfs/yourPath/metrics.properties --conf spark.metrics.conf=./metrics.properties

参考： Azure Databricks Jobs API - SparkSubmitTask

此 article 给出了一个示例，说明如何使用 Spark 可配置指标系统监控 Apache Spark 组件。具体来说，它展示了如何设置新源和启用接收器。

如何访问 databricks 中的 spark executor、worker 和 master 级别指标？

How to access spark executor, worker and master level metrics in databricks?

apache-spark

databricks

azure-databricks