过滤 Apache Spark Log4J 消息以仅保留包含特定字符串的消息

Question

我有一些使用 Apache Spark 用 Java 编写的代码，我想禁用所有 Log4j 日志消息级别（ERROR、WARN 等）并仅保留包含特定字符串的 INFO 级别。换句话说，我有这些日志：

19/04/21 19:09:40 INFO Instrumentation: [e10c0eb5] {"seed":26,"impurity":"entropy","featuresCol":"indexedFeatures","maxDepth":5,"labelCol":"indexedLabel","numTrees":10}
19/04/21 19:09:40 INFO Instrumentation: [752ad4c3] {"seed":26,"impurity":"entropy","featuresCol":"indexedFeatures","maxDepth":5,"labelCol":"indexedLabel","numTrees":12}
19/04/21 19:09:40 INFO Instrumentation: [d9d09329] {"seed":26,"impurity":"entropy","featuresCol":"indexedFeatures","maxDepth":5,"labelCol":"indexedLabel","numTrees":11}
19/04/21 19:09:40 INFO SparkContext: Starting job: take at DecisionTreeMetadata.scala:112
19/04/21 19:09:40 INFO SparkContext: Starting job: take at DecisionTreeMetadata.scala:112
19/04/21 19:09:40 INFO SparkContext: Starting job: take at DecisionTreeMetadata.scala:112
19/04/21 19:09:40 INFO DAGScheduler: Got job 5 (take at DecisionTreeMetadata.scala:112) with 1 output partitions
19/04/21 19:09:40 INFO DAGScheduler: Final stage: ResultStage 6 (take at DecisionTreeMetadata.scala:112)
19/04/21 19:09:40 INFO DAGScheduler: Parents of final stage: List()
19/04/21 19:09:40 INFO DAGScheduler: Missing parents: List()

我只想保留以 "INFO Instrumentation" 开头的那些。我有这个示例代码：

    /*Logger.getLogger("org").setLevel(Level.OFF);
    Logger.getLogger("akka").setLevel(Level.OFF);*/








    SparkSession sparkSession = new SparkSession
            .Builder()
            .appName("Random Forest Classifier")
            .master("local[*]")
            .config("spark.ui.port", "40000")
            .getOrCreate();

我想更改前两个注释行以应用我的过滤器，这是否可行？如果可以，该怎么做？？

我的问题已经解决了；事实证明，我不需要任何过滤器或附加器；我只是禁用 "org" 和 "akka" 的所有日志，然后我只启用此 class 的信息级别："org.apache.spark.ml.util"，如下所示：

    Logger.getLogger("org").setLevel(Level.OFF);
    Logger.getLogger("akka").setLevel(Level.OFF);


    Logger.getLogger("org.apache.spark.ml.util").setLevel(Level.INFO);

感谢您的帮助。

Answer 1

您可以向包含您希望忽略的消息的记录器添加过滤器。
过滤器实现将过滤掉不符合给定谓词的消息，例如：

import org.apache.log4j.Level;
import org.apache.log4j.spi.Filter;
import org.apache.log4j.spi.LoggingEvent;

public class MyLog4jFilter extends Filter {

    /**
     * Custom filter to only log INFO events with the 'Instrumentation:' prefix in their message
     */
    @Override
    public int decide(LoggingEvent event) {
        if(event.getLevel() == Level.INFO && event.getMessage().trim().startsWith("Instrumentation:"))
           return ACCEPT;
        else 
           return DENY;
    }

}

参考文献：

过滤 Apache Spark Log4J 消息以仅保留包含特定字符串的消息

Filter Apache Spark Log4J Messages to keep only those which contain specific string

java

logging

log4j