Elasticsearch：docFreq是如何计算的

Question

我想了解 docFreq 是如何计算的。是每个索引，每个映射每个字段？

将 explain 设置为 true 时，我从查询中得到了这些结果。当hit处于mapping ListedName.standard docFreq如下图

 {
              "value" : 16.316673,
              "description" : """weight(ListedName.standard:"eagle pointe" in 48) [PerFieldSimilarity], result of:""",
              "details" : [
                {
                  "value" : 16.316673,
                  "description" : "score(doc=48,freq=1.0 = phraseFreq=1.0\n), product of:",
                  "details" : [
                    {
                      "value" : 3.0,
                      "description" : "boost",
                      "details" : [ ]
                    },
                    {
                      "value" : 5.4388914,
                      "description" : "idf(), sum of:",
                      "details" : [
                        {
                          "value" : 1.7870536,
                          "description" : "idf, computed as log(1 + (docCount - docFreq + 0.5) / (docFreq + 0.5)) from:",
                          "details" : [
                            {
                              "value" : 35.0,
                              "description" : "docFreq",
                              "details" : [ ]
                            },
                            {
                              "value" : 211.0,
                              "description" : "docCount",
                              "details" : [ ]
                            }
                          ]
                        },
                        {
                          "value" : 3.651838,
                          "description" : "idf, computed as log(1 + (docCount - docFreq + 0.5) / (docFreq + 0.5)) from:",
                          "details" : [
                            {
                              "value" : 5.0,
                              "description" : "docFreq",
                              "details" : [ ]
                            },
                            {
                              "value" : 211.0,
                              "description" : "docCount",
                              "details" : [ ]
                            }
                          ]
                        }
                      ]
                    },
                    {
                      "value" : 1.0,
                      "description" : "tfNorm, computed as (freq * (k1 + 1)) / (freq + k1) from:",
                      "details" : [
                        {
                          "value" : 1.0,
                          "description" : "phraseFreq=1.0",
                          "details" : [ ]
                        },
                        {
                          "value" : 0.0,
                          "description" : "parameter k1",
                          "details" : [ ]
                        },
                        {
                          "value" : 0.0,
                          "description" : "parameter b (norms omitted for field)",
                          "details" : [ ]
                        }
                      ]
                    }
                  ]
                }
              ]
            },

而当命中位于 mapping Line1 docFreq 很高，如下所示

  {
              "value" : 1.1640041,
              "description" : """weight(Line1:"eagle pointe" in 148) [PerFieldSimilarity], result of:""",
              "details" : [
                {
                  "value" : 1.1640041,
                  "description" : "score(doc=148,freq=1.0 = phraseFreq=1.0\n), product of:",
                  "details" : [
                    {
                      "value" : 3.0,
                      "description" : "boost",
                      "details" : [ ]
                    },
                    {
                      "value" : 0.38800138,
                      "description" : "idf(), sum of:",
                      "details" : [
                        {
                          "value" : 0.18813552,
                          "description" : "idf, computed as log(1 + (docCount - docFreq + 0.5) / (docFreq + 0.5)) from:",
                          "details" : [
                            {
                              "value" : 171.0,
                              "description" : "docFreq",
                              "details" : [ ]
                            },
                            {
                              "value" : 206.0,
                              "description" : "docCount",
                              "details" : [ ]
                            }
                          ]
                        },
                        {
                          "value" : 0.19986586,
                          "description" : "idf, computed as log(1 + (docCount - docFreq + 0.5) / (docFreq + 0.5)) from:",
                          "details" : [
                            {
                              "value" : 169.0,
                              "description" : "docFreq",
                              "details" : [ ]
                            },
                            {
                              "value" : 206.0,
                              "description" : "docCount",
                              "details" : [ ]
                            }
                          ]
                        }
                      ]
                    },
                    {
                      "value" : 1.0,
                      "description" : "tfNorm, computed as (freq * (k1 + 1)) / (freq + k1) from:",
                      "details" : [
                        {
                          "value" : 1.0,
                          "description" : "phraseFreq=1.0",
                          "details" : [ ]
                        },
                        {
                          "value" : 0.0,
                          "description" : "parameter k1",
                          "details" : [ ]
                        },
                        {
                          "value" : 0.0,
                          "description" : "parameter b (norms omitted for field)",
                          "details" : [ ]
                        }
                      ]
                    }
                  ]
                }
              ]
            }

Answer 1

这应该取决于评分模型（参见Similarity）是如何定义的，相似度算法可以在per-index或per-field的基础上设置。

Elasticsearch allows you to configure a scoring algorithm or similarity per field. The similarity setting provides a simple way of choosing a similarity algorithm other than the default BM25, such as TF/IDF.

现在，我们可以在评分说明输出中看到：

weight(<field>:"eagle pointe" in 48) [PerFieldSimilarity]

在这种情况下，docFreq 似乎仅限于在该字段中包含术语的文档数量。但是，我没有找到任何关于此的扩展信息，我不确定背后的逻辑，因为它应该取决于 class 相似性定义本身，而不是在特定的设置自定义的事实领域。

可以为整个索引设置默认相似度并在映射设置中为每个字段指定一个相似度：请参阅 Elasticsearch Reference [7.2] » Index modules » Similarity module。

您可能想要检查哪个相似性被用作默认值，以及是否有任何字段映射覆盖它。为了进行测试，我会尝试将默认值重置为 "classic" (tf-idf) 并删除这两个字段的任何现有覆盖以仔细检查 docFreq 是否在各个字段之间保持一致（哪个可能是一个错误）。

比照。 Lucene's TFIDFSimilarity

Elasticsearch：docFreq是如何计算的

Elasticsearch: How is docFreq calculated

lucene

elasticsearch