将 log4net 文件推送到 ElasticSearch

Question

我已经编写了 c# WebAPI，并希望使用 log4net 将请求记录到文件中。将日志写入日志文件后，我想将这些日志文件（将使用 DelegateHandler 捕获所有请求）推送到 ElasticSearch。我到处搜索，但找不到解决问题的方法。以下是我拥有的 log4net.config 文件（ElasticAppender 不起作用）

   <log4net>
     <root>
       <appender-ref ref="FileAppender" />
       <appender-ref ref="ElasticSearchAppender" />
     </root>
     <appender name="FileAppender" type="log4net.Appender.RollingFileAppender">
       <file type="log4net.Util.PatternString" value="Logs/API_.log" />
       <appendToFile value="true" />
       <rollingStyle value="Composite" />
       <datePattern value="yyyy-MM-dd" />
       <maximumFileSize value="10MB" />
       <preserveLogFileNameExtension value="true"/>
       <staticLogFileName value="false" />
       <layout type="log4net.Layout.PatternLayout">
          <conversionPattern value="%date{HH:mm:ss.fff}, %message%newline" />
       </layout>
       <filter type="log4net.Filter.LevelRangeFilter">
          <levelMin value="DEBUG" />
          <levelMax value="ERROR" />
       </filter>
     </appender>
     <appender name="ElasticSearchAppender" type="log4net.ElasticSearch.ElasticSearchAppender, log4stash">
       <Server>elasticnode01</Server>
       <Port>9200</Port>
       <IndexName>apilogentry</IndexName>
       <ElasticFilters>
          <Filter type="log4net.ElasticSearch.Filters.RenameKeyFilter, log4stash">
            <Key>processId</Key>
            <RenameTo>ServiceName</RenameTo>
          </Filter>
          <Filter type="log4net.ElasticSearch.Filters.RenameKeyFilter, log4stash">
            <Key>Message</Key>
            <RenameTo>message</RenameTo>
         </Filter>
         <Grok>
            <SourceKey>message</SourceKey>
            <Pattern>%{NUMBER:ResponseCode} %{WORD:Method} %{URIPATHPARAM:Url} %{NUMBER:ElapsedMls} %{GREEDYDATA:StatusMessage}</Pattern>
         </Grok>
      </ElasticFilters>
      <filter type="log4net.Filter.LevelRangeFilter">
        <levelMin value="INFO" />
        <levelMax value="ERROR" />
      </filter>
    </appender>
 </log4net>

这是我用来收集数据（我想要记录）的 class。

  public class ApiLogEntry
{
    public long ApiLogEntryId { get; set; }             // The (database) ID for the API log entry.
    public string Application { get; set; }             // The application that made the request.
    public string User { get; set; }                    // The user that made the request.
    public string Machine { get; set; }                 // The machine that made the request.
    public string RequestIpAddress { get; set; }        // The IP address that made the request.
    public string RequestContentType { get; set; }      // The request content type.
    public string RequestContentBody { get; set; }      // The request content body.
    public string RequestUri { get; set; }              // The request URI.
    public string RequestMethod { get; set; }           // The request method (GET, POST, etc).
    public string RequestRouteTemplate { get; set; }    // The request route template.
    public string RequestRouteData { get; set; }        // The request route data.
    public string RequestHeaders { get; set; }          // The request headers.
    public DateTime? RequestTimestamp { get; set; }     // The request timestamp.
    public string ResponseContentType { get; set; }     // The response content type.
    public string ResponseContentBody { get; set; }     // The response content body.
    public int? ResponseStatusCode { get; set; }        // The response status code.
    public string ResponseHeaders { get; set; }         // The response headers.
    public DateTime? ResponseTimestamp { get; set; }    // The response timestamp.
}

这是处理所有 HTTP 请求并应记录信息的 class。现在信息正在记录到文件中。

public class ApiLogHandler : DelegatingHandler
{
    ILogger _logger;
    protected override async Task<HttpResponseMessage> SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
    {
        var apiLogEntry = CreateApiLogEntryWithRequestData(request);
        if (request.Content != null)
        {
            await request.Content.ReadAsStringAsync()
                .ContinueWith(task =>
                {
                    apiLogEntry.RequestContentBody = task.Result;
                }, cancellationToken);
        }

        return await base.SendAsync(request, cancellationToken)
            .ContinueWith(task =>
            {
                var response = task.Result;

                // Update the API log entry with response info
                apiLogEntry.ResponseStatusCode = (int)response.StatusCode;
                apiLogEntry.ResponseTimestamp = DateTime.Now;

                if (response.Content != null)
                {
                  //  apiLogEntry.ResponseContentBody = response.Content.ReadAsStringAsync().Result;
                    apiLogEntry.ResponseContentType = response.Content.Headers.ContentType.MediaType;
                    apiLogEntry.ResponseHeaders = SerializeHeaders(response.Content.Headers);
                }

                // TODO: Save the API log entry to the database


                _logger.Info(apiLogEntry);

                return response;
            }, cancellationToken);
    }

    private ApiLogEntry CreateApiLogEntryWithRequestData(HttpRequestMessage request)
    {
        var context = ((HttpContextBase)request.Properties["MS_HttpContext"]);
        var routeData = request.GetRouteData();

        return new ApiLogEntry
        {
            Application = "Search.API",
            User = context.User.Identity.Name,
            Machine = Environment.MachineName,
            RequestContentType = context.Request.ContentType,
            RequestRouteTemplate = routeData.Route.RouteTemplate,
            //RequestRouteData = SerializeRouteData(routeData),
            RequestIpAddress = context.Request.UserHostAddress,
            RequestMethod = request.Method.Method,
            RequestHeaders = SerializeHeaders(request.Headers),
            RequestTimestamp = DateTime.Now,
            RequestUri = request.RequestUri.ToString()
        };
    }

    private string SerializeRouteData(IHttpRouteData routeData)
    {
        return JsonConvert.SerializeObject(routeData, Formatting.None, new JsonSerializerSettings()
        {
            ReferenceLoopHandling = ReferenceLoopHandling.Ignore
        });

        //return JsonConvert.SerializeObject(routeData, Formatting.Indented, new JsonSerializerSettings
        //{
        //    PreserveReferencesHandling = PreserveReferencesHandling.Objects
        //});
    }

    private string SerializeHeaders(HttpHeaders headers)
    {
        var dict = new Dictionary<string, string>();

        foreach (var item in headers.ToList())
        {
            if (item.Value != null)
            {
                var header = String.Empty;
                foreach (var value in item.Value)
                {
                    header += value + " ";
                }

                // Trim the trailing space and add item to the dictionary
                header = header.TrimEnd(" ".ToCharArray());
                dict.Add(item.Key, header);
            }
        }
        return JsonConvert.SerializeObject(dict, Formatting.Indented);
    }

    public ApiLogHandler(ILogger logger)
    {
        _logger = logger;
    }
}

如何将这些文件的输出输出到 ElasticSearch？我已经阅读了有关 FileBeat 和 logstash 的信息，但不确定如何获得这一切运行。有人对我可能缺少的东西有任何想法吗？（我知道 log4net.config 中的 grok 条目不正确（我正在尝试映射到 ApiLogEntry class 但不知道如何继续）。

TIA

Answer 1

遵循文档 Log4net.ElasticSearch NuGet Lib 你应该能够实现你的目标。

Answer 2

Elastic Search Log4Net appender 的文档does not recommend using this approach in production。

它推荐的是使用 Filebeat to push the log file data via LogStash (for data transformation/parsing) into an Elastic Search database. At the time of writing all of these components could be used freely on a self hosted Basic installation。

将 log4net 文件推送到 ElasticSearch

Pushing log4net files to ElasticSearch

c#

log4net

elasticsearch