Httpclient 计算多个 url 远程文件大小

Question

我要计算总文件大小为 1000+ url。虽然任务是运行我想显示当前数量和总数。但是如果我添加您可以在我的代码中看到的信息，我会收到 ssl 错误。

这是我遇到的错误：

    "Unhandled exception. System.Net.Http.HttpRequestException: The SSL connection could not be established, see inner exception.
 ---> System.IO.IOException:  Received an unexpected EOF or 0 bytes from the transport stream.
   at System.Net.Security.SslStream.ReceiveBlobAsync[TIOAdapter](TIOAdapter adapter)
   at System.Net.Security.SslStream.ForceAuthenticationAsync[TIOAdapter](TIOAdapter adapter, Boolean receiveFirst, Byte[] reAuthenticationData, Boolean isApm)
   at System.Net.Http.ConnectHelper.EstablishSslConnectionAsyncCore(Boolean async, Stream stream, SslClientAuthenticationOptions sslOptions, CancellationToken cancellationToken)
   --- End of inner exception stack trace ---
   at System.Net.Http.ConnectHelper.EstablishSslConnectionAsyncCore(Boolean async, Stream stream, SslClientAuthenticationOptions sslOptions, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.ConnectAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.CreateHttp11ConnectionAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.GetHttpConnectionAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)
   at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.HttpClient.SendAsyncCore(HttpRequestMessage request, HttpCompletionOption completionOption, Boolean async, Boolean emitTelemetryStartStop, CancellationToken cancellationToken)"

您能否建议显示计数的最佳方式？

 #region GET REMOTE FILE SIZE
public static async Task<string> GetRemoteFileSize(this List<string> urls) {
    long sum = 0;
    long localSum = 0;
    var tasks = new List<Task<long?>>();
    for(int i = 0; i < urls.Count; i++) {
        var url = urls[i];
        tasks.Add(GetTotalBytes(url));
        //if i will add here the counts, it's not getting the actual count
        Console.Title = $"{i} of {urls.Count}";
    }
    
    for (int i = 0; i < tasks.Count; i++) {
        Task<long?> task = tasks[i];
        localSum += Convert.ToInt32(await task.ConfigureAwait(false));
        Console.Title = $"{i} of {urls.Count}"; //if I will add this, I'm getting the SSL error even if I already added this " ServerCertificateCustomValidationCallback = (sender, cert, chain, sslPolicyErrors) => { return true; }," to my httpclient handler
    }
    await Task.WhenAll(tasks.ToArray()).ConfigureAwait(false);
    return Interlocked.Add(ref sum, localSum).FormatFileSize();
}
private static async Task<long?> GetTotalBytes(string url) {

    Uri uri = new Uri(url, UriKind.Absolute);
    uri.SetServiceManagerConnection();

    using var request = new HttpRequestMessage(HttpMethod.Head, uri);
    request.AddAllCommonHeaders(uri);
    using var response = await Client.SendAsync(request, HttpCompletionOption.ResponseHeadersRead).ConfigureAwait(false);
    HttpStatusCode statusCode = response.StatusCode;

    if (!response.IsSuccessStatusCode) {
        throw new InvalidDataException($"Error occured: {statusCode} {(int)statusCode}");
    }

    return response.Content.Headers.ContentLength.GetValueOrDefault();
}
#endregion GET REMOTE FILE SIZE

//我用的class

static string FormatBytes(this long bytes) {
        var unit = 1024;
        if (bytes < unit) { return $"{bytes} B"; }
        var exp = (int)(Math.Log(bytes) / Math.Log(unit));
        return $"{bytes / Math.Pow(unit, exp):F2} {("KMGTPE")[exp - 1]}B";
    }
    public static string FormatFileSize(this long bytes) => bytes.FormatBytes();

 public static void SetServiceManagerConnection(this Uri uri) {
        var sp = ServicePointManager.FindServicePoint(uri);
        sp.ConnectionLimit = 20; //default 2 //The number of connections per end point.
        sp.UseNagleAlgorithm = false; //Nagle’s algorithm is a means of improving the efficiency of TCP/IP networks by reducing the number of packets that need to be sent over the network
        sp.Expect100Continue = false; //save bandwidth before sending huge object for post and put request to ensure remote end point is up and running.
        sp.ConnectionLeaseTimeout = (int)TimeSpan.FromMinutes(1).TotalMilliseconds;//60 * 1000; // 1 minute
    }

更新：这个有效：

  static HttpClient Client { get; set; } = new(new HttpClientHandler { 
        Proxy = null, 
        UseProxy = false,
        SslProtocols = (SslProtocols)(SecurityProtocolType.Tls | SecurityProtocolType.Tls11 | SecurityProtocolType.Tls12 | SecurityProtocolType.Tls13)
    });

Answer 1

更新： 当 HTTPS 出现 TLS 握手错误时，此错误很典型。您应该指定 SecurityProtocol

System.Net.ServicePointManager.SecurityProtocol = 
SecurityProtocolType.Tls | SecurityProtocolType.Tls11 | SecurityProtocolType.Tls12 | SecurityProtocolType.Tls13;

参考：https://docs.microsoft.com/en-us/dotnet/api/system.net.securityprotocoltype

Answer 2

如其他人所述，您遇到了 TLS 错误。

一旦你解决了这个问题，你正在做的操作的一个更快的方法是 HTTP HEAD，这个 returns 给你只是 headers，在那里你可以看到尺寸。

注意： 并非所有服务器都允许 HEAD，例如 AWS。

这个问题的答案展示了如何用 HttpClient 做到这一点： HTTP HEAD request with HttpClient in .NET 4.5 and C#

Httpclient 计算多个 url 远程文件大小

Httpclient calculating multiple url remote file size

c#

.net-5