.NET HttpClient - 在响应 header 有不正确的 Content-Length 时接受部分响应

Question

我正在使用 .NET Core 3.1 开发 ASP.NET Web 应用程序。该应用程序从具有错误的外部网络服务器下载 mp3 文件：响应 header 中的 Content-Length 报告字节数高于 mp3 的实际字节数。

这是一个使用 curl 从该服务器下载文件的示例：

curl -sSL -D - "http://example.com/test.mp3" -o /dev/null
HTTP/1.1 200 OK
Cache-Control: private
Pragma: no-cache
Content-Length: 50561024
Content-Type: audio/mpeg
Content-Range: bytes 0-50561023/50561024
Expires: 0
Accept-Ranges: 0-50561023
Server: Microsoft-IIS/10.0
Content-Transfer-Encoding: binary
X-AspNet-Version: 4.0.30319
X-Powered-By: ASP.NET
Date: Fri, 03 Jan 2020 23:43:54 GMT

curl: (18) transfer closed with 266240 bytes remaining to read

所以即使 curl 报告传输不完整，mp3 已完全下载 50294784 字节，我可以在我尝试过的任何音频播放器中打开它。

我在我的网络应用程序中想要的是与 curl 相同的行为：忽略不正确的 Content-Length 并下载 mp3，直到服务器关闭传输。

现在我只是使用 HttpClient 异步下载 mp3:

internal static HttpClient httpClient = new HttpClient() { Timeout = new TimeSpan( 0, 15, 0 ) };

using( var response = await httpClient.GetAsync( downloadableMp3.Uri, HttpCompletionOption.ResponseContentRead ) )
using( var streamToReadFrom = await response.Content.ReadAsStreamAsync() )

然而，与 curl 不同的是，当传输关闭得太早时，传输会作为一个整体中止：

Task <SchedulerTaskWrapper FAILED System.Net.Http.HttpRequestException: Error while copying content to a stream.
 ---> System.IO.IOException: The response ended prematurely.
   at System.Net.Http.HttpConnection.FillAsync()
   at System.Net.Http.HttpConnection.CopyToContentLengthAsync(Stream destination, UInt64 length, Int32 bufferSize, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnection.ContentLengthReadStream.CompleteCopyToAsync(Task copyTask, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionResponseContent.SerializeToStreamAsync(Stream stream, TransportContext context, CancellationToken cancellationToken)
   at System.Net.Http.HttpContent.LoadIntoBufferAsyncCore(Task serializeToStreamTask, MemoryStream tempBuffer)
   --- End of inner exception stack trace ---
   at System.Net.Http.HttpContent.LoadIntoBufferAsyncCore(Task serializeToStreamTask, MemoryStream tempBuffer)
   at System.Net.Http.HttpClient.FinishSendAsyncBuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)

有什么方法可以将 HttpClient 配置为 "ignore" 不正确的 Content-Length 并仍然获取 mp3？

Answer 1

如果您查看方法 SendAsyncCore in dotnet runtime repo, you can see quite large code that implements core functionality of sending requests and handling responses. If the server sends the content-length header, this method internally creates ContentLengthReadStream. This stream expects a fixed number of bytes and is read until the expected amount is reached. If the content-length is greater than the real amount of bytes then ContentLengthReadStream 会抛出异常并显示消息 The response ended prematurely。

由于所有这些方法都非常严格和内部化，因此没有扩展或更改此功能的余地。但是有一个解决方法。您可以手动将流读入缓冲区，直到抛出异常。流的正常终止条件是 Read 方法 returns 零字节。如果 content-length 正确，也应包含此条件。

using var resp = await httpClient.GetAsync("http://example.com/test.mp3", HttpCompletionOption.ResponseHeadersRead);
using var contentStream = await resp.Content.ReadAsStreamAsync();

var bufferSize = 2048;
var buffer = new byte[bufferSize];
var result = new List<byte>();

try
{
    var readBytes = 0;
    while ((readBytes = contentStream.Read(buffer)) != 0)
    {
        for (int i = 0; i < readBytes; i++)
        {
            result.Add(buffer[i]);
        }
    }
}
catch (IOException ex)
{
    if (!ex.Message.StartsWith("The response ended prematurely"))
    {
        throw;
    }
}

以上代码将整个响应字节加载到列表 result 中。对于大内容，这可能不是一个好的解决方案。

另请注意，在这种情况下您不应使用 HttpCompletionOption.ResponseContentRead，因为如果您调用 GetAsync 方法，它会立即尝试读取内容。由于我们要稍后阅读内容，因此应将其更改为HttpCompletionOption.ResponseHeadersRead。这意味着 GetAsync 在读取 headers 时完成操作（而内容尚未读取）。

.NET HttpClient - 在响应 header 有不正确的 Content-Length 时接受部分响应

.NET HttpClient - Accept partial response when response header has an incorrect Content-Length

c#

dotnet-httpclient

.net-core

asp.net-core