将 PUT HTTP 请求代理到 AWS S3 失败

Question

我正在创建一个代理，它可以简单地将 HTTP 请求中继到 Amazon S3。代理现在使用带有 ngx_aws_auth 模块的 Nginx，该模块将必要的 headers 添加到 HTTP 请求中，因此客户端不需要。

Nginx 配置：

http {
    include       mime.types;
    default_type  application/octet-stream;
    sendfile        on;
    client_body_in_file_only clean;
    client_body_buffer_size 32K;

    client_max_body_size 300M;

    send_timeout 300s;

    keepalive_timeout  65;

    gzip  on;

 server {
    listen     8000;

    location / {
      proxy_pass https://test-bucket.s3.amazonaws.com;

      aws_access_key xxxxx;
      aws_secret_key yyyyyy;
      s3_bucket test-bucket;

      proxy_set_header Authorization $s3_auth_token;
      proxy_set_header x-amz-date $aws_date;
    }
  }
}

几乎一切都很好，除非我尝试上传更大的文件。

获取：

$ curl -vX GET http://192.168.99.131:8000/a.txt ; echo
* About to connect() to 192.168.99.131 port 8000 (#0)
*   Trying 192.168.99.131...
* Connected to 192.168.99.131 (192.168.99.131) port 8000 (#0)
> GET /a.txt HTTP/1.1
> User-Agent: curl/7.29.0
> Host: 192.168.99.131:8000
> Accept: */*
>
< HTTP/1.1 200 OK
< Server: nginx/1.6.3
< Date: Sun, 30 Aug 2015 18:18:50 GMT
< Content-Type: application/x-www-form-urlencoded
< Content-Length: 1
< Connection: keep-alive
< x-amz-id-2: 1F/eQtkJU=
< x-amz-request-id: A970D9BCF5D5890B
< Last-Modified: Sat, 29 Aug 2015 00:58:33 GMT
< ETag: "0cc175b9c0f1b6a831c399e269772661"
< Accept-Ranges: bytes
<
* Connection #0 to host 192.168.99.131 left intact
a

PUT 小文件：

$ echo 'a'>a;curl -vX PUT -d @a http://192.168.99.131:8000/a.txt
* About to connect() to 192.168.99.131 port 8000 (#0)
*   Trying 192.168.99.131...
* Connected to 192.168.99.131 (192.168.99.131) port 8000 (#0)
> PUT /a.txt HTTP/1.1
> User-Agent: curl/7.29.0
> Host: 192.168.99.131:8000
> Accept: */*
> Content-Length: 1
> Content-Type: application/x-www-form-urlencoded
>
* upload completely sent off: 1 out of 1 bytes
< HTTP/1.1 200 OK
< Server: nginx/1.6.3
< Date: Sun, 30 Aug 2015 18:20:03 GMT
< Content-Length: 0
< Connection: keep-alive
< x-amz-id-2: GF/hVpsr+r2R/j7qqUf0=
< x-amz-request-id: F499D02F2D28B6EB
< ETag: "0cc175b9c0f1b6a831c399e269772661"
<
* Connection #0 to host 192.168.99.131 left intact

PUT 更大的文件而不调整 header:

curl -vvvX PUT -d @1408133479717.jpg http://192.168.99.131:8000/a.jpg
* About to connect() to 192.168.99.131 port 8000 (#0)
*   Trying 192.168.99.131...
* Connected to 192.168.99.131 (192.168.99.131) port 8000 (#0)
> PUT /a.jpg HTTP/1.1
> User-Agent: curl/7.29.0
> Host: 192.168.99.131:8000
> Accept: */*
> Content-Length: 320356
> Content-Type: application/x-www-form-urlencoded
> Expect: 100-continue
>
< HTTP/1.1 100 Continue
< HTTP/1.1 200 OK
< Server: nginx/1.6.3
< Date: Sun, 30 Aug 2015 18:33:45 GMT
< Content-Length: 0
< Connection: keep-alive
< x-amz-id-2: im+nMggFzCCp+/mfLdWDr/ZwJFs=
< x-amz-request-id: 3B86E54874009D73
< ETag: "8d04fbe9d623bf03bb828594ca272063"
<
* Connection #0 to host 192.168.99.131 left intact

本次上传部分文件大小为320356字节，原文件为825210字节。

PUT 更大的文件并调整 header:

[vagrant@gripper nginx]$ curl -vvvX PUT -H "Content-Length: 825210" -d @1408133479717.jpg http://192.168.99.131:8000/a.jpg
* About to connect() to 192.168.99.131 port 8000 (#0)
*   Trying 192.168.99.131...
* Connected to 192.168.99.131 (192.168.99.131) port 8000 (#0)
> PUT /a.jpg HTTP/1.1
> User-Agent: curl/7.29.0
> Host: 192.168.99.131:8000
> Accept: */*
> Content-Length: 825210
> Content-Type: application/x-www-form-urlencoded
> Expect: 100-continue
>
< HTTP/1.1 100 Continue
* Empty reply from server
* Connection #0 to host 192.168.99.131 left intact
curl: (52) Empty reply from server

我的问题是：

如何使用 curl 上传正确大小的文件？

Answer 1

听起来您可能运行遇到了 client_max_body_size 的任何问题。也就是说，默认值为 1MB，所以我很惊讶你的 825KB 上传失败，除非你在你的 nginx conf 中修改了这个值。

http://nginx.org/en/docs/http/ngx_http_core_module.html#client_max_body_size

How to edit nginx.conf to increase file size upload

将 PUT HTTP 请求代理到 AWS S3 失败

Proxying PUT HTTP requests to AWS S3 fails

proxy

curl

http

nginx

amazon-s3