Go中如何自定义http.Client或http.Transport超时重试？
How to customize http.Client or http.Transport in Go to retry after timeout?
我想为标准 http.Client 实现自定义 http.Transport，如果客户端超时，它将自动重试。
P.S。出于某种原因，自定义 http.Transport 是 必备 。我已经检查过 hashicorp/go-retryablehttp，但是它不允许我使用自己的 http.Transport。
这是我的尝试，自定义组件：
type CustomTransport struct {
http.RoundTripper
// ... private fields
}
func NewCustomTransport(upstream *http.Transport) *CustomTransport {
upstream.TLSClientConfig = &tls.Config{InsecureSkipVerify: true}
// ... other customizations for transport
return &CustomTransport{upstream}
}
func (ct *CustomTransport) RoundTrip(req *http.Request) (resp *http.Response, err error) {
req.Header.Set("Secret", "Blah blah blah")
// ... other customizations for each request
for i := 1; i <= 5; i++ {
resp, err = ct.RoundTripper.RoundTrip(req)
if errors.Is(err, context.DeadlineExceeded) {
log.Warnf("#%d got timeout will retry - %v", i, err)
//time.Sleep(time.Duration(100*i) * time.Millisecond)
continue
} else {
break
}
}
log.Debugf("got final result: %v", err)
return resp, err
}
来电号码：
func main() {
transport := NewCustomTransport(http.DefaultTransport.(*http.Transport))
client := &http.Client{
Timeout: 8 * time.Second,
Transport: transport,
}
apiUrl := "https://httpbin.org/delay/10"
log.Debugf("begin to get %q", apiUrl)
start := time.Now()
resp, err := client.Get(apiUrl)
if err != nil {
log.Warnf("client got error: %v", err)
} else {
defer resp.Body.Close()
}
log.Debugf("end to get %q, time cost: %v", apiUrl, time.Since(start))
if resp != nil {
data, err := httputil.DumpResponse(resp, true)
if err != nil {
log.Warnf("fail to dump resp: %v", err)
}
fmt.Println(string(data))
}
}
我的实现没有按预期工作，一旦客户端超时，重试实际上不会发生。请参阅下面的日志：
2020-07-15T00:53:22.586 DEBUG begin to get "https://httpbin.org/delay/10"
2020-07-15T00:53:30.590 WARN #1 got timeout will retry - context deadline exceeded
2020-07-15T00:53:30.590 WARN #2 got timeout will retry - context deadline exceeded
2020-07-15T00:53:30.590 WARN #3 got timeout will retry - context deadline exceeded
2020-07-15T00:53:30.590 WARN #4 got timeout will retry - context deadline exceeded
2020-07-15T00:53:30.590 WARN #5 got timeout will retry - context deadline exceeded
2020-07-15T00:53:30.590 DEBUG got final result: context deadline exceeded
2020-07-15T00:53:30.590 WARN client got error: Get "https://httpbin.org/delay/10": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
2020-07-15T00:53:30.590 DEBUG end to get "https://httpbin.org/delay/10", time cost: 8.004182786s
你能告诉我如何解决这个问题，或者任何 methods/ideas 来实现这样的 http.Client 吗？
不需要自定义http.Client之类的东西。您可以简单地将获取操作包装到重试中——有很多可用的模块可以做到这一点：
package main
import (
"io"
"log"
"net/http"
"os"
"time"
"github.com/avast/retry-go"
)
func main() {
r, err := fetchDataWithRetries("http://nonexistant.example.com")
if err != nil {
log.Printf("Error fetching data: %s", err)
os.Exit(1)
}
defer r.Body.Close()
io.Copy(os.Stdout, r.Body)
}
// fetchDataWithRetries is your wrapped retrieval.
// It works with a static configuration for the retries,
// but obviously, you can generalize this function further.
func fetchDataWithRetries(url string) (r *http.Response, err error) {
retry.Do(
// The actual function that does "stuff"
func() error {
log.Printf("Retrieving data from '%s'", url)
r, err = http.Get(url)
return err
},
// A function to decide whether you actually want to
// retry or not. In this case, it would make sense
// to actually stop retrying, since the host does not exist.
// Return true if you want to retry, false if not.
retry.RetryIf(
func(error) bool {
log.Printf("Retrieving data: %s", err)
log.Printf("Deciding whether to retry")
return true
}),
retry.OnRetry(func(try uint, orig error) {
log.Printf("Retrying to fetch data. Try: %d", try+2)
}),
retry.Attempts(3),
// Basically, we are setting up a delay
// which randoms between 2 and 4 seconds.
retry.Delay(3*time.Second),
retry.MaxJitter(1*time.Second),
)
return
}
Note that the Timeout field of http.Client is more or less obsolete. Best practice now is to use http.Request.Context() for timeouts. – Flimzy
感谢@Flimzy 的启发！我试图使用上下文代替 http.Client 方式进行超时控制。这是代码：
func (ct *CustomTransport) RoundTrip(req *http.Request) (resp *http.Response, err error) {
req.Header.Set("Secret", "Blah blah blah")
// ... other customizations for each request
for i := 1; i <= 5; i++ {
ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
defer cancel()
//reqT := req.WithContext(ctx)
resp, err = ct.RoundTripper.RoundTrip(req.WithContext(ctx))
if errors.Is(err, context.DeadlineExceeded) {
log.Warnf("#%d got timeout will retry - %v", i, err)
//time.Sleep(time.Duration(100*i) * time.Millisecond)
continue
} else {
break
}
}
根据日志，它有效（注意日志中的时间戳，它实际上重试了）：
2020-07-16T00:06:12.788+0800 DEBUG begin to get "https://httpbin.org/delay/10"
2020-07-16T00:06:20.794+0800 WARN #1 got timeout will retry - context deadline exceeded
2020-07-16T00:06:28.794+0800 WARN #2 got timeout will retry - context deadline exceeded
2020-07-16T00:06:36.799+0800 WARN #3 got timeout will retry - context deadline exceeded
2020-07-16T00:06:44.803+0800 WARN #4 got timeout will retry - context deadline exceeded
2020-07-16T00:06:52.809+0800 WARN #5 got timeout will retry - context deadline exceeded
2020-07-16T00:06:52.809+0800 DEBUG got final result: context deadline exceeded
2020-07-16T00:06:52.809+0800 WARN client got error: Get "https://httpbin.org/delay/10": context deadline exceeded
2020-07-16T00:06:52.809+0800 DEBUG end to get "https://httpbin.org/delay/10", time cost: 40.019334668s
我想为标准 http.Client 实现自定义 http.Transport，如果客户端超时，它将自动重试。
P.S。出于某种原因，自定义 http.Transport 是 必备 。我已经检查过 hashicorp/go-retryablehttp，但是它不允许我使用自己的 http.Transport。
这是我的尝试，自定义组件：
type CustomTransport struct {
http.RoundTripper
// ... private fields
}
func NewCustomTransport(upstream *http.Transport) *CustomTransport {
upstream.TLSClientConfig = &tls.Config{InsecureSkipVerify: true}
// ... other customizations for transport
return &CustomTransport{upstream}
}
func (ct *CustomTransport) RoundTrip(req *http.Request) (resp *http.Response, err error) {
req.Header.Set("Secret", "Blah blah blah")
// ... other customizations for each request
for i := 1; i <= 5; i++ {
resp, err = ct.RoundTripper.RoundTrip(req)
if errors.Is(err, context.DeadlineExceeded) {
log.Warnf("#%d got timeout will retry - %v", i, err)
//time.Sleep(time.Duration(100*i) * time.Millisecond)
continue
} else {
break
}
}
log.Debugf("got final result: %v", err)
return resp, err
}
来电号码：
func main() {
transport := NewCustomTransport(http.DefaultTransport.(*http.Transport))
client := &http.Client{
Timeout: 8 * time.Second,
Transport: transport,
}
apiUrl := "https://httpbin.org/delay/10"
log.Debugf("begin to get %q", apiUrl)
start := time.Now()
resp, err := client.Get(apiUrl)
if err != nil {
log.Warnf("client got error: %v", err)
} else {
defer resp.Body.Close()
}
log.Debugf("end to get %q, time cost: %v", apiUrl, time.Since(start))
if resp != nil {
data, err := httputil.DumpResponse(resp, true)
if err != nil {
log.Warnf("fail to dump resp: %v", err)
}
fmt.Println(string(data))
}
}
我的实现没有按预期工作，一旦客户端超时，重试实际上不会发生。请参阅下面的日志：
2020-07-15T00:53:22.586 DEBUG begin to get "https://httpbin.org/delay/10"
2020-07-15T00:53:30.590 WARN #1 got timeout will retry - context deadline exceeded
2020-07-15T00:53:30.590 WARN #2 got timeout will retry - context deadline exceeded
2020-07-15T00:53:30.590 WARN #3 got timeout will retry - context deadline exceeded
2020-07-15T00:53:30.590 WARN #4 got timeout will retry - context deadline exceeded
2020-07-15T00:53:30.590 WARN #5 got timeout will retry - context deadline exceeded
2020-07-15T00:53:30.590 DEBUG got final result: context deadline exceeded
2020-07-15T00:53:30.590 WARN client got error: Get "https://httpbin.org/delay/10": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
2020-07-15T00:53:30.590 DEBUG end to get "https://httpbin.org/delay/10", time cost: 8.004182786s
你能告诉我如何解决这个问题，或者任何 methods/ideas 来实现这样的 http.Client 吗？
不需要自定义http.Client之类的东西。您可以简单地将获取操作包装到重试中——有很多可用的模块可以做到这一点：
package main
import (
"io"
"log"
"net/http"
"os"
"time"
"github.com/avast/retry-go"
)
func main() {
r, err := fetchDataWithRetries("http://nonexistant.example.com")
if err != nil {
log.Printf("Error fetching data: %s", err)
os.Exit(1)
}
defer r.Body.Close()
io.Copy(os.Stdout, r.Body)
}
// fetchDataWithRetries is your wrapped retrieval.
// It works with a static configuration for the retries,
// but obviously, you can generalize this function further.
func fetchDataWithRetries(url string) (r *http.Response, err error) {
retry.Do(
// The actual function that does "stuff"
func() error {
log.Printf("Retrieving data from '%s'", url)
r, err = http.Get(url)
return err
},
// A function to decide whether you actually want to
// retry or not. In this case, it would make sense
// to actually stop retrying, since the host does not exist.
// Return true if you want to retry, false if not.
retry.RetryIf(
func(error) bool {
log.Printf("Retrieving data: %s", err)
log.Printf("Deciding whether to retry")
return true
}),
retry.OnRetry(func(try uint, orig error) {
log.Printf("Retrying to fetch data. Try: %d", try+2)
}),
retry.Attempts(3),
// Basically, we are setting up a delay
// which randoms between 2 and 4 seconds.
retry.Delay(3*time.Second),
retry.MaxJitter(1*time.Second),
)
return
}
Note that the Timeout field of http.Client is more or less obsolete. Best practice now is to use http.Request.Context() for timeouts. – Flimzy
感谢@Flimzy 的启发！我试图使用上下文代替 http.Client 方式进行超时控制。这是代码：
func (ct *CustomTransport) RoundTrip(req *http.Request) (resp *http.Response, err error) {
req.Header.Set("Secret", "Blah blah blah")
// ... other customizations for each request
for i := 1; i <= 5; i++ {
ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
defer cancel()
//reqT := req.WithContext(ctx)
resp, err = ct.RoundTripper.RoundTrip(req.WithContext(ctx))
if errors.Is(err, context.DeadlineExceeded) {
log.Warnf("#%d got timeout will retry - %v", i, err)
//time.Sleep(time.Duration(100*i) * time.Millisecond)
continue
} else {
break
}
}
根据日志，它有效（注意日志中的时间戳，它实际上重试了）：
2020-07-16T00:06:12.788+0800 DEBUG begin to get "https://httpbin.org/delay/10"
2020-07-16T00:06:20.794+0800 WARN #1 got timeout will retry - context deadline exceeded
2020-07-16T00:06:28.794+0800 WARN #2 got timeout will retry - context deadline exceeded
2020-07-16T00:06:36.799+0800 WARN #3 got timeout will retry - context deadline exceeded
2020-07-16T00:06:44.803+0800 WARN #4 got timeout will retry - context deadline exceeded
2020-07-16T00:06:52.809+0800 WARN #5 got timeout will retry - context deadline exceeded
2020-07-16T00:06:52.809+0800 DEBUG got final result: context deadline exceeded
2020-07-16T00:06:52.809+0800 WARN client got error: Get "https://httpbin.org/delay/10": context deadline exceeded
2020-07-16T00:06:52.809+0800 DEBUG end to get "https://httpbin.org/delay/10", time cost: 40.019334668s