AWS Kinesis 吞吐量低
Low Throughput of AWS Kinesis
我用python的boto.kinesis module to write records to AWS Kinesis. The maximum throughput that is reached is about 40 puts/sec. However, according to the Kinesis FAQ:
Each shard can support up to 1000 PUT records per second.
所以我目前的方法只达到了理论上可能的 4%，这似乎低得离谱。
有没有人知道如何提高吞吐量？
设置：Kinesis Stream 是一个具有一个分片的实例。生产者位于与 Kinesis Stream 相同区域的专用 AWS EC2 实例 (t3.medium) 上。它创建大约 20 个字符长度的字符串，并通过 boto.kinesis.Connection.put_record("my_stream", my_message).
将它们发送到 Kinesis Stream
简化代码：
from boto import kinesis
import time
connection = kinesis.connect_to_region(REGION)
stream = connection.create_stream("my_stream", shard_count=1)
time.sleep(60) # wait a minute until stream is created
for i in range(NUM_MESSAGES):
my_message = "This is message %d" % i
connection.put_record(my_message, "my_stream", "partition_key")
http://docs.aws.amazon.com/kinesis/latest/dev/service-sizes-and-limits.html
限额是 records/second
您应该使用 putRecords 来提高写入吞吐量。你这样做的方式是在同一个调用中放置多个记录。所以你继续追加，最后你做 put 记录。
我用python的boto.kinesis module to write records to AWS Kinesis. The maximum throughput that is reached is about 40 puts/sec. However, according to the Kinesis FAQ:
Each shard can support up to 1000 PUT records per second.
所以我目前的方法只达到了理论上可能的 4%，这似乎低得离谱。
有没有人知道如何提高吞吐量？
设置：Kinesis Stream 是一个具有一个分片的实例。生产者位于与 Kinesis Stream 相同区域的专用 AWS EC2 实例 (t3.medium) 上。它创建大约 20 个字符长度的字符串，并通过 boto.kinesis.Connection.put_record("my_stream", my_message).
简化代码：
from boto import kinesis
import time
connection = kinesis.connect_to_region(REGION)
stream = connection.create_stream("my_stream", shard_count=1)
time.sleep(60) # wait a minute until stream is created
for i in range(NUM_MESSAGES):
my_message = "This is message %d" % i
connection.put_record(my_message, "my_stream", "partition_key")
http://docs.aws.amazon.com/kinesis/latest/dev/service-sizes-and-limits.html
限额是 records/second 您应该使用 putRecords 来提高写入吞吐量。你这样做的方式是在同一个调用中放置多个记录。所以你继续追加，最后你做 put 记录。