What are the typical write throughput to s3 you see

[Henry Larson <ne...@gmail.com>]

I am running my flume (flume 0.9.4-cdh3u4) flow in BE mode and both agent
and collector, that are on separate machines, are large EC2 instances

The collector writes to the local filesystem as well as a S3 bucket in the
same region.

The agent, when configured in E2E mode has been tested to handle 4MB/s and
so is the agent<->collector speed.

Using 'iftop' I notice that there are two outgoing connections by the
collector to S3 and the write throughput to s3 hovers around 40KB/s - 50KB/s

1. What are the typical write throughput to s3 you have seen using the
collector?

2. I was wondering if it would be possible to spawn more of these threads
(that write to S3) to maximize the instance utilization?

How do I do that?

I would rather not spawn more collectors, if I could increase the write
throughput another way.