You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@nifi.apache.org by pradeepbill <pr...@gmail.com> on 2017/04/25 20:41:17 UTC

puts3object super slow

Hello folks, 
I have a listenSyslogProcessor -> puts3object processor, and our network
bandwidth is 1GBPS, but I see a super slow performance on the s3 writes,our
s3 bucket has no policy, here are the stats

Total Size written – 17MB
Total Time for Queue to empty – 21 mins
Min/Max/Mean – 0.0/9.29MB/1.08MB

Its absolutely unusable, I dont know where the slowness is , either the
puts3processor api?, or network ? , or environment ? .Where can I debug, the
nifi-app.log has no errors/exceptions.Please advice.

Thanks
Pradeep



--
View this message in context: http://apache-nifi-developer-list.39713.n7.nabble.com/puts3object-super-slow-tp15554.html
Sent from the Apache NiFi Developer List mailing list archive at Nabble.com.

Re: puts3object super slow

Posted by pradeepbill <pr...@gmail.com>.

thanks Bryan, i took your hint, and looks like the network might be the
culprit.I did a test and 95M took almost an hour.



--
View this message in context: http://apache-nifi-developer-list.39713.n7.nabble.com/puts3object-super-slow-tp15554p15556.html
Sent from the Apache NiFi Developer List mailing list archive at Nabble.com.

Re: puts3object super slow

Posted by Bryan Rosander <br...@apache.org>.

Hi Pradeep,

How does the upload speed compare to uploading without NiFi?  Maybe try the
enhanced uploader [1] to the same bucket from the same machine to see if
the problem is internal to NiFi or maybe due to something else?

[1] http://docs.aws.amazon.com/AmazonS3/latest/UG/enhanced-uploader.html

Thanks,
Bryan

On Tue, Apr 25, 2017 at 4:41 PM, pradeepbill <pr...@gmail.com> wrote:

> Hello folks,
> I have a listenSyslogProcessor -> puts3object processor, and our network
> bandwidth is 1GBPS, but I see a super slow performance on the s3 writes,our
> s3 bucket has no policy, here are the stats
>
> Total Size written – 17MB
> Total Time for Queue to empty – 21 mins
> Min/Max/Mean – 0.0/9.29MB/1.08MB
>
> Its absolutely unusable, I dont know where the slowness is , either the
> puts3processor api?, or network ? , or environment ? .Where can I debug,
> the
> nifi-app.log has no errors/exceptions.Please advice.
>
> Thanks
> Pradeep
>
>
>
> --
> View this message in context: http://apache-nifi-developer-
> list.39713.n7.nabble.com/puts3object-super-slow-tp15554.html
> Sent from the Apache NiFi Developer List mailing list archive at
> Nabble.com.
>