You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hadoop.apache.org by Shashi Vishwakarma <sh...@gmail.com> on 2015/09/04 17:36:09 UTC
Performance impact of curl command on linux server
Hi
I have been using curl command for ingesting data into HDFS using WebHDFS.
Admins are saying that resource utilization for curl is very high which is
hampering server performance.
I would like to know if curl can use so much of resource which could lead
to performance issue.
Just to give more idea , I have been ingesting around 25Gb of data into
HDFS. Not to sure if curl can cause so much of issue with such a small data
set.
Any suggestion would be a great help.
Thanks,
Shashi
Re: Performance impact of curl command on linux server
Posted by Harsh J <ha...@cloudera.com>.
That depends - what resource did it consume, concerning your admins? CPU?
Were you uploading using the chunked technique?
On Fri, Sep 4, 2015 at 9:06 PM Shashi Vishwakarma <sh...@gmail.com>
wrote:
> Hi
>
> I have been using curl command for ingesting data into HDFS using WebHDFS.
> Admins are saying that resource utilization for curl is very high which is
> hampering server performance.
>
> I would like to know if curl can use so much of resource which could lead
> to performance issue.
>
> Just to give more idea , I have been ingesting around 25Gb of data into
> HDFS. Not to sure if curl can cause so much of issue with such a small data
> set.
>
> Any suggestion would be a great help.
>
> Thanks,
>
> Shashi
>
Re: Performance impact of curl command on linux server
Posted by Harsh J <ha...@cloudera.com>.
That depends - what resource did it consume, concerning your admins? CPU?
Were you uploading using the chunked technique?
On Fri, Sep 4, 2015 at 9:06 PM Shashi Vishwakarma <sh...@gmail.com>
wrote:
> Hi
>
> I have been using curl command for ingesting data into HDFS using WebHDFS.
> Admins are saying that resource utilization for curl is very high which is
> hampering server performance.
>
> I would like to know if curl can use so much of resource which could lead
> to performance issue.
>
> Just to give more idea , I have been ingesting around 25Gb of data into
> HDFS. Not to sure if curl can cause so much of issue with such a small data
> set.
>
> Any suggestion would be a great help.
>
> Thanks,
>
> Shashi
>
Re: Performance impact of curl command on linux server
Posted by Harsh J <ha...@cloudera.com>.
That depends - what resource did it consume, concerning your admins? CPU?
Were you uploading using the chunked technique?
On Fri, Sep 4, 2015 at 9:06 PM Shashi Vishwakarma <sh...@gmail.com>
wrote:
> Hi
>
> I have been using curl command for ingesting data into HDFS using WebHDFS.
> Admins are saying that resource utilization for curl is very high which is
> hampering server performance.
>
> I would like to know if curl can use so much of resource which could lead
> to performance issue.
>
> Just to give more idea , I have been ingesting around 25Gb of data into
> HDFS. Not to sure if curl can cause so much of issue with such a small data
> set.
>
> Any suggestion would be a great help.
>
> Thanks,
>
> Shashi
>
Re: Performance impact of curl command on linux server
Posted by Harsh J <ha...@cloudera.com>.
That depends - what resource did it consume, concerning your admins? CPU?
Were you uploading using the chunked technique?
On Fri, Sep 4, 2015 at 9:06 PM Shashi Vishwakarma <sh...@gmail.com>
wrote:
> Hi
>
> I have been using curl command for ingesting data into HDFS using WebHDFS.
> Admins are saying that resource utilization for curl is very high which is
> hampering server performance.
>
> I would like to know if curl can use so much of resource which could lead
> to performance issue.
>
> Just to give more idea , I have been ingesting around 25Gb of data into
> HDFS. Not to sure if curl can cause so much of issue with such a small data
> set.
>
> Any suggestion would be a great help.
>
> Thanks,
>
> Shashi
>