You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@accumulo.apache.org by Wyatt Frelot <Wy...@altamiracorp.com> on 2015/02/06 21:05:45 UTC

ACCUMULO VAGRANT CLUSTER

Good day all,

I have completed the development of a Vagrant Accumulo Cluster. It automatically installs and configures Hadoop 2.4.1, Accumulo 1.6.1, and ZooKeeper 3.4.6 with "basic" settings. I tried to designed it being able to update the cluster with minimum effort. Hopefully that is the case.

Cluster Information: Vagrant cluster initializes 4 nodes w/ 1.5GB of RAM on a private network. If you look in the “Vagrantfile", I believe this information would be easy to see and adjust if needed. All of the node related info is located on under the “Additional Information” header of the link below. Nodes use typical vagrant usernames and passwords.

Setup Instructions: http://wjfrelo.github.io/AccumuloCluster/
I included as much information as I thought would be useful. Feel free to provide feedback on the value of the instructions.

Testing: I have been able to get it to work with minimum adjustments on Linux Platforms (Ubuntu, CentOS), Window 7 & 8, and Mac OSX. Main problems have occurred on window platforms (ssh and virtualbox path issues. Workarounds are identified in instructions)

Hope it is helpful and useful.


--
Wyatt J. Frelot

Re: hdfs cpu usage

Posted by Ara Ebrahimi <ar...@argyledata.com>.

Hi,

On all nodes. Nope, this is the behavior we’ve been seeing since we started testing a few weeks ago. Yes the behavior persists after restarts.

Ara.

On Feb 7, 2015, at 10:50 AM, Keith Turner <ke...@deenlo.com>> wrote:

Is this happening on just one node, or all nodes? Have you run w/o problem on google cloud env before? Do you see the problem if you restart the vms?

On Fri, Feb 6, 2015 at 9:34 PM, Ara Ebrahimi <ar...@argyledata.com>> wrote:
Hi,

We’re seeing some weird behavior from the hdfs daemon on google cloud environment when we use accumulo Scanner to sequentially scan a table. Top reports 200-300% cpu usage for the hdfs daemon. Accumulo is also around 500%. iostat %util is low. avgrq-sz is low, rMB/s is low, there’s lots of free memory. It seems like something causes the hdfs daemon to consume a lot of cpu and not to send enough read requests to the disk (ssd actually, so disk is super fast and vastly under-utilized). The process which sends scan requests to accumulo is 500% active (using 3 query batch threads and aggressive scan-batch-size/read-ahead-threashold values). So it seems like somehow hdfs is the bottleneck. On another cluster we rarely see hdfs daemon going over 10% cpu usage. Any idea what the issue could be?

Thanks,
Ara.

________________________________

This message is for the designated recipient only and may contain privileged, proprietary, or otherwise confidential information. If you have received it in error, please notify the sender immediately and delete the original. Any other use of the e-mail by you is prohibited. Thank you in advance for your cooperation.

________________________________

Re: hdfs cpu usage

Posted by Keith Turner <ke...@deenlo.com>.

Is this happening on just one node, or all nodes?  Have you run w/o problem
on google cloud env before?  Do you see the problem if you restart the vms?

On Fri, Feb 6, 2015 at 9:34 PM, Ara Ebrahimi <ar...@argyledata.com>
wrote:

> Hi,
>
> We’re seeing some weird behavior from the hdfs daemon on google cloud
> environment when we use accumulo Scanner to sequentially scan a table. Top
> reports 200-300% cpu usage for the hdfs daemon. Accumulo is also around
> 500%. iostat %util is low. avgrq-sz is low, rMB/s is low, there’s lots of
> free memory. It seems like something causes the hdfs daemon to consume a
> lot of cpu and not to send enough read requests to the disk (ssd actually,
> so disk is super fast and vastly under-utilized). The process which sends
> scan requests to accumulo is 500% active (using 3 query batch threads and
> aggressive scan-batch-size/read-ahead-threashold values). So it seems like
> somehow hdfs is the bottleneck. On another cluster we rarely see hdfs
> daemon going over 10% cpu usage. Any idea what the issue could be?
>
> Thanks,
> Ara.
>
>
>
> ________________________________
>
> This message is for the designated recipient only and may contain
> privileged, proprietary, or otherwise confidential information. If you have
> received it in error, please notify the sender immediately and delete the
> original. Any other use of the e-mail by you is prohibited. Thank you in
> advance for your cooperation.
>
> ________________________________
>

Re: hdfs cpu usage

Posted by Ara Ebrahimi <ar...@argyledata.com>.

Nope. This is for a full table scan. The same config in our in-premise cluster performs well while on google cloud we see this weird hdfs cpu usage issue.

Ara.

> On Feb 9, 2015, at 9:31 AM, Adam Fuchs <af...@apache.org> wrote:
>
> Ara,
>
> What kind of query load are you generating within your batch scanners?
> Are you using an iterator that seeks around a lot? Are you grabbing
> many small batches (only a few keys per range) from the batch scanner?
> As a wild guess, this could be the result of lots of seeks with a low
> cache hit rate, which would induce CPU load in HDFS fetching blocks
> and CPU load in Accumulo decrypting/decompressing those blocks. The
> monitor page will show you seek rates and cache hit rates.
>
> Adam
>
>
> On Sat, Feb 7, 2015 at 8:48 PM, Ara Ebrahimi
> <ar...@argyledata.com> wrote:
>> 2.4.0.2.1.
>>
>> Yeah seems like I need to do that. I was hoping I’d get some advice based on
>> prior experience with google cloud environment.
>>
>> Ara.
>>
>> On Feb 7, 2015, at 11:23 AM, Josh Elser <jo...@gmail.com> wrote:
>>
>> What version of Hadoop are you using?
>>
>> Have you considered hooking up a profiler to the Datanode on GCE to see
>> where the time is being spent? That might help shed some light on the
>> situation.
>>
>> Ara Ebrahimi wrote:
>>
>> Hi,
>>
>> We’re seeing some weird behavior from the hdfs daemon on google cloud
>> environment when we use accumulo Scanner to sequentially scan a table. Top
>> reports 200-300% cpu usage for the hdfs daemon. Accumulo is also around
>> 500%. iostat %util is low. avgrq-sz is low, rMB/s is low, there’s lots of
>> free memory. It seems like something causes the hdfs daemon to consume a lot
>> of cpu and not to send enough read requests to the disk (ssd actually, so
>> disk is super fast and vastly under-utilized). The process which sends scan
>> requests to accumulo is 500% active (using 3 query batch threads and
>> aggressive scan-batch-size/read-ahead-threashold values). So it seems like
>> somehow hdfs is the bottleneck. On another cluster we rarely see hdfs daemon
>> going over 10% cpu usage. Any idea what the issue could be?
>>
>> Thanks,
>> Ara.
>>
>>
>>
>> ________________________________
>>
>> This message is for the designated recipient only and may contain
>> privileged, proprietary, or otherwise confidential information. If you have
>> received it in error, please notify the sender immediately and delete the
>> original. Any other use of the e-mail by you is prohibited. Thank you in
>> advance for your cooperation.
>>
>> ________________________________
>>
>>
>>
>>
>> ________________________________
>>
>> This message is for the designated recipient only and may contain
>> privileged, proprietary, or otherwise confidential information. If you have
>> received it in error, please notify the sender immediately and delete the
>> original. Any other use of the e-mail by you is prohibited. Thank you in
>> advance for your cooperation.
>>
>> ________________________________
>>
>>
>>
>>
>>
>> ________________________________
>>
>> This message is for the designated recipient only and may contain
>> privileged, proprietary, or otherwise confidential information. If you have
>> received it in error, please notify the sender immediately and delete the
>> original. Any other use of the e-mail by you is prohibited. Thank you in
>> advance for your cooperation.
>>
>> ________________________________
>
>
>
> ________________________________
>
> This message is for the designated recipient only and may contain privileged, proprietary, or otherwise confidential information. If you have received it in error, please notify the sender immediately and delete the original. Any other use of the e-mail by you is prohibited. Thank you in advance for your cooperation.
>
> ________________________________




________________________________

This message is for the designated recipient only and may contain privileged, proprietary, or otherwise confidential information. If you have received it in error, please notify the sender immediately and delete the original. Any other use of the e-mail by you is prohibited. Thank you in advance for your cooperation.

________________________________

Re: hdfs cpu usage

Posted by Adam Fuchs <af...@apache.org>.

Ara,

What kind of query load are you generating within your batch scanners?
Are you using an iterator that seeks around a lot? Are you grabbing
many small batches (only a few keys per range) from the batch scanner?
As a wild guess, this could be the result of lots of seeks with a low
cache hit rate, which would induce CPU load in HDFS fetching blocks
and CPU load in Accumulo decrypting/decompressing those blocks. The
monitor page will show you seek rates and cache hit rates.

Adam


On Sat, Feb 7, 2015 at 8:48 PM, Ara Ebrahimi
<ar...@argyledata.com> wrote:
> 2.4.0.2.1.
>
> Yeah seems like I need to do that. I was hoping I’d get some advice based on
> prior experience with google cloud environment.
>
> Ara.
>
> On Feb 7, 2015, at 11:23 AM, Josh Elser <jo...@gmail.com> wrote:
>
> What version of Hadoop are you using?
>
> Have you considered hooking up a profiler to the Datanode on GCE to see
> where the time is being spent? That might help shed some light on the
> situation.
>
> Ara Ebrahimi wrote:
>
> Hi,
>
> We’re seeing some weird behavior from the hdfs daemon on google cloud
> environment when we use accumulo Scanner to sequentially scan a table. Top
> reports 200-300% cpu usage for the hdfs daemon. Accumulo is also around
> 500%. iostat %util is low. avgrq-sz is low, rMB/s is low, there’s lots of
> free memory. It seems like something causes the hdfs daemon to consume a lot
> of cpu and not to send enough read requests to the disk (ssd actually, so
> disk is super fast and vastly under-utilized). The process which sends scan
> requests to accumulo is 500% active (using 3 query batch threads and
> aggressive scan-batch-size/read-ahead-threashold values). So it seems like
> somehow hdfs is the bottleneck. On another cluster we rarely see hdfs daemon
> going over 10% cpu usage. Any idea what the issue could be?
>
> Thanks,
> Ara.
>
>
>
> ________________________________
>
> This message is for the designated recipient only and may contain
> privileged, proprietary, or otherwise confidential information. If you have
> received it in error, please notify the sender immediately and delete the
> original. Any other use of the e-mail by you is prohibited. Thank you in
> advance for your cooperation.
>
> ________________________________
>
>
>
>
> ________________________________
>
> This message is for the designated recipient only and may contain
> privileged, proprietary, or otherwise confidential information. If you have
> received it in error, please notify the sender immediately and delete the
> original. Any other use of the e-mail by you is prohibited. Thank you in
> advance for your cooperation.
>
> ________________________________
>
>
>
>
>
> ________________________________
>
> This message is for the designated recipient only and may contain
> privileged, proprietary, or otherwise confidential information. If you have
> received it in error, please notify the sender immediately and delete the
> original. Any other use of the e-mail by you is prohibited. Thank you in
> advance for your cooperation.
>
> ________________________________

Re: hdfs cpu usage

Posted by Ara Ebrahimi <ar...@argyledata.com>.

2.4.0.2.1.

Yeah seems like I need to do that. I was hoping I’d get some advice based on prior experience with google cloud environment.

Ara.

On Feb 7, 2015, at 11:23 AM, Josh Elser <jo...@gmail.com>> wrote:

What version of Hadoop are you using?

Have you considered hooking up a profiler to the Datanode on GCE to see
where the time is being spent? That might help shed some light on the
situation.

Ara Ebrahimi wrote:
Hi,

Thanks,
Ara.

________________________________

Re: hdfs cpu usage

Posted by Josh Elser <jo...@gmail.com>.

What version of Hadoop are you using?

Have you considered hooking up a profiler to the Datanode on GCE to see 
where the time is being spent? That might help shed some light on the 
situation.

Ara Ebrahimi wrote:
> Hi,
>
> We’re seeing some weird behavior from the hdfs daemon on google cloud environment when we use accumulo Scanner to sequentially scan a table. Top reports 200-300% cpu usage for the hdfs daemon. Accumulo is also around 500%. iostat %util is low. avgrq-sz is low, rMB/s is low, there’s lots of free memory. It seems like something causes the hdfs daemon to consume a lot of cpu and not to send enough read requests to the disk (ssd actually, so disk is super fast and vastly under-utilized). The process which sends scan requests to accumulo is 500% active (using 3 query batch threads and aggressive scan-batch-size/read-ahead-threashold values). So it seems like somehow hdfs is the bottleneck. On another cluster we rarely see hdfs daemon going over 10% cpu usage. Any idea what the issue could be?
>
> Thanks,
> Ara.
>
>
>
> ________________________________
>
> This message is for the designated recipient only and may contain privileged, proprietary, or otherwise confidential information. If you have received it in error, please notify the sender immediately and delete the original. Any other use of the e-mail by you is prohibited. Thank you in advance for your cooperation.
>
> ________________________________

hdfs cpu usage

Posted by Ara Ebrahimi <ar...@argyledata.com>.

Hi,

We’re seeing some weird behavior from the hdfs daemon on google cloud environment when we use accumulo Scanner to sequentially scan a table. Top reports 200-300% cpu usage for the hdfs daemon. Accumulo is also around 500%. iostat %util is low. avgrq-sz is low, rMB/s is low, there’s lots of free memory. It seems like something causes the hdfs daemon to consume a lot of cpu and not to send enough read requests to the disk (ssd actually, so disk is super fast and vastly under-utilized). The process which sends scan requests to accumulo is 500% active (using 3 query batch threads and aggressive scan-batch-size/read-ahead-threashold values). So it seems like somehow hdfs is the bottleneck. On another cluster we rarely see hdfs daemon going over 10% cpu usage. Any idea what the issue could be?

Thanks,
Ara.



________________________________

This message is for the designated recipient only and may contain privileged, proprietary, or otherwise confidential information. If you have received it in error, please notify the sender immediately and delete the original. Any other use of the e-mail by you is prohibited. Thank you in advance for your cooperation.

________________________________

Re: ACCUMULO VAGRANT CLUSTER

Posted by Wyatt Frelot <Wy...@altamiracorp.com>.

Josh,

That is ultimately my goal. Would love some collaboration and  feedback to further develop it prior to release. If ultimately, we believe that it is good we can move it in at that point.

Wyatt
> On Feb 6, 2015, at 15:49, Josh Elser <jo...@gmail.com> wrote:
> 
> Cool stuff, Wyatt. Thanks for sharing.
> 
> Any interest in contributing this upstream to Accumulo?
> 
> https://issues.apache.org/jira/browse/ACCUMULO-3560
> 
> I know David M has also done some work with Vagrant in the past, so maybe we could get some collaboration in place. I've only hacked on it to date.
> 
> Wyatt Frelot wrote:
>> Good day all,
>> 
>> I have completed the development of a Vagrant Accumulo Cluster. It
>> automatically installs and configures Hadoop 2.4.1, Accumulo 1.6.1, and
>> ZooKeeper 3.4.6 with "basic" settings. I tried to designed it being able
>> to update the cluster with minimum effort. Hopefully that is the case.
>> 
>> Cluster Information: Vagrant cluster initializes 4 nodes w/ 1.5GB of RAM
>> on a private network. If you look in the “Vagrantfile", I believe this
>> information would be easy to see and adjust if needed. All of the node
>> related info is located on under the “Additional Information” header of
>> the link below. Nodes use typical vagrant usernames and passwords.
>> 
>> Setup Instructions: http://wjfrelo.github.io/AccumuloCluster/
>> I included as much information as I thought would be useful. Feel free
>> to provide feedback on the value of the instructions.
>> 
>> Testing: I have been able to get it to work with minimum adjustments on
>> Linux Platforms (Ubuntu, CentOS), Window 7 & 8, and Mac OSX. Main
>> problems have occurred on window platforms (ssh and virtualbox path
>> issues. Workarounds are identified in instructions)
>> 
>> Hope it is helpful and useful.
>> 
>> 
>> --
>> Wyatt J. Frelot

Re: ACCUMULO VAGRANT CLUSTER

Posted by Josh Elser <jo...@gmail.com>.

Cool stuff, Wyatt. Thanks for sharing.

Any interest in contributing this upstream to Accumulo?

https://issues.apache.org/jira/browse/ACCUMULO-3560

I know David M has also done some work with Vagrant in the past, so 
maybe we could get some collaboration in place. I've only hacked on it 
to date.

Wyatt Frelot wrote:
> Good day all,
>
> I have completed the development of a Vagrant Accumulo Cluster. It
> automatically installs and configures Hadoop 2.4.1, Accumulo 1.6.1, and
> ZooKeeper 3.4.6 with "basic" settings. I tried to designed it being able
> to update the cluster with minimum effort. Hopefully that is the case.
>
> Cluster Information: Vagrant cluster initializes 4 nodes w/ 1.5GB of RAM
> on a private network. If you look in the “Vagrantfile", I believe this
> information would be easy to see and adjust if needed. All of the node
> related info is located on under the “Additional Information” header of
> the link below. Nodes use typical vagrant usernames and passwords.
>
> Setup Instructions: http://wjfrelo.github.io/AccumuloCluster/
> I included as much information as I thought would be useful. Feel free
> to provide feedback on the value of the instructions.
>
> Testing: I have been able to get it to work with minimum adjustments on
> Linux Platforms (Ubuntu, CentOS), Window 7 & 8, and Mac OSX. Main
> problems have occurred on window platforms (ssh and virtualbox path
> issues. Workarounds are identified in instructions)
>
> Hope it is helpful and useful.
>
>
> --
> Wyatt J. Frelot

Re: ACCUMULO VAGRANT CLUSTER

Posted by Wyatt Frelot <Wy...@altamiracorp.com>.

Mike,

I have thought about Docker. But, my use case requires a lower learning a this point. I thought Docker would require the users to go a little deeper than what I would prefer. With Vagrant, I didn’t think the learning curve was as steep.

No, I didn’t see the docker build you mentioned, but I will definitely pay attention to it form now on. Thanks!

Wyatt
On Feb 6, 2015, at 15:12, Mike Atlas <mi...@weft.io>> wrote:

Thanks - any chance you might consider doing this as docker containers?

Also, your github page does not link to your repo correctly, it should be https://github.com/wjfrelo/vagrantAccumuloCluster instead of https://github.com/wjfrelo/AccumuloCluster

Have you seen Steffen Roegner's docker builds (and Salt formulae) for Accumulo? https://github.com/sroegner/docker-builds

On Fri, Feb 6, 2015 at 3:05 PM, Wyatt Frelot <Wy...@altamiracorp.com>> wrote:
Good day all,

I have completed the development of a Vagrant Accumulo Cluster. It automatically installs and configures Hadoop 2.4.1, Accumulo 1.6.1, and ZooKeeper 3.4.6 with "basic" settings. I tried to designed it being able to update the cluster with minimum effort. Hopefully that is the case.

Cluster Information: Vagrant cluster initializes 4 nodes w/ 1.5GB of RAM on a private network. If you look in the “Vagrantfile", I believe this information would be easy to see and adjust if needed. All of the node related info is located on under the “Additional Information” header of the link below. Nodes use typical vagrant usernames and passwords.

Setup Instructions: http://wjfrelo.github.io/AccumuloCluster/
I included as much information as I thought would be useful. Feel free to provide feedback on the value of the instructions.

Testing: I have been able to get it to work with minimum adjustments on Linux Platforms (Ubuntu, CentOS), Window 7 & 8, and Mac OSX. Main problems have occurred on window platforms (ssh and virtualbox path issues. Workarounds are identified in instructions)

Hope it is helpful and useful.

--
Wyatt J. Frelot

Re: ACCUMULO VAGRANT CLUSTER

Posted by Mike Atlas <mi...@weft.io>.

Thanks - any chance you might consider doing this as docker containers?

Also, your github page does not link to your repo correctly, it should be
https://github.com/wjfrelo/vagrantAccumuloCluster instead of
https://github.com/wjfrelo/AccumuloCluster

Have you seen Steffen Roegner's docker builds (and Salt formulae) for
Accumulo? https://github.com/sroegner/docker-builds



On Fri, Feb 6, 2015 at 3:05 PM, Wyatt Frelot <Wy...@altamiracorp.com>
wrote:

>  Good day all,
>
>  I have completed the development of a Vagrant Accumulo Cluster. It
> automatically installs and configures Hadoop 2.4.1, Accumulo 1.6.1, and
> ZooKeeper 3.4.6 with "basic" settings. I tried to designed it being able to
> update the cluster with minimum effort. Hopefully that is the case.
>
> Cluster Information: Vagrant cluster initializes 4 nodes w/ 1.5GB of RAM
> on a private network. If you look in the “Vagrantfile", I believe this
> information would be easy to see and adjust if needed. All of the node
> related info is located on under the “Additional Information” header of the
> link below. Nodes use typical vagrant usernames and passwords.
>
> Setup Instructions: http://wjfrelo.github.io/AccumuloCluster/
> I included as much information as I thought would be useful. Feel free to
> provide feedback on the value of the instructions.
>
> Testing: I have been able to get it to work with minimum adjustments on
> Linux Platforms (Ubuntu, CentOS), Window 7 & 8, and Mac OSX. Main problems
> have occurred on window platforms (ssh and virtualbox path issues.
> Workarounds are identified in instructions)
>
> Hope it is helpful and useful.
>
>
> --
> Wyatt J. Frelot
>