Re: Accumulo with Isilon

[Donald Miner <dm...@clearedgeit.com>]

Disclaimer: I worked for EMC/Pivotal and have personally worked on a few
Isilon/HDFS systems and am an overall fan of the technology.

To my knowledge, nobody has done this... but it is something I have been
wanting to see. I think there are some situations where it could really
speed up performance of Accumulo (or HBase, for that matter). First, the
throughput of Isilon for a single file is faster than a single harddrive's
throughput. So, for example, I imagine compaction might be faster if you
don't run into a bad network bottleneck. Individual small scans might be
faster as well, since you are only reading a few files. I think larger
scale things like MapReduce or many compactions at once might end up being
slower. Either way, this is all speculation, which is why I've always
wanted to see it done.

For those of you that have not heard of Isilon/HDFS, I think it is a pretty
awesome solution. It basically takes Isilon's scale out NAS and puts a
protocol facade on top so that it looks like HDFS over the wire. That is,
HDFS clients communicate with it as if there was a namenode or data nodes
that existed, but in reality, Isilon is storing data the way it wants to
store it. For example, hypothetically you could point Accumulo to the
Isilon facade NameNode and Accumulo would know no difference. One really
cool thing is just because you are using Isilon/HDFS, doesn't mean the core
functionality of Isilon goes away. You can ingest (or modify) data via NFS
or CIFS and have the data magically appear in HDFS. More
information/marketing is here
http://www.emc.com/big-data/scale-out-storage-hadoop.htm

I hope some others on here might have some relevant thoughts...... I know
we've talked about alternative HDFS a few times (gluster, ceph), but never
a NAS appliance like this.

I'll give you some good technical contact emails at EMC/Pivotal off list
following this email.

-d


On Wed, Apr 30, 2014 at 5:39 PM, <Bo...@l-3com.com> wrote:

>  Has anyone used Accumulo with an Isilon platform for their name/data
> nodes?  Any advice and/or lessons learned would be appreciated.
>
>
>
> Bob Thorman
>
> Chief Engineer, PED Products
>
> L-3 Communications, ComCept
>
> (972) 772-7501 voice
>
> (972) 772-7510 fax
>
> (214) 793-2922 mobile
>
> bob.thorman@ncct.af.smil.mil
>
>
>



-- 

Donald Miner
Chief Technology Officer
ClearEdge IT Solutions, LLC
Cell: 443 799 7807
www.clearedgeit.com