Support for Alluxio

[Edmon Begoli <eb...@gmail.com>]

Is it possible to use Alluxio as a filesystem backend in Drill's storage
formats, and if so how?

Thanks.

RE: Support for Alluxio

[Ralph Schlosser <ra...@ericsson.com>]

Hi Oscar, 

I'd definitely be interested in your approach to containerization and running on Drill, Alluxio etc. on Kubernetes. A while ago I made an attempt at dockerizing Alluxio + Drill but hit a road block wrt. networking issues. Would be great if you have something you can share.


Regards,
Ralph

-----Original Message-----
From: Oscar Morante [mailto:spacepluk@gmail.com] 
Sent: 28 September 2016 08:26
To: user@drill.apache.org
Subject: Re: Support for Alluxio

Hi there,
I've been looking into this.  It's almost working except for some issues when reading lots of parquet files, but I'm not sure if that is Alluxio's fault.

I have a couple of WIP Docker images that's I'm using on Kubernetes.  I can share if you want, but it's all very hackish right now and I'm still changing things very often.

Cheers,

On Wed, Sep 14, 2016 at 11:13:49AM -0700, Sudheesh Katkam wrote:
>Based on the documentation [1], Alluxio not only uses HDFS as a underlying storage system but also provides a “Hadoop API” to clients, and Drill uses this API. So it should be possible.
>
>Create a storage plugin named “alluxio” whose contents match “dfs” plugin, and then make changes. Not sure what exactly, but [2] should be helpful. See “S3 Storage Plugin” as an example as well [3]. Once you get things to work, please contribute by adding a section to the Drill documentation.
>
>Thank you,
>Sudheesh
>
>[1] 
>http://www.alluxio.org/docs/master/en/File-System-API.html#hadoop-api 
><http://www.alluxio.org/docs/master/en/File-System-API.html#hadoop-api>
>[2] http://www.alluxio.org/docs/master/en/Configuration-Settings.html 
><http://www.alluxio.org/docs/master/en/Configuration-Settings.html>
>[3] http://drill.apache.org/docs/s3-storage-plugin/ 
><http://drill.apache.org/docs/s3-storage-plugin/>
>
>> On Sep 14, 2016, at 10:07 AM, Edmon Begoli <eb...@gmail.com> wrote:
>>
>> Is it possible to use Alluxio as a filesystem backend in Drill's 
>> storage formats, and if so how?
>>
>> Thanks.
>

--
Oscar Morante
"Self-education is, I firmly believe, the only kind of education there is."
                                                          -- Isaac Asimov.

Re: Support for Alluxio

[Oscar Morante <sp...@gmail.com>]

Hi there,
I've been looking into this.  It's almost working except for some issues 
when reading lots of parquet files, but I'm not sure if that is 
Alluxio's fault.

I have a couple of WIP Docker images that's I'm using on Kubernetes.  I 
can share if you want, but it's all very hackish right now and I'm still 
changing things very often.

Cheers,

On Wed, Sep 14, 2016 at 11:13:49AM -0700, Sudheesh Katkam wrote:
>Based on the documentation [1], Alluxio not only uses HDFS as a underlying storage system but also provides a “Hadoop API” to clients, and Drill uses this API. So it should be possible.
>
>Create a storage plugin named “alluxio” whose contents match “dfs” plugin, and then make changes. Not sure what exactly, but [2] should be helpful. See “S3 Storage Plugin” as an example as well [3]. Once you get things to work, please contribute by adding a section to the Drill documentation.
>
>Thank you,
>Sudheesh
>
>[1] http://www.alluxio.org/docs/master/en/File-System-API.html#hadoop-api <http://www.alluxio.org/docs/master/en/File-System-API.html#hadoop-api>
>[2] http://www.alluxio.org/docs/master/en/Configuration-Settings.html <http://www.alluxio.org/docs/master/en/Configuration-Settings.html>
>[3] http://drill.apache.org/docs/s3-storage-plugin/ <http://drill.apache.org/docs/s3-storage-plugin/>
>
>> On Sep 14, 2016, at 10:07 AM, Edmon Begoli <eb...@gmail.com> wrote:
>>
>> Is it possible to use Alluxio as a filesystem backend in Drill's storage
>> formats, and if so how?
>>
>> Thanks.
>

-- 
Oscar Morante
"Self-education is, I firmly believe, the only kind of education there is."
                                                          -- Isaac Asimov.

Re: Support for Alluxio

[Sudheesh Katkam <sk...@maprtech.com>]

Based on the documentation [1], Alluxio not only uses HDFS as a underlying storage system but also provides a “Hadoop API” to clients, and Drill uses this API. So it should be possible.

Create a storage plugin named “alluxio” whose contents match “dfs” plugin, and then make changes. Not sure what exactly, but [2] should be helpful. See “S3 Storage Plugin” as an example as well [3]. Once you get things to work, please contribute by adding a section to the Drill documentation.

Thank you,
Sudheesh

[1] http://www.alluxio.org/docs/master/en/File-System-API.html#hadoop-api <http://www.alluxio.org/docs/master/en/File-System-API.html#hadoop-api>
[2] http://www.alluxio.org/docs/master/en/Configuration-Settings.html <http://www.alluxio.org/docs/master/en/Configuration-Settings.html>
[3] http://drill.apache.org/docs/s3-storage-plugin/ <http://drill.apache.org/docs/s3-storage-plugin/>

> On Sep 14, 2016, at 10:07 AM, Edmon Begoli <eb...@gmail.com> wrote:
> 
> Is it possible to use Alluxio as a filesystem backend in Drill's storage
> formats, and if so how?
> 
> Thanks.