You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@drill.apache.org by Elijah Zupancic <el...@apache.org> on 2016/05/06 16:43:42 UTC
Manta Object Store Support
I'm trying to get started contributing to Apache Drill. I've got the
project checked out and it is building to my satisfaction. Right now, I'm
trying to add support for the open source object store Manta (
https://github.com/joyent/manta). I thought that this would be a good
learning project.
Initially, I want to add support in the same way that S3 has support.
However, I can't seem to find a reference to the S3 storage driver in the
code base. Is the s3 storage driver part of a different project? How would
you suggest that I get started?
Thank you,
Elijah Zupancic
Re: Manta Object Store Support
Posted by Jacques Nadeau <ja...@dremio.com>.
It should "just work". Nothing in the logs?
--
Jacques Nadeau
CTO and Co-Founder, Dremio
On Wed, May 25, 2016 at 7:01 PM, Elijah Zupancic <el...@apache.org> wrote:
> Hi Tomer,
>
> Thanks for your advice about creating a Hadoop FileSystem implementation.
> I just finished a prototypical implementation of a Hadoop file system for
> Manta: https://github.com/dekobon/hadoop-manta <
> https://github.com/dekobon/hadoop-manta>
>
> I see the example for enabling S3 with Apache Drill and I’ve verified that
> it works. However, when I attempt to replicate the configuration of S3 for
> Manta, I’m unable to get the Hadoop FileSystem driver to load. I’ve
> verified that the FileSystem driver works in Hadoop by checking all of the
> hdfs dfs -* commands and I’ve got a fair bit of automated testing around it.
>
> What’s the magic to get it turned on with Drill? Do I need to do something
> to make the jar load other than copy it into jars/3rdparty? Right now, I’m
> just testing in drill-embedded for what it is worth.
>
> Thanks,
> Elijah Zupancic
>
> > On May 6, 2016, at 4:11 PM, Tomer Shiran <ts...@dremio.com> wrote:
> >
> > Does Manta have a Hadoop FileSystem API implementation? That's what Drill
> > uses for S3, HDFS, MapR-FS, Azure Blob Storage, etc. You could
> potentially
> > write a Drill storage plugin, but you get a lot for free if you already
> > have the file system implementation.
> >
> > On Fri, May 6, 2016 at 9:43 AM, Elijah Zupancic <el...@apache.org>
> wrote:
> >
> >> I'm trying to get started contributing to Apache Drill. I've got the
> >> project checked out and it is building to my satisfaction. Right now,
> I'm
> >> trying to add support for the open source object store Manta (
> >> https://github.com/joyent/manta). I thought that this would be a good
> >> learning project.
> >>
> >> Initially, I want to add support in the same way that S3 has support.
> >> However, I can't seem to find a reference to the S3 storage driver in
> the
> >> code base. Is the s3 storage driver part of a different project? How
> would
> >> you suggest that I get started?
> >>
> >> Thank you,
> >> Elijah Zupancic
> >>
>
>
Re: Manta Object Store Support
Posted by Elijah Zupancic <el...@apache.org>.
Hi Tomer,
Thanks for your advice about creating a Hadoop FileSystem implementation. I just finished a prototypical implementation of a Hadoop file system for Manta: https://github.com/dekobon/hadoop-manta <https://github.com/dekobon/hadoop-manta>
I see the example for enabling S3 with Apache Drill and I’ve verified that it works. However, when I attempt to replicate the configuration of S3 for Manta, I’m unable to get the Hadoop FileSystem driver to load. I’ve verified that the FileSystem driver works in Hadoop by checking all of the hdfs dfs -* commands and I’ve got a fair bit of automated testing around it.
What’s the magic to get it turned on with Drill? Do I need to do something to make the jar load other than copy it into jars/3rdparty? Right now, I’m just testing in drill-embedded for what it is worth.
Thanks,
Elijah Zupancic
> On May 6, 2016, at 4:11 PM, Tomer Shiran <ts...@dremio.com> wrote:
>
> Does Manta have a Hadoop FileSystem API implementation? That's what Drill
> uses for S3, HDFS, MapR-FS, Azure Blob Storage, etc. You could potentially
> write a Drill storage plugin, but you get a lot for free if you already
> have the file system implementation.
>
> On Fri, May 6, 2016 at 9:43 AM, Elijah Zupancic <el...@apache.org> wrote:
>
>> I'm trying to get started contributing to Apache Drill. I've got the
>> project checked out and it is building to my satisfaction. Right now, I'm
>> trying to add support for the open source object store Manta (
>> https://github.com/joyent/manta). I thought that this would be a good
>> learning project.
>>
>> Initially, I want to add support in the same way that S3 has support.
>> However, I can't seem to find a reference to the S3 storage driver in the
>> code base. Is the s3 storage driver part of a different project? How would
>> you suggest that I get started?
>>
>> Thank you,
>> Elijah Zupancic
>>
Re: Manta Object Store Support
Posted by Tomer Shiran <ts...@dremio.com>.
Does Manta have a Hadoop FileSystem API implementation? That's what Drill
uses for S3, HDFS, MapR-FS, Azure Blob Storage, etc. You could potentially
write a Drill storage plugin, but you get a lot for free if you already
have the file system implementation.
On Fri, May 6, 2016 at 9:43 AM, Elijah Zupancic <el...@apache.org> wrote:
> I'm trying to get started contributing to Apache Drill. I've got the
> project checked out and it is building to my satisfaction. Right now, I'm
> trying to add support for the open source object store Manta (
> https://github.com/joyent/manta). I thought that this would be a good
> learning project.
>
> Initially, I want to add support in the same way that S3 has support.
> However, I can't seem to find a reference to the S3 storage driver in the
> code base. Is the s3 storage driver part of a different project? How would
> you suggest that I get started?
>
> Thank you,
> Elijah Zupancic
>
Re: Manta Object Store Support
Posted by Elijah Zupancic <el...@apache.org>.
Please disregard this. I’ve figured out how to get it working. Once I realized that error logging for Hadoop Filesystem drivers wasn’t enabled by logback by default, then everything clicked into place. I’ll be back to ask more questions.
Thank you for your time,
Elijah Zupancic
> On May 6, 2016, at 9:43 AM, Elijah Zupancic <el...@apache.org> wrote:
>
> I'm trying to get started contributing to Apache Drill. I've got the project checked out and it is building to my satisfaction. Right now, I'm trying to add support for the open source object store Manta (https://github.com/joyent/manta <https://github.com/joyent/manta>). I thought that this would be a good learning project.
>
> Initially, I want to add support in the same way that S3 has support. However, I can't seem to find a reference to the S3 storage driver in the code base. Is the s3 storage driver part of a different project? How would you suggest that I get started?
>
> Thank you,
> Elijah Zupancic