You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@arrow.apache.org by Renato Marroquín Mogrovejo <re...@gmail.com> on 2018/07/25 10:49:22 UTC
arrow:io:S3ReadableFile
Hi Arrow experts,
I am in the middle of implementing a S3ReadableFile class, and I am
wondering if this is can be accomplished by using the hdfs client? or is it
just that it isn't a feature that users needed so far?
Any pointers/ideas are highly appreciated!
Thanks!
Renato M.
Re: arrow:io:S3ReadableFile
Posted by Renato Marroquín Mogrovejo <re...@gmail.com>.
Thanks Wes and Uwe for your suggestions!
2018-07-25 16:58 GMT+02:00 Uwe L. Korn <uw...@xhochy.com>:
> Hello Renato,
>
> I don't think that the hdfs client will give you the necessary interface
> to use Hadoop's S3 implementation. If it does, this might be a simple way
> to support more filesystems just by using their Hadoop implementation.
>
> In general, it would be preferred to have native (C/C++) implementations
> of these file classes that don't need to call into the JVM.
>
> Uwe
>
> On Wed, Jul 25, 2018, at 12:49 PM, Renato Marroquín Mogrovejo wrote:
> > Hi Arrow experts,
> >
> > I am in the middle of implementing a S3ReadableFile class, and I am
> > wondering if this is can be accomplished by using the hdfs client? or is
> it
> > just that it isn't a feature that users needed so far?
> > Any pointers/ideas are highly appreciated!
> > Thanks!
> >
> >
> > Renato M.
>
Re: arrow:io:S3ReadableFile
Posted by "Uwe L. Korn" <uw...@xhochy.com>.
Hello Renato,
I don't think that the hdfs client will give you the necessary interface to use Hadoop's S3 implementation. If it does, this might be a simple way to support more filesystems just by using their Hadoop implementation.
In general, it would be preferred to have native (C/C++) implementations of these file classes that don't need to call into the JVM.
Uwe
On Wed, Jul 25, 2018, at 12:49 PM, Renato Marroquín Mogrovejo wrote:
> Hi Arrow experts,
>
> I am in the middle of implementing a S3ReadableFile class, and I am
> wondering if this is can be accomplished by using the hdfs client? or is it
> just that it isn't a feature that users needed so far?
> Any pointers/ideas are highly appreciated!
> Thanks!
>
>
> Renato M.
Re: arrow:io:S3ReadableFile
Posted by Wes McKinney <we...@gmail.com>.
hey Renato,
I would recommend following whatever TensorFlow has done. We can even
reuse their code (Apache 2.0):
https://github.com/tensorflow/tensorflow/tree/master/tensorflow/core/platform/s3
- Wes
On Wed, Jul 25, 2018 at 6:49 AM, Renato Marroquín Mogrovejo
<re...@gmail.com> wrote:
> Hi Arrow experts,
>
> I am in the middle of implementing a S3ReadableFile class, and I am
> wondering if this is can be accomplished by using the hdfs client? or is it
> just that it isn't a feature that users needed so far?
> Any pointers/ideas are highly appreciated!
> Thanks!
>
>
> Renato M.