You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@arrow.apache.org by Akbar <ed...@gmail.com> on 2020/08/01 04:02:35 UTC
Questions | filesystem legacy
Hello,
Documentation on legacy file system interface is not quite clear. I’m not sure if the HDFS API layers are still relevant and supported. I know the HDFS API are operational
The two questions I have
1. if there is an equivalent to HadoopFileSystem.download in the new file system interface (pyarrow.fs.HadoopFileSystem)
2. Will support for HDFS API be removed - I ask this based on the legacy tag on the documentation
Sent from my iPhone
Re: Questions | filesystem legacy
Posted by Joris Van den Bossche <jo...@gmail.com>.
Hi Akbar,
The documentation regarding the legacy and new file system interface is
indeed somewhat lacking. So in general, we have the older, and now legacy,
filesystems in pyarrow.filesystem (
https://arrow.apache.org/docs/python/filesystems_deprecated.html) and a new
implementation in pyarrow.fs (
https://arrow.apache.org/docs/python/filesystems.html,
https://arrow.apache.org/docs/python/api/filesystems.html).
We need to document this better (and actually deprecate), but the long term
goal is certainly to eventually remove pyarrow.filesystem.
So regarding your specific HDFS related questions:
- There is also a HadoopFileSystem in the new interface (
https://arrow.apache.org/docs/python/generated/pyarrow.fs.HadoopFileSystem.html),
so in general support for HDFS is not limited to the deprecated API
- The available methods on the new interface are different though, and
there is no "download" method anymore. However (although I am not fully
familiar with this), I think you can achieve more or less the same with the
"open_input_file" method of the new interface (which returns a NativeFile
object, which has a download method).
Best,
Joris
On Tue, 4 Aug 2020 at 04:19, Akbar <ed...@gmail.com> wrote:
> apologies - I made of mess of that email. Let me try again
>
>
> - *Question 1* - pyarrow.HadoopFileSystem.download
> <https://arrow.apache.org/docs/python/generated/pyarrow.HadoopFileSystem.download.html> -
> is listed under Filesystem Interface (Legacy) (and so are all the HDFS
> APIs) - does this mean support for this is limited?
> - *Question 2* - is there an equivalent to
> pyarrow.HadoopFileSystem.download in the newer
> pyarrow.fs.HadoopFileSystem
> <https://arrow.apache.org/docs/python/generated/pyarrow.fs.HadoopFileSystem.html#pyarrow.fs.HadoopFileSystem>
> .
>
>
> I want to be able to fetch/download an HDFS entire file or folder to the
> HOST OS filesystem - Let me know if you have any guidance
>
>
> On Fri, 31 Jul 2020 at 23:02, Akbar <ed...@gmail.com> wrote:
>
>>
>> Hello,
>>
>> Documentation on legacy file system interface is not quite clear. I’m not
>> sure if the HDFS API layers are still relevant and supported. I know the
>> HDFS API are operational
>>
>> The two questions I have
>> 1. if there is an equivalent to HadoopFileSystem.download in the new file
>> system interface (pyarrow.fs.HadoopFileSystem)
>> 2. Will support for HDFS API be removed - I ask this based on the legacy
>> tag on the documentation
>>
>> Sent from my iPhone
>
>
Re: Questions | filesystem legacy
Posted by Akbar <ed...@gmail.com>.
apologies - I made of mess of that email. Let me try again
- *Question 1* - pyarrow.HadoopFileSystem.download
<https://arrow.apache.org/docs/python/generated/pyarrow.HadoopFileSystem.download.html
> -
is listed under Filesystem Interface (Legacy) (and so are all the HDFS
APIs) - does this mean support for this is limited?
- *Question 2* - is there an equivalent to
pyarrow.HadoopFileSystem.download in the newer
pyarrow.fs.HadoopFileSystem
<https://arrow.apache.org/docs/python/generated/pyarrow.fs.HadoopFileSystem.html#pyarrow.fs.HadoopFileSystem>
.
I want to be able to fetch/download an HDFS entire file or folder to the
HOST OS filesystem - Let me know if you have any guidance
On Fri, 31 Jul 2020 at 23:02, Akbar <ed...@gmail.com> wrote:
>
> Hello,
>
> Documentation on legacy file system interface is not quite clear. I’m not
> sure if the HDFS API layers are still relevant and supported. I know the
> HDFS API are operational
>
> The two questions I have
> 1. if there is an equivalent to HadoopFileSystem.download in the new file
> system interface (pyarrow.fs.HadoopFileSystem)
> 2. Will support for HDFS API be removed - I ask this based on the legacy
> tag on the documentation
>
> Sent from my iPhone