You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@arrow.apache.org by Jerry Adair <Je...@sas.com.INVALID> on 2022/10/26 17:58:35 UTC
RE: Apache Arrow filesystem question
Hi-
I am curious about the Apache Arrow filesystem support. I noticed that there are classes for S3 and GCS. I am currently using those classes to read/write from/to those cloud storage options. However I noticed that there is no class for Azure (ADLS). And so I am curious why that is? Does anyone know?
Thanks!
Jerry
Re: Apache Arrow filesystem question
Posted by Jacob Wujciak <ja...@voltrondata.com.INVALID>.
The azure implementation is underway but did not make it into 10.0.0:
https://issues.apache.org/jira/browse/ARROW-2034
On Wed, Oct 26, 2022 at 7:59 PM Jerry Adair <Je...@sas.com.invalid>
wrote:
> Hi-
>
> I am curious about the Apache Arrow filesystem support. I noticed that
> there are classes for S3 and GCS. I am currently using those classes to
> read/write from/to those cloud storage options. However I noticed that
> there is no class for Azure (ADLS). And so I am curious why that is? Does
> anyone know?
>
> Thanks!
> Jerry
>
RE: Apache Arrow filesystem question
Posted by Jerry Adair <Je...@sas.com.INVALID>.
I agree. I already utilize the AWS and GCS classes. We have a customer need for ADLS & Parquet. I've recommended that we wait for the new implementation to become available in the next Arrow/Parquet C++ library release. In a way I am making a "bet" that the feature makes it into that release (11.0.0), so hopefully the contributors will make that happen.
-----Original Message-----
From: Wes McKinney <we...@gmail.com>
Sent: Thursday, October 27, 2022 4:42 PM
To: dev@arrow.apache.org
Subject: Re: Apache Arrow filesystem question
[You don't often get email from wesmckinn@gmail.com. Learn why this is important at https://aka.ms/LearnAboutSenderIdentification ]
EXTERNAL
I definitely think it would be a good thing to have a C++ ADLS filesystem interface that is on par in quality with our S3 and GCS C++ interfaces - these should also provide material performance benefits to Python users over a pure-Python interface (I'm not sure if pyarrow's S3 interface via C++ has been studied relative to s3fs but there should be performance benefits in many datasets interactions).
On Wed, Oct 26, 2022 at 1:09 PM Lee, David <Da...@blackrock.com.invalid> wrote:
>
> Or this..
>
> https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Farro
> w.apache.org%2Fdocs%2Fpython%2Ffilesystems.html%23using-fsspec-compati
> ble-filesystems-with-arrow&data=05%7C01%7CJerry.Adair%40sas.com%7C
> b0f839fa00934e9b49ce08dab8644346%7Cb1c14d5c362545b3a4309552373a0c2f%7C
> 0%7C0%7C638025038018602431%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDA
> iLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sd
> ata=gnpmJKK3%2FcuYbnNlhEr%2F4notr7lCf0F9dJFTH8IeqYA%3D&reserved=0
>
> import adlfs
> # ... load your credentials and configure the filesystem fs =
> adlfs.AzureBlobFileSystem(account_name=account_name,
> account_key=account_key)
>
>
> -----Original Message-----
> From: Lee, David
> Sent: Wednesday, October 26, 2022 11:04 AM
> To: dev@arrow.apache.org
> Subject: RE: Apache Arrow filesystem question
>
> There are fsspec implementations to support other filesystems.
>
> https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpypi
> .org%2Fproject%2Fpyarrowfs-adlgen2%2F&data=05%7C01%7CJerry.Adair%4
> 0sas.com%7Cb0f839fa00934e9b49ce08dab8644346%7Cb1c14d5c362545b3a4309552
> 373a0c2f%7C0%7C0%7C638025038018602431%7CUnknown%7CTWFpbGZsb3d8eyJWIjoi
> MC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7
> C%7C&sdata=Zhzd2XQa1pAkKYolRG4%2Bvgpr%2Fk7tmKSv6fL11Kg5xIg%3D&
> reserved=0
>
> -----Original Message-----
> From: Jerry Adair <Je...@sas.com.INVALID>
> Sent: Wednesday, October 26, 2022 10:59 AM
> To: dev@arrow.apache.org
> Subject: RE: Apache Arrow filesystem question
>
> External Email: Use caution with links and attachments
>
>
> Hi-
>
> I am curious about the Apache Arrow filesystem support. I noticed that there are classes for S3 and GCS. I am currently using those classes to read/write from/to those cloud storage options. However I noticed that there is no class for Azure (ADLS). And so I am curious why that is? Does anyone know?
>
> Thanks!
> Jerry
>
>
> This message may contain information that is confidential or privileged. If you are not the intended recipient, please advise the sender immediately and delete this message. See https://nam02.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.blackrock.com%2Fcorporate%2Fcompliance%2Femail-disclaimers&data=05%7C01%7CJerry.Adair%40sas.com%7Cb0f839fa00934e9b49ce08dab8644346%7Cb1c14d5c362545b3a4309552373a0c2f%7C0%7C0%7C638025038018602431%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=a7lIrYIWXA3WNA%2Bc%2BRFvY952NWyjb3f03WLxUeOfOY4%3D&reserved=0 for further information. Please refer to https://nam02.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.blackrock.com%2Fcorporate%2Fcompliance%2Fprivacy-policy&data=05%7C01%7CJerry.Adair%40sas.com%7Cb0f839fa00934e9b49ce08dab8644346%7Cb1c14d5c362545b3a4309552373a0c2f%7C0%7C0%7C638025038018602431%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=X93wWDdHJ6oPQT1S8sZqURTZl1borPzC8VCRisblioI%3D&reserved=0 for more information about BlackRock's Privacy Policy.
>
>
> For a list of BlackRock's office addresses worldwide, see https://nam02.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.blackrock.com%2Fcorporate%2Fabout-us%2Fcontacts-locations&data=05%7C01%7CJerry.Adair%40sas.com%7Cb0f839fa00934e9b49ce08dab8644346%7Cb1c14d5c362545b3a4309552373a0c2f%7C0%7C0%7C638025038018602431%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=okDt4J9JZ78nIOx913vltrblW%2B9XFTJpUpNOv%2BthNwA%3D&reserved=0.
>
> (c) 2022 BlackRock, Inc. All rights reserved.
Re: Apache Arrow filesystem question
Posted by Wes McKinney <we...@gmail.com>.
I definitely think it would be a good thing to have a C++ ADLS
filesystem interface that is on par in quality with our S3 and GCS C++
interfaces — these should also provide material performance benefits
to Python users over a pure-Python interface (I'm not sure if
pyarrow's S3 interface via C++ has been studied relative to s3fs but
there should be performance benefits in many datasets interactions).
On Wed, Oct 26, 2022 at 1:09 PM Lee, David
<Da...@blackrock.com.invalid> wrote:
>
> Or this..
>
> https://arrow.apache.org/docs/python/filesystems.html#using-fsspec-compatible-filesystems-with-arrow
>
> import adlfs
> # ... load your credentials and configure the filesystem
> fs = adlfs.AzureBlobFileSystem(account_name=account_name, account_key=account_key)
>
>
> -----Original Message-----
> From: Lee, David
> Sent: Wednesday, October 26, 2022 11:04 AM
> To: dev@arrow.apache.org
> Subject: RE: Apache Arrow filesystem question
>
> There are fsspec implementations to support other filesystems.
>
> https://pypi.org/project/pyarrowfs-adlgen2/
>
> -----Original Message-----
> From: Jerry Adair <Je...@sas.com.INVALID>
> Sent: Wednesday, October 26, 2022 10:59 AM
> To: dev@arrow.apache.org
> Subject: RE: Apache Arrow filesystem question
>
> External Email: Use caution with links and attachments
>
>
> Hi-
>
> I am curious about the Apache Arrow filesystem support. I noticed that there are classes for S3 and GCS. I am currently using those classes to read/write from/to those cloud storage options. However I noticed that there is no class for Azure (ADLS). And so I am curious why that is? Does anyone know?
>
> Thanks!
> Jerry
>
>
> This message may contain information that is confidential or privileged. If you are not the intended recipient, please advise the sender immediately and delete this message. See http://www.blackrock.com/corporate/compliance/email-disclaimers for further information. Please refer to http://www.blackrock.com/corporate/compliance/privacy-policy for more information about BlackRock’s Privacy Policy.
>
>
> For a list of BlackRock's office addresses worldwide, see http://www.blackrock.com/corporate/about-us/contacts-locations.
>
> © 2022 BlackRock, Inc. All rights reserved.
RE: Apache Arrow filesystem question
Posted by "Lee, David" <Da...@blackrock.com.INVALID>.
Or this..
https://arrow.apache.org/docs/python/filesystems.html#using-fsspec-compatible-filesystems-with-arrow
import adlfs
# ... load your credentials and configure the filesystem
fs = adlfs.AzureBlobFileSystem(account_name=account_name, account_key=account_key)
-----Original Message-----
From: Lee, David
Sent: Wednesday, October 26, 2022 11:04 AM
To: dev@arrow.apache.org
Subject: RE: Apache Arrow filesystem question
There are fsspec implementations to support other filesystems.
https://pypi.org/project/pyarrowfs-adlgen2/
-----Original Message-----
From: Jerry Adair <Je...@sas.com.INVALID>
Sent: Wednesday, October 26, 2022 10:59 AM
To: dev@arrow.apache.org
Subject: RE: Apache Arrow filesystem question
External Email: Use caution with links and attachments
Hi-
I am curious about the Apache Arrow filesystem support. I noticed that there are classes for S3 and GCS. I am currently using those classes to read/write from/to those cloud storage options. However I noticed that there is no class for Azure (ADLS). And so I am curious why that is? Does anyone know?
Thanks!
Jerry
This message may contain information that is confidential or privileged. If you are not the intended recipient, please advise the sender immediately and delete this message. See http://www.blackrock.com/corporate/compliance/email-disclaimers for further information. Please refer to http://www.blackrock.com/corporate/compliance/privacy-policy for more information about BlackRock’s Privacy Policy.
For a list of BlackRock's office addresses worldwide, see http://www.blackrock.com/corporate/about-us/contacts-locations.
© 2022 BlackRock, Inc. All rights reserved.
RE: Apache Arrow filesystem question
Posted by "Lee, David" <Da...@blackrock.com.INVALID>.
There are fsspec implementations to support other filesystems.
https://pypi.org/project/pyarrowfs-adlgen2/
-----Original Message-----
From: Jerry Adair <Je...@sas.com.INVALID>
Sent: Wednesday, October 26, 2022 10:59 AM
To: dev@arrow.apache.org
Subject: RE: Apache Arrow filesystem question
External Email: Use caution with links and attachments
Hi-
I am curious about the Apache Arrow filesystem support. I noticed that there are classes for S3 and GCS. I am currently using those classes to read/write from/to those cloud storage options. However I noticed that there is no class for Azure (ADLS). And so I am curious why that is? Does anyone know?
Thanks!
Jerry
This message may contain information that is confidential or privileged. If you are not the intended recipient, please advise the sender immediately and delete this message. See http://www.blackrock.com/corporate/compliance/email-disclaimers for further information. Please refer to http://www.blackrock.com/corporate/compliance/privacy-policy for more information about BlackRock’s Privacy Policy.
For a list of BlackRock's office addresses worldwide, see http://www.blackrock.com/corporate/about-us/contacts-locations.
© 2022 BlackRock, Inc. All rights reserved.