You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by GitBox <gi...@apache.org> on 2021/09/01 09:24:29 UTC
[GitHub] [arrow] Nlte opened a new pull request #11040: ARROW-13699: [Python][Docs] Improve filesystem documentation
Nlte opened a new pull request #11040:
URL: https://github.com/apache/arrow/pull/11040
Following https://issues.apache.org/jira/browse/ARROW-13699
Here are small improvement proposals for the python filesystem docs.
The goal is to make it more explicit what filesystem comes natively with PyArrow and what falls under fsspec third party libraries.
- Added sections for each native PyArrow filesystem
- Extended the fsspec section with additional code examples
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] github-actions[bot] commented on pull request #11040: ARROW-13699: [Python][Docs] Improve filesystem documentation
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #11040:
URL: https://github.com/apache/arrow/pull/11040#issuecomment-909417725
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] Nlte commented on a change in pull request #11040: ARROW-13699: [Python][Docs] Improve filesystem documentation
Posted by GitBox <gi...@apache.org>.
Nlte commented on a change in pull request #11040:
URL: https://github.com/apache/arrow/pull/11040#discussion_r708454673
##########
File path: docs/source/python/filesystems.rst
##########
@@ -200,10 +240,34 @@ For example::
import pyarrow.dataset as ds
ds.dataset("data/", filesystem=fs)
+::
Review comment:
That makes sense yes, I've pushed the change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] Nlte commented on a change in pull request #11040: ARROW-13699: [Python][Docs] Improve filesystem documentation
Posted by GitBox <gi...@apache.org>.
Nlte commented on a change in pull request #11040:
URL: https://github.com/apache/arrow/pull/11040#discussion_r708453822
##########
File path: docs/source/python/filesystems.rst
##########
@@ -200,10 +240,34 @@ For example::
import pyarrow.dataset as ds
ds.dataset("data/", filesystem=fs)
+::
+
+ # Similarly for Azure Blob Storage
+ import adlfs
+ # ... load your credentials and configure the filesystem
+ fs = adlfs.AzureBlobFileSystem(account_name=account_name, account_key=account_key)
+
+ import pyarrow.dataset as ds
+ ds.dataset("data/", filesystem=fs)
Review comment:
Yes I have just tried with a couple of partitions in an azure container (the equivalent of s3 bucket).
I have added the "container" bit in a recent commit to make that more clear.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] Nlte commented on a change in pull request #11040: ARROW-13699: [Python][Docs] Improve filesystem documentation
Posted by GitBox <gi...@apache.org>.
Nlte commented on a change in pull request #11040:
URL: https://github.com/apache/arrow/pull/11040#discussion_r708453822
##########
File path: docs/source/python/filesystems.rst
##########
@@ -200,10 +240,34 @@ For example::
import pyarrow.dataset as ds
ds.dataset("data/", filesystem=fs)
+::
+
+ # Similarly for Azure Blob Storage
+ import adlfs
+ # ... load your credentials and configure the filesystem
+ fs = adlfs.AzureBlobFileSystem(account_name=account_name, account_key=account_key)
+
+ import pyarrow.dataset as ds
+ ds.dataset("data/", filesystem=fs)
Review comment:
Yes I have just tried with a couple of partitions in an azure container (the equivalent of s3 bucket).
I have added the "mycontainer" bit in a recent commit, before the prefix, to make that more clear.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] amol- commented on a change in pull request #11040: ARROW-13699: [Python][Docs] Improve filesystem documentation
Posted by GitBox <gi...@apache.org>.
amol- commented on a change in pull request #11040:
URL: https://github.com/apache/arrow/pull/11040#discussion_r708107559
##########
File path: docs/source/python/filesystems.rst
##########
@@ -200,10 +240,34 @@ For example::
import pyarrow.dataset as ds
ds.dataset("data/", filesystem=fs)
+::
Review comment:
Might make sense to move the "Similarly for Azure Blob Storage" phrase from a comment to an actual phrase before the `::` here? I guess it would lead to an easier to follow flow of documentation
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] jorisvandenbossche commented on pull request #11040: ARROW-13699: [Python][Docs] Improve filesystem documentation
Posted by GitBox <gi...@apache.org>.
jorisvandenbossche commented on pull request #11040:
URL: https://github.com/apache/arrow/pull/11040#issuecomment-920098621
Thanks @Nlte for the nice doc update!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] github-actions[bot] commented on pull request #11040: ARROW-13699: [Python][Docs] Improve filesystem documentation
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #11040:
URL: https://github.com/apache/arrow/pull/11040#issuecomment-909417725
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] nealrichardson closed pull request #11040: ARROW-13699: [Python][Docs] Improve filesystem documentation
Posted by GitBox <gi...@apache.org>.
nealrichardson closed pull request #11040:
URL: https://github.com/apache/arrow/pull/11040
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] pitrou commented on a change in pull request #11040: ARROW-13699: [Python][Docs] Improve filesystem documentation
Posted by GitBox <gi...@apache.org>.
pitrou commented on a change in pull request #11040:
URL: https://github.com/apache/arrow/pull/11040#discussion_r707541657
##########
File path: docs/source/python/filesystems.rst
##########
@@ -200,10 +240,34 @@ For example::
import pyarrow.dataset as ds
ds.dataset("data/", filesystem=fs)
+::
+
+ # Similarly for Azure Blob Storage
+ import adlfs
+ # ... load your credentials and configure the filesystem
+ fs = adlfs.AzureBlobFileSystem(account_name=account_name, account_key=account_key)
+
+ import pyarrow.dataset as ds
+ ds.dataset("data/", filesystem=fs)
Review comment:
Nice! Did you check this actually works?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] pitrou commented on pull request #11040: ARROW-13699: [Python][Docs] Improve filesystem documentation
Posted by GitBox <gi...@apache.org>.
pitrou commented on pull request #11040:
URL: https://github.com/apache/arrow/pull/11040#issuecomment-918412754
@amol- Do you want to take a look here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org