You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by GitBox <gi...@apache.org> on 2021/09/01 09:24:29 UTC

[GitHub] [arrow] Nlte opened a new pull request #11040: ARROW-13699: [Python][Docs] Improve filesystem documentation

Nlte opened a new pull request #11040:
URL: https://github.com/apache/arrow/pull/11040


   Following https://issues.apache.org/jira/browse/ARROW-13699
   Here are small improvement proposals for the python filesystem docs.
   
   The goal is to make it more explicit what filesystem comes natively with PyArrow and what falls under fsspec third party libraries.
   
   - Added sections for each native PyArrow filesystem
   - Extended the fsspec section with additional code examples
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] github-actions[bot] commented on pull request #11040: ARROW-13699: [Python][Docs] Improve filesystem documentation

Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #11040:
URL: https://github.com/apache/arrow/pull/11040#issuecomment-909417725






-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] Nlte commented on a change in pull request #11040: ARROW-13699: [Python][Docs] Improve filesystem documentation

Posted by GitBox <gi...@apache.org>.
Nlte commented on a change in pull request #11040:
URL: https://github.com/apache/arrow/pull/11040#discussion_r708454673



##########
File path: docs/source/python/filesystems.rst
##########
@@ -200,10 +240,34 @@ For example::
    import pyarrow.dataset as ds
    ds.dataset("data/", filesystem=fs)
 
+::

Review comment:
       That makes sense yes, I've pushed the change.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] Nlte commented on a change in pull request #11040: ARROW-13699: [Python][Docs] Improve filesystem documentation

Posted by GitBox <gi...@apache.org>.
Nlte commented on a change in pull request #11040:
URL: https://github.com/apache/arrow/pull/11040#discussion_r708453822



##########
File path: docs/source/python/filesystems.rst
##########
@@ -200,10 +240,34 @@ For example::
    import pyarrow.dataset as ds
    ds.dataset("data/", filesystem=fs)
 
+::
+
+   # Similarly for Azure Blob Storage
+   import adlfs
+   # ... load your credentials and configure the filesystem
+   fs = adlfs.AzureBlobFileSystem(account_name=account_name, account_key=account_key)
+
+   import pyarrow.dataset as ds
+   ds.dataset("data/", filesystem=fs)

Review comment:
       Yes I have just tried with a couple of partitions in an azure container (the equivalent of s3 bucket).
   I have added the "container" bit in a recent commit to make that more clear.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] Nlte commented on a change in pull request #11040: ARROW-13699: [Python][Docs] Improve filesystem documentation

Posted by GitBox <gi...@apache.org>.
Nlte commented on a change in pull request #11040:
URL: https://github.com/apache/arrow/pull/11040#discussion_r708453822



##########
File path: docs/source/python/filesystems.rst
##########
@@ -200,10 +240,34 @@ For example::
    import pyarrow.dataset as ds
    ds.dataset("data/", filesystem=fs)
 
+::
+
+   # Similarly for Azure Blob Storage
+   import adlfs
+   # ... load your credentials and configure the filesystem
+   fs = adlfs.AzureBlobFileSystem(account_name=account_name, account_key=account_key)
+
+   import pyarrow.dataset as ds
+   ds.dataset("data/", filesystem=fs)

Review comment:
       Yes I have just tried with a couple of partitions in an azure container (the equivalent of s3 bucket).
   I have added the "mycontainer" bit in a recent commit, before the prefix, to make that more clear.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] amol- commented on a change in pull request #11040: ARROW-13699: [Python][Docs] Improve filesystem documentation

Posted by GitBox <gi...@apache.org>.
amol- commented on a change in pull request #11040:
URL: https://github.com/apache/arrow/pull/11040#discussion_r708107559



##########
File path: docs/source/python/filesystems.rst
##########
@@ -200,10 +240,34 @@ For example::
    import pyarrow.dataset as ds
    ds.dataset("data/", filesystem=fs)
 
+::

Review comment:
       Might make sense to move the "Similarly for Azure Blob Storage" phrase from a comment to an actual phrase before the `::` here? I guess it would lead to an easier to follow flow of documentation




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] jorisvandenbossche commented on pull request #11040: ARROW-13699: [Python][Docs] Improve filesystem documentation

Posted by GitBox <gi...@apache.org>.
jorisvandenbossche commented on pull request #11040:
URL: https://github.com/apache/arrow/pull/11040#issuecomment-920098621


   Thanks @Nlte for the nice doc update!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] github-actions[bot] commented on pull request #11040: ARROW-13699: [Python][Docs] Improve filesystem documentation

Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #11040:
URL: https://github.com/apache/arrow/pull/11040#issuecomment-909417725






-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] nealrichardson closed pull request #11040: ARROW-13699: [Python][Docs] Improve filesystem documentation

Posted by GitBox <gi...@apache.org>.
nealrichardson closed pull request #11040:
URL: https://github.com/apache/arrow/pull/11040


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] pitrou commented on a change in pull request #11040: ARROW-13699: [Python][Docs] Improve filesystem documentation

Posted by GitBox <gi...@apache.org>.
pitrou commented on a change in pull request #11040:
URL: https://github.com/apache/arrow/pull/11040#discussion_r707541657



##########
File path: docs/source/python/filesystems.rst
##########
@@ -200,10 +240,34 @@ For example::
    import pyarrow.dataset as ds
    ds.dataset("data/", filesystem=fs)
 
+::
+
+   # Similarly for Azure Blob Storage
+   import adlfs
+   # ... load your credentials and configure the filesystem
+   fs = adlfs.AzureBlobFileSystem(account_name=account_name, account_key=account_key)
+
+   import pyarrow.dataset as ds
+   ds.dataset("data/", filesystem=fs)

Review comment:
       Nice! Did you check this actually works?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] pitrou commented on pull request #11040: ARROW-13699: [Python][Docs] Improve filesystem documentation

Posted by GitBox <gi...@apache.org>.
pitrou commented on pull request #11040:
URL: https://github.com/apache/arrow/pull/11040#issuecomment-918412754


   @amol- Do you want to take a look here?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org