You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@arrow.apache.org by "Tom-Newton (via GitHub)" <gi...@apache.org> on 2026/01/30 10:03:26 UTC

[I] [Python][FS][Azure] Pickling `SubTreeFileSystem(base_path, AzureFileSystem(...))` is lossy [arrow]

Tom-Newton opened a new issue, #49078:
URL: https://github.com/apache/arrow/issues/49078

   ### Describe the bug, including details regarding any error messages, version, and platform.
   
   ### Reproduce:
   ```
   import pyarrow.fs
   
   azure_fs = pyarrow.fs.AzureFileSystem(account_name="test", sas_token="test")
   print(azure_fs.__reduce__())
   
   subtree_fs = pyarrow.fs.SubTreeFileSystem("/tmp", azure_fs)
   print(subtree_fs.base_fs.__reduce__())
   ```
   
   Returns
   ```
   (<cyfunction AzureFileSystem._reconstruct at 0x79d22010c940>, ({'account_name': 'test', 'account_key': '', 'blob_storage_authority': '.blob.core.windows.net', 'blob_storage_scheme': 'https', 'client_id': '', 'client_secret': '', 'dfs_storage_authority': '.dfs.core.windows.net', 'dfs_storage_scheme': 'https', 'sas_token': 'test', 'tenant_id': ''},))
   (<cyfunction AzureFileSystem._reconstruct at 0x79d22010c940>, ({'account_name': 'test', 'account_key': '', 'blob_storage_authority': '.blob.core.windows.net', 'blob_storage_scheme': 'https', 'client_id': '', 'client_secret': '', 'dfs_storage_authority': '.dfs.core.windows.net', 'dfs_storage_scheme': 'https', 'sas_token': '', 'tenant_id': ''},))
   ```
   Notice how the first result the `sas_token` is not empty but the second one is. 
   
   ### Cause:
   The `sas_token` and a couple of the other values returned by `AzureFileSystem.__reduce__` read from `self` of the python side `AzureFileSystem` object. When constructing a `SubTreeFileSystem`, the python side `AzureFileSystem` object is discarded and the `SubTreeFileSystem` only holds a pointer to the `CAzureFileSystem`. Therefore its not possible to reconstruct a python side `AzureFileSystem` including the attributes on `self` of the original `AzureFileSystem`. 
   
   ### Component(s)
   
   Python


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] [Python][FS][Azure] Pickling `SubTreeFileSystem(base_path, AzureFileSystem(...))` is lossy [arrow]

Posted by "raulcd (via GitHub)" <gi...@apache.org>.
raulcd closed issue #49078: [Python][FS][Azure] Pickling `SubTreeFileSystem(base_path, AzureFileSystem(...))` is lossy
URL: https://github.com/apache/arrow/issues/49078


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] [Python][FS][Azure] Pickling `SubTreeFileSystem(base_path, AzureFileSystem(...))` is lossy [arrow]

Posted by "raulcd (via GitHub)" <gi...@apache.org>.
raulcd commented on issue #49078:
URL: https://github.com/apache/arrow/issues/49078#issuecomment-3983492175

   Issue resolved by pull request 49140
   https://github.com/apache/arrow/pull/49140


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org