You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@arrow.apache.org by "djouallah (via GitHub)" <gi...@apache.org> on 2023/11/15 03:31:09 UTC
[I] add support for Fabric OneLake, it is already supported by Delta_rs [arrow]
djouallah opened a new issue, #38726:
URL: https://github.com/apache/arrow/issues/38726
### Describe the enhancement requested
Delta_rs added support for Fabric OneLake recently, it will be nice to add the support for pyarrow dataset to read parquet and csv etc from OneLake Fabric
https://github.com/delta-io/delta-rs/pull/1642
currently I am using this code to read from dataset and save ad delta table but it works only with local path
```
import pyarrow.dataset as ds
from deltalake.writer import write_deltalake
aadToken = mssparkutils.credentials.getToken('storage')
storage_options={"bearer_token": aadToken, "use_fabric_endpoint": "true"}
sf=100
rowgroup = 2000000
nbr_rowgroup_File = 8 * rowgroup
for tbl in ['lineitem','nation','region','customer','supplier','orders','part','partsupp'] :
print(tbl)
dataset = ds.dataset(f'/lakehouse/default/Files/{sf}/{tbl}',format="parquet")
write_deltalake(f"abfss://Rust@onelake.dfs.fabric.microsoft.com/test.Lakehouse/Tables/{tbl}"\
,dataset\
,mode='overwrite',overwrite_schema=True,max_rows_per_file =nbr_rowgroup_File,min_rows_per_group=rowgroup,max_rows_per_group=rowgroup\
,storage_options=storage_options)
```
### Component(s)
Format, Integration, Python
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] add support for Fabric OneLake, it is already supported by Delta_rs [arrow]
Posted by "AlenkaF (via GitHub)" <gi...@apache.org>.
AlenkaF commented on issue #38726:
URL: https://github.com/apache/arrow/issues/38726#issuecomment-1837977200
As the support for Azure Blob Storage is being in progress in Apache Arrow C++ (https://github.com/search?q=repo%3Aapache%2Farrow+%5BC%2B%2B%5D%5BFS%5D%5BAzure%5D&type=issues) and it will be available in Python as a follow-up also, would that be something we can use to read from OneLake Fabric?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] add support for Fabric OneLake, it is already supported by Delta_rs [arrow]
Posted by "thisisnic (via GitHub)" <gi...@apache.org>.
thisisnic commented on issue #38726:
URL: https://github.com/apache/arrow/issues/38726#issuecomment-3921965145
This issue has been marked as stale because it has had no activity in the past 2 years. Please remove the stale label or comment below, or this issue will be closed in 14 days.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] add support for Fabric OneLake, it is already supported by Delta_rs [arrow]
Posted by "thisisnic (via GitHub)" <gi...@apache.org>.
thisisnic closed issue #38726: add support for Fabric OneLake, it is already supported by Delta_rs
URL: https://github.com/apache/arrow/issues/38726
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org