You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "Fokko (via GitHub)" <gi...@apache.org> on 2023/03/01 11:42:02 UTC

[GitHub] [iceberg] Fokko opened a new pull request, #6970: Python: Fix ADLS configuration

Fokko opened a new pull request, #6970:
URL: https://github.com/apache/iceberg/pull/6970

   The configuration keys weren't extracted according to the documentation.
   
   cc @cccs-eric 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] Fokko commented on a diff in pull request #6970: Python: Fix ADLS configuration

Posted by "Fokko (via GitHub)" <gi...@apache.org>.
Fokko commented on code in PR #6970:
URL: https://github.com/apache/iceberg/pull/6970#discussion_r1124987895


##########
python/pyiceberg/io/fsspec.py:
##########
@@ -112,8 +112,15 @@ def _s3(properties: Properties) -> AbstractFileSystem:
 def _adlfs(properties: Properties) -> AbstractFileSystem:
     from adlfs import AzureBlobFileSystem
 
-    fs = AzureBlobFileSystem(**properties)
-    return fs
+    return AzureBlobFileSystem(
+        connection_string=properties.get("adlfs.connection-string"),
+        account_name=properties.get("adlfs.account-name"),
+        account_key=properties.get("adlfs.account-key"),
+        sas_token=properties.get("adlfs.sas-token"),
+        tenant_id=properties.get("adlfs.tenant-id"),
+        client_id=properties.get("adlfs.client-id"),
+        client_secret=properties.get("adlfs.client-secret"),
+    )
 
 
 SCHEME_TO_FS = {

Review Comment:
   I wasn't sure if `wasb` is supported. I think we should confirm that first



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] alaturqua commented on a diff in pull request #6970: Python: Fix ADLS configuration

Posted by "alaturqua (via GitHub)" <gi...@apache.org>.
alaturqua commented on code in PR #6970:
URL: https://github.com/apache/iceberg/pull/6970#discussion_r1123016616


##########
python/pyiceberg/io/fsspec.py:
##########
@@ -112,8 +112,15 @@ def _s3(properties: Properties) -> AbstractFileSystem:
 def _adlfs(properties: Properties) -> AbstractFileSystem:
     from adlfs import AzureBlobFileSystem
 
-    fs = AzureBlobFileSystem(**properties)
-    return fs
+    return AzureBlobFileSystem(
+        connection_string=properties.get("adlfs.connection-string"),
+        account_name=properties.get("adlfs.account-name"),
+        account_key=properties.get("adlfs.account-key"),
+        sas_token=properties.get("adlfs.sas-token"),
+        tenant_id=properties.get("adlfs.tenant-id"),
+        client_id=properties.get("adlfs.client-id"),
+        client_secret=properties.get("adlfs.client-secret"),
+    )
 
 
 SCHEME_TO_FS = {

Review Comment:
   I would like to suggest adding here WASB/WASBS support as well:
   
   ```
   SCHEME_TO_FS = {
       "s3": _s3,
       "s3a": _s3,
       "s3n": _s3,
       "abfs": _adlfs,
       "abfss": _adlfs,
       "wasb": _adlfs,
       "wasbs":_adlfs
   }
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] alaturqua commented on a diff in pull request #6970: Python: Fix ADLS configuration

Posted by "alaturqua (via GitHub)" <gi...@apache.org>.
alaturqua commented on code in PR #6970:
URL: https://github.com/apache/iceberg/pull/6970#discussion_r1122763057


##########
python/tests/conftest.py:
##########
@@ -1277,8 +1277,8 @@ def adlfs_fsspec_fileio(request: pytest.FixtureRequest) -> Generator[FsspecFileI
     azurite_account_key = request.config.getoption("--adlfs.account-key")
     azurite_connection_string = f"DefaultEndpointsProtocol=http;AccountName={azurite_account_name};AccountKey={azurite_account_key};BlobEndpoint={azurite_url}/{azurite_account_name};"
     properties = {
-        "connection_string": azurite_connection_string,
-        "account_name": azurite_account_name,
+        "adlfs.connection-string": azurite_connection_string,
+        "adlfs.account-name": azurite_account_name,

Review Comment:
   @Fokko 
   
   I believe the simplest approach would be to utilize connection-string, as it can accommodate various possibilities such as a SAS Token. 
   
   For example Trino/Starburst uses account-name and access-key  Azure Storage.
   Check please Hive Connector with Azure Storage in Trino as reference.
   https://trino.io/docs/current/connector/hive-azure.html#
   
   Based on the information provided, my recommendation would be to allow for both a connection-string or an account-name and account-key to be used. This would provide flexibility for users who may prefer one method over the other.
   
   
   For the Azure Storage URL's there are following possibilites. I guess this is for the other PR relevant.
   
   - abfs
   - abfss
   - wasb 
   - wasbs
   - adl



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] alaturqua commented on a diff in pull request #6970: Python: Fix ADLS configuration

Posted by "alaturqua (via GitHub)" <gi...@apache.org>.
alaturqua commented on code in PR #6970:
URL: https://github.com/apache/iceberg/pull/6970#discussion_r1125007908


##########
python/pyiceberg/io/fsspec.py:
##########
@@ -112,8 +112,15 @@ def _s3(properties: Properties) -> AbstractFileSystem:
 def _adlfs(properties: Properties) -> AbstractFileSystem:
     from adlfs import AzureBlobFileSystem
 
-    fs = AzureBlobFileSystem(**properties)
-    return fs
+    return AzureBlobFileSystem(
+        connection_string=properties.get("adlfs.connection-string"),
+        account_name=properties.get("adlfs.account-name"),
+        account_key=properties.get("adlfs.account-key"),
+        sas_token=properties.get("adlfs.sas-token"),
+        tenant_id=properties.get("adlfs.tenant-id"),
+        client_id=properties.get("adlfs.client-id"),
+        client_secret=properties.get("adlfs.client-secret"),
+    )
 
 
 SCHEME_TO_FS = {

Review Comment:
   I just tested wasb on pandas. Pandas is using adlfs as well. It is not supported:
   
   code:
   ```
   import pandas as pd
   storage_options={'account_key': "<account_key>"}
   
   df = pd.read_parquet('wasb://<container_name@<account_name>.blob.core.windows.net/path_to_file.parquet', storage_options=storage_options)
   df.head()
   ```
   
   output:
   ```
   File [c:\Python\Python39\lib\site-packages\pandas\io\parquet.py:493](file:///C:/Python/Python39/lib/site-packages/pandas/io/parquet.py:493), in read_parquet(path, engine, columns, storage_options, use_nullable_dtypes, **kwargs)
       446 """
       447 Load a parquet object from the file path, returning a DataFrame.
       448 
      (...)
       489 DataFrame
       490 """
       491 impl = get_engine(engine)
   --> 493 return impl.read(
       494     path,
       495     columns=columns,
       496     storage_options=storage_options,
       497     use_nullable_dtypes=use_nullable_dtypes,
       498     **kwargs,
       499 )
   
   File [c:\Python\Python39\lib\site-packages\pandas\io\parquet.py:233](file:///C:/Python/Python39/lib/site-packages/pandas/io/parquet.py:233), in PyArrowImpl.read(self, path, columns, use_nullable_dtypes, storage_options, **kwargs)
   ...
   --> 208         raise ValueError("Protocol not known: %s" % protocol)
       209     bit = known_implementations[protocol]
       210     try:
   
   ValueError: Protocol not known: wasb
   ```
   
   The same works with abfs protocol.
   
   Using following url:
   `abfs://<container_name@<account_name>.dfs.core.windows.net/path_to_file.parquet`
   
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] cccs-eric commented on a diff in pull request #6970: Python: Fix ADLS configuration

Posted by "cccs-eric (via GitHub)" <gi...@apache.org>.
cccs-eric commented on code in PR #6970:
URL: https://github.com/apache/iceberg/pull/6970#discussion_r1124992862


##########
python/pyiceberg/io/fsspec.py:
##########
@@ -112,8 +112,15 @@ def _s3(properties: Properties) -> AbstractFileSystem:
 def _adlfs(properties: Properties) -> AbstractFileSystem:
     from adlfs import AzureBlobFileSystem
 
-    fs = AzureBlobFileSystem(**properties)
-    return fs
+    return AzureBlobFileSystem(
+        connection_string=properties.get("adlfs.connection-string"),
+        account_name=properties.get("adlfs.account-name"),
+        account_key=properties.get("adlfs.account-key"),
+        sas_token=properties.get("adlfs.sas-token"),
+        tenant_id=properties.get("adlfs.tenant-id"),
+        client_id=properties.get("adlfs.client-id"),
+        client_secret=properties.get("adlfs.client-secret"),
+    )
 
 
 SCHEME_TO_FS = {

Review Comment:
   Correct.  And the current implementation of Iceberg only supports `abfs` and `az`.  If we want to add `adl`, we would need to add code that checks for protocol and create an instance of `adlfs.AzureBlobFileSystem` or `adlfs.AzureDatalakeFileSystem`.  Right now we only use `adlfs.AzureBlobFileSystem`.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] Fokko commented on a diff in pull request #6970: Python: Fix ADLS configuration

Posted by "Fokko (via GitHub)" <gi...@apache.org>.
Fokko commented on code in PR #6970:
URL: https://github.com/apache/iceberg/pull/6970#discussion_r1124958773


##########
python/pyiceberg/io/fsspec.py:
##########
@@ -112,8 +112,15 @@ def _s3(properties: Properties) -> AbstractFileSystem:
 def _adlfs(properties: Properties) -> AbstractFileSystem:
     from adlfs import AzureBlobFileSystem
 
-    fs = AzureBlobFileSystem(**properties)
-    return fs
+    return AzureBlobFileSystem(
+        connection_string=properties.get("adlfs.connection-string"),
+        account_name=properties.get("adlfs.account-name"),
+        account_key=properties.get("adlfs.account-key"),
+        sas_token=properties.get("adlfs.sas-token"),
+        tenant_id=properties.get("adlfs.tenant-id"),
+        client_id=properties.get("adlfs.client-id"),
+        client_secret=properties.get("adlfs.client-secret"),
+    )
 
 
 SCHEME_TO_FS = {

Review Comment:
   I've added them, except the `adl` one. Are you in the position to test the `wasb` endpoint?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] Fokko commented on a diff in pull request #6970: Python: Fix ADLS configuration

Posted by "Fokko (via GitHub)" <gi...@apache.org>.
Fokko commented on code in PR #6970:
URL: https://github.com/apache/iceberg/pull/6970#discussion_r1125426311


##########
python/pyiceberg/io/fsspec.py:
##########
@@ -112,8 +112,15 @@ def _s3(properties: Properties) -> AbstractFileSystem:
 def _adlfs(properties: Properties) -> AbstractFileSystem:
     from adlfs import AzureBlobFileSystem
 
-    fs = AzureBlobFileSystem(**properties)
-    return fs
+    return AzureBlobFileSystem(
+        connection_string=properties.get("adlfs.connection-string"),
+        account_name=properties.get("adlfs.account-name"),
+        account_key=properties.get("adlfs.account-key"),
+        sas_token=properties.get("adlfs.sas-token"),
+        tenant_id=properties.get("adlfs.tenant-id"),
+        client_id=properties.get("adlfs.client-id"),
+        client_secret=properties.get("adlfs.client-secret"),
+    )
 
 
 SCHEME_TO_FS = {

Review Comment:
   @alaturqua Thanks for giving this a try. I'll leave out wasb for now, and we can pick that up in a separate PR. If you feel that this is something that's pressing, feel free to raise an issue so we can keep track of it.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] alaturqua commented on a diff in pull request #6970: Python: Fix ADLS configuration

Posted by "alaturqua (via GitHub)" <gi...@apache.org>.
alaturqua commented on code in PR #6970:
URL: https://github.com/apache/iceberg/pull/6970#discussion_r1123016616


##########
python/pyiceberg/io/fsspec.py:
##########
@@ -112,8 +112,15 @@ def _s3(properties: Properties) -> AbstractFileSystem:
 def _adlfs(properties: Properties) -> AbstractFileSystem:
     from adlfs import AzureBlobFileSystem
 
-    fs = AzureBlobFileSystem(**properties)
-    return fs
+    return AzureBlobFileSystem(
+        connection_string=properties.get("adlfs.connection-string"),
+        account_name=properties.get("adlfs.account-name"),
+        account_key=properties.get("adlfs.account-key"),
+        sas_token=properties.get("adlfs.sas-token"),
+        tenant_id=properties.get("adlfs.tenant-id"),
+        client_id=properties.get("adlfs.client-id"),
+        client_secret=properties.get("adlfs.client-secret"),
+    )
 
 
 SCHEME_TO_FS = {

Review Comment:
   @Fokko I would like to suggest adding here WASB/WASBS support as well:
   
   ```
   SCHEME_TO_FS = {
       "s3": _s3,
       "s3a": _s3,
       "s3n": _s3,
       "abfs": _adlfs,
       "abfss": _adlfs,
       "wasb": _adlfs,
       "wasbs":_adlfs
   }
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] Fokko commented on a diff in pull request #6970: Python: Fix ADLS configuration

Posted by "Fokko (via GitHub)" <gi...@apache.org>.
Fokko commented on code in PR #6970:
URL: https://github.com/apache/iceberg/pull/6970#discussion_r1124960556


##########
python/pyiceberg/io/fsspec.py:
##########
@@ -112,8 +112,15 @@ def _s3(properties: Properties) -> AbstractFileSystem:
 def _adlfs(properties: Properties) -> AbstractFileSystem:
     from adlfs import AzureBlobFileSystem
 
-    fs = AzureBlobFileSystem(**properties)
-    return fs
+    return AzureBlobFileSystem(
+        connection_string=properties.get("adlfs.connection-string"),
+        account_name=properties.get("adlfs.account-name"),
+        account_key=properties.get("adlfs.account-key"),
+        sas_token=properties.get("adlfs.sas-token"),
+        tenant_id=properties.get("adlfs.tenant-id"),
+        client_id=properties.get("adlfs.client-id"),
+        client_secret=properties.get("adlfs.client-secret"),
+    )
 
 
 SCHEME_TO_FS = {

Review Comment:
   Hmm, I don't see `wasb` being mentioned in the adlfs repository: https://github.com/search?q=repo%3Afsspec%2Fadlfs%20wasb&type=code



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] cccs-eric commented on a diff in pull request #6970: Python: Fix ADLS configuration

Posted by "cccs-eric (via GitHub)" <gi...@apache.org>.
cccs-eric commented on code in PR #6970:
URL: https://github.com/apache/iceberg/pull/6970#discussion_r1123014890


##########
python/tests/conftest.py:
##########
@@ -1277,8 +1277,8 @@ def adlfs_fsspec_fileio(request: pytest.FixtureRequest) -> Generator[FsspecFileI
     azurite_account_key = request.config.getoption("--adlfs.account-key")
     azurite_connection_string = f"DefaultEndpointsProtocol=http;AccountName={azurite_account_name};AccountKey={azurite_account_key};BlobEndpoint={azurite_url}/{azurite_account_name};"
     properties = {
-        "connection_string": azurite_connection_string,
-        "account_name": azurite_account_name,
+        "adlfs.connection-string": azurite_connection_string,
+        "adlfs.account-name": azurite_account_name,

Review Comment:
   When I wrote that code, I only supported what I could test, which is Azure Gen2 Datalake (or Storage Account).  Gen2 is nice since it provides file system semantics, file-level security, and scale combined with blob storage benefits.  That's why `pyiceberg.io.fsspec` only uses a `AzureBlobFileSystem` instance (which supports `abfs` and `az` protocols).  Could that be extended to support Gen1?  Probably since fsspec supports it, but I wasn't in a position to test it.
   
   As for the required configuration, Azure supports multiple ways to authenticate and that's why I left it open, to support all possibilites:
   ```python
   def _adlfs(properties: Properties) -> AbstractFileSystem:
       from adlfs import AzureBlobFileSystem
   
       fs = AzureBlobFileSystem(**properties)
       return fs
   ```
   Authentication is handled by `AzureBlobFileSystem`, who delegates to [Azure Identity](https://github.com/Azure/azure-sdk-for-python/tree/main/sdk/identity/azure-identity).
   
   But I understand that this does not provide good documentation for a user, so I'm all for listing configuration options.  The question is do we want to limit those options to only a few (so we code checks as @alaturqua suggested), or we provide examples in the documentation and we delegate validation to `AzureBlobFileSystem`?
   
   It can get tricky if we want to code checks for every possibility.  As an example, here is the documentation for `account_key` and `sas_token`:
   ```
       account_key: str
           The storage account key. This is used for shared key authentication.
           If any of account key, sas token or client_id is specified, anonymous access
           will be used.
       sas_token: str
           A shared access signature token to use to authenticate requests
           instead of the account key. If account key and sas token are both
           specified, account key will be used to sign. If any of account key, sas token
           or client_id are specified, anonymous access will be used.
   ```
   I'm not saying it cannot be done, but it requires attention.
   
   I don't know what is Iceberg philosophy when it comes to using an external library, i.e. should Iceberg retain full control or delegation is fine?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] alaturqua commented on a diff in pull request #6970: Python: Fix ADLS configuration

Posted by "alaturqua (via GitHub)" <gi...@apache.org>.
alaturqua commented on code in PR #6970:
URL: https://github.com/apache/iceberg/pull/6970#discussion_r1123016616


##########
python/pyiceberg/io/fsspec.py:
##########
@@ -112,8 +112,15 @@ def _s3(properties: Properties) -> AbstractFileSystem:
 def _adlfs(properties: Properties) -> AbstractFileSystem:
     from adlfs import AzureBlobFileSystem
 
-    fs = AzureBlobFileSystem(**properties)
-    return fs
+    return AzureBlobFileSystem(
+        connection_string=properties.get("adlfs.connection-string"),
+        account_name=properties.get("adlfs.account-name"),
+        account_key=properties.get("adlfs.account-key"),
+        sas_token=properties.get("adlfs.sas-token"),
+        tenant_id=properties.get("adlfs.tenant-id"),
+        client_id=properties.get("adlfs.client-id"),
+        client_secret=properties.get("adlfs.client-secret"),
+    )
 
 
 SCHEME_TO_FS = {

Review Comment:
   @Fokko I would like to suggest adding here WASB/WASBS support as well:
   
   ```
   SCHEME_TO_FS = {
       "s3": _s3,
       "s3a": _s3,
       "s3n": _s3,
       "abfs": _adlfs,
       "abfss": _adlfs,
       "wasb": _adlfs,
       "wasbs": _adlfs
   }
   ```
   
   This requires another endpoint for wasb[s]: 
   <file_system>@<account_name>.blob.core.windows.net.
   
   abfs[s] endpoint is:
    <file_system>@<account_name>.dfs.core.windows.net
   
   Maybe adlfs handles this atomatically
   
   there is another scheme adl for data lake storage gen 1. But we are not using it.
   adl://<data_lake_storage_gen1_name>.azuredatalakestore.net/<path>/<file_name>
   
   
    
    
    
    



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] alaturqua commented on a diff in pull request #6970: Python: Fix ADLS configuration

Posted by "alaturqua (via GitHub)" <gi...@apache.org>.
alaturqua commented on code in PR #6970:
URL: https://github.com/apache/iceberg/pull/6970#discussion_r1125007908


##########
python/pyiceberg/io/fsspec.py:
##########
@@ -112,8 +112,15 @@ def _s3(properties: Properties) -> AbstractFileSystem:
 def _adlfs(properties: Properties) -> AbstractFileSystem:
     from adlfs import AzureBlobFileSystem
 
-    fs = AzureBlobFileSystem(**properties)
-    return fs
+    return AzureBlobFileSystem(
+        connection_string=properties.get("adlfs.connection-string"),
+        account_name=properties.get("adlfs.account-name"),
+        account_key=properties.get("adlfs.account-key"),
+        sas_token=properties.get("adlfs.sas-token"),
+        tenant_id=properties.get("adlfs.tenant-id"),
+        client_id=properties.get("adlfs.client-id"),
+        client_secret=properties.get("adlfs.client-secret"),
+    )
 
 
 SCHEME_TO_FS = {

Review Comment:
   I just tested wasb on pandas. Pandas is using adlfs as well. It is not supported:
   
   code:
   ```
   import pandas as pd
   storage_options={'account_key': "<account_key>"}
   
   df = pd.read_parquet('wasb://<container_name>@<account_name>.blob.core.windows.net/path_to_file.parquet', storage_options=storage_options)
   df.head()
   ```
   
   output:
   ```
   File [c:\Python\Python39\lib\site-packages\pandas\io\parquet.py:493](file:///C:/Python/Python39/lib/site-packages/pandas/io/parquet.py:493), in read_parquet(path, engine, columns, storage_options, use_nullable_dtypes, **kwargs)
       446 """
       447 Load a parquet object from the file path, returning a DataFrame.
       448 
      (...)
       489 DataFrame
       490 """
       491 impl = get_engine(engine)
   --> 493 return impl.read(
       494     path,
       495     columns=columns,
       496     storage_options=storage_options,
       497     use_nullable_dtypes=use_nullable_dtypes,
       498     **kwargs,
       499 )
   
   File [c:\Python\Python39\lib\site-packages\pandas\io\parquet.py:233](file:///C:/Python/Python39/lib/site-packages/pandas/io/parquet.py:233), in PyArrowImpl.read(self, path, columns, use_nullable_dtypes, storage_options, **kwargs)
   ...
   --> 208         raise ValueError("Protocol not known: %s" % protocol)
       209     bit = known_implementations[protocol]
       210     try:
   
   ValueError: Protocol not known: wasb
   ```
   
   The same works with abfs protocol.
   
   Using following url:
   `abfs://<container_name>@<account_name>.dfs.core.windows.net/path_to_file.parquet`
   
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] Fokko commented on a diff in pull request #6970: Python: Fix ADLS configuration

Posted by "Fokko (via GitHub)" <gi...@apache.org>.
Fokko commented on code in PR #6970:
URL: https://github.com/apache/iceberg/pull/6970#discussion_r1125426390


##########
python/tests/conftest.py:
##########
@@ -1277,8 +1277,8 @@ def adlfs_fsspec_fileio(request: pytest.FixtureRequest) -> Generator[FsspecFileI
     azurite_account_key = request.config.getoption("--adlfs.account-key")
     azurite_connection_string = f"DefaultEndpointsProtocol=http;AccountName={azurite_account_name};AccountKey={azurite_account_key};BlobEndpoint={azurite_url}/{azurite_account_name};"
     properties = {
-        "connection_string": azurite_connection_string,
-        "account_name": azurite_account_name,
+        "adlfs.connection-string": azurite_connection_string,
+        "adlfs.account-name": azurite_account_name,

Review Comment:
   I think we're on the same page. I'll merge this PR now, and feel free to open up any follow-up PRs if you feel like it should be done differently.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] alaturqua commented on a diff in pull request #6970: Python: Fix ADLS configuration

Posted by "alaturqua (via GitHub)" <gi...@apache.org>.
alaturqua commented on code in PR #6970:
URL: https://github.com/apache/iceberg/pull/6970#discussion_r1123016616


##########
python/pyiceberg/io/fsspec.py:
##########
@@ -112,8 +112,15 @@ def _s3(properties: Properties) -> AbstractFileSystem:
 def _adlfs(properties: Properties) -> AbstractFileSystem:
     from adlfs import AzureBlobFileSystem
 
-    fs = AzureBlobFileSystem(**properties)
-    return fs
+    return AzureBlobFileSystem(
+        connection_string=properties.get("adlfs.connection-string"),
+        account_name=properties.get("adlfs.account-name"),
+        account_key=properties.get("adlfs.account-key"),
+        sas_token=properties.get("adlfs.sas-token"),
+        tenant_id=properties.get("adlfs.tenant-id"),
+        client_id=properties.get("adlfs.client-id"),
+        client_secret=properties.get("adlfs.client-secret"),
+    )
 
 
 SCHEME_TO_FS = {

Review Comment:
   @Fokko I would like to suggest adding here WASB/WASBS support as well:
   
   ```
   SCHEME_TO_FS = {
       "s3": _s3,
       "s3a": _s3,
       "s3n": _s3,
       "abfs": _adlfs,
       "abfss": _adlfs,
       "wasb": _adlfs,
       "wasbs": _adlfs
   }
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] alaturqua commented on a diff in pull request #6970: Python: Fix ADLS configuration

Posted by "alaturqua (via GitHub)" <gi...@apache.org>.
alaturqua commented on code in PR #6970:
URL: https://github.com/apache/iceberg/pull/6970#discussion_r1123016616


##########
python/pyiceberg/io/fsspec.py:
##########
@@ -112,8 +112,15 @@ def _s3(properties: Properties) -> AbstractFileSystem:
 def _adlfs(properties: Properties) -> AbstractFileSystem:
     from adlfs import AzureBlobFileSystem
 
-    fs = AzureBlobFileSystem(**properties)
-    return fs
+    return AzureBlobFileSystem(
+        connection_string=properties.get("adlfs.connection-string"),
+        account_name=properties.get("adlfs.account-name"),
+        account_key=properties.get("adlfs.account-key"),
+        sas_token=properties.get("adlfs.sas-token"),
+        tenant_id=properties.get("adlfs.tenant-id"),
+        client_id=properties.get("adlfs.client-id"),
+        client_secret=properties.get("adlfs.client-secret"),
+    )
 
 
 SCHEME_TO_FS = {

Review Comment:
   @Fokko I would like to suggest adding here WASB/WASBS support as well:
   
   ```
   SCHEME_TO_FS = {
       "s3": _s3,
       "s3a": _s3,
       "s3n": _s3,
       "abfs": _adlfs,
       "abfss": _adlfs,
       "wasb": _adlfs,
       "wasbs": _adlfs
   }
   ```
   
   This requires another endpoint for wasb[s]: 
   wasb[s]://<file_system>@<account_name>.blob.core.windows.net.
   
   abfs[s] endpoint is:
    abfs[s]://<file_system>@<account_name>.dfs.core.windows.net
   
   Maybe adlfs handles this atomatically
   
   there is another scheme adl for data lake storage gen 1. But we are not using it.
   adl://<data_lake_storage_gen1_name>.azuredatalakestore.net/<path>/<file_name>
   
   
    
    
    
    



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] alaturqua commented on a diff in pull request #6970: Python: Fix ADLS configuration

Posted by "alaturqua (via GitHub)" <gi...@apache.org>.
alaturqua commented on code in PR #6970:
URL: https://github.com/apache/iceberg/pull/6970#discussion_r1125007908


##########
python/pyiceberg/io/fsspec.py:
##########
@@ -112,8 +112,15 @@ def _s3(properties: Properties) -> AbstractFileSystem:
 def _adlfs(properties: Properties) -> AbstractFileSystem:
     from adlfs import AzureBlobFileSystem
 
-    fs = AzureBlobFileSystem(**properties)
-    return fs
+    return AzureBlobFileSystem(
+        connection_string=properties.get("adlfs.connection-string"),
+        account_name=properties.get("adlfs.account-name"),
+        account_key=properties.get("adlfs.account-key"),
+        sas_token=properties.get("adlfs.sas-token"),
+        tenant_id=properties.get("adlfs.tenant-id"),
+        client_id=properties.get("adlfs.client-id"),
+        client_secret=properties.get("adlfs.client-secret"),
+    )
 
 
 SCHEME_TO_FS = {

Review Comment:
   I just tested wasb on pandas. Pandas is using adlfs as well. It is not supported:
   
   code:
   ```
   import pandas as pd
   storage_options={'account_key': "<account_key>"}
   
   df = pd.read_parquet('wasb://<container_name>@<account_name>.blob.core.windows.net/path/to/file.parquet', storage_options=storage_options)
   df.head()
   ```
   
   output:
   ```
   File [c:\Python\Python39\lib\site-packages\pandas\io\parquet.py:493](file:///C:/Python/Python39/lib/site-packages/pandas/io/parquet.py:493), in read_parquet(path, engine, columns, storage_options, use_nullable_dtypes, **kwargs)
       446 """
       447 Load a parquet object from the file path, returning a DataFrame.
       448 
      (...)
       489 DataFrame
       490 """
       491 impl = get_engine(engine)
   --> 493 return impl.read(
       494     path,
       495     columns=columns,
       496     storage_options=storage_options,
       497     use_nullable_dtypes=use_nullable_dtypes,
       498     **kwargs,
       499 )
   
   File [c:\Python\Python39\lib\site-packages\pandas\io\parquet.py:233](file:///C:/Python/Python39/lib/site-packages/pandas/io/parquet.py:233), in PyArrowImpl.read(self, path, columns, use_nullable_dtypes, storage_options, **kwargs)
   ...
   --> 208         raise ValueError("Protocol not known: %s" % protocol)
       209     bit = known_implementations[protocol]
       210     try:
   
   ValueError: Protocol not known: wasb
   ```
   
   The same works with abfs protocol.
   
   Using following url:
   `abfs://<container_name>@<account_name>.dfs.core.windows.net/path/to/file.parquet`
   
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] Fokko commented on a diff in pull request #6970: Python: Fix ADLS configuration

Posted by "Fokko (via GitHub)" <gi...@apache.org>.
Fokko commented on code in PR #6970:
URL: https://github.com/apache/iceberg/pull/6970#discussion_r1122680469


##########
python/tests/conftest.py:
##########
@@ -1277,8 +1277,8 @@ def adlfs_fsspec_fileio(request: pytest.FixtureRequest) -> Generator[FsspecFileI
     azurite_account_key = request.config.getoption("--adlfs.account-key")
     azurite_connection_string = f"DefaultEndpointsProtocol=http;AccountName={azurite_account_name};AccountKey={azurite_account_key};BlobEndpoint={azurite_url}/{azurite_account_name};"
     properties = {
-        "connection_string": azurite_connection_string,
-        "account_name": azurite_account_name,
+        "adlfs.connection-string": azurite_connection_string,
+        "adlfs.account-name": azurite_account_name,

Review Comment:
   What would be your suggestion? I noticed that you can't set `endpoint` using the constructor. This can only be set using the connection string.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] cccs-eric commented on a diff in pull request #6970: Python: Fix ADLS configuration

Posted by "cccs-eric (via GitHub)" <gi...@apache.org>.
cccs-eric commented on code in PR #6970:
URL: https://github.com/apache/iceberg/pull/6970#discussion_r1121690588


##########
python/tests/conftest.py:
##########
@@ -1277,8 +1277,8 @@ def adlfs_fsspec_fileio(request: pytest.FixtureRequest) -> Generator[FsspecFileI
     azurite_account_key = request.config.getoption("--adlfs.account-key")
     azurite_connection_string = f"DefaultEndpointsProtocol=http;AccountName={azurite_account_name};AccountKey={azurite_account_key};BlobEndpoint={azurite_url}/{azurite_account_name};"
     properties = {
-        "connection_string": azurite_connection_string,
-        "account_name": azurite_account_name,
+        "adlfs.connection-string": azurite_connection_string,
+        "adlfs.account-name": azurite_account_name,

Review Comment:
   This is likely not needed since `connection-string` already contains the account name



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] alaturqua commented on a diff in pull request #6970: Python: Fix ADLS configuration

Posted by "alaturqua (via GitHub)" <gi...@apache.org>.
alaturqua commented on code in PR #6970:
URL: https://github.com/apache/iceberg/pull/6970#discussion_r1124985772


##########
python/pyiceberg/io/fsspec.py:
##########
@@ -112,8 +112,15 @@ def _s3(properties: Properties) -> AbstractFileSystem:
 def _adlfs(properties: Properties) -> AbstractFileSystem:
     from adlfs import AzureBlobFileSystem
 
-    fs = AzureBlobFileSystem(**properties)
-    return fs
+    return AzureBlobFileSystem(
+        connection_string=properties.get("adlfs.connection-string"),
+        account_name=properties.get("adlfs.account-name"),
+        account_key=properties.get("adlfs.account-key"),
+        sas_token=properties.get("adlfs.sas-token"),
+        tenant_id=properties.get("adlfs.tenant-id"),
+        client_id=properties.get("adlfs.client-id"),
+        client_secret=properties.get("adlfs.client-secret"),
+    )
 
 
 SCHEME_TO_FS = {

Review Comment:
   I think your changes are not pushed yet.
   
   It looks like they only support:
   
   - az
   - abfs
   - adl
   
   https://github.com/fsspec/adlfs/blob/2d56e411906519f31ee614e601469f67845b41bd/docs/index.md#fsspec-protocols
   
   I created an issue with question on their repository:
   https://github.com/fsspec/adlfs/issues/403
   
   I don't know, if they are supporting wasb implicitly. Would need to test. I could take a look tomorrow or on monday.
   
   
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] alaturqua commented on a diff in pull request #6970: Python: Fix ADLS configuration

Posted by "alaturqua (via GitHub)" <gi...@apache.org>.
alaturqua commented on code in PR #6970:
URL: https://github.com/apache/iceberg/pull/6970#discussion_r1124985772


##########
python/pyiceberg/io/fsspec.py:
##########
@@ -112,8 +112,15 @@ def _s3(properties: Properties) -> AbstractFileSystem:
 def _adlfs(properties: Properties) -> AbstractFileSystem:
     from adlfs import AzureBlobFileSystem
 
-    fs = AzureBlobFileSystem(**properties)
-    return fs
+    return AzureBlobFileSystem(
+        connection_string=properties.get("adlfs.connection-string"),
+        account_name=properties.get("adlfs.account-name"),
+        account_key=properties.get("adlfs.account-key"),
+        sas_token=properties.get("adlfs.sas-token"),
+        tenant_id=properties.get("adlfs.tenant-id"),
+        client_id=properties.get("adlfs.client-id"),
+        client_secret=properties.get("adlfs.client-secret"),
+    )
 
 
 SCHEME_TO_FS = {

Review Comment:
   I think your changes are not pushed yet.
   
   It looks like they only support:
   
   - az
   - abfs
   - adl
   
   https://github.com/fsspec/adlfs/blob/2d56e411906519f31ee614e601469f67845b41bd/docs/index.md#fsspec-protocols
   
   I don't know, if they are supporting wasb implicitly. Would need to test. I could take a look tomorrow or on monday.
   
   
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] cccs-eric commented on a diff in pull request #6970: Python: Fix ADLS configuration

Posted by "cccs-eric (via GitHub)" <gi...@apache.org>.
cccs-eric commented on code in PR #6970:
URL: https://github.com/apache/iceberg/pull/6970#discussion_r1124992862


##########
python/pyiceberg/io/fsspec.py:
##########
@@ -112,8 +112,15 @@ def _s3(properties: Properties) -> AbstractFileSystem:
 def _adlfs(properties: Properties) -> AbstractFileSystem:
     from adlfs import AzureBlobFileSystem
 
-    fs = AzureBlobFileSystem(**properties)
-    return fs
+    return AzureBlobFileSystem(
+        connection_string=properties.get("adlfs.connection-string"),
+        account_name=properties.get("adlfs.account-name"),
+        account_key=properties.get("adlfs.account-key"),
+        sas_token=properties.get("adlfs.sas-token"),
+        tenant_id=properties.get("adlfs.tenant-id"),
+        client_id=properties.get("adlfs.client-id"),
+        client_secret=properties.get("adlfs.client-secret"),
+    )
 
 
 SCHEME_TO_FS = {

Review Comment:
   Correct.  And the current implementation of PyIceberg only supports `abfs` and `az`.  If we want to add `adl`, we would need to add code that checks for protocol and create an instance of `adlfs.AzureBlobFileSystem` or `adlfs.AzureDatalakeFileSystem`.  Right now we only use `adlfs.AzureBlobFileSystem`.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] cccs-eric commented on pull request #6970: Python: Fix ADLS configuration

Posted by "cccs-eric (via GitHub)" <gi...@apache.org>.
cccs-eric commented on PR #6970:
URL: https://github.com/apache/iceberg/pull/6970#issuecomment-1450109465

   Thanks for pointing this out @Fokko .  IMHO, if we are going to list `connection-string` as an ADLS options in Iceberg's documentation, maybe we should add some more details about it.  There is a catch with this one since it will overwrite other values like `account-name` (see https://github.com/fsspec/adlfs/blob/f15c37a43afd87a04f01b61cd90294dd57181e1d/adlfs/spec.py#L350-L354).  So a user could be mislead by configuring both `connection-string` and `account-name` with different values.  If I'm right, conftest.py could be changed to only configure `connection-string`.  Not sure if we should copy AzureBlobFileSystem's doc over here, or simply link to it.
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] Fokko commented on a diff in pull request #6970: Python: Fix ADLS configuration

Posted by "Fokko (via GitHub)" <gi...@apache.org>.
Fokko commented on code in PR #6970:
URL: https://github.com/apache/iceberg/pull/6970#discussion_r1122680469


##########
python/tests/conftest.py:
##########
@@ -1277,8 +1277,8 @@ def adlfs_fsspec_fileio(request: pytest.FixtureRequest) -> Generator[FsspecFileI
     azurite_account_key = request.config.getoption("--adlfs.account-key")
     azurite_connection_string = f"DefaultEndpointsProtocol=http;AccountName={azurite_account_name};AccountKey={azurite_account_key};BlobEndpoint={azurite_url}/{azurite_account_name};"
     properties = {
-        "connection_string": azurite_connection_string,
-        "account_name": azurite_account_name,
+        "adlfs.connection-string": azurite_connection_string,
+        "adlfs.account-name": azurite_account_name,

Review Comment:
   What would be your suggestion? I'm less of an Azure user than you :)
   
   I noticed that you can't set `endpoint` using the constructor. This can only be set using the connection string. So the connection string seems to offer more options.
   
   cc @alaturqua



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] alaturqua commented on a diff in pull request #6970: Python: Fix ADLS configuration

Posted by "alaturqua (via GitHub)" <gi...@apache.org>.
alaturqua commented on code in PR #6970:
URL: https://github.com/apache/iceberg/pull/6970#discussion_r1122763057


##########
python/tests/conftest.py:
##########
@@ -1277,8 +1277,8 @@ def adlfs_fsspec_fileio(request: pytest.FixtureRequest) -> Generator[FsspecFileI
     azurite_account_key = request.config.getoption("--adlfs.account-key")
     azurite_connection_string = f"DefaultEndpointsProtocol=http;AccountName={azurite_account_name};AccountKey={azurite_account_key};BlobEndpoint={azurite_url}/{azurite_account_name};"
     properties = {
-        "connection_string": azurite_connection_string,
-        "account_name": azurite_account_name,
+        "adlfs.connection-string": azurite_connection_string,
+        "adlfs.account-name": azurite_account_name,

Review Comment:
   @Fokko 
   
   I believe the simplest approach would be to utilize connection-string, as it can accommodate various possibilities such as a SAS Token. 
   
   For example Trino/Starburst uses account-name and access-key  Azure Storage.
   Check please Hive Connector with Azure Storage in Trino as reference.
   https://trino.io/docs/current/connector/hive-azure.html#
   
   Based on the information provided, my recommendation would be to allow for both a connection-string or an account-name and account-key to be used. This would provide flexibility for users who may prefer one method over the other. The implementation could check and force to use one of them either connection-string or account-name + accunt-key.
   
   
   For the Azure Storage URL's there are following possibilites. I guess this is for the other PR relevant.
   
   - abfs
   - abfss
   - wasb 
   - wasbs
   - adl



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] Fokko commented on a diff in pull request #6970: Python: Fix ADLS configuration

Posted by "Fokko (via GitHub)" <gi...@apache.org>.
Fokko commented on code in PR #6970:
URL: https://github.com/apache/iceberg/pull/6970#discussion_r1124957788


##########
python/tests/conftest.py:
##########
@@ -1277,8 +1277,8 @@ def adlfs_fsspec_fileio(request: pytest.FixtureRequest) -> Generator[FsspecFileI
     azurite_account_key = request.config.getoption("--adlfs.account-key")
     azurite_connection_string = f"DefaultEndpointsProtocol=http;AccountName={azurite_account_name};AccountKey={azurite_account_key};BlobEndpoint={azurite_url}/{azurite_account_name};"
     properties = {
-        "connection_string": azurite_connection_string,
-        "account_name": azurite_account_name,
+        "adlfs.connection-string": azurite_connection_string,
+        "adlfs.account-name": azurite_account_name,

Review Comment:
   Thanks for the responses @alaturqua and @cccs-eric 
   
   > The implementation could check and force to use one of them either connection-string or account-name + accunt-key.
   
   This is also how it originally was intended, I'm all for it.
   
   > For the Azure Storage URL's there are following possibilites. I guess this is for the other PR relevant.
   
   I've added those, did you have a chance of testing these?
   
   > I don't know what is Iceberg philosophy when it comes to using an external library, i.e. should Iceberg retain full control or delegation is fine?
   
   We are liberal in the options that we accept, but we want to unify them. For example, for S3 there are two FileIO's, PyArrow and S3FS. You can switch between them without having to change your configuration. Therefore we have to do some mapping between the names.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] Fokko merged pull request #6970: Python: Fix ADLS configuration

Posted by "Fokko (via GitHub)" <gi...@apache.org>.
Fokko merged PR #6970:
URL: https://github.com/apache/iceberg/pull/6970


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org