You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "njriasan (via GitHub)" <gi...@apache.org> on 2024/04/12 00:05:39 UTC
[I] Enable reading WASB and WASBS file paths with ABFS and ABFSS [iceberg]
njriasan opened a new issue, #10127:
URL: https://github.com/apache/iceberg/issues/10127
### Feature Request / Improvement
When you setup a managed Snowflake Iceberg table on an Azure account, they will provide locations that use `wasbs://` and not `abfss://`. `wasb` is currently deprecated by Azure and everyone is encouraged to use ABFS instead. While Snowflake should really change this behavior, in the spirit of allowing people to "update" Iceberg tables without rewriting all the metadata files, it would be great if the iceberg library could handle this automatically.
Its my understanding that you can convert a `wasbs` URI to an `abfss` URI by just making two changes:
- Replacing `wasbs://` with `abfss://`
- Updating `blob.core.windows.net` with `dfs.core.windows.net`.
If this change could be replaced when loading the location and file information from metadata files, then every user could effortlessly update to using `abfs`.
### Query engine
Snowflake
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Enable reading WASB and WASBS file paths with ABFS and ABFSS [iceberg]
Posted by "njriasan (via GitHub)" <gi...@apache.org>.
njriasan commented on issue #10127:
URL: https://github.com/apache/iceberg/issues/10127#issuecomment-2050746955
I'm also interesting in making this contribution to Iceberg if others are okay with this change and can give me some pointers about the best way to ensure this can be applied to every file path.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Enable reading WASB and WASBS file paths with ABFS and ABFSS [iceberg]
Posted by "RussellSpitzer (via GitHub)" <gi...@apache.org>.
RussellSpitzer closed issue #10127: Enable reading WASB and WASBS file paths with ABFS and ABFSS
URL: https://github.com/apache/iceberg/issues/10127
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Enable reading WASB and WASBS file paths with ABFS and ABFSS [iceberg]
Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.
kevinjqliu commented on issue #10127:
URL: https://github.com/apache/iceberg/issues/10127#issuecomment-3189306663
FYI #11830 resolves this issued
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Enable reading WASB and WASBS file paths with ABFS and ABFSS [iceberg]
Posted by "ms1111 (via GitHub)" <gi...@apache.org>.
ms1111 commented on issue #10127:
URL: https://github.com/apache/iceberg/issues/10127#issuecomment-2065633615
Is there a difference in that wasbs is for blob storage, and abfss requires hierarchical storage (ADLSv2) to be enabled? If you try to read a blob storage account using the ADLS SDK, certain things will not work.
([iceberg-azure](https://github.com/apache/iceberg/tree/main/azure/src/main/java/org/apache/iceberg/azure) seems to require abfss & ADLSv2. I was actually wishing it'd work with wasbs, using the blob storage APIs, because ADLSv2 doesn't support automatic blob versioning.)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Enable reading WASB and WASBS file paths with ABFS and ABFSS [iceberg]
Posted by "njriasan (via GitHub)" <gi...@apache.org>.
njriasan commented on issue #10127:
URL: https://github.com/apache/iceberg/issues/10127#issuecomment-2050746432
I marked this as Snowflake because this is the offending writer, but I think this generally would apply to all engines.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Enable reading WASB and WASBS file paths with ABFS and ABFSS [iceberg]
Posted by "liko9 (via GitHub)" <gi...@apache.org>.
liko9 commented on issue #10127:
URL: https://github.com/apache/iceberg/issues/10127#issuecomment-3189322127
I'm concerned that this allows the mixing of paths and protocols. The current implementation in main allows a location specified as abfss://container@blob.core.windows.net to work - but the MS Azure approach is that abfss is for ADLSv2 via dfs.core.windows.net.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Enable reading WASB and WASBS file paths with ABFS and ABFSS [iceberg]
Posted by "dennishuo (via GitHub)" <gi...@apache.org>.
dennishuo commented on issue #10127:
URL: https://github.com/apache/iceberg/issues/10127#issuecomment-2080319607
A cursory check through https://github.com/apache/iceberg/blob/main/azure/src/main/java/org/apache/iceberg/azure/adlsv2 seems to only reveal three methods used:
- `fileClient.getProperties`: https://github.com/Azure/azure-sdk-for-java/blob/0aa45226a625aa19da7183800bb90531eb1f1ee2/sdk/storage/azure-storage-file-datalake/src/main/java/com/azure/storage/file/datalake/DataLakePathClient.java#L1206
- `fileClient.openInputStream`: https://github.com/Azure/azure-sdk-for-java/blob/0aa45226a625aa19da7183800bb90531eb1f1ee2/sdk/storage/azure-storage-file-datalake/src/main/java/com/azure/storage/file/datalake/DataLakeFileClient.java#L1139
- `fileClient.getOutputStream`: https://github.com/Azure/azure-sdk-for-java/blob/0aa45226a625aa19da7183800bb90531eb1f1ee2/sdk/storage/azure-storage-file-datalake/src/main/java/com/azure/storage/file/datalake/DataLakeFileClient.java#L1189
And all three appear to just delegate through to the internal `blockBlobClient` instead of touching `dataLakeFileAsyncClient`, so it does seem like simply accepting `wasbs` in `ADLSFileIO` should be sufficient. Would be good to have a second set of eyes confirm though.
It could also be safest to just make `ADLSLocation` permissive in accepting `wasbs://` without changing `ResolvingFileIO` if people want `wasbs://` to still automatically fallthrough to HadoopFileIO in usual scenarios.
@njriasan I'm happy to help review a PR if you wanted to take a stab at this (we'll naturally still need a committer to also review).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Enable reading WASB and WASBS file paths with ABFS and ABFSS [iceberg]
Posted by "liko9 (via GitHub)" <gi...@apache.org>.
liko9 commented on issue #10127:
URL: https://github.com/apache/iceberg/issues/10127#issuecomment-2452011720
Should we reopen this issue since it [was reverted](https://github.com/apache/iceberg/pull/11344) and still exists as a problem?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Enable reading WASB and WASBS file paths with ABFS and ABFSS [iceberg]
Posted by "dennishuo (via GitHub)" <gi...@apache.org>.
dennishuo commented on issue #10127:
URL: https://github.com/apache/iceberg/issues/10127#issuecomment-2080302481
@ms1111 raises a good point, as there are some known incompatibilities in low-level Blob vs ADLS APIs: https://learn.microsoft.com/en-us/azure/storage/blobs/data-lake-storage-known-issues
However, it looks like the discrepancies may or may not be fundamental to Iceberg/Hadoop use cases depending on what subset of each APIs the different client-side implementations use.
It looks like while the legacy Hadoop `wasbs` impl is being deprecated, some of the underlying Blob APIs are still being used both by the legacy `BlobClient` as well as newer `DataLake*Client`. For example, the `DataLakeFileSystemClientBuilder` appears to always simultaneously create both an internal "datalake client" as well as a "blob client", pointing respectively at `dfs.core.windows.net` and `blob.core.windows.net`: https://github.com/Azure/azure-sdk-for-java/blob/0aa45226a625aa19da7183800bb90531eb1f1ee2/sdk/storage/azure-storage-file-datalake/src/main/java/com/azure/storage/file/datalake/DataLakeFileSystemClientBuilder.java#L175
public DataLakeFileSystemClientBuilder endpoint(String endpoint) {
// Ensure endpoint provided is dfs endpoint
endpoint = DataLakeImplUtils.endpointToDesiredEndpoint(endpoint, "dfs", "blob");
blobContainerClientBuilder.endpoint(DataLakeImplUtils.endpointToDesiredEndpoint(endpoint, "blob", "dfs"));
And the helper function doesn't really care if the input is `dfs` or `blob` already, it just constructs with the desired one: https://github.com/Azure/azure-sdk-for-java/blob/21eb8bc8b4cce3e365bbcb9d139b07a3a554a2d9/sdk/storage/azure-storage-file-datalake/src/main/java/com/azure/storage/file/datalake/implementation/util/DataLakeImplUtils.java#L16
public static String endpointToDesiredEndpoint(String endpoint, String desiredEndpoint, String currentEndpoint) {
// Add the . on either side to prevent overwriting an account name.
String desiredStringToMatch = "." + desiredEndpoint + ".";
String currentRegexToMatch = "\\." + currentEndpoint + "\\.";
if (endpoint.contains(desiredStringToMatch)) {
return endpoint;
} else {
return endpoint.replaceFirst(currentRegexToMatch, desiredStringToMatch);
}
}
And then some of the methods just delegate through to the "blob client" instead of the "datalake client": https://github.com/Azure/azure-sdk-for-java/blob/0aa45226a625aa19da7183800bb90531eb1f1ee2/sdk/storage/azure-storage-file-datalake/src/main/java/com/azure/storage/file/datalake/DataLakeFileClient.java#L1079
public FileReadResponse readWithResponse(OutputStream stream, FileRange range, DownloadRetryOptions options,
DataLakeRequestConditions requestConditions, boolean getRangeContentMd5, Duration timeout, Context context) {
return DataLakeImplUtils.returnOrConvertException(() -> {
BlobDownloadResponse response = blockBlobClient.downloadWithResponse(stream, Transforms.toBlobRange(range),
Transforms.toBlobDownloadRetryOptions(options), Transforms.toBlobRequestConditions(requestConditions),
getRangeContentMd5, timeout, context);
return Transforms.toFileReadResponse(response);
}, LOGGER);
}
While it'll be important for all service providers to also migrate to no longer producing `wasbs://` paths by default and ensuring users all have ADLSv2 enabled, it seems to me the Iceberg metadata/manifest-list/manifest files will be fairly sticky for lots of large Iceberg deployments with exabytes of data involved, so even if new data is migrated quickly, the libraries really need to support ingesting `wasbs://` scheme for the foreseeable future.
Are there any Azure experts who can confirm that `ADLSFileIO` strictly adheres to the subset of `DataLakeFileSystemClient` that only depends on the `Blob` semantics? If so, could that mean it's possible to use `DataLakeFileSystemClient` even for blob storage accounts that don't enable ADLSv2?
If it'll be guaranteed to remain drop-in compatible, it seems like one approach could be to include `wasbs://` in `ResolvingFileIO` to also map to `ADLSFileIO` and then make `ADLSLocation` more permissive to accept `wasbs://` URIs. Looks like it may not technically even need to do any explicit path-translation to `abfss` since the scheme prefix `abfss` is discarded anyways, and the code I pasted above from Azure client libraries seems to not actually care too much about whether you're trying to configure the client with `dfs.core.windows.net` or `blob.core.windows.net`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Enable reading WASB and WASBS file paths with ABFS and ABFSS [iceberg]
Posted by "musmanir (via GitHub)" <gi...@apache.org>.
musmanir commented on issue #10127:
URL: https://github.com/apache/iceberg/issues/10127#issuecomment-3262274236
> I'm concerned that this allows the mixing of paths and protocols. The current implementation in main allows a location specified as abfss://[container@blob.core.windows.net](mailto:container@blob.core.windows.net) to work - but the MS Azure approach is that abfss is for ADLSv2 via dfs.core.windows.net.
I am seeing this problem already. I tried to read snowflake managed iceberg tables from Trino. And trno is complaining that metadata file should contain `.dfs.core.windows.net` but instead it contains `.blob.core.windows.net`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org