You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@opendal.apache.org by "BaurzhanSakhariev (via GitHub)" <gi...@apache.org> on 2024/12/16 16:27:55 UTC
BlockCountExceedsLimit on azblob when uploading large files
GitHub user BaurzhanSakhariev created a discussion: BlockCountExceedsLimit on azblob when uploading large files
Hi, I have a question about uploading large files (>5GB)
We use OpenDAL java binding 0.46.3 and `getting “BlockCountExceedsLimit”, message: “The uncommitted block count cannot exceed the maximum limit of 100,000 blocks. ` (https://github.com/crate/crate/issues/17136)
I couldn't create an isolated small test. Tried modifying `BlockingWriteTest` to repeat write 100K times and also played around with the size of the generated data but tests passes.
https://learn.microsoft.com/en-us/troubleshoot/azure/azure-storage/blobs/connectivity/invalid-blob-or-block-content-or-invalid-block-list suggested to mitigate that by committing blocks (put block API)
Looks like it has been implemented already in https://github.com/apache/opendal/pull/4181 and released in 0.45.1 so version we use (0.46.3) includes it.
Is it a bug or I'm missing some configuration parameter for azblob?
GitHub link: https://github.com/apache/opendal/discussions/5420
----
This is an automatically sent email for dev@opendal.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@opendal.apache.org
Re: BlockCountExceedsLimit on azblob when uploading large files
Posted by "BaurzhanSakhariev (via GitHub)" <gi...@apache.org>.
GitHub user BaurzhanSakhariev added a comment to the discussion: BlockCountExceedsLimit on azblob when uploading large files
Opened a feature request: https://github.com/apache/opendal/issues/5421
GitHub link: https://github.com/apache/opendal/discussions/5420#discussioncomment-11598671
----
This is an automatically sent email for dev@opendal.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@opendal.apache.org
Re: BlockCountExceedsLimit on azblob when uploading large files
Posted by "BaurzhanSakhariev (via GitHub)" <gi...@apache.org>.
GitHub user BaurzhanSakhariev edited a comment on the discussion: BlockCountExceedsLimit on azblob when uploading large files
>I can patch OperatorOutputStream
Hm, MAX_BYTES is private and used across public methods, in order to have a custom chunk size I need to copy the whole implementation (which is fine for an interim solution). Delegate is not an option as it will use hardcoded 16KiB.
To create a copy, I also need to copy a constructor with
`final long op = operator.nativeHandle ` which doesn't work as it's not public.
I looked at public API (including duplicate) but didn't find a way to get a `nativeHandle` so that I can expose it and further create custom copy.
Is there public API to get the `nativeHandle` (maybe I missed it)?
GitHub link: https://github.com/apache/opendal/discussions/5420#discussioncomment-11598514
----
This is an automatically sent email for dev@opendal.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@opendal.apache.org
Re: BlockCountExceedsLimit on azblob when uploading large files
Posted by "BaurzhanSakhariev (via GitHub)" <gi...@apache.org>.
GitHub user BaurzhanSakhariev edited a comment on the discussion: BlockCountExceedsLimit on azblob when uploading large files
>I can patch OperatorOutputStream
Hm, MAX_BYTES is private and used across public methods, in order to have a custom chunk size I need to copy the whole implementation (which is fine for an interim solution). Delegate is not an option as it will use hardcoded 16KiB.
To create a copy, I also need to copy a constructor with
`final long op = operator.nativeHandle ` which doesn't work as it's not public.
I looked at public API (including duplicate) but didn't find a way to get a `nativeHandle` so that I can expose it and further create custom copy.
Is there public API to get the `nativeHandle` (maybe I missed it)?
UPD: I think I can use reflection for now to provide a custom value.
Opened a feature request: https://github.com/apache/opendal/issues/5421
GitHub link: https://github.com/apache/opendal/discussions/5420#discussioncomment-11598514
----
This is an automatically sent email for dev@opendal.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@opendal.apache.org
Re: BlockCountExceedsLimit on azblob when uploading large files
Posted by "BaurzhanSakhariev (via GitHub)" <gi...@apache.org>.
GitHub user BaurzhanSakhariev deleted a comment on the discussion: BlockCountExceedsLimit on azblob when uploading large files
>I can patch OperatorOutputStream
Hm, MAX_BYTES is private and used across public methods, in order to have a custom chunk size I need to copy the whole implementation (which is fine for an interim solution). Delegate is not an option as it will use hardcoded 16KiB.
To create a copy, I also need to copy a constructor with
`final long op = operator.nativeHandle ` which doesn't work as it's not public.
I looked at public API (including duplicate) but didn't find a way to get a `nativeHandle` so that I can expose it and further create custom copy.
Is there public API to get the `nativeHandle` (maybe I missed it)?
UPD: I think I can use reflection for now to provide a custom value.
Opened a feature request: https://github.com/apache/opendal/issues/5421
GitHub link: https://github.com/apache/opendal/discussions/5420#discussioncomment-11598514
----
This is an automatically sent email for dev@opendal.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@opendal.apache.org
Re: BlockCountExceedsLimit on azblob when uploading large files
Posted by "BaurzhanSakhariev (via GitHub)" <gi...@apache.org>.
GitHub user BaurzhanSakhariev edited a comment on the discussion: BlockCountExceedsLimit on azblob when uploading large files
>I can patch OperatorOutputStream
Hm, MAX_BYTES is private and used across public methods, in order to have a custom chunk size I need to copy the whole implementation (which is fine for an interim solution). Delegate is not an option as it will use hardcoded 16KiB.
To create a copy, I also need to copy a constructor with
`final long op = operator.nativeHandle ` which doesn't work as it's not public.
I looked at public API (including duplicate) but didn't find a way to get a `nativeHandle` so that I can expose it and further create custom copy.
Is there public API to get the `nativeHandle` (maybe I missed it)?
Opened a feature request: https://github.com/apache/opendal/issues/5421
GitHub link: https://github.com/apache/opendal/discussions/5420#discussioncomment-11598514
----
This is an automatically sent email for dev@opendal.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@opendal.apache.org
Re: BlockCountExceedsLimit on azblob when uploading large files
Posted by "BaurzhanSakhariev (via GitHub)" <gi...@apache.org>.
GitHub user BaurzhanSakhariev added a comment to the discussion: BlockCountExceedsLimit on azblob when uploading large files
Hi @Xuanwo, thanks for the pointers, looks like I can patch `OperatorOutputStream` locally as an interim solution.
I will give it a try and report back.
GitHub link: https://github.com/apache/opendal/discussions/5420#discussioncomment-11590392
----
This is an automatically sent email for dev@opendal.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@opendal.apache.org
Re: BlockCountExceedsLimit on azblob when uploading large files
Posted by "BaurzhanSakhariev (via GitHub)" <gi...@apache.org>.
GitHub user BaurzhanSakhariev added a comment to the discussion: BlockCountExceedsLimit on azblob when uploading large files
>I can patch OperatorOutputStream
Hm, MAX_BYTES is private and used across public methods, in order to have a custom chunk size I need to copy the whole implementation (which is fine for an interim solution).
To create a copy, I also need to copy a constructor with
`final long op = operator.nativeHandle ` which doesn't work as it's not public.
I looked at public API (including duplicate) but didn't find a way to get a `nativeHandle` so that I can expose it and further create custom copy.
Is there public API to get the `nativeHandle` (maybe I missed it)?
GitHub link: https://github.com/apache/opendal/discussions/5420#discussioncomment-11598514
----
This is an automatically sent email for dev@opendal.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@opendal.apache.org
Re: BlockCountExceedsLimit on azblob when uploading large files
Posted by "Xuanwo (via GitHub)" <gi...@apache.org>.
GitHub user Xuanwo added a comment to the discussion: BlockCountExceedsLimit on azblob when uploading large files
Hi, it looks like we didn't expose the `chunk` API to java side: https://github.com/apache/opendal/blob/e9df3ebf832317a3d732c61e0e9e6733f8b91325/bindings/java/src/operator_output_stream.rs#L44-L52
Instead, we expose a very small chunk `16KiB`: https://github.com/apache/opendal/blob/e9df3ebf832317a3d732c61e0e9e6733f8b91325/bindings/java/src/main/java/org/apache/opendal/OperatorOutputStream.java#L38-L41
We will need to expose https://docs.rs/opendal/0.50.2/opendal/operator_functions/struct.FunctionWriter.html#method.chunk to java side so that you can configure it to 8MiB or 16MiB instead.
Also cc @tisonkun @G-XD for ideas.
GitHub link: https://github.com/apache/opendal/discussions/5420#discussioncomment-11583389
----
This is an automatically sent email for dev@opendal.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@opendal.apache.org