You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@beam.apache.org by 池上有希乃 <ik...@cloud-ace.jp> on 2021/12/21 01:50:12 UTC
Is FileIO for Azure Blob Storage GA?
Hi
I think I will use Apache Beam for my business.
So there is a question.
Whether FileIO for Azure Blob Storage is GA phase or not?
Thanks,
Yukino Ikegami
Re: Is FileIO for Azure Blob Storage GA?
Posted by Piper H <po...@gmail.com>.
池上 有希乃
wait for your good news. :)
Piper
On Mon, Dec 27, 2021 at 9:33 AM 池上有希乃 <ik...@cloud-ace.jp> wrote:
> Hello Yu, Brian,
>
> Thanks a lot!
> I'll develop the system using FileIO for Azure Blob Storage.
> If any, I'll give feedback to this community.
>
> Thanks,
> Yukino Ikegami
>
> 2021年12月22日(水) 10:26 Yu Watanabe <yu...@gmail.com>:
>
>> Hello Brian.
>>
>> Thank you for the follow up.
>>
>> Ikegami-san.
>>
>> As a reference, I will also put PR for the python implementation which
>> was released in 2.25.0 .
>>
>> https://issues.apache.org/jira/browse/BEAM-6807
>> https://github.com/apache/beam/pull/12492
>>
>> However, it is still not on the io FileSystem IO list so not sure with
>> the release state though.
>>
>> https://beam.apache.org/documentation/io/built-in/
>>
>> Thanks,
>> Yu
>>
>>
>> On Wed, Dec 22, 2021 at 9:38 AM Brian Hulette <bh...@google.com>
>> wrote:
>>
>>> The package is marked experimental [1], which is how we indicate a
>>> feature isn't GA and may change. That being said, a lot of important Beam
>>> features are still marked experimental but are actually quite solid and
>>> stable, so there's some nuance here.
>>>
>>> @Pablo Estrada <pa...@google.com> may be able to comment more since
>>> he was involved with the initial contribution, but he may be out for the
>>> holidays for a little while.
>>>
>>> [1]
>>> https://beam.apache.org/releases/javadoc/2.34.0/org/apache/beam/sdk/io/azure/blobstore/package-summary.html
>>>
>>> On Tue, Dec 21, 2021 at 4:47 AM Yu Watanabe <yu...@gmail.com>
>>> wrote:
>>>
>>>> Hello .
>>>>
>>>> Looks like unit tests are passed in these PRs
>>>> .
>>>> https://issues.apache.org/jira/browse/BEAM-10378
>>>>
>>>> But the integration test with the WordCount example has not passed yet.
>>>>
>>>> https://github.com/apache/beam/pull/12649
>>>>
>>>> SInce AzureBlobStoreFileSystemRegistrar class is available in 2.34.0 ,
>>>> you could test in your prototype environment.
>>>>
>>>>
>>>> https://beam.apache.org/releases/javadoc/2.34.0/org/apache/beam/sdk/io/azure/blobstore/AzureBlobStoreFileSystemRegistrar.html
>>>>
>>>> Thanks,
>>>> Yu
>>>>
>>>>
>>>>
>>>> Official doc FileSystem class for azure blob storage is not listed in
>>>> the official doc.
>>>> https://beam.apache.org/documentation/io/built-in/
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> On Tue, Dec 21, 2021 at 10:50 AM 池上有希乃 <ik...@cloud-ace.jp>
>>>> wrote:
>>>>
>>>>> Hi
>>>>>
>>>>> I think I will use Apache Beam for my business.
>>>>> So there is a question.
>>>>> Whether FileIO for Azure Blob Storage is GA phase or not?
>>>>>
>>>>> Thanks,
>>>>> Yukino Ikegami
>>>>>
>>>>
>>>>
>>>> --
>>>> Yu Watanabe
>>>>
>>>> linkedin: www.linkedin.com/in/yuwatanabe1/
>>>> twitter: twitter.com/yuwtennis
>>>>
>>>>
>>>
>>
>> --
>> Yu Watanabe
>>
>> linkedin: www.linkedin.com/in/yuwatanabe1/
>> twitter: twitter.com/yuwtennis
>>
>>
>
>
> --
> ＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝
> クラウドエース株式会社
> 技術本部システム開発部データMLディビジョン
> 池上 有希乃
>
> 〒100-0004 東京都千代田区大手町2-6-2 日本ビルヂング11F 1106
> TEL: 03-6280-5939 FAX: 03-6800-3954
> Email: ikegami.yukino@cloud-ace.jp
> URL: https://www.cloud-ace.jp/
>
> <https://cloud-ace.jp/opendx2021/?utm_source=odx2021_all>
> ＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝
>
Re: Is FileIO for Azure Blob Storage GA?
Posted by 池上有希乃 <ik...@cloud-ace.jp>.
Hello Yu, Brian,
Thanks a lot!
I'll develop the system using FileIO for Azure Blob Storage.
If any, I'll give feedback to this community.
Thanks,
Yukino Ikegami
2021年12月22日(水) 10:26 Yu Watanabe <yu...@gmail.com>:
> Hello Brian.
>
> Thank you for the follow up.
>
> Ikegami-san.
>
> As a reference, I will also put PR for the python implementation which
> was released in 2.25.0 .
>
> https://issues.apache.org/jira/browse/BEAM-6807
> https://github.com/apache/beam/pull/12492
>
> However, it is still not on the io FileSystem IO list so not sure with the
> release state though.
>
> https://beam.apache.org/documentation/io/built-in/
>
> Thanks,
> Yu
>
>
> On Wed, Dec 22, 2021 at 9:38 AM Brian Hulette <bh...@google.com> wrote:
>
>> The package is marked experimental [1], which is how we indicate a
>> feature isn't GA and may change. That being said, a lot of important Beam
>> features are still marked experimental but are actually quite solid and
>> stable, so there's some nuance here.
>>
>> @Pablo Estrada <pa...@google.com> may be able to comment more since he
>> was involved with the initial contribution, but he may be out for the
>> holidays for a little while.
>>
>> [1]
>> https://beam.apache.org/releases/javadoc/2.34.0/org/apache/beam/sdk/io/azure/blobstore/package-summary.html
>>
>> On Tue, Dec 21, 2021 at 4:47 AM Yu Watanabe <yu...@gmail.com>
>> wrote:
>>
>>> Hello .
>>>
>>> Looks like unit tests are passed in these PRs
>>> .
>>> https://issues.apache.org/jira/browse/BEAM-10378
>>>
>>> But the integration test with the WordCount example has not passed yet.
>>>
>>> https://github.com/apache/beam/pull/12649
>>>
>>> SInce AzureBlobStoreFileSystemRegistrar class is available in 2.34.0 ,
>>> you could test in your prototype environment.
>>>
>>>
>>> https://beam.apache.org/releases/javadoc/2.34.0/org/apache/beam/sdk/io/azure/blobstore/AzureBlobStoreFileSystemRegistrar.html
>>>
>>> Thanks,
>>> Yu
>>>
>>>
>>>
>>> Official doc FileSystem class for azure blob storage is not listed in
>>> the official doc.
>>> https://beam.apache.org/documentation/io/built-in/
>>>
>>>
>>>
>>>
>>>
>>> On Tue, Dec 21, 2021 at 10:50 AM 池上有希乃 <ik...@cloud-ace.jp>
>>> wrote:
>>>
>>>> Hi
>>>>
>>>> I think I will use Apache Beam for my business.
>>>> So there is a question.
>>>> Whether FileIO for Azure Blob Storage is GA phase or not?
>>>>
>>>> Thanks,
>>>> Yukino Ikegami
>>>>
>>>
>>>
>>> --
>>> Yu Watanabe
>>>
>>> linkedin: www.linkedin.com/in/yuwatanabe1/
>>> twitter: twitter.com/yuwtennis
>>>
>>>
>>
>
> --
> Yu Watanabe
>
> linkedin: www.linkedin.com/in/yuwatanabe1/
> twitter: twitter.com/yuwtennis
>
>
--
＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝
クラウドエース株式会社
技術本部システム開発部データMLディビジョン
池上 有希乃
〒100-0004 東京都千代田区大手町2-6-2 日本ビルヂング11F 1106
TEL: 03-6280-5939 FAX: 03-6800-3954
Email: ikegami.yukino@cloud-ace.jp
URL: https://www.cloud-ace.jp/
<https://cloud-ace.jp/opendx2021/?utm_source=odx2021_all>
＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝＝
Re: Is FileIO for Azure Blob Storage GA?
Posted by Yu Watanabe <yu...@gmail.com>.
Hello Brian.
Thank you for the follow up.
Ikegami-san.
As a reference, I will also put PR for the python implementation which was
released in 2.25.0 .
https://issues.apache.org/jira/browse/BEAM-6807
https://github.com/apache/beam/pull/12492
However, it is still not on the io FileSystem IO list so not sure with the
release state though.
https://beam.apache.org/documentation/io/built-in/
Thanks,
Yu
On Wed, Dec 22, 2021 at 9:38 AM Brian Hulette <bh...@google.com> wrote:
> The package is marked experimental [1], which is how we indicate a feature
> isn't GA and may change. That being said, a lot of important Beam features
> are still marked experimental but are actually quite solid and stable, so
> there's some nuance here.
>
> @Pablo Estrada <pa...@google.com> may be able to comment more since he
> was involved with the initial contribution, but he may be out for the
> holidays for a little while.
>
> [1]
> https://beam.apache.org/releases/javadoc/2.34.0/org/apache/beam/sdk/io/azure/blobstore/package-summary.html
>
> On Tue, Dec 21, 2021 at 4:47 AM Yu Watanabe <yu...@gmail.com> wrote:
>
>> Hello .
>>
>> Looks like unit tests are passed in these PRs
>> .
>> https://issues.apache.org/jira/browse/BEAM-10378
>>
>> But the integration test with the WordCount example has not passed yet.
>>
>> https://github.com/apache/beam/pull/12649
>>
>> SInce AzureBlobStoreFileSystemRegistrar class is available in 2.34.0 ,
>> you could test in your prototype environment.
>>
>>
>> https://beam.apache.org/releases/javadoc/2.34.0/org/apache/beam/sdk/io/azure/blobstore/AzureBlobStoreFileSystemRegistrar.html
>>
>> Thanks,
>> Yu
>>
>>
>>
>> Official doc FileSystem class for azure blob storage is not listed in the
>> official doc.
>> https://beam.apache.org/documentation/io/built-in/
>>
>>
>>
>>
>>
>> On Tue, Dec 21, 2021 at 10:50 AM 池上有希乃 <ik...@cloud-ace.jp>
>> wrote:
>>
>>> Hi
>>>
>>> I think I will use Apache Beam for my business.
>>> So there is a question.
>>> Whether FileIO for Azure Blob Storage is GA phase or not?
>>>
>>> Thanks,
>>> Yukino Ikegami
>>>
>>
>>
>> --
>> Yu Watanabe
>>
>> linkedin: www.linkedin.com/in/yuwatanabe1/
>> twitter: twitter.com/yuwtennis
>>
>>
>
--
Yu Watanabe
linkedin: www.linkedin.com/in/yuwatanabe1/
twitter: twitter.com/yuwtennis
Re: Is FileIO for Azure Blob Storage GA?
Posted by Brian Hulette <bh...@google.com>.
The package is marked experimental [1], which is how we indicate a feature
isn't GA and may change. That being said, a lot of important Beam features
are still marked experimental but are actually quite solid and stable, so
there's some nuance here.
@Pablo Estrada <pa...@google.com> may be able to comment more since he
was involved with the initial contribution, but he may be out for the
holidays for a little while.
[1]
https://beam.apache.org/releases/javadoc/2.34.0/org/apache/beam/sdk/io/azure/blobstore/package-summary.html
On Tue, Dec 21, 2021 at 4:47 AM Yu Watanabe <yu...@gmail.com> wrote:
> Hello .
>
> Looks like unit tests are passed in these PRs
> .
> https://issues.apache.org/jira/browse/BEAM-10378
>
> But the integration test with the WordCount example has not passed yet.
>
> https://github.com/apache/beam/pull/12649
>
> SInce AzureBlobStoreFileSystemRegistrar class is available in 2.34.0 , you
> could test in your prototype environment.
>
>
> https://beam.apache.org/releases/javadoc/2.34.0/org/apache/beam/sdk/io/azure/blobstore/AzureBlobStoreFileSystemRegistrar.html
>
> Thanks,
> Yu
>
>
>
> Official doc FileSystem class for azure blob storage is not listed in the
> official doc.
> https://beam.apache.org/documentation/io/built-in/
>
>
>
>
>
> On Tue, Dec 21, 2021 at 10:50 AM 池上有希乃 <ik...@cloud-ace.jp>
> wrote:
>
>> Hi
>>
>> I think I will use Apache Beam for my business.
>> So there is a question.
>> Whether FileIO for Azure Blob Storage is GA phase or not?
>>
>> Thanks,
>> Yukino Ikegami
>>
>
>
> --
> Yu Watanabe
>
> linkedin: www.linkedin.com/in/yuwatanabe1/
> twitter: twitter.com/yuwtennis
>
>
Re: Is FileIO for Azure Blob Storage GA?
Posted by Yu Watanabe <yu...@gmail.com>.
Hello .
Looks like unit tests are passed in these PRs
.
https://issues.apache.org/jira/browse/BEAM-10378
But the integration test with the WordCount example has not passed yet.
https://github.com/apache/beam/pull/12649
SInce AzureBlobStoreFileSystemRegistrar class is available in 2.34.0 , you
could test in your prototype environment.
https://beam.apache.org/releases/javadoc/2.34.0/org/apache/beam/sdk/io/azure/blobstore/AzureBlobStoreFileSystemRegistrar.html
Thanks,
Yu
Official doc FileSystem class for azure blob storage is not listed in the
official doc.
https://beam.apache.org/documentation/io/built-in/
On Tue, Dec 21, 2021 at 10:50 AM 池上有希乃 <ik...@cloud-ace.jp> wrote:
> Hi
>
> I think I will use Apache Beam for my business.
> So there is a question.
> Whether FileIO for Azure Blob Storage is GA phase or not?
>
> Thanks,
> Yukino Ikegami
>
--
Yu Watanabe
linkedin: www.linkedin.com/in/yuwatanabe1/
twitter: twitter.com/yuwtennis