You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@beam.apache.org by Sachin Mittal <sj...@gmail.com> on 2023/05/22 12:00:01 UTC

Re: Difference between sdk.io.aws2.kinesis.KinesisIO vs sdk.io.kinesis.KinesisIO

Hi,
Can anyone tell me if enhanced fanout consumer for kinesis is released or
not.
If I see this PR
https://github.com/apache/beam/pull/23540 looks like this feature is merged.
I also see this issue now closed
https://github.com/apache/beam/issues/19967

However I am not able to find the mention of the same in release notes:
https://github.com/apache/beam/releases/tag/v2.47.0

Any knowledge in this would be appreciated.

Thanks
Sachin


On Mon, Sep 5, 2022 at 12:35 PM Moritz Mack <mm...@talend.com> wrote:

> Hi Sachin,
>
>
>
> I’d recommend migrating to the new AWS 2 IOs in
> beam-sdks-java-io-amazon-web-services2 (using Amazon’s Java SDK v2) rather
> soon.
>
> The previous ones (beam-sdks-java-io-amazon-web-services and
> beam-sdks-java-io-kinesis) are both deprecated and not actively maintained
> anymore.
>
>
>
> Please have a look at these notes in the changelog:
>
> https://github.com/apache/beam/blob/master/CHANGES.md#2410---2022-08-23
>
> https://github.com/apache/beam/blob/master/CHANGES.md#2380---2022-04-20
>
>
>
> Currently, neither one supports enhanced fan-out consumers yet.  It’s
> certainly something we’d like to support for the new v2 modules, but I
> personally didn’t have time to start looking into it so far.
>
> https://github.com/apache/beam/issues/19967
>
>
>
> Don’t hesitate to reach out in case you are facing any issues when
> migrating!
>
>
>
> Kind regards,
>
> Moritz
>
>
>
>
>
>
>
> On 31.08.22, 10:05, "Sachin Mittal" <sj...@gmail.com> wrote:
>
>
>
> Hello folks, We are consuming from a kinesis stream in our beam
> application. So far we were using sdk. io. kinesis. KinesisIO to read from
> kinesis. What I understand is that this library does not use Enhanced
> Fan-Out Consumer provided by AWS.
>
> Hello folks,
>
> We are consuming from a kinesis stream in our beam application.
>
> So far we were using *sdk.io
> <https://urldefense.com/v3/__http:/sdk.io__;!!CiXD_PY!VUENr73r_r9hNui_jnBewR2JWX7t2IwDXe6_FLFUq43GIFb3-uSIOC9DWmtR3XKaJzp3O-sg6N0iBA$>*.kinesis.KinesisIO
> to read from kinesis.
>
> What I understand is that this library does not use Enhanced Fan-Out
> Consumer provided by AWS.
>
>
>
> Recently I saw this library *sdk.io.aws2*.kinesis.KinesisIO and I
> wanted to understand what is the purpose of this new KinesisIO to read from
> and write to kinesis ?
>
>
>
> Does this new library use Enhanced Fan-Out Consumer ?
>
>
>
> When and how should we decide if we want to migrate to new library or
> continue to use the current one ?
>
>
>
> Thanks
>
> Sachin
>
>
>
>
>

Re: Difference between sdk.io.aws2.kinesis.KinesisIO vs sdk.io.kinesis.KinesisIO

Posted by Pavel Solomin <p....@gmail.com>.
Just to add: If you want to try out a pre-release version, you can use
SNAPSHOT artifacts. In Maven pom.xml it would look like this:

    <repositories>
        <repository>
            <id>apache.snapshots</id>
            <name>Apache Development Snapshot Repository</name>
            <url>https://repository.apache.org/content/repositories/snapshots/</url>
            <releases>
                <enabled>false</enabled>
            </releases>
            <snapshots>
                <enabled>true</enabled>
            </snapshots>
        </repository>
    </repositories>

...

        <dependency>
            <groupId>org.apache.beam</groupId>
            <artifactId>beam-sdks-java-io-amazon-web-services2</artifactId>
            <version>2.48.0-SNAPSHOT</version>
        </dependency>

        < other beam sdk dependencies >

Best Regards,
Pavel Solomin

Tel: +351 962 950 692 | Skype: pavel_solomin | Linkedin
<https://www.linkedin.com/in/pavelsolomin>





On Mon, 22 May 2023 at 13:07, Moritz Mack <mm...@talend.com> wrote:

> Hi Sachin,
>
>
>
> Enhanced Fanout for Kinesis will be released with Beam 2.48.0. See the I/O
> section:
>
> https://github.com/apache/beam/blob/master/CHANGES.md#2480---unreleased
>
>
>
> Best,
>
> Moritz
>
>
>
> On 22.05.23, 14:00, "Sachin Mittal" <sj...@gmail.com> wrote:
>
>
>
> Hi, Can anyone tell me if enhanced fanout consumer for kinesis is released
> or not. If I see this PR https: //github. com/apache/beam/pull/23540 looks
> like this feature is merged. I also see this issue now closed https:
> //github. com/apache/beam/issues/19967However
>
> Hi,
>
> Can anyone tell me if enhanced fanout consumer for kinesis is released or
> not.
>
> If I see this PR
>
> https://github.com/apache/beam/pull/23540
> <https://urldefense.com/v3/__https:/github.com/apache/beam/pull/23540__;!!CiXD_PY!UoiR6oywupQ0vxlUpxBE-F-bRravkguTQOPk-a7djBb24sHZZW7QL3i7V1-o6KXmsbexU5y_aYgF9Q$>
> looks like this feature is merged.
>
> I also see this issue now closed
>
> https://github.com/apache/beam/issues/19967
> <https://urldefense.com/v3/__https:/github.com/apache/beam/issues/19967__;!!CiXD_PY!UoiR6oywupQ0vxlUpxBE-F-bRravkguTQOPk-a7djBb24sHZZW7QL3i7V1-o6KXmsbexU5yFb_csSA$>
>
>
>
> However I am not able to find the mention of the same in release notes:
>
> https://github.com/apache/beam/releases/tag/v2.47.0
> <https://urldefense.com/v3/__https:/github.com/apache/beam/releases/tag/v2.47.0__;!!CiXD_PY!UoiR6oywupQ0vxlUpxBE-F-bRravkguTQOPk-a7djBb24sHZZW7QL3i7V1-o6KXmsbexU5yjitVjHQ$>
>
>
>
> Any knowledge in this would be appreciated.
>
>
>
> Thanks
>
> Sachin
>
>
>
>
>
> On Mon, Sep 5, 2022 at 12:35 PM Moritz Mack <mm...@talend.com> wrote:
>
> Hi Sachin,
>
>
>
> I’d recommend migrating to the new AWS 2 IOs in
> beam-sdks-java-io-amazon-web-services2 (using Amazon’s Java SDK v2) rather
> soon.
>
> The previous ones (beam-sdks-java-io-amazon-web-services and
> beam-sdks-java-io-kinesis) are both deprecated and not actively maintained
> anymore.
>
>
>
> Please have a look at these notes in the changelog:
>
> https://github.com/apache/beam/blob/master/CHANGES.md#2410---2022-08-23
> <https://urldefense.com/v3/__https:/github.com/apache/beam/blob/master/CHANGES.md*2410---2022-08-23__;Iw!!CiXD_PY!UoiR6oywupQ0vxlUpxBE-F-bRravkguTQOPk-a7djBb24sHZZW7QL3i7V1-o6KXmsbexU5zG-ZfSOw$>
>
> https://github.com/apache/beam/blob/master/CHANGES.md#2380---2022-04-20
> <https://urldefense.com/v3/__https:/github.com/apache/beam/blob/master/CHANGES.md*2380---2022-04-20__;Iw!!CiXD_PY!UoiR6oywupQ0vxlUpxBE-F-bRravkguTQOPk-a7djBb24sHZZW7QL3i7V1-o6KXmsbexU5xH6M7dfw$>
>
>
>
> Currently, neither one supports enhanced fan-out consumers yet.  It’s
> certainly something we’d like to support for the new v2 modules, but I
> personally didn’t have time to start looking into it so far.
>
> https://github.com/apache/beam/issues/19967
> <https://urldefense.com/v3/__https:/github.com/apache/beam/issues/19967__;!!CiXD_PY!UoiR6oywupQ0vxlUpxBE-F-bRravkguTQOPk-a7djBb24sHZZW7QL3i7V1-o6KXmsbexU5yFb_csSA$>
>
>
>
> Don’t hesitate to reach out in case you are facing any issues when
> migrating!
>
>
>
> Kind regards,
>
> Moritz
>
>
>
>
>
>
>
> On 31.08.22, 10:05, "Sachin Mittal" <sj...@gmail.com> wrote:
>
>
>
> Hello folks, We are consuming from a kinesis stream in our beam
> application. So far we were using sdk. io. kinesis. KinesisIO to read from
> kinesis. What I understand is that this library does not use Enhanced
> Fan-Out Consumer provided by AWS.
>
> Hello folks,
>
> We are consuming from a kinesis stream in our beam application.
>
> So far we were using *sdk.io
> <https://urldefense.com/v3/__http:/sdk.io__;!!CiXD_PY!VUENr73r_r9hNui_jnBewR2JWX7t2IwDXe6_FLFUq43GIFb3-uSIOC9DWmtR3XKaJzp3O-sg6N0iBA$>*.kinesis.KinesisIO
> to read from kinesis.
>
> What I understand is that this library does not use Enhanced Fan-Out
> Consumer provided by AWS.
>
>
>
> Recently I saw this library *sdk.io.aws2*.kinesis.KinesisIO and I
> wanted to understand what is the purpose of this new KinesisIO to read from
> and write to kinesis ?
>
>
>
> Does this new library use Enhanced Fan-Out Consumer ?
>
>
>
> When and how should we decide if we want to migrate to new library or
> continue to use the current one ?
>
>
>
> Thanks
>
> Sachin
>
>
>
>
>
> *As a recipient of an email from the Talend Group, your personal data will
> be processed by our systems. Please see our Privacy Notice
> <https://www.talend.com/privacy-policy/>*for more information about our
> collection and use of your personal information, our security practices,
> and your data protection rights, including any rights you may have to
> object to automated-decision making or profiling we use to analyze support
> or marketing related communications. To manage or discontinue promotional
> communications, use the communication preferences portal
> <https://info.talend.com/emailpreferencesen.html>. To exercise your data
> protection rights, use the privacy request form
> <https://talend.my.onetrust.com/webform/ef906c5a-de41-4ea0-ba73-96c079cdd15a/b191c71d-f3cb-4a42-9815-0c3ca021704cl>.
> Contact us here <https://www.talend.com/contact/>or by mail to either of
> our co-headquarters: Talend, Inc.: 400 South El Camino Real, Ste 1400, San
> Mateo, CA 94402; Talend SAS: 5/7 rue Salomon De Rothschild, 92150 Suresnes,
> France
>

Re: Difference between sdk.io.aws2.kinesis.KinesisIO vs sdk.io.kinesis.KinesisIO

Posted by Moritz Mack <mm...@talend.com>.
Hi Sachin,

Enhanced Fanout for Kinesis will be released with Beam 2.48.0. See the I/O section:
https://github.com/apache/beam/blob/master/CHANGES.md#2480---unreleased

Best,
Moritz

On 22.05.23, 14:00, "Sachin Mittal" <sj...@gmail.com> wrote:

Hi, Can anyone tell me if enhanced fanout consumer for kinesis is released or not. If I see this PR https: //github. com/apache/beam/pull/23540 looks like this feature is merged. I also see this issue now closed https: //github. com/apache/beam/issues/19967However

Hi,
Can anyone tell me if enhanced fanout consumer for kinesis is released or not.
If I see this PR
https://github.com/apache/beam/pull/23540<https://urldefense.com/v3/__https:/github.com/apache/beam/pull/23540__;!!CiXD_PY!UoiR6oywupQ0vxlUpxBE-F-bRravkguTQOPk-a7djBb24sHZZW7QL3i7V1-o6KXmsbexU5y_aYgF9Q$> looks like this feature is merged.
I also see this issue now closed
https://github.com/apache/beam/issues/19967<https://urldefense.com/v3/__https:/github.com/apache/beam/issues/19967__;!!CiXD_PY!UoiR6oywupQ0vxlUpxBE-F-bRravkguTQOPk-a7djBb24sHZZW7QL3i7V1-o6KXmsbexU5yFb_csSA$>

However I am not able to find the mention of the same in release notes:
https://github.com/apache/beam/releases/tag/v2.47.0<https://urldefense.com/v3/__https:/github.com/apache/beam/releases/tag/v2.47.0__;!!CiXD_PY!UoiR6oywupQ0vxlUpxBE-F-bRravkguTQOPk-a7djBb24sHZZW7QL3i7V1-o6KXmsbexU5yjitVjHQ$>

Any knowledge in this would be appreciated.

Thanks
Sachin


On Mon, Sep 5, 2022 at 12:35 PM Moritz Mack <mm...@talend.com>> wrote:
Hi Sachin,

I’d recommend migrating to the new AWS 2 IOs in beam-sdks-java-io-amazon-web-services2 (using Amazon’s Java SDK v2) rather soon.
The previous ones (beam-sdks-java-io-amazon-web-services and beam-sdks-java-io-kinesis) are both deprecated and not actively maintained anymore.

Please have a look at these notes in the changelog:
https://github.com/apache/beam/blob/master/CHANGES.md#2410---2022-08-23<https://urldefense.com/v3/__https:/github.com/apache/beam/blob/master/CHANGES.md*2410---2022-08-23__;Iw!!CiXD_PY!UoiR6oywupQ0vxlUpxBE-F-bRravkguTQOPk-a7djBb24sHZZW7QL3i7V1-o6KXmsbexU5zG-ZfSOw$>
https://github.com/apache/beam/blob/master/CHANGES.md#2380---2022-04-20<https://urldefense.com/v3/__https:/github.com/apache/beam/blob/master/CHANGES.md*2380---2022-04-20__;Iw!!CiXD_PY!UoiR6oywupQ0vxlUpxBE-F-bRravkguTQOPk-a7djBb24sHZZW7QL3i7V1-o6KXmsbexU5xH6M7dfw$>

Currently, neither one supports enhanced fan-out consumers yet.  It’s certainly something we’d like to support for the new v2 modules, but I personally didn’t have time to start looking into it so far.
https://github.com/apache/beam/issues/19967<https://urldefense.com/v3/__https:/github.com/apache/beam/issues/19967__;!!CiXD_PY!UoiR6oywupQ0vxlUpxBE-F-bRravkguTQOPk-a7djBb24sHZZW7QL3i7V1-o6KXmsbexU5yFb_csSA$>

Don’t hesitate to reach out in case you are facing any issues when migrating!

Kind regards,
Moritz



On 31.08.22, 10:05, "Sachin Mittal" <sj...@gmail.com>> wrote:

Hello folks, We are consuming from a kinesis stream in our beam application. So far we were using sdk. io. kinesis. KinesisIO to read from kinesis. What I understand is that this library does not use Enhanced Fan-Out Consumer provided by AWS. 
Hello folks,
We are consuming from a kinesis stream in our beam application.
So far we were using sdk.io<https://urldefense.com/v3/__http:/sdk.io__;!!CiXD_PY!VUENr73r_r9hNui_jnBewR2JWX7t2IwDXe6_FLFUq43GIFb3-uSIOC9DWmtR3XKaJzp3O-sg6N0iBA$>.kinesis.KinesisIO to read from kinesis.
What I understand is that this library does not use Enhanced Fan-Out Consumer provided by AWS.

Recently I saw this library sdk.io.aws2.kinesis.KinesisIO and I wanted to understand what is the purpose of this new KinesisIO to read from and write to kinesis ?

Does this new library use Enhanced Fan-Out Consumer ?

When and how should we decide if we want to migrate to new library or continue to use the current one ?

Thanks
Sachin



As a recipient of an email from the Talend Group, your personal data will be processed by our systems. Please see our Privacy Notice <https://www.talend.com/privacy-policy/> for more information about our collection and use of your personal information, our security practices, and your data protection rights, including any rights you may have to object to automated-decision making or profiling we use to analyze support or marketing related communications. To manage or discontinue promotional communications, use the communication preferences portal<https://info.talend.com/emailpreferencesen.html>. To exercise your data protection rights, use the privacy request form<https://talend.my.onetrust.com/webform/ef906c5a-de41-4ea0-ba73-96c079cdd15a/b191c71d-f3cb-4a42-9815-0c3ca021704cl>. Contact us here <https://www.talend.com/contact/> or by mail to either of our co-headquarters: Talend, Inc.: 400 South El Camino Real, Ste 1400, San Mateo, CA 94402; Talend SAS: 5/7 rue Salomon De Rothschild, 92150 Suresnes, France