You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@beam.apache.org by Jonothan Farr <jo...@gmail.com> on 2020/04/17 23:48:17 UTC
Support for AWS SDK v2 and enhanced fanout in KinesisIO
Hi, I wanted to try out KinesisIO with enhanced fanout for a PoC I was
working on, so I ended up submitting that as
https://github.com/apache/beam/pull/9899.
But since that still needs work to be fully functional (right now it does
not handle resharding), I figured I could at least contribute the updates
to make KinesisIO compatible with the AWS SDK v2, so I split that out and
submitted it as: https://github.com/apache/beam/pull/11318.
Now some totally reasonable concerns have been raised about maintaining two
separate codebases, and having to dual-commit every bugfix or feature
enhancement. That seems to already be the case for all of the other AWS IOs
in amazon-web-services2 (dynamodb, sns, sqs), but do we want to continue
that trend? What do members of the community think?
One solution is obviously to rewrite the AWS IOs with an added abstraction
layer so that the SDK-specific code can be factored out. However, the
KinesisIO class itself already has dependencies on the v1 AWS SDK so I'm
not aware of a way that this could be done so that it's
backwards-compatible. This means that everyone currently using KinesisIO
would most likely have to update their code to use the new interface
whether they wanted to use the new v2 SDK or continue to use v1. I think a
rewrite would also be a non-trivial task with more than a few details to be
worked out first.
From my perspective, the options are:
- Merge #11318 now and the community can use the v2 AWS SDK with KinesisIO
and I can continue working on enhanced fanout while a rewrite is being
worked on, but the community now has to dual-commit any changes to
KinesisIO.
- Close #11318 and continue working on #9899 and I alone have to
dual-commit the changes until it can be merged, at which point everyone can
use enhanced fanout while the rewrite is being worked on. Then the
community will also have to dual-commit but only after enhanced fanout is
available.
- Close both PRs and just wait for a rewrite, and no one can use the v2 AWS
SDK or enhanced fanout in KinesisIO until after it's done. (Not to imply
that I wouldn't be interested in contributing to that effort, just for the
sake of these 2 PRs.)
- Don't do the rewrite at all and just live with dual-committing until the
KinesisIO based on the v1 SDK reaches its end of life. No one has to update
their code to continue using the v1 SDK, only if they want to move to v2.
There may be other options or considerations I'm missing here. What do
others think?
Thanks,
Jonothan
Re: Support for AWS SDK v2 and enhanced fanout in KinesisIO
Posted by Ismaël Mejía <ie...@gmail.com>.
Oups I meant 'would you like to start rolling this plan?' sorry.
On Fri, May 8, 2020 at 12:03 AM Ismaël Mejía <ie...@gmail.com> wrote:
>
> Achieving good abstractions will prove elusive since the APIs differ
> and we will end up with a ton of extra maintenance work that should be
> not Beam's responsibility. I know that similar code (almost copy
> pasteable) is not nice to have but we should consider this as a
> temporary measure and probably try to make deprecation (and possible
> removal) of older versions as soon as possible.
> Please remember that this was already discussed in the past [1] and
> the soft consensus was around rapid deprecation AWS SDK v1 IOs already
> available in their v2 version and only do improvements on v1 IOs for
> security related issues and dependency updates. This has not happened
> yet but the only missing thing is someone to take action.
>
> +cammach84@gmail.com would you have to me to start rolling this plan?
> otherwise someone else can jump and do it too.
>
> [1] https://lists.apache.org/thread.html/130cb60e6bcdd58c5afdd0c375663eaf05e705aab9ee0196535cd17f%40%3Cdev.beam.apache.org%3E
>
> On Thu, May 7, 2020 at 11:05 PM Luke Cwik <lc...@google.com> wrote:
> >
> > I think you should try and share as much as is reasonable. Using what is shared between AWS V1 and V2 SDKs would be a good signal as to what should be shared in Beam. There might be some places where a trivial wrapper could help but I wouldn't try to create a bunch of grand abstractions that fit both versions of the AWS SDKs.
> >
> > On Wed, May 6, 2020 at 9:59 AM Alexey Romanenko <ar...@gmail.com> wrote:
> >>
> >> I’d like to get back to this question, fairly raised by Jonothan a while ago, since actually it affects not only KinesisIO but also all other AWS IOs in general, that uses AWS SDK of two different versions.
> >>
> >> My personal opinion - I’d strongly like to avoid a copy-pasted code for two different versions of the same IO (and double support of this), that is SDK-independent. In this case, if we wish too provide IO for two AWS SDK versions, we have to extract this core logic into one common and SDK-independent framework that could used by any of SDK versions then.
> >>
> >> In the same time, if it will require a lot of efforts to achieve that, then probably we need to decide to leave only one IO based on single AWS SDK version (I guess it will be V2 as more modern) and deprecate old version (and then remove it to avoid a confusion for users).
> >>
> >> What others think about that?
> >>
> >> On 18 Apr 2020, at 01:48, Jonothan Farr <jo...@gmail.com> wrote:
> >>
> >> Hi, I wanted to try out KinesisIO with enhanced fanout for a PoC I was working on, so I ended up submitting that as https://github.com/apache/beam/pull/9899.
> >>
> >> But since that still needs work to be fully functional (right now it does not handle resharding), I figured I could at least contribute the updates to make KinesisIO compatible with the AWS SDK v2, so I split that out and submitted it as: https://github.com/apache/beam/pull/11318.
> >>
> >> Now some totally reasonable concerns have been raised about maintaining two separate codebases, and having to dual-commit every bugfix or feature enhancement. That seems to already be the case for all of the other AWS IOs in amazon-web-services2 (dynamodb, sns, sqs), but do we want to continue that trend? What do members of the community think?
> >>
> >> One solution is obviously to rewrite the AWS IOs with an added abstraction layer so that the SDK-specific code can be factored out. However, the KinesisIO class itself already has dependencies on the v1 AWS SDK so I'm not aware of a way that this could be done so that it's backwards-compatible. This means that everyone currently using KinesisIO would most likely have to update their code to use the new interface whether they wanted to use the new v2 SDK or continue to use v1. I think a rewrite would also be a non-trivial task with more than a few details to be worked out first.
> >>
> >> From my perspective, the options are:
> >>
> >> - Merge #11318 now and the community can use the v2 AWS SDK with KinesisIO and I can continue working on enhanced fanout while a rewrite is being worked on, but the community now has to dual-commit any changes to KinesisIO.
> >> - Close #11318 and continue working on #9899 and I alone have to dual-commit the changes until it can be merged, at which point everyone can use enhanced fanout while the rewrite is being worked on. Then the community will also have to dual-commit but only after enhanced fanout is available.
> >> - Close both PRs and just wait for a rewrite, and no one can use the v2 AWS SDK or enhanced fanout in KinesisIO until after it's done. (Not to imply that I wouldn't be interested in contributing to that effort, just for the sake of these 2 PRs.)
> >> - Don't do the rewrite at all and just live with dual-committing until the KinesisIO based on the v1 SDK reaches its end of life. No one has to update their code to continue using the v1 SDK, only if they want to move to v2.
> >>
> >> There may be other options or considerations I'm missing here. What do others think?
> >>
> >> Thanks,
> >> Jonothan
> >>
> >>
Re: Support for AWS SDK v2 and enhanced fanout in KinesisIO
Posted by Luke Cwik <lc...@google.com>.
You'll want to reach out to the users@ thread for getting feedback on
deprecation.
On Mon, May 11, 2020 at 9:53 AM Alexey Romanenko <ar...@gmail.com>
wrote:
>
> Thanks Ismaël for recalling of this thread, I think we should start to
> take some efforts to deprecate the AWS SDK V1 IOs that are already
> implemeneted using V2 (if there are no other objections). In this case, it
> would make sense to abstract some common code ONLY if ,for some reasons, we
> would like to keep the both versions of IO (which is not a case for now,
> afaik).
>
> > On 8 May 2020, at 00:03, Ismaël Mejía <ie...@gmail.com> wrote:
> >
> > Achieving good abstractions will prove elusive since the APIs differ
> > and we will end up with a ton of extra maintenance work that should be
> > not Beam's responsibility. I know that similar code (almost copy
> > pasteable) is not nice to have but we should consider this as a
> > temporary measure and probably try to make deprecation (and possible
> > removal) of older versions as soon as possible.
> > Please remember that this was already discussed in the past [1] and
> > the soft consensus was around rapid deprecation AWS SDK v1 IOs already
> > available in their v2 version and only do improvements on v1 IOs for
> > security related issues and dependency updates. This has not happened
> > yet but the only missing thing is someone to take action.
> >
> > +cammach84@gmail.com would you have to me to start rolling this plan?
> > otherwise someone else can jump and do it too.
> >
> > [1]
> https://lists.apache.org/thread.html/130cb60e6bcdd58c5afdd0c375663eaf05e705aab9ee0196535cd17f%40%3Cdev.beam.apache.org%3E
> >
> > On Thu, May 7, 2020 at 11:05 PM Luke Cwik <lc...@google.com> wrote:
> >>
> >> I think you should try and share as much as is reasonable. Using what
> is shared between AWS V1 and V2 SDKs would be a good signal as to what
> should be shared in Beam. There might be some places where a trivial
> wrapper could help but I wouldn't try to create a bunch of grand
> abstractions that fit both versions of the AWS SDKs.
> >>
> >> On Wed, May 6, 2020 at 9:59 AM Alexey Romanenko <
> aromanenko.dev@gmail.com> wrote:
> >>>
> >>> I’d like to get back to this question, fairly raised by Jonothan a
> while ago, since actually it affects not only KinesisIO but also all other
> AWS IOs in general, that uses AWS SDK of two different versions.
> >>>
> >>> My personal opinion - I’d strongly like to avoid a copy-pasted code
> for two different versions of the same IO (and double support of this),
> that is SDK-independent. In this case, if we wish too provide IO for two
> AWS SDK versions, we have to extract this core logic into one common and
> SDK-independent framework that could used by any of SDK versions then.
> >>>
> >>> In the same time, if it will require a lot of efforts to achieve that,
> then probably we need to decide to leave only one IO based on single AWS
> SDK version (I guess it will be V2 as more modern) and deprecate old
> version (and then remove it to avoid a confusion for users).
> >>>
> >>> What others think about that?
> >>>
> >>> On 18 Apr 2020, at 01:48, Jonothan Farr <jo...@gmail.com>
> wrote:
> >>>
> >>> Hi, I wanted to try out KinesisIO with enhanced fanout for a PoC I was
> working on, so I ended up submitting that as
> https://github.com/apache/beam/pull/9899.
> >>>
> >>> But since that still needs work to be fully functional (right now it
> does not handle resharding), I figured I could at least contribute the
> updates to make KinesisIO compatible with the AWS SDK v2, so I split that
> out and submitted it as: https://github.com/apache/beam/pull/11318.
> >>>
> >>> Now some totally reasonable concerns have been raised about
> maintaining two separate codebases, and having to dual-commit every bugfix
> or feature enhancement. That seems to already be the case for all of the
> other AWS IOs in amazon-web-services2 (dynamodb, sns, sqs), but do we want
> to continue that trend? What do members of the community think?
> >>>
> >>> One solution is obviously to rewrite the AWS IOs with an added
> abstraction layer so that the SDK-specific code can be factored out.
> However, the KinesisIO class itself already has dependencies on the v1 AWS
> SDK so I'm not aware of a way that this could be done so that it's
> backwards-compatible. This means that everyone currently using KinesisIO
> would most likely have to update their code to use the new interface
> whether they wanted to use the new v2 SDK or continue to use v1. I think a
> rewrite would also be a non-trivial task with more than a few details to be
> worked out first.
> >>>
> >>> From my perspective, the options are:
> >>>
> >>> - Merge #11318 now and the community can use the v2 AWS SDK with
> KinesisIO and I can continue working on enhanced fanout while a rewrite is
> being worked on, but the community now has to dual-commit any changes to
> KinesisIO.
> >>> - Close #11318 and continue working on #9899 and I alone have to
> dual-commit the changes until it can be merged, at which point everyone can
> use enhanced fanout while the rewrite is being worked on. Then the
> community will also have to dual-commit but only after enhanced fanout is
> available.
> >>> - Close both PRs and just wait for a rewrite, and no one can use the
> v2 AWS SDK or enhanced fanout in KinesisIO until after it's done. (Not to
> imply that I wouldn't be interested in contributing to that effort, just
> for the sake of these 2 PRs.)
> >>> - Don't do the rewrite at all and just live with dual-committing until
> the KinesisIO based on the v1 SDK reaches its end of life. No one has to
> update their code to continue using the v1 SDK, only if they want to move
> to v2.
> >>>
> >>> There may be other options or considerations I'm missing here. What do
> others think?
> >>>
> >>> Thanks,
> >>> Jonothan
> >>>
> >>>
>
>
Re: Support for AWS SDK v2 and enhanced fanout in KinesisIO
Posted by Alexey Romanenko <ar...@gmail.com>.
Thanks Ismaël for recalling of this thread, I think we should start to take some efforts to deprecate the AWS SDK V1 IOs that are already implemeneted using V2 (if there are no other objections). In this case, it would make sense to abstract some common code ONLY if ,for some reasons, we would like to keep the both versions of IO (which is not a case for now, afaik).
> On 8 May 2020, at 00:03, Ismaël Mejía <ie...@gmail.com> wrote:
>
> Achieving good abstractions will prove elusive since the APIs differ
> and we will end up with a ton of extra maintenance work that should be
> not Beam's responsibility. I know that similar code (almost copy
> pasteable) is not nice to have but we should consider this as a
> temporary measure and probably try to make deprecation (and possible
> removal) of older versions as soon as possible.
> Please remember that this was already discussed in the past [1] and
> the soft consensus was around rapid deprecation AWS SDK v1 IOs already
> available in their v2 version and only do improvements on v1 IOs for
> security related issues and dependency updates. This has not happened
> yet but the only missing thing is someone to take action.
>
> +cammach84@gmail.com would you have to me to start rolling this plan?
> otherwise someone else can jump and do it too.
>
> [1] https://lists.apache.org/thread.html/130cb60e6bcdd58c5afdd0c375663eaf05e705aab9ee0196535cd17f%40%3Cdev.beam.apache.org%3E
>
> On Thu, May 7, 2020 at 11:05 PM Luke Cwik <lc...@google.com> wrote:
>>
>> I think you should try and share as much as is reasonable. Using what is shared between AWS V1 and V2 SDKs would be a good signal as to what should be shared in Beam. There might be some places where a trivial wrapper could help but I wouldn't try to create a bunch of grand abstractions that fit both versions of the AWS SDKs.
>>
>> On Wed, May 6, 2020 at 9:59 AM Alexey Romanenko <ar...@gmail.com> wrote:
>>>
>>> I’d like to get back to this question, fairly raised by Jonothan a while ago, since actually it affects not only KinesisIO but also all other AWS IOs in general, that uses AWS SDK of two different versions.
>>>
>>> My personal opinion - I’d strongly like to avoid a copy-pasted code for two different versions of the same IO (and double support of this), that is SDK-independent. In this case, if we wish too provide IO for two AWS SDK versions, we have to extract this core logic into one common and SDK-independent framework that could used by any of SDK versions then.
>>>
>>> In the same time, if it will require a lot of efforts to achieve that, then probably we need to decide to leave only one IO based on single AWS SDK version (I guess it will be V2 as more modern) and deprecate old version (and then remove it to avoid a confusion for users).
>>>
>>> What others think about that?
>>>
>>> On 18 Apr 2020, at 01:48, Jonothan Farr <jo...@gmail.com> wrote:
>>>
>>> Hi, I wanted to try out KinesisIO with enhanced fanout for a PoC I was working on, so I ended up submitting that as https://github.com/apache/beam/pull/9899.
>>>
>>> But since that still needs work to be fully functional (right now it does not handle resharding), I figured I could at least contribute the updates to make KinesisIO compatible with the AWS SDK v2, so I split that out and submitted it as: https://github.com/apache/beam/pull/11318.
>>>
>>> Now some totally reasonable concerns have been raised about maintaining two separate codebases, and having to dual-commit every bugfix or feature enhancement. That seems to already be the case for all of the other AWS IOs in amazon-web-services2 (dynamodb, sns, sqs), but do we want to continue that trend? What do members of the community think?
>>>
>>> One solution is obviously to rewrite the AWS IOs with an added abstraction layer so that the SDK-specific code can be factored out. However, the KinesisIO class itself already has dependencies on the v1 AWS SDK so I'm not aware of a way that this could be done so that it's backwards-compatible. This means that everyone currently using KinesisIO would most likely have to update their code to use the new interface whether they wanted to use the new v2 SDK or continue to use v1. I think a rewrite would also be a non-trivial task with more than a few details to be worked out first.
>>>
>>> From my perspective, the options are:
>>>
>>> - Merge #11318 now and the community can use the v2 AWS SDK with KinesisIO and I can continue working on enhanced fanout while a rewrite is being worked on, but the community now has to dual-commit any changes to KinesisIO.
>>> - Close #11318 and continue working on #9899 and I alone have to dual-commit the changes until it can be merged, at which point everyone can use enhanced fanout while the rewrite is being worked on. Then the community will also have to dual-commit but only after enhanced fanout is available.
>>> - Close both PRs and just wait for a rewrite, and no one can use the v2 AWS SDK or enhanced fanout in KinesisIO until after it's done. (Not to imply that I wouldn't be interested in contributing to that effort, just for the sake of these 2 PRs.)
>>> - Don't do the rewrite at all and just live with dual-committing until the KinesisIO based on the v1 SDK reaches its end of life. No one has to update their code to continue using the v1 SDK, only if they want to move to v2.
>>>
>>> There may be other options or considerations I'm missing here. What do others think?
>>>
>>> Thanks,
>>> Jonothan
>>>
>>>
Re: Support for AWS SDK v2 and enhanced fanout in KinesisIO
Posted by Ismaël Mejía <ie...@gmail.com>.
Achieving good abstractions will prove elusive since the APIs differ
and we will end up with a ton of extra maintenance work that should be
not Beam's responsibility. I know that similar code (almost copy
pasteable) is not nice to have but we should consider this as a
temporary measure and probably try to make deprecation (and possible
removal) of older versions as soon as possible.
Please remember that this was already discussed in the past [1] and
the soft consensus was around rapid deprecation AWS SDK v1 IOs already
available in their v2 version and only do improvements on v1 IOs for
security related issues and dependency updates. This has not happened
yet but the only missing thing is someone to take action.
+cammach84@gmail.com would you have to me to start rolling this plan?
otherwise someone else can jump and do it too.
[1] https://lists.apache.org/thread.html/130cb60e6bcdd58c5afdd0c375663eaf05e705aab9ee0196535cd17f%40%3Cdev.beam.apache.org%3E
On Thu, May 7, 2020 at 11:05 PM Luke Cwik <lc...@google.com> wrote:
>
> I think you should try and share as much as is reasonable. Using what is shared between AWS V1 and V2 SDKs would be a good signal as to what should be shared in Beam. There might be some places where a trivial wrapper could help but I wouldn't try to create a bunch of grand abstractions that fit both versions of the AWS SDKs.
>
> On Wed, May 6, 2020 at 9:59 AM Alexey Romanenko <ar...@gmail.com> wrote:
>>
>> I’d like to get back to this question, fairly raised by Jonothan a while ago, since actually it affects not only KinesisIO but also all other AWS IOs in general, that uses AWS SDK of two different versions.
>>
>> My personal opinion - I’d strongly like to avoid a copy-pasted code for two different versions of the same IO (and double support of this), that is SDK-independent. In this case, if we wish too provide IO for two AWS SDK versions, we have to extract this core logic into one common and SDK-independent framework that could used by any of SDK versions then.
>>
>> In the same time, if it will require a lot of efforts to achieve that, then probably we need to decide to leave only one IO based on single AWS SDK version (I guess it will be V2 as more modern) and deprecate old version (and then remove it to avoid a confusion for users).
>>
>> What others think about that?
>>
>> On 18 Apr 2020, at 01:48, Jonothan Farr <jo...@gmail.com> wrote:
>>
>> Hi, I wanted to try out KinesisIO with enhanced fanout for a PoC I was working on, so I ended up submitting that as https://github.com/apache/beam/pull/9899.
>>
>> But since that still needs work to be fully functional (right now it does not handle resharding), I figured I could at least contribute the updates to make KinesisIO compatible with the AWS SDK v2, so I split that out and submitted it as: https://github.com/apache/beam/pull/11318.
>>
>> Now some totally reasonable concerns have been raised about maintaining two separate codebases, and having to dual-commit every bugfix or feature enhancement. That seems to already be the case for all of the other AWS IOs in amazon-web-services2 (dynamodb, sns, sqs), but do we want to continue that trend? What do members of the community think?
>>
>> One solution is obviously to rewrite the AWS IOs with an added abstraction layer so that the SDK-specific code can be factored out. However, the KinesisIO class itself already has dependencies on the v1 AWS SDK so I'm not aware of a way that this could be done so that it's backwards-compatible. This means that everyone currently using KinesisIO would most likely have to update their code to use the new interface whether they wanted to use the new v2 SDK or continue to use v1. I think a rewrite would also be a non-trivial task with more than a few details to be worked out first.
>>
>> From my perspective, the options are:
>>
>> - Merge #11318 now and the community can use the v2 AWS SDK with KinesisIO and I can continue working on enhanced fanout while a rewrite is being worked on, but the community now has to dual-commit any changes to KinesisIO.
>> - Close #11318 and continue working on #9899 and I alone have to dual-commit the changes until it can be merged, at which point everyone can use enhanced fanout while the rewrite is being worked on. Then the community will also have to dual-commit but only after enhanced fanout is available.
>> - Close both PRs and just wait for a rewrite, and no one can use the v2 AWS SDK or enhanced fanout in KinesisIO until after it's done. (Not to imply that I wouldn't be interested in contributing to that effort, just for the sake of these 2 PRs.)
>> - Don't do the rewrite at all and just live with dual-committing until the KinesisIO based on the v1 SDK reaches its end of life. No one has to update their code to continue using the v1 SDK, only if they want to move to v2.
>>
>> There may be other options or considerations I'm missing here. What do others think?
>>
>> Thanks,
>> Jonothan
>>
>>
Re: Support for AWS SDK v2 and enhanced fanout in KinesisIO
Posted by Luke Cwik <lc...@google.com>.
I think you should try and share as much as is reasonable. Using what is
shared between AWS V1 and V2 SDKs would be a good signal as to what should
be shared in Beam. There might be some places where a trivial wrapper could
help but I wouldn't try to create a bunch of grand abstractions that fit
both versions of the AWS SDKs.
On Wed, May 6, 2020 at 9:59 AM Alexey Romanenko <ar...@gmail.com>
wrote:
> I’d like to get back to this question, fairly raised by Jonothan a while
> ago, since actually it affects not only KinesisIO but also all other AWS
> IOs in general, that uses AWS SDK of two different versions.
>
> My personal opinion - I’d strongly like to avoid a copy-pasted code for
> two different versions of the same IO (and double support of this), that is
> SDK-independent. In this case, if we wish too provide IO for two AWS SDK
> versions, we have to extract this core logic into one common and
> SDK-independent framework that could used by any of SDK versions then.
>
> In the same time, if it will require a lot of efforts to achieve that,
> then probably we need to decide to leave only one IO based on single AWS
> SDK version (I guess it will be V2 as more modern) and deprecate old
> version (and then remove it to avoid a confusion for users).
>
> What others think about that?
>
> On 18 Apr 2020, at 01:48, Jonothan Farr <jo...@gmail.com> wrote:
>
> Hi, I wanted to try out KinesisIO with enhanced fanout for a PoC I was
> working on, so I ended up submitting that as
> https://github.com/apache/beam/pull/9899.
>
> But since that still needs work to be fully functional (right now it does
> not handle resharding), I figured I could at least contribute the updates
> to make KinesisIO compatible with the AWS SDK v2, so I split that out and
> submitted it as: https://github.com/apache/beam/pull/11318.
>
> Now some totally reasonable concerns have been raised about maintaining
> two separate codebases, and having to dual-commit every bugfix or feature
> enhancement. That seems to already be the case for all of the other AWS IOs
> in amazon-web-services2 (dynamodb, sns, sqs), but do we want to continue
> that trend? What do members of the community think?
>
> One solution is obviously to rewrite the AWS IOs with an added abstraction
> layer so that the SDK-specific code can be factored out. However, the
> KinesisIO class itself already has dependencies on the v1 AWS SDK so I'm
> not aware of a way that this could be done so that it's
> backwards-compatible. This means that everyone currently using KinesisIO
> would most likely have to update their code to use the new interface
> whether they wanted to use the new v2 SDK or continue to use v1. I think a
> rewrite would also be a non-trivial task with more than a few details to be
> worked out first.
>
> From my perspective, the options are:
>
> - Merge #11318 now and the community can use the v2 AWS SDK with KinesisIO
> and I can continue working on enhanced fanout while a rewrite is being
> worked on, but the community now has to dual-commit any changes to
> KinesisIO.
> - Close #11318 and continue working on #9899 and I alone have to
> dual-commit the changes until it can be merged, at which point everyone can
> use enhanced fanout while the rewrite is being worked on. Then the
> community will also have to dual-commit but only after enhanced fanout is
> available.
> - Close both PRs and just wait for a rewrite, and no one can use the v2
> AWS SDK or enhanced fanout in KinesisIO until after it's done. (Not to
> imply that I wouldn't be interested in contributing to that effort, just
> for the sake of these 2 PRs.)
> - Don't do the rewrite at all and just live with dual-committing until the
> KinesisIO based on the v1 SDK reaches its end of life. No one has to update
> their code to continue using the v1 SDK, only if they want to move to v2.
>
> There may be other options or considerations I'm missing here. What do
> others think?
>
> Thanks,
> Jonothan
>
>
>
Re: Support for AWS SDK v2 and enhanced fanout in KinesisIO
Posted by Alexey Romanenko <ar...@gmail.com>.
I’d like to get back to this question, fairly raised by Jonothan a while ago, since actually it affects not only KinesisIO but also all other AWS IOs in general, that uses AWS SDK of two different versions.
My personal opinion - I’d strongly like to avoid a copy-pasted code for two different versions of the same IO (and double support of this), that is SDK-independent. In this case, if we wish too provide IO for two AWS SDK versions, we have to extract this core logic into one common and SDK-independent framework that could used by any of SDK versions then.
In the same time, if it will require a lot of efforts to achieve that, then probably we need to decide to leave only one IO based on single AWS SDK version (I guess it will be V2 as more modern) and deprecate old version (and then remove it to avoid a confusion for users).
What others think about that?
> On 18 Apr 2020, at 01:48, Jonothan Farr <jo...@gmail.com> wrote:
>
> Hi, I wanted to try out KinesisIO with enhanced fanout for a PoC I was working on, so I ended up submitting that as https://github.com/apache/beam/pull/9899 <https://github.com/apache/beam/pull/9899>.
>
> But since that still needs work to be fully functional (right now it does not handle resharding), I figured I could at least contribute the updates to make KinesisIO compatible with the AWS SDK v2, so I split that out and submitted it as: https://github.com/apache/beam/pull/11318 <https://github.com/apache/beam/pull/11318>.
>
> Now some totally reasonable concerns have been raised about maintaining two separate codebases, and having to dual-commit every bugfix or feature enhancement. That seems to already be the case for all of the other AWS IOs in amazon-web-services2 (dynamodb, sns, sqs), but do we want to continue that trend? What do members of the community think?
>
> One solution is obviously to rewrite the AWS IOs with an added abstraction layer so that the SDK-specific code can be factored out. However, the KinesisIO class itself already has dependencies on the v1 AWS SDK so I'm not aware of a way that this could be done so that it's backwards-compatible. This means that everyone currently using KinesisIO would most likely have to update their code to use the new interface whether they wanted to use the new v2 SDK or continue to use v1. I think a rewrite would also be a non-trivial task with more than a few details to be worked out first.
>
> From my perspective, the options are:
>
> - Merge #11318 now and the community can use the v2 AWS SDK with KinesisIO and I can continue working on enhanced fanout while a rewrite is being worked on, but the community now has to dual-commit any changes to KinesisIO.
> - Close #11318 and continue working on #9899 and I alone have to dual-commit the changes until it can be merged, at which point everyone can use enhanced fanout while the rewrite is being worked on. Then the community will also have to dual-commit but only after enhanced fanout is available.
> - Close both PRs and just wait for a rewrite, and no one can use the v2 AWS SDK or enhanced fanout in KinesisIO until after it's done. (Not to imply that I wouldn't be interested in contributing to that effort, just for the sake of these 2 PRs.)
> - Don't do the rewrite at all and just live with dual-committing until the KinesisIO based on the v1 SDK reaches its end of life. No one has to update their code to continue using the v1 SDK, only if they want to move to v2.
>
> There may be other options or considerations I'm missing here. What do others think?
>
> Thanks,
> Jonothan