KinesisIO Tests - are they run anywhere?

[Piotr Szuberski <pi...@polidea.com>]

I'm writing KinesisIO external transform with python wrapper and I found that the tests aren't executed anywhere in Jenkins. Am I wrong or there is a reason for that?

Re: KinesisIO Tests - are they run anywhere?

[Ismaël Mejía <ie...@gmail.com>]

Just for reference we already run localstack for DynamoDB tests via
testcontainers [1] so everything should be ready (at least module wise) to run
other simulated ITs.  Being able to run all our AWS related ITs with localstack
is a really simple and nice contribution to do if anyone is interested.

About real AWS runs there have been conversations about AWS credits donation in
the past but sadly with no progress. Most recent ones were at the
beginning of this
year so it is not a 'forgotten' subject it is just hard to find someone to
donate the credits or contribute a credit card for this purpose [2].
Contributions welcome ;)

[1] https://www.testcontainers.org/modules/localstack/
[2] https://aws.amazon.com/blogs/opensource/aws-promotional-credits-open-source-projects/

On Tue, Jul 14, 2020 at 11:00 PM Kenneth Knowles <ke...@apache.org> wrote:
>
> Localstack looks really cool. We have to consider resource consumption and latency for where it fits in the testing plan. It definitely seems useful for having ITs easily. For individual IOs, an in-process implementation can be much more performant without losing much accuracy (since it is also just another implementation that is not the real one) so if that is available I would also use one of those (any time you would mock, use an in-process fake instead). Like direct runner vs ULR. You don't need the ULR to test your transform's correctness. Localstack's README discusses the pro/con of each pretty well. Fault injection is pretty compelling and process isolation might reduce dependency troubles.
>
> Kenn
>
> On Fri, Jul 10, 2020 at 4:45 AM Alexey Romanenko <ar...@gmail.com> wrote:
>>
>> I think that we should get back to this question since that time we have more and more AWS-related IO connectors (mostly in Java SDK afaik).
>>
>> It would be great to have a dedicated Beam’s credentials to run all our AWS-releasted ITs against real AWS instance, but till then, I’m +1 to run such tests against 3rd party implementations, for example, “localstack”, as most comprehensive one. Of course we can observe some potential discrepancy in behaviour between real AWS and other implementations, but if it’s not principal things then it should not be a stopper. I believe that regular running ITs, especially for IO connectors, is a very important.
>>
>> On 9 Jul 2020, at 22:18, Luke Cwik <lc...@google.com> wrote:
>>
>> It has come up a few times[1, 2, 3, 4] and there have also been a few comments over time about whether someone could donate AWS resources to the project.
>>
>> 1: https://issues.apache.org/jira/browse/BEAM-601
>> 2: https://issues.apache.org/jira/browse/BEAM-3373
>> 3: https://issues.apache.org/jira/browse/BEAM-3550
>> 4: https://issues.apache.org/jira/browse/BEAM-3032
>>
>> On Thu, Jul 9, 2020 at 1:02 PM Mani Kolbe <ma...@gmail.com> wrote:
>>>
>>> Have you guys considered using localstack to run AWS service based integration tests?
>>>
>>> https://github.com/localstack/localstack
>>>
>>> On Thu, 9 Jul, 2020, 5:25 PM Piotr Szuberski, <pi...@polidea.com> wrote:
>>>>
>>>> Yeah, I meant KinesisIOIT tests. I'll do the same with the cross-language it tests then. Thanks for your reply :)
>>>>
>>>> On 2020/07/08 17:13:11, Alexey Romanenko <ar...@gmail.com> wrote:
>>>> > If you mean Java KinesisIO tests, then unit tests are running on Jenkins [1] and ITs are not running since it requires AWS credentials that we don’t have dedicated to Beam for the moment.
>>>> >
>>>> > In the same time, you can run KinesisIOIT with your own credentials, like we do in Talend (a company that I work for).
>>>> >
>>>> > [1] https://ci-beam.apache.org/job/beam_PreCommit_Java_Commit/12209/testReport/org.apache.beam.sdk.io.kinesis/ <https://ci-beam.apache.org/job/beam_PreCommit_Java_Commit/12209/testReport/org.apache.beam.sdk.io.kinesis/>
>>>> >
>>>> > > On 8 Jul 2020, at 13:11, Piotr Szuberski <pi...@polidea.com> wrote:
>>>> > >
>>>> > > I'm writing KinesisIO external transform with python wrapper and I found that the tests aren't executed anywhere in Jenkins. Am I wrong or there is a reason for that?
>>>> >
>>>> >
>>
>>

Re: KinesisIO Tests - are they run anywhere?

[Kenneth Knowles <ke...@apache.org>]

Localstack looks really cool. We have to consider resource consumption and
latency for where it fits in the testing plan. It definitely seems useful
for having ITs easily. For individual IOs, an in-process implementation can
be much more performant without losing much accuracy (since it is also just
another implementation that is not the real one) so if that is available I
would also use one of those (any time you would mock, use an in-process
fake instead). Like direct runner vs ULR. You don't need the ULR to test
your transform's correctness. Localstack's README discusses the pro/con of
each pretty well. Fault injection is pretty compelling and process
isolation might reduce dependency troubles.

Kenn

On Fri, Jul 10, 2020 at 4:45 AM Alexey Romanenko <ar...@gmail.com>
wrote:

> I think that we should get back to this question since that time we have
> more and more AWS-related IO connectors (mostly in Java SDK afaik).
>
> It would be great to have a dedicated Beam’s credentials to run all our
> AWS-releasted ITs against real AWS instance, but till then, I’m +1 to run
> such tests against 3rd party implementations, for example, “localstack”, as
> most comprehensive one. Of course we can observe some potential discrepancy
> in behaviour between real AWS and other implementations, but if it’s not
> principal things then it should not be a stopper. I believe that regular
> running ITs, especially for IO connectors, is a very important.
>
> On 9 Jul 2020, at 22:18, Luke Cwik <lc...@google.com> wrote:
>
> It has come up a few times[1, 2, 3, 4] and there have also been a few
> comments over time about whether someone could donate AWS resources to the
> project.
>
> 1: https://issues.apache.org/jira/browse/BEAM-601
> 2: https://issues.apache.org/jira/browse/BEAM-3373
> 3: https://issues.apache.org/jira/browse/BEAM-3550
> 4: https://issues.apache.org/jira/browse/BEAM-3032
>
> On Thu, Jul 9, 2020 at 1:02 PM Mani Kolbe <ma...@gmail.com> wrote:
>
>> Have you guys considered using localstack to run AWS service based
>> integration tests?
>>
>> https://github.com/localstack/localstack
>>
>> On Thu, 9 Jul, 2020, 5:25 PM Piotr Szuberski, <
>> piotr.szuberski@polidea.com> wrote:
>>
>>> Yeah, I meant KinesisIOIT tests. I'll do the same with the
>>> cross-language it tests then. Thanks for your reply :)
>>>
>>> On 2020/07/08 17:13:11, Alexey Romanenko <ar...@gmail.com>
>>> wrote:
>>> > If you mean Java KinesisIO tests, then unit tests are running on
>>> Jenkins [1] and ITs are not running since it requires AWS credentials that
>>> we don’t have dedicated to Beam for the moment.
>>> >
>>> > In the same time, you can run KinesisIOIT with your own credentials,
>>> like we do in Talend (a company that I work for).
>>> >
>>> > [1]
>>> https://ci-beam.apache.org/job/beam_PreCommit_Java_Commit/12209/testReport/org.apache.beam.sdk.io.kinesis/
>>> <
>>> https://ci-beam.apache.org/job/beam_PreCommit_Java_Commit/12209/testReport/org.apache.beam.sdk.io.kinesis/
>>> >
>>> >
>>> > > On 8 Jul 2020, at 13:11, Piotr Szuberski <
>>> piotr.szuberski@polidea.com> wrote:
>>> > >
>>> > > I'm writing KinesisIO external transform with python wrapper and I
>>> found that the tests aren't executed anywhere in Jenkins. Am I wrong or
>>> there is a reason for that?
>>> >
>>> >
>>>
>>
>

Re: KinesisIO Tests - are they run anywhere?

[Alexey Romanenko <ar...@gmail.com>]

I think that we should get back to this question since that time we have more and more AWS-related IO connectors (mostly in Java SDK afaik).

It would be great to have a dedicated Beam’s credentials to run all our AWS-releasted ITs against real AWS instance, but till then, I’m +1 to run such tests against 3rd party implementations, for example, “localstack”, as most comprehensive one. Of course we can observe some potential discrepancy in behaviour between real AWS and other implementations, but if it’s not principal things then it should not be a stopper. I believe that regular running ITs, especially for IO connectors, is a very important. 

> On 9 Jul 2020, at 22:18, Luke Cwik <lc...@google.com> wrote:
> 
> It has come up a few times[1, 2, 3, 4] and there have also been a few comments over time about whether someone could donate AWS resources to the project.
> 
> 1: https://issues.apache.org/jira/browse/BEAM-601 <https://issues.apache.org/jira/browse/BEAM-601>
> 2: https://issues.apache.org/jira/browse/BEAM-3373 <https://issues.apache.org/jira/browse/BEAM-3373>
> 3: https://issues.apache.org/jira/browse/BEAM-3550 <https://issues.apache.org/jira/browse/BEAM-3550>
> 4: https://issues.apache.org/jira/browse/BEAM-3032 <https://issues.apache.org/jira/browse/BEAM-3032>
> On Thu, Jul 9, 2020 at 1:02 PM Mani Kolbe <manikolbe@gmail.com <ma...@gmail.com>> wrote:
> Have you guys considered using localstack to run AWS service based integration tests?
> 
> https://github.com/localstack/localstack <https://github.com/localstack/localstack>
> 
> On Thu, 9 Jul, 2020, 5:25 PM Piotr Szuberski, <piotr.szuberski@polidea.com <ma...@polidea.com>> wrote:
> Yeah, I meant KinesisIOIT tests. I'll do the same with the cross-language it tests then. Thanks for your reply :)
> 
> On 2020/07/08 17:13:11, Alexey Romanenko <aromanenko.dev@gmail.com <ma...@gmail.com>> wrote: 
> > If you mean Java KinesisIO tests, then unit tests are running on Jenkins [1] and ITs are not running since it requires AWS credentials that we don’t have dedicated to Beam for the moment.
> > 
> > In the same time, you can run KinesisIOIT with your own credentials, like we do in Talend (a company that I work for).
> > 
> > [1] https://ci-beam.apache.org/job/beam_PreCommit_Java_Commit/12209/testReport/org.apache.beam.sdk.io.kinesis/ <https://ci-beam.apache.org/job/beam_PreCommit_Java_Commit/12209/testReport/org.apache.beam.sdk.io.kinesis/> <https://ci-beam.apache.org/job/beam_PreCommit_Java_Commit/12209/testReport/org.apache.beam.sdk.io.kinesis/ <https://ci-beam.apache.org/job/beam_PreCommit_Java_Commit/12209/testReport/org.apache.beam.sdk.io.kinesis/>>
> > 
> > > On 8 Jul 2020, at 13:11, Piotr Szuberski <piotr.szuberski@polidea.com <ma...@polidea.com>> wrote:
> > > 
> > > I'm writing KinesisIO external transform with python wrapper and I found that the tests aren't executed anywhere in Jenkins. Am I wrong or there is a reason for that?
> > 
> > 

Re: KinesisIO Tests - are they run anywhere?

[Luke Cwik <lc...@google.com>]

It has come up a few times[1, 2, 3, 4] and there have also been a few
comments over time about whether someone could donate AWS resources to the
project.

1: https://issues.apache.org/jira/browse/BEAM-601
2: https://issues.apache.org/jira/browse/BEAM-3373
3: https://issues.apache.org/jira/browse/BEAM-3550
4: https://issues.apache.org/jira/browse/BEAM-3032

On Thu, Jul 9, 2020 at 1:02 PM Mani Kolbe <ma...@gmail.com> wrote:

> Have you guys considered using localstack to run AWS service based
> integration tests?
>
> https://github.com/localstack/localstack
>
> On Thu, 9 Jul, 2020, 5:25 PM Piotr Szuberski, <pi...@polidea.com>
> wrote:
>
>> Yeah, I meant KinesisIOIT tests. I'll do the same with the cross-language
>> it tests then. Thanks for your reply :)
>>
>> On 2020/07/08 17:13:11, Alexey Romanenko <ar...@gmail.com>
>> wrote:
>> > If you mean Java KinesisIO tests, then unit tests are running on
>> Jenkins [1] and ITs are not running since it requires AWS credentials that
>> we don’t have dedicated to Beam for the moment.
>> >
>> > In the same time, you can run KinesisIOIT with your own credentials,
>> like we do in Talend (a company that I work for).
>> >
>> > [1]
>> https://ci-beam.apache.org/job/beam_PreCommit_Java_Commit/12209/testReport/org.apache.beam.sdk.io.kinesis/
>> <
>> https://ci-beam.apache.org/job/beam_PreCommit_Java_Commit/12209/testReport/org.apache.beam.sdk.io.kinesis/
>> >
>> >
>> > > On 8 Jul 2020, at 13:11, Piotr Szuberski <pi...@polidea.com>
>> wrote:
>> > >
>> > > I'm writing KinesisIO external transform with python wrapper and I
>> found that the tests aren't executed anywhere in Jenkins. Am I wrong or
>> there is a reason for that?
>> >
>> >
>>
>

Re: KinesisIO Tests - are they run anywhere?

[Mani Kolbe <ma...@gmail.com>]

Have you guys considered using localstack to run AWS service based
integration tests?

https://github.com/localstack/localstack

On Thu, 9 Jul, 2020, 5:25 PM Piotr Szuberski, <pi...@polidea.com>
wrote:

> Yeah, I meant KinesisIOIT tests. I'll do the same with the cross-language
> it tests then. Thanks for your reply :)
>
> On 2020/07/08 17:13:11, Alexey Romanenko <ar...@gmail.com>
> wrote:
> > If you mean Java KinesisIO tests, then unit tests are running on Jenkins
> [1] and ITs are not running since it requires AWS credentials that we don’t
> have dedicated to Beam for the moment.
> >
> > In the same time, you can run KinesisIOIT with your own credentials,
> like we do in Talend (a company that I work for).
> >
> > [1]
> https://ci-beam.apache.org/job/beam_PreCommit_Java_Commit/12209/testReport/org.apache.beam.sdk.io.kinesis/
> <
> https://ci-beam.apache.org/job/beam_PreCommit_Java_Commit/12209/testReport/org.apache.beam.sdk.io.kinesis/
> >
> >
> > > On 8 Jul 2020, at 13:11, Piotr Szuberski <pi...@polidea.com>
> wrote:
> > >
> > > I'm writing KinesisIO external transform with python wrapper and I
> found that the tests aren't executed anywhere in Jenkins. Am I wrong or
> there is a reason for that?
> >
> >
>

Re: KinesisIO Tests - are they run anywhere?

[Piotr Szuberski <pi...@polidea.com>]

Yeah, I meant KinesisIOIT tests. I'll do the same with the cross-language it tests then. Thanks for your reply :)

On 2020/07/08 17:13:11, Alexey Romanenko <ar...@gmail.com> wrote: 
> If you mean Java KinesisIO tests, then unit tests are running on Jenkins [1] and ITs are not running since it requires AWS credentials that we don’t have dedicated to Beam for the moment.
> 
> In the same time, you can run KinesisIOIT with your own credentials, like we do in Talend (a company that I work for).
> 
> [1] https://ci-beam.apache.org/job/beam_PreCommit_Java_Commit/12209/testReport/org.apache.beam.sdk.io.kinesis/ <https://ci-beam.apache.org/job/beam_PreCommit_Java_Commit/12209/testReport/org.apache.beam.sdk.io.kinesis/>
> 
> > On 8 Jul 2020, at 13:11, Piotr Szuberski <pi...@polidea.com> wrote:
> > 
> > I'm writing KinesisIO external transform with python wrapper and I found that the tests aren't executed anywhere in Jenkins. Am I wrong or there is a reason for that?
> 
> 

Re: KinesisIO Tests - are they run anywhere?

[Alexey Romanenko <ar...@gmail.com>]

If you mean Java KinesisIO tests, then unit tests are running on Jenkins [1] and ITs are not running since it requires AWS credentials that we don’t have dedicated to Beam for the moment.

In the same time, you can run KinesisIOIT with your own credentials, like we do in Talend (a company that I work for).

[1] https://ci-beam.apache.org/job/beam_PreCommit_Java_Commit/12209/testReport/org.apache.beam.sdk.io.kinesis/ <https://ci-beam.apache.org/job/beam_PreCommit_Java_Commit/12209/testReport/org.apache.beam.sdk.io.kinesis/>

> On 8 Jul 2020, at 13:11, Piotr Szuberski <pi...@polidea.com> wrote:
> 
> I'm writing KinesisIO external transform with python wrapper and I found that the tests aren't executed anywhere in Jenkins. Am I wrong or there is a reason for that?