You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@beam.apache.org by Jason Kuster <ja...@google.com.INVALID> on 2017/03/21 21:28:35 UTC
First IO IT Running!
Hi all,
Exciting news! As of yesterday, we have checked in the Jenkins
configuration for our first continuously running IO Integration Test! You
can check it out in Jenkins here[1]. We’re also publishing results to a
database, and we’ve turned up a basic dashboarding system where you can see
the results here[2]. Caveat: there are only two runs, and we’ll be tweaking
the underlying system still, so don’t panic that we’re up and to the right
currently. ;)
This is the first test running continuously on top of the performance / IO
testing infrastructure described in this doc[3]. Initial support for Beam
is now present in PerfKit Benchmarker; given what they had already, it was
easiest to add support for Dataflow and Java. We need your help to add
additional support! The doc lists a number of JIRA issues to build out
support for other systems. I’m happy to work with people to help them
understand what is necessary for these tasks; just send an email to the
list if you need help and I’ll help you move forwards.
Looking forward to it!
Jason
[1] https://builds.apache.org/job/beam_PerformanceTests_JDBC/
[2]
https://apache-beam-testing.appspot.com/explore?dashboard=5714163003293696
[3]
https://docs.google.com/document/d/1PsjGPSN6FuorEEPrKEP3u3m16tyOzph5FnL2DhaRDz0/edit?ts=58a78e73
--
-------
Jason Kuster
Apache Beam / Google Cloud Dataflow
Re: First IO IT Running!
Posted by Kenneth Knowles <kl...@google.com.INVALID>.
This is a really exciting development! I would definitely like to help out.
Still ingesting the docs and JIRAs.
On Tue, Mar 21, 2017 at 3:01 PM, Stephen Sisk <si...@google.com.invalid>
wrote:
> I'm really excited to see these tests are running!
>
> These Jdbc tests are testing against a postgres instance - that instance is
> running on the kubernetes cluster I've set up for beam IO ITs as discussed
> in the "Hosting data stores for IO transform testing" thread[0]. I set up
> that postgres instance using the kubernetes scripts for Jdbc[1]. Anyone can
> run their own kubernetes cluster and do the same thing for themselves to
> run the ITs. (I'd actually to love to hear about that if anyone does it.)
>
> I'm excited to get a few more ITs using this infrastructure so we can test
> it out/smooth out the remaining rough edges in creating ITs. I'm happy to
> answer questions about that on the mailing list, but we obviously have to
> have the process written down - the Testing IO Transforms in Apache Beam
> doc [2] covers how to do this, but is still rough. I'm working on getting
> that up on the website and ironing out the rough edges [3], but generally
> reading that doc plus checking out how the JdbcIO or ElasticsearchIO tests
> work should give you a sense of how to get it working. I'm also thinking we
> might want to simplify the way we do data loading, so I don't consider this
> process fully stabilized, but I'll port code written according to the
> current standards to the new standards if we make changes.
>
> ElasticsearchIO has all the prerequisites, so I'd like to get them going in
> the near future. I know JB has started on this in his RedisIO PR, and the
> HadoopInputFormatIO also has ITs & k8 scripts, so there's more in the pipe.
> For now, each datastore has to be manually set up, but I'd like to automate
> that process - I'll file a JIRA ticket shortly for that.
>
> Thanks,
> Stephen
> [0] Hosting data stores for IO transform testing -
> https://lists.apache.org/thread.html/9fd3c51cb679706efa4d0df2111a6a
> c438b851818b639aba644607af@%3Cdev.beam.apache.org%3E
> [1] Postgres k8 scripts -
> https://github.com/apache/beam/tree/master/sdks/java/io/
> jdbc/src/test/resources/kubernetes
> [2] IO testing guide -
> https://docs.google.com/document/d/153J9jPQhMCNi_eBzJfhAg-
> NprQ7vbf1jNVRgdqeEE8I/edit?usp=sharing
> [3] Jira for IO guide - https://issues.apache.org/jira/browse/BEAM-1025
>
> On Tue, Mar 21, 2017 at 2:28 PM Jason Kuster <jasonkuster@google.com.
> invalid>
> wrote:
>
> > Hi all,
> >
> > Exciting news! As of yesterday, we have checked in the Jenkins
> > configuration for our first continuously running IO Integration Test! You
> > can check it out in Jenkins here[1]. We’re also publishing results to a
> > database, and we’ve turned up a basic dashboarding system where you can
> see
> > the results here[2]. Caveat: there are only two runs, and we’ll be
> tweaking
> > the underlying system still, so don’t panic that we’re up and to the
> right
> > currently. ;)
> >
> > This is the first test running continuously on top of the performance /
> IO
> > testing infrastructure described in this doc[3]. Initial support for
> Beam
> > is now present in PerfKit Benchmarker; given what they had already, it
> was
> > easiest to add support for Dataflow and Java. We need your help to add
> > additional support! The doc lists a number of JIRA issues to build out
> > support for other systems. I’m happy to work with people to help them
> > understand what is necessary for these tasks; just send an email to the
> > list if you need help and I’ll help you move forwards.
> >
> > Looking forward to it!
> >
> > Jason
> >
> > [1] https://builds.apache.org/job/beam_PerformanceTests_JDBC/
> > [2]
> > https://apache-beam-testing.appspot.com/explore?dashboard=
> 5714163003293696
> > [3]
> >
> > https://docs.google.com/document/d/1PsjGPSN6FuorEEPrKEP3u3m16tyOz
> ph5FnL2DhaRDz0/edit?ts=58a78e73
> >
> > --
> > -------
> > Jason Kuster
> > Apache Beam / Google Cloud Dataflow
> >
>
Re: First IO IT Running!
Posted by Ismaël Mejía <ie...@apache.org>.
Excellent news, I am eager to see more IOs/Runners been included in
the Integration Tests, and I will be glad to contribute in anything I
can.
Congratulations for this important milestone.
Ismaël
ps. I will try to reproduce the Kubernetes setup so I will be
eventually annoying you with questions.
On Wed, Mar 22, 2017 at 11:28 AM, Aljoscha Krettek <al...@apache.org> wrote:
> Great news! I can’t wait to also have support for this for the Flink Runner. Which is partially blocked by me or others working on the Flink Runner, I guess… :-(
>> On 22 Mar 2017, at 05:15, Jean-Baptiste Onofré <jb...@nanthrax.net> wrote:
>>
>> Awesome !!! Great news !
>>
>> Thanks guys for that !
>>
>> I started to implement IT in JMS, MQTT, Redis, Cassandra IOs. I keep you posted.
>>
>> Regards
>> JB
>>
>> On 03/21/2017 11:01 PM, Stephen Sisk wrote:
>>> I'm really excited to see these tests are running!
>>>
>>> These Jdbc tests are testing against a postgres instance - that instance is
>>> running on the kubernetes cluster I've set up for beam IO ITs as discussed
>>> in the "Hosting data stores for IO transform testing" thread[0]. I set up
>>> that postgres instance using the kubernetes scripts for Jdbc[1]. Anyone can
>>> run their own kubernetes cluster and do the same thing for themselves to
>>> run the ITs. (I'd actually to love to hear about that if anyone does it.)
>>>
>>> I'm excited to get a few more ITs using this infrastructure so we can test
>>> it out/smooth out the remaining rough edges in creating ITs. I'm happy to
>>> answer questions about that on the mailing list, but we obviously have to
>>> have the process written down - the Testing IO Transforms in Apache Beam
>>> doc [2] covers how to do this, but is still rough. I'm working on getting
>>> that up on the website and ironing out the rough edges [3], but generally
>>> reading that doc plus checking out how the JdbcIO or ElasticsearchIO tests
>>> work should give you a sense of how to get it working. I'm also thinking we
>>> might want to simplify the way we do data loading, so I don't consider this
>>> process fully stabilized, but I'll port code written according to the
>>> current standards to the new standards if we make changes.
>>>
>>> ElasticsearchIO has all the prerequisites, so I'd like to get them going in
>>> the near future. I know JB has started on this in his RedisIO PR, and the
>>> HadoopInputFormatIO also has ITs & k8 scripts, so there's more in the pipe.
>>> For now, each datastore has to be manually set up, but I'd like to automate
>>> that process - I'll file a JIRA ticket shortly for that.
>>>
>>> Thanks,
>>> Stephen
>>> [0] Hosting data stores for IO transform testing -
>>> https://lists.apache.org/thread.html/9fd3c51cb679706efa4d0df2111a6ac438b851818b639aba644607af@%3Cdev.beam.apache.org%3E
>>> [1] Postgres k8 scripts -
>>> https://github.com/apache/beam/tree/master/sdks/java/io/jdbc/src/test/resources/kubernetes
>>> [2] IO testing guide -
>>> https://docs.google.com/document/d/153J9jPQhMCNi_eBzJfhAg-NprQ7vbf1jNVRgdqeEE8I/edit?usp=sharing
>>> [3] Jira for IO guide - https://issues.apache.org/jira/browse/BEAM-1025
>>>
>>> On Tue, Mar 21, 2017 at 2:28 PM Jason Kuster <ja...@google.com.invalid>
>>> wrote:
>>>
>>>> Hi all,
>>>>
>>>> Exciting news! As of yesterday, we have checked in the Jenkins
>>>> configuration for our first continuously running IO Integration Test! You
>>>> can check it out in Jenkins here[1]. We’re also publishing results to a
>>>> database, and we’ve turned up a basic dashboarding system where you can see
>>>> the results here[2]. Caveat: there are only two runs, and we’ll be tweaking
>>>> the underlying system still, so don’t panic that we’re up and to the right
>>>> currently. ;)
>>>>
>>>> This is the first test running continuously on top of the performance / IO
>>>> testing infrastructure described in this doc[3]. Initial support for Beam
>>>> is now present in PerfKit Benchmarker; given what they had already, it was
>>>> easiest to add support for Dataflow and Java. We need your help to add
>>>> additional support! The doc lists a number of JIRA issues to build out
>>>> support for other systems. I’m happy to work with people to help them
>>>> understand what is necessary for these tasks; just send an email to the
>>>> list if you need help and I’ll help you move forwards.
>>>>
>>>> Looking forward to it!
>>>>
>>>> Jason
>>>>
>>>> [1] https://builds.apache.org/job/beam_PerformanceTests_JDBC/
>>>> [2]
>>>> https://apache-beam-testing.appspot.com/explore?dashboard=5714163003293696
>>>> [3]
>>>>
>>>> https://docs.google.com/document/d/1PsjGPSN6FuorEEPrKEP3u3m16tyOzph5FnL2DhaRDz0/edit?ts=58a78e73
>>>>
>>>> --
>>>> -------
>>>> Jason Kuster
>>>> Apache Beam / Google Cloud Dataflow
>>>>
>>>
>>
>> --
>> Jean-Baptiste Onofré
>> jbonofre@apache.org
>> http://blog.nanthrax.net
>> Talend - http://www.talend.com
>
Re: First IO IT Running!
Posted by Aljoscha Krettek <al...@apache.org>.
Great news! I can’t wait to also have support for this for the Flink Runner. Which is partially blocked by me or others working on the Flink Runner, I guess… :-(
> On 22 Mar 2017, at 05:15, Jean-Baptiste Onofré <jb...@nanthrax.net> wrote:
>
> Awesome !!! Great news !
>
> Thanks guys for that !
>
> I started to implement IT in JMS, MQTT, Redis, Cassandra IOs. I keep you posted.
>
> Regards
> JB
>
> On 03/21/2017 11:01 PM, Stephen Sisk wrote:
>> I'm really excited to see these tests are running!
>>
>> These Jdbc tests are testing against a postgres instance - that instance is
>> running on the kubernetes cluster I've set up for beam IO ITs as discussed
>> in the "Hosting data stores for IO transform testing" thread[0]. I set up
>> that postgres instance using the kubernetes scripts for Jdbc[1]. Anyone can
>> run their own kubernetes cluster and do the same thing for themselves to
>> run the ITs. (I'd actually to love to hear about that if anyone does it.)
>>
>> I'm excited to get a few more ITs using this infrastructure so we can test
>> it out/smooth out the remaining rough edges in creating ITs. I'm happy to
>> answer questions about that on the mailing list, but we obviously have to
>> have the process written down - the Testing IO Transforms in Apache Beam
>> doc [2] covers how to do this, but is still rough. I'm working on getting
>> that up on the website and ironing out the rough edges [3], but generally
>> reading that doc plus checking out how the JdbcIO or ElasticsearchIO tests
>> work should give you a sense of how to get it working. I'm also thinking we
>> might want to simplify the way we do data loading, so I don't consider this
>> process fully stabilized, but I'll port code written according to the
>> current standards to the new standards if we make changes.
>>
>> ElasticsearchIO has all the prerequisites, so I'd like to get them going in
>> the near future. I know JB has started on this in his RedisIO PR, and the
>> HadoopInputFormatIO also has ITs & k8 scripts, so there's more in the pipe.
>> For now, each datastore has to be manually set up, but I'd like to automate
>> that process - I'll file a JIRA ticket shortly for that.
>>
>> Thanks,
>> Stephen
>> [0] Hosting data stores for IO transform testing -
>> https://lists.apache.org/thread.html/9fd3c51cb679706efa4d0df2111a6ac438b851818b639aba644607af@%3Cdev.beam.apache.org%3E
>> [1] Postgres k8 scripts -
>> https://github.com/apache/beam/tree/master/sdks/java/io/jdbc/src/test/resources/kubernetes
>> [2] IO testing guide -
>> https://docs.google.com/document/d/153J9jPQhMCNi_eBzJfhAg-NprQ7vbf1jNVRgdqeEE8I/edit?usp=sharing
>> [3] Jira for IO guide - https://issues.apache.org/jira/browse/BEAM-1025
>>
>> On Tue, Mar 21, 2017 at 2:28 PM Jason Kuster <ja...@google.com.invalid>
>> wrote:
>>
>>> Hi all,
>>>
>>> Exciting news! As of yesterday, we have checked in the Jenkins
>>> configuration for our first continuously running IO Integration Test! You
>>> can check it out in Jenkins here[1]. We’re also publishing results to a
>>> database, and we’ve turned up a basic dashboarding system where you can see
>>> the results here[2]. Caveat: there are only two runs, and we’ll be tweaking
>>> the underlying system still, so don’t panic that we’re up and to the right
>>> currently. ;)
>>>
>>> This is the first test running continuously on top of the performance / IO
>>> testing infrastructure described in this doc[3]. Initial support for Beam
>>> is now present in PerfKit Benchmarker; given what they had already, it was
>>> easiest to add support for Dataflow and Java. We need your help to add
>>> additional support! The doc lists a number of JIRA issues to build out
>>> support for other systems. I’m happy to work with people to help them
>>> understand what is necessary for these tasks; just send an email to the
>>> list if you need help and I’ll help you move forwards.
>>>
>>> Looking forward to it!
>>>
>>> Jason
>>>
>>> [1] https://builds.apache.org/job/beam_PerformanceTests_JDBC/
>>> [2]
>>> https://apache-beam-testing.appspot.com/explore?dashboard=5714163003293696
>>> [3]
>>>
>>> https://docs.google.com/document/d/1PsjGPSN6FuorEEPrKEP3u3m16tyOzph5FnL2DhaRDz0/edit?ts=58a78e73
>>>
>>> --
>>> -------
>>> Jason Kuster
>>> Apache Beam / Google Cloud Dataflow
>>>
>>
>
> --
> Jean-Baptiste Onofré
> jbonofre@apache.org
> http://blog.nanthrax.net
> Talend - http://www.talend.com
Re: First IO IT Running!
Posted by Jean-Baptiste Onofré <jb...@nanthrax.net>.
Awesome !!! Great news !
Thanks guys for that !
I started to implement IT in JMS, MQTT, Redis, Cassandra IOs. I keep you posted.
Regards
JB
On 03/21/2017 11:01 PM, Stephen Sisk wrote:
> I'm really excited to see these tests are running!
>
> These Jdbc tests are testing against a postgres instance - that instance is
> running on the kubernetes cluster I've set up for beam IO ITs as discussed
> in the "Hosting data stores for IO transform testing" thread[0]. I set up
> that postgres instance using the kubernetes scripts for Jdbc[1]. Anyone can
> run their own kubernetes cluster and do the same thing for themselves to
> run the ITs. (I'd actually to love to hear about that if anyone does it.)
>
> I'm excited to get a few more ITs using this infrastructure so we can test
> it out/smooth out the remaining rough edges in creating ITs. I'm happy to
> answer questions about that on the mailing list, but we obviously have to
> have the process written down - the Testing IO Transforms in Apache Beam
> doc [2] covers how to do this, but is still rough. I'm working on getting
> that up on the website and ironing out the rough edges [3], but generally
> reading that doc plus checking out how the JdbcIO or ElasticsearchIO tests
> work should give you a sense of how to get it working. I'm also thinking we
> might want to simplify the way we do data loading, so I don't consider this
> process fully stabilized, but I'll port code written according to the
> current standards to the new standards if we make changes.
>
> ElasticsearchIO has all the prerequisites, so I'd like to get them going in
> the near future. I know JB has started on this in his RedisIO PR, and the
> HadoopInputFormatIO also has ITs & k8 scripts, so there's more in the pipe.
> For now, each datastore has to be manually set up, but I'd like to automate
> that process - I'll file a JIRA ticket shortly for that.
>
> Thanks,
> Stephen
> [0] Hosting data stores for IO transform testing -
> https://lists.apache.org/thread.html/9fd3c51cb679706efa4d0df2111a6ac438b851818b639aba644607af@%3Cdev.beam.apache.org%3E
> [1] Postgres k8 scripts -
> https://github.com/apache/beam/tree/master/sdks/java/io/jdbc/src/test/resources/kubernetes
> [2] IO testing guide -
> https://docs.google.com/document/d/153J9jPQhMCNi_eBzJfhAg-NprQ7vbf1jNVRgdqeEE8I/edit?usp=sharing
> [3] Jira for IO guide - https://issues.apache.org/jira/browse/BEAM-1025
>
> On Tue, Mar 21, 2017 at 2:28 PM Jason Kuster <ja...@google.com.invalid>
> wrote:
>
>> Hi all,
>>
>> Exciting news! As of yesterday, we have checked in the Jenkins
>> configuration for our first continuously running IO Integration Test! You
>> can check it out in Jenkins here[1]. We\u2019re also publishing results to a
>> database, and we\u2019ve turned up a basic dashboarding system where you can see
>> the results here[2]. Caveat: there are only two runs, and we\u2019ll be tweaking
>> the underlying system still, so don\u2019t panic that we\u2019re up and to the right
>> currently. ;)
>>
>> This is the first test running continuously on top of the performance / IO
>> testing infrastructure described in this doc[3]. Initial support for Beam
>> is now present in PerfKit Benchmarker; given what they had already, it was
>> easiest to add support for Dataflow and Java. We need your help to add
>> additional support! The doc lists a number of JIRA issues to build out
>> support for other systems. I\u2019m happy to work with people to help them
>> understand what is necessary for these tasks; just send an email to the
>> list if you need help and I\u2019ll help you move forwards.
>>
>> Looking forward to it!
>>
>> Jason
>>
>> [1] https://builds.apache.org/job/beam_PerformanceTests_JDBC/
>> [2]
>> https://apache-beam-testing.appspot.com/explore?dashboard=5714163003293696
>> [3]
>>
>> https://docs.google.com/document/d/1PsjGPSN6FuorEEPrKEP3u3m16tyOzph5FnL2DhaRDz0/edit?ts=58a78e73
>>
>> --
>> -------
>> Jason Kuster
>> Apache Beam / Google Cloud Dataflow
>>
>
--
Jean-Baptiste Onofr�
jbonofre@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com
Re: First IO IT Running!
Posted by Stephen Sisk <si...@google.com.INVALID>.
I'm really excited to see these tests are running!
These Jdbc tests are testing against a postgres instance - that instance is
running on the kubernetes cluster I've set up for beam IO ITs as discussed
in the "Hosting data stores for IO transform testing" thread[0]. I set up
that postgres instance using the kubernetes scripts for Jdbc[1]. Anyone can
run their own kubernetes cluster and do the same thing for themselves to
run the ITs. (I'd actually to love to hear about that if anyone does it.)
I'm excited to get a few more ITs using this infrastructure so we can test
it out/smooth out the remaining rough edges in creating ITs. I'm happy to
answer questions about that on the mailing list, but we obviously have to
have the process written down - the Testing IO Transforms in Apache Beam
doc [2] covers how to do this, but is still rough. I'm working on getting
that up on the website and ironing out the rough edges [3], but generally
reading that doc plus checking out how the JdbcIO or ElasticsearchIO tests
work should give you a sense of how to get it working. I'm also thinking we
might want to simplify the way we do data loading, so I don't consider this
process fully stabilized, but I'll port code written according to the
current standards to the new standards if we make changes.
ElasticsearchIO has all the prerequisites, so I'd like to get them going in
the near future. I know JB has started on this in his RedisIO PR, and the
HadoopInputFormatIO also has ITs & k8 scripts, so there's more in the pipe.
For now, each datastore has to be manually set up, but I'd like to automate
that process - I'll file a JIRA ticket shortly for that.
Thanks,
Stephen
[0] Hosting data stores for IO transform testing -
https://lists.apache.org/thread.html/9fd3c51cb679706efa4d0df2111a6ac438b851818b639aba644607af@%3Cdev.beam.apache.org%3E
[1] Postgres k8 scripts -
https://github.com/apache/beam/tree/master/sdks/java/io/jdbc/src/test/resources/kubernetes
[2] IO testing guide -
https://docs.google.com/document/d/153J9jPQhMCNi_eBzJfhAg-NprQ7vbf1jNVRgdqeEE8I/edit?usp=sharing
[3] Jira for IO guide - https://issues.apache.org/jira/browse/BEAM-1025
On Tue, Mar 21, 2017 at 2:28 PM Jason Kuster <ja...@google.com.invalid>
wrote:
> Hi all,
>
> Exciting news! As of yesterday, we have checked in the Jenkins
> configuration for our first continuously running IO Integration Test! You
> can check it out in Jenkins here[1]. We’re also publishing results to a
> database, and we’ve turned up a basic dashboarding system where you can see
> the results here[2]. Caveat: there are only two runs, and we’ll be tweaking
> the underlying system still, so don’t panic that we’re up and to the right
> currently. ;)
>
> This is the first test running continuously on top of the performance / IO
> testing infrastructure described in this doc[3]. Initial support for Beam
> is now present in PerfKit Benchmarker; given what they had already, it was
> easiest to add support for Dataflow and Java. We need your help to add
> additional support! The doc lists a number of JIRA issues to build out
> support for other systems. I’m happy to work with people to help them
> understand what is necessary for these tasks; just send an email to the
> list if you need help and I’ll help you move forwards.
>
> Looking forward to it!
>
> Jason
>
> [1] https://builds.apache.org/job/beam_PerformanceTests_JDBC/
> [2]
> https://apache-beam-testing.appspot.com/explore?dashboard=5714163003293696
> [3]
>
> https://docs.google.com/document/d/1PsjGPSN6FuorEEPrKEP3u3m16tyOzph5FnL2DhaRDz0/edit?ts=58a78e73
>
> --
> -------
> Jason Kuster
> Apache Beam / Google Cloud Dataflow
>