You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@beam.apache.org by ssisk <gi...@git.apache.org> on 2017/01/25 01:44:28 UTC
[GitHub] beam pull request #1840: BEAM-1310 Add integration tests for JdbcIO
GitHub user ssisk opened a pull request:
https://github.com/apache/beam/pull/1840
BEAM-1310 Add integration tests for JdbcIO
Be sure to do all of the following to help us incorporate your contribution
quickly and easily:
- [X ] Make sure the PR title is formatted like:
`[BEAM-<Jira issue #>] Description of pull request`
- [X ] Make sure tests pass via `mvn clean verify`. (Even better, enable
Travis-CI on your fork and ensure the whole test matrix passes).
- [X ] Replace `<Jira issue #>` in the title with the actual Jira issue
number, if there is one.
- [X ] If this contribution is large, please file an Apache
[Individual Contributor License Agreement](https://www.apache.org/licenses/icla.txt).
---
Hi @tgroh can you please take a look?
This adds an integration test & data loading script for Jdbc IO
You'll note there are TODOs for verifying the contents of the rows in the tests - I will get to those, but wanted to get a first, useful test in so that we can continue working on the end to end integration test infrastructure.
This IT demonstrates:
* Having a separate load script for read tests
* Using TestPipelineOptions that are not test based, but rather data source based.
* Writing kubernetes scripts for instantiating an instance of the data source.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/ssisk/beam io-testing
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/beam/pull/1840.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1840
----
commit 21561ce64db15e2ba6b13186cc0092ba419b253e
Author: Stephen Sisk <si...@google.com>
Date: 2016-12-23T01:35:40Z
Add an example IT that relies on an external data store.
commit 83fcafcec6ababa664bbb2098fb440d86873ba20
Author: Stephen Sisk <si...@google.com>
Date: 2017-01-11T19:21:47Z
Update JdbcIOIT example so it uses real pipeline options & demonstrates a bug in JdbcIO
commit ba57d38e92fc6b79a192738f9f40925d2f3ae6f9
Author: Stephen Sisk <si...@google.com>
Date: 2017-01-12T19:48:37Z
JdbcIOIT now runs successfully. Fixed errors in Serialization & tableName
It is intentional that we use static inner classes rather than anonymous classes -
anonymous classes pull in their containing class, and thus the containing class
must be serializable.
There was also a small error in using an outdated table name in verification.
Also updated a few comments.
commit 82b35cf42387f39380145c77b9f15725fe9fff06
Author: Stephen Sisk <si...@google.com>
Date: 2016-12-23T01:35:40Z
Add an example IT that relies on an external data store.
commit b3456996ad2363de6d27181b03cdd08634a32aad
Author: Stephen Sisk <si...@google.com>
Date: 2017-01-11T19:21:47Z
Update JdbcIOIT example so it uses real pipeline options & demonstrates a bug in JdbcIO
commit 12963c6d657821b50491138b577f9ae0f7286eb6
Author: Stephen Sisk <si...@google.com>
Date: 2017-01-12T19:48:37Z
JdbcIOIT now runs successfully. Fixed errors in Serialization & tableName
It is intentional that we use static inner classes rather than anonymous classes -
anonymous classes pull in their containing class, and thus the containing class
must be serializable.
There was also a small error in using an outdated table name in verification.
Also updated a few comments.
commit e59949ae41ea60698490bf4c836382bcd1e05bcc
Author: Stephen Sisk <si...@google.com>
Date: 2017-01-13T01:27:17Z
Merge branch 'io-testing' of github.com:ssisk/beam into io-testing
commit 1e95f9253ad9cd6295f1377850f064edcd317032
Author: Stephen Sisk <si...@google.com>
Date: 2016-12-23T01:35:40Z
Add an example IT that relies on an external data store.
commit 760413fe328b0f7feed637e39bfe2990168b9cbf
Author: Stephen Sisk <si...@google.com>
Date: 2017-01-11T19:21:47Z
Update JdbcIOIT example so it uses real pipeline options & demonstrates a bug in JdbcIO
commit 8720379930b6fdc772d7e32facb55c9ded5b55ae
Author: Stephen Sisk <si...@google.com>
Date: 2017-01-12T19:48:37Z
JdbcIOIT now runs successfully. Fixed errors in Serialization & tableName
It is intentional that we use static inner classes rather than anonymous classes -
anonymous classes pull in their containing class, and thus the containing class
must be serializable.
There was also a small error in using an outdated table name in verification.
Also updated a few comments.
commit 8c364ff6cfbc1d34f6f2b283be6ee7beff61c773
Author: Stephen Sisk <si...@google.com>
Date: 2017-01-13T02:16:40Z
Switch JdbcTestOptions -> PostgresTestOptions & add comments explaining use.
commit 35ee94279b37e9d0d9c21dd6799200b07ca8817f
Author: Stephen Sisk <si...@google.com>
Date: 2017-01-18T00:16:16Z
Add example kubernetes & mesos scripts for setting up postgres instance.
commit 170881085f0b5f0098dd9087a04462d48e15a5f7
Author: Stephen Sisk <si...@google.com>
Date: 2016-12-23T01:35:40Z
Add an example IT that relies on an external data store.
commit 0f60e160d6c0ebf1b059f599813d551180b6c1cb
Author: Stephen Sisk <si...@google.com>
Date: 2017-01-11T19:21:47Z
Update JdbcIOIT example so it uses real pipeline options & demonstrates a bug in JdbcIO
commit 007cde2e75220c9c7d10da939eab5715f2be70a8
Author: Stephen Sisk <si...@google.com>
Date: 2017-01-12T19:48:37Z
JdbcIOIT now runs successfully. Fixed errors in Serialization & tableName
It is intentional that we use static inner classes rather than anonymous classes -
anonymous classes pull in their containing class, and thus the containing class
must be serializable.
There was also a small error in using an outdated table name in verification.
Also updated a few comments.
commit 5dfc7cc97f7739ce4847b0a18fc2a4208df7b0c1
Author: Stephen Sisk <si...@google.com>
Date: 2017-01-25T00:24:30Z
Add JdbcIOIT testWrite, uses data loading script
JdbcTestDataSet allows users to load data for the read tests,
and is shared with JdbcIOTest
commit 226b7c01abea254be48a68e5ebc060587e093878
Author: Stephen Sisk <si...@google.com>
Date: 2017-01-25T00:41:39Z
Commit cleanup: remove TODO/old mesos files
commit e8ae3e80faa0d6028538055e8682a3df9fc4c52e
Author: Stephen Sisk <si...@google.com>
Date: 2017-01-25T01:22:00Z
Fix bad copy paste - now correctly verifies writes
commit dc46b4660e5e5a65a90f8da35ac6a73d52edff13
Author: Stephen Sisk <si...@google.com>
Date: 2017-01-25T01:29:07Z
Remove outdated comment
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] beam pull request #1840: BEAM-1310 Add integration tests for JdbcIO
Posted by ssisk <gi...@git.apache.org>.
Github user ssisk closed the pull request at:
https://github.com/apache/beam/pull/1840
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---