You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@beam.apache.org by "ASF GitHub Bot (JIRA)" <ji...@apache.org> on 2017/01/25 01:45:26 UTC
[jira] [Commented] (BEAM-1310) Add running integration tests for
JdbcIO
[ https://issues.apache.org/jira/browse/BEAM-1310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15837006#comment-15837006 ]
ASF GitHub Bot commented on BEAM-1310:
--------------------------------------
GitHub user ssisk opened a pull request:
https://github.com/apache/beam/pull/1840
BEAM-1310 Add integration tests for JdbcIO
Be sure to do all of the following to help us incorporate your contribution
quickly and easily:
- [X ] Make sure the PR title is formatted like:
`[BEAM-<Jira issue #>] Description of pull request`
- [X ] Make sure tests pass via `mvn clean verify`. (Even better, enable
Travis-CI on your fork and ensure the whole test matrix passes).
- [X ] Replace `<Jira issue #>` in the title with the actual Jira issue
number, if there is one.
- [X ] If this contribution is large, please file an Apache
[Individual Contributor License Agreement](https://www.apache.org/licenses/icla.txt).
---
Hi @tgroh can you please take a look?
This adds an integration test & data loading script for Jdbc IO
You'll note there are TODOs for verifying the contents of the rows in the tests - I will get to those, but wanted to get a first, useful test in so that we can continue working on the end to end integration test infrastructure.
This IT demonstrates:
* Having a separate load script for read tests
* Using TestPipelineOptions that are not test based, but rather data source based.
* Writing kubernetes scripts for instantiating an instance of the data source.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/ssisk/beam io-testing
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/beam/pull/1840.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1840
----
commit 21561ce64db15e2ba6b13186cc0092ba419b253e
Author: Stephen Sisk <si...@google.com>
Date: 2016-12-23T01:35:40Z
Add an example IT that relies on an external data store.
commit 83fcafcec6ababa664bbb2098fb440d86873ba20
Author: Stephen Sisk <si...@google.com>
Date: 2017-01-11T19:21:47Z
Update JdbcIOIT example so it uses real pipeline options & demonstrates a bug in JdbcIO
commit ba57d38e92fc6b79a192738f9f40925d2f3ae6f9
Author: Stephen Sisk <si...@google.com>
Date: 2017-01-12T19:48:37Z
JdbcIOIT now runs successfully. Fixed errors in Serialization & tableName
It is intentional that we use static inner classes rather than anonymous classes -
anonymous classes pull in their containing class, and thus the containing class
must be serializable.
There was also a small error in using an outdated table name in verification.
Also updated a few comments.
commit 82b35cf42387f39380145c77b9f15725fe9fff06
Author: Stephen Sisk <si...@google.com>
Date: 2016-12-23T01:35:40Z
Add an example IT that relies on an external data store.
commit b3456996ad2363de6d27181b03cdd08634a32aad
Author: Stephen Sisk <si...@google.com>
Date: 2017-01-11T19:21:47Z
Update JdbcIOIT example so it uses real pipeline options & demonstrates a bug in JdbcIO
commit 12963c6d657821b50491138b577f9ae0f7286eb6
Author: Stephen Sisk <si...@google.com>
Date: 2017-01-12T19:48:37Z
JdbcIOIT now runs successfully. Fixed errors in Serialization & tableName
It is intentional that we use static inner classes rather than anonymous classes -
anonymous classes pull in their containing class, and thus the containing class
must be serializable.
There was also a small error in using an outdated table name in verification.
Also updated a few comments.
commit e59949ae41ea60698490bf4c836382bcd1e05bcc
Author: Stephen Sisk <si...@google.com>
Date: 2017-01-13T01:27:17Z
Merge branch 'io-testing' of github.com:ssisk/beam into io-testing
commit 1e95f9253ad9cd6295f1377850f064edcd317032
Author: Stephen Sisk <si...@google.com>
Date: 2016-12-23T01:35:40Z
Add an example IT that relies on an external data store.
commit 760413fe328b0f7feed637e39bfe2990168b9cbf
Author: Stephen Sisk <si...@google.com>
Date: 2017-01-11T19:21:47Z
Update JdbcIOIT example so it uses real pipeline options & demonstrates a bug in JdbcIO
commit 8720379930b6fdc772d7e32facb55c9ded5b55ae
Author: Stephen Sisk <si...@google.com>
Date: 2017-01-12T19:48:37Z
JdbcIOIT now runs successfully. Fixed errors in Serialization & tableName
It is intentional that we use static inner classes rather than anonymous classes -
anonymous classes pull in their containing class, and thus the containing class
must be serializable.
There was also a small error in using an outdated table name in verification.
Also updated a few comments.
commit 8c364ff6cfbc1d34f6f2b283be6ee7beff61c773
Author: Stephen Sisk <si...@google.com>
Date: 2017-01-13T02:16:40Z
Switch JdbcTestOptions -> PostgresTestOptions & add comments explaining use.
commit 35ee94279b37e9d0d9c21dd6799200b07ca8817f
Author: Stephen Sisk <si...@google.com>
Date: 2017-01-18T00:16:16Z
Add example kubernetes & mesos scripts for setting up postgres instance.
commit 170881085f0b5f0098dd9087a04462d48e15a5f7
Author: Stephen Sisk <si...@google.com>
Date: 2016-12-23T01:35:40Z
Add an example IT that relies on an external data store.
commit 0f60e160d6c0ebf1b059f599813d551180b6c1cb
Author: Stephen Sisk <si...@google.com>
Date: 2017-01-11T19:21:47Z
Update JdbcIOIT example so it uses real pipeline options & demonstrates a bug in JdbcIO
commit 007cde2e75220c9c7d10da939eab5715f2be70a8
Author: Stephen Sisk <si...@google.com>
Date: 2017-01-12T19:48:37Z
JdbcIOIT now runs successfully. Fixed errors in Serialization & tableName
It is intentional that we use static inner classes rather than anonymous classes -
anonymous classes pull in their containing class, and thus the containing class
must be serializable.
There was also a small error in using an outdated table name in verification.
Also updated a few comments.
commit 5dfc7cc97f7739ce4847b0a18fc2a4208df7b0c1
Author: Stephen Sisk <si...@google.com>
Date: 2017-01-25T00:24:30Z
Add JdbcIOIT testWrite, uses data loading script
JdbcTestDataSet allows users to load data for the read tests,
and is shared with JdbcIOTest
commit 226b7c01abea254be48a68e5ebc060587e093878
Author: Stephen Sisk <si...@google.com>
Date: 2017-01-25T00:41:39Z
Commit cleanup: remove TODO/old mesos files
commit e8ae3e80faa0d6028538055e8682a3df9fc4c52e
Author: Stephen Sisk <si...@google.com>
Date: 2017-01-25T01:22:00Z
Fix bad copy paste - now correctly verifies writes
commit dc46b4660e5e5a65a90f8da35ac6a73d52edff13
Author: Stephen Sisk <si...@google.com>
Date: 2017-01-25T01:29:07Z
Remove outdated comment
----
> Add running integration tests for JdbcIO
> ----------------------------------------
>
> Key: BEAM-1310
> URL: https://issues.apache.org/jira/browse/BEAM-1310
> Project: Beam
> Issue Type: Bug
> Components: sdk-java-core
> Reporter: Stephen Sisk
> Assignee: Stephen Sisk
>
> Jdbc IO could use some integration tests! We'd like to have them run against a real list instance of postgres.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)