You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@beam.apache.org by ke...@apache.org on 2017/03/09 00:13:36 UTC
[1/3] beam git commit: Jdbc k8 script: postgres data store only
accessible inside test project
Repository: beam
Updated Branches:
refs/heads/master f14b27409 -> c37e55080
Jdbc k8 script: postgres data store only accessible inside test project
Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/6e24a867
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/6e24a867
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/6e24a867
Branch: refs/heads/master
Commit: 6e24a867cca6771dc41a6429e26c57f2432af15c
Parents: 32e17d4
Author: Stephen Sisk <si...@google.com>
Authored: Mon Mar 6 16:01:26 2017 -0800
Committer: Kenneth Knowles <kl...@google.com>
Committed: Wed Mar 8 16:12:16 2017 -0800
----------------------------------------------------------------------
.../jdbc/src/test/resources/kubernetes/postgres-pod-no-vol.yml | 4 ++--
.../src/test/resources/kubernetes/postgres-service-public.yml | 3 ++-
2 files changed, 4 insertions(+), 3 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/beam/blob/6e24a867/sdks/java/io/jdbc/src/test/resources/kubernetes/postgres-pod-no-vol.yml
----------------------------------------------------------------------
diff --git a/sdks/java/io/jdbc/src/test/resources/kubernetes/postgres-pod-no-vol.yml b/sdks/java/io/jdbc/src/test/resources/kubernetes/postgres-pod-no-vol.yml
index 1e0c12a..17cad93 100644
--- a/sdks/java/io/jdbc/src/test/resources/kubernetes/postgres-pod-no-vol.yml
+++ b/sdks/java/io/jdbc/src/test/resources/kubernetes/postgres-pod-no-vol.yml
@@ -24,8 +24,8 @@ spec:
- name: postgres
image: postgres
env:
- - name: POSTGRES_PASS
- value: sroim3
+ - name: POSTGRES_PASSWORD
+ value: uuinkks
- name: PGDATA
value: /var/lib/postgresql/data/pgdata
ports:
http://git-wip-us.apache.org/repos/asf/beam/blob/6e24a867/sdks/java/io/jdbc/src/test/resources/kubernetes/postgres-service-public.yml
----------------------------------------------------------------------
diff --git a/sdks/java/io/jdbc/src/test/resources/kubernetes/postgres-service-public.yml b/sdks/java/io/jdbc/src/test/resources/kubernetes/postgres-service-public.yml
index 4da79f8..24d8eee 100644
--- a/sdks/java/io/jdbc/src/test/resources/kubernetes/postgres-service-public.yml
+++ b/sdks/java/io/jdbc/src/test/resources/kubernetes/postgres-service-public.yml
@@ -22,6 +22,7 @@ metadata:
spec:
ports:
- port: 5432
+ nodePort: 31234
selector:
name: postgres-no-pv
- type: LoadBalancer
+ type: NodePort
[2/3] beam git commit: Jdbc k8 & data loading: add teardown and
update names/docs
Posted by ke...@apache.org.
Jdbc k8 & data loading: add teardown and update names/docs
Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/32e17d4d
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/32e17d4d
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/32e17d4d
Branch: refs/heads/master
Commit: 32e17d4d6a84e349cc55732371245e8ad126a747
Parents: 5d120bd
Author: Stephen Sisk <si...@google.com>
Authored: Mon Mar 6 15:59:31 2017 -0800
Committer: Kenneth Knowles <kl...@google.com>
Committed: Wed Mar 8 16:12:16 2017 -0800
----------------------------------------------------------------------
sdks/java/io/jdbc/src/test/README.md | 32 ++++++++++++++++++++
.../beam/sdk/io/jdbc/JdbcTestDataSet.java | 3 +-
.../kubernetes/setup-postgres-service.sh | 20 ------------
.../jdbc/src/test/resources/kubernetes/setup.sh | 20 ++++++++++++
.../src/test/resources/kubernetes/teardown.sh | 20 ++++++++++++
5 files changed, 74 insertions(+), 21 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/beam/blob/32e17d4d/sdks/java/io/jdbc/src/test/README.md
----------------------------------------------------------------------
diff --git a/sdks/java/io/jdbc/src/test/README.md b/sdks/java/io/jdbc/src/test/README.md
new file mode 100644
index 0000000..5a7ac99
--- /dev/null
+++ b/sdks/java/io/jdbc/src/test/README.md
@@ -0,0 +1,32 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+-->
+
+These are instructions for maintaining postgres as needed for Integration Tests (JdbcIOIT).
+
+You can always ignore these instructions if you have your own postgres cluster to test against.
+
+Setting up Postgres
+-------------------
+1. Setup kubectl so it is configured to work with your kubernetes cluster
+1. Run the postgres setup script
+ src/test/resources/kubernetes/setup.sh
+1. Do the data loading - create the data store instance by following the instructions in JdbcTestDataSet
+
+... and your postgres instances are set up!
+
http://git-wip-us.apache.org/repos/asf/beam/blob/32e17d4d/sdks/java/io/jdbc/src/test/java/org/apache/beam/sdk/io/jdbc/JdbcTestDataSet.java
----------------------------------------------------------------------
diff --git a/sdks/java/io/jdbc/src/test/java/org/apache/beam/sdk/io/jdbc/JdbcTestDataSet.java b/sdks/java/io/jdbc/src/test/java/org/apache/beam/sdk/io/jdbc/JdbcTestDataSet.java
index 11cc2be..2a5d6f4 100644
--- a/sdks/java/io/jdbc/src/test/java/org/apache/beam/sdk/io/jdbc/JdbcTestDataSet.java
+++ b/sdks/java/io/jdbc/src/test/java/org/apache/beam/sdk/io/jdbc/JdbcTestDataSet.java
@@ -41,8 +41,9 @@ public class JdbcTestDataSet {
* Use this to create the read tables before IT read tests.
*
* <p>To invoke this class, you can use this command line:
+ * (run from the jdbc root directory)
* mvn test-compile exec:java -Dexec.mainClass=org.apache.beam.sdk.io.jdbc.JdbcTestDataSet \
- * -Dexec.args="--postgresIp=1.1.1.1 --postgresUsername=postgres
+ * -Dexec.args="--postgresServerName=127.0.0.1 --postgresUsername=postgres \
* --postgresDatabaseName=myfancydb \
* --postgresPassword=yourpassword --postgresSsl=false" \
* -Dexec.classpathScope=test
http://git-wip-us.apache.org/repos/asf/beam/blob/32e17d4d/sdks/java/io/jdbc/src/test/resources/kubernetes/setup-postgres-service.sh
----------------------------------------------------------------------
diff --git a/sdks/java/io/jdbc/src/test/resources/kubernetes/setup-postgres-service.sh b/sdks/java/io/jdbc/src/test/resources/kubernetes/setup-postgres-service.sh
deleted file mode 100644
index eece773..0000000
--- a/sdks/java/io/jdbc/src/test/resources/kubernetes/setup-postgres-service.sh
+++ /dev/null
@@ -1,20 +0,0 @@
-#!/bin/bash
-#
-# Licensed to the Apache Software Foundation (ASF) under one or more
-# contributor license agreements. See the NOTICE file distributed with
-# this work for additional information regarding copyright ownership.
-# The ASF licenses this file to You under the Apache License, Version 2.0
-# (the "License"); you may not use this file except in compliance with
-# the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-
-kubectl.sh create -f postgres-pod-no-vol.yml
-kubectl.sh create -f postgres-service-public.yml
http://git-wip-us.apache.org/repos/asf/beam/blob/32e17d4d/sdks/java/io/jdbc/src/test/resources/kubernetes/setup.sh
----------------------------------------------------------------------
diff --git a/sdks/java/io/jdbc/src/test/resources/kubernetes/setup.sh b/sdks/java/io/jdbc/src/test/resources/kubernetes/setup.sh
new file mode 100644
index 0000000..7d9b2e2
--- /dev/null
+++ b/sdks/java/io/jdbc/src/test/resources/kubernetes/setup.sh
@@ -0,0 +1,20 @@
+#!/bin/bash
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+kubectl create -f postgres-pod-no-vol.yml
+kubectl create -f postgres-service-public.yml
http://git-wip-us.apache.org/repos/asf/beam/blob/32e17d4d/sdks/java/io/jdbc/src/test/resources/kubernetes/teardown.sh
----------------------------------------------------------------------
diff --git a/sdks/java/io/jdbc/src/test/resources/kubernetes/teardown.sh b/sdks/java/io/jdbc/src/test/resources/kubernetes/teardown.sh
new file mode 100644
index 0000000..451562e
--- /dev/null
+++ b/sdks/java/io/jdbc/src/test/resources/kubernetes/teardown.sh
@@ -0,0 +1,20 @@
+#!/bin/bash
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+kubectl delete service postgres-no-pv
+kubectl delete pod postgres-no-pv
[3/3] beam git commit: This closes #2174: updates to JdbcIO k8
scripts & data loading
Posted by ke...@apache.org.
This closes #2174: updates to JdbcIO k8 scripts & data loading
Jdbc k8 script: postgres data store only accessible inside test project
Jdbc k8 & data loading: add teardown and update names/docs
Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/c37e5508
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/c37e5508
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/c37e5508
Branch: refs/heads/master
Commit: c37e55080b2a783910aca87c7640663948e23893
Parents: f14b274 6e24a86
Author: Kenneth Knowles <kl...@google.com>
Authored: Wed Mar 8 16:12:41 2017 -0800
Committer: Kenneth Knowles <kl...@google.com>
Committed: Wed Mar 8 16:12:41 2017 -0800
----------------------------------------------------------------------
sdks/java/io/jdbc/src/test/README.md | 32 ++++++++++++++++++++
.../beam/sdk/io/jdbc/JdbcTestDataSet.java | 3 +-
.../kubernetes/postgres-pod-no-vol.yml | 4 +--
.../kubernetes/postgres-service-public.yml | 3 +-
.../kubernetes/setup-postgres-service.sh | 20 ------------
.../jdbc/src/test/resources/kubernetes/setup.sh | 20 ++++++++++++
.../src/test/resources/kubernetes/teardown.sh | 20 ++++++++++++
7 files changed, 78 insertions(+), 24 deletions(-)
----------------------------------------------------------------------