You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@edgent.apache.org by dl...@apache.org on 2016/12/22 01:35:16 UTC
[1/2] incubator-edgent git commit: [Edgent-325] improve jdbc, kafka,
and java7 test doc
Repository: incubator-edgent
Updated Branches:
refs/heads/master 99b7c8e21 -> be0d4b199
[Edgent-325] improve jdbc, kafka, and java7 test doc
Project: http://git-wip-us.apache.org/repos/asf/incubator-edgent/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-edgent/commit/0d588ae6
Tree: http://git-wip-us.apache.org/repos/asf/incubator-edgent/tree/0d588ae6
Diff: http://git-wip-us.apache.org/repos/asf/incubator-edgent/diff/0d588ae6
Branch: refs/heads/master
Commit: 0d588ae6d509540f414dac96a4a2785800f1c3ba
Parents: 99b7c8e
Author: Dale LaBossiere <dl...@us.ibm.com>
Authored: Wed Dec 21 17:06:33 2016 -0500
Committer: Dale LaBossiere <dl...@us.ibm.com>
Committed: Wed Dec 21 17:06:33 2016 -0500
----------------------------------------------------------------------
DEVELOPMENT.md | 65 ++++++++++++++++++--
.../test/connectors/jdbc/JdbcStreamsTest.java | 10 ++-
.../kafka/KafkaStreamsTestManual.java | 43 +++++++++++++
scripts/connectors/kafka/README | 4 +-
scripts/connectors/kafka/README-kafka | 25 ++++++++
5 files changed, 137 insertions(+), 10 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-edgent/blob/0d588ae6/DEVELOPMENT.md
----------------------------------------------------------------------
diff --git a/DEVELOPMENT.md b/DEVELOPMENT.md
index 00be84b..f64a0ed 100644
--- a/DEVELOPMENT.md
+++ b/DEVELOPMENT.md
@@ -104,8 +104,7 @@ The build process has been tested on Linux and macOS.
To build on Windows probably needs some changes, please get involved and contribute them!
-
-**Continuous Integration**
+### Continuous Integration
When a pull request is opened on the GitHub mirror site, the Travis CI service runs a full build.
@@ -160,6 +159,57 @@ Running the `reports` target produces two reports:
* `builds/distributions/reports/tests/index.html` - JUnit test report
* `builds/distributions/reports/coverage/index.html` - Code coverage report.
+### Testing the Kafka Connector
+
+The kafka connector tests aren't run by default as the connector must
+connect to a running Kafka/Zookeeper config.
+
+There are apparently ways to embedd Kafka and Zookeeper for testing purposes but
+we're not there yet. Contributions welcome!
+
+Setting up the servers is easy.
+Follow the steps in the [KafkaStreamsTestManual](connectors/kafka/src/test/java/org/apache/edgent/test/connectors/kafka/KafkaStreamsTestManual.java) javadoc.
+
+Once kafka/zookeeper are running you can run the tests and samples:
+```sh
+#### run the kafka tests
+./gradlew connectors:kafka:test --tests '*.*Manual'
+
+#### run the sample
+cd java8/scripts/connectors/kafka
+cat README
+./runkafkasample.sh sub
+./runkafkasample.sh pub
+```
+
+### Testing the JDBC Connector
+
+The JDBC connector tests are written to run against Apache Derby
+as the backing dbms and the derby jar needs to be on the classpath.
+The tests are skipped if derby can't be loaded.
+The test harness adds $DERBY_HOME/db/lib/derby.jar to the classpath.
+See [JdbcStreamsTest](connectors/jdbc/src/test/java/org/apache/edgent/test/connectors/jdbc/JdbcStreamsTest.java)
+for more info but the following should suffice:
+
+```sh
+export DERBY_HOME=$JAVA_HOME/db
+
+#### if JAVA_HOME isn't set - e.g., on OSX...
+export DERBY_HOME=`/usr/libexec/java_home`/db
+```
+
+Once DERBY_HOME is set the tests and samples can be run as follows
+```sh
+#### run the jdbc tests
+./gradlew connectors:jdbc:test
+
+#### run the sample
+cd java8/scripts/connectors/jdbc
+cat README
+./runjdbcsample.sh writer
+./runjdbcsample.sh reader
+```
+
### Testing Edgent with Java7
All of the standard build system _tasks_ above must be run with
@@ -173,14 +223,17 @@ See [JAVA_SUPPORT](JAVA_SUPPORT.md) for information about what
Edgent features are supported in the different environments.
``` sh
- # run with JAVA_HOME set for Java8
+ # run with JAVA_HOME/PATH set for Java8
$ ./gradlew test7Compile # compile the Edgent tests to operate in a Java7 environment
- # run with JAVA_HOME set for Java7
-$ ./gradlew test7Run # run the tests with a Java7 VM
+$ sh # muck with EVs for Java7 in a subshell
+ $ export JAVA_HOME=`/usr/libexec/java_home -v 1.7` # on OSX
+ $ export PATH=$JAVA_HOME/bin:$PATH
+ $ ./gradlew test7Run # run the tests with a Java7 VM
+ $ exit
# run with JAVA_HOME set for Java8
-$ ./gradlew test7Reports # generate the JUnit and coverage tests
+$ ./gradlew test7Reports # generate the JUnit and coverage reports
```
### Publish to Maven Repository
http://git-wip-us.apache.org/repos/asf/incubator-edgent/blob/0d588ae6/connectors/jdbc/src/test/java/org/apache/edgent/test/connectors/jdbc/JdbcStreamsTest.java
----------------------------------------------------------------------
diff --git a/connectors/jdbc/src/test/java/org/apache/edgent/test/connectors/jdbc/JdbcStreamsTest.java b/connectors/jdbc/src/test/java/org/apache/edgent/test/connectors/jdbc/JdbcStreamsTest.java
index 4b1e8ca..a15b9d9 100644
--- a/connectors/jdbc/src/test/java/org/apache/edgent/test/connectors/jdbc/JdbcStreamsTest.java
+++ b/connectors/jdbc/src/test/java/org/apache/edgent/test/connectors/jdbc/JdbcStreamsTest.java
@@ -52,9 +52,15 @@ import org.junit.Test;
* Manually install Derby for other JDKs if required.
* Arrange for the classpath to be configured by one of:
* <ul>
- * <li>set the DERBY_HOME environment variable. build.xml adds
- * $DERBY_HOME/lib/derby.jar to the classpath when running the tests.</li>
* <li>manually add derby.jar to the classpath</li>
+ * <li>set the DERBY_HOME environment variable. connectors/jdbc/build.gradle adds
+ * $DERBY_HOME/lib/derby.jar to the classpath when running the tests.
+ * e.g., try
+ * <ul>
+ * <li> export DERBY_HOME=$JAVA_HOME/db</li>
+ * <li> OSX: export DERBY_HOME=`/usr/libexec/java_home`/db
+ * </ul>
+ * </li>
* </ul>
* The tests are "skipped" if the dbms's jdbc driver can't be found.
*/
http://git-wip-us.apache.org/repos/asf/incubator-edgent/blob/0d588ae6/connectors/kafka/src/test/java/org/apache/edgent/test/connectors/kafka/KafkaStreamsTestManual.java
----------------------------------------------------------------------
diff --git a/connectors/kafka/src/test/java/org/apache/edgent/test/connectors/kafka/KafkaStreamsTestManual.java b/connectors/kafka/src/test/java/org/apache/edgent/test/connectors/kafka/KafkaStreamsTestManual.java
index 4328af8..0a9c3ab 100644
--- a/connectors/kafka/src/test/java/org/apache/edgent/test/connectors/kafka/KafkaStreamsTestManual.java
+++ b/connectors/kafka/src/test/java/org/apache/edgent/test/connectors/kafka/KafkaStreamsTestManual.java
@@ -37,6 +37,49 @@ import org.apache.edgent.topology.Topology;
import org.apache.edgent.topology.plumbing.PlumbingStreams;
import org.junit.Test;
+/**
+ * Test the Kafka connector.
+ * <p>
+ * The tests expect a Kafka/Zookeeper running on the local host at their
+ * default ports: 9092 and 2181 respectively.
+ * <p>
+ * The following system properties may be used to override that:
+ * <ul>
+ * <li>org.apache.edgent.test.connectors.kafka.bootstrap.servers=localhost:9092</li>
+ * <li>org.apache.edgent.test.connectors.kafka.zookeeper.connect=localhost:2181</li>
+ * </ul>
+ * <p>
+ * Setting up a Kafka/Zookeeper config on the default localhost ports is simple
+ * and well documented at https://kafka.apache.org/quickstart. This should do it:
+ * <p>
+ * After downloading kafka:
+ * <pre>{@code
+ * tar zxf ~/Downloads/kafka_2.11-0.10.1.0.tgz
+ * cd kafka_2.11-0.10.1.0/
+ *
+ * # start the servers (best in separate windows)
+ * bin/zookeeper-server-start.sh config/zookeeper.properties
+ * bin/kafka-server-start.sh config/server.properties
+ * }</pre>
+ *
+ * <p>
+ * The tests and sample require certain test topics. Create them:
+ * <pre>{@code
+ * # create our kafka test and sample topics
+ * bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic testTopic1
+ * bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic testTopic2
+ * bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic kafkaSampleTopic
+ * bin/kafka-topics.sh --list --zookeeper localhost:2181
+ *
+ * # quick verify
+ * bin/kafka-console-producer.sh --broker-list localhost:9092 --topic testTopic1
+ * hi
+ * there
+ * ^D
+ * bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic testTopic1 --from-beginning
+ * ... you should see the "hi" and "there" messages.
+ * }</pre>
+ */
public class KafkaStreamsTestManual extends ConnectorTestBase {
private static final int PUB_DELAY_MSEC = 4*1000;
private static final int SEC_TIMEOUT = 10;
http://git-wip-us.apache.org/repos/asf/incubator-edgent/blob/0d588ae6/scripts/connectors/kafka/README
----------------------------------------------------------------------
diff --git a/scripts/connectors/kafka/README b/scripts/connectors/kafka/README
index f038193..b1c6bec 100644
--- a/scripts/connectors/kafka/README
+++ b/scripts/connectors/kafka/README
@@ -6,8 +6,8 @@ By default the samples require the following kafka broker configuration:
- kafka topic "kafkaSampleTopic" exists
- no authentication
-See http://kafka.apache.org for the code and setup information for
-a Kafka broker.
+See README-kafka for information about setting up a kafka server
+and creating the topic.
The source code for the samples is in the <edgent-release>/samples directory.
http://git-wip-us.apache.org/repos/asf/incubator-edgent/blob/0d588ae6/scripts/connectors/kafka/README-kafka
----------------------------------------------------------------------
diff --git a/scripts/connectors/kafka/README-kafka b/scripts/connectors/kafka/README-kafka
new file mode 100644
index 0000000..e253963
--- /dev/null
+++ b/scripts/connectors/kafka/README-kafka
@@ -0,0 +1,25 @@
+Setting up a Kafka/Zookeeper config on the default localhost ports is simple
+and well documented at https://kafka.apache.org/quickstart. This should do it:
+
+After downloading kafka:
+
+tar zxf ~/Downloads/kafka_2.11-0.10.1.0.tgz
+cd kafka_2.11-0.10.1.0/
+
+# start the servers (best in separate windows)
+bin/zookeeper-server-start.sh config/zookeeper.properties
+bin/kafka-server-start.sh config/server.properties
+
+The sample requires a topic. Create it:
+
+# create our kafka sample topic
+bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic kafkaSampleTopic
+bin/kafka-topics.sh --list --zookeeper localhost:2181
+
+# quick verify
+bin/kafka-console-producer.sh --broker-list localhost:9092 --topic kafkaSampleTopic
+hi
+there
+^D
+bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic kafkaSampleTopic --from-beginning
+... you should see the "hi" and "there" messages.
[2/2] incubator-edgent git commit: tweak
Posted by dl...@apache.org.
tweak
Project: http://git-wip-us.apache.org/repos/asf/incubator-edgent/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-edgent/commit/be0d4b19
Tree: http://git-wip-us.apache.org/repos/asf/incubator-edgent/tree/be0d4b19
Diff: http://git-wip-us.apache.org/repos/asf/incubator-edgent/diff/be0d4b19
Branch: refs/heads/master
Commit: be0d4b19960201f4ecf7c58266f701ba517b62bd
Parents: 0d588ae
Author: Dale LaBossiere <dl...@us.ibm.com>
Authored: Wed Dec 21 17:10:22 2016 -0500
Committer: Dale LaBossiere <dl...@us.ibm.com>
Committed: Wed Dec 21 17:10:22 2016 -0500
----------------------------------------------------------------------
.../edgent/test/connectors/kafka/KafkaStreamsTestManual.java | 3 +--
1 file changed, 1 insertion(+), 2 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-edgent/blob/be0d4b19/connectors/kafka/src/test/java/org/apache/edgent/test/connectors/kafka/KafkaStreamsTestManual.java
----------------------------------------------------------------------
diff --git a/connectors/kafka/src/test/java/org/apache/edgent/test/connectors/kafka/KafkaStreamsTestManual.java b/connectors/kafka/src/test/java/org/apache/edgent/test/connectors/kafka/KafkaStreamsTestManual.java
index 0a9c3ab..74e5b63 100644
--- a/connectors/kafka/src/test/java/org/apache/edgent/test/connectors/kafka/KafkaStreamsTestManual.java
+++ b/connectors/kafka/src/test/java/org/apache/edgent/test/connectors/kafka/KafkaStreamsTestManual.java
@@ -63,12 +63,11 @@ import org.junit.Test;
* }</pre>
*
* <p>
- * The tests and sample require certain test topics. Create them:
+ * Create the test's topics:
* <pre>{@code
* # create our kafka test and sample topics
* bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic testTopic1
* bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic testTopic2
- * bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic kafkaSampleTopic
* bin/kafka-topics.sh --list --zookeeper localhost:2181
*
* # quick verify