[jira] [Comment Edited] (QUARKS-19) Instructions for setting up Kafka and MQTT (Mosquitto) and running manual tests

["May Wone (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/QUARKS-19?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15187796#comment-15187796 ] 

May Wone edited comment on QUARKS-19 at 3/10/16 7:39 AM:
---------------------------------------------------------

h1. How to 1) install Kafka and 2) run Quarks Kafka manual tests on a Linux cluster machine

h2. Download Kafka
Go to http://kafka.apache.org/documentation.html\#quickstart

Download Kafka kafka\_2.11-0.9.0.0.tgz file to directory say: ~/kafka
*cd ~/kafka*
*tar –xvf kafka\_2.11-0.9.0.0.tgz**

*cd kafka\_2.11-0.9.0.0*

h2. Start zookeeper
Follow the Kafka page directions to start the zookeeper and Kafka servers, i.e., from directory kafka\_2.11-0.9.0.0:

*./bin/zookeeper-server-start.sh config/zookeeper.properties*

Note: If there are errors, see Trouble-shooting sections for Symptom 1 and Symptom 2. 

h2. Start Kafka server
*./bin/kafka-server-start.sh config/server.properties*

Note: If there are errors, see Trouble-shooting sectionsfor Symptom 3.

h2. Run Quarks manual tests
>From a Quarks development directory, i.e. ...\git\quarks (which contains file build.xml):

*ant -Dtest.base.pattern='\*\*/\*TestManual.java' test*

Look for these results:
{code}
test.run:
    [mkdir] Created dir: /homes/hny7/maywone/git/quarks/connectors/kafka/unittests
    [mkdir] Created dir: /homes/hny7/maywone/git/quarks/connectors/kafka/unittests/testrun1997679054
[jacoco:coverage] Enhancing junit with coverage
    [junit] Running quarks.test.connectors.kafka.KafkaStreamsTestManual
    [junit] Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 17.235 sec

{code}

h2. Trouble-shooting

h3. Symptom 1

Starting the zookeeper results in:
{code}
-bash-4.2$ ./bin/zookeeper-server-start.sh config/zookeeper.properties
<?xml version="1.0" ?>
<verbosegc xmlns="http://www.ibm.com/j9/verbosegc" version="R28_20160106_1341_B284759_CMPRSS">
JVMJ9VM007E Command-line option unrecognised: -Xloggc:/homes/hny7/maywone/kafka2/kafka_2.11-0.9.0.0/bin/../logs/zookeeper-gc.log
</verbosegc>
Error: Could not create the Java Virtual Machine.
Error: A fatal exception has occurred. Program will exit.

{code}
h4. Work around 1

 This type of error may indicate your machine’s JVM does not support this option.

 Edit kafka-run-class.sh to delete any unsupported option(s). For example, replace

{code}
KAFKA_GC_LOG_OPTS="-Xloggc:$LOG_DIR/$GC_LOG_FILE_NAME -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintGCTimeStamps "
{code}
with 
{code}
KAFKA_GC_LOG_OPTS="-verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintGCTimeStamps "
{code}


h3. Symptom 2

Starting the zookeeper results in:

{code}
[2016-02-19 13:43:42,002] ERROR Severe unrecoverable error, exiting (org.apache.zookeeper.server.SyncRequestProcessor)
java.io.FileNotFoundException: /tmp/zookeeper/version-2/log.94 (Permission denied)
        at java.io.FileOutputStream.open(FileOutputStream.java:286)
        at java.io.FileOutputStream.<init>(FileOutputStream.java:226)
        at java.io.FileOutputStream.<init>(FileOutputStream.java:173)
        at org.apache.zookeeper.server.persistence.FileTxnLog.append(FileTxnLog.java:205)
        at org.apache.zookeeper.server.persistence.FileTxnSnapLog.append(FileTxnSnapLog.java:314)
        at org.apache.zookeeper.server.ZKDatabase.append(ZKDatabase.java:476)
        at org.apache

{code}

h4. Work-around 2

Modify zookeeper.properties to replace 
{code}
dataDir=/tmp/zookeeper 
{code}
with 
{code}
dataDir=~/tmp/zookeeper
{code}
Apparently, this zookeeper property value specifies an absolute path.

h3. Symptom 3

Starting the Kafka server results in:
{code}
[2016-02-19 13:51:57,161] FATAL Fatal error during KafkaServer startup. Prepare to shutdown (kafka.server.KafkaServer)
java.io.FileNotFoundException: /tmp/kafka-logs/.lock (Permission denied)
        at java.io.RandomAccessFile.open0(Native Method)
        at java.io.RandomAccessFile.open(RandomAccessFile.java:333)
        at java.io.RandomAccessFile.<init>(RandomAccessFile.java:257)
        at kafka.utils.FileLock.<init>(FileLock.scala:29)
        at kafka.log.LogManager$$anonfun$lockLogDirs$1.apply(LogManager.scala:96)
        at kafka.log.LogManager$$anonfun$lockLogDirs$1.apply(LogManager.scala:95)
        at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:245)
        at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:245)
        at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
        at scala.collection.mutable.WrappedArray.foreach(WrappedArray.scala:35)
        at scala.collection.TraversableLike$class.map(TraversableLike.scala:245)
        at scala.collection.AbstractTraversable.map(Traversable.scala:104)
        at kafka.log.LogManager.lockLogDirs(LogManager.scala:95)
        at kafka.log.LogManager.<init>(LogManager.scala:57)
        at kafka.server.KafkaServer.createLogManager(KafkaServer.scala:589)
        at kafka.server.KafkaServer.startup(KafkaServer.scala:171)
        at kafka.server.KafkaServerStartable.startup(KafkaServerStartable.scala:37)
        at kafka.Kafka$.main(Kafka.scala:67)
        at kafka.Kafka.main(Kafka.scala)

{code}

h4. Workaround 3

 Modify server.properties to replace 
{code}
log.dirs=/tmp/kafka-logs
{code}
with 
{code}
log.dirs=tmp/kafka-logs
{code}
 Apparently this server property value specifies a relative path.



was (Author: mnwone):
KAFKA
How to 1) Install Kafka and 2) Run Quarks Kafka manual tests on a Linux cluster machine

Go to http://kafka.apache.org/documentation.html\#quickstart

Here’s an outline of the steps from the quick start guide:

1.  Download Kafka kafka\_2.11-0.9.0.0.tgz file to directory say: ~/kafka

 *tar –xvf kafka\_2.11-0.9.0.0.tgz**

 *cd kafka\_2.11-0.9.0.0*

2.  My notes: On cluster machine, these additional modifications were needed, because my userid has permission issues accessing /tmp and the JVM does not support all the options specified in the kafka scripts. (Your machine may not need these changes.)

    2a.  kafka-run-class.sh – remove any unsupported JVM options (see Troubleshooting section \#1 for an example). If zookeeper (step 3a) starts ok, then this is not applicable.

    2b.  zookeeper.properties – replace *dataDir=/tmp/zookeeper* with *dataDir=~/tmp/zookeeper* (see Trouble-shooting section \#2).

    2c.  Server.properties - replace *log.dirs=/tmp/kafka-logs* with *log.dirs=tmp/kafka-logs* (see Trouble-shooting section \#3).

3.  Follow the Kafka page directions to start the zookeeper and Kafka servers, i.e., from directory kafka\_2.11-0.9.0.0:

    3a.  *./bin/zookeeper-server-start.sh config/zookeeper.properties*

    3b.  *./bin/kafka-server-start.sh config/server.properties*

4.  To manually execute the Quarks \*xxxTestManual.java tests (per text in build.xml), run:

*ant -Dtest.base.pattern='\*\*/\*TestManual.java' test*

Look for these results:

test.run:
 \[mkdir\] Created dir: /homes/hny7/maywone/git/quarks/connectors/kafka/unittests

 \[mkdir\] Created dir: /homes/hny7/maywone/git/quarks/connectors/kafka/unittests/testrun1997679054

 \[jacoco:coverage\] Enhancing junit with coverage

 \[junit\] Running quarks.test.connectors.kafka.KafkaStreamsTestManual

 \[junit\] Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 17.235 sec

Trouble-shooting

Symptom \#1:

Starting the zookeeper results in:

 -bash-4.2$ ./bin/zookeeper-server-start.sh config/zookeeper.properties

 &lt;?xml version="1.0" ?&gt;

 &lt;verbosegc xmlns="http://www.ibm.com/j9/verbosegc" version="R28\_20160106\_1341\_B284759\_CMPRSS"&gt;

 JVMJ9VM007E Command-line option unrecognised: -Xloggc:/homes/hny7/maywone/kafka2/kafka\_2.11-0.9.0.0/bin/../logs/zookeeper-gc.log

 &lt;/verbosegc&gt;

 Error: Could not create the Java Virtual Machine.

 Error: A fatal exception has occurred. Program will exit.

Work around:

 This type of error may indicate your machine’s JVM does not support this option.

 Edit kafka-run-class.sh to delete any unsupported option(s). For example, replace

 KAFKA\_GC\_LOG\_OPTS="-Xloggc:$LOG\_DIR/$GC\_LOG\_FILE\_NAME -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintGCTimeStamps "

 with KAFKA\_GC\_LOG\_OPTS="-verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintGCTimeStamps "

-----------------------------------------------------------------------------------------

Symptom \#2

Starting the zookeeper results in:

 \[2016-02-19 13:43:42,002\] ERROR Severe unrecoverable error, exiting (org.apache.zookeeper.server.SyncRequestProcessor)

 java.io.FileNotFoundException: /tmp/zookeeper/version-2/log.94 (Permission denied)

 at java.io.FileOutputStream.open(FileOutputStream.java:286)

 at java.io.FileOutputStream.&lt;init&gt;(FileOutputStream.java:226)

 at java.io.FileOutputStream.&lt;init&gt;(FileOutputStream.java:173)

 at org.apache.zookeeper.server.persistence.FileTxnLog.append(FileTxnLog.java:205)

 at org.apache.zookeeper.server.persistence.FileTxnSnapLog.append(FileTxnSnapLog.java:314)

 at org.apache.zookeeper.server.ZKDatabase.append(ZKDatabase.java:476)

 at org.apache.zookeeper.server.SyncRequestProcessor.run(SyncRequestProcessor.java:140)

 &lt;/verbosegc&gt;

Work-around:

Modify zookeeper.properties to replace *dataDir=/tmp/zookeeper* with *dataDir=~/tmp/zookeeper*

Apparently, this zookeeper property value specifies an absolute path.

-----------------------------------------------------------------------------------------
Symptom \#3:

Starting the Kafka server results in:

 \[2016-02-19 13:51:57,161\] FATAL Fatal error during KafkaServer startup. Prepare to shutdown (kafka.server.KafkaServer)

 java.io.FileNotFoundException: /tmp/kafka-logs/.lock (Permission denied)

 at java.io.RandomAccessFile.open0(Native Method)

 at java.io.RandomAccessFile.open(RandomAccessFile.java:333)

 at java.io.RandomAccessFile.&lt;init&gt;(RandomAccessFile.java:257)

 at kafka.utils.FileLock.&lt;init&gt;(FileLock.scala:29)

 at kafka.log.LogManager$$anonfun$lockLogDirs$1.apply(LogManager.scala:96)

 at kafka.log.LogManager$$anonfun$lockLogDirs$1.apply(LogManager.scala:95)

 at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:245)

 at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:245)

 at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)

 at scala.collection.mutable.WrappedArray.foreach(WrappedArray.scala:35)

 at scala.collection.TraversableLike$class.map(TraversableLike.scala:245)

 at scala.collection.AbstractTraversable.map(Traversable.scala:104)

 at kafka.log.LogManager.lockLogDirs(LogManager.scala:95)

 at kafka.log.LogManager.&lt;init&gt;(LogManager.scala:57)

 at kafka.server.KafkaServer.createLogManager(KafkaServer.scala:589)

 at kafka.server.KafkaServer.startup(KafkaServer.scala:171)

 at kafka.server.KafkaServerStartable.startup(KafkaServerStartable.scala:37)

 at kafka.Kafka$.main(Kafka.scala:67)

 at kafka.Kafka.main(Kafka.scala)

Workaround:

 Modify server.properties to replace *log.dirs=/tmp/kafka-l*ogs with *log.dirs=tmp/kafka-logs*

 Apparently this server property value specifies a relative path.


> Instructions for setting up Kafka and MQTT (Mosquitto) and running manual tests
> -------------------------------------------------------------------------------
>
>                 Key: QUARKS-19
>                 URL: https://issues.apache.org/jira/browse/QUARKS-19
>             Project: Quarks
>          Issue Type: Improvement
>          Components: Documentation
>            Reporter: May Wone
>            Assignee: May Wone
>            Priority: Minor
>
> Will attach instructions for setting up Kafka and Mosquitto (MQTT) and running the Quarks manual tests. 
> I'd welcome feedback on the contents.
> Also, I'm thinking about where to put this information - my initial take is on a 'Quarks wiki'.   Comments? 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)