You are viewing a plain text version of this content. The canonical link for it is here.

Posted to reviews@bahir.apache.org by zubairnabi-intech <gi...@git.apache.org> on 2017/11/29 05:55:27 UTC

[GitHub] bahir pull request #55: [BAHIR-104] Multi-topic MQTT DStream in Python is no...

GitHub user zubairnabi-intech opened a pull request:

    https://github.com/apache/bahir/pull/55

    [BAHIR-104] Multi-topic MQTT DStream in Python is now a PairRDD.

    Fixing [BAHIR-104]. 
    List of topics passed in Python converted to a Java String array before passing it to Py4J. 
    Test added for this change.

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/intech-iiot/bahir mqtt-python-paired-stream

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/bahir/pull/55.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #55
    
----
commit 98d52c70db95e455b677a888aba3f5c369cab2a1
Author: Zubair Nabi <zu...@intechww.com>
Date:   2017-11-29T05:45:28Z

    [BAHIR-104] Multi-topic MQTT DStream in Python is now a PairRDD.

----


---

[GitHub] bahir issue #55: [BAHIR-104] Multi-topic MQTT DStream in Python is now a Pai...

Posted by ckadner <gi...@git.apache.org>.

Github user ckadner commented on the issue:

    https://github.com/apache/bahir/pull/55
  
    @emlaver -- could you take a look at the build failure? Thanks
    
    ```
    CloudantChangesDFSuite:
    - load and save data from Cloudant database *** FAILED ***
      0 did not equal 1967 (CloudantChangesDFSuite.scala:51)
    ```
    
    ... and with a bit more log context:
    
    ```
    [INFO] --- scalatest-maven-plugin:1.0:test (test) @ spark-sql-cloudant_2.11 ---
    Discovery starting.
    
    Sql-cloudant tests that require Cloudant databases have been enabled by
    the environment variables CLOUDANT_USER and CLOUDANT_PASSWORD.
            
    Discovery completed in 187 milliseconds.
    Run starting. Expected test count is: 22
    CloudantOptionSuite:
    - invalid api receiver option throws an error message
    - empty username option throws an error message
    - empty password option throws an error message
    - empty databaseName throws an error message
    ClientSparkFunSuite:
    CloudantChangesDFSuite:
    - load and save data from Cloudant database *** FAILED ***
      0 did not equal 1967 (CloudantChangesDFSuite.scala:51)
    - load and count data from Cloudant search index
    - load data and verify deleted doc is not in results
    - load data and count rows in filtered dataframe
    - save filtered dataframe to database
    - save dataframe to database using createDBOnSave=true option
    - load and count data from view
    - load data from view with MapReduce function
    - load data and verify total count of selector, filter, and view option
    CloudantSparkSQLSuite:
    - verify results from temp view of database n_airportcodemapping
    - verify results from temp view of index in n_flight
    CloudantAllDocsDFSuite:
    - load and save data from Cloudant database
    - load and count data from Cloudant search index
    - load data and count rows in filtered dataframe
    - save filtered dataframe to database
    - save dataframe to database using createDBOnSave=true option
    - load and count data from view
    - load data from view with MapReduce function
    Run completed in 3 minutes, 8 seconds.
    Total number of tests run: 22
    Suites: completed 6, aborted 0
    Tests: succeeded 21, failed 1, canceled 0, ignored 0, pending 0
    *** 1 TEST FAILED ***
    [INFO] ------------------------------------------------------------------------
    [INFO] Reactor Summary:
    [INFO] 
    [INFO] Apache Bahir - Parent POM .......................... SUCCESS [  4.355 s]
    [INFO] Apache Bahir - Spark SQL Cloudant DataSource ....... FAILURE [06:50 min]
    [INFO] Apache Bahir - Spark Streaming Akka ................ SKIPPED
    [INFO] Apache Bahir - Spark SQL Streaming Akka ............ SKIPPED
    [INFO] Apache Bahir - Spark Streaming MQTT ................ SKIPPED
    [INFO] Apache Bahir - Spark SQL Streaming MQTT ............ SKIPPED
    [INFO] Apache Bahir - Spark Streaming Twitter ............. SKIPPED
    [INFO] Apache Bahir - Spark Streaming ZeroMQ .............. SKIPPED
    [INFO] Apache Bahir - Spark Streaming Google PubSub ....... SKIPPED
    [INFO] Apache Bahir - Spark Extensions Distribution ....... SKIPPED
    [INFO] ------------------------------------------------------------------------
    [INFO] BUILD FAILURE
    [INFO] ------------------------------------------------------------------------
    [INFO] Total time: 06:55 min
    [INFO] Finished at: 2017-12-05T14:33:29-08:00
    [INFO] Final Memory: 67M/2606M
    [INFO] ------------------------------------------------------------------------
    [ERROR] Failed to execute goal org.scalatest:scalatest-maven-plugin:1.0:test (test) on project spark-sql-cloudant_2.11: There are test failures -> [Help 1]
    [ERROR] 
    [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
    [ERROR] Re-run Maven using the -X switch to enable full debug logging.
    [ERROR] 
    [ERROR] For more information about the errors and possible solutions, please read the following articles:
    [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
    [ERROR] 
    [ERROR] After correcting the problems, you can resume the build with the command
    [ERROR]   mvn <goals> -rf :spark-sql-cloudant_2.11
    ```


---

[GitHub] bahir issue #55: [BAHIR-104] Multi-topic MQTT DStream in Python is now a Pai...

Posted by ApacheBahir <gi...@git.apache.org>.

Github user ApacheBahir commented on the issue:

    https://github.com/apache/bahir/pull/55
  
    
    Refer to this link for build results (access rights to CI server needed): 
    http://169.45.79.58:8080/job/bahir_spark_pr_builder/106/



---

[GitHub] bahir issue #55: [BAHIR-104] Multi-topic MQTT DStream in Python is now a Pai...

Posted by ckadner <gi...@git.apache.org>.

Github user ckadner commented on the issue:

    https://github.com/apache/bahir/pull/55
  
    We still have failing tests in `sql-cloudant` but all other modules were built and tested successfully.
    
    ```
    [INFO] ------------------------------------------------------------------------
    [INFO] Reactor Summary:
    [INFO] 
    [INFO] Apache Bahir - Parent POM .......................... SUCCESS [  4.356 s]
    [INFO] Apache Bahir - Spark SQL Cloudant DataSource ....... FAILURE [03:13 min]
    [INFO] Apache Bahir - Spark Streaming Akka ................ SUCCESS [ 27.906 s]
    [INFO] Apache Bahir - Spark SQL Streaming Akka ............ SUCCESS [03:49 min]
    [INFO] Apache Bahir - Spark Streaming MQTT ................ SUCCESS [01:53 min]
    [INFO] Apache Bahir - Spark SQL Streaming MQTT ............ SUCCESS [02:05 min]
    [INFO] Apache Bahir - Spark Streaming Twitter ............. SUCCESS [ 25.999 s]
    [INFO] Apache Bahir - Spark Streaming ZeroMQ .............. SUCCESS [ 18.989 s]
    [INFO] Apache Bahir - Spark Streaming Google PubSub ....... SUCCESS [ 53.220 s]
    [INFO] Apache Bahir - Spark Extensions Distribution ....... SUCCESS [  3.500 s]
    [INFO] ------------------------------------------------------------------------
    [INFO] BUILD FAILURE
    [INFO] ------------------------------------------------------------------------
    [INFO] Total time: 13:16 min
    [INFO] Finished at: 2017-12-07T17:02:57-08:00
    [INFO] Final Memory: 183M/6960M
    [INFO] ------------------------------------------------------------------------
    ```


---

[GitHub] bahir issue #55: [BAHIR-104] Multi-topic MQTT DStream in Python is now a Pai...

Posted by zubairnabi-intech <gi...@git.apache.org>.

Github user zubairnabi-intech commented on the issue:

    https://github.com/apache/bahir/pull/55
  
    The build failure is actually unrelated to this change. A Cloudant test in Scala is failing.


---

[GitHub] bahir issue #55: [BAHIR-104] Multi-topic MQTT DStream in Python is now a Pai...

Posted by ckadner <gi...@git.apache.org>.

Github user ckadner commented on the issue:

    https://github.com/apache/bahir/pull/55
  
    Any further comments?
    
    @anntinutj - you commented on [BAHIR-104](https://issues.apache.org/jira/browse/BAHIR-104)
    @fbeneventi - I saw you worked on this before (https://github.com/fbeneventi/bahir/commit/3755ecc)


---

[GitHub] bahir issue #55: [BAHIR-104] Multi-topic MQTT DStream in Python is now a Pai...

Posted by ApacheBahir <gi...@git.apache.org>.

Github user ApacheBahir commented on the issue:

    https://github.com/apache/bahir/pull/55
  
    Build failed, see build log for details
     



---

[GitHub] bahir issue #55: [BAHIR-104] Multi-topic MQTT DStream in Python is now a Pai...

Posted by ckadner <gi...@git.apache.org>.

Github user ckadner commented on the issue:

    https://github.com/apache/bahir/pull/55
  
    **LGTM**
    
    I ran the (new) Python tests manually and they completed successfully.
    
    **Before:**
    ```
    [bahir] (master *=)$ streaming-mqtt/python-tests/run-python-tests.sh
    ----------------------------------------------------------------------
    Ran 1 test in 22.871s
    OK
    ```
    
    
    
    **After:**
    ```
    [bahir] (pr-55_BAHIR-104_python_pairRDD *=)$ streaming-mqtt/python-tests/run-python-tests.sh
    ----------------------------------------------------------------------
    Ran 2 tests in 27.593s
    OK
    
    ```


---

[GitHub] bahir issue #55: [BAHIR-104] Multi-topic MQTT DStream in Python is now a Pai...

Posted by ApacheBahir <gi...@git.apache.org>.

Github user ApacheBahir commented on the issue:

    https://github.com/apache/bahir/pull/55
  
    Can one of the admins verify this patch?


---

[GitHub] bahir issue #55: [BAHIR-104] Multi-topic MQTT DStream in Python is now a Pai...

Posted by ckadner <gi...@git.apache.org>.

Github user ckadner commented on the issue:

    https://github.com/apache/bahir/pull/55
  
    The problem is that one failing test in `sql-cloudant` causes all remaining tests to be skipped. Which means this PR can't be tested.
    
    @emlaver -- If you had an actual fix, could you create a PR for which we can quickly merge? Then this PR could be rebased on that latest code.
    
    If that is not possible, I will have to look into changing the Jenkins build to run the tests separately from (and after) the compile/package build and tell Maven to keep running after test failures.


---

[GitHub] bahir issue #55: [BAHIR-104] Multi-topic MQTT DStream in Python is now a Pai...

Posted by emlaver <gi...@git.apache.org>.

Github user emlaver commented on the issue:

    https://github.com/apache/bahir/pull/55
  
    @ckadner Hi yes I will look into fixing this sporadically failing test. This shouldn't hold up this PR, right?


---

[GitHub] bahir issue #55: [BAHIR-104] Multi-topic MQTT DStream in Python is now a Pai...

Posted by ckadner <gi...@git.apache.org>.

Github user ckadner commented on the issue:

    https://github.com/apache/bahir/pull/55
  
    Actually, our Maven build does not kick of the Python tests, so we may have to test this PR "manually" for the time being.
    
    I will start on making the build changes independently.


---

[GitHub] bahir issue #55: [BAHIR-104] Multi-topic MQTT DStream in Python is now a Pai...

Posted by ckadner <gi...@git.apache.org>.

Github user ckadner commented on the issue:

    https://github.com/apache/bahir/pull/55
  
    retest this please


---

[GitHub] bahir issue #55: [BAHIR-104] Multi-topic MQTT DStream in Python is now a Pai...

Posted by ApacheBahir <gi...@git.apache.org>.

Github user ApacheBahir commented on the issue:

    https://github.com/apache/bahir/pull/55
  
    Build failed, see build log for details
     



---