You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@carbondata.apache.org by vandana7 <gi...@git.apache.org> on 2018/04/18 11:29:37 UTC

[GitHub] carbondata pull request #2182: [CARBONDATA-2354] fixed streaming example

GitHub user vandana7 opened a pull request:

    https://github.com/apache/carbondata/pull/2182

    [CARBONDATA-2354] fixed streaming example

    

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/vandana7/incubator-carbondata streaming_example

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/carbondata/pull/2182.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #2182
    
----
commit 795b16e3e4bd9232cca54b758d1dbeeb20123d3e
Author: vandana <va...@...>
Date:   2018-04-18T11:27:17Z

    fixed streaming example

----


---

[GitHub] carbondata pull request #2182: [CARBONDATA-2354] fixed streaming example

Posted by jackylk <gi...@git.apache.org>.

Github user jackylk commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2182#discussion_r183084661
  
    --- Diff: docs/streaming-guide.md ---
    @@ -26,64 +26,64 @@ Start spark-shell in new terminal, type :paste, then copy and run the following
      import java.io.File
      import org.apache.spark.sql.{CarbonEnv, SparkSession}
      import org.apache.spark.sql.CarbonSession._
    - import org.apache.spark.sql.streaming.{ProcessingTime, StreamingQuery}
    - import org.apache.carbondata.core.util.path.CarbonStorePath
    - 
    - val warehouse = new File("./warehouse").getCanonicalPath
    - val metastore = new File("./metastore").getCanonicalPath
    - 
    - val spark = SparkSession
    -   .builder()
    -   .master("local")
    -   .appName("StreamExample")
    -   .config("spark.sql.warehouse.dir", warehouse)
    -   .getOrCreateCarbonSession(warehouse, metastore)
    -
    - spark.sparkContext.setLogLevel("ERROR")
    -
    - // drop table if exists previously
    - spark.sql(s"DROP TABLE IF EXISTS carbon_table")
    - // Create target carbon table and populate with initial data
    - spark.sql(
    -   s"""
    -      | CREATE TABLE carbon_table (
    -      | col1 INT,
    -      | col2 STRING
    -      | )
    -      | STORED BY 'carbondata'
    -      | TBLPROPERTIES('streaming'='true')""".stripMargin)
    -
    - val carbonTable = CarbonEnv.getCarbonTable(Some("default"), "carbon_table")(spark)
    - val tablePath = CarbonStorePath.getCarbonTablePath(carbonTable.getAbsoluteTableIdentifier)
    - 
    - // batch load
    - var qry: StreamingQuery = null
    - val readSocketDF = spark.readStream
    -   .format("socket")
    -   .option("host", "localhost")
    -   .option("port", 9099)
    -   .load()
    -
    - // Write data from socket stream to carbondata file
    - qry = readSocketDF.writeStream
    -   .format("carbondata")
    -   .trigger(ProcessingTime("5 seconds"))
    -   .option("checkpointLocation", tablePath.getStreamingCheckpointDir)
    -   .option("dbName", "default")
    -   .option("tableName", "carbon_table")
    -   .start()
    -
    - // start new thread to show data
    - new Thread() {
    -   override def run(): Unit = {
    -     do {
    -       spark.sql("select * from carbon_table").show(false)
    -       Thread.sleep(10000)
    -     } while (true)
    -   }
    - }.start()
    -
    - qry.awaitTermination()
    +  import org.apache.spark.sql.streaming.{ProcessingTime, StreamingQuery}
    --- End diff --
    
    indentation is not correct


---

[GitHub] carbondata issue #2182: [CARBONDATA-2354] fixed streaming example

Posted by CarbonDataQA <gi...@git.apache.org>.

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2182
  
    Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/5953/



---

[GitHub] carbondata issue #2182: [CARBONDATA-2354] fixed streaming example

Posted by chenliang613 <gi...@git.apache.org>.

Github user chenliang613 commented on the issue:

    https://github.com/apache/carbondata/pull/2182
  
    retest this please


---

[GitHub] carbondata issue #2182: [CARBONDATA-2354] fixed streaming example

Posted by vandana7 <gi...@git.apache.org>.

Github user vandana7 commented on the issue:

    https://github.com/apache/carbondata/pull/2182
  
    @jackylk please review 


---

[GitHub] carbondata issue #2182: [CARBONDATA-2354] fixed streaming example

Posted by CarbonDataQA <gi...@git.apache.org>.

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2182
  
    Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/4139/



---

[GitHub] carbondata issue #2182: [CARBONDATA-2354] fixed streaming example

Posted by CarbonDataQA <gi...@git.apache.org>.

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2182
  
    Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/4797/



---

[GitHub] carbondata issue #2182: [CARBONDATA-2354] fixed streaming example

Posted by CarbonDataQA <gi...@git.apache.org>.

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2182
  
    Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/5136/



---

[GitHub] carbondata issue #2182: [CARBONDATA-2354] fixed streaming example

Posted by CarbonDataQA <gi...@git.apache.org>.

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2182
  
    Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/3911/



---

[GitHub] carbondata issue #2182: [CARBONDATA-2354] fixed streaming example

Posted by CarbonDataQA <gi...@git.apache.org>.

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2182
  
    Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/3915/



---

[GitHub] carbondata issue #2182: [CARBONDATA-2354] fixed streaming example

Posted by CarbonDataQA <gi...@git.apache.org>.

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2182
  
    Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/3933/



---