[jira] [Commented] (CARBONDATA-1990) Null values shown when the basic word count example is tried on carbon streaming table

["anubhav tarar (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/CARBONDATA-1990?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16396620#comment-16396620 ] 

anubhav tarar commented on CARBONDATA-1990:
-------------------------------------------

i am closing this bug as we should use text method instead of giving format as csv and we also will need to convert dataframe to string by appending commas

> Null values shown when the basic word count example is tried on carbon streaming table
> --------------------------------------------------------------------------------------
>
>                 Key: CARBONDATA-1990
>                 URL: https://issues.apache.org/jira/browse/CARBONDATA-1990
>             Project: CarbonData
>          Issue Type: Bug
>          Components: spark-integration
>    Affects Versions: 1.3.0
>         Environment: spark-2.1
>            Reporter: anubhav tarar
>            Priority: Major
>             Fix For: 1.4.0
>
>
> import org.apache.spark.sql.SparkSession
> import org.apache.spark.sql.CarbonSession._
> import org.apache.spark.sql.types._
> import org.apache.spark.sql.{AnalysisException, DataFrame, DataFrameWriter, Row, SaveMode}
> val carbon = SparkSession.builder().config(sc.getConf) .getOrCreateCarbonSession("hdfs://localhost:54311/newCarbonStore","/tmp")
>  carbon.sql("drop table if exists stream_table_csvSource")
>  carbon.sql("create table stream_table_csvSource(word string,count string) stored by 'carbondata' TBLPROPERTIES('streaming'='true')")
> import carbon.sqlContext.implicits._
> val userSchema = StructType(
> Array(StructField("name", StringType)))
> val lines = carbon.readStream
> .format("csv")
> .option("path", "file:///home/anubhav/Documents/csv")
> .option("sep",",")
> .schema(userSchema)
> .load()
> lines.printSchema
> val words = lines.as[String].flatMap(_.split("\n"))
> val wordCounts = words.groupBy("value").count()
> val qry = wordCounts.writeStream
>               .outputMode("complete")
>              .format("carbondata")
>              .option("checkpointLocation","file/home/anubhav/Downloads/checkpointlist155")
>             .option("dbName", "default")
>             .option("tableName", "stream_table_csvSource")
>             .start()
>           qry.awaitTermination()
> then i close my shell and select data from table there are null values shown  
> scala> carbon.sql("select * from stream_table_csvSource").show
> 18/01/05 14:14:34 ERROR CarbonProperties: Executor task launch worker-0 Configured value for property carbon.number.of.cores.while.loading is wrong. Falling back to the default value 2
> +-------+-----+
> |   word|count|
> +-------+-----+
> |anubhav| null|
> |geetika| null|
> | prince| null|
> +-------+-----+



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)