You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@carbondata.apache.org by "Ravindra Pesala (JIRA)" <ji...@apache.org> on 2018/02/03 10:25:00 UTC
[jira] [Updated] (CARBONDATA-2000) Unable to save a dataframe
result as carbondata streaming table
[ https://issues.apache.org/jira/browse/CARBONDATA-2000?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ravindra Pesala updated CARBONDATA-2000:
----------------------------------------
Fix Version/s: (was: 1.3.0)
1.4.0
> Unable to save a dataframe result as carbondata streaming table
> ---------------------------------------------------------------
>
> Key: CARBONDATA-2000
> URL: https://issues.apache.org/jira/browse/CARBONDATA-2000
> Project: CarbonData
> Issue Type: Bug
> Components: spark-integration
> Affects Versions: 1.3.0
> Environment: spark-2.1
> Reporter: anubhav tarar
> Assignee: anubhav tarar
> Priority: Trivial
> Fix For: 1.4.0
>
>
> 1.create carbonsession
> import org.apache.spark.sql.SparkSession
> import org.apache.spark.sql.CarbonSession._
> val carbon = SparkSession.builder().config(sc.getConf) .getOrCreateCarbonSession("hdfs://localhost:54311/newCarbonStore","/tmp"
> 2.create a dataframe with carbonsession
> import carbon.sqlContext.implicits._
> carbon.sql("drop table if exists streamingtable");
> val df =carbon.sparkContext.parallelize(1 to 5).toDF("colId")
> 3.register dataframe as carbon streaming table
> df.write.format("carbondata").option("tableName","streamingTable").option("streaming","true").mode(SaveMode.Overwrite).save
> 4,desc formatted the table
> carbon.sql("describe formatted streamingTable").show(100)
> +--------------------+--------------------+--------------------+
> | col_name| data_type| comment|
> +--------------------+--------------------+--------------------+
> |colid ...|int ...|MEASURE,null ...|
> | ...| ...| ...|
> |##Detailed Table ...| ...| ...|
> |Database Name ...|default ...| ...|
> |Table Name ...|streamingtable ...| ...|
> |CARBON Store Path...|hdfs://localhost:...| ...|
> |Comment ...| ...| ...|
> |Table Block Size ...|1024 MB ...| ...|
> |Table Data Size ...|316 ...| ...|
> |Table Index Size ...|283 ...| ...|
> |Last Update Time ...|1515393447642 ...| ...|
> |SORT_SCOPE ...|LOCAL_SORT ...|LOCAL_SORT ...|
> |Streaming ...|false ...| ...|
> |SORT_SCOPE ...|LOCAL_SORT ...|LOCAL_SORT ...|
> | ...| ...| ...|
> |##Detailed Column...| ...| ...|
> |ADAPTIVE ...| ...| ...|
> |SORT_COLUMNS ...| ...| ...|
> +--------------------+--------------------+--------------------+
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)