You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@carbondata.apache.org by "anubhav tarar (JIRA)" <ji...@apache.org> on 2018/01/08 06:54:03 UTC
[jira] [Created] (CARBONDATA-2000) Unable to save a dataframe
result as carbondata streaming table
anubhav tarar created CARBONDATA-2000:
-----------------------------------------
Summary: Unable to save a dataframe result as carbondata streaming table
Key: CARBONDATA-2000
URL: https://issues.apache.org/jira/browse/CARBONDATA-2000
Project: CarbonData
Issue Type: Bug
Components: spark-integration
Affects Versions: 1.3.0
Environment: spark-2.1
Reporter: anubhav tarar
Assignee: anubhav tarar
Priority: Trivial
Fix For: 1.3.0
1.create carbonsession
import org.apache.spark.sql.SparkSession
import org.apache.spark.sql.CarbonSession._
val carbon = SparkSession.builder().config(sc.getConf) .getOrCreateCarbonSession("hdfs://localhost:54311/newCarbonStore","/tmp"
2.create a dataframe with carbonsession
import carbon.sqlContext.implicits._
carbon.sql("drop table if exists streamingtable");
val df =carbon.sparkContext.parallelize(1 to 5).toDF("colId")
3.register dataframe as carbon streaming table
df.write.format("carbondata").option("tableName","streamingTable").option("streaming","true").mode(SaveMode.Overwrite).save
4,desc formatted the table
carbon.sql("describe formatted streamingTable").show(100)
+--------------------+--------------------+--------------------+
| col_name| data_type| comment|
+--------------------+--------------------+--------------------+
|colid ...|int ...|MEASURE,null ...|
| ...| ...| ...|
|##Detailed Table ...| ...| ...|
|Database Name ...|default ...| ...|
|Table Name ...|streamingtable ...| ...|
|CARBON Store Path...|hdfs://localhost:...| ...|
|Comment ...| ...| ...|
|Table Block Size ...|1024 MB ...| ...|
|Table Data Size ...|316 ...| ...|
|Table Index Size ...|283 ...| ...|
|Last Update Time ...|1515393447642 ...| ...|
|SORT_SCOPE ...|LOCAL_SORT ...|LOCAL_SORT ...|
|Streaming ...|false ...| ...|
|SORT_SCOPE ...|LOCAL_SORT ...|LOCAL_SORT ...|
| ...| ...| ...|
|##Detailed Column...| ...| ...|
|ADAPTIVE ...| ...| ...|
|SORT_COLUMNS ...| ...| ...|
+--------------------+--------------------+--------------------+
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)