You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@carbondata.apache.org by haomarch <ma...@126.com> on 2020/05/12 08:52:23 UTC

[Disscussion] Support GloabalSort in the CDC Flow

If there is GloabalSort table in the CDC Flow. The following exception will
be throwed:

Exception in thread "main" java.lang.RuntimeException: column: id specified
in sort columns does not exist in schema
	at
org.apache.carbondata.sdk.file.CarbonWriterBuilder.buildTableSchema(CarbonWriterBuilder.java:828)
	at
org.apache.carbondata.sdk.file.CarbonWriterBuilder.buildCarbonTable(CarbonWriterBuilder.java:794)
	at
org.apache.carbondata.sdk.file.CarbonWriterBuilder.buildLoadModel(CarbonWriterBuilder.java:720)
	at
org.apache.spark.sql.carbondata.execution.datasources.CarbonSparkDataSourceUtil$.prepareLoadModel(CarbonSparkDataSourceUtil.scala:281)
	at
org.apache.spark.sql.carbondata.execution.datasources.SparkCarbonFileFormat.prepareWrite(SparkCarbonFileFormat.scala:141)
	at
org.apache.spark.sql.execution.command.mutation.merge.CarbonMergeDataSetCommand.processIUD(CarbonMergeDataSetCommand.scala:269)
	at
org.apache.spark.sql.execution.command.mutation.merge.CarbonMergeDataSetCommand.processData(CarbonMergeDataSetCommand.scala:152)



--
Sent from: http://apache-carbondata-dev-mailing-list-archive.1130556.n5.nabble.com/

Re: [Disscussion] Support GloabalSort in the CDC Flow

Posted by haomarch <ma...@126.com>.

Shall we
use NO_SORT to write delta files?
while using Gloabal to write base files?

It will improve the update performance a lot.



--
Sent from: http://apache-carbondata-dev-mailing-list-archive.1130556.n5.nabble.com/

Re: [Disscussion] Support GloabalSort in the CDC Flow

Posted by David CaiQiang <da...@gmail.com>.

In my opinion, this is an issue if it can't work.

Better to change the topic title to use ‘question'/'issue' instead of
'discussion'.



-----
Best Regards
David Cai
--
Sent from: http://apache-carbondata-dev-mailing-list-archive.1130556.n5.nabble.com/