You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@carbondata.apache.org by GitBox <gi...@apache.org> on 2021/08/18 17:05:26 UTC
[GitHub] [carbondata] Lior-AI opened a new issue #4206: Cannot create table with partitions in Spark in EMR
Lior-AI opened a new issue #4206:
URL: https://github.com/apache/carbondata/issues/4206
I am running spark in EMR
> Release label:emr-5.24.1
Hadoop distribution:Amazon 2.8.5
Applications:
Hive 2.3.4, Pig 0.17.0, Hue 4.4.0, Flink 1.8.0, Spark 2.4.2, Presto 0.219, JupyterHub 0.9.6
Jar complied with:
>apache-carbondata:2.2.0
spark:2.4.5
hadoop:2.8.3
When trying to create a table like this:
```
CREATE TABLE IF NOT EXISTS will_not_work(
timestamp string,
name string
)
PARTITIONED BY (dt string, hr string)
STORED AS carbondata
LOCATION 's3a://my-bucket/CarbonDataTests/will_not_work
```
I get the following error:
```
org.apache.carbondata.common.exceptions.sql.MalformedCarbonCommandException: Partition is not supported for external table
at org.apache.spark.sql.parser.CarbonSparkSqlParserUtil$.buildTableInfoFromCatalogTable(CarbonSparkSqlParserUtil.scala:219)
at org.apache.spark.sql.CarbonSource$.createTableInfo(CarbonSource.scala:235)
at org.apache.spark.sql.CarbonSource$.createTableMeta(CarbonSource.scala:394)
at org.apache.spark.sql.execution.command.table.CarbonCreateDataSourceTableCommand.processMetadata(CarbonCreateDataSourceTableCommand.scala:69)
at org.apache.spark.sql.execution.command.MetadataCommand$$anonfun$run$1.apply(package.scala:137)
at org.apache.spark.sql.execution.command.MetadataCommand$$anonfun$run$1.apply(package.scala:137)
at org.apache.spark.sql.execution.command.Auditable$class.runWithAudit(package.scala:118)
at org.apache.spark.sql.execution.command.MetadataCommand.runWithAudit(package.scala:134)
at org.apache.spark.sql.execution.command.MetadataCommand.run(package.scala:137)
at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult$lzycompute(commands.scala:70)
at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult(commands.scala:68)
at org.apache.spark.sql.execution.command.ExecutedCommandExec.executeCollect(commands.scala:79)
at org.apache.spark.sql.Dataset$$anonfun$6.apply(Dataset.scala:194)
at org.apache.spark.sql.Dataset$$anonfun$6.apply(Dataset.scala:194)
at org.apache.spark.sql.Dataset$$anonfun$53.apply(Dataset.scala:3364)
at org.apache.spark.sql.execution.SQLExecution$$anonfun$withNewExecutionId$1.apply(SQLExecution.scala:78)
at org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:125)
at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:73)
at org.apache.spark.sql.Dataset.withAction(Dataset.scala:3363)
at org.apache.spark.sql.Dataset.<init>(Dataset.scala:194)
at org.apache.spark.sql.Dataset$.ofRows(Dataset.scala:79)
at org.apache.spark.sql.SparkSession.sql(SparkSession.scala:643)
... 64 elided
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] Lior-AI closed issue #4206: Cannot create table with partitions in Spark in EMR
Posted by GitBox <gi...@apache.org>.
Lior-AI closed issue #4206:
URL: https://github.com/apache/carbondata/issues/4206
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] nihal0107 commented on issue #4206: Cannot create table with partitions in Spark in EMR
Posted by GitBox <gi...@apache.org>.
nihal0107 commented on issue #4206:
URL: https://github.com/apache/carbondata/issues/4206#issuecomment-903767216
Hi, As you can see the error message is `partition is not supported for external table`.
Whenever you create a table with location then it will be an external table and we are not supporting partition for the external table. Partition is only supported for the transactional table. please go through other details about partitions
https://github.com/apache/carbondata/blob/master/docs/ddl-of-carbondata.md#partition
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] Lior-AI commented on issue #4206: Cannot create table with partitions in Spark in EMR
Posted by GitBox <gi...@apache.org>.
Lior-AI commented on issue #4206:
URL: https://github.com/apache/carbondata/issues/4206#issuecomment-911815814
Solved in https://github.com/apache/carbondata/commit/42f69827e0a577b6128417104c0a49cd5bf21ad7
but now there is a different problem : https://github.com/apache/carbondata/issues/4212
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org