You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@carbondata.apache.org by GitBox <gi...@apache.org> on 2020/05/05 16:47:11 UTC
[GitHub] [carbondata] VenuReddy2103 opened a new pull request #3745: [CARBONDATA-3793]Data load with partition columns fail with InvalidLoadOptionException when load option 'header' is set to 'true'
VenuReddy2103 opened a new pull request #3745:
URL: https://github.com/apache/carbondata/pull/3745
### Why is this PR needed?
Data load with partition columns fail with InvalidLoadOptionException when load option `header` is set to `true`.
**Callstack:**
`2020-05-05 21:49:35 AUDIT audit:97 - {"time":"5 May, 2020 9:49:35 PM IST","username":"root1","opName":"LOAD DATA","opId":"199081091980878","opStatus":"FAILED","opTime":"1734 ms","table":"default.source","extraInfo":{"Exception":"org.apache.carbondata.common.exceptions.sql.InvalidLoadOptionException","Message":"When 'header' option is true, 'fileheader' option is not required."}}
Exception in thread "main" org.apache.carbondata.common.exceptions.sql.InvalidLoadOptionException: When 'header' option is true, 'fileheader' option is not required.
at org.apache.carbondata.processing.loading.model.CarbonLoadModelBuilder.build(CarbonLoadModelBuilder.java:203)
at org.apache.carbondata.processing.loading.model.CarbonLoadModelBuilder.build(CarbonLoadModelBuilder.java:126)
at org.apache.spark.sql.execution.datasources.SparkCarbonTableFormat.prepareWrite(SparkCarbonTableFormat.scala:132)
at org.apache.spark.sql.execution.datasources.FileFormatWriter$.write(FileFormatWriter.scala:103)
at org.apache.spark.sql.execution.command.management.CarbonInsertIntoHadoopFsRelationCommand.run(CarbonInsertIntoHadoopFsRelationCommand.scala:160)
at org.apache.spark.sql.execution.command.DataWritingCommandExec.sideEffectResult$lzycompute(commands.scala:104)
at org.apache.spark.sql.execution.command.DataWritingCommandExec.sideEffectResult(commands.scala:102)`
### What changes were proposed in this PR?
In `SparkCarbonTableFormat.prepareWrite()` method after adding the `fileheader` option with header columns, to `optionsFinal`, need to set `header` option to `false`.
### Does this PR introduce any user interface change?
- No
### Is any new testcase added?
- No
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3745: [CARBONDATA-3793]Data load with partition columns fail with InvalidLoadOptionException when load option 'header' is set to 'true'
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on pull request #3745:
URL: https://github.com/apache/carbondata/pull/3745#issuecomment-625700976
Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/2975/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3745: [CARBONDATA-3793]Data load with partition columns fail with InvalidLoadOptionException when load option 'header' is set to 'true'
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on pull request #3745:
URL: https://github.com/apache/carbondata/pull/3745#issuecomment-626671668
Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/2997/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] VenuReddy2103 commented on a change in pull request #3745: [CARBONDATA-3793]Data load with partition columns fail with InvalidLoadOptionException when load option 'header' is set to 'true'
Posted by GitBox <gi...@apache.org>.
VenuReddy2103 commented on a change in pull request #3745:
URL: https://github.com/apache/carbondata/pull/3745#discussion_r421957392
##########
File path: integration/spark/src/main/scala/org/apache/spark/sql/execution/datasources/SparkCarbonTableFormat.scala
##########
@@ -126,6 +126,7 @@ with Serializable {
optionsFinal.put(
"fileheader",
dataSchema.fields.map(_.name.toLowerCase).mkString(",") + "," + partitionStr)
+ optionsFinal.put("header", "false")
Review comment:
done
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3745: [CARBONDATA-3793]Data load with partition columns fail with InvalidLoadOptionException when load option 'header' is set to 'true'
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on pull request #3745:
URL: https://github.com/apache/carbondata/pull/3745#issuecomment-625702052
Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/1257/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3745: [CARBONDATA-3793]Data load with partition columns fail with InvalidLoadOptionException when load option 'header' is set to 'true'
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on pull request #3745:
URL: https://github.com/apache/carbondata/pull/3745#issuecomment-626549803
Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/1268/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3745: [CARBONDATA-3793]Data load with partition columns fail with InvalidLoadOptionException when load option 'header' is set to 'true'
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on pull request #3745:
URL: https://github.com/apache/carbondata/pull/3745#issuecomment-624245981
Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/2944/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3745: [CARBONDATA-3793]Data load with partition columns fail with InvalidLoadOptionException when load option 'header' is set to 'true'
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on pull request #3745:
URL: https://github.com/apache/carbondata/pull/3745#issuecomment-626650923
Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/1278/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] akashrn5 commented on a change in pull request #3745: [CARBONDATA-3793]Data load with partition columns fail with InvalidLoadOptionException when load option 'header' is set to 'true'
Posted by GitBox <gi...@apache.org>.
akashrn5 commented on a change in pull request #3745:
URL: https://github.com/apache/carbondata/pull/3745#discussion_r421406251
##########
File path: integration/spark/src/main/scala/org/apache/spark/sql/execution/datasources/SparkCarbonTableFormat.scala
##########
@@ -126,6 +126,7 @@ with Serializable {
optionsFinal.put(
"fileheader",
dataSchema.fields.map(_.name.toLowerCase).mkString(",") + "," + partitionStr)
+ optionsFinal.put("header", "false")
Review comment:
i think you can remove from filling in optionsLocal
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3745: [CARBONDATA-3793]Data load with partition columns fail with InvalidLoadOptionException when load option 'header' is set to 'true'
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on pull request #3745:
URL: https://github.com/apache/carbondata/pull/3745#issuecomment-624244314
Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/1226/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] akashrn5 commented on a change in pull request #3745: [CARBONDATA-3793]Data load with partition columns fail with InvalidLoadOptionException when load option 'header' is set to 'true'
Posted by GitBox <gi...@apache.org>.
akashrn5 commented on a change in pull request #3745:
URL: https://github.com/apache/carbondata/pull/3745#discussion_r421350945
##########
File path: integration/spark/src/main/scala/org/apache/spark/sql/execution/datasources/SparkCarbonTableFormat.scala
##########
@@ -126,6 +126,7 @@ with Serializable {
optionsFinal.put(
"fileheader",
dataSchema.fields.map(_.name.toLowerCase).mkString(",") + "," + partitionStr)
+ optionsFinal.put("header", "false")
Review comment:
please refer line 77 in org.apache.carbondata.processing.loading.model.CarbonLoadModelBuilder#build(java.util.Map<java.lang.String,java.lang.String>, long, java.lang.String), can be handled like that
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3745: [CARBONDATA-3793]Data load with partition columns fail with InvalidLoadOptionException when load option 'header' is set to 'true'
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on pull request #3745:
URL: https://github.com/apache/carbondata/pull/3745#issuecomment-626551710
Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/2986/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] akashrn5 commented on pull request #3745: [CARBONDATA-3793]Data load with partition columns fail with InvalidLoadOptionException when load option 'header' is set to 'true'
Posted by GitBox <gi...@apache.org>.
akashrn5 commented on pull request #3745:
URL: https://github.com/apache/carbondata/pull/3745#issuecomment-626294615
@VenuReddy2103 can you please add one test case
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org