You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@carbondata.apache.org by QiangCai <gi...@git.apache.org> on 2017/04/25 12:53:47 UTC
[GitHub] incubator-carbondata pull request #849: fix SparkSessionExample
GitHub user QiangCai opened a pull request:
https://github.com/apache/incubator-carbondata/pull/849
fix SparkSessionExample
revert old code to fix SparkSessionExample.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/QiangCai/incubator-carbondata InserIntoForSparkSession
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-carbondata/pull/849.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #849
----
commit 73512e397d7f7b882e1ef78f73c62c0d58d8a340
Author: QiangCai <qi...@qq.com>
Date: 2017-04-25T12:51:16Z
fix SparkSessionExample
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-carbondata pull request #849: [CARBONDATA-992] Fix error of Exampl...
Posted by jackylk <gi...@git.apache.org>.
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/incubator-carbondata/pull/849#discussion_r113793937
--- Diff: integration/spark2/src/main/scala/org/apache/spark/sql/CarbonDatasourceHadoopRelation.scala ---
@@ -75,4 +75,16 @@ case class CarbonDatasourceHadoopRelation(
", " + "Table name :" + carbonTable.getFactTableName + ", Schema :" + tableSchema + " ]"
}
+ override def insert(data: DataFrame, overwrite: Boolean): Unit = {
+ if (carbonRelation.output.size > CarbonCommonConstants.DEFAULT_MAX_NUMBER_OF_COLUMNS) {
+ sys.error("Maximum supported column by carbon is: " +
+ CarbonCommonConstants.DEFAULT_MAX_NUMBER_OF_COLUMNS)
+ }
+ if(data.logicalPlan.output.size >= carbonRelation.output.size) {
--- End diff --
what if the size equals but schema is different?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-carbondata pull request #849: [CARBONDATA-992] Fix error of Exampl...
Posted by jackylk <gi...@git.apache.org>.
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/incubator-carbondata/pull/849#discussion_r113793605
--- Diff: examples/spark2/src/main/scala/org/apache/carbondata/examples/CarbonSessionExample.scala ---
@@ -34,7 +34,10 @@ object CarbonSessionExample {
val metastoredb = s"$rootPath/examples/spark2/target"
CarbonProperties.getInstance()
- .addProperty(CarbonCommonConstants.CARBON_TIMESTAMP_FORMAT, "yyyy/MM/dd")
+ .addProperty(CarbonCommonConstants.CARBON_TIMESTAMP_FORMAT, "yyyy/MM/dd HH:mm:ss")
+
+ CarbonProperties.getInstance()
+ .addProperty(CarbonCommonConstants.CARBON_DATE_FORMAT, "yyyy/MM/dd")
--- End diff --
you can use `addProperty` directly on return value of last `addProperty`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-carbondata issue #849: Fix SparkSessionExample
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/incubator-carbondata/pull/849
Build Failed with Spark 1.6.2, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/1791/
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-carbondata issue #849: [CARBONDATA-992] Fix error of Example
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/incubator-carbondata/pull/849
Build Success with Spark 1.6.2, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/1874/
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-carbondata issue #849: [CARBONDATA-992] Fix error of Example
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/incubator-carbondata/pull/849
Build Success with Spark 1.6.2, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/1822/
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-carbondata issue #849: [CARBONDATA-992] Fix error of Example
Posted by chenliang613 <gi...@git.apache.org>.
Github user chenliang613 commented on the issue:
https://github.com/apache/incubator-carbondata/pull/849
LGTM
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-carbondata pull request #849: [CARBONDATA-992] Fix error of Exampl...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/incubator-carbondata/pull/849
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-carbondata pull request #849: [CARBONDATA-992] Fix error of Exampl...
Posted by QiangCai <gi...@git.apache.org>.
Github user QiangCai commented on a diff in the pull request:
https://github.com/apache/incubator-carbondata/pull/849#discussion_r114125771
--- Diff: integration/spark2/src/main/scala/org/apache/spark/sql/CarbonDatasourceHadoopRelation.scala ---
@@ -75,4 +75,16 @@ case class CarbonDatasourceHadoopRelation(
", " + "Table name :" + carbonTable.getFactTableName + ", Schema :" + tableSchema + " ]"
}
+ override def insert(data: DataFrame, overwrite: Boolean): Unit = {
+ if (carbonRelation.output.size > CarbonCommonConstants.DEFAULT_MAX_NUMBER_OF_COLUMNS) {
+ sys.error("Maximum supported column by carbon is: " +
+ CarbonCommonConstants.DEFAULT_MAX_NUMBER_OF_COLUMNS)
+ }
+ if(data.logicalPlan.output.size >= carbonRelation.output.size) {
+ LoadTableByInsert(this, data.logicalPlan).run(sparkSession)
+ } else {
+ sys.error("Cannot insert into target table because column number are different")
--- End diff --
I remember this error message is same with hive.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-carbondata pull request #849: [CARBONDATA-992] Fix error of Exampl...
Posted by jackylk <gi...@git.apache.org>.
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/incubator-carbondata/pull/849#discussion_r113794079
--- Diff: integration/spark2/src/main/scala/org/apache/spark/sql/CarbonDatasourceHadoopRelation.scala ---
@@ -75,4 +75,16 @@ case class CarbonDatasourceHadoopRelation(
", " + "Table name :" + carbonTable.getFactTableName + ", Schema :" + tableSchema + " ]"
}
+ override def insert(data: DataFrame, overwrite: Boolean): Unit = {
+ if (carbonRelation.output.size > CarbonCommonConstants.DEFAULT_MAX_NUMBER_OF_COLUMNS) {
+ sys.error("Maximum supported column by carbon is: " +
+ CarbonCommonConstants.DEFAULT_MAX_NUMBER_OF_COLUMNS)
+ }
+ if(data.logicalPlan.output.size >= carbonRelation.output.size) {
+ LoadTableByInsert(this, data.logicalPlan).run(sparkSession)
+ } else {
+ sys.error("Cannot insert into target table because column number are different")
--- End diff --
Better to give more detail information in the log to describe the schema difference
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-carbondata issue #849: [CARBONDATA-992] Fix error of Example
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/incubator-carbondata/pull/849
Build Success with Spark 1.6.2, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/1875/
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-carbondata issue #849: Fix SparkSessionExample
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/incubator-carbondata/pull/849
Build Success with Spark 1.6.2, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/1796/
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-carbondata pull request #849: [CARBONDATA-992] Fix error of Exampl...
Posted by QiangCai <gi...@git.apache.org>.
Github user QiangCai commented on a diff in the pull request:
https://github.com/apache/incubator-carbondata/pull/849#discussion_r114124975
--- Diff: examples/spark2/src/main/scala/org/apache/carbondata/examples/CarbonSessionExample.scala ---
@@ -34,7 +34,10 @@ object CarbonSessionExample {
val metastoredb = s"$rootPath/examples/spark2/target"
CarbonProperties.getInstance()
- .addProperty(CarbonCommonConstants.CARBON_TIMESTAMP_FORMAT, "yyyy/MM/dd")
+ .addProperty(CarbonCommonConstants.CARBON_TIMESTAMP_FORMAT, "yyyy/MM/dd HH:mm:ss")
+
+ CarbonProperties.getInstance()
+ .addProperty(CarbonCommonConstants.CARBON_DATE_FORMAT, "yyyy/MM/dd")
--- End diff --
ok
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-carbondata pull request #849: [CARBONDATA-992] Fix error of Exampl...
Posted by QiangCai <gi...@git.apache.org>.
Github user QiangCai commented on a diff in the pull request:
https://github.com/apache/incubator-carbondata/pull/849#discussion_r114125190
--- Diff: integration/spark2/src/main/scala/org/apache/spark/sql/CarbonDatasourceHadoopRelation.scala ---
@@ -75,4 +75,16 @@ case class CarbonDatasourceHadoopRelation(
", " + "Table name :" + carbonTable.getFactTableName + ", Schema :" + tableSchema + " ]"
}
+ override def insert(data: DataFrame, overwrite: Boolean): Unit = {
+ if (carbonRelation.output.size > CarbonCommonConstants.DEFAULT_MAX_NUMBER_OF_COLUMNS) {
+ sys.error("Maximum supported column by carbon is: " +
+ CarbonCommonConstants.DEFAULT_MAX_NUMBER_OF_COLUMNS)
+ }
+ if(data.logicalPlan.output.size >= carbonRelation.output.size) {
--- End diff --
for different schema, need convert column datatype.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-carbondata issue #849: [CARBONDATA-992] Fix error of Example
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/incubator-carbondata/pull/849
Build Success with Spark 1.6.2, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/1855/
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-carbondata issue #849: [CARBONDATA-992] Fix error of Example
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/incubator-carbondata/pull/849
Build Failed with Spark 1.6.2, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/1852/
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-carbondata issue #849: Fix SparkSessionExample
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/incubator-carbondata/pull/849
Build Success with Spark 1.6.2, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/1798/
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-carbondata issue #849: [CARBONDATA-992] Fix error of Example
Posted by QiangCai <gi...@git.apache.org>.
Github user QiangCai commented on the issue:
https://github.com/apache/incubator-carbondata/pull/849
@ravipesala please review the code.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---