You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@carbondata.apache.org by ravipesala <gi...@git.apache.org> on 2017/09/06 15:19:05 UTC
[GitHub] carbondata pull request #1337: Fix update fail when carbon.update.persist.en...
GitHub user ravipesala opened a pull request:
https://github.com/apache/carbondata/pull/1337
Fix update fail when carbon.update.persist.enable'='false'
The UDF for getting segementid while loading the data is not handled so when it needs to reexecute the rdd when persist enable is false it is not getting tupleId from carbon
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/ravipesala/incubator-carbondata update-fail
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/carbondata/pull/1337.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1337
----
commit df368b963d38b25a49d12dbbc62b660acae31154
Author: Ravindra Pesala <ra...@gmail.com>
Date: 2017-09-06T15:12:34Z
Fix update fail when carbon.update.persist.enable'='false'
----
---
[GitHub] carbondata issue #1337: [CARBONDATA-1445] Fix update fail when carbon.update...
Posted by zzcclp <gi...@git.apache.org>.
Github user zzcclp commented on the issue:
https://github.com/apache/carbondata/pull/1337
retest this please.
---
[GitHub] carbondata issue #1337: [CARBONDATA-1445] Fix update fail when carbon.update...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1337
Build Success with Spark 1.6, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/60/
---
[GitHub] carbondata issue #1337: [CARBONDATA-1445] Fix update fail when carbon.update...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/1337
SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/802/
---
[GitHub] carbondata issue #1337: [CARBONDATA-1445] Fix update fail when carbon.update...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/1337
SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/642/
---
[GitHub] carbondata issue #1337: [CARBONDATA-1445] Fix update fail when carbon.update...
Posted by zzcclp <gi...@git.apache.org>.
Github user zzcclp commented on the issue:
https://github.com/apache/carbondata/pull/1337
LGTM, it works correctly now.
---
[GitHub] carbondata issue #1337: [CARBONDATA-1445] Fix update fail when carbon.update...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1337
Build Success with Spark 1.6, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/15/
---
[GitHub] carbondata issue #1337: [CARBONDATA-1445] Fix update fail when carbon.update...
Posted by zzcclp <gi...@git.apache.org>.
Github user zzcclp commented on the issue:
https://github.com/apache/carbondata/pull/1337
retest this please
---
[GitHub] carbondata issue #1337: [CARBONDATA-1445] Fix update fail when carbon.update...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/1337
Build Success with Spark 1.6, Please check CI http://144.76.159.231:8080/job/ApacheCarbonPRBuilder1/3/
---
[GitHub] carbondata issue #1337: [CARBONDATA-1445] Fix update fail when carbon.update...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1337
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/6/
---
[GitHub] carbondata issue #1337: [CARBONDATA-1445] Fix update fail when carbon.update...
Posted by zzcclp <gi...@git.apache.org>.
Github user zzcclp commented on the issue:
https://github.com/apache/carbondata/pull/1337
@ravipesala @chenliang613 will this pr be merged into branch-1.2?
---
[GitHub] carbondata issue #1337: [CARBONDATA-1445] Fix update fail when carbon.update...
Posted by zzcclp <gi...@git.apache.org>.
Github user zzcclp commented on the issue:
https://github.com/apache/carbondata/pull/1337
retest this please
---
[GitHub] carbondata issue #1337: [CARBONDATA-1445] Fix update fail when carbon.update...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/1337
SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/626/
---
[GitHub] carbondata issue #1337: [CARBONDATA-1445] Fix update fail when carbon.update...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/1337
retest this please
---
[GitHub] carbondata issue #1337: [CARBONDATA-1445] Fix update fail when carbon.update...
Posted by zzcclp <gi...@git.apache.org>.
Github user zzcclp commented on the issue:
https://github.com/apache/carbondata/pull/1337
retest this please
---
[GitHub] carbondata issue #1337: [CARBONDATA-1445] Fix update fail when carbon.update...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/1337
SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/580/
---
[GitHub] carbondata issue #1337: [CARBONDATA-1445] Fix update fail when carbon.update...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/1337
SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/643/
---
[GitHub] carbondata issue #1337: [CARBONDATA-1445] Fix update fail when carbon.update...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1337
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/157/
---
[GitHub] carbondata issue #1337: [CARBONDATA-1445] Fix update fail when carbon.update...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/1337
SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/616/
---
[GitHub] carbondata issue #1337: [CARBONDATA-1445] Fix update fail when carbon.update...
Posted by zzcclp <gi...@git.apache.org>.
Github user zzcclp commented on the issue:
https://github.com/apache/carbondata/pull/1337
retest this please
---
[GitHub] carbondata pull request #1337: [CARBONDATA-1445] Fix update fail when carbon...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1337#discussion_r137800762
--- Diff: integration/spark-common-test/src/test/scala/org/apache/carbondata/spark/testsuite/iud/UpdateCarbonTableTestCase.scala ---
@@ -448,6 +448,40 @@ class UpdateCarbonTableTestCase extends QueryTest with BeforeAndAfterAll {
sql("DROP TABLE IF EXISTS default.carbon1")
}
+ test("""CARBONDATA-1445 carbon.update.persist.enable=false it will fail to update data""") {
+ CarbonProperties.getInstance()
+ .addProperty(CarbonCommonConstants.isPersistEnabled, "false")
+ import sqlContext.implicits._
+ val df = sqlContext.sparkContext.parallelize(0 to 50)
+ .map(x => ("a", x.toString, (x % 2).toString, x, x.toLong, x * 2))
+ .toDF("stringField1", "stringField2", "stringField3", "intField", "longField", "int2Field")
+ sql("DROP TABLE IF EXISTS default.study_carbondata ")
+ sql(s""" CREATE TABLE IF NOT EXISTS default.study_carbondata (
+ | stringField1 string,
+ | stringField2 string,
+ | stringField3 string,
+ | intField int,
+ | longField bigint,
+ | int2Field int) STORED BY 'carbondata'""".stripMargin)
+ df.write
+ .format("carbondata")
+ .option("tableName", "study_carbondata")
+ .option("compress", "true") // just valid when tempCSV is true
+ .option("tempCSV", "false")
+ .option("single_pass", "true")
+ .option("sort_scope", "LOCAL_SORT")
+ .mode(SaveMode.Append)
+ .save()
+ sql("""
+ UPDATE default.study_carbondata a
--- End diff --
Added
---
[GitHub] carbondata issue #1337: [CARBONDATA-1445] Fix update fail when carbon.update...
Posted by zzcclp <gi...@git.apache.org>.
Github user zzcclp commented on the issue:
https://github.com/apache/carbondata/pull/1337
retest this please
---
[GitHub] carbondata issue #1337: [CARBONDATA-1445] Fix update fail when carbon.update...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/1337
SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/703/
---
[GitHub] carbondata issue #1337: [CARBONDATA-1445] Fix update fail when carbon.update...
Posted by zzcclp <gi...@git.apache.org>.
Github user zzcclp commented on the issue:
https://github.com/apache/carbondata/pull/1337
retest this please
---
[GitHub] carbondata issue #1337: [CARBONDATA-1445] Fix update fail when carbon.update...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/1337
SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/794/
---
[GitHub] carbondata issue #1337: [CARBONDATA-1445] Fix update fail when carbon.update...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/1337
SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/621/
---
[GitHub] carbondata issue #1337: [CARBONDATA-1445] Fix update fail when carbon.update...
Posted by zzcclp <gi...@git.apache.org>.
Github user zzcclp commented on the issue:
https://github.com/apache/carbondata/pull/1337
what's problem with this pr, it always builds failed with Spark 2.1.0?
---
[GitHub] carbondata issue #1337: [CARBONDATA-1445] Fix update fail when carbon.update...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1337
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/126/
---
[GitHub] carbondata issue #1337: [CARBONDATA-1445] Fix update fail when carbon.update...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/1337
Build Success with Spark 1.6, Please check CI http://144.76.159.231:8080/job/ApacheCarbonPRBuilder1/23/
---
[GitHub] carbondata pull request #1337: [CARBONDATA-1445] Fix update fail when carbon...
Posted by jackylk <gi...@git.apache.org>.
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1337#discussion_r137448549
--- Diff: integration/spark-common-test/src/test/scala/org/apache/carbondata/spark/testsuite/iud/UpdateCarbonTableTestCase.scala ---
@@ -448,6 +448,40 @@ class UpdateCarbonTableTestCase extends QueryTest with BeforeAndAfterAll {
sql("DROP TABLE IF EXISTS default.carbon1")
}
+ test("""CARBONDATA-1445 carbon.update.persist.enable=false it will fail to update data""") {
+ CarbonProperties.getInstance()
+ .addProperty(CarbonCommonConstants.isPersistEnabled, "false")
+ import sqlContext.implicits._
+ val df = sqlContext.sparkContext.parallelize(0 to 50)
+ .map(x => ("a", x.toString, (x % 2).toString, x, x.toLong, x * 2))
+ .toDF("stringField1", "stringField2", "stringField3", "intField", "longField", "int2Field")
+ sql("DROP TABLE IF EXISTS default.study_carbondata ")
+ sql(s""" CREATE TABLE IF NOT EXISTS default.study_carbondata (
+ | stringField1 string,
+ | stringField2 string,
+ | stringField3 string,
+ | intField int,
+ | longField bigint,
+ | int2Field int) STORED BY 'carbondata'""".stripMargin)
+ df.write
+ .format("carbondata")
+ .option("tableName", "study_carbondata")
+ .option("compress", "true") // just valid when tempCSV is true
+ .option("tempCSV", "false")
+ .option("single_pass", "true")
+ .option("sort_scope", "LOCAL_SORT")
+ .mode(SaveMode.Append)
+ .save()
+ sql("""
+ UPDATE default.study_carbondata a
--- End diff --
Please add assert to check the updated value
---
[GitHub] carbondata issue #1337: [CARBONDATA-1445] Fix update fail when carbon.update...
Posted by zzcclp <gi...@git.apache.org>.
Github user zzcclp commented on the issue:
https://github.com/apache/carbondata/pull/1337
retest this please
---
[GitHub] carbondata issue #1337: [CARBONDATA-1445] Fix update fail when carbon.update...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/1337
SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/582/
---
[GitHub] carbondata issue #1337: [CARBONDATA-1445] Fix update fail when carbon.update...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/1337
retest this please
---
[GitHub] carbondata issue #1337: [CARBONDATA-1445] Fix update fail when carbon.update...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/1337
SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/655/
---
[GitHub] carbondata issue #1337: [CARBONDATA-1445] Fix update fail when carbon.update...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/1337
SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/619/
---
[GitHub] carbondata issue #1337: [CARBONDATA-1445] Fix update fail when carbon.update...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1337
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/183/
---
[GitHub] carbondata pull request #1337: [CARBONDATA-1445] Fix update fail when carbon...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/carbondata/pull/1337
---
[GitHub] carbondata issue #1337: [CARBONDATA-1445] Fix update fail when carbon.update...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/1337
SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/785/
---
[GitHub] carbondata issue #1337: [CARBONDATA-1445] Fix update fail when carbon.update...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1337
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/97/
---
[GitHub] carbondata issue #1337: [CARBONDATA-1445] Fix update fail when carbon.update...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/1337
SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/729/
---
[GitHub] carbondata issue #1337: [CARBONDATA-1445] Fix update fail when carbon.update...
Posted by zzcclp <gi...@git.apache.org>.
Github user zzcclp commented on the issue:
https://github.com/apache/carbondata/pull/1337
retest this please
---
[GitHub] carbondata issue #1337: [CARBONDATA-1445] Fix update fail when carbon.update...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1337
Build Success with Spark 1.6, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/41/
---
[GitHub] carbondata issue #1337: [CARBONDATA-1445] Fix update fail when carbon.update...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1337
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/172/
---
[GitHub] carbondata issue #1337: [CARBONDATA-1445] Fix update fail when carbon.update...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1337
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/141/
---
[GitHub] carbondata issue #1337: [CARBONDATA-1445] Fix update fail when carbon.update...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1337
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/166/
---
[GitHub] carbondata issue #1337: [CARBONDATA-1445] Fix update fail when carbon.update...
Posted by zzcclp <gi...@git.apache.org>.
Github user zzcclp commented on the issue:
https://github.com/apache/carbondata/pull/1337
retest this please
---
[GitHub] carbondata issue #1337: [CARBONDATA-1445] Fix update fail when carbon.update...
Posted by zzcclp <gi...@git.apache.org>.
Github user zzcclp commented on the issue:
https://github.com/apache/carbondata/pull/1337
retest this please
---
[GitHub] carbondata issue #1337: [CARBONDATA-1445] Fix update fail when carbon.update...
Posted by zzcclp <gi...@git.apache.org>.
Github user zzcclp commented on the issue:
https://github.com/apache/carbondata/pull/1337
retest this please.
---
[GitHub] carbondata issue #1337: [CARBONDATA-1445] Fix update fail when carbon.update...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/1337
SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/702/
---
[GitHub] carbondata issue #1337: [CARBONDATA-1445] Fix update fail when carbon.update...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1337
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/125/
---
[GitHub] carbondata issue #1337: [CARBONDATA-1445] Fix update fail when carbon.update...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/1337
retest this please
---
[GitHub] carbondata issue #1337: [CARBONDATA-1445] Fix update fail when carbon.update...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/1337
SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/649/
---
[GitHub] carbondata issue #1337: [CARBONDATA-1445] Fix update fail when carbon.update...
Posted by jackylk <gi...@git.apache.org>.
Github user jackylk commented on the issue:
https://github.com/apache/carbondata/pull/1337
LGTM
---
[GitHub] carbondata issue #1337: [CARBONDATA-1445] Fix update fail when carbon.update...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/1337
SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/814/
---
[GitHub] carbondata issue #1337: [CARBONDATA-1445] Fix update fail when carbon.update...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1337
Build Success with Spark 1.6, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/48/
---