You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@carbondata.apache.org by geetikagupta16 <gi...@git.apache.org> on 2018/01/17 08:03:30 UTC
[GitHub] carbondata pull request #1819: Refactored code to set bad.records.action par...
GitHub user geetikagupta16 opened a pull request:
https://github.com/apache/carbondata/pull/1819
Refactored code to set bad.records.action parameter using SET command
Be sure to do all of the following checklist to help us incorporate
your contribution quickly and easily:
- [ ] Any interfaces changed?
- [ ] Any backward compatibility impacted?
- [ ] Document update required?
- [ ] Testing done
Please provide details on
- Whether new unit test cases have been added or why no new tests are required?
- How it is tested? Please attach test report.
- Is it a performance related change? Please attach the performance test report.
- Any additional information to help reviewers in testing this change.
- [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/geetikagupta16/incubator-carbondata CARBONDATA-1964
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/carbondata/pull/1819.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1819
----
commit 3d8901402a164e8d812cb20a93ce2eae637601fb
Author: Geetika Gupta <ge...@...>
Date: 2018-01-17T08:01:56Z
Refactored code to set bad.records.action parameter using SET command
----
---
[GitHub] carbondata issue #1819: [CARBONDATA-1964] Fixed bug to set bad.records.actio...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/1819
SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/3210/
---
[GitHub] carbondata issue #1819: [CARBONDATA-1964] Fixed bug to set bad.records.actio...
Posted by geetikagupta16 <gi...@git.apache.org>.
Github user geetikagupta16 commented on the issue:
https://github.com/apache/carbondata/pull/1819
retest this please
---
[GitHub] carbondata issue #1819: [CARBONDATA-1964] Fixed bug to set bad.records.actio...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1819
Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/1648/
---
[GitHub] carbondata issue #1819: [CARBONDATA-1964] Fixed bug to set bad.records.actio...
Posted by manishgupta88 <gi...@git.apache.org>.
Github user manishgupta88 commented on the issue:
https://github.com/apache/carbondata/pull/1819
LGTM...will merge once SDV build passes
---
[GitHub] carbondata pull request #1819: [CARBONDATA-1964] Fixed bug to set bad.record...
Posted by manishgupta88 <gi...@git.apache.org>.
Github user manishgupta88 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1819#discussion_r164342705
--- Diff: integration/spark-common-cluster-test/src/test/scala/org/apache/carbondata/cluster/sdv/generated/DataLoadingTestCase.scala ---
@@ -745,6 +744,34 @@ class DataLoadingTestCase extends QueryTest with BeforeAndAfterAll {
sql(s"""drop table uniqdata""").collect
}
+ test("BadRecord_Dataload_026", Include) {
+ dropTable("sales")
+ sql(
+ """CREATE TABLE IF NOT EXISTS sales(ID BigInt, date Timestamp, country String,
+ actual_price Double, Quantity int, sold_price Decimal(19,2)) STORED BY 'carbondata'""")
+ CarbonProperties.getInstance()
+ .addProperty(CarbonCommonConstants.CARBON_TIMESTAMP_FORMAT, "yyyy/MM/dd")
+ intercept[Exception] {
+ sql(s"LOAD DATA local inpath '${ resourcesPath }/badrecords/datasample.csv' INTO TABLE sales OPTIONS ('bad_records_logger_enable'='true','bad_records_action'='fail', 'DELIMITER'=" +
+ " ',', 'QUOTECHAR'= '\"')")
+ }
+ checkAnswer(s"""select count(*) from sales""",
+ Seq(Row(0)), "BadRecord_Dataload_026")
+ sql("drop table sales")
--- End diff --
use drop table if exists everywhere in your newly added test cases
---
[GitHub] carbondata issue #1819: [CARBONDATA-1964] Fixed bug to set bad.records.actio...
Posted by geetikagupta16 <gi...@git.apache.org>.
Github user geetikagupta16 commented on the issue:
https://github.com/apache/carbondata/pull/1819
@mohammadshahidkhan I have made some changes. Please check
---
[GitHub] carbondata issue #1819: [CARBONDATA-1964] Fixed bug to set bad.records.actio...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1819
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/3275/
---
[GitHub] carbondata issue #1819: [CARBONDATA-1964] Fixed bug to set bad.records.actio...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1819
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/2886/
---
[GitHub] carbondata issue #1819: [CARBONDATA-1964] Fixed bug to set bad.records.actio...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1819
Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/1701/
---
[GitHub] carbondata issue #1819: [CARBONDATA-1964] Fixed bug to set bad.records.actio...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1819
Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/2017/
---
[GitHub] carbondata issue #1819: [CARBONDATA-1964] Fixed bug to set bad.records.actio...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1819
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/3308/
---
[GitHub] carbondata issue #1819: [CARBONDATA-1964] Fixed bug to set bad.records.actio...
Posted by geetikagupta16 <gi...@git.apache.org>.
Github user geetikagupta16 commented on the issue:
https://github.com/apache/carbondata/pull/1819
@manishgupta88 The above test cases are passed on my local machine but they are failing on the CI. I am unable to figure out the cause for the same.
---
[GitHub] carbondata issue #1819: [CARBONDATA-1964] Fixed bug to set bad.records.actio...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/1819
SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/3188/
---
[GitHub] carbondata issue #1819: [CARBONDATA-1964] Fixed bug to set bad.records.actio...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1819
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/3254/
---
[GitHub] carbondata issue #1819: [CARBONDATA-1964] Fixed bug to set bad.records.actio...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/1819
SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/3242/
---
[GitHub] carbondata issue #1819: [CARBONDATA-1964] Fixed bug to set bad.records.actio...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/1819
SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/3175/
---
[GitHub] carbondata issue #1819: [CARBONDATA-1964] Fixed bug to set bad.records.actio...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1819
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/3214/
---
[GitHub] carbondata issue #1819: [CARBONDATA-1964] Fixed bug to set bad.records.actio...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1819
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/2946/
---
[GitHub] carbondata pull request #1819: [CARBONDATA-1964] Fixed bug to set bad.record...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/carbondata/pull/1819
---
[GitHub] carbondata issue #1819: [CARBONDATA-1964] Fixed bug to set bad.records.actio...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1819
Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/1653/
---
[GitHub] carbondata issue #1819: [CARBONDATA-1964] Fixed bug to set bad.records.actio...
Posted by geetikagupta16 <gi...@git.apache.org>.
Github user geetikagupta16 commented on the issue:
https://github.com/apache/carbondata/pull/1819
retest sdv please
---
[GitHub] carbondata issue #1819: [CARBONDATA-1964] Fixed bug to set bad.records.actio...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/1819
SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/2956/
---
[GitHub] carbondata issue #1819: [CARBONDATA-1964] Fixed bug to set bad.records.actio...
Posted by manishgupta88 <gi...@git.apache.org>.
Github user manishgupta88 commented on the issue:
https://github.com/apache/carbondata/pull/1819
LGTM
---
[GitHub] carbondata issue #1819: [CARBONDATA-1964] Fixed bug to set bad.records.actio...
Posted by geetikagupta16 <gi...@git.apache.org>.
Github user geetikagupta16 commented on the issue:
https://github.com/apache/carbondata/pull/1819
retest sdv please
---
[GitHub] carbondata issue #1819: [CARBONDATA-1964] Fixed bug to set bad.records.actio...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1819
Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/1981/
---
[GitHub] carbondata issue #1819: [CARBONDATA-1964] Fixed bug to set bad.records.actio...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1819
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/2935/
---
[GitHub] carbondata issue #1819: [CARBONDATA-1964] Fixed bug to set bad.records.actio...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1819
Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/1720/
---
[GitHub] carbondata issue #1819: [CARBONDATA-1964] Fixed bug to set bad.records.actio...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1819
Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/2040/
---
[GitHub] carbondata issue #1819: [CARBONDATA-1964] Fixed bug to set bad.records.actio...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1819
Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/2071/
---
[GitHub] carbondata issue #1819: [CARBONDATA-1964] Fixed bug to set bad.records.actio...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1819
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/2880/
---
[GitHub] carbondata issue #1819: [CARBONDATA-1964] Fixed bug to set bad.records.actio...
Posted by mohammadshahidkhan <gi...@git.apache.org>.
Github user mohammadshahidkhan commented on the issue:
https://github.com/apache/carbondata/pull/1819
@geetikagupta16
org/apache/carbondata/spark/rdd/CarbonDataRDDFactory.scala:507
if (loadStatus == SegmentStatus.LOAD_PARTIAL_SUCCESS &&
status(0)._2._2.failureCauses == FailureCauses.BAD_RECORDS &&
**carbonLoadModel.getBadRecordsAction.split(",")(1) == LoggerAction.FAIL.name**)
The problem is checking equally without ignoring the case.
The same problem will arise even with load options option 'bad_records_action'='fail'.
Please correct there as well and add test case to cover this scenario.
---
[GitHub] carbondata issue #1819: [CARBONDATA-1964] Fixed bug to set bad.records.actio...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1819
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/3252/
---
[GitHub] carbondata issue #1819: [CARBONDATA-1964] Fixed bug to set bad.records.actio...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/1819
SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/3208/
---
[GitHub] carbondata issue #1819: [CARBONDATA-1964] Fixed bug to set bad.records.actio...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1819
Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/2019/
---
[GitHub] carbondata issue #1819: [CARBONDATA-1964] Fixed bug to set bad.records.actio...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1819
Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/1712/
---