You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@carbondata.apache.org by Indhumathi27 <gi...@git.apache.org> on 2018/03/14 13:40:09 UTC
[GitHub] carbondata pull request #2065: [CARBONDATA-2256] Adding sdv Testcases for SE...
GitHub user Indhumathi27 opened a pull request:
https://github.com/apache/carbondata/pull/2065
[CARBONDATA-2256] Adding sdv Testcases for SET_Parameter_Dynamically_…
…Feature
Be sure to do all of the following checklist to help us incorporate
your contribution quickly and easily:
- [ ] Any interfaces changed?
- [ ] Any backward compatibility impacted?
- [ ] Document update required?
- [x] Testing done
sdv testcase added
- [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/Indhumathi27/carbondata sdv_SET
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/carbondata/pull/2065.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #2065
----
commit 996aeba6a5f8e3f7b2ce04eb27fbdfe5d3e0611c
Author: Indhumathi M <in...@...>
Date: 2018-03-14T13:32:59Z
[CARBONDATA-2256] Adding sdv Testcases for SET_Parameter_Dynamically_Feature
----
---
[GitHub] carbondata issue #2065: [CARBONDATA-2256] Adding sdv Testcases for SET_Param...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2065
Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/3105/
---
[GitHub] carbondata issue #2065: [CARBONDATA-2256] Adding sdv Testcases for SET_Param...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2065
Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/3236/
---
[GitHub] carbondata issue #2065: [CARBONDATA-2256] Adding sdv Testcases for SET_Param...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2065
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/4607/
---
[GitHub] carbondata issue #2065: [CARBONDATA-2256] Adding sdv Testcases for SET_Param...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2065
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/4445/
---
[GitHub] carbondata issue #2065: [CARBONDATA-2256] Adding sdv Testcases for SET_Param...
Posted by Indhumathi27 <gi...@git.apache.org>.
Github user Indhumathi27 commented on the issue:
https://github.com/apache/carbondata/pull/2065
retes this please
---
[GitHub] carbondata issue #2065: [CARBONDATA-2256] Adding sdv Testcases for SET_Param...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/2065
SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/3898/
---
[GitHub] carbondata issue #2065: [CARBONDATA-2256] Adding sdv Testcases for SET_Param...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2065
Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/3389/
---
[GitHub] carbondata issue #2065: [CARBONDATA-2256] Adding sdv Testcases for SET_Param...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2065
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/4377/
---
[GitHub] carbondata issue #2065: [CARBONDATA-2256] Adding sdv Testcases for SET_Param...
Posted by Indhumathi27 <gi...@git.apache.org>.
Github user Indhumathi27 commented on the issue:
https://github.com/apache/carbondata/pull/2065
@kunal642 added a test scenario where the same property is added to set command and load command and the load command should have higher priority!!
---
[GitHub] carbondata issue #2065: [CARBONDATA-2256] Adding sdv Testcases for SET_Param...
Posted by Indhumathi27 <gi...@git.apache.org>.
Github user Indhumathi27 commented on the issue:
https://github.com/apache/carbondata/pull/2065
Retest sdv please
---
[GitHub] carbondata issue #2065: [CARBONDATA-2256] Adding sdv Testcases for SET_Param...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2065
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/4614/
---
[GitHub] carbondata issue #2065: [CARBONDATA-2256] Adding sdv Testcases for SET_Param...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2065
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/4479/
---
[GitHub] carbondata pull request #2065: [CARBONDATA-2256] Adding sdv Testcases for SE...
Posted by kunal642 <gi...@git.apache.org>.
Github user kunal642 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2065#discussion_r179413817
--- Diff: integration/spark-common-cluster-test/src/test/scala/org/apache/carbondata/cluster/sdv/generated/SetParameterTestCase.scala ---
@@ -0,0 +1,267 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.carbondata.cluster.sdv.generated
+
+import org.apache.spark.sql.Row
+import org.apache.spark.sql.common.util._
+import org.scalatest.BeforeAndAfterAll
+
+import org.apache.carbondata.core.datastore.filesystem.{CarbonFile, CarbonFileFilter}
+import org.apache.carbondata.core.datastore.impl.FileFactory
+
+/**
+ * Test Class for SetParameterTestcase to verify all scenarios
+ */
+
+class SetParameterTestCase extends QueryTest with BeforeAndAfterAll {
+
+ override def beforeAll(): Unit = {
+ cleanAllTable()
+ }
+
+ private def cleanAllTable(): Unit = {
+ sql("drop table if exists carbon_table")
+ sql("drop table if exists emptyColumnValues")
+ sql("drop table if exists carbon_table_bad_record_logger")
+ sql("drop table if exists carbon_table_single_pass")
+ sql("drop table if exists carbon_table_disable_bad_record_logger")
+ sql("drop table if exists carbon_table_load")
+ sql("RESET")
+ }
+
+ override def afterAll(): Unit = {
+ cleanAllTable()
+ }
+
+ test("TC_001-test SET property for Bad Record Logger Enable=FALSE") {
+ sql("drop table if exists carbon_table_disable_bad_record_logger")
+ sql("SET carbon.options.bad.records.logger.enable=false")
+ sql(
+ "create table carbon_table_disable_bad_record_logger(empno int, empname String, designation String, " +
+ "doj Timestamp," +
+ "workgroupcategory int, workgroupcategoryname String, deptno int, deptname String," +
+ "projectcode int, projectjoindate Timestamp, projectenddate Timestamp,attendance int," +
+ "utilization int,salary int) STORED BY 'org.apache.carbondata.format'")
+ sql(
+ s"""LOAD DATA LOCAL INPATH '$resourcesPath/Data/sortcolumns/data.csv' into table
+ |carbon_table_disable_bad_record_logger options('FILEHEADER'='empno,empname,designation,doj,workgroupcategory,workgroupcategoryname,deptno,deptname,projectcode,projectjoindate,projectenddate,attendance,utilization,salary',
+ |'BAD_RECORDS_ACTION'='REDIRECT','BAD_RECORD_PATH'='$resourcesPath')"""
+ .stripMargin)
+ assert(getLogFileCount("default", "carbon_table_disable_bad_record_logger", "0") == 0)
+ }
+
+ test("TC_002-test SET property for Bad Record Logger Enable=TRUE") {
+ sql("drop table if exists carbon_table_bad_record_logger")
+ sql("SET carbon.options.bad.records.logger.enable=true")
+ sql(
+ "create table carbon_table_bad_record_logger(empno int, empname String, designation String," +
+ " doj Timestamp," +
+ "workgroupcategory int, workgroupcategoryname String, deptno int, deptname String," +
+ "projectcode int, projectjoindate Timestamp, projectenddate Timestamp,attendance int," +
+ "utilization int,salary int) STORED BY 'org.apache.carbondata.format'")
+ sql(
+ s"""LOAD DATA LOCAL INPATH '$resourcesPath/Data/sortcolumns/data.csv' into table
+ |carbon_table_bad_record_logger options('FILEHEADER'='empno,empname,designation,doj,workgroupcategory,workgroupcategoryname,deptno,deptname,projectcode,projectjoindate,projectenddate,attendance,utilization,salary',
+ |'BAD_RECORDS_ACTION'='REDIRECT','BAD_RECORD_PATH'='$resourcesPath')"""
+ .stripMargin)
+ assert(getLogFileCount("default", "carbon_table_bad_record_logger", "0") >= 1)
+ }
+
+ test("TC_003-test SET property for Bad Record Action=FORCE") {
+ sql("drop table if exists carbon_table")
+ sql("SET carbon.options.bad.records.action=force")
+ sql(
+ "create table carbon_table(empno int, empname String, designation String, doj Timestamp," +
+ "workgroupcategory int, workgroupcategoryname String, deptno int, deptname String," +
+ "projectcode int, projectjoindate Timestamp, projectenddate Timestamp,attendance int," +
+ "utilization int,salary int) STORED BY 'org.apache.carbondata.format'")
+ sql(
+ s"""LOAD DATA LOCAL INPATH '$resourcesPath/Data/sortcolumns/data.csv' into table
+ |carbon_table options('FILEHEADER'='empno,empname,designation,doj,workgroupcategory,workgroupcategoryname,deptno,deptname,projectcode,projectjoindate,projectenddate,attendance,utilization,salary')"""
+ .stripMargin)
+ checkAnswer(
+ s"""select count(*) from carbon_table""",
+ Seq(Row(11)), "SetParameterTestCase-TC_003-test SET property for Bad Record Action=FORCE")
+ }
+
+ test("TC_004-test SET property for Bad Record Action=REDIRECT") {
+ sql("drop table if exists carbon_table")
+ sql("SET carbon.options.bad.records.logger.enable=true")
+ sql("SET carbon.options.bad.records.action=redirect")
+ sql(s"SET carbon.options.bad.record.path=$resourcesPath")
+ sql(
+ "create table carbon_table(empno int, empname String, designation String, doj date," +
+ "workgroupcategory int, workgroupcategoryname String, deptno int, deptname String," +
+ "projectcode int, projectjoindate Timestamp, projectenddate Timestamp,attendance int," +
+ "utilization int,salary int) STORED BY 'org.apache.carbondata.format'")
+ sql(
+ s"""LOAD DATA LOCAL INPATH '$resourcesPath/Data/sortcolumns/data.csv' into table
+ |carbon_table options('FILEHEADER'='empno,empname,designation,doj,workgroupcategory,workgroupcategoryname,deptno,deptname,projectcode,projectjoindate,projectenddate,attendance,utilization,salary')"""
+ .stripMargin)
+ checkAnswer(
+ s"""select count(*) from carbon_table""",
+ Seq(Row(0)), "SetParameterTestCase-TC_004-test SET property for Bad Record Action=REDIRECT")
+ }
+
+ test("TC_005-test SET property for Bad Record Action=IGNORE") {
+ sql("drop table if exists carbon_table")
+ sql("SET carbon.options.bad.records.logger.enable=true")
+ sql("SET carbon.options.bad.records.action=ignore")
+ sql(
+ "create table carbon_table(empno int, empname String, designation String, doj date," +
+ "workgroupcategory int, workgroupcategoryname String, deptno int, deptname String," +
+ "projectcode int, projectjoindate Timestamp, projectenddate Timestamp,attendance int," +
+ "utilization int,salary int) STORED BY 'org.apache.carbondata.format'")
+ sql(
+ s"""LOAD DATA LOCAL INPATH '$resourcesPath/Data/sortcolumns/data.csv' into table
+ |carbon_table options('FILEHEADER'='empno,empname,designation,doj,workgroupcategory,workgroupcategoryname,deptno,deptname,projectcode,projectjoindate,projectenddate,attendance,utilization,salary')"""
+ .stripMargin)
+ checkAnswer(
+ s"""select count(*) from carbon_table""",
+ Seq(Row(0)), "SetParameterTestCase-TC_005-test SET property for Bad Record Action=IGNORE")
+ }
+
+ test("TC_006-test SET property for Bad Record Action=FAIL") {
+ sql("drop table if exists carbon_table")
+ sql("SET carbon.options.bad.records.logger.enable=true")
+ sql("SET carbon.options.bad.records.action=fail")
+ sql(
+ "create table carbon_table(empno int, empname String, designation String, doj date," +
+ "workgroupcategory int, workgroupcategoryname String, deptno int, deptname String," +
+ "projectcode int, projectjoindate Timestamp, projectenddate Timestamp,attendance int," +
+ "utilization int,salary int) STORED BY 'org.apache.carbondata.format'")
+ val exMessage = intercept[Exception] {
+ sql(
+ s"""LOAD DATA LOCAL INPATH '$resourcesPath/Data/sortcolumns/data.csv' into table
+ |carbon_table options('FILEHEADER'='empno,empname,designation,doj,workgroupcategory,workgroupcategoryname,deptno,deptname,projectcode,projectjoindate,projectenddate,attendance,utilization,salary')"""
+ .stripMargin)
+ }
+ assert(exMessage.getMessage.contains("Data load failed due to bad record"))
+ }
+
+ test("TC_007-test SET property IS__EMPTY_DATA_BAD_RECORD=FALSE") {
+ sql("drop table if exists emptyColumnValues")
+ sql("RESET")
+ sql("SET carbon.options.bad.records.logger.enable=true")
+ sql("SET carbon.options.is.empty.data.badrecord=false")
+ sql(
+ """CREATE TABLE IF NOT EXISTS emptyColumnValues(ID int,CUST_ID int,cust_name string) STORED
+ BY 'org.apache.carbondata.format'
+ """)
+ sql(
+ s"""LOAD DATA LOCAL INPATH '$resourcesPath/Data/badrecord/doubleqoute.csv' into table
+ |emptyColumnValues options('SINGLE_PASS'='true')"""
+ .stripMargin)
+ checkAnswer(
+ s"""select count(*) from emptyColumnValues""",
+ Seq(Row(1)), "SetParameterTestCase-TC_007-test SET property IS__EMPTY_DATA_BAD_RECORD=FALSE")
+ }
+
+ test("TC_008-test SET property IS__EMPTY_DATA_BAD_RECORD=TRUE") {
+ sql("drop table if exists emptyColumnValues")
+ sql("SET carbon.options.bad.records.logger.enable=true")
+ sql("SET carbon.options.is.empty.data.badrecord=true")
+ sql("SET carbon.options.bad.records.action=redirect")
+ sql(s"SET carbon.options.bad.record.path=$resourcesPath")
+ sql(
+ """CREATE TABLE IF NOT EXISTS emptyColumnValues(ID int,CUST_ID int,cust_name string) STORED
+ BY 'org.apache.carbondata.format'
+ """)
+ sql(
+ s"""LOAD DATA LOCAL INPATH '$resourcesPath/Data/badrecord/doubleqoute.csv' into table
+ |emptyColumnValues options('SINGLE_PASS'='true')"""
+ .stripMargin)
+ checkAnswer(
+ s"""select count(*) from emptyColumnValues""",
+ Seq(Row(1)), "SetParameterTestCase-TC_008-test SET property IS__EMPTY_DATA_BAD_RECORD=TRUE")
+ }
+
+ test("TC_009-test SET property for Single Pass") {
+ sql("drop table if exists carbon_table_single_pass")
+ sql("SET carbon.options.single.pass=true")
+ sql(
+ "create table carbon_table_single_pass(empno int, empname String, designation String, doj " +
+ "Timestamp,workgroupcategory int, workgroupcategoryname String, deptno int, deptname " +
+ "String," +
+ "projectcode int, projectjoindate Timestamp, projectenddate Timestamp,attendance int," +
+ "utilization int,salary int) STORED BY 'org.apache.carbondata.format'")
+ sql(
+ s"""LOAD DATA LOCAL INPATH '$resourcesPath/Data/sortcolumns/data.csv' into table
+ |carbon_table_single_pass options('ALL_DICTIONARY_PATH '='$resourcesPath','FILEHEADER'='empno,empname,designation,doj,workgroupcategory,workgroupcategoryname,deptno,deptname,projectcode,projectjoindate,projectenddate,attendance,utilization,salary')"""
+ .stripMargin)
+ sql(s"""select * from carbon_table_single_pass""")
+ }
+
+ test("TC_010-test SET property for Sort Scope-Local_Sort") {
+ sql("drop table if exists carbon_table")
+ sql("SET carbon.options.bad.records.logger.enable=true")
+ sql("SET carbon.options.sort.scope=local_sort")
+ sql(
+ "create table carbon_table(empno int, empname String, designation String, doj Timestamp," +
+ "workgroupcategory int) STORED BY 'org.apache.carbondata.format'")
+ checkExistence(sql("DESC FORMATTED carbon_table"), true, "LOCAL_SORT")
+ val sortscope=sql("DESC FORMATTED carbon_table").collect().filter(_.getString(1).trim.equals("LOCAL_SORT"))
+ assertResult(1)(sortscope.length)
+ assertResult("LOCAL_SORT")(sortscope(0).getString(1).trim)
+ }
+
+ test("TC_011-test SET property to Enable Unsafe Sort") {
+ sql("drop table if exists carbon_table")
+ sql("SET carbon.options.bad.records.logger.enable=true")
+ sql("SET enable.unsafe.sort=true")
+ sql(
+ "create table carbon_table(empno int, empname String, designation String, doj Timestamp," +
+ "workgroupcategory int, workgroupcategoryname String, deptno int, deptname String," +
+ "projectcode int, projectjoindate Timestamp, projectenddate Timestamp,attendance int," +
+ "utilization int,salary int) STORED BY 'org.apache.carbondata.format'")
+ sql(
+ s"""LOAD DATA LOCAL INPATH '$resourcesPath/Data/sortcolumns/data.csv' into table
+ |carbon_table options('FILEHEADER'='empno,empname,designation,doj,workgroupcategory,workgroupcategoryname,deptno,deptname,projectcode,projectjoindate,projectenddate,attendance,utilization,salary')"""
+ .stripMargin)
+ sql(s"""select * from carbon_table""")
--- End diff --
remove this line
---
[GitHub] carbondata issue #2065: [CARBONDATA-2256] Adding sdv Testcases for SET_Param...
Posted by Indhumathi27 <gi...@git.apache.org>.
Github user Indhumathi27 commented on the issue:
https://github.com/apache/carbondata/pull/2065
Retest sdv please
---
[GitHub] carbondata issue #2065: [CARBONDATA-2256] Adding sdv Testcases for SET_Param...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2065
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/4438/
---
[GitHub] carbondata issue #2065: [CARBONDATA-2256] Adding sdv Testcases for SET_Param...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2065
Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/3239/
---
[GitHub] carbondata issue #2065: [CARBONDATA-2256] Adding sdv Testcases for SET Param...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2065
Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/3619/
---
[GitHub] carbondata issue #2065: [CARBONDATA-2256] Adding sdv Testcases for SET_Param...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2065
Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/3143/
---
[GitHub] carbondata issue #2065: [CARBONDATA-2256] Adding sdv Testcases for SET_Param...
Posted by Indhumathi27 <gi...@git.apache.org>.
Github user Indhumathi27 commented on the issue:
https://github.com/apache/carbondata/pull/2065
Retest SDV please
---
[GitHub] carbondata issue #2065: [CARBONDATA-2256] Adding sdv Testcases for SET_Param...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2065
Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/3382/
---
[GitHub] carbondata issue #2065: [CARBONDATA-2256] Adding sdv Testcases for SET_Param...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/2065
SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/4116/
---
[GitHub] carbondata issue #2065: [CARBONDATA-2256] Adding sdv Testcases for SET_Param...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2065
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/4472/
---
[GitHub] carbondata issue #2065: [CARBONDATA-2256] Adding sdv Testcases for SET_Param...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2065
Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/3370/
---
[GitHub] carbondata issue #2065: [CARBONDATA-2256] Adding sdv Testcases for SET_Param...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/2065
SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/3967/
---
[GitHub] carbondata issue #2065: [CARBONDATA-2256] Adding sdv Testcases for SET_Param...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/2065
SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/4114/
---
[GitHub] carbondata issue #2065: [CARBONDATA-2256] Adding sdv Testcases for SET_Param...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2065
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/4426/
---
[GitHub] carbondata issue #2065: [CARBONDATA-2256] Adding sdv Testcases for SET_Param...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2065
Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/3249/
---
[GitHub] carbondata issue #2065: [CARBONDATA-2256] Adding sdv Testcases for SET_Param...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2065
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/4602/
---
[GitHub] carbondata issue #2065: [CARBONDATA-2256] Adding sdv Testcases for SET_Param...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/2065
SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/3985/
---
[GitHub] carbondata issue #2065: [CARBONDATA-2256] Adding sdv Testcases for SET Param...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2065
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/4842/
---
[GitHub] carbondata issue #2065: [CARBONDATA-2256] Adding sdv Testcases for SET_Param...
Posted by kunal642 <gi...@git.apache.org>.
Github user kunal642 commented on the issue:
https://github.com/apache/carbondata/pull/2065
can you add a test scenario where the same property is added to set command and load command and the load command should have higher priority!!
---
[GitHub] carbondata issue #2065: [CARBONDATA-2256] Adding sdv Testcases for SET_Param...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2065
Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/3212/
---
[GitHub] carbondata pull request #2065: [CARBONDATA-2256] Adding sdv Testcases for SE...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/carbondata/pull/2065
---
[GitHub] carbondata issue #2065: [CARBONDATA-2256] Adding sdv Testcases for SET_Param...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2065
Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/3205/
---
[GitHub] carbondata issue #2065: [CARBONDATA-2256] Adding sdv Testcases for SET_Param...
Posted by Indhumathi27 <gi...@git.apache.org>.
Github user Indhumathi27 commented on the issue:
https://github.com/apache/carbondata/pull/2065
Retest this please
---
[GitHub] carbondata issue #2065: [CARBONDATA-2256] Adding sdv Testcases for SET_Param...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/2065
SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/4107/
---
[GitHub] carbondata issue #2065: [CARBONDATA-2256] Adding sdv Testcases for SET_Param...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/2065
SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/3995/
---
[GitHub] carbondata issue #2065: [CARBONDATA-2256] Adding sdv Testcases for SET_Param...
Posted by Indhumathi27 <gi...@git.apache.org>.
Github user Indhumathi27 commented on the issue:
https://github.com/apache/carbondata/pull/2065
Retest this please
---
[GitHub] carbondata issue #2065: [CARBONDATA-2256] Adding sdv Testcases for SET_Param...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2065
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/4596/
---
[GitHub] carbondata issue #2065: [CARBONDATA-2256] Adding sdv Testcases for SET_Param...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2065
Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/3230/
---
[GitHub] carbondata issue #2065: [CARBONDATA-2256] Adding sdv Testcases for SET Param...
Posted by kunal642 <gi...@git.apache.org>.
Github user kunal642 commented on the issue:
https://github.com/apache/carbondata/pull/2065
LGTM
---
[GitHub] carbondata issue #2065: [CARBONDATA-2256] Adding sdv Testcases for SET_Param...
Posted by Indhumathi27 <gi...@git.apache.org>.
Github user Indhumathi27 commented on the issue:
https://github.com/apache/carbondata/pull/2065
Retest this please
---
[GitHub] carbondata issue #2065: [CARBONDATA-2256] Adding sdv Testcases for SET_Param...
Posted by Indhumathi27 <gi...@git.apache.org>.
Github user Indhumathi27 commented on the issue:
https://github.com/apache/carbondata/pull/2065
Retest sdv please
---
[GitHub] carbondata pull request #2065: [CARBONDATA-2256] Adding sdv Testcases for SE...
Posted by kunal642 <gi...@git.apache.org>.
Github user kunal642 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2065#discussion_r179413870
--- Diff: integration/spark-common-cluster-test/src/test/scala/org/apache/carbondata/cluster/sdv/generated/SetParameterTestCase.scala ---
@@ -0,0 +1,267 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.carbondata.cluster.sdv.generated
+
+import org.apache.spark.sql.Row
+import org.apache.spark.sql.common.util._
+import org.scalatest.BeforeAndAfterAll
+
+import org.apache.carbondata.core.datastore.filesystem.{CarbonFile, CarbonFileFilter}
+import org.apache.carbondata.core.datastore.impl.FileFactory
+
+/**
+ * Test Class for SetParameterTestcase to verify all scenarios
+ */
+
+class SetParameterTestCase extends QueryTest with BeforeAndAfterAll {
+
+ override def beforeAll(): Unit = {
+ cleanAllTable()
+ }
+
+ private def cleanAllTable(): Unit = {
+ sql("drop table if exists carbon_table")
+ sql("drop table if exists emptyColumnValues")
+ sql("drop table if exists carbon_table_bad_record_logger")
+ sql("drop table if exists carbon_table_single_pass")
+ sql("drop table if exists carbon_table_disable_bad_record_logger")
+ sql("drop table if exists carbon_table_load")
+ sql("RESET")
+ }
+
+ override def afterAll(): Unit = {
+ cleanAllTable()
+ }
+
+ test("TC_001-test SET property for Bad Record Logger Enable=FALSE") {
+ sql("drop table if exists carbon_table_disable_bad_record_logger")
+ sql("SET carbon.options.bad.records.logger.enable=false")
+ sql(
+ "create table carbon_table_disable_bad_record_logger(empno int, empname String, designation String, " +
+ "doj Timestamp," +
+ "workgroupcategory int, workgroupcategoryname String, deptno int, deptname String," +
+ "projectcode int, projectjoindate Timestamp, projectenddate Timestamp,attendance int," +
+ "utilization int,salary int) STORED BY 'org.apache.carbondata.format'")
+ sql(
+ s"""LOAD DATA LOCAL INPATH '$resourcesPath/Data/sortcolumns/data.csv' into table
+ |carbon_table_disable_bad_record_logger options('FILEHEADER'='empno,empname,designation,doj,workgroupcategory,workgroupcategoryname,deptno,deptname,projectcode,projectjoindate,projectenddate,attendance,utilization,salary',
+ |'BAD_RECORDS_ACTION'='REDIRECT','BAD_RECORD_PATH'='$resourcesPath')"""
+ .stripMargin)
+ assert(getLogFileCount("default", "carbon_table_disable_bad_record_logger", "0") == 0)
+ }
+
+ test("TC_002-test SET property for Bad Record Logger Enable=TRUE") {
+ sql("drop table if exists carbon_table_bad_record_logger")
+ sql("SET carbon.options.bad.records.logger.enable=true")
+ sql(
+ "create table carbon_table_bad_record_logger(empno int, empname String, designation String," +
+ " doj Timestamp," +
+ "workgroupcategory int, workgroupcategoryname String, deptno int, deptname String," +
+ "projectcode int, projectjoindate Timestamp, projectenddate Timestamp,attendance int," +
+ "utilization int,salary int) STORED BY 'org.apache.carbondata.format'")
+ sql(
+ s"""LOAD DATA LOCAL INPATH '$resourcesPath/Data/sortcolumns/data.csv' into table
+ |carbon_table_bad_record_logger options('FILEHEADER'='empno,empname,designation,doj,workgroupcategory,workgroupcategoryname,deptno,deptname,projectcode,projectjoindate,projectenddate,attendance,utilization,salary',
+ |'BAD_RECORDS_ACTION'='REDIRECT','BAD_RECORD_PATH'='$resourcesPath')"""
+ .stripMargin)
+ assert(getLogFileCount("default", "carbon_table_bad_record_logger", "0") >= 1)
+ }
+
+ test("TC_003-test SET property for Bad Record Action=FORCE") {
+ sql("drop table if exists carbon_table")
+ sql("SET carbon.options.bad.records.action=force")
+ sql(
+ "create table carbon_table(empno int, empname String, designation String, doj Timestamp," +
+ "workgroupcategory int, workgroupcategoryname String, deptno int, deptname String," +
+ "projectcode int, projectjoindate Timestamp, projectenddate Timestamp,attendance int," +
+ "utilization int,salary int) STORED BY 'org.apache.carbondata.format'")
+ sql(
+ s"""LOAD DATA LOCAL INPATH '$resourcesPath/Data/sortcolumns/data.csv' into table
+ |carbon_table options('FILEHEADER'='empno,empname,designation,doj,workgroupcategory,workgroupcategoryname,deptno,deptname,projectcode,projectjoindate,projectenddate,attendance,utilization,salary')"""
+ .stripMargin)
+ checkAnswer(
+ s"""select count(*) from carbon_table""",
+ Seq(Row(11)), "SetParameterTestCase-TC_003-test SET property for Bad Record Action=FORCE")
+ }
+
+ test("TC_004-test SET property for Bad Record Action=REDIRECT") {
+ sql("drop table if exists carbon_table")
+ sql("SET carbon.options.bad.records.logger.enable=true")
+ sql("SET carbon.options.bad.records.action=redirect")
+ sql(s"SET carbon.options.bad.record.path=$resourcesPath")
+ sql(
+ "create table carbon_table(empno int, empname String, designation String, doj date," +
+ "workgroupcategory int, workgroupcategoryname String, deptno int, deptname String," +
+ "projectcode int, projectjoindate Timestamp, projectenddate Timestamp,attendance int," +
+ "utilization int,salary int) STORED BY 'org.apache.carbondata.format'")
+ sql(
+ s"""LOAD DATA LOCAL INPATH '$resourcesPath/Data/sortcolumns/data.csv' into table
+ |carbon_table options('FILEHEADER'='empno,empname,designation,doj,workgroupcategory,workgroupcategoryname,deptno,deptname,projectcode,projectjoindate,projectenddate,attendance,utilization,salary')"""
+ .stripMargin)
+ checkAnswer(
+ s"""select count(*) from carbon_table""",
+ Seq(Row(0)), "SetParameterTestCase-TC_004-test SET property for Bad Record Action=REDIRECT")
+ }
+
+ test("TC_005-test SET property for Bad Record Action=IGNORE") {
+ sql("drop table if exists carbon_table")
+ sql("SET carbon.options.bad.records.logger.enable=true")
+ sql("SET carbon.options.bad.records.action=ignore")
+ sql(
+ "create table carbon_table(empno int, empname String, designation String, doj date," +
+ "workgroupcategory int, workgroupcategoryname String, deptno int, deptname String," +
+ "projectcode int, projectjoindate Timestamp, projectenddate Timestamp,attendance int," +
+ "utilization int,salary int) STORED BY 'org.apache.carbondata.format'")
+ sql(
+ s"""LOAD DATA LOCAL INPATH '$resourcesPath/Data/sortcolumns/data.csv' into table
+ |carbon_table options('FILEHEADER'='empno,empname,designation,doj,workgroupcategory,workgroupcategoryname,deptno,deptname,projectcode,projectjoindate,projectenddate,attendance,utilization,salary')"""
+ .stripMargin)
+ checkAnswer(
+ s"""select count(*) from carbon_table""",
+ Seq(Row(0)), "SetParameterTestCase-TC_005-test SET property for Bad Record Action=IGNORE")
+ }
+
+ test("TC_006-test SET property for Bad Record Action=FAIL") {
+ sql("drop table if exists carbon_table")
+ sql("SET carbon.options.bad.records.logger.enable=true")
+ sql("SET carbon.options.bad.records.action=fail")
+ sql(
+ "create table carbon_table(empno int, empname String, designation String, doj date," +
+ "workgroupcategory int, workgroupcategoryname String, deptno int, deptname String," +
+ "projectcode int, projectjoindate Timestamp, projectenddate Timestamp,attendance int," +
+ "utilization int,salary int) STORED BY 'org.apache.carbondata.format'")
+ val exMessage = intercept[Exception] {
+ sql(
+ s"""LOAD DATA LOCAL INPATH '$resourcesPath/Data/sortcolumns/data.csv' into table
+ |carbon_table options('FILEHEADER'='empno,empname,designation,doj,workgroupcategory,workgroupcategoryname,deptno,deptname,projectcode,projectjoindate,projectenddate,attendance,utilization,salary')"""
+ .stripMargin)
+ }
+ assert(exMessage.getMessage.contains("Data load failed due to bad record"))
+ }
+
+ test("TC_007-test SET property IS__EMPTY_DATA_BAD_RECORD=FALSE") {
+ sql("drop table if exists emptyColumnValues")
+ sql("RESET")
+ sql("SET carbon.options.bad.records.logger.enable=true")
+ sql("SET carbon.options.is.empty.data.badrecord=false")
+ sql(
+ """CREATE TABLE IF NOT EXISTS emptyColumnValues(ID int,CUST_ID int,cust_name string) STORED
+ BY 'org.apache.carbondata.format'
+ """)
+ sql(
+ s"""LOAD DATA LOCAL INPATH '$resourcesPath/Data/badrecord/doubleqoute.csv' into table
+ |emptyColumnValues options('SINGLE_PASS'='true')"""
+ .stripMargin)
+ checkAnswer(
+ s"""select count(*) from emptyColumnValues""",
+ Seq(Row(1)), "SetParameterTestCase-TC_007-test SET property IS__EMPTY_DATA_BAD_RECORD=FALSE")
+ }
+
+ test("TC_008-test SET property IS__EMPTY_DATA_BAD_RECORD=TRUE") {
+ sql("drop table if exists emptyColumnValues")
+ sql("SET carbon.options.bad.records.logger.enable=true")
+ sql("SET carbon.options.is.empty.data.badrecord=true")
+ sql("SET carbon.options.bad.records.action=redirect")
+ sql(s"SET carbon.options.bad.record.path=$resourcesPath")
+ sql(
+ """CREATE TABLE IF NOT EXISTS emptyColumnValues(ID int,CUST_ID int,cust_name string) STORED
+ BY 'org.apache.carbondata.format'
+ """)
+ sql(
+ s"""LOAD DATA LOCAL INPATH '$resourcesPath/Data/badrecord/doubleqoute.csv' into table
+ |emptyColumnValues options('SINGLE_PASS'='true')"""
+ .stripMargin)
+ checkAnswer(
+ s"""select count(*) from emptyColumnValues""",
+ Seq(Row(1)), "SetParameterTestCase-TC_008-test SET property IS__EMPTY_DATA_BAD_RECORD=TRUE")
+ }
+
+ test("TC_009-test SET property for Single Pass") {
+ sql("drop table if exists carbon_table_single_pass")
+ sql("SET carbon.options.single.pass=true")
+ sql(
+ "create table carbon_table_single_pass(empno int, empname String, designation String, doj " +
+ "Timestamp,workgroupcategory int, workgroupcategoryname String, deptno int, deptname " +
+ "String," +
+ "projectcode int, projectjoindate Timestamp, projectenddate Timestamp,attendance int," +
+ "utilization int,salary int) STORED BY 'org.apache.carbondata.format'")
+ sql(
+ s"""LOAD DATA LOCAL INPATH '$resourcesPath/Data/sortcolumns/data.csv' into table
+ |carbon_table_single_pass options('ALL_DICTIONARY_PATH '='$resourcesPath','FILEHEADER'='empno,empname,designation,doj,workgroupcategory,workgroupcategoryname,deptno,deptname,projectcode,projectjoindate,projectenddate,attendance,utilization,salary')"""
+ .stripMargin)
+ sql(s"""select * from carbon_table_single_pass""")
--- End diff --
remove this line
---
[GitHub] carbondata issue #2065: [CARBONDATA-2256] Adding sdv Testcases for SET_Param...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2065
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/4339/
---
[GitHub] carbondata issue #2065: [CARBONDATA-2256] Adding sdv Testcases for SET_Param...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/2065
SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/3987/
---
[GitHub] carbondata issue #2065: [CARBONDATA-2256] Adding sdv Testcases for SET_Param...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/2065
SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/3988/
---
[GitHub] carbondata issue #2065: [CARBONDATA-2256] Adding sdv Testcases for SET_Param...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2065
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/4463/
---
[GitHub] carbondata issue #2065: [CARBONDATA-2256] Adding sdv Testcases for SET_Param...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2065
Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/3240/
---
[GitHub] carbondata issue #2065: [CARBONDATA-2256] Adding sdv Testcases for SET_Param...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/2065
SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/4131/
---
[GitHub] carbondata issue #2065: [CARBONDATA-2256] Adding sdv Testcases for SET_Param...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2065
Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/3192/
---
[GitHub] carbondata issue #2065: [CARBONDATA-2256] Adding sdv Testcases for SET_Param...
Posted by Indhumathi27 <gi...@git.apache.org>.
Github user Indhumathi27 commented on the issue:
https://github.com/apache/carbondata/pull/2065
Retest sdv please
---
[GitHub] carbondata issue #2065: [CARBONDATA-2256] Adding sdv Testcases for SET_Param...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/2065
SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/3980/
---
[GitHub] carbondata issue #2065: [CARBONDATA-2256] Adding sdv Testcases for SET_Param...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/2065
SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/4093/
---
[GitHub] carbondata issue #2065: [CARBONDATA-2256] Adding sdv Testcases for SET Param...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/2065
SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/4312/
---
[GitHub] carbondata issue #2065: [CARBONDATA-2256] Adding sdv Testcases for SET_Param...
Posted by Indhumathi27 <gi...@git.apache.org>.
Github user Indhumathi27 commented on the issue:
https://github.com/apache/carbondata/pull/2065
Retest this please
---
[GitHub] carbondata issue #2065: [CARBONDATA-2256] Adding sdv Testcases for SET_Param...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2065
Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/3375/
---
[GitHub] carbondata issue #2065: [CARBONDATA-2256] Adding sdv Testcases for SET_Param...
Posted by Indhumathi27 <gi...@git.apache.org>.
Github user Indhumathi27 commented on the issue:
https://github.com/apache/carbondata/pull/2065
Retest this please
---
[GitHub] carbondata issue #2065: [CARBONDATA-2256] Adding sdv Testcases for SET_Param...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/2065
SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/4108/
---