You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@carbondata.apache.org by Indhumathi27 <gi...@git.apache.org> on 2018/04/18 09:11:52 UTC
[GitHub] carbondata pull request #2180: [CARBONDATA-2356] Added UT & SDV Testcases fo...
GitHub user Indhumathi27 opened a pull request:
https://github.com/apache/carbondata/pull/2180
[CARBONDATA-2356] Added UT & SDV Testcases for LuceneDataMap
- [X] Any interfaces changed?
NA
- [X] Any backward compatibility impacted?
NA
- [X] Document update required?
NA
- [X] Testing done
Please provide details on
- Whether new unit test cases have been added or why no new tests are required?
- How it is tested? Please attach test report.
- Is it a performance related change? Please attach the performance test report.
- Any additional information to help reviewers in testing this change.
NA
- [X] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.
NA
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/Indhumathi27/carbondata lucsdvtest
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/carbondata/pull/2180.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #2180
----
commit a5c86c4c13049d42fac4aa0963da7b6fd385f42c
Author: Indhumathi27 <in...@...>
Date: 2018-04-18T06:10:48Z
[CARBONDATA-2356] Added UT & SDV Testcases for LuceneDataMap
----
---
[GitHub] carbondata issue #2180: [CARBONDATA-2356] Added UT Scenarios for LuceneDataM...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2180
Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/3930/
---
[GitHub] carbondata issue #2180: [CARBONDATA-2356] Added UT Scenarios for LuceneDataM...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/2180
SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/4571/
---
[GitHub] carbondata pull request #2180: [CARBONDATA-2356] Added UT Scenarios for Luce...
Posted by Indhumathi27 <gi...@git.apache.org>.
Github user Indhumathi27 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2180#discussion_r183212178
--- Diff: integration/spark-common-test/src/test/scala/org/apache/carbondata/datamap/lucene/LuceneFineGrainDataMapSuite.scala ---
@@ -197,13 +197,353 @@ class LuceneFineGrainDataMapSuite extends QueryTest with BeforeAndAfterAll {
sql("DROP TABLE IF EXISTS datamap_test3")
}
+ test("test lucene fine grain data map for create datamap with Duplicate Columns") {
+ sql("DROP TABLE IF EXISTS datamap_test_table")
+ sql(
+ """
+ | CREATE TABLE datamap_test_table(id INT, name STRING, city STRING, age INT)
+ | STORED BY 'carbondata'
+ | TBLPROPERTIES('SORT_COLUMNS'='city,name', 'SORT_SCOPE'='LOCAL_SORT')
+ """.stripMargin)
+ val exception_duplicate_column: Exception = intercept[MalformedDataMapCommandException] {
+ sql(
+ s"""
+ | CREATE DATAMAP dm ON TABLE datamap_test_table
+ | USING 'lucene'
+ | DMProperties('TEXT_COLUMNS'='name')
+ """.stripMargin)
+ sql(
+ s"""
+ | CREATE DATAMAP dm1 ON TABLE datamap_test_table
+ | USING 'lucene'
+ | DMProperties('TEXT_COLUMNS'='name')
+ """.stripMargin)
+ }
+ assert(exception_duplicate_column.getMessage
+ .contains("Create lucene datamap dm1 failed, datamap already exists on column(s) name"))
+ sql("drop datamap if exists dm on table datamap_test_table")
+ }
+
+ test("test lucene fine grain data map with wildcard matching ") {
+ sql("DROP TABLE IF EXISTS datamap_test_table")
+ sql(
+ """
+ | CREATE TABLE datamap_test_table(id INT, name STRING, city STRING, age INT)
+ | STORED BY 'carbondata'
+ | TBLPROPERTIES('SORT_COLUMNS'='city,name', 'SORT_SCOPE'='LOCAL_SORT')
+ """.stripMargin)
+ sql(
+ s"""
+ | CREATE DATAMAP dm ON TABLE datamap_test_table
+ | USING 'lucene'
+ | DMProperties('TEXT_COLUMNS'='Name , cIty')
--- End diff --
okay
---
[GitHub] carbondata issue #2180: [CARBONDATA-2356] Added UT Scenarios for LuceneDataM...
Posted by Indhumathi27 <gi...@git.apache.org>.
Github user Indhumathi27 commented on the issue:
https://github.com/apache/carbondata/pull/2180
Retest this please
---
[GitHub] carbondata issue #2180: [CARBONDATA-2356] Added UT Scenarios for LuceneDataM...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2180
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/5436/
---
[GitHub] carbondata issue #2180: [CARBONDATA-2356] Added UT Scenarios for LuceneDataM...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2180
Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/4034/
---
[GitHub] carbondata pull request #2180: [CARBONDATA-2356] Added UT Scenarios for Luce...
Posted by jackylk <gi...@git.apache.org>.
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2180#discussion_r183211288
--- Diff: datamap/lucene/src/main/java/org/apache/carbondata/datamap/lucene/LuceneFineGrainDataMap.java ---
@@ -157,6 +162,14 @@ private String getQueryString(Expression expression) {
return null;
}
+ //maximum records to return
+ if (strQuery.contains(",") && strQuery.indexOf(",") != strQuery.length() - 1) {
--- End diff --
It seems there is no testcase for `maxDoc`, do I miss anything?
---
[GitHub] carbondata issue #2180: [CARBONDATA-2356] Added UT Scenarios for LuceneDataM...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2180
Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/4259/
---
[GitHub] carbondata issue #2180: [CARBONDATA-2356] Added UT Scenarios for LuceneDataM...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2180
Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/4283/
---
[GitHub] carbondata issue #2180: [CARBONDATA-2356] Added UT Scenarios for LuceneDataM...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/2180
SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/4583/
---
[GitHub] carbondata issue #2180: [CARBONDATA-2356] Added UT Scenarios for LuceneDataM...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2180
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/5425/
---
[GitHub] carbondata pull request #2180: [CARBONDATA-2356] Added UT Scenarios for Luce...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/carbondata/pull/2180
---
[GitHub] carbondata pull request #2180: [CARBONDATA-2356] Added UT Scenarios for Luce...
Posted by Indhumathi27 <gi...@git.apache.org>.
Github user Indhumathi27 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2180#discussion_r183140729
--- Diff: datamap/lucene/src/main/java/org/apache/carbondata/datamap/lucene/LuceneFineGrainDataMap.java ---
@@ -182,7 +182,7 @@ private String getQueryString(Expression expression) {
// execute index search
TopDocs result;
try {
- result = indexSearcher.search(query, indexReader.maxDoc());
+ result = indexSearcher.search(query, Integer.MAX_VALUE);
--- End diff --
Added
---
[GitHub] carbondata pull request #2180: [CARBONDATA-2356] Added UT Scenarios for Luce...
Posted by jackylk <gi...@git.apache.org>.
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2180#discussion_r183211234
--- Diff: integration/spark-common-test/src/test/scala/org/apache/carbondata/datamap/lucene/LuceneFineGrainDataMapSuite.scala ---
@@ -197,13 +197,353 @@ class LuceneFineGrainDataMapSuite extends QueryTest with BeforeAndAfterAll {
sql("DROP TABLE IF EXISTS datamap_test3")
}
+ test("test lucene fine grain data map for create datamap with Duplicate Columns") {
+ sql("DROP TABLE IF EXISTS datamap_test_table")
+ sql(
+ """
+ | CREATE TABLE datamap_test_table(id INT, name STRING, city STRING, age INT)
+ | STORED BY 'carbondata'
+ | TBLPROPERTIES('SORT_COLUMNS'='city,name', 'SORT_SCOPE'='LOCAL_SORT')
+ """.stripMargin)
+ val exception_duplicate_column: Exception = intercept[MalformedDataMapCommandException] {
+ sql(
+ s"""
+ | CREATE DATAMAP dm ON TABLE datamap_test_table
+ | USING 'lucene'
+ | DMProperties('TEXT_COLUMNS'='name')
+ """.stripMargin)
+ sql(
+ s"""
+ | CREATE DATAMAP dm1 ON TABLE datamap_test_table
+ | USING 'lucene'
+ | DMProperties('TEXT_COLUMNS'='name')
+ """.stripMargin)
+ }
+ assert(exception_duplicate_column.getMessage
+ .contains("Create lucene datamap dm1 failed, datamap already exists on column(s) name"))
+ sql("drop datamap if exists dm on table datamap_test_table")
+ }
+
+ test("test lucene fine grain data map with wildcard matching ") {
+ sql("DROP TABLE IF EXISTS datamap_test_table")
+ sql(
+ """
+ | CREATE TABLE datamap_test_table(id INT, name STRING, city STRING, age INT)
+ | STORED BY 'carbondata'
+ | TBLPROPERTIES('SORT_COLUMNS'='city,name', 'SORT_SCOPE'='LOCAL_SORT')
+ """.stripMargin)
+ sql(
+ s"""
+ | CREATE DATAMAP dm ON TABLE datamap_test_table
+ | USING 'lucene'
+ | DMProperties('TEXT_COLUMNS'='Name , cIty')
+ """.stripMargin)
+ sql(s"LOAD DATA LOCAL INPATH '$file2' INTO TABLE datamap_test_table OPTIONS('header'='false')")
+ checkAnswer(sql("SELECT * FROM datamap_test_table WHERE TEXT_MATCH('name:n99*')"),
+ sql("select * from datamap_test_table where name like 'n99%'"))
+ checkAnswer(sql("SELECT * FROM datamap_test_table WHERE TEXT_MATCH('name:n*9')"),
+ sql(s"select * from datamap_test_table where name like 'n%9'"))
+ sql("drop datamap if exists dm on table datamap_test_table")
+ }
+
+ test("test lucene fine grain data map with TEXT_MATCH 'AND' Filter ") {
+ sql("DROP TABLE IF EXISTS datamap_test_table")
+ sql(
+ """
+ | CREATE TABLE datamap_test_table(id INT, name STRING, city STRING, age INT)
+ | STORED BY 'carbondata'
+ | TBLPROPERTIES('SORT_COLUMNS'='city,name', 'SORT_SCOPE'='LOCAL_SORT')
+ """.stripMargin)
+ sql(
+ s"""
+ | CREATE DATAMAP dm ON TABLE datamap_test_table
+ | USING 'lucene'
+ | DMProperties('TEXT_COLUMNS'='Name , cIty')
--- End diff --
change to name, city, non-capital
---
[GitHub] carbondata issue #2180: [CARBONDATA-2356] Added UT Scenarios for LuceneDataM...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2180
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/5256/
---
[GitHub] carbondata issue #2180: [CARBONDATA-2356] Added UT Scenarios for LuceneDataM...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2180
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/5271/
---
[GitHub] carbondata pull request #2180: [CARBONDATA-2356] Added UT Scenarios for Luce...
Posted by jackylk <gi...@git.apache.org>.
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2180#discussion_r183083696
--- Diff: integration/spark-common-test/src/test/scala/org/apache/carbondata/datamap/lucene/LuceneFineGrainDataMapSuite.scala ---
@@ -197,13 +197,334 @@ class LuceneFineGrainDataMapSuite extends QueryTest with BeforeAndAfterAll {
sql("DROP TABLE IF EXISTS datamap_test3")
}
+ test("test lucene fine grain data map for create datamap with Duplicate Columns") {
+ sql("DROP TABLE IF EXISTS datamap_test_table")
+ sql(
+ """
+ | CREATE TABLE datamap_test_table(id INT, name STRING, city STRING, age INT)
+ | STORED BY 'carbondata'
+ | TBLPROPERTIES('SORT_COLUMNS'='city,name', 'SORT_SCOPE'='LOCAL_SORT')
+ """.stripMargin)
+ val exception_duplicate_column: Exception = intercept[Exception] {
--- End diff --
Can you change Exception to a more specific exception
---
[GitHub] carbondata issue #2180: [CARBONDATA-2356] Added UT Scenarios for LuceneDataM...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2180
Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/4268/
---
[GitHub] carbondata issue #2180: [CARBONDATA-2356] Added UT Scenarios for LuceneDataM...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2180
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/5235/
---
[GitHub] carbondata pull request #2180: [CARBONDATA-2356] Added UT Scenarios for Luce...
Posted by jackylk <gi...@git.apache.org>.
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2180#discussion_r183211362
--- Diff: datamap/lucene/src/main/java/org/apache/carbondata/datamap/lucene/LuceneFineGrainDataMap.java ---
@@ -157,6 +162,14 @@ private String getQueryString(Expression expression) {
return null;
}
+ //maximum records to return
+ if (strQuery.contains(",") && strQuery.indexOf(",") != strQuery.length() - 1) {
+ max_doc = Integer.parseInt(strQuery.substring(strQuery.lastIndexOf(",") + 1).trim());
--- End diff --
parseInt may give NumberFormatException, please handle it
---
[GitHub] carbondata issue #2180: [CARBONDATA-2356] Added UT Scenarios for LuceneDataM...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2180
Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/4038/
---
[GitHub] carbondata issue #2180: [CARBONDATA-2356] Added UT & SDV Testcases for Lucen...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2180
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/5125/
---
[GitHub] carbondata issue #2180: [CARBONDATA-2356] Added UT Scenarios for LuceneDataM...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2180
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/5245/
---
[GitHub] carbondata pull request #2180: [CARBONDATA-2356] Added UT Scenarios for Luce...
Posted by Indhumathi27 <gi...@git.apache.org>.
Github user Indhumathi27 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2180#discussion_r183212273
--- Diff: datamap/lucene/src/main/java/org/apache/carbondata/datamap/lucene/LuceneFineGrainDataMap.java ---
@@ -157,6 +162,14 @@ private String getQueryString(Expression expression) {
return null;
}
+ //maximum records to return
+ if (strQuery.contains(",") && strQuery.indexOf(",") != strQuery.length() - 1) {
+ max_doc = Integer.parseInt(strQuery.substring(strQuery.lastIndexOf(",") + 1).trim());
+ strQuery = strQuery.substring(0, strQuery.indexOf(","));
+ } else if (strQuery.contains(",") && strQuery.indexOf(",") == strQuery.length() - 1) {
+ strQuery = strQuery.substring(0, strQuery.indexOf(","));
--- End diff --
If user doesnot specify the value in TEXT_MATCH UDF, like TEXT_MATCH("name:a", ) , default value should be Integer.MAX_VALUE
---
[GitHub] carbondata pull request #2180: [CARBONDATA-2356] Added UT Scenarios for Luce...
Posted by jackylk <gi...@git.apache.org>.
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2180#discussion_r183211383
--- Diff: datamap/lucene/src/main/java/org/apache/carbondata/datamap/lucene/LuceneFineGrainDataMap.java ---
@@ -157,6 +162,14 @@ private String getQueryString(Expression expression) {
return null;
}
+ //maximum records to return
+ if (strQuery.contains(",") && strQuery.indexOf(",") != strQuery.length() - 1) {
--- End diff --
What should be the correct format in the strQuery?
Please add description for it.
---
[GitHub] carbondata pull request #2180: [CARBONDATA-2356] Added UT Scenarios for Luce...
Posted by Indhumathi27 <gi...@git.apache.org>.
Github user Indhumathi27 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2180#discussion_r184353464
--- Diff: core/src/main/java/org/apache/carbondata/core/scan/expression/ColumnExpression.java ---
@@ -132,5 +132,8 @@ public void setCarbonColumn(CarbonColumn carbonColumn) {
public void findAndSetChild(Expression oldExpr, Expression newExpr) {
}
+ @Override public int getMaxDoc() {
--- End diff --
removed
---
[GitHub] carbondata issue #2180: [CARBONDATA-2356] Added UT Scenarios for LuceneDataM...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2180
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/5402/
---
[GitHub] carbondata pull request #2180: [CARBONDATA-2356] Added UT Scenarios for Luce...
Posted by jackylk <gi...@git.apache.org>.
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2180#discussion_r183083335
--- Diff: datamap/lucene/src/main/java/org/apache/carbondata/datamap/lucene/LuceneFineGrainDataMap.java ---
@@ -182,7 +182,7 @@ private String getQueryString(Expression expression) {
// execute index search
TopDocs result;
try {
- result = indexSearcher.search(query, indexReader.maxDoc());
+ result = indexSearcher.search(query, Integer.MAX_VALUE);
--- End diff --
I think it is better to pass this variable from the TEXT_MATCH UDF, like TEXT_MATCH("name:a", 100), so that user can decide the value. If user does not give the value, default value can be Integer.MAX_VALUE
---
[GitHub] carbondata issue #2180: [CARBONDATA-2356] Added UT Scenarios for LuceneDataM...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/2180
SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/4566/
---
[GitHub] carbondata issue #2180: [CARBONDATA-2356] Added UT Scenarios for LuceneDataM...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2180
Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/4113/
---
[GitHub] carbondata pull request #2180: [CARBONDATA-2356] Added UT Scenarios for Luce...
Posted by jackylk <gi...@git.apache.org>.
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2180#discussion_r183211232
--- Diff: integration/spark-common-test/src/test/scala/org/apache/carbondata/datamap/lucene/LuceneFineGrainDataMapSuite.scala ---
@@ -197,13 +197,353 @@ class LuceneFineGrainDataMapSuite extends QueryTest with BeforeAndAfterAll {
sql("DROP TABLE IF EXISTS datamap_test3")
}
+ test("test lucene fine grain data map for create datamap with Duplicate Columns") {
+ sql("DROP TABLE IF EXISTS datamap_test_table")
+ sql(
+ """
+ | CREATE TABLE datamap_test_table(id INT, name STRING, city STRING, age INT)
+ | STORED BY 'carbondata'
+ | TBLPROPERTIES('SORT_COLUMNS'='city,name', 'SORT_SCOPE'='LOCAL_SORT')
+ """.stripMargin)
+ val exception_duplicate_column: Exception = intercept[MalformedDataMapCommandException] {
+ sql(
+ s"""
+ | CREATE DATAMAP dm ON TABLE datamap_test_table
+ | USING 'lucene'
+ | DMProperties('TEXT_COLUMNS'='name')
+ """.stripMargin)
+ sql(
+ s"""
+ | CREATE DATAMAP dm1 ON TABLE datamap_test_table
+ | USING 'lucene'
+ | DMProperties('TEXT_COLUMNS'='name')
+ """.stripMargin)
+ }
+ assert(exception_duplicate_column.getMessage
+ .contains("Create lucene datamap dm1 failed, datamap already exists on column(s) name"))
+ sql("drop datamap if exists dm on table datamap_test_table")
+ }
+
+ test("test lucene fine grain data map with wildcard matching ") {
+ sql("DROP TABLE IF EXISTS datamap_test_table")
+ sql(
+ """
+ | CREATE TABLE datamap_test_table(id INT, name STRING, city STRING, age INT)
+ | STORED BY 'carbondata'
+ | TBLPROPERTIES('SORT_COLUMNS'='city,name', 'SORT_SCOPE'='LOCAL_SORT')
+ """.stripMargin)
+ sql(
+ s"""
+ | CREATE DATAMAP dm ON TABLE datamap_test_table
+ | USING 'lucene'
+ | DMProperties('TEXT_COLUMNS'='Name , cIty')
--- End diff --
change to `name, city`, non-capital
---
[GitHub] carbondata pull request #2180: [CARBONDATA-2356] Added UT Scenarios for Luce...
Posted by jackylk <gi...@git.apache.org>.
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2180#discussion_r184357465
--- Diff: integration/spark2/src/main/scala/org/apache/spark/sql/optimizer/CarbonFilters.scala ---
@@ -141,7 +141,9 @@ object CarbonFilters {
case FalseExpr() =>
Some(new FalseExpression(null))
case TextMatch(queryString) =>
- Some(new MatchExpression(queryString))
+ Some(new MatchExpression(queryString, null))
+ case TextMatchDoc(queryString, maxDoc) =>
--- End diff --
please rename to TextMatchLimit
---
[GitHub] carbondata pull request #2180: [CARBONDATA-2356] Added UT Scenarios for Luce...
Posted by jackylk <gi...@git.apache.org>.
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2180#discussion_r184358059
--- Diff: integration/spark2/src/main/scala/org/apache/spark/sql/execution/strategy/CarbonLateDecodeStrategy.scala ---
@@ -536,6 +536,13 @@ private[sql] class CarbonLateDecodeStrategy extends SparkStrategy {
}
Some(TextMatch(u.children.head.toString()))
+ case u: ScalaUDF if u.function.isInstanceOf[TextMatchMaxDocUDF] =>
+ if (u.children.size > 2) {
+ throw new MalformedCarbonCommandException(
+ "TEXT_MATCH UDF syntax: TEXT_MATCH('luceneQuerySyntax')")
+ }
+ Some(TextMatchDoc(u.children.head.toString(), u.children.last.toString()))
--- End diff --
rename to TextMatchLimit
---
[GitHub] carbondata pull request #2180: [CARBONDATA-2356] Added UT Scenarios for Luce...
Posted by jackylk <gi...@git.apache.org>.
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2180#discussion_r183211315
--- Diff: datamap/lucene/src/main/java/org/apache/carbondata/datamap/lucene/LuceneFineGrainDataMap.java ---
@@ -85,6 +85,11 @@
this.analyzer = analyzer;
}
+ /**
+ * default maximum records to return
+ */
+ private int max_doc = Integer.MAX_VALUE;
--- End diff --
It seems there is no testcase for this parameter when user creating datamap, do I miss anything?
---
[GitHub] carbondata issue #2180: [CARBONDATA-2356] Added UT Scenarios for LuceneDataM...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2180
Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/4275/
---
[GitHub] carbondata issue #2180: [CARBONDATA-2356] Added UT Scenarios for LuceneDataM...
Posted by Indhumathi27 <gi...@git.apache.org>.
Github user Indhumathi27 commented on the issue:
https://github.com/apache/carbondata/pull/2180
Retest this please
---
[GitHub] carbondata pull request #2180: [CARBONDATA-2356] Added UT Scenarios for Luce...
Posted by jackylk <gi...@git.apache.org>.
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2180#discussion_r183211275
--- Diff: datamap/lucene/src/main/java/org/apache/carbondata/datamap/lucene/LuceneFineGrainDataMap.java ---
@@ -85,6 +85,11 @@
this.analyzer = analyzer;
}
+ /**
+ * default maximum records to return
+ */
+ private int max_doc = Integer.MAX_VALUE;
--- End diff --
I think a local variable is enough, not required to make it class member
---
[GitHub] carbondata issue #2180: [CARBONDATA-2356] Added UT Scenarios for LuceneDataM...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2180
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/5449/
---
[GitHub] carbondata issue #2180: [CARBONDATA-2356] Added UT Scenarios for LuceneDataM...
Posted by Indhumathi27 <gi...@git.apache.org>.
Github user Indhumathi27 commented on the issue:
https://github.com/apache/carbondata/pull/2180
Retest this please
---
[GitHub] carbondata issue #2180: [CARBONDATA-2356] Added UT Scenarios for LuceneDataM...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2180
Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/4235/
---
[GitHub] carbondata issue #2180: [CARBONDATA-2356] Added UT Scenarios for LuceneDataM...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2180
Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/3935/
---
[GitHub] carbondata issue #2180: [CARBONDATA-2356] Added UT Scenarios for LuceneDataM...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2180
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/5399/
---
[GitHub] carbondata pull request #2180: [CARBONDATA-2356] Added UT Scenarios for Luce...
Posted by akashrn5 <gi...@git.apache.org>.
Github user akashrn5 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2180#discussion_r183205830
--- Diff: datamap/lucene/src/main/java/org/apache/carbondata/datamap/lucene/LuceneFineGrainDataMap.java ---
@@ -182,7 +182,7 @@ private String getQueryString(Expression expression) {
// execute index search
TopDocs result;
try {
- result = indexSearcher.search(query, indexReader.maxDoc());
+ result = indexSearcher.search(query, Integer.MAX_VALUE);
--- End diff --
@Indhumathi27 i think what jacky means is, when user fires sql like "select * from table wjere text_match('name:a',100)", he should fire with number of records in result from lucene search, if he doesnot give in sql, then we will pass integer max , so please handle according to that in udf
---
[GitHub] carbondata pull request #2180: [CARBONDATA-2356] Added UT Scenarios for Luce...
Posted by Indhumathi27 <gi...@git.apache.org>.
Github user Indhumathi27 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2180#discussion_r183212201
--- Diff: datamap/lucene/src/main/java/org/apache/carbondata/datamap/lucene/LuceneFineGrainDataMap.java ---
@@ -85,6 +85,11 @@
this.analyzer = analyzer;
}
+ /**
+ * default maximum records to return
+ */
+ private int max_doc = Integer.MAX_VALUE;
--- End diff --
Added one testcase for TEXT_MATCH UDF, like TEXT_MATCH("name:a", 10)
---
[GitHub] carbondata issue #2180: [CARBONDATA-2356] Added UT Scenarios for LuceneDataM...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2180
Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/4025/
---
[GitHub] carbondata issue #2180: [CARBONDATA-2356] Added UT Scenarios for LuceneDataM...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2180
Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/4090/
---
[GitHub] carbondata issue #2180: [CARBONDATA-2356] Added UT Scenarios for LuceneDataM...
Posted by jackylk <gi...@git.apache.org>.
Github user jackylk commented on the issue:
https://github.com/apache/carbondata/pull/2180
LGTM
---
[GitHub] carbondata issue #2180: [CARBONDATA-2356] Added UT & SDV Testcases for Lucen...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2180
Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/3927/
---
[GitHub] carbondata issue #2180: [CARBONDATA-2356] Added UT Scenarios for LuceneDataM...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2180
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/5292/
---
[GitHub] carbondata pull request #2180: [CARBONDATA-2356] Added UT Scenarios for Luce...
Posted by jackylk <gi...@git.apache.org>.
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2180#discussion_r183211461
--- Diff: datamap/lucene/src/main/java/org/apache/carbondata/datamap/lucene/LuceneFineGrainDataMap.java ---
@@ -157,6 +162,14 @@ private String getQueryString(Expression expression) {
return null;
}
+ //maximum records to return
+ if (strQuery.contains(",") && strQuery.indexOf(",") != strQuery.length() - 1) {
+ max_doc = Integer.parseInt(strQuery.substring(strQuery.lastIndexOf(",") + 1).trim());
+ strQuery = strQuery.substring(0, strQuery.indexOf(","));
+ } else if (strQuery.contains(",") && strQuery.indexOf(",") == strQuery.length() - 1) {
+ strQuery = strQuery.substring(0, strQuery.indexOf(","));
--- End diff --
I am not sure why this if branch is needed
---
[GitHub] carbondata issue #2180: [CARBONDATA-2356] Added UT Scenarios for LuceneDataM...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/2180
SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/4537/
---
[GitHub] carbondata issue #2180: [CARBONDATA-2356] Added UT Scenarios for LuceneDataM...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2180
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/5220/
---
[GitHub] carbondata issue #2180: [CARBONDATA-2356] Added UT Scenarios for LuceneDataM...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2180
Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/4075/
---
[GitHub] carbondata pull request #2180: [CARBONDATA-2356] Added UT Scenarios for Luce...
Posted by Indhumathi27 <gi...@git.apache.org>.
Github user Indhumathi27 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2180#discussion_r183458964
--- Diff: datamap/lucene/src/main/java/org/apache/carbondata/datamap/lucene/LuceneFineGrainDataMap.java ---
@@ -182,7 +182,7 @@ private String getQueryString(Expression expression) {
// execute index search
TopDocs result;
try {
- result = indexSearcher.search(query, indexReader.maxDoc());
+ result = indexSearcher.search(query, Integer.MAX_VALUE);
--- End diff --
@jackylk Tired to implement Text_Match UDF with variable number of arguments(either 1 0r 2).but it doesnt work.Only either one works
---
[GitHub] carbondata issue #2180: [CARBONDATA-2356] Added UT Scenarios for LuceneDataM...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/2180
SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/4581/
---
[GitHub] carbondata issue #2180: [CARBONDATA-2356] Added UT Scenarios for LuceneDataM...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2180
Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/4105/
---
[GitHub] carbondata issue #2180: [CARBONDATA-2356] Added UT Scenarios for LuceneDataM...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2180
Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/4112/
---
[GitHub] carbondata issue #2180: [CARBONDATA-2356] Added UT Scenarios for LuceneDataM...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2180
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/5151/
---
[GitHub] carbondata issue #2180: [CARBONDATA-2356] Added UT Scenarios for LuceneDataM...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2180
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/5265/
---
[GitHub] carbondata issue #2180: [CARBONDATA-2356] Added UT Scenarios for LuceneDataM...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2180
Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/4231/
---
[GitHub] carbondata issue #2180: [CARBONDATA-2356] Added UT Scenarios for LuceneDataM...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2180
Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/4084/
---
[GitHub] carbondata issue #2180: [CARBONDATA-2356] Added UT Scenarios for LuceneDataM...
Posted by Indhumathi27 <gi...@git.apache.org>.
Github user Indhumathi27 commented on the issue:
https://github.com/apache/carbondata/pull/2180
Retest this please
---
[GitHub] carbondata issue #2180: [CARBONDATA-2356] Added UT & SDV Testcases for Lucen...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2180
Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/3904/
---
[GitHub] carbondata issue #2180: [CARBONDATA-2356] Added UT Scenarios for LuceneDataM...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2180
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/5285/
---
[GitHub] carbondata issue #2180: [CARBONDATA-2356] Added UT Scenarios for LuceneDataM...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2180
Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/4063/
---
[GitHub] carbondata issue #2180: [CARBONDATA-2356] Added UT Scenarios for LuceneDataM...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/2180
SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/4541/
---
[GitHub] carbondata issue #2180: [CARBONDATA-2356] Added UT & SDV Testcases for Lucen...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2180
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/5149/
---
[GitHub] carbondata issue #2180: [CARBONDATA-2356] Added UT Scenarios for LuceneDataM...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/2180
SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/4562/
---
[GitHub] carbondata pull request #2180: [CARBONDATA-2356] Added UT Scenarios for Luce...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2180#discussion_r184322994
--- Diff: core/src/main/java/org/apache/carbondata/core/scan/expression/ColumnExpression.java ---
@@ -132,5 +132,8 @@ public void setCarbonColumn(CarbonColumn carbonColumn) {
public void findAndSetChild(Expression oldExpr, Expression newExpr) {
}
+ @Override public int getMaxDoc() {
--- End diff --
Don't add this method here, it is very specific to lucene. So only add this in that lucene expression.
---
[GitHub] carbondata issue #2180: [CARBONDATA-2356] Added UT Scenarios for LuceneDataM...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2180
Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/4108/
---
[GitHub] carbondata issue #2180: [CARBONDATA-2356] Added UT Scenarios for LuceneDataM...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2180
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/5288/
---
[GitHub] carbondata issue #2180: [CARBONDATA-2356] Added UT Scenarios for LuceneDataM...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2180
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/5293/
---
[GitHub] carbondata pull request #2180: [CARBONDATA-2356] Added UT Scenarios for Luce...
Posted by jackylk <gi...@git.apache.org>.
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2180#discussion_r183211277
--- Diff: datamap/lucene/src/main/java/org/apache/carbondata/datamap/lucene/LuceneFineGrainDataMap.java ---
@@ -85,6 +85,11 @@
this.analyzer = analyzer;
}
+ /**
+ * default maximum records to return
+ */
+ private int max_doc = Integer.MAX_VALUE;
--- End diff --
And change to `maxDoc`
---
[GitHub] carbondata issue #2180: [CARBONDATA-2356] Added UT Scenarios for LuceneDataM...
Posted by Indhumathi27 <gi...@git.apache.org>.
Github user Indhumathi27 commented on the issue:
https://github.com/apache/carbondata/pull/2180
Retest this please
---
[GitHub] carbondata pull request #2180: [CARBONDATA-2356] Added UT Scenarios for Luce...
Posted by Indhumathi27 <gi...@git.apache.org>.
Github user Indhumathi27 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2180#discussion_r183212203
--- Diff: datamap/lucene/src/main/java/org/apache/carbondata/datamap/lucene/LuceneFineGrainDataMap.java ---
@@ -157,6 +162,14 @@ private String getQueryString(Expression expression) {
return null;
}
+ //maximum records to return
+ if (strQuery.contains(",") && strQuery.indexOf(",") != strQuery.length() - 1) {
--- End diff --
okay
---
[GitHub] carbondata pull request #2180: [CARBONDATA-2356] Added UT Scenarios for Luce...
Posted by jackylk <gi...@git.apache.org>.
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2180#discussion_r184357724
--- Diff: integration/spark2/src/main/scala/org/apache/spark/sql/execution/strategy/CarbonLateDecodeStrategy.scala ---
@@ -536,6 +536,13 @@ private[sql] class CarbonLateDecodeStrategy extends SparkStrategy {
}
Some(TextMatch(u.children.head.toString()))
+ case u: ScalaUDF if u.function.isInstanceOf[TextMatchMaxDocUDF] =>
+ if (u.children.size > 2) {
+ throw new MalformedCarbonCommandException(
+ "TEXT_MATCH UDF syntax: TEXT_MATCH('luceneQuerySyntax')")
--- End diff --
This message is not correct, should be TEXT_MATCH_WITH_LIMIT
---
[GitHub] carbondata issue #2180: [CARBONDATA-2356] Added UT Scenarios for LuceneDataM...
Posted by Indhumathi27 <gi...@git.apache.org>.
Github user Indhumathi27 commented on the issue:
https://github.com/apache/carbondata/pull/2180
Retest this please
---
[GitHub] carbondata pull request #2180: [CARBONDATA-2356] Added UT Scenarios for Luce...
Posted by Indhumathi27 <gi...@git.apache.org>.
Github user Indhumathi27 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2180#discussion_r184057789
--- Diff: datamap/lucene/src/main/java/org/apache/carbondata/datamap/lucene/LuceneFineGrainDataMap.java ---
@@ -182,7 +182,7 @@ private String getQueryString(Expression expression) {
// execute index search
TopDocs result;
try {
- result = indexSearcher.search(query, indexReader.maxDoc());
+ result = indexSearcher.search(query, Integer.MAX_VALUE);
--- End diff --
@jackylk Implemented new function TEXT_MATCH_WITH_LIMIT to support user to fire query with number of records.TEXT_MATCH will return Integer.MaxValue number of records.
---
[GitHub] carbondata pull request #2180: [CARBONDATA-2356] Added UT Scenarios for Luce...
Posted by Indhumathi27 <gi...@git.apache.org>.
Github user Indhumathi27 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2180#discussion_r183140665
--- Diff: integration/spark-common-test/src/test/scala/org/apache/carbondata/datamap/lucene/LuceneFineGrainDataMapSuite.scala ---
@@ -197,13 +197,334 @@ class LuceneFineGrainDataMapSuite extends QueryTest with BeforeAndAfterAll {
sql("DROP TABLE IF EXISTS datamap_test3")
}
+ test("test lucene fine grain data map for create datamap with Duplicate Columns") {
+ sql("DROP TABLE IF EXISTS datamap_test_table")
+ sql(
+ """
+ | CREATE TABLE datamap_test_table(id INT, name STRING, city STRING, age INT)
+ | STORED BY 'carbondata'
+ | TBLPROPERTIES('SORT_COLUMNS'='city,name', 'SORT_SCOPE'='LOCAL_SORT')
+ """.stripMargin)
+ val exception_duplicate_column: Exception = intercept[Exception] {
--- End diff --
Changed Exception to a more specific exception.Please check
---