You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@carbondata.apache.org by akashrn5 <gi...@git.apache.org> on 2018/03/29 14:52:23 UTC
[GitHub] carbondata pull request #2113: [WIP][LUCENE_DATAMAP]load issue in lucene dat...
GitHub user akashrn5 opened a pull request:
https://github.com/apache/carbondata/pull/2113
[WIP][LUCENE_DATAMAP]load issue in lucene datamap, make multiple directory based on taskId
Be sure to do all of the following checklist to help us incorporate
your contribution quickly and easily:
- [ ] Any interfaces changed?
- [ ] Any backward compatibility impacted?
- [ ] Document update required?
- [ ] Testing done
Please provide details on
- Whether new unit test cases have been added or why no new tests are required?
- How it is tested? Please attach test report.
- Is it a performance related change? Please attach the performance test report.
- Any additional information to help reviewers in testing this change.
- [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/akashrn5/incubator-carbondata lucene
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/carbondata/pull/2113.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #2113
----
commit 4453e9218d2997923fa1ab3d60e4cac74cc14dba
Author: akashrn5 <ak...@...>
Date: 2018-03-29T14:29:36Z
load issue in lucene datamap, make multiple directory based on taskId
----
---
[GitHub] carbondata issue #2113: [CARBONDATA-2347][LUCENE_DATAMAP]load issue in lucen...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2113
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/5105/
---
[GitHub] carbondata issue #2113: [CARBONDATA-2347][LUCENE_DATAMAP]load issue in lucen...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2113
Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/3887/
---
[GitHub] carbondata pull request #2113: [CARBONDATA-2347][LUCENE_DATAMAP]load issue i...
Posted by jackylk <gi...@git.apache.org>.
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2113#discussion_r181394048
--- Diff: core/src/main/java/org/apache/carbondata/core/constants/CarbonCommonConstants.java ---
@@ -1642,6 +1642,16 @@
public static final String CARBON_SEARCH_MODE_THREAD_DEFAULT = "3";
+ /**
+ * compression mode used by lucene for index writing
+ */
+ public static final String CARBON_LUCENE_COMPRESSION_MODE = "carbon.lucene.compression.mode";
--- End diff --
what are the options available for this property?
---
[GitHub] carbondata issue #2113: [WIP][LUCENE_DATAMAP]load issue in lucene datamap, m...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2113
Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/3706/
---
[GitHub] carbondata pull request #2113: [CARBONDATA-2347][LUCENE_DATAMAP]load issue i...
Posted by akashrn5 <gi...@git.apache.org>.
Github user akashrn5 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2113#discussion_r181437595
--- Diff: core/src/main/java/org/apache/carbondata/core/constants/CarbonCommonConstants.java ---
@@ -1642,6 +1642,16 @@
public static final String CARBON_SEARCH_MODE_THREAD_DEFAULT = "3";
+ /**
+ * compression mode used by lucene for index writing
+ */
+ public static final String CARBON_LUCENE_COMPRESSION_MODE = "carbon.lucene.compression.mode";
--- End diff --
SPEED and COMPRESSION, by default the property value will be SPEED
---
[GitHub] carbondata issue #2113: [WIP][LUCENE_DATAMAP]load issue in lucene datamap, m...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/2113
SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/4434/
---
[GitHub] carbondata pull request #2113: [CARBONDATA-2347][LUCENE_DATAMAP]load issue i...
Posted by akashrn5 <gi...@git.apache.org>.
Github user akashrn5 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2113#discussion_r181438240
--- Diff: datamap/lucene/pom.xml ---
@@ -141,6 +141,34 @@
</execution>
</executions>
</plugin>
+ <plugin>
--- End diff --
this was added to include test suite in main CI
---
[GitHub] carbondata issue #2113: [CARBONDATA-2347][LUCENE_DATAMAP]load issue in lucen...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2113
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/5057/
---
[GitHub] carbondata issue #2113: [WIP][LUCENE_DATAMAP]load issue in lucene datamap, m...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2113
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/4976/
---
[GitHub] carbondata pull request #2113: [CARBONDATA-2347][LUCENE_DATAMAP]load issue i...
Posted by jackylk <gi...@git.apache.org>.
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2113#discussion_r181660289
--- Diff: core/src/main/java/org/apache/carbondata/core/metadata/schema/table/CarbonTable.java ---
@@ -856,6 +858,25 @@ public boolean hasAggregationDataMap() {
return false;
}
+ /**
+ * whether this table has Lucene DataMap or not
+ */
+ public boolean hasLuceneDataMap() {
--- End diff --
remove this now, we should add blocking feature in other PR
---
[GitHub] carbondata pull request #2113: [CARBONDATA-2347][LUCENE_DATAMAP]load issue i...
Posted by akashrn5 <gi...@git.apache.org>.
Github user akashrn5 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2113#discussion_r181439768
--- Diff: integration/spark2/src/main/scala/org/apache/spark/sql/CarbonEnv.scala ---
@@ -173,6 +174,10 @@ object CarbonEnv {
.addListener(classOf[AlterTableDropPartitionPostStatusEvent],
AlterTableDropPartitionPostStatusListener)
.addListener(classOf[AlterTableDropPartitionMetaEvent], AlterTableDropPartitionMetaListener)
+ .addListener(classOf[AlterTableRenamePreEvent], LuceneRenameTablePreListener)
--- End diff --
this listener class is added to block alter operation on lucene datamap, if we are blocking alter operation for all the datamaps, then this may not be required.
---
[GitHub] carbondata pull request #2113: [CARBONDATA-2347][LUCENE_DATAMAP]load issue i...
Posted by jackylk <gi...@git.apache.org>.
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2113#discussion_r181399425
--- Diff: integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/datamap/CarbonCreateDataMapCommand.scala ---
@@ -69,11 +69,33 @@ case class CarbonCreateDataMapCommand(
}
dataMapSchema = new DataMapSchema(dataMapName, dmClassName)
- if (mainTable != null &&
- mainTable.isStreamingTable &&
- !(dataMapSchema.getProviderName.equalsIgnoreCase(DataMapClassProvider.PREAGGREGATE.toString)
- || dataMapSchema.getProviderName
- .equalsIgnoreCase(DataMapClassProvider.TIMESERIES.toString))) {
+ if (dataMapSchema.getProviderName.equalsIgnoreCase(DataMapClassProvider.LUCENEFG.toString) ||
--- End diff --
I think we should abstract interface for it. We can not add if check for every new datamap added
---
[GitHub] carbondata issue #2113: [WIP][LUCENE_DATAMAP]load issue in lucene datamap, m...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/2113
SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/4393/
---
[GitHub] carbondata issue #2113: [WIP][LUCENE_DATAMAP]load issue in lucene datamap, m...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/2113
SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/4315/
---
[GitHub] carbondata issue #2113: [CARBONDATA-2347][LUCENE_DATAMAP]load issue in lucen...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2113
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/5113/
---
[GitHub] carbondata pull request #2113: [CARBONDATA-2347][LUCENE_DATAMAP]load issue i...
Posted by jackylk <gi...@git.apache.org>.
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2113#discussion_r181713300
--- Diff: core/src/main/java/org/apache/carbondata/core/metadata/schema/table/DiskBasedDMSchemaStorageProvider.java ---
@@ -129,9 +130,11 @@ public DiskBasedDMSchemaStorageProvider(String storePath) {
return dataMapSchemas;
}
- @Override public void dropSchema(String dataMapName) throws IOException {
- String schemaPath =
- storePath + CarbonCommonConstants.FILE_SEPARATOR + dataMapName + ".dmschema";
+ @Override public void dropSchema(String dataMapName, String tableName, String dataMapProviderName)
+ throws IOException {
+ String schemaPath = storePath + CarbonCommonConstants.FILE_SEPARATOR + tableName
--- End diff --
make an utility function and use it in all place including testcase
---
[GitHub] carbondata issue #2113: [CARBONDATA-2347][LUCENE_DATAMAP]load issue in lucen...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2113
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/5073/
---
[GitHub] carbondata issue #2113: [CARBONDATA-2347][LUCENE_DATAMAP]load issue in lucen...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2113
Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/3836/
---
[GitHub] carbondata issue #2113: [WIP][LUCENE_DATAMAP]load issue in lucene datamap, m...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2113
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/4838/
---
[GitHub] carbondata issue #2113: [WIP][LUCENE_DATAMAP]load issue in lucene datamap, m...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2113
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/4932/
---
[GitHub] carbondata pull request #2113: [CARBONDATA-2347][LUCENE_DATAMAP]load issue i...
Posted by jackylk <gi...@git.apache.org>.
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2113#discussion_r181660484
--- Diff: core/src/main/java/org/apache/carbondata/core/metadata/schema/table/DataMapSchema.java ---
@@ -143,11 +142,12 @@ public void setChildSchema(TableSchema childSchema) {
/**
* Return true if this datamap is an Index DataMap
+ *
* @return
*/
public boolean isIndexDataMap() {
- if (providerName.equalsIgnoreCase(DataMapClassProvider.PREAGGREGATE.getShortName()) ||
- providerName.equalsIgnoreCase(DataMapClassProvider.TIMESERIES.getShortName())) {
+ if (providerName.equalsIgnoreCase(DataMapClassProvider.PREAGGREGATE.getShortName())
+ || providerName.equalsIgnoreCase(DataMapClassProvider.TIMESERIES.getShortName())) {
--- End diff --
no need to modify
---
[GitHub] carbondata pull request #2113: [CARBONDATA-2347][LUCENE_DATAMAP]load issue i...
Posted by jackylk <gi...@git.apache.org>.
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2113#discussion_r181713494
--- Diff: datamap/lucene/src/main/java/org/apache/carbondata/datamap/lucene/LuceneDataMapFactoryBase.java ---
@@ -205,9 +234,12 @@ public void clear(Segment segment) {
/**
* Clear all datamaps from memory
*/
- @Override
- public void clear() {
-
+ @Override public void clear() {
--- End diff --
move @override to previous line, please follow this in future
---
[GitHub] carbondata issue #2113: [WIP][LUCENE_DATAMAP]load issue in lucene datamap, m...
Posted by akashrn5 <gi...@git.apache.org>.
Github user akashrn5 commented on the issue:
https://github.com/apache/carbondata/pull/2113
retest this please
---
[GitHub] carbondata issue #2113: [CARBONDATA-2347][LUCENE_DATAMAP]load issue in lucen...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2113
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/5056/
---
[GitHub] carbondata issue #2113: [WIP][LUCENE_DATAMAP]load issue in lucene datamap, m...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/2113
SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/4436/
---
[GitHub] carbondata issue #2113: [WIP][LUCENE_DATAMAP]load issue in lucene datamap, m...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/2113
SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/4386/
---
[GitHub] carbondata issue #2113: [WIP][LUCENE_DATAMAP]load issue in lucene datamap, m...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/2113
SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/4299/
---
[GitHub] carbondata issue #2113: [WIP][LUCENE_DATAMAP]load issue in lucene datamap, m...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2113
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/4794/
---
[GitHub] carbondata issue #2113: [CARBONDATA-2347][LUCENE_DATAMAP]load issue in lucen...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2113
Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/3883/
---
[GitHub] carbondata issue #2113: [WIP][LUCENE_DATAMAP]load issue in lucene datamap, m...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2113
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/4967/
---
[GitHub] carbondata issue #2113: [CARBONDATA-2347][LUCENE_DATAMAP]load issue in lucen...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2113
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/5095/
---
[GitHub] carbondata pull request #2113: [CARBONDATA-2347][LUCENE_DATAMAP]load issue i...
Posted by jackylk <gi...@git.apache.org>.
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2113#discussion_r181658482
--- Diff: core/src/main/java/org/apache/carbondata/core/constants/CarbonCommonConstants.java ---
@@ -1642,6 +1642,16 @@
public static final String CARBON_SEARCH_MODE_THREAD_DEFAULT = "3";
+ /**
+ * compression mode used by lucene for index writing
+ */
+ public static final String CARBON_LUCENE_COMPRESSION_MODE = "carbon.lucene.compression.mode";
--- End diff --
mention it in the description in line 1646
---
[GitHub] carbondata pull request #2113: [CARBONDATA-2347][LUCENE_DATAMAP]load issue i...
Posted by jackylk <gi...@git.apache.org>.
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2113#discussion_r181659113
--- Diff: core/src/main/java/org/apache/carbondata/core/metadata/SegmentFileStore.java ---
@@ -681,10 +681,10 @@ private static void deletePhysicalPartition(List<PartitionSpec> partitionSpecs,
FileFactory.deleteAllCarbonFilesOfDir(FileFactory.getCarbonFile(location.toString()));
}
} else {
- // delete the segment folder if it is empty
+ // delete the segment folder
CarbonFile file = FileFactory.getCarbonFile(location.toString());
--- End diff --
change `file` to `segmentPath`
---
[GitHub] carbondata pull request #2113: [CARBONDATA-2347][LUCENE_DATAMAP]load issue i...
Posted by jackylk <gi...@git.apache.org>.
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2113#discussion_r181663757
--- Diff: integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/datamap/CarbonDataMapShowCommand.scala ---
@@ -40,35 +42,48 @@ case class CarbonDataMapShowCommand(tableIdentifier: Option[TableIdentifier])
override def output: Seq[Attribute] = {
Seq(AttributeReference("DataMapName", StringType, nullable = false)(),
AttributeReference("ClassName", StringType, nullable = false)(),
- AttributeReference("Associated Table", StringType, nullable = false)())
+ AttributeReference("Associated Table", StringType, nullable = false)(),
+ AttributeReference("DMProperties", StringType, nullable = false)())
}
override def processData(sparkSession: SparkSession): Seq[Row] = {
+ val finalSchemaList: util.List[DataMapSchema] = new util.ArrayList[DataMapSchema]()
--- End diff --
rename to datamapSchemaList
---
[GitHub] carbondata issue #2113: [WIP][LUCENE_DATAMAP]load issue in lucene datamap, m...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2113
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/4994/
---
[GitHub] carbondata issue #2113: [WIP][LUCENE_DATAMAP]load issue in lucene datamap, m...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2113
Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/3615/
---
[GitHub] carbondata pull request #2113: [CARBONDATA-2347][LUCENE_DATAMAP]load issue i...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2113#discussion_r181727647
--- Diff: integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/datamap/CarbonCreateDataMapCommand.scala ---
@@ -69,8 +69,29 @@ case class CarbonCreateDataMapCommand(
}
dataMapSchema = new DataMapSchema(dataMapName, dmClassName)
- if (mainTable != null &&
- mainTable.isStreamingTable &&
+ if (dataMapSchema.getProviderName.equalsIgnoreCase(DataMapClassProvider.LUCENE.toString)) {
--- End diff --
This all validation code should be inside the luceneDatamapFactory init method. Please don't add any lucene specific code outside of lucene classes
---
[GitHub] carbondata issue #2113: [CARBONDATA-2347][LUCENE_DATAMAP]load issue in lucen...
Posted by jackylk <gi...@git.apache.org>.
Github user jackylk commented on the issue:
https://github.com/apache/carbondata/pull/2113
retest this please
---
[GitHub] carbondata issue #2113: [WIP][LUCENE_DATAMAP]load issue in lucene datamap, m...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2113
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/4667/
---
[GitHub] carbondata pull request #2113: [CARBONDATA-2347][LUCENE_DATAMAP]load issue i...
Posted by jackylk <gi...@git.apache.org>.
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2113#discussion_r181665850
--- Diff: integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/mutation/CarbonProjectForDeleteCommand.scala ---
@@ -17,15 +17,18 @@
package org.apache.spark.sql.execution.command.mutation
+import scala.collection.JavaConverters._
--- End diff --
seems no need to modify this file
---
[GitHub] carbondata issue #2113: [WIP][LUCENE_DATAMAP]load issue in lucene datamap, m...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2113
Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/3571/
---
[GitHub] carbondata issue #2113: [WIP][LUCENE_DATAMAP]load issue in lucene datamap, m...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/2113
SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/4432/
---
[GitHub] carbondata issue #2113: [CARBONDATA-2347][LUCENE_DATAMAP]load issue in lucen...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2113
Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/3851/
---
[GitHub] carbondata pull request #2113: [CARBONDATA-2347][LUCENE_DATAMAP]load issue i...
Posted by jackylk <gi...@git.apache.org>.
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2113#discussion_r181665909
--- Diff: integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/mutation/CarbonProjectForUpdateCommand.scala ---
@@ -17,6 +17,8 @@
package org.apache.spark.sql.execution.command.mutation
+import scala.collection.JavaConverters._
--- End diff --
seems no need to modify this file
---
[GitHub] carbondata issue #2113: [CARBONDATA-2347][LUCENE_DATAMAP]load issue in lucen...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2113
Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/3890/
---
[GitHub] carbondata issue #2113: [CARBONDATA-2347][LUCENE_DATAMAP]load issue in lucen...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2113
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/5089/
---
[GitHub] carbondata issue #2113: [WIP][LUCENE_DATAMAP]load issue in lucene datamap, m...
Posted by akashrn5 <gi...@git.apache.org>.
Github user akashrn5 commented on the issue:
https://github.com/apache/carbondata/pull/2113
retest this please
---
[GitHub] carbondata issue #2113: [WIP][LUCENE_DATAMAP]load issue in lucene datamap, m...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/2113
SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/4296/
---
[GitHub] carbondata issue #2113: [WIP][LUCENE_DATAMAP]load issue in lucene datamap, m...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/2113
SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/4390/
---
[GitHub] carbondata issue #2113: [CARBONDATA-2347][LUCENE_DATAMAP]load issue in lucen...
Posted by jackylk <gi...@git.apache.org>.
Github user jackylk commented on the issue:
https://github.com/apache/carbondata/pull/2113
LGTM
---
[GitHub] carbondata issue #2113: [WIP][LUCENE_DATAMAP]load issue in lucene datamap, m...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2113
Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/3700/
---
[GitHub] carbondata issue #2113: [CARBONDATA-2347][LUCENE_DATAMAP]load issue in lucen...
Posted by akashrn5 <gi...@git.apache.org>.
Github user akashrn5 commented on the issue:
https://github.com/apache/carbondata/pull/2113
retest this please
---
[GitHub] carbondata issue #2113: [CARBONDATA-2347][LUCENE_DATAMAP]load issue in lucen...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2113
Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/3888/
---
[GitHub] carbondata issue #2113: [WIP][LUCENE_DATAMAP]load issue in lucene datamap, m...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2113
Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/3713/
---
[GitHub] carbondata issue #2113: [WIP][LUCENE_DATAMAP]load issue in lucene datamap, m...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2113
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/4963/
---
[GitHub] carbondata issue #2113: [WIP][LUCENE_DATAMAP]load issue in lucene datamap, m...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2113
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/4705/
---
[GitHub] carbondata pull request #2113: [CARBONDATA-2347][LUCENE_DATAMAP]load issue i...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2113#discussion_r181725781
--- Diff: core/src/main/java/org/apache/carbondata/core/datamap/IndexDataMapProvider.java ---
@@ -99,25 +103,24 @@ private DataMapFactory createIndexDataMapFactory(DataMapSchema dataMapSchema)
return dataMapFactory;
}
- private DataMapFactory getDataMapFactoryByShortName(String providerName)
+ public static DataMapFactory getDataMapFactoryByShortName(String providerName)
throws MalformedDataMapCommandException {
+ DataMapRegistry.registerDataMap(DataMapClassProvider.LUCENE.getClassName(),
--- End diff --
What is the need of specifing the LUCENE here, these classes are supposed to be independent on any specific datamap.
---
[GitHub] carbondata issue #2113: [CARBONDATA-2347][LUCENE_DATAMAP]load issue in lucen...
Posted by akashrn5 <gi...@git.apache.org>.
Github user akashrn5 commented on the issue:
https://github.com/apache/carbondata/pull/2113
retest sdv please
---
[GitHub] carbondata issue #2113: [CARBONDATA-2347][LUCENE_DATAMAP]load issue in lucen...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2113
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/5035/
---
[GitHub] carbondata issue #2113: [CARBONDATA-2347][LUCENE_DATAMAP]load issue in lucen...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2113
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/5062/
---
[GitHub] carbondata issue #2113: [CARBONDATA-2347][LUCENE_DATAMAP]load issue in lucen...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2113
Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/3881/
---
[GitHub] carbondata issue #2113: [WIP][LUCENE_DATAMAP]load issue in lucene datamap, m...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2113
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/4796/
---
[GitHub] carbondata issue #2113: [WIP][LUCENE_DATAMAP]load issue in lucene datamap, m...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2113
Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/3551/
---
[GitHub] carbondata issue #2113: [WIP][LUCENE_DATAMAP]load issue in lucene datamap, m...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2113
Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/3772/
---
[GitHub] carbondata issue #2113: [WIP][LUCENE_DATAMAP]load issue in lucene datamap, m...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2113
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/4984/
---
[GitHub] carbondata issue #2113: [WIP][LUCENE_DATAMAP]load issue in lucene datamap, m...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/2113
SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/4298/
---
[GitHub] carbondata pull request #2113: [CARBONDATA-2347][LUCENE_DATAMAP]load issue i...
Posted by jackylk <gi...@git.apache.org>.
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2113#discussion_r181662181
--- Diff: datamap/lucene/src/main/java/org/apache/carbondata/datamap/lucene/LuceneFineGrainDataMapFactory.java ---
@@ -62,7 +62,17 @@
public List<FineGrainDataMap> getDataMaps(DataMapDistributable distributable,
ReadCommittedScope readCommittedScope)
throws IOException {
- return getDataMaps(distributable.getSegment(), readCommittedScope);
+ List<FineGrainDataMap> lstDataMap = new ArrayList<>();
+ FineGrainDataMap dataMap = new LuceneFineGrainDataMap(analyzer);
+ String indexPath = ((LuceneDataMapDistributable) distributable).getIndexPath();
+ try {
+ dataMap.init(new DataMapModel(indexPath));
+ } catch (MemoryException e) {
+ LOGGER.error("failed to get lucene datamap , detail is {}" + e.getMessage());
--- End diff --
you can use String.format() to format the log message
---
[GitHub] carbondata issue #2113: [WIP][LUCENE_DATAMAP]load issue in lucene datamap, m...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2113
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/4926/
---
[GitHub] carbondata pull request #2113: [CARBONDATA-2347][LUCENE_DATAMAP]load issue i...
Posted by jackylk <gi...@git.apache.org>.
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2113#discussion_r181397988
--- Diff: integration/spark2/src/main/scala/org/apache/spark/sql/CarbonEnv.scala ---
@@ -173,6 +174,10 @@ object CarbonEnv {
.addListener(classOf[AlterTableDropPartitionPostStatusEvent],
AlterTableDropPartitionPostStatusListener)
.addListener(classOf[AlterTableDropPartitionMetaEvent], AlterTableDropPartitionMetaListener)
+ .addListener(classOf[AlterTableRenamePreEvent], LuceneRenameTablePreListener)
--- End diff --
Is this required? Ideally, lucene datamap is a separate module which should not have intrusive modification in other modules
---
[GitHub] carbondata pull request #2113: [CARBONDATA-2347][LUCENE_DATAMAP]load issue i...
Posted by jackylk <gi...@git.apache.org>.
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2113#discussion_r181661716
--- Diff: datamap/lucene/pom.xml ---
@@ -141,6 +141,34 @@
</execution>
</executions>
</plugin>
+ <plugin>
--- End diff --
Can you move the testcase to spark2 module and add `<scope>test</scope>` for spark2 in the pom
---
[GitHub] carbondata issue #2113: [CARBONDATA-2347][LUCENE_DATAMAP]load issue in lucen...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2113
Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/3869/
---
[GitHub] carbondata issue #2113: [WIP][LUCENE_DATAMAP]load issue in lucene datamap, m...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/2113
SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/4297/
---
[GitHub] carbondata issue #2113: [WIP][LUCENE_DATAMAP]load issue in lucene datamap, m...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/2113
SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/4179/
---
[GitHub] carbondata issue #2113: [WIP][LUCENE_DATAMAP]load issue in lucene datamap, m...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/2113
SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/4428/
---
[GitHub] carbondata pull request #2113: [CARBONDATA-2347][LUCENE_DATAMAP]load issue i...
Posted by jackylk <gi...@git.apache.org>.
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2113#discussion_r181662903
--- Diff: datamap/lucene/src/test/scala/org/apache/carbondata/datamap/lucene/LuceneFineGrainDataMapSuite.scala ---
@@ -57,7 +65,7 @@ class LuceneFineGrainDataMapSuite extends QueryTest with BeforeAndAfterAll {
var exception = intercept[MalformedDataMapCommandException](sql(
s"""
| CREATE DATAMAP dm1 ON TABLE datamap_test
- | USING 'org.apache.carbondata.datamap.lucene.LuceneFineGrainDataMapFactory'
+ | USING 'lucenefg'
--- End diff --
please use `lucene` only
---
[GitHub] carbondata issue #2113: [WIP][LUCENE_DATAMAP]load issue in lucene datamap, m...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2113
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/4846/
---
[GitHub] carbondata pull request #2113: [CARBONDATA-2347][LUCENE_DATAMAP]load issue i...
Posted by jackylk <gi...@git.apache.org>.
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2113#discussion_r181666147
--- Diff: integration/spark2/src/main/scala/org/apache/spark/sql/execution/strategy/DDLStrategy.scala ---
@@ -244,6 +244,13 @@ class DDLStrategy(sparkSession: SparkSession) extends SparkStrategy {
throw new MalformedCarbonCommandException("Unsupported operation on unmanaged table")
}
+ // TODO remove this limitation after streaming table support 'Lucene' DataMap
+ // if the table has 'Lucene' DataMap, it doesn't support streaming now
+ if (carbonTable.hasLuceneDataMap) {
--- End diff --
no need to do this now, do it later
---
[GitHub] carbondata issue #2113: [CARBONDATA-2347][LUCENE_DATAMAP]load issue in lucen...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2113
Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/3892/
---
[GitHub] carbondata issue #2113: [WIP][LUCENE_DATAMAP]load issue in lucene datamap, m...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/2113
SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/4419/
---
[GitHub] carbondata issue #2113: [WIP][LUCENE_DATAMAP]load issue in lucene datamap, m...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2113
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/4989/
---
[GitHub] carbondata issue #2113: [WIP][LUCENE_DATAMAP]load issue in lucene datamap, m...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/2113
SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/4263/
---
[GitHub] carbondata issue #2113: [CARBONDATA-2347][LUCENE_DATAMAP]load issue in lucen...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2113
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/5078/
---
[GitHub] carbondata issue #2113: [WIP][LUCENE_DATAMAP]load issue in lucene datamap, m...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2113
Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/3705/
---
[GitHub] carbondata issue #2113: [WIP][LUCENE_DATAMAP]load issue in lucene datamap, m...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/2113
SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/4426/
---
[GitHub] carbondata issue #2113: [WIP][LUCENE_DATAMAP]load issue in lucene datamap, m...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/2113
SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/4392/
---
[GitHub] carbondata issue #2113: [WIP][LUCENE_DATAMAP]load issue in lucene datamap, m...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2113
Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/3530/
---
[GitHub] carbondata issue #2113: [CARBONDATA-2347][LUCENE_DATAMAP]load issue in lucen...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2113
Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/3860/
---
[GitHub] carbondata pull request #2113: [CARBONDATA-2347][LUCENE_DATAMAP]load issue i...
Posted by jackylk <gi...@git.apache.org>.
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2113#discussion_r181665755
--- Diff: integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/datamap/CarbonDataMapShowCommand.scala ---
@@ -40,35 +42,48 @@ case class CarbonDataMapShowCommand(tableIdentifier: Option[TableIdentifier])
override def output: Seq[Attribute] = {
Seq(AttributeReference("DataMapName", StringType, nullable = false)(),
AttributeReference("ClassName", StringType, nullable = false)(),
- AttributeReference("Associated Table", StringType, nullable = false)())
+ AttributeReference("Associated Table", StringType, nullable = false)(),
+ AttributeReference("DMProperties", StringType, nullable = false)())
}
override def processData(sparkSession: SparkSession): Seq[Row] = {
+ val finalSchemaList: util.List[DataMapSchema] = new util.ArrayList[DataMapSchema]()
tableIdentifier match {
case Some(table) =>
Checker.validateTableExists(table.database, table.table, sparkSession)
val carbonTable = CarbonEnv.getCarbonTable(table)(sparkSession)
if (carbonTable.hasDataMapSchema) {
- val schemaList = carbonTable.getTableInfo.getDataMapSchemaList
- convertToRow(schemaList)
- } else {
- convertToRow(DataMapStoreManager.getInstance().getAllDataMapSchemas(carbonTable))
+ finalSchemaList.addAll(carbonTable.getTableInfo.getDataMapSchemaList)
+ }
+ val indexSchemas = DataMapStoreManager.getInstance().getAllDataMapSchemas(carbonTable)
+ if (!indexSchemas.isEmpty) {
+ finalSchemaList.addAll(indexSchemas)
}
+ convertToRow(finalSchemaList)
case _ =>
convertToRow(DataMapStoreManager.getInstance().getAllDataMapSchemas)
}
-
}
private def convertToRow(schemaList: util.List[DataMapSchema]) = {
if (schemaList != null && schemaList.size() > 0) {
schemaList.asScala.map { s =>
var table = "(NA)"
val relationIdentifier = s.getRelationIdentifier
- if (relationIdentifier != null) {
+ var dmProperties = "(NA)"
+ val isFGorCGdm =
--- End diff --
For show datamap command, I think just show the shortname and the main table
And we should have desc datamap command, which I think we should add in other PR
---
[GitHub] carbondata issue #2113: [CARBONDATA-2347][LUCENE_DATAMAP]load issue in lucen...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2113
Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/3827/
---
[GitHub] carbondata issue #2113: [WIP][LUCENE_DATAMAP]load issue in lucene datamap, m...
Posted by akashrn5 <gi...@git.apache.org>.
Github user akashrn5 commented on the issue:
https://github.com/apache/carbondata/pull/2113
When the loading is triggered after creation of table and datamap , it failed with lock exception.
Analysis:
The index write location for the lucene is same, and to IndexWriter will take a lock file called write.lock in write location while writing the index files. In carbon loading the writer tasks are launched parallel and those many writers are opened,
Since the write.lock file is acquired by one writer, all other tasks will fail and dataloading will fail.
Solution
so in this PR, the write path can be configured on the fly, with task id and timestamp mapping.
Handling in reader part:
In query side, there was a single indexSearch object which does search on index file present in a single location.
Now with thsi dataload fix, there will be multiple directories which will have index files, and cannot give a single path to search object, as serch object is one.
Probable Solutions:
1. create those many number of search objects, as number of index file directories created in data load. and finally return all the final Blocklets.
2. you can have one searcher in one executor to search the local folder, this part im working how to do
---
[GitHub] carbondata pull request #2113: [CARBONDATA-2347][LUCENE_DATAMAP]load issue i...
Posted by jackylk <gi...@git.apache.org>.
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2113#discussion_r181396888
--- Diff: datamap/lucene/pom.xml ---
@@ -141,6 +141,34 @@
</execution>
</executions>
</plugin>
+ <plugin>
--- End diff --
I realize that in this pom, it should not depend on carbon-spark2, please modify the dependency in this pom
---
[GitHub] carbondata issue #2113: [CARBONDATA-2347][LUCENE_DATAMAP]load issue in lucen...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2113
Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/3819/
---
[GitHub] carbondata pull request #2113: [CARBONDATA-2347][LUCENE_DATAMAP]load issue i...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/carbondata/pull/2113
---
[GitHub] carbondata issue #2113: [CARBONDATA-2347][LUCENE_DATAMAP]load issue in lucen...
Posted by akashrn5 <gi...@git.apache.org>.
Github user akashrn5 commented on the issue:
https://github.com/apache/carbondata/pull/2113
retest this please
---
[GitHub] carbondata issue #2113: [WIP][LUCENE_DATAMAP]load issue in lucene datamap, m...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/2113
SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/4280/
---
[GitHub] carbondata issue #2113: [CARBONDATA-2347][LUCENE_DATAMAP]load issue in lucen...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2113
Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/3840/
---
[GitHub] carbondata issue #2113: [WIP][LUCENE_DATAMAP]load issue in lucene datamap, m...
Posted by akashrn5 <gi...@git.apache.org>.
Github user akashrn5 commented on the issue:
https://github.com/apache/carbondata/pull/2113
retest this please
---
[GitHub] carbondata pull request #2113: [CARBONDATA-2347][LUCENE_DATAMAP]load issue i...
Posted by jackylk <gi...@git.apache.org>.
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2113#discussion_r181663104
--- Diff: integration/spark-common-test/src/test/scala/org/apache/carbondata/spark/testsuite/datamap/CGDataMapTestCase.scala ---
@@ -402,7 +402,7 @@ class CGDataMapTestCase extends QueryTest with BeforeAndAfterAll {
sql(s"create datamap test_cg_datamap1 on table datamap_store_test1 using '${classOf[CGDataMapFactory].getName}' as select id, name from datamap_store_test")
- val loc = CarbonProperties.getInstance().getSystemFolderLocation + "/test_cg_datamap1.dmschema"
+ val loc = CarbonProperties.getInstance().getSystemFolderLocation + "/datamap_store_test1_test_cg_datamap1.dmschema"
--- End diff --
Please make a utility function to construct the dmschema path
---
[GitHub] carbondata issue #2113: [CARBONDATA-2347][LUCENE_DATAMAP]load issue in lucen...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2113
Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/3855/
---
[GitHub] carbondata issue #2113: [WIP][LUCENE_DATAMAP]load issue in lucene datamap, m...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/2113
SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/4391/
---
[GitHub] carbondata pull request #2113: [CARBONDATA-2347][LUCENE_DATAMAP]load issue i...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2113#discussion_r181629595
--- Diff: core/src/main/java/org/apache/carbondata/core/datamap/DataMapStoreManager.java ---
@@ -246,17 +247,24 @@ public TableDataMap getDataMap(CarbonTable table, DataMapSchema dataMapSchema) {
public TableDataMap createAndRegisterDataMap(CarbonTable table,
DataMapSchema dataMapSchema) throws MalformedDataMapCommandException, IOException {
DataMapFactory dataMapFactory;
- try {
- // try to create datamap by reflection to test whether it is a valid DataMapFactory class
- Class<? extends DataMapFactory> factoryClass =
- (Class<? extends DataMapFactory>) Class.forName(dataMapSchema.getProviderName());
- dataMapFactory = factoryClass.newInstance();
- } catch (ClassNotFoundException e) {
- throw new MalformedDataMapCommandException(
- "DataMap '" + dataMapSchema.getProviderName() + "' not found");
- } catch (Throwable e) {
- throw new MetadataProcessException(
- "failed to create DataMap '" + dataMapSchema.getProviderName() + "'", e);
+ if (dataMapSchema.getProviderName()
+ .equalsIgnoreCase(DataMapClassProvider.LUCENEFG.getShortName()) || dataMapSchema
+ .getProviderName().equalsIgnoreCase(DataMapClassProvider.LUCENECG.getShortName())) {
--- End diff --
Why is this special check? why can't it instantiated from old code?
---
[GitHub] carbondata issue #2113: [CARBONDATA-2347][LUCENE_DATAMAP]load issue in lucen...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2113
Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/3866/
---
[GitHub] carbondata issue #2113: [WIP][LUCENE_DATAMAP]load issue in lucene datamap, m...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2113
Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/3478/
---
[GitHub] carbondata pull request #2113: [CARBONDATA-2347][LUCENE_DATAMAP]load issue i...
Posted by jackylk <gi...@git.apache.org>.
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2113#discussion_r181661362
--- Diff: core/src/main/java/org/apache/carbondata/core/metadata/schema/table/DiskBasedDMSchemaStorageProvider.java ---
@@ -129,9 +130,9 @@ public DiskBasedDMSchemaStorageProvider(String storePath) {
return dataMapSchemas;
}
- @Override public void dropSchema(String dataMapName) throws IOException {
- String schemaPath =
- storePath + CarbonCommonConstants.FILE_SEPARATOR + dataMapName + ".dmschema";
+ @Override public void dropSchema(String dataMapName,String tableName) throws IOException {
+ String schemaPath = storePath + CarbonCommonConstants.FILE_SEPARATOR + tableName
+ + CarbonCommonConstants.UNDERSCORE + dataMapName + ".dmschema";
--- End diff --
I think it is better to record the datamap short name in the path also, so that we can know what kind of datamap it is by looking at the file name
---
[GitHub] carbondata issue #2113: [WIP][LUCENE_DATAMAP]load issue in lucene datamap, m...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/2113
SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/4427/
---
[GitHub] carbondata issue #2113: [WIP][LUCENE_DATAMAP]load issue in lucene datamap, m...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2113
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/4758/
---
[GitHub] carbondata issue #2113: [WIP][LUCENE_DATAMAP]load issue in lucene datamap, m...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2113
Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/3719/
---
[GitHub] carbondata issue #2113: [WIP][LUCENE_DATAMAP]load issue in lucene datamap, m...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2113
Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/3441/
---
[GitHub] carbondata issue #2113: [WIP][LUCENE_DATAMAP]load issue in lucene datamap, m...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/2113
SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/4308/
---
[GitHub] carbondata issue #2113: [CARBONDATA-2347][LUCENE_DATAMAP]load issue in lucen...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2113
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/5109/
---
[GitHub] carbondata issue #2113: [CARBONDATA-2347][LUCENE_DATAMAP]load issue in lucen...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2113
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/5044/
---
[GitHub] carbondata issue #2113: [WIP][LUCENE_DATAMAP]load issue in lucene datamap, m...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/2113
SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/4399/
---
[GitHub] carbondata issue #2113: [WIP][LUCENE_DATAMAP]load issue in lucene datamap, m...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2113
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/4938/
---
[GitHub] carbondata pull request #2113: [CARBONDATA-2347][LUCENE_DATAMAP]load issue i...
Posted by jackylk <gi...@git.apache.org>.
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2113#discussion_r181660460
--- Diff: core/src/main/java/org/apache/carbondata/core/metadata/schema/table/DataMapSchema.java ---
@@ -143,11 +142,12 @@ public void setChildSchema(TableSchema childSchema) {
/**
* Return true if this datamap is an Index DataMap
+ *
--- End diff --
no need to modify
---
[GitHub] carbondata issue #2113: [WIP][LUCENE_DATAMAP]load issue in lucene datamap, m...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2113
Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/3708/
---
[GitHub] carbondata issue #2113: [WIP][LUCENE_DATAMAP]load issue in lucene datamap, m...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/2113
SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/4213/
---
[GitHub] carbondata issue #2113: [WIP][LUCENE_DATAMAP]load issue in lucene datamap, m...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2113
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/4923/
---
[GitHub] carbondata issue #2113: [WIP][LUCENE_DATAMAP]load issue in lucene datamap, m...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2113
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/4778/
---
[GitHub] carbondata issue #2113: [CARBONDATA-2347][LUCENE_DATAMAP]load issue in lucen...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2113
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/5083/
---
[GitHub] carbondata issue #2113: [WIP][LUCENE_DATAMAP]load issue in lucene datamap, m...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2113
Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/3777/
---
[GitHub] carbondata issue #2113: [WIP][LUCENE_DATAMAP]load issue in lucene datamap, m...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2113
Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/3573/
---
[GitHub] carbondata issue #2113: [WIP][LUCENE_DATAMAP]load issue in lucene datamap, m...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2113
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/4795/
---
[GitHub] carbondata issue #2113: [WIP][LUCENE_DATAMAP]load issue in lucene datamap, m...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2113
Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/3624/
---