You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@carbondata.apache.org by GitBox <gi...@apache.org> on 2020/02/12 09:23:56 UTC
[GitHub] [carbondata] jackylk opened a new pull request #3614:
[CARBONDATA-3693] Separate Index command from DataMap command
jackylk opened a new pull request #3614: [CARBONDATA-3693] Separate Index command from DataMap command
URL: https://github.com/apache/carbondata/pull/3614
### Why is this PR needed?
INDEX DDL is common in databases, carbondata should change its index related SQL syntax as other database.
### What changes were proposed in this PR?
1. Add support for: CREATE INDEX, DROP INDEX, SHOW INDEXES, REFRESH INDEX
2. Rename related class, change from XXXDataMap to XXXIndex
### Does this PR introduce any user interface change?
- No
### Is any new testcase added?
- No
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3614:
[CARBONDATA-3693] Separate Index command from DataMap command
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3614: [CARBONDATA-3693] Separate Index command from DataMap command
URL: https://github.com/apache/carbondata/pull/3614#issuecomment-590796621
Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/2161/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3614:
[CARBONDATA-3693] Separate Index command from DataMap command
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3614: [CARBONDATA-3693] Separate Index command from DataMap command
URL: https://github.com/apache/carbondata/pull/3614#issuecomment-601731435
Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/2525/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3614:
[CARBONDATA-3693] Separate Index command from DataMap command
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3614: [CARBONDATA-3693] Separate Index command from DataMap command
URL: https://github.com/apache/carbondata/pull/3614#issuecomment-596108812
Build Failed with Spark 2.4.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.4/668/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3614:
[CARBONDATA-3693] Separate Index command from DataMap command
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3614: [CARBONDATA-3693] Separate Index command from DataMap command
URL: https://github.com/apache/carbondata/pull/3614#issuecomment-591739324
Build Success with Spark 2.4.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.4/504/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3614:
[CARBONDATA-3693] Separate Index command from DataMap command
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3614: [CARBONDATA-3693] Separate Index command from DataMap command
URL: https://github.com/apache/carbondata/pull/3614#issuecomment-596159100
Build Failed with Spark 2.4.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.4/670/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] akashrn5 commented on issue #3614: [CARBONDATA-3693]
Separate Index command from DataMap command
Posted by GitBox <gi...@apache.org>.
akashrn5 commented on issue #3614: [CARBONDATA-3693] Separate Index command from DataMap command
URL: https://github.com/apache/carbondata/pull/3614#issuecomment-590695826
@jackylk please rebase
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3614:
[CARBONDATA-3693] Separate Index command from DataMap command
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3614: [CARBONDATA-3693] Separate Index command from DataMap command
URL: https://github.com/apache/carbondata/pull/3614#issuecomment-589960258
Build Success with Spark 2.4.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.4/401/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] akashrn5 commented on a change in pull request #3614:
[CARBONDATA-3693] Separate Index command from DataMap command
Posted by GitBox <gi...@apache.org>.
akashrn5 commented on a change in pull request #3614: [CARBONDATA-3693] Separate Index command from DataMap command
URL: https://github.com/apache/carbondata/pull/3614#discussion_r384282806
##########
File path: core/src/main/java/org/apache/carbondata/core/index/IndexLevel.java
##########
@@ -15,17 +15,17 @@
* limitations under the License.
*/
-package org.apache.carbondata.core.datamap;
+package org.apache.carbondata.core.index;
import org.apache.carbondata.common.annotations.InterfaceAudience;
import org.apache.carbondata.common.annotations.InterfaceStability;
/**
- * Index level of the datamap
+ * Index level
Review comment:
no need of this comment i think
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3614:
[CARBONDATA-3693] Separate Index command from DataMap command
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3614: [CARBONDATA-3693] Separate Index command from DataMap command
URL: https://github.com/apache/carbondata/pull/3614#issuecomment-592471657
Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/2229/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3614:
[CARBONDATA-3693] Separate Index command from DataMap command
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3614: [CARBONDATA-3693] Separate Index command from DataMap command
URL: https://github.com/apache/carbondata/pull/3614#issuecomment-596105176
Build Failed with Spark 2.4.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.4/666/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] Indhumathi27 commented on issue #3614:
[CARBONDATA-3693] Separate Index command from DataMap command
Posted by GitBox <gi...@apache.org>.
Indhumathi27 commented on issue #3614: [CARBONDATA-3693] Separate Index command from DataMap command
URL: https://github.com/apache/carbondata/pull/3614#issuecomment-600425692
@jackylk Can you please rebase the code
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] jackylk closed pull request #3614: [CARBONDATA-3693]
Separate Index command from DataMap command
Posted by GitBox <gi...@apache.org>.
jackylk closed pull request #3614: [CARBONDATA-3693] Separate Index command from DataMap command
URL: https://github.com/apache/carbondata/pull/3614
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] jackylk commented on issue #3614: [CARBONDATA-3693]
Separate Index command from DataMap command
Posted by GitBox <gi...@apache.org>.
jackylk commented on issue #3614: [CARBONDATA-3693] Separate Index command from DataMap command
URL: https://github.com/apache/carbondata/pull/3614#issuecomment-591734910
retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3614:
[CARBONDATA-3693] Separate Index command from DataMap command
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3614: [CARBONDATA-3693] Separate Index command from DataMap command
URL: https://github.com/apache/carbondata/pull/3614#issuecomment-596162421
Build Failed with Spark 2.4.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.4/671/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3614:
[CARBONDATA-3693] Separate Index command from DataMap command
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3614: [CARBONDATA-3693] Separate Index command from DataMap command
URL: https://github.com/apache/carbondata/pull/3614#issuecomment-590940395
Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/2172/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3614:
[CARBONDATA-3693] Separate Index command from DataMap command
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3614: [CARBONDATA-3693] Separate Index command from DataMap command
URL: https://github.com/apache/carbondata/pull/3614#issuecomment-590890828
Build Failed with Spark 2.4.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.4/469/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] jackylk commented on a change in pull request #3614:
[CARBONDATA-3693] Separate Index command from DataMap command
Posted by GitBox <gi...@apache.org>.
jackylk commented on a change in pull request #3614: [CARBONDATA-3693] Separate Index command from DataMap command
URL: https://github.com/apache/carbondata/pull/3614#discussion_r385613902
##########
File path: core/src/main/java/org/apache/carbondata/core/datamap/DataMapStoreManager.java
##########
@@ -168,6 +186,38 @@ public DataMapSchema getDataMapSchema(String dataMapName)
return provider.retrieveSchema(dataMapName);
}
+ public List<DataMapSchema> getAllMVSchemas() throws IOException {
+ return provider.retrieveAllSchemas().stream()
+ .filter(schema -> !schema.isIndexDataMap())
+ .collect(Collectors.toList());
+ }
+
+ public DataMapSchema getIndexSchema(String indexName)
+ throws NoSuchIndexException, IOException {
+ try {
+ DataMapSchema schema = provider.retrieveSchema(indexName);
+ if (!schema.isIndexDataMap()) {
+ throw new NoSuchIndexException(indexName);
+ }
+ return schema;
+ } catch (NoSuchDataMapException e) {
Review comment:
log added. If reading schema failed, IOexception will be thrown
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3614:
[CARBONDATA-3693] Separate Index command from DataMap command
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3614: [CARBONDATA-3693] Separate Index command from DataMap command
URL: https://github.com/apache/carbondata/pull/3614#issuecomment-594396920
Build Failed with Spark 2.4.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.4/602/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3614:
[CARBONDATA-3693] Separate Index command from DataMap command
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3614: [CARBONDATA-3693] Separate Index command from DataMap command
URL: https://github.com/apache/carbondata/pull/3614#issuecomment-594326418
Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/2301/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3614:
[CARBONDATA-3693] Separate Index command from DataMap command
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3614: [CARBONDATA-3693] Separate Index command from DataMap command
URL: https://github.com/apache/carbondata/pull/3614#issuecomment-596056898
Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/2362/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3614:
[CARBONDATA-3693] Separate Index command from DataMap command
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3614: [CARBONDATA-3693] Separate Index command from DataMap command
URL: https://github.com/apache/carbondata/pull/3614#issuecomment-595707309
Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/2356/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3614:
[CARBONDATA-3693] Separate Index command from DataMap command
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3614: [CARBONDATA-3693] Separate Index command from DataMap command
URL: https://github.com/apache/carbondata/pull/3614#issuecomment-589985676
Build Success with Spark 2.4.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.4/403/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] akashrn5 commented on a change in pull request #3614:
[CARBONDATA-3693] Separate Index command from DataMap command
Posted by GitBox <gi...@apache.org>.
akashrn5 commented on a change in pull request #3614: [CARBONDATA-3693] Separate Index command from DataMap command
URL: https://github.com/apache/carbondata/pull/3614#discussion_r384036324
##########
File path: core/src/main/java/org/apache/carbondata/core/datamap/DataMapStoreManager.java
##########
@@ -168,6 +186,38 @@ public DataMapSchema getDataMapSchema(String dataMapName)
return provider.retrieveSchema(dataMapName);
}
+ public List<DataMapSchema> getAllMVSchemas() throws IOException {
+ return provider.retrieveAllSchemas().stream()
+ .filter(schema -> !schema.isIndexDataMap())
+ .collect(Collectors.toList());
+ }
+
+ public DataMapSchema getIndexSchema(String indexName)
+ throws NoSuchIndexException, IOException {
+ try {
+ DataMapSchema schema = provider.retrieveSchema(indexName);
+ if (!schema.isIndexDataMap()) {
+ throw new NoSuchIndexException(indexName);
+ }
+ return schema;
+ } catch (NoSuchDataMapException e) {
+ throw new NoSuchIndexException(indexName);
+ }
+ }
+
+ public DataMapSchema getMVSchema(String mvName)
+ throws NoSuchMaterializedViewException, IOException {
+ try {
+ DataMapSchema schema = provider.retrieveSchema(mvName);
+ if (schema == null || schema.isIndexDataMap()) {
+ throw new NoSuchMaterializedViewException(mvName);
Review comment:
please do Logger.error before throwing exception
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3614:
[CARBONDATA-3693] Separate Index command from DataMap command
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3614: [CARBONDATA-3693] Separate Index command from DataMap command
URL: https://github.com/apache/carbondata/pull/3614#issuecomment-592822066
Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/2234/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3614:
[CARBONDATA-3693] Separate Index command from DataMap command
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3614: [CARBONDATA-3693] Separate Index command from DataMap command
URL: https://github.com/apache/carbondata/pull/3614#issuecomment-596076530
Build Failed with Spark 2.4.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.4/661/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3614:
[CARBONDATA-3693] Separate Index command from DataMap command
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3614: [CARBONDATA-3693] Separate Index command from DataMap command
URL: https://github.com/apache/carbondata/pull/3614#issuecomment-591315246
Build Success with Spark 2.4.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.4/491/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3614:
[CARBONDATA-3693] Separate Index command from DataMap command
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3614: [CARBONDATA-3693] Separate Index command from DataMap command
URL: https://github.com/apache/carbondata/pull/3614#issuecomment-596225894
Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/2384/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] akashrn5 commented on a change in pull request #3614:
[CARBONDATA-3693] Separate Index command from DataMap command
Posted by GitBox <gi...@apache.org>.
akashrn5 commented on a change in pull request #3614: [CARBONDATA-3693] Separate Index command from DataMap command
URL: https://github.com/apache/carbondata/pull/3614#discussion_r386206213
##########
File path: core/src/main/java/org/apache/carbondata/core/datamap/status/DataMapStatusManager.java
##########
@@ -142,13 +142,13 @@ public static DataMapSchema getDataMapSchema(String dataMapName)
}
/**
- * This method will remove all segments of dataMap table in case of Insert-Overwrite/Update/Delete
+ * This method will remove all segments of MV table in case of Insert-Overwrite/Update/Delete
* operations on main table
*
* @param allDataMapSchemas of main carbon table
* @throws IOException
*/
- public static void truncateDataMap(List<DataMapSchema> allDataMapSchemas)
+ public static void truncateMVTable(List<DataMapSchema> allDataMapSchemas)
Review comment:
ok
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] jackylk commented on a change in pull request #3614:
[CARBONDATA-3693] Separate Index command from DataMap command
Posted by GitBox <gi...@apache.org>.
jackylk commented on a change in pull request #3614: [CARBONDATA-3693] Separate Index command from DataMap command
URL: https://github.com/apache/carbondata/pull/3614#discussion_r385618823
##########
File path: core/src/main/java/org/apache/carbondata/core/datamap/status/DiskBasedDataMapStatusProvider.java
##########
@@ -31,11 +31,11 @@
import org.apache.carbondata.common.logging.LogServiceFactory;
import org.apache.carbondata.core.constants.CarbonCommonConstants;
-import org.apache.carbondata.core.datamap.dev.DataMapSyncStatus;
import org.apache.carbondata.core.datastore.impl.FileFactory;
import org.apache.carbondata.core.fileoperations.AtomicFileOperationFactory;
import org.apache.carbondata.core.fileoperations.AtomicFileOperations;
import org.apache.carbondata.core.fileoperations.FileWriteOperation;
+import org.apache.carbondata.core.index.dev.DataMapSyncStatus;
Review comment:
no, package name is changed
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3614:
[CARBONDATA-3693] Separate Index command from DataMap command
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3614: [CARBONDATA-3693] Separate Index command from DataMap command
URL: https://github.com/apache/carbondata/pull/3614#issuecomment-596087497
Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/2370/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] akashrn5 commented on issue #3614: [CARBONDATA-3693]
Separate Index command from DataMap command
Posted by GitBox <gi...@apache.org>.
akashrn5 commented on issue #3614: [CARBONDATA-3693] Separate Index command from DataMap command
URL: https://github.com/apache/carbondata/pull/3614#issuecomment-590694064
@jackylk plese resolve conflicts
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] akashrn5 removed a comment on issue #3614:
[CARBONDATA-3693] Separate Index command from DataMap command
Posted by GitBox <gi...@apache.org>.
akashrn5 removed a comment on issue #3614: [CARBONDATA-3693] Separate Index command from DataMap command
URL: https://github.com/apache/carbondata/pull/3614#issuecomment-590694064
@jackylk plese resolve conflicts
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3614:
[CARBONDATA-3693] Separate Index command from DataMap command
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3614: [CARBONDATA-3693] Separate Index command from DataMap command
URL: https://github.com/apache/carbondata/pull/3614#issuecomment-594319286
Build Success with Spark 2.4.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.4/594/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3614:
[CARBONDATA-3693] Separate Index command from DataMap command
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3614: [CARBONDATA-3693] Separate Index command from DataMap command
URL: https://github.com/apache/carbondata/pull/3614#issuecomment-587365131
Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/2033/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3614:
[CARBONDATA-3693] Separate Index command from DataMap command
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3614: [CARBONDATA-3693] Separate Index command from DataMap command
URL: https://github.com/apache/carbondata/pull/3614#issuecomment-585284339
Build Success with Spark 2.4.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.4/270/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] jackylk commented on a change in pull request #3614:
[CARBONDATA-3693] Separate Index command from DataMap command
Posted by GitBox <gi...@apache.org>.
jackylk commented on a change in pull request #3614: [CARBONDATA-3693] Separate Index command from DataMap command
URL: https://github.com/apache/carbondata/pull/3614#discussion_r385616001
##########
File path: core/src/main/java/org/apache/carbondata/core/datamap/status/DataMapStatusManager.java
##########
@@ -142,13 +142,13 @@ public static DataMapSchema getDataMapSchema(String dataMapName)
}
/**
- * This method will remove all segments of dataMap table in case of Insert-Overwrite/Update/Delete
+ * This method will remove all segments of MV table in case of Insert-Overwrite/Update/Delete
* operations on main table
*
* @param allDataMapSchemas of main carbon table
* @throws IOException
*/
- public static void truncateDataMap(List<DataMapSchema> allDataMapSchemas)
+ public static void truncateMVTable(List<DataMapSchema> allDataMapSchemas)
Review comment:
DataMapSchema can not be changed in this PR since Index and MV still both use it. Metadata unification will be done in another PR.
I think @niuge01 is doing it
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3614:
[CARBONDATA-3693] Separate Index command from DataMap command
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3614: [CARBONDATA-3693] Separate Index command from DataMap command
URL: https://github.com/apache/carbondata/pull/3614#issuecomment-590166319
Build Success with Spark 2.4.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.4/418/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] akashrn5 commented on a change in pull request #3614:
[CARBONDATA-3693] Separate Index command from DataMap command
Posted by GitBox <gi...@apache.org>.
akashrn5 commented on a change in pull request #3614: [CARBONDATA-3693] Separate Index command from DataMap command
URL: https://github.com/apache/carbondata/pull/3614#discussion_r384283718
##########
File path: core/src/main/java/org/apache/carbondata/core/index/TableIndex.java
##########
@@ -338,7 +339,7 @@ public Void call() throws IOException {
try {
executorService.awaitTermination(2, TimeUnit.HOURS);
} catch (InterruptedException e) {
- LOG.error("Error in pruning datamap in multi-thread: " + e.getMessage());
+ LOG.error("Error in pruning in multi-thread: " + e.getMessage());
Review comment:
change message , include pruning index
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3614:
[CARBONDATA-3693] Separate Index command from DataMap command
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3614: [CARBONDATA-3693] Separate Index command from DataMap command
URL: https://github.com/apache/carbondata/pull/3614#issuecomment-585310941
Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/1973/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3614:
[CARBONDATA-3693] Separate Index command from DataMap command
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3614: [CARBONDATA-3693] Separate Index command from DataMap command
URL: https://github.com/apache/carbondata/pull/3614#issuecomment-590724995
Build Success with Spark 2.4.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.4/454/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] akashrn5 commented on a change in pull request #3614:
[CARBONDATA-3693] Separate Index command from DataMap command
Posted by GitBox <gi...@apache.org>.
akashrn5 commented on a change in pull request #3614: [CARBONDATA-3693] Separate Index command from DataMap command
URL: https://github.com/apache/carbondata/pull/3614#discussion_r384281648
##########
File path: core/src/main/java/org/apache/carbondata/core/index/DistributableIndexFormat.java
##########
@@ -29,16 +29,17 @@
import org.apache.carbondata.common.logging.LogServiceFactory;
import org.apache.carbondata.core.constants.CarbonCommonConstants;
-import org.apache.carbondata.core.datamap.dev.expr.DataMapDistributableWrapper;
+import org.apache.carbondata.core.datamap.DataMapStoreManager;
+import org.apache.carbondata.core.datamap.Segment;
import org.apache.carbondata.core.datastore.impl.FileFactory;
+import org.apache.carbondata.core.index.dev.expr.IndexDistributableWrapper;
import org.apache.carbondata.core.indexstore.ExtendedBlocklet;
import org.apache.carbondata.core.indexstore.PartitionSpec;
import org.apache.carbondata.core.metadata.schema.table.CarbonTable;
import org.apache.carbondata.core.readcommitter.LatestFilesReadCommittedScope;
import org.apache.carbondata.core.readcommitter.ReadCommittedScope;
import org.apache.carbondata.core.readcommitter.TableStatusReadCommittedScope;
import org.apache.carbondata.core.scan.filter.resolver.FilterResolverIntf;
-
Review comment:
revert
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] jackylk commented on a change in pull request #3614:
[CARBONDATA-3693] Separate Index command from DataMap command
Posted by GitBox <gi...@apache.org>.
jackylk commented on a change in pull request #3614: [CARBONDATA-3693] Separate Index command from DataMap command
URL: https://github.com/apache/carbondata/pull/3614#discussion_r385612622
##########
File path: core/src/main/java/org/apache/carbondata/core/datamap/DataMapProvider.java
##########
@@ -178,7 +180,7 @@ public boolean rebuild() throws IOException, NoSuchDataMapException {
== SegmentStatus.INSERT_OVERWRITE_IN_PROGRESS) && SegmentStatusManager
.isLoadInProgress(dataMapTableAbsoluteTableIdentifier,
loadMetaDetail.getLoadName())) {
- throw new RuntimeException("Rebuild to datamap " + dataMapSchema.getDataMapName()
+ throw new RuntimeException("Rebuild to " + dataMapSchema.getDataMapName()
Review comment:
fixed
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3614:
[CARBONDATA-3693] Separate Index command from DataMap command
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3614: [CARBONDATA-3693] Separate Index command from DataMap command
URL: https://github.com/apache/carbondata/pull/3614#issuecomment-596087372
Build Failed with Spark 2.4.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.4/664/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] akashrn5 commented on a change in pull request #3614:
[CARBONDATA-3693] Separate Index command from DataMap command
Posted by GitBox <gi...@apache.org>.
akashrn5 commented on a change in pull request #3614: [CARBONDATA-3693] Separate Index command from DataMap command
URL: https://github.com/apache/carbondata/pull/3614#discussion_r384033877
##########
File path: core/src/main/java/org/apache/carbondata/core/datamap/DataMapProvider.java
##########
@@ -48,25 +49,26 @@
import org.apache.log4j.Logger;
/**
- * DataMap is a accelerator for certain type of query. Developer can add new DataMap
+ * Index is a accelerator for certain type of query. Developer can add new Index
* implementation to improve query performance.
*
- * Currently two types of DataMap are supported
+ * Currently two types of Index are supported
* <ol>
- * <li> MVDataMap: materialized view type of DataMap to accelerate olap style query,
+ * <li> MVDataMap: materialized view type of Index to accelerate olap style query,
* like SPJG query (select, predicate, join, groupby) </li>
- * <li> DataMap: index type of DataMap to accelerate filter query </li>
+ * <li> Index: index type of Index to accelerate filter query </li>
* </ol>
*
* <p>
* In following command <br>
- * {@code CREATE DATAMAP dm ON TABLE main USING 'provider'}, <br>
- * the <b>provider</b> string can be a short name or class name of the DataMap implementation.
+ * {@code CREATE INDEX index ON main AS 'provider'}, <br>
Review comment:
`on table ` should not be removed here, its required
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3614:
[CARBONDATA-3693] Separate Index command from DataMap command
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3614: [CARBONDATA-3693] Separate Index command from DataMap command
URL: https://github.com/apache/carbondata/pull/3614#issuecomment-591748167
Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/2203/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] akashrn5 commented on a change in pull request #3614:
[CARBONDATA-3693] Separate Index command from DataMap command
Posted by GitBox <gi...@apache.org>.
akashrn5 commented on a change in pull request #3614: [CARBONDATA-3693] Separate Index command from DataMap command
URL: https://github.com/apache/carbondata/pull/3614#discussion_r384033343
##########
File path: core/src/main/java/org/apache/carbondata/core/constants/CarbonCommonConstants.java
##########
@@ -460,10 +460,9 @@ private CarbonCommonConstants() {
public static final String LOCAL_DICTIONARY_EXCLUDE = "local_dictionary_exclude";
/**
- * DMPROPERTY for Index DataMap, like lucene, bloomfilter DataMap,
- * to indicate a list of column name to be indexed
+ * Internal property to store for index column names
*/
- public static final String INDEX_COLUMNS = "INDEX_COLUMNS";
+ public static final String INDEX_COLUMNS = "_INDEX_COLUMNS";
Review comment:
why this change is required?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3614:
[CARBONDATA-3693] Separate Index command from DataMap command
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3614: [CARBONDATA-3693] Separate Index command from DataMap command
URL: https://github.com/apache/carbondata/pull/3614#issuecomment-596108910
Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/2374/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3614:
[CARBONDATA-3693] Separate Index command from DataMap command
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3614: [CARBONDATA-3693] Separate Index command from DataMap command
URL: https://github.com/apache/carbondata/pull/3614#issuecomment-590746991
Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/2155/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] jackylk commented on a change in pull request #3614:
[CARBONDATA-3693] Separate Index command from DataMap command
Posted by GitBox <gi...@apache.org>.
jackylk commented on a change in pull request #3614: [CARBONDATA-3693] Separate Index command from DataMap command
URL: https://github.com/apache/carbondata/pull/3614#discussion_r385613902
##########
File path: core/src/main/java/org/apache/carbondata/core/datamap/DataMapStoreManager.java
##########
@@ -168,6 +186,38 @@ public DataMapSchema getDataMapSchema(String dataMapName)
return provider.retrieveSchema(dataMapName);
}
+ public List<DataMapSchema> getAllMVSchemas() throws IOException {
+ return provider.retrieveAllSchemas().stream()
+ .filter(schema -> !schema.isIndexDataMap())
+ .collect(Collectors.toList());
+ }
+
+ public DataMapSchema getIndexSchema(String indexName)
+ throws NoSuchIndexException, IOException {
+ try {
+ DataMapSchema schema = provider.retrieveSchema(indexName);
+ if (!schema.isIndexDataMap()) {
+ throw new NoSuchIndexException(indexName);
+ }
+ return schema;
+ } catch (NoSuchDataMapException e) {
Review comment:
log added
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3614:
[CARBONDATA-3693] Separate Index command from DataMap command
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3614: [CARBONDATA-3693] Separate Index command from DataMap command
URL: https://github.com/apache/carbondata/pull/3614#issuecomment-590965050
Build Failed with Spark 2.4.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.4/475/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3614:
[CARBONDATA-3693] Separate Index command from DataMap command
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3614: [CARBONDATA-3693] Separate Index command from DataMap command
URL: https://github.com/apache/carbondata/pull/3614#issuecomment-587522824
Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/2044/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3614:
[CARBONDATA-3693] Separate Index command from DataMap command
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3614: [CARBONDATA-3693] Separate Index command from DataMap command
URL: https://github.com/apache/carbondata/pull/3614#issuecomment-591441468
Build Success with Spark 2.4.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.4/501/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] akashrn5 commented on a change in pull request #3614:
[CARBONDATA-3693] Separate Index command from DataMap command
Posted by GitBox <gi...@apache.org>.
akashrn5 commented on a change in pull request #3614: [CARBONDATA-3693] Separate Index command from DataMap command
URL: https://github.com/apache/carbondata/pull/3614#discussion_r384280832
##########
File path: core/src/main/java/org/apache/carbondata/core/datamap/status/DataMapStatus.java
##########
@@ -18,7 +18,7 @@
package org.apache.carbondata.core.datamap.status;
/**
- * DataMap status
+ * Index status
*/
public enum DataMapStatus {
Review comment:
can make this IndexStatus
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3614:
[CARBONDATA-3693] Separate Index command from DataMap command
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3614: [CARBONDATA-3693] Separate Index command from DataMap command
URL: https://github.com/apache/carbondata/pull/3614#issuecomment-591330074
Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/2191/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3614:
[CARBONDATA-3693] Separate Index command from DataMap command
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3614: [CARBONDATA-3693] Separate Index command from DataMap command
URL: https://github.com/apache/carbondata/pull/3614#issuecomment-590821296
Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/2163/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3614:
[CARBONDATA-3693] Separate Index command from DataMap command
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3614: [CARBONDATA-3693] Separate Index command from DataMap command
URL: https://github.com/apache/carbondata/pull/3614#issuecomment-585141874
Build Success with Spark 2.4.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.4/259/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3614:
[CARBONDATA-3693] Separate Index command from DataMap command
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3614: [CARBONDATA-3693] Separate Index command from DataMap command
URL: https://github.com/apache/carbondata/pull/3614#issuecomment-596037509
Build Failed with Spark 2.4.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.4/656/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3614:
[CARBONDATA-3693] Separate Index command from DataMap command
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3614: [CARBONDATA-3693] Separate Index command from DataMap command
URL: https://github.com/apache/carbondata/pull/3614#issuecomment-585113664
Build Failed with Spark 2.4.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.4/255/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3614:
[CARBONDATA-3693] Separate Index command from DataMap command
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3614: [CARBONDATA-3693] Separate Index command from DataMap command
URL: https://github.com/apache/carbondata/pull/3614#issuecomment-602026055
Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/2534/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] jackylk commented on a change in pull request #3614:
[CARBONDATA-3693] Separate Index command from DataMap command
Posted by GitBox <gi...@apache.org>.
jackylk commented on a change in pull request #3614: [CARBONDATA-3693] Separate Index command from DataMap command
URL: https://github.com/apache/carbondata/pull/3614#discussion_r385616968
##########
File path: core/src/main/java/org/apache/carbondata/core/index/TableIndex.java
##########
@@ -338,7 +339,7 @@ public Void call() throws IOException {
try {
executorService.awaitTermination(2, TimeUnit.HOURS);
} catch (InterruptedException e) {
- LOG.error("Error in pruning datamap in multi-thread: " + e.getMessage());
+ LOG.error("Error in pruning in multi-thread: " + e.getMessage());
Review comment:
fixed
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] akashrn5 commented on a change in pull request #3614:
[CARBONDATA-3693] Separate Index command from DataMap command
Posted by GitBox <gi...@apache.org>.
akashrn5 commented on a change in pull request #3614: [CARBONDATA-3693] Separate Index command from DataMap command
URL: https://github.com/apache/carbondata/pull/3614#discussion_r384281103
##########
File path: core/src/main/java/org/apache/carbondata/core/datamap/status/DiskBasedDataMapStatusProvider.java
##########
@@ -31,11 +31,11 @@
import org.apache.carbondata.common.logging.LogServiceFactory;
import org.apache.carbondata.core.constants.CarbonCommonConstants;
-import org.apache.carbondata.core.datamap.dev.DataMapSyncStatus;
import org.apache.carbondata.core.datastore.impl.FileFactory;
import org.apache.carbondata.core.fileoperations.AtomicFileOperationFactory;
import org.apache.carbondata.core.fileoperations.AtomicFileOperations;
import org.apache.carbondata.core.fileoperations.FileWriteOperation;
+import org.apache.carbondata.core.index.dev.DataMapSyncStatus;
Review comment:
can revert this change
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3614:
[CARBONDATA-3693] Separate Index command from DataMap command
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3614: [CARBONDATA-3693] Separate Index command from DataMap command
URL: https://github.com/apache/carbondata/pull/3614#issuecomment-592821671
Build Failed with Spark 2.4.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.4/534/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3614:
[CARBONDATA-3693] Separate Index command from DataMap command
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3614: [CARBONDATA-3693] Separate Index command from DataMap command
URL: https://github.com/apache/carbondata/pull/3614#issuecomment-590796162
Build Success with Spark 2.4.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.4/462/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] jackylk commented on issue #3614: [CARBONDATA-3693]
Separate Index command from DataMap command
Posted by GitBox <gi...@apache.org>.
jackylk commented on issue #3614: [CARBONDATA-3693] Separate Index command from DataMap command
URL: https://github.com/apache/carbondata/pull/3614#issuecomment-592807190
retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3614:
[CARBONDATA-3693] Separate Index command from DataMap command
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3614: [CARBONDATA-3693] Separate Index command from DataMap command
URL: https://github.com/apache/carbondata/pull/3614#issuecomment-595856295
Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/2359/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3614:
[CARBONDATA-3693] Separate Index command from DataMap command
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3614: [CARBONDATA-3693] Separate Index command from DataMap command
URL: https://github.com/apache/carbondata/pull/3614#issuecomment-585113999
Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/1958/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3614:
[CARBONDATA-3693] Separate Index command from DataMap command
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3614: [CARBONDATA-3693] Separate Index command from DataMap command
URL: https://github.com/apache/carbondata/pull/3614#issuecomment-591199786
Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/2175/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3614:
[CARBONDATA-3693] Separate Index command from DataMap command
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3614: [CARBONDATA-3693] Separate Index command from DataMap command
URL: https://github.com/apache/carbondata/pull/3614#issuecomment-601739739
Build Failed with Spark 2.4.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.4/818/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] akashrn5 commented on a change in pull request #3614:
[CARBONDATA-3693] Separate Index command from DataMap command
Posted by GitBox <gi...@apache.org>.
akashrn5 commented on a change in pull request #3614: [CARBONDATA-3693] Separate Index command from DataMap command
URL: https://github.com/apache/carbondata/pull/3614#discussion_r384281012
##########
File path: core/src/main/java/org/apache/carbondata/core/datamap/status/DataMapStatusManager.java
##########
@@ -142,13 +142,13 @@ public static DataMapSchema getDataMapSchema(String dataMapName)
}
/**
- * This method will remove all segments of dataMap table in case of Insert-Overwrite/Update/Delete
+ * This method will remove all segments of MV table in case of Insert-Overwrite/Update/Delete
* operations on main table
*
* @param allDataMapSchemas of main carbon table
* @throws IOException
*/
- public static void truncateDataMap(List<DataMapSchema> allDataMapSchemas)
+ public static void truncateMVTable(List<DataMapSchema> allDataMapSchemas)
Review comment:
can we change change to IndexSchema
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3614:
[CARBONDATA-3693] Separate Index command from DataMap command
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3614: [CARBONDATA-3693] Separate Index command from DataMap command
URL: https://github.com/apache/carbondata/pull/3614#issuecomment-602027087
Build Failed with Spark 2.4.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.4/827/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3614:
[CARBONDATA-3693] Separate Index command from DataMap command
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3614: [CARBONDATA-3693] Separate Index command from DataMap command
URL: https://github.com/apache/carbondata/pull/3614#issuecomment-594357828
Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/2305/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] jackylk commented on a change in pull request #3614:
[CARBONDATA-3693] Separate Index command from DataMap command
Posted by GitBox <gi...@apache.org>.
jackylk commented on a change in pull request #3614: [CARBONDATA-3693] Separate Index command from DataMap command
URL: https://github.com/apache/carbondata/pull/3614#discussion_r385609753
##########
File path: core/src/main/java/org/apache/carbondata/core/datamap/DataMapProvider.java
##########
@@ -48,25 +49,26 @@
import org.apache.log4j.Logger;
/**
- * DataMap is a accelerator for certain type of query. Developer can add new DataMap
+ * Index is a accelerator for certain type of query. Developer can add new Index
* implementation to improve query performance.
*
- * Currently two types of DataMap are supported
+ * Currently two types of Index are supported
* <ol>
- * <li> MVDataMap: materialized view type of DataMap to accelerate olap style query,
+ * <li> MVDataMap: materialized view type of Index to accelerate olap style query,
* like SPJG query (select, predicate, join, groupby) </li>
- * <li> DataMap: index type of DataMap to accelerate filter query </li>
+ * <li> Index: index type of Index to accelerate filter query </li>
* </ol>
*
* <p>
* In following command <br>
- * {@code CREATE DATAMAP dm ON TABLE main USING 'provider'}, <br>
- * the <b>provider</b> string can be a short name or class name of the DataMap implementation.
+ * {@code CREATE INDEX index ON main AS 'provider'}, <br>
Review comment:
I have make ON TABLE optional
so user can give `CREATE INDEX index ON TABLE main` or `CREATE INDEX index ON main`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3614:
[CARBONDATA-3693] Separate Index command from DataMap command
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3614: [CARBONDATA-3693] Separate Index command from DataMap command
URL: https://github.com/apache/carbondata/pull/3614#issuecomment-585843181
Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/1982/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] akashrn5 commented on a change in pull request #3614:
[CARBONDATA-3693] Separate Index command from DataMap command
Posted by GitBox <gi...@apache.org>.
akashrn5 commented on a change in pull request #3614: [CARBONDATA-3693] Separate Index command from DataMap command
URL: https://github.com/apache/carbondata/pull/3614#discussion_r384034368
##########
File path: core/src/main/java/org/apache/carbondata/core/datamap/DataMapProvider.java
##########
@@ -178,7 +180,7 @@ public boolean rebuild() throws IOException, NoSuchDataMapException {
== SegmentStatus.INSERT_OVERWRITE_IN_PROGRESS) && SegmentStatusManager
.isLoadInProgress(dataMapTableAbsoluteTableIdentifier,
loadMetaDetail.getLoadName())) {
- throw new RuntimeException("Rebuild to datamap " + dataMapSchema.getDataMapName()
+ throw new RuntimeException("Rebuild to " + dataMapSchema.getDataMapName()
Review comment:
`Rebuild to` can be renamed to `Refresh`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] ajantha-bhat commented on issue #3614:
[CARBONDATA-3693] Separate Index command from DataMap command
Posted by GitBox <gi...@apache.org>.
ajantha-bhat commented on issue #3614: [CARBONDATA-3693] Separate Index command from DataMap command
URL: https://github.com/apache/carbondata/pull/3614#issuecomment-591307790
retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3614:
[CARBONDATA-3693] Separate Index command from DataMap command
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3614: [CARBONDATA-3693] Separate Index command from DataMap command
URL: https://github.com/apache/carbondata/pull/3614#issuecomment-596076221
Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/2367/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3614:
[CARBONDATA-3693] Separate Index command from DataMap command
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3614: [CARBONDATA-3693] Separate Index command from DataMap command
URL: https://github.com/apache/carbondata/pull/3614#issuecomment-593088362
Build Success with Spark 2.4.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.4/548/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] jackylk commented on a change in pull request #3614:
[CARBONDATA-3693] Separate Index command from DataMap command
Posted by GitBox <gi...@apache.org>.
jackylk commented on a change in pull request #3614: [CARBONDATA-3693] Separate Index command from DataMap command
URL: https://github.com/apache/carbondata/pull/3614#discussion_r385609753
##########
File path: core/src/main/java/org/apache/carbondata/core/datamap/DataMapProvider.java
##########
@@ -48,25 +49,26 @@
import org.apache.log4j.Logger;
/**
- * DataMap is a accelerator for certain type of query. Developer can add new DataMap
+ * Index is a accelerator for certain type of query. Developer can add new Index
* implementation to improve query performance.
*
- * Currently two types of DataMap are supported
+ * Currently two types of Index are supported
* <ol>
- * <li> MVDataMap: materialized view type of DataMap to accelerate olap style query,
+ * <li> MVDataMap: materialized view type of Index to accelerate olap style query,
* like SPJG query (select, predicate, join, groupby) </li>
- * <li> DataMap: index type of DataMap to accelerate filter query </li>
+ * <li> Index: index type of Index to accelerate filter query </li>
* </ol>
*
* <p>
* In following command <br>
- * {@code CREATE DATAMAP dm ON TABLE main USING 'provider'}, <br>
- * the <b>provider</b> string can be a short name or class name of the DataMap implementation.
+ * {@code CREATE INDEX index ON main AS 'provider'}, <br>
Review comment:
I have make the syntax to `CREATE INDEX index_name ON [TABLE] main_table`
so user can give
`CREATE INDEX index ON TABLE main` or
`CREATE INDEX index ON main`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3614:
[CARBONDATA-3693] Separate Index command from DataMap command
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3614: [CARBONDATA-3693] Separate Index command from DataMap command
URL: https://github.com/apache/carbondata/pull/3614#issuecomment-589966869
Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/2102/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] akashrn5 commented on a change in pull request #3614:
[CARBONDATA-3693] Separate Index command from DataMap command
Posted by GitBox <gi...@apache.org>.
akashrn5 commented on a change in pull request #3614: [CARBONDATA-3693] Separate Index command from DataMap command
URL: https://github.com/apache/carbondata/pull/3614#discussion_r384035745
##########
File path: core/src/main/java/org/apache/carbondata/core/datamap/DataMapStoreManager.java
##########
@@ -168,6 +186,38 @@ public DataMapSchema getDataMapSchema(String dataMapName)
return provider.retrieveSchema(dataMapName);
}
+ public List<DataMapSchema> getAllMVSchemas() throws IOException {
+ return provider.retrieveAllSchemas().stream()
+ .filter(schema -> !schema.isIndexDataMap())
+ .collect(Collectors.toList());
+ }
+
+ public DataMapSchema getIndexSchema(String indexName)
+ throws NoSuchIndexException, IOException {
+ try {
+ DataMapSchema schema = provider.retrieveSchema(indexName);
+ if (!schema.isIndexDataMap()) {
+ throw new NoSuchIndexException(indexName);
+ }
+ return schema;
+ } catch (NoSuchDataMapException e) {
Review comment:
please add `LOGGER.error()` here and if no schema file is present, it will throw IO exception, if while reading schema failed IOexception is thrown, always saying NoSuchIndexException is misleading i think
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3614:
[CARBONDATA-3693] Separate Index command from DataMap command
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3614: [CARBONDATA-3693] Separate Index command from DataMap command
URL: https://github.com/apache/carbondata/pull/3614#issuecomment-590929249
Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/2168/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3614:
[CARBONDATA-3693] Separate Index command from DataMap command
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3614: [CARBONDATA-3693] Separate Index command from DataMap command
URL: https://github.com/apache/carbondata/pull/3614#issuecomment-589992593
Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/2104/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3614:
[CARBONDATA-3693] Separate Index command from DataMap command
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3614: [CARBONDATA-3693] Separate Index command from DataMap command
URL: https://github.com/apache/carbondata/pull/3614#issuecomment-590921139
Build Failed with Spark 2.4.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.4/473/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3614:
[CARBONDATA-3693] Separate Index command from DataMap command
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3614: [CARBONDATA-3693] Separate Index command from DataMap command
URL: https://github.com/apache/carbondata/pull/3614#issuecomment-595856139
Build Failed with Spark 2.4.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.4/654/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3614:
[CARBONDATA-3693] Separate Index command from DataMap command
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3614: [CARBONDATA-3693] Separate Index command from DataMap command
URL: https://github.com/apache/carbondata/pull/3614#issuecomment-590784619
Build Failed with Spark 2.4.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.4/460/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3614:
[CARBONDATA-3693] Separate Index command from DataMap command
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3614: [CARBONDATA-3693] Separate Index command from DataMap command
URL: https://github.com/apache/carbondata/pull/3614#issuecomment-592472131
Build Failed with Spark 2.4.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.4/529/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3614:
[CARBONDATA-3693] Separate Index command from DataMap command
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3614: [CARBONDATA-3693] Separate Index command from DataMap command
URL: https://github.com/apache/carbondata/pull/3614#issuecomment-595708396
Build Failed with Spark 2.4.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.4/650/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3614:
[CARBONDATA-3693] Separate Index command from DataMap command
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3614: [CARBONDATA-3693] Separate Index command from DataMap command
URL: https://github.com/apache/carbondata/pull/3614#issuecomment-596094180
Build Failed with Spark 2.4.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.4/665/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] jackylk commented on a change in pull request #3614:
[CARBONDATA-3693] Separate Index command from DataMap command
Posted by GitBox <gi...@apache.org>.
jackylk commented on a change in pull request #3614: [CARBONDATA-3693] Separate Index command from DataMap command
URL: https://github.com/apache/carbondata/pull/3614#discussion_r385617240
##########
File path: core/src/main/java/org/apache/carbondata/core/index/dev/BlockletSerializer.java
##########
@@ -15,19 +15,19 @@
* limitations under the License.
*/
-package org.apache.carbondata.core.datamap.dev;
+package org.apache.carbondata.core.index.dev;
import java.io.DataInputStream;
import java.io.DataOutputStream;
import java.io.IOException;
import org.apache.carbondata.common.annotations.InterfaceAudience;
-import org.apache.carbondata.core.datamap.dev.fgdatamap.FineGrainBlocklet;
import org.apache.carbondata.core.datastore.impl.FileFactory;
+import org.apache.carbondata.core.index.dev.fgindex.FineGrainBlocklet;
/**
* A serializer/deserializer for {@link FineGrainBlocklet}, it is used after prune the data
- * by {@link org.apache.carbondata.core.datamap.dev.fgdatamap.FineGrainDataMap}
+ * by {@link FineGrainIndex}
Review comment:
fixed
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] jackylk commented on a change in pull request #3614:
[CARBONDATA-3693] Separate Index command from DataMap command
Posted by GitBox <gi...@apache.org>.
jackylk commented on a change in pull request #3614: [CARBONDATA-3693] Separate Index command from DataMap command
URL: https://github.com/apache/carbondata/pull/3614#discussion_r385615266
##########
File path: core/src/main/java/org/apache/carbondata/core/datamap/DataMapStoreManager.java
##########
@@ -168,6 +186,38 @@ public DataMapSchema getDataMapSchema(String dataMapName)
return provider.retrieveSchema(dataMapName);
}
+ public List<DataMapSchema> getAllMVSchemas() throws IOException {
+ return provider.retrieveAllSchemas().stream()
+ .filter(schema -> !schema.isIndexDataMap())
+ .collect(Collectors.toList());
+ }
+
+ public DataMapSchema getIndexSchema(String indexName)
+ throws NoSuchIndexException, IOException {
+ try {
+ DataMapSchema schema = provider.retrieveSchema(indexName);
+ if (!schema.isIndexDataMap()) {
+ throw new NoSuchIndexException(indexName);
+ }
+ return schema;
+ } catch (NoSuchDataMapException e) {
+ throw new NoSuchIndexException(indexName);
+ }
+ }
+
+ public DataMapSchema getMVSchema(String mvName)
+ throws NoSuchMaterializedViewException, IOException {
+ try {
+ DataMapSchema schema = provider.retrieveSchema(mvName);
+ if (schema == null || schema.isIndexDataMap()) {
+ throw new NoSuchMaterializedViewException(mvName);
Review comment:
In this case, no materialized view is found. I think throwing exception is enough. What kind of log do you suggest?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3614:
[CARBONDATA-3693] Separate Index command from DataMap command
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3614: [CARBONDATA-3693] Separate Index command from DataMap command
URL: https://github.com/apache/carbondata/pull/3614#issuecomment-594403095
Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/2309/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3614:
[CARBONDATA-3693] Separate Index command from DataMap command
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3614: [CARBONDATA-3693] Separate Index command from DataMap command
URL: https://github.com/apache/carbondata/pull/3614#issuecomment-591454698
Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/2200/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3614:
[CARBONDATA-3693] Separate Index command from DataMap command
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3614: [CARBONDATA-3693] Separate Index command from DataMap command
URL: https://github.com/apache/carbondata/pull/3614#issuecomment-602006326
Build Failed with Spark 2.4.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.4/823/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3614:
[CARBONDATA-3693] Separate Index command from DataMap command
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3614: [CARBONDATA-3693] Separate Index command from DataMap command
URL: https://github.com/apache/carbondata/pull/3614#issuecomment-590168725
Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/2118/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] jackylk commented on a change in pull request #3614:
[CARBONDATA-3693] Separate Index command from DataMap command
Posted by GitBox <gi...@apache.org>.
jackylk commented on a change in pull request #3614: [CARBONDATA-3693] Separate Index command from DataMap command
URL: https://github.com/apache/carbondata/pull/3614#discussion_r385609150
##########
File path: core/src/main/java/org/apache/carbondata/core/constants/CarbonCommonConstants.java
##########
@@ -460,10 +460,9 @@ private CarbonCommonConstants() {
public static final String LOCAL_DICTIONARY_EXCLUDE = "local_dictionary_exclude";
/**
- * DMPROPERTY for Index DataMap, like lucene, bloomfilter DataMap,
- * to indicate a list of column name to be indexed
+ * Internal property to store for index column names
*/
- public static final String INDEX_COLUMNS = "INDEX_COLUMNS";
+ public static final String INDEX_COLUMNS = "_INDEX_COLUMNS";
Review comment:
This is a temporary change, index metadata should also unified using main table's table property. This will be done in another PR
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3614:
[CARBONDATA-3693] Separate Index command from DataMap command
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3614: [CARBONDATA-3693] Separate Index command from DataMap command
URL: https://github.com/apache/carbondata/pull/3614#issuecomment-594357541
Build Failed with Spark 2.4.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.4/598/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3614:
[CARBONDATA-3693] Separate Index command from DataMap command
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3614: [CARBONDATA-3693] Separate Index command from DataMap command
URL: https://github.com/apache/carbondata/pull/3614#issuecomment-596104788
Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/2372/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] akashrn5 commented on a change in pull request #3614:
[CARBONDATA-3693] Separate Index command from DataMap command
Posted by GitBox <gi...@apache.org>.
akashrn5 commented on a change in pull request #3614: [CARBONDATA-3693] Separate Index command from DataMap command
URL: https://github.com/apache/carbondata/pull/3614#discussion_r386205955
##########
File path: core/src/main/java/org/apache/carbondata/core/datamap/DataMapStoreManager.java
##########
@@ -168,6 +186,38 @@ public DataMapSchema getDataMapSchema(String dataMapName)
return provider.retrieveSchema(dataMapName);
}
+ public List<DataMapSchema> getAllMVSchemas() throws IOException {
+ return provider.retrieveAllSchemas().stream()
+ .filter(schema -> !schema.isIndexDataMap())
+ .collect(Collectors.toList());
+ }
+
+ public DataMapSchema getIndexSchema(String indexName)
+ throws NoSuchIndexException, IOException {
+ try {
+ DataMapSchema schema = provider.retrieveSchema(indexName);
+ if (!schema.isIndexDataMap()) {
+ throw new NoSuchIndexException(indexName);
+ }
+ return schema;
+ } catch (NoSuchDataMapException e) {
+ throw new NoSuchIndexException(indexName);
+ }
+ }
+
+ public DataMapSchema getMVSchema(String mvName)
+ throws NoSuchMaterializedViewException, IOException {
+ try {
+ DataMapSchema schema = provider.retrieveSchema(mvName);
+ if (schema == null || schema.isIndexDataMap()) {
+ throw new NoSuchMaterializedViewException(mvName);
Review comment:
just log error with message and the throwable, if we directly throw may be sometimes if the cause is different it will mislead
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] jackylk commented on a change in pull request #3614:
[CARBONDATA-3693] Separate Index command from DataMap command
Posted by GitBox <gi...@apache.org>.
jackylk commented on a change in pull request #3614: [CARBONDATA-3693] Separate Index command from DataMap command
URL: https://github.com/apache/carbondata/pull/3614#discussion_r385616775
##########
File path: core/src/main/java/org/apache/carbondata/core/index/IndexLevel.java
##########
@@ -15,17 +15,17 @@
* limitations under the License.
*/
-package org.apache.carbondata.core.datamap;
+package org.apache.carbondata.core.index;
import org.apache.carbondata.common.annotations.InterfaceAudience;
import org.apache.carbondata.common.annotations.InterfaceStability;
/**
- * Index level of the datamap
+ * Index level
Review comment:
fixed
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] akashrn5 commented on a change in pull request #3614:
[CARBONDATA-3693] Separate Index command from DataMap command
Posted by GitBox <gi...@apache.org>.
akashrn5 commented on a change in pull request #3614: [CARBONDATA-3693] Separate Index command from DataMap command
URL: https://github.com/apache/carbondata/pull/3614#discussion_r384286761
##########
File path: core/src/main/java/org/apache/carbondata/core/index/dev/BlockletSerializer.java
##########
@@ -15,19 +15,19 @@
* limitations under the License.
*/
-package org.apache.carbondata.core.datamap.dev;
+package org.apache.carbondata.core.index.dev;
import java.io.DataInputStream;
import java.io.DataOutputStream;
import java.io.IOException;
import org.apache.carbondata.common.annotations.InterfaceAudience;
-import org.apache.carbondata.core.datamap.dev.fgdatamap.FineGrainBlocklet;
import org.apache.carbondata.core.datastore.impl.FileFactory;
+import org.apache.carbondata.core.index.dev.fgindex.FineGrainBlocklet;
/**
* A serializer/deserializer for {@link FineGrainBlocklet}, it is used after prune the data
- * by {@link org.apache.carbondata.core.datamap.dev.fgdatamap.FineGrainDataMap}
+ * by {@link FineGrainIndex}
Review comment:
@link is not working for it, please correct it
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] jackylk commented on a change in pull request #3614:
[CARBONDATA-3693] Separate Index command from DataMap command
Posted by GitBox <gi...@apache.org>.
jackylk commented on a change in pull request #3614: [CARBONDATA-3693] Separate Index command from DataMap command
URL: https://github.com/apache/carbondata/pull/3614#discussion_r385614720
##########
File path: core/src/main/java/org/apache/carbondata/core/datamap/DataMapStoreManager.java
##########
@@ -168,6 +186,38 @@ public DataMapSchema getDataMapSchema(String dataMapName)
return provider.retrieveSchema(dataMapName);
}
+ public List<DataMapSchema> getAllMVSchemas() throws IOException {
+ return provider.retrieveAllSchemas().stream()
+ .filter(schema -> !schema.isIndexDataMap())
+ .collect(Collectors.toList());
+ }
+
+ public DataMapSchema getIndexSchema(String indexName)
+ throws NoSuchIndexException, IOException {
+ try {
+ DataMapSchema schema = provider.retrieveSchema(indexName);
+ if (!schema.isIndexDataMap()) {
+ throw new NoSuchIndexException(indexName);
+ }
+ return schema;
+ } catch (NoSuchDataMapException e) {
+ throw new NoSuchIndexException(indexName);
+ }
+ }
+
+ public DataMapSchema getMVSchema(String mvName)
+ throws NoSuchMaterializedViewException, IOException {
+ try {
+ DataMapSchema schema = provider.retrieveSchema(mvName);
+ if (schema == null || schema.isIndexDataMap()) {
+ throw new NoSuchMaterializedViewException(mvName);
+ }
+ return schema;
+ } catch (NoSuchDataMapException e) {
+ throw new NoSuchMaterializedViewException(mvName);
Review comment:
log added
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] jackylk commented on issue #3614: [CARBONDATA-3693]
Separate Index command from DataMap command
Posted by GitBox <gi...@apache.org>.
jackylk commented on issue #3614: [CARBONDATA-3693] Separate Index command from DataMap command
URL: https://github.com/apache/carbondata/pull/3614#issuecomment-590718957
@akashrn5 rebased
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3614:
[CARBONDATA-3693] Separate Index command from DataMap command
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3614: [CARBONDATA-3693] Separate Index command from DataMap command
URL: https://github.com/apache/carbondata/pull/3614#issuecomment-596226159
Build Failed with Spark 2.4.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.4/678/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3614:
[CARBONDATA-3693] Separate Index command from DataMap command
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3614: [CARBONDATA-3693] Separate Index command from DataMap command
URL: https://github.com/apache/carbondata/pull/3614#issuecomment-596119968
Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/2375/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3614:
[CARBONDATA-3693] Separate Index command from DataMap command
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3614: [CARBONDATA-3693] Separate Index command from DataMap command
URL: https://github.com/apache/carbondata/pull/3614#issuecomment-593091305
Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/2248/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3614:
[CARBONDATA-3693] Separate Index command from DataMap command
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3614: [CARBONDATA-3693] Separate Index command from DataMap command
URL: https://github.com/apache/carbondata/pull/3614#issuecomment-587513012
Build Success with Spark 2.4.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.4/342/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] jackylk commented on a change in pull request #3614:
[CARBONDATA-3693] Separate Index command from DataMap command
Posted by GitBox <gi...@apache.org>.
jackylk commented on a change in pull request #3614: [CARBONDATA-3693] Separate Index command from DataMap command
URL: https://github.com/apache/carbondata/pull/3614#discussion_r385617862
##########
File path: core/src/main/java/org/apache/carbondata/core/index/DistributableIndexFormat.java
##########
@@ -29,16 +29,17 @@
import org.apache.carbondata.common.logging.LogServiceFactory;
import org.apache.carbondata.core.constants.CarbonCommonConstants;
-import org.apache.carbondata.core.datamap.dev.expr.DataMapDistributableWrapper;
+import org.apache.carbondata.core.datamap.DataMapStoreManager;
+import org.apache.carbondata.core.datamap.Segment;
import org.apache.carbondata.core.datastore.impl.FileFactory;
+import org.apache.carbondata.core.index.dev.expr.IndexDistributableWrapper;
import org.apache.carbondata.core.indexstore.ExtendedBlocklet;
import org.apache.carbondata.core.indexstore.PartitionSpec;
import org.apache.carbondata.core.metadata.schema.table.CarbonTable;
import org.apache.carbondata.core.readcommitter.LatestFilesReadCommittedScope;
import org.apache.carbondata.core.readcommitter.ReadCommittedScope;
import org.apache.carbondata.core.readcommitter.TableStatusReadCommittedScope;
import org.apache.carbondata.core.scan.filter.resolver.FilterResolverIntf;
-
Review comment:
This line is not required
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3614:
[CARBONDATA-3693] Separate Index command from DataMap command
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3614: [CARBONDATA-3693] Separate Index command from DataMap command
URL: https://github.com/apache/carbondata/pull/3614#issuecomment-591187927
Build Failed with Spark 2.4.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.4/476/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3614:
[CARBONDATA-3693] Separate Index command from DataMap command
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3614: [CARBONDATA-3693] Separate Index command from DataMap command
URL: https://github.com/apache/carbondata/pull/3614#issuecomment-596158872
Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/2376/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] akashrn5 commented on a change in pull request #3614:
[CARBONDATA-3693] Separate Index command from DataMap command
Posted by GitBox <gi...@apache.org>.
akashrn5 commented on a change in pull request #3614: [CARBONDATA-3693] Separate Index command from DataMap command
URL: https://github.com/apache/carbondata/pull/3614#discussion_r386206139
##########
File path: core/src/main/java/org/apache/carbondata/core/datamap/status/DataMapStatus.java
##########
@@ -18,7 +18,7 @@
package org.apache.carbondata.core.datamap.status;
/**
- * DataMap status
+ * Index status
*/
public enum DataMapStatus {
Review comment:
yes, since it is in core part and not specific to MV module, i thought can change this
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3614:
[CARBONDATA-3693] Separate Index command from DataMap command
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3614: [CARBONDATA-3693] Separate Index command from DataMap command
URL: https://github.com/apache/carbondata/pull/3614#issuecomment-587344079
Build Success with Spark 2.4.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.4/331/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3614:
[CARBONDATA-3693] Separate Index command from DataMap command
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3614: [CARBONDATA-3693] Separate Index command from DataMap command
URL: https://github.com/apache/carbondata/pull/3614#issuecomment-585160538
Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/1962/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] jackylk commented on a change in pull request #3614:
[CARBONDATA-3693] Separate Index command from DataMap command
Posted by GitBox <gi...@apache.org>.
jackylk commented on a change in pull request #3614: [CARBONDATA-3693] Separate Index command from DataMap command
URL: https://github.com/apache/carbondata/pull/3614#discussion_r385615513
##########
File path: core/src/main/java/org/apache/carbondata/core/datamap/status/DataMapStatus.java
##########
@@ -18,7 +18,7 @@
package org.apache.carbondata.core.datamap.status;
/**
- * DataMap status
+ * Index status
*/
public enum DataMapStatus {
Review comment:
But MV also uses this, right?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3614:
[CARBONDATA-3693] Separate Index command from DataMap command
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3614: [CARBONDATA-3693] Separate Index command from DataMap command
URL: https://github.com/apache/carbondata/pull/3614#issuecomment-596119881
Build Failed with Spark 2.4.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.4/669/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3614:
[CARBONDATA-3693] Separate Index command from DataMap command
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3614: [CARBONDATA-3693] Separate Index command from DataMap command
URL: https://github.com/apache/carbondata/pull/3614#issuecomment-602007334
Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/2530/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3614:
[CARBONDATA-3693] Separate Index command from DataMap command
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3614: [CARBONDATA-3693] Separate Index command from DataMap command
URL: https://github.com/apache/carbondata/pull/3614#issuecomment-596057275
Build Failed with Spark 2.4.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.4/657/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3614:
[CARBONDATA-3693] Separate Index command from DataMap command
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3614: [CARBONDATA-3693] Separate Index command from DataMap command
URL: https://github.com/apache/carbondata/pull/3614#issuecomment-596162117
Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/2377/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3614:
[CARBONDATA-3693] Separate Index command from DataMap command
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3614: [CARBONDATA-3693] Separate Index command from DataMap command
URL: https://github.com/apache/carbondata/pull/3614#issuecomment-596094303
Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/2371/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3614:
[CARBONDATA-3693] Separate Index command from DataMap command
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3614: [CARBONDATA-3693] Separate Index command from DataMap command
URL: https://github.com/apache/carbondata/pull/3614#issuecomment-596037617
Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/2361/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] akashrn5 commented on a change in pull request #3614:
[CARBONDATA-3693] Separate Index command from DataMap command
Posted by GitBox <gi...@apache.org>.
akashrn5 commented on a change in pull request #3614: [CARBONDATA-3693] Separate Index command from DataMap command
URL: https://github.com/apache/carbondata/pull/3614#discussion_r384036122
##########
File path: core/src/main/java/org/apache/carbondata/core/datamap/DataMapStoreManager.java
##########
@@ -168,6 +186,38 @@ public DataMapSchema getDataMapSchema(String dataMapName)
return provider.retrieveSchema(dataMapName);
}
+ public List<DataMapSchema> getAllMVSchemas() throws IOException {
+ return provider.retrieveAllSchemas().stream()
+ .filter(schema -> !schema.isIndexDataMap())
+ .collect(Collectors.toList());
+ }
+
+ public DataMapSchema getIndexSchema(String indexName)
+ throws NoSuchIndexException, IOException {
+ try {
+ DataMapSchema schema = provider.retrieveSchema(indexName);
+ if (!schema.isIndexDataMap()) {
+ throw new NoSuchIndexException(indexName);
+ }
+ return schema;
+ } catch (NoSuchDataMapException e) {
+ throw new NoSuchIndexException(indexName);
+ }
+ }
+
+ public DataMapSchema getMVSchema(String mvName)
+ throws NoSuchMaterializedViewException, IOException {
+ try {
+ DataMapSchema schema = provider.retrieveSchema(mvName);
+ if (schema == null || schema.isIndexDataMap()) {
+ throw new NoSuchMaterializedViewException(mvName);
+ }
+ return schema;
+ } catch (NoSuchDataMapException e) {
+ throw new NoSuchMaterializedViewException(mvName);
Review comment:
please add LOGGER.error() here and if no schema file is present, it will throw IO exception, if while reading schema failed IOexception is thrown, always saying NoSuchMaterializedViewException is misleading i think
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3614:
[CARBONDATA-3693] Separate Index command from DataMap command
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on issue #3614: [CARBONDATA-3693] Separate Index command from DataMap command
URL: https://github.com/apache/carbondata/pull/3614#issuecomment-585816220
Build Success with Spark 2.4.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.4/279/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services