You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@carbondata.apache.org by ravipesala <gi...@git.apache.org> on 2018/01/03 14:08:09 UTC
[GitHub] carbondata pull request #1757: [CARBONDATA-1977] Fix aggregation table loadi...
GitHub user ravipesala opened a pull request:
https://github.com/apache/carbondata/pull/1757
[CARBONDATA-1977] Fix aggregation table loading after loading of partition table.
Aggregate tables are not loading for the partition tables. Because the load events are not fired during the partition table load.
Be sure to do all of the following checklist to help us incorporate
your contribution quickly and easily:
- [X] Any interfaces changed?
- [X] Any backward compatibility impacted?
- [X] Document update required?
- [X] Testing done
Tests added
- [X] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/ravipesala/incubator-carbondata aggregation-table-load-partition
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/carbondata/pull/1757.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1757
----
commit 24d64b51777f9f2bdbbbc1ca35be326db9367114
Author: ravipesala <ra...@...>
Date: 2018-01-03T11:32:33Z
Fix aggregation table loading after loading of partition table.
----
---
[GitHub] carbondata pull request #1757: [CARBONDATA-1977][PARTITION] Fix aggregation ...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/carbondata/pull/1757
---
[GitHub] carbondata issue #1757: [CARBONDATA-1977][PARTITION] Fix aggregation table l...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1757
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/2572/
---
[GitHub] carbondata issue #1757: [CARBONDATA-1977][PARTITION] Fix aggregation table l...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1757
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/2584/
---
[GitHub] carbondata issue #1757: [CARBONDATA-1977][PARTITION] Fix aggregation table l...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1757
Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/1348/
---
[GitHub] carbondata issue #1757: [CARBONDATA-1977][PARTITION] Fix aggregation table l...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1757
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/2586/
---
[GitHub] carbondata pull request #1757: [CARBONDATA-1977][PARTITION] Fix aggregation ...
Posted by jackylk <gi...@git.apache.org>.
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1757#discussion_r159605621
--- Diff: processing/src/main/java/org/apache/carbondata/processing/loading/events/LoadEvents.java ---
@@ -0,0 +1,152 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.carbondata.processing.loading.events;
+
+import java.util.Map;
+
+import org.apache.carbondata.core.metadata.CarbonTableIdentifier;
+import org.apache.carbondata.events.Event;
+import org.apache.carbondata.processing.loading.model.CarbonLoadModel;
+
+public class LoadEvents {
+ /**
+ * Class for handling operations before start of a load process.
+ * Example usage: For validation purpose
+ */
+ public static class LoadTablePreExecutionEvent extends Event {
+ private CarbonTableIdentifier carbonTableIdentifier;
+ private CarbonLoadModel carbonLoadModel;
+ private String factPath;
+ private boolean isDataFrameDefined;
+ private Map<String, String> optionsFinal;
+ // userProvidedOptions are needed if we need only the load options given by user
+ private Map<String, String> userProvidedOptions;
+ private boolean isOverWriteTable;
+
+ public LoadTablePreExecutionEvent(CarbonTableIdentifier carbonTableIdentifier,
+ CarbonLoadModel carbonLoadModel, String factPath, boolean isDataFrameDefined,
+ Map<String, String> optionsFinal, Map<String, String> userProvidedOptions,
+ boolean isOverWriteTable) {
+ this.carbonTableIdentifier = carbonTableIdentifier;
+ this.carbonLoadModel = carbonLoadModel;
+ this.factPath = factPath;
+ this.isDataFrameDefined = isDataFrameDefined;
+ this.optionsFinal = optionsFinal;
+ this.userProvidedOptions = userProvidedOptions;
+ this.isOverWriteTable = isOverWriteTable;
+ }
+
+ public CarbonTableIdentifier getCarbonTableIdentifier() {
+ return carbonTableIdentifier;
+ }
+
+ public CarbonLoadModel getCarbonLoadModel() {
+ return carbonLoadModel;
+ }
+
+ public String getFactPath() {
+ return factPath;
+ }
+
+ public boolean isDataFrameDefined() {
+ return isDataFrameDefined;
+ }
+
+ public Map<String, String> getOptionsFinal() {
+ return optionsFinal;
+ }
+
+ public Map<String, String> getUserProvidedOptions() {
+ return userProvidedOptions;
+ }
+
+ public boolean isOverWriteTable() {
+ return isOverWriteTable;
+ }
+ }
+
+ /**
+ * Class for handling operations after data load completion and before final
+ * commit of load operation. Example usage: For loading pre-aggregate tables
--- End diff --
I remember, we will move all command execution out of event listener, right?
Otherwise, metadata and data process may be wrong
---
[GitHub] carbondata issue #1757: [CARBONDATA-1977][PARTITION] Fix aggregation table l...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1757
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/2541/
---
[GitHub] carbondata issue #1757: [CARBONDATA-1977][PARTITION] Fix aggregation table l...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1757
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/2521/
---
[GitHub] carbondata issue #1757: [CARBONDATA-1977][PARTITION] Fix aggregation table l...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1757
Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/1341/
---
[GitHub] carbondata issue #1757: [CARBONDATA-1977][PARTITION] Fix aggregation table l...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1757
Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/1333/
---
[GitHub] carbondata pull request #1757: [CARBONDATA-1977][PARTITION] Fix aggregation ...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1757#discussion_r159665801
--- Diff: processing/src/main/java/org/apache/carbondata/processing/loading/events/LoadEvents.java ---
@@ -0,0 +1,152 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.carbondata.processing.loading.events;
+
+import java.util.Map;
+
+import org.apache.carbondata.core.metadata.CarbonTableIdentifier;
+import org.apache.carbondata.events.Event;
+import org.apache.carbondata.processing.loading.model.CarbonLoadModel;
+
+public class LoadEvents {
+ /**
+ * Class for handling operations before start of a load process.
+ * Example usage: For validation purpose
+ */
+ public static class LoadTablePreExecutionEvent extends Event {
+ private CarbonTableIdentifier carbonTableIdentifier;
+ private CarbonLoadModel carbonLoadModel;
+ private String factPath;
+ private boolean isDataFrameDefined;
+ private Map<String, String> optionsFinal;
+ // userProvidedOptions are needed if we need only the load options given by user
+ private Map<String, String> userProvidedOptions;
+ private boolean isOverWriteTable;
+
+ public LoadTablePreExecutionEvent(CarbonTableIdentifier carbonTableIdentifier,
+ CarbonLoadModel carbonLoadModel, String factPath, boolean isDataFrameDefined,
+ Map<String, String> optionsFinal, Map<String, String> userProvidedOptions,
+ boolean isOverWriteTable) {
+ this.carbonTableIdentifier = carbonTableIdentifier;
+ this.carbonLoadModel = carbonLoadModel;
+ this.factPath = factPath;
+ this.isDataFrameDefined = isDataFrameDefined;
+ this.optionsFinal = optionsFinal;
+ this.userProvidedOptions = userProvidedOptions;
+ this.isOverWriteTable = isOverWriteTable;
+ }
+
+ public CarbonTableIdentifier getCarbonTableIdentifier() {
+ return carbonTableIdentifier;
+ }
+
+ public CarbonLoadModel getCarbonLoadModel() {
+ return carbonLoadModel;
+ }
+
+ public String getFactPath() {
+ return factPath;
+ }
+
+ public boolean isDataFrameDefined() {
+ return isDataFrameDefined;
+ }
+
+ public Map<String, String> getOptionsFinal() {
+ return optionsFinal;
+ }
+
+ public Map<String, String> getUserProvidedOptions() {
+ return userProvidedOptions;
+ }
+
+ public boolean isOverWriteTable() {
+ return isOverWriteTable;
+ }
+ }
+
+ /**
+ * Class for handling operations after data load completion and before final
+ * commit of load operation. Example usage: For loading pre-aggregate tables
--- End diff --
Load always be there in processdata, so there should not be any issue
---
[GitHub] carbondata issue #1757: [CARBONDATA-1977][PARTITION] Fix aggregation table l...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/1757
SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/2701/
---
[GitHub] carbondata issue #1757: [CARBONDATA-1977][PARTITION] Fix aggregation table l...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1757
Build Failed with Spark 2.2.0, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/1291/
---
[GitHub] carbondata issue #1757: [CARBONDATA-1977][PARTITION] Fix aggregation table l...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1757
Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/1312/
---
[GitHub] carbondata issue #1757: [CARBONDATA-1977][PARTITION] Fix aggregation table l...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/1757
SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/2685/
---
[GitHub] carbondata issue #1757: [CARBONDATA-1977][PARTITION] Fix aggregation table l...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1757
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/2557/
---
[GitHub] carbondata issue #1757: [CARBONDATA-1977][PARTITION] Fix aggregation table l...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/1757
SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/2750/
---
[GitHub] carbondata issue #1757: [CARBONDATA-1977][PARTITION] Fix aggregation table l...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/1757
SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/2749/
---
[GitHub] carbondata issue #1757: [CARBONDATA-1977][PARTITION] Fix aggregation table l...
Posted by jackylk <gi...@git.apache.org>.
Github user jackylk commented on the issue:
https://github.com/apache/carbondata/pull/1757
LGTM
---
[GitHub] carbondata issue #1757: [CARBONDATA-1977][PARTITION] Fix aggregation table l...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/1757
SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/2729/
---