You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@carbondata.apache.org by mohammadshahidkhan <gi...@git.apache.org> on 2018/06/20 14:27:17 UTC
[GitHub] carbondata pull request #2389: [CARBONDATA-2623][DataMap] Add DataMap Pre an...
GitHub user mohammadshahidkhan opened a pull request:
https://github.com/apache/carbondata/pull/2389
[CARBONDATA-2623][DataMap] Add DataMap Pre and Pevent listener
Added Pre and Post Execution Events for index datamap
- [X] Any interfaces changed?
None
- [X] Any backward compatibility impacted?
None
- [X] Document update required?
None
- [X] Testing done
Please provide details on
- Whether new unit test cases have been added or why no new tests are required?
- How it is tested? Please attach test report.
- Is it a performance related change? Please attach the performance test report.
- Any additional information to help reviewers in testing this change.
Test addition is not applicable
- [X] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.
NA
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/mohammadshahidkhan/incubator-carbondata bloom_internal_event
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/carbondata/pull/2389.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #2389
----
commit 0938102f3457e659e00783d1bf385c1b677ecfbc
Author: mohammadshahidkhan <mo...@...>
Date: 2018-06-20T14:22:51Z
[CARBONDATA-2623][DataMap] Add DataMap Pre and Pevent listener
----
---
[GitHub] carbondata issue #2389: [CARBONDATA-2623][DataMap] Add DataMap pre and post ...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2389
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/6444/
---
[GitHub] carbondata issue #2389: [CARBONDATA-2623][DataMap] Add DataMap Pre and Peven...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/2389
SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/5359/
---
[GitHub] carbondata issue #2389: [CARBONDATA-2623][DataMap] Add DataMap Pre and Peven...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2389
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/6421/
---
[GitHub] carbondata issue #2389: [CARBONDATA-2623][DataMap] Add DataMap pre and post ...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2389
Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/5271/
---
[GitHub] carbondata pull request #2389: [CARBONDATA-2623][DataMap] Add DataMap pre an...
Posted by manishgupta88 <gi...@git.apache.org>.
Github user manishgupta88 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2389#discussion_r197019227
--- Diff: integration/spark2/src/main/scala/org/apache/carbondata/spark/rdd/CarbonTableCompactor.scala ---
@@ -272,6 +284,13 @@ class CarbonTableCompactor(carbonLoadModel: CarbonLoadModel,
mergedLoadName)
OperationListenerBus.getInstance()
.fireEvent(compactionLoadStatusPostEvent, operationContext)
+ if (null != tableDataMaps) {
+ val buildDataMapPostExecutionEvent: BuildDataMapPostExecutionEvent =
+ new BuildDataMapPostExecutionEvent(sqlContext.sparkSession,
--- End diff --
Above LoadDataMapPreExecutionEvent is used and here BuildDataMapPostExecutionEvent is used. Check if this is correct. I think above it should be BuildDataMapPreExecutionEvent OR here it should be LoadDataMapPostExecutionEvent
---
[GitHub] carbondata pull request #2389: [CARBONDATA-2623][DataMap] Add DataMap pre an...
Posted by manishgupta88 <gi...@git.apache.org>.
Github user manishgupta88 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2389#discussion_r197019345
--- Diff: integration/spark-common/src/main/scala/org/apache/carbondata/events/DataMapEvents.scala ---
@@ -0,0 +1,76 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.carbondata.events
+
+import org.apache.spark.sql.SparkSession
+
+import org.apache.carbondata.core.metadata.AbsoluteTableIdentifier
+
+/**
+ * For handling operation's after finish of index creation over table with index datamap
+ * example: bloom datamap, Lucene datamap
+ */
+case class CreateDataMapPostExecutionEvent(sparkSession: SparkSession,
+ storePath: String) extends Event with CreateDataMapEventsInfo
+
+/**
+ * For handling operation's before start of update index datmap status over table with index datamap
+ * example: bloom datamap, Lucene datamap
+ */
+case class UpdateDataMapPreExecutionEvent(sparkSession: SparkSession,
+ storePath: String) extends Event with CreateDataMapEventsInfo
+
+/**
+ * For handling operation's after finish of update index datmap status over table with index
+ * datamap
+ * example: bloom datamap, Lucene datamap
+ */
+case class UpdateDataMapPostExecutionEvent(sparkSession: SparkSession,
+ storePath: String) extends Event with CreateDataMapEventsInfo
+
+/**
+ * For handling operation's before start of index build over table with index datamap
+ * example: bloom datamap, Lucene datamap
+ */
+case class BuildDataMapPreExecutionEvent(sparkSession: SparkSession,
+ identifier: AbsoluteTableIdentifier, dataMapNames: scala.collection.mutable.Seq[String])
+ extends Event with BuildDataMapEventsInfo
+
+/**
+ * For handling operation's before start of index build during load over table with index datamap
+ * example: bloom datamap, Lucene datamap
+ */
+case class LoadDataMapPreExecutionEvent(sparkSession: SparkSession,
--- End diff --
check if we can remove LoadDataMap event and instead use BuildDataMap events
---
[GitHub] carbondata pull request #2389: [CARBONDATA-2623][DataMap] Add DataMap pre an...
Posted by mohammadshahidkhan <gi...@git.apache.org>.
Github user mohammadshahidkhan commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2389#discussion_r197025960
--- Diff: integration/spark2/src/main/scala/org/apache/carbondata/spark/rdd/CarbonTableCompactor.scala ---
@@ -272,6 +284,13 @@ class CarbonTableCompactor(carbonLoadModel: CarbonLoadModel,
mergedLoadName)
OperationListenerBus.getInstance()
.fireEvent(compactionLoadStatusPostEvent, operationContext)
+ if (null != tableDataMaps) {
+ val buildDataMapPostExecutionEvent: BuildDataMapPostExecutionEvent =
+ new BuildDataMapPostExecutionEvent(sqlContext.sparkSession,
--- End diff --
Changed to BuildDataMapPreExecutionEvent
---
[GitHub] carbondata issue #2389: [CARBONDATA-2623][DataMap] Add DataMap pre and post ...
Posted by manishgupta88 <gi...@git.apache.org>.
Github user manishgupta88 commented on the issue:
https://github.com/apache/carbondata/pull/2389
LGTM
---
[GitHub] carbondata pull request #2389: [CARBONDATA-2623][DataMap] Add DataMap pre an...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/carbondata/pull/2389
---
[GitHub] carbondata issue #2389: [CARBONDATA-2623][DataMap] Add DataMap pre and post ...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2389
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/6440/
---
[GitHub] carbondata pull request #2389: [CARBONDATA-2623][DataMap] Add DataMap pre an...
Posted by mohammadshahidkhan <gi...@git.apache.org>.
Github user mohammadshahidkhan commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2389#discussion_r197026107
--- Diff: integration/spark2/src/main/scala/org/apache/carbondata/spark/rdd/CarbonTableCompactor.scala ---
@@ -272,6 +284,13 @@ class CarbonTableCompactor(carbonLoadModel: CarbonLoadModel,
mergedLoadName)
OperationListenerBus.getInstance()
.fireEvent(compactionLoadStatusPostEvent, operationContext)
+ if (null != tableDataMaps) {
+ val buildDataMapPostExecutionEvent: BuildDataMapPostExecutionEvent =
+ new BuildDataMapPostExecutionEvent(sqlContext.sparkSession,
--- End diff --
Fixed changed to BuildDataMapPreExecutionEvent .
---
[GitHub] carbondata issue #2389: [CARBONDATA-2623][DataMap] Add DataMap pre and post ...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2389
Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/5276/
---
[GitHub] carbondata pull request #2389: [CARBONDATA-2623][DataMap] Add DataMap pre an...
Posted by manishgupta88 <gi...@git.apache.org>.
Github user manishgupta88 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2389#discussion_r197027345
--- Diff: integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/datamap/CarbonCreateDataMapCommand.scala ---
@@ -128,7 +140,17 @@ case class CarbonCreateDataMapCommand(
if (mainTable != null && !deferredRebuild) {
dataMapProvider.rebuild()
if (dataMapSchema.isIndexDataMap) {
+ val operationContext: OperationContext = new OperationContext()
+ val systemFolderLocation: String = CarbonProperties.getInstance().getSystemFolderLocation
+ val createDataMapPreExecutionEvent: CreateDataMapPreExecutionEvent =
+ new CreateDataMapPreExecutionEvent(sparkSession, systemFolderLocation)
+ OperationListenerBus.getInstance().fireEvent(createDataMapPreExecutionEvent,
+ operationContext)
DataMapStatusManager.enableDataMap(dataMapName)
+ val createDataMapPostExecutionEvent: CreateDataMapPostExecutionEvent =
+ new CreateDataMapPostExecutionEvent(sparkSession, systemFolderLocation)
+ OperationListenerBus.getInstance().fireEvent(createDataMapPostExecutionEvent,
+ operationContext)
--- End diff --
here it should be UpdateDataMapPreExecutionEvent and UpdateDataMapPostExecutionEvent instead of createDataMapPreExecutionEvent and createDataMapPostExecutionEvent
---
[GitHub] carbondata issue #2389: [CARBONDATA-2623][DataMap] Add DataMap Pre and Peven...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2389
Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/5255/
---