You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@carbondata.apache.org by kunal642 <gi...@git.apache.org> on 2017/03/15 09:41:04 UTC
[GitHub] incubator-carbondata pull request #656: [CARBONDATA-773] Fixed multiple Dict...
GitHub user kunal642 opened a pull request:
https://github.com/apache/incubator-carbondata/pull/656
[CARBONDATA-773] Fixed multiple DictionaryServer instances issue
Fixed the issue where multiple dictionary server instances were being created for parallel load.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/kunal642/incubator-carbondata dictionary_fix
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-carbondata/pull/656.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #656
----
commit 3a29c2f995f35df7d166e992891ad6b2f27a7823
Author: kunal642 <ku...@knoldus.in>
Date: 2017-03-15T09:08:31Z
fixed multiple dictionary server issue
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-carbondata issue #656: [WIP][CARBONDATA-773] Fixed multiple Dictio...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/incubator-carbondata/pull/656
Build Failed with Spark 1.6.2, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/1536/
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-carbondata issue #656: [WIP][CARBONDATA-773] Fixed multiple Dictio...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/incubator-carbondata/pull/656
Build Success with Spark 1.6.2, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/1538/
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-carbondata issue #656: [CARBONDATA-773] Fixed multiple DictionaryS...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/incubator-carbondata/pull/656
Build Success with Spark 1.6.2, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/1208/
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-carbondata issue #656: [WIP] Fixed multiple DictionaryServer insta...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/incubator-carbondata/pull/656
Build Success with Spark 1.6.2, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/1189/
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-carbondata issue #656: [CARBONDATA-773] Fixed multiple DictionaryS...
Posted by kunal642 <gi...@git.apache.org>.
Github user kunal642 commented on the issue:
https://github.com/apache/incubator-carbondata/pull/656
@ravipesala changes made. Please review
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-carbondata pull request #656: [CARBONDATA-773] Fixed multiple Dict...
Posted by gvramana <gi...@git.apache.org>.
Github user gvramana commented on a diff in the pull request:
https://github.com/apache/incubator-carbondata/pull/656#discussion_r106593403
--- Diff: core/src/main/java/org/apache/carbondata/core/dictionary/server/DictionaryServer.java ---
@@ -43,42 +43,71 @@
private EventLoopGroup boss;
private EventLoopGroup worker;
+ private int port;
+ private static DictionaryServer INSTANCE = null;
+
+ private DictionaryServer(int port) {
+ startServer(port);
+ }
+
+ public static DictionaryServer getInstance(int port) {
+ if (INSTANCE == null) {
--- End diff --
Dictionary server creation should be thread safe, use double check lock .
if(INSTANCE == null){
aquirelock
if(INSTANCE==null){
createinstance
}
releaselock
}
return instance
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-carbondata issue #656: [WIP][CARBONDATA-773] Fixed multiple Dictio...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/incubator-carbondata/pull/656
Build Success with Spark 1.6.2, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/1535/
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-carbondata pull request #656: [CARBONDATA-773] Fixed multiple Dict...
Posted by gvramana <gi...@git.apache.org>.
Github user gvramana commented on a diff in the pull request:
https://github.com/apache/incubator-carbondata/pull/656#discussion_r106596051
--- Diff: integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/carbonTableSchema.scala ---
@@ -524,8 +524,9 @@ case class LoadTable(
@throws[Exception]
def call: DictionaryServer = {
Thread.currentThread().setName("Dictionary server")
- val server: DictionaryServer = new DictionaryServer
- server.startServer(dictionaryServerPort.toInt)
+ val server: DictionaryServer = DictionaryServer
--- End diff --
Dataloading shouldnot continue if dictionary server is failure.
port initialization should happen and then port should be passed to LoadRDD
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-carbondata issue #656: [CARBONDATA-773] Fixed multiple DictionaryS...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/incubator-carbondata/pull/656
Build Success with Spark 1.6.2, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/1153/
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-carbondata issue #656: [WIP] Fixed multiple DictionaryServer insta...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/incubator-carbondata/pull/656
Build Failed with Spark 1.6.2, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/1188/
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-carbondata issue #656: [CARBONDATA-773] Fixed multiple DictionaryS...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/incubator-carbondata/pull/656
Build Failed with Spark 1.6.2, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/1204/
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-carbondata issue #656: [WIP][CARBONDATA-773] Fixed multiple Dictio...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/incubator-carbondata/pull/656
Build Success with Spark 1.6.2, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/1537/
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-carbondata issue #656: [CARBONDATA-773] Fixed multiple DictionaryS...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/incubator-carbondata/pull/656
@kunal642 It is not all about avoid multiple server instances, we always shutdown after load completes so even if you avoid creating multiple instance of server you cannot do second load as you already shutdown the server. So we need to come with proper solution.
Solution 1 -> Start the dictionary server once per driver and avoid shutting down after load completes. But make sure that dictionary data is flushed to store after load completion of that table.
Solution 2 -> Always starts new dictionary server on new port (first check the free port availability).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-carbondata issue #656: [CARBONDATA-773] Fixed multiple DictionaryS...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/incubator-carbondata/pull/656
Build Success with Spark 1.6.2, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/1357/
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-carbondata issue #656: [CARBONDATA-773] Fixed multiple DictionaryS...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/incubator-carbondata/pull/656
Build Failed with Spark 1.6.2, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/1155/
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-carbondata pull request #656: [CARBONDATA-773] Fixed multiple Dict...
Posted by gvramana <gi...@git.apache.org>.
Github user gvramana commented on a diff in the pull request:
https://github.com/apache/incubator-carbondata/pull/656#discussion_r106593774
--- Diff: core/src/main/java/org/apache/carbondata/core/dictionary/server/DictionaryServer.java ---
@@ -43,42 +43,71 @@
private EventLoopGroup boss;
private EventLoopGroup worker;
+ private int port;
+ private static DictionaryServer INSTANCE = null;
+
+ private DictionaryServer(int port) {
+ startServer(port);
+ }
+
+ public static DictionaryServer getInstance(int port) {
+ if (INSTANCE == null) {
+ INSTANCE = new DictionaryServer(port);
+ }
+ return INSTANCE;
+ }
/**
* start dictionary server
*
* @param port
* @throws Exception
*/
- public void startServer(int port) {
- long start = System.currentTimeMillis();
+ private void startServer(int port) {
dictionaryServerHandler = new DictionaryServerHandler();
boss = new NioEventLoopGroup();
worker = new NioEventLoopGroup();
// Configure the server.
+ this.port = bindToPort(port, 0);
+ }
+
+ private int bindToPort(int port, int retries) {
+ long start = System.currentTimeMillis();
--- End diff --
No need for recursion, directly use loop
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-carbondata issue #656: [CARBONDATA-773] Fixed multiple DictionaryS...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/incubator-carbondata/pull/656
Build Failed with Spark 1.6.2, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/1355/
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-carbondata pull request #656: [CARBONDATA-773] Fixed multiple Dict...
Posted by gvramana <gi...@git.apache.org>.
Github user gvramana commented on a diff in the pull request:
https://github.com/apache/incubator-carbondata/pull/656#discussion_r106595337
--- Diff: integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/carbonTableSchema.scala ---
@@ -514,7 +514,7 @@ case class LoadTable(
val dictionaryServerPort = CarbonProperties.getInstance()
.getProperty(CarbonCommonConstants.DICTIONARY_SERVER_PORT,
CarbonCommonConstants.DICTIONARY_SERVER_PORT_DEFAULT)
- carbonLoadModel.setDictionaryServerPort(Integer.parseInt(dictionaryServerPort))
+ carbonLoadModel.setDictionaryServerPort(dictionaryServerPort.toInt)
--- End diff --
remove this
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-carbondata pull request #656: [CARBONDATA-773] Fixed multiple Dict...
Posted by gvramana <gi...@git.apache.org>.
Github user gvramana commented on a diff in the pull request:
https://github.com/apache/incubator-carbondata/pull/656#discussion_r106595169
--- Diff: integration/spark/src/main/scala/org/apache/carbondata/spark/rdd/CarbonDataRDDFactory.scala ---
@@ -963,15 +962,14 @@ object CarbonDataRDDFactory {
.getTableName
}")
LOGGER.error("Dataload failed due to failure in table status updation.")
- shutDownDictionaryServer(carbonLoadModel, result, false)
throw new Exception(errorMessage)
}
} else if (!carbonLoadModel.isRetentionRequest) {
// TODO : Handle it
LOGGER.info("********Database updated**********")
}
- shutDownDictionaryServer(carbonLoadModel, result)
+ writeDictionary(carbonLoadModel, result)
--- End diff --
Create a new WriteTableDictionary method/message to ensure only that only one table dictionary is flushed, while allowing other dictionary generation to continue.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-carbondata issue #656: [CARBONDATA-773] Fixed multiple DictionaryS...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/incubator-carbondata/pull/656
Build Failed with Spark 1.6.2, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/1247/
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-carbondata issue #656: [CARBONDATA-773] Fixed multiple DictionaryS...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/incubator-carbondata/pull/656
Build Success with Spark 1.6.2, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/1205/
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-carbondata pull request #656: [CARBONDATA-773] Fixed multiple Dict...
Posted by gvramana <gi...@git.apache.org>.
Github user gvramana commented on a diff in the pull request:
https://github.com/apache/incubator-carbondata/pull/656#discussion_r106594167
--- Diff: integration/spark/src/main/scala/org/apache/carbondata/spark/rdd/CarbonDataRDDFactory.scala ---
@@ -963,15 +962,14 @@ object CarbonDataRDDFactory {
.getTableName
}")
LOGGER.error("Dataload failed due to failure in table status updation.")
- shutDownDictionaryServer(carbonLoadModel, result, false)
throw new Exception(errorMessage)
}
} else if (!carbonLoadModel.isRetentionRequest) {
// TODO : Handle it
LOGGER.info("********Database updated**********")
}
- shutDownDictionaryServer(carbonLoadModel, result)
+ writeDictionary(carbonLoadModel, result)
--- End diff --
Write dictionary should happen before metadata update, if any failure in writedictionary should fail data load.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-carbondata issue #656: [CARBONDATA-773] Fixed multiple DictionaryS...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/incubator-carbondata/pull/656
Build Failed with Spark 1.6.2, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/1206/
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---