You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@tajo.apache.org by navis <gi...@git.apache.org> on 2015/03/18 10:08:52 UTC
[GitHub] tajo pull request: TAJO-1412 Add polling API in TajoMasterClientPr...
GitHub user navis opened a pull request:
https://github.com/apache/tajo/pull/432
TAJO-1412 Add polling API in TajoMasterClientProtocol to wait result effectively
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/navis/tajo TAJO-1412
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/tajo/pull/432.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #432
----
commit d8088f73bd61b061f6999283b4f83fbd01d1f7e1
Author: navis.ryu <na...@apache.org>
Date: 2015-03-18T09:01:40Z
TAJO-1412 Add polling API in TajoMasterClientProtocol to wait result effectively
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] tajo pull request: TAJO-1412 Add polling API in TajoMasterClientPr...
Posted by jihoonson <gi...@git.apache.org>.
Github user jihoonson commented on a diff in the pull request:
https://github.com/apache/tajo/pull/432#discussion_r28409499
--- Diff: tajo-core/src/main/java/org/apache/tajo/master/QueryInfo.java ---
@@ -119,14 +119,37 @@ public int getQueryMasterClientPort() {
return queryMasterClientPort;
}
- public TajoProtos.QueryState getQueryState() {
+ public synchronized TajoProtos.QueryState getQueryState() {
--- End diff --
It seems that all ```synchronized``` declaration is redundant.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] tajo pull request: TAJO-1412 Add polling API in TajoMasterClientPr...
Posted by navis <gi...@git.apache.org>.
Github user navis commented on a diff in the pull request:
https://github.com/apache/tajo/pull/432#discussion_r28468938
--- Diff: tajo-core/src/main/java/org/apache/tajo/master/QueryInfo.java ---
@@ -119,14 +119,37 @@ public int getQueryMasterClientPort() {
return queryMasterClientPort;
}
- public TajoProtos.QueryState getQueryState() {
+ public synchronized TajoProtos.QueryState getQueryState() {
--- End diff --
queryState is volatile, so it does not need a lock. But for waitState method, I need a lock to synchronize. I just choosen "this" not to add any other object.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] tajo pull request: TAJO-1412 Add polling API in TajoMasterClientPr...
Posted by jihoonson <gi...@git.apache.org>.
Github user jihoonson commented on the pull request:
https://github.com/apache/tajo/pull/432#issuecomment-93625436
I'll finish my review soon.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] tajo pull request: TAJO-1412 Add polling API in TajoMasterClientPr...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/tajo/pull/432
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] tajo pull request: TAJO-1412 Add polling API in TajoMasterClientPr...
Posted by jihoonson <gi...@git.apache.org>.
Github user jihoonson commented on a diff in the pull request:
https://github.com/apache/tajo/pull/432#discussion_r28477237
--- Diff: tajo-core/src/main/java/org/apache/tajo/master/QueryInfo.java ---
@@ -119,14 +119,37 @@ public int getQueryMasterClientPort() {
return queryMasterClientPort;
}
- public TajoProtos.QueryState getQueryState() {
+ public synchronized TajoProtos.QueryState getQueryState() {
--- End diff --
Oh, I got your point.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] tajo pull request: TAJO-1412 Add polling API in TajoMasterClientPr...
Posted by jihoonson <gi...@git.apache.org>.
Github user jihoonson commented on the pull request:
https://github.com/apache/tajo/pull/432#issuecomment-89796828
Thanks for your contribution.
The patch looks good to me. I left one comment.
If you agree, I'll commit.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] tajo pull request: TAJO-1412 Add polling API in TajoMasterClientPr...
Posted by jihoonson <gi...@git.apache.org>.
Github user jihoonson commented on a diff in the pull request:
https://github.com/apache/tajo/pull/432#discussion_r28477732
--- Diff: tajo-core/src/main/java/org/apache/tajo/master/QueryInfo.java ---
@@ -119,14 +119,37 @@ public int getQueryMasterClientPort() {
return queryMasterClientPort;
}
- public TajoProtos.QueryState getQueryState() {
+ public synchronized TajoProtos.QueryState getQueryState() {
return queryState;
}
- public void setQueryState(TajoProtos.QueryState queryState) {
+ public synchronized boolean isTerminalState() {
+ return isTerminalState(queryState);
+ }
+
+ public static boolean isTerminalState(TajoProtos.QueryState state) {
+ return state == TajoProtos.QueryState.QUERY_FAILED ||
+ state == TajoProtos.QueryState.QUERY_ERROR ||
+ state == TajoProtos.QueryState.QUERY_KILLED ||
+ state == TajoProtos.QueryState.QUERY_SUCCEEDED;
+ }
+
+ public synchronized void setQueryState(TajoProtos.QueryState queryState) {
this.queryState = queryState;
+ notifyAll();
}
+ public synchronized boolean waitState(TajoProtos.QueryState expect, long timeout)
+ throws InterruptedException {
+ long prev = System.currentTimeMillis();
+ while (timeout > 0 && expect != queryState && !isTerminalState()) {
+ wait(timeout);
--- End diff --
I think that this approach has a potential problem of process hang. Once wait() is called, it can wake up only when setQueryState() is called. During query processing, setQueryState() is called only when TajoMaster receives heartbeats from workers. That means, if some heartbeats are lost, this process cannot wake up even though waiting time exceeds the given timeout.
This will be not a big problem because we usually expect to receive at least one heartbeat from a lot of workers. However, I think that it is still a problem because the process's work can be different from our intention.
What do you think?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] tajo pull request: TAJO-1412 Add polling API in TajoMasterClientPr...
Posted by navis <gi...@git.apache.org>.
Github user navis commented on a diff in the pull request:
https://github.com/apache/tajo/pull/432#discussion_r28479952
--- Diff: tajo-core/src/main/java/org/apache/tajo/master/QueryInfo.java ---
@@ -119,14 +119,37 @@ public int getQueryMasterClientPort() {
return queryMasterClientPort;
}
- public TajoProtos.QueryState getQueryState() {
+ public synchronized TajoProtos.QueryState getQueryState() {
return queryState;
}
- public void setQueryState(TajoProtos.QueryState queryState) {
+ public synchronized boolean isTerminalState() {
+ return isTerminalState(queryState);
+ }
+
+ public static boolean isTerminalState(TajoProtos.QueryState state) {
+ return state == TajoProtos.QueryState.QUERY_FAILED ||
+ state == TajoProtos.QueryState.QUERY_ERROR ||
+ state == TajoProtos.QueryState.QUERY_KILLED ||
+ state == TajoProtos.QueryState.QUERY_SUCCEEDED;
+ }
+
+ public synchronized void setQueryState(TajoProtos.QueryState queryState) {
this.queryState = queryState;
+ notifyAll();
}
+ public synchronized boolean waitState(TajoProtos.QueryState expect, long timeout)
+ throws InterruptedException {
+ long prev = System.currentTimeMillis();
+ while (timeout > 0 && expect != queryState && !isTerminalState()) {
+ wait(timeout);
--- End diff --
wait(timeout) waits on lock for timeout msec. Isn't it?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] tajo pull request: TAJO-1412 Add polling API in TajoMasterClientPr...
Posted by jihoonson <gi...@git.apache.org>.
Github user jihoonson commented on the pull request:
https://github.com/apache/tajo/pull/432#issuecomment-93780346
Hi @navis. I've reviewed this patch carefully.
Sorry to say, but I think that this is not appropriate for our architecture.
When a client calls ```pollQueryStatus()```, the process who waits for the query state to be changed is ```TajoMaster```'s rpc server. As you know, we will support multi-tenancy soon. In multi-tenant environment, master's rpc server will be getting slower as more clients submit queries and wait for their results. IMHO, polling in only client will be better than blocking master.
What do you think?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] tajo pull request: TAJO-1412 Add polling API in TajoMasterClientPr...
Posted by jihoonson <gi...@git.apache.org>.
Github user jihoonson commented on a diff in the pull request:
https://github.com/apache/tajo/pull/432#discussion_r28481129
--- Diff: tajo-core/src/main/java/org/apache/tajo/master/QueryInfo.java ---
@@ -119,14 +119,37 @@ public int getQueryMasterClientPort() {
return queryMasterClientPort;
}
- public TajoProtos.QueryState getQueryState() {
+ public synchronized TajoProtos.QueryState getQueryState() {
return queryState;
}
- public void setQueryState(TajoProtos.QueryState queryState) {
+ public synchronized boolean isTerminalState() {
+ return isTerminalState(queryState);
+ }
+
+ public static boolean isTerminalState(TajoProtos.QueryState state) {
+ return state == TajoProtos.QueryState.QUERY_FAILED ||
+ state == TajoProtos.QueryState.QUERY_ERROR ||
+ state == TajoProtos.QueryState.QUERY_KILLED ||
+ state == TajoProtos.QueryState.QUERY_SUCCEEDED;
+ }
+
+ public synchronized void setQueryState(TajoProtos.QueryState queryState) {
this.queryState = queryState;
+ notifyAll();
}
+ public synchronized boolean waitState(TajoProtos.QueryState expect, long timeout)
+ throws InterruptedException {
+ long prev = System.currentTimeMillis();
+ while (timeout > 0 && expect != queryState && !isTerminalState()) {
+ wait(timeout);
--- End diff --
Oh, you are right. I made so much mistakes due to lack of sleep.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] tajo pull request: TAJO-1412 Add polling API in TajoMasterClientPr...
Posted by jihoonson <gi...@git.apache.org>.
Github user jihoonson commented on a diff in the pull request:
https://github.com/apache/tajo/pull/432#discussion_r27778283
--- Diff: tajo-core/src/main/java/org/apache/tajo/master/QueryInfo.java ---
@@ -119,14 +119,37 @@ public int getQueryMasterClientPort() {
return queryMasterClientPort;
}
- public TajoProtos.QueryState getQueryState() {
+ public synchronized TajoProtos.QueryState getQueryState() {
--- End diff --
Declaring ```synchronized``` is redundant. QueryState is changed by only the QueryInProgress.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---