You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@ambari.apache.org by "JiaLiangC (via GitHub)" <gi...@apache.org> on 2023/03/24 07:06:24 UTC
[GitHub] [ambari] JiaLiangC opened a new pull request, #3672: AMBARI-25912: use LinkedBlockingQueue in AgentsRegistrationQueue for performance improvement
JiaLiangC opened a new pull request, #3672:
URL: https://github.com/apache/ambari/pull/3672
## What changes were proposed in this pull request?
#### Background
The AgentRegisteringQueueChecker is an interceptor used during the long connection between the agent and server, mainly for requesting congestion control. It internally uses AgentsRegistrationQueue to store the sessionID of connections.
The sessionID is added to the AgentsRegistrationQueue when the ambari agent registers to the server or sends a heartbeat.
When the long connection between the agent and server is disconnected, or when the server completes the heartbeat request, the sessionID is removed from the AgentsRegistrationQueue.
When the queue is full, the server returns "not allowed" to the agent.
#### Solution
AgentsRegistrationQueue uses ArrayBlockingQueue, which uses the same lock for both writing and consuming, making it easy to negatively impact performance when dealing with large clusters. Therefore, we are switching to using LinkedBlockingQueue in order to improve performance.
## How was this patch tested?
unit tests
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@ambari.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@ambari.apache.org
For additional commands, e-mail: dev-help@ambari.apache.org
[GitHub] [ambari] virajjasani merged pull request #3672: AMBARI-25912: use LinkedBlockingQueue in AgentsRegistrationQueue for performance improvement
Posted by "virajjasani (via GitHub)" <gi...@apache.org>.
virajjasani merged PR #3672:
URL: https://github.com/apache/ambari/pull/3672
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@ambari.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@ambari.apache.org
For additional commands, e-mail: dev-help@ambari.apache.org
[GitHub] [ambari] virajjasani commented on pull request #3672: AMBARI-25912: use LinkedBlockingQueue in AgentsRegistrationQueue for performance improvement
Posted by "virajjasani (via GitHub)" <gi...@apache.org>.
virajjasani commented on PR #3672:
URL: https://github.com/apache/ambari/pull/3672#issuecomment-1685561142
thanks for the review @bhavikpatel9977
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@ambari.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@ambari.apache.org
For additional commands, e-mail: dev-help@ambari.apache.org
[GitHub] [ambari] JiaLiangC commented on pull request #3672: AMBARI-25912: use LinkedBlockingQueue in AgentsRegistrationQueue for performance improvement
Posted by "JiaLiangC (via GitHub)" <gi...@apache.org>.
JiaLiangC commented on PR #3672:
URL: https://github.com/apache/ambari/pull/3672#issuecomment-1683404166
@virajjasani Could you please review this PR?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@ambari.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@ambari.apache.org
For additional commands, e-mail: dev-help@ambari.apache.org
[GitHub] [ambari] bhavikpatel9977 commented on pull request #3672: AMBARI-25912: use LinkedBlockingQueue in AgentsRegistrationQueue for performance improvement
Posted by "bhavikpatel9977 (via GitHub)" <gi...@apache.org>.
bhavikpatel9977 commented on PR #3672:
URL: https://github.com/apache/ambari/pull/3672#issuecomment-1683382788
@JiaLiangC Thanks for the detailed explanation.
+lgtm
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@ambari.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@ambari.apache.org
For additional commands, e-mail: dev-help@ambari.apache.org
[GitHub] [ambari] bhavikpatel9977 commented on pull request #3672: AMBARI-25912: use LinkedBlockingQueue in AgentsRegistrationQueue for performance improvement
Posted by "bhavikpatel9977 (via GitHub)" <gi...@apache.org>.
bhavikpatel9977 commented on PR #3672:
URL: https://github.com/apache/ambari/pull/3672#issuecomment-1682120761
Can you also share any benchmark report?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@ambari.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@ambari.apache.org
For additional commands, e-mail: dev-help@ambari.apache.org
[GitHub] [ambari] JiaLiangC commented on pull request #3672: AMBARI-25912: use LinkedBlockingQueue in AgentsRegistrationQueue for performance improvement
Posted by "JiaLiangC (via GitHub)" <gi...@apache.org>.
JiaLiangC commented on PR #3672:
URL: https://github.com/apache/ambari/pull/3672#issuecomment-1683192552
@bhavikpatel9977 ok.
The lock contention of LinkedBlockingQueue is much lower than ArrayBlockingQueue in high-concurrency situations, resulting in a significant performance improvement. LinkedBlockingQueue is the most commonly used queue in concurrent scenarios. There have been numerous tests conducted on this. You can find more information in this link: https://stackoverflow.com/questions/5102570/implementation-of-blockingqueue-what-are-the-differences-between-synchronousque
```
public class Test {
static ExecutorService e = Executors.newFixedThreadPool(2);
static int N = 1000000;
public static void main(String[] args) throws Exception {
for (int i = 0; i < 10; i++) {
int length = (i == 0) ? 1 : i * 5;
System.out.print(length + "\t");
System.out.print(doTest(new LinkedBlockingQueue<Integer>(length), N) + "\t");
System.out.print(doTest(new ArrayBlockingQueue<Integer>(length), N) + "\t");
System.out.print(doTest(new SynchronousQueue<Integer>(), N));
System.out.println();
}
e.shutdown();
}
private static long doTest(final BlockingQueue<Integer> q, final int n) throws Exception {
long t = System.nanoTime();
e.submit(new Runnable() {
public void run() {
for (int i = 0; i < n; i++)
try { q.put(i); } catch (InterruptedException ex) {}
}
});
Long r = e.submit(new Callable<Long>() {
public Long call() {
long sum = 0;
for (int i = 0; i < n; i++)
try { sum += q.take(); } catch (InterruptedException ex) {}
return sum;
}
}).get();
t = System.nanoTime() - t;
return (long)(1000000000.0 * N / t); // Throughput, items/sec
}
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@ambari.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@ambari.apache.org
For additional commands, e-mail: dev-help@ambari.apache.org