You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@storm.apache.org by GitBox <gi...@apache.org> on 2020/07/21 19:28:30 UTC
[GitHub] [storm] Ethanlm opened a new pull request #3313: [STORM-3679] Fix the misuse of nodeId as hostname in LoadAwareShuffleGrouping
Ethanlm opened a new pull request #3313:
URL: https://github.com/apache/storm/pull/3313
## What is the purpose of the change
LoadAwareShuffleGrouping misuses nodeId as the hostname to cause consistent invalid DNS queries for "hostnames" like 2a1f2cf3-c701-4621-9e93-640b4e63be48-10.215.73.209.
This causes excessive unnecessary loads on nscd and DNS. Also because of this bug, every target tasks will be treated as at least RACK_LOCAL because if an ip address can't be determined, YahooDNSToSwitchMapping treats it as DEFAULT_RACK. This doesn't impact WORKER_LOCAL and HOST_LOCAL though.
## How was the change tested
Before the change,
added logs and showed that the "hostname" is actually an ID. And tcp dump
```
18:42:21.913688 IP <host1>.42462 > <dns>: 43104+ A? 2a1f2cf3-c701-4621-9e93-640b4e63be48-<ip1>. (68)
18:42:21.914181 IP <dns> > <host1>.42462: 43104 NXDomain 0/1/0 (143)
```
After the change,
determined from additional added debug logs (removed after the coding is done) that the hostnames are correct and no invalid queries shown in TCP dump.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [storm] Ethanlm merged pull request #3313: [STORM-3679] Fix the misuse of nodeId as hostname in LoadAwareShuffleGrouping
Posted by GitBox <gi...@apache.org>.
Ethanlm merged pull request #3313:
URL: https://github.com/apache/storm/pull/3313
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org