You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@helix.apache.org by GitBox <gi...@apache.org> on 2020/04/15 04:18:07 UTC
[GitHub] [helix] pkuwm opened a new pull request #955: Fix MSD routing data
update methods
pkuwm opened a new pull request #955: Fix MSD routing data update methods
URL: https://github.com/apache/helix/pull/955
### Issues
- [ ] My PR addresses the following Helix issues and references them in the PR description:
Fixes part of #939
### Description
- [ ] Here are some details about my PR, including screenshots of any UI changes:
1. Routing data update methods in ZkMetadataStoreDirectory writes routing data to ZK and refresh routing cache. However, it should not refresh routing data if writing to ZK fails. Refreshing routing should only be proceeded if writing to ZK is successful.
2. The test is flaky because of removing namespace when refreshing routing data. It is caused by race condition between the read request after updating and data change callback to refresh routing (which is second refresh). Namespace should only be removed when exception is thrown.
### Tests
- [ ] The following tests are written for this issue:
Fix TestZkMetadataStoreDirectory
- [ ] The following is the result of the "mvn test" command on the appropriate module:
Before: TestZkMetadataStoreDirectory failed within 5 attempts.
After: 50 attempts of running TestZkMetadataStoreDirectory passed
In helix-rest:
```
[ERROR] Failures:
[ERROR] TestClusterAccessor.testGetControllerLeadershipHistory:364 expected:<true> but was:<false>
[ERROR] TestJobAccessor.testDeleteJob:206 ยป Helix Fail to stop the workflow/queue Test...
[INFO]
[ERROR] Tests run: 155, Failures: 2, Errors: 0, Skipped: 15
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 45.246 s
[INFO] Finished at: 2020-04-14T21:12:45-07:00
[INFO] ------------------------------------------------------------------------
[INFO] Tests run: 28, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 24.402 s - in org.apache.helix.rest.server.TestClusterAccessor
[INFO]
[INFO] Results:
[INFO]
[INFO] Tests run: 28, Failures: 0, Errors: 0, Skipped: 0
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 30.602 s
[INFO] Finished at: 2020-04-14T21:14:10-07:00
[INFO] ------------------------------------------------------------------------
[INFO] Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 20.28 s - in org.apache.helix.rest.server.TestJobAccessor
[INFO]
[INFO] Results:
[INFO]
[INFO] Tests run: 8, Failures: 0, Errors: 0, Skipped: 0
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 26.061 s
[INFO] Finished at: 2020-04-14T21:17:40-07:00
[INFO] ------------------------------------------------------------------------
```
### Commits
- [ ] My commits all reference appropriate Apache Helix GitHub issues in their subject lines. In addition, my commits follow the guidelines from "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation (Optional)
- [ ] In case of new functionality, my PR adds documentation in the following wiki page:
(Link the GitHub wiki you added)
### Code Quality
- [ ] My diff has been formatted using helix-style.xml
(helix-style-intellij.xml if IntelliJ IDE is used)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
[GitHub] [helix] NealSun96 commented on a change in pull request #955: Fix
MSD routing data refreshing for update methods
Posted by GitBox <gi...@apache.org>.
NealSun96 commented on a change in pull request #955: Fix MSD routing data refreshing for update methods
URL: https://github.com/apache/helix/pull/955#discussion_r408970017
##########
File path: helix-rest/src/main/java/org/apache/helix/rest/metadatastore/ZkMetadataStoreDirectory.java
##########
@@ -304,42 +314,35 @@ public boolean deleteShardingKey(String namespace, String realm, String sharding
*/
@Override
public void refreshRoutingData(String namespace) {
- // Safe to ignore the callback if any of the maps are null.
- // If routingDataMap is null, then it will be populated by the constructor anyway
- // If routingDataMap is not null, then it's safe for the callback function to update it
- if (_routingZkAddressMap == null || _realmToShardingKeysMap == null
- || _routingDataReaderMap == null || _routingDataWriterMap == null) {
- LOG.warn(
- "refreshRoutingData callback called before ZKMetadataStoreDirectory was fully initialized. Skipping refresh!");
- return;
- }
-
// Check if namespace exists; otherwise, return as a NOP and log it
if (!_routingZkAddressMap.containsKey(namespace)) {
LOG.error(
"Failed to refresh internally-cached routing data! Namespace not found: " + namespace);
return;
}
- // Remove the raw data first in case of failure on creation
- _realmToShardingKeysMap.remove(namespace);
- // Remove routing data first in case of failure on creation
- _routingDataMap.remove(namespace);
-
Map<String, List<String>> rawRoutingData;
try {
rawRoutingData = _routingDataReaderMap.get(namespace).getRoutingData();
- _realmToShardingKeysMap.put(namespace, rawRoutingData);
} catch (InvalidRoutingDataException e) {
LOG.error("Failed to refresh cached routing data for namespace {}", namespace, e);
+ _realmToShardingKeysMap.remove(namespace);
+ _routingDataMap.remove(namespace);
return;
}
+ TrieRoutingData trieRoutingData;
try {
- _routingDataMap.put(namespace, new TrieRoutingData(rawRoutingData));
+ trieRoutingData = new TrieRoutingData(rawRoutingData);
} catch (InvalidRoutingDataException e) {
LOG.warn("TrieRoutingData is not created for namespace {}", namespace, e);
+ _realmToShardingKeysMap.remove(namespace);
Review comment:
The raw data shouldn't be removed here. The raw data is allowed to contain invalid routing data.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
[GitHub] [helix] pkuwm commented on a change in pull request #955: Fix MSD
routing data refreshing for update methods
Posted by GitBox <gi...@apache.org>.
pkuwm commented on a change in pull request #955: Fix MSD routing data refreshing for update methods
URL: https://github.com/apache/helix/pull/955#discussion_r409025076
##########
File path: helix-rest/src/main/java/org/apache/helix/rest/metadatastore/ZkMetadataStoreDirectory.java
##########
@@ -304,42 +314,35 @@ public boolean deleteShardingKey(String namespace, String realm, String sharding
*/
@Override
public void refreshRoutingData(String namespace) {
- // Safe to ignore the callback if any of the maps are null.
- // If routingDataMap is null, then it will be populated by the constructor anyway
- // If routingDataMap is not null, then it's safe for the callback function to update it
- if (_routingZkAddressMap == null || _realmToShardingKeysMap == null
- || _routingDataReaderMap == null || _routingDataWriterMap == null) {
- LOG.warn(
- "refreshRoutingData callback called before ZKMetadataStoreDirectory was fully initialized. Skipping refresh!");
- return;
- }
-
// Check if namespace exists; otherwise, return as a NOP and log it
if (!_routingZkAddressMap.containsKey(namespace)) {
LOG.error(
"Failed to refresh internally-cached routing data! Namespace not found: " + namespace);
return;
}
- // Remove the raw data first in case of failure on creation
- _realmToShardingKeysMap.remove(namespace);
- // Remove routing data first in case of failure on creation
- _routingDataMap.remove(namespace);
-
Map<String, List<String>> rawRoutingData;
try {
rawRoutingData = _routingDataReaderMap.get(namespace).getRoutingData();
- _realmToShardingKeysMap.put(namespace, rawRoutingData);
} catch (InvalidRoutingDataException e) {
LOG.error("Failed to refresh cached routing data for namespace {}", namespace, e);
+ _realmToShardingKeysMap.remove(namespace);
+ _routingDataMap.remove(namespace);
return;
}
+ TrieRoutingData trieRoutingData;
try {
- _routingDataMap.put(namespace, new TrieRoutingData(rawRoutingData));
+ trieRoutingData = new TrieRoutingData(rawRoutingData);
} catch (InvalidRoutingDataException e) {
LOG.warn("TrieRoutingData is not created for namespace {}", namespace, e);
+ _realmToShardingKeysMap.remove(namespace);
+ _routingDataMap.remove(namespace);
+ return;
}
+
+ _realmToShardingKeysMap.put(namespace, rawRoutingData);
+ _routingDataMap.put(namespace, trieRoutingData);
Review comment:
My logic here is like a validation first and update. We will discuss the behavior first and make the change.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
[GitHub] [helix] NealSun96 commented on a change in pull request #955: Fix
MSD routing data refreshing for update methods
Posted by GitBox <gi...@apache.org>.
NealSun96 commented on a change in pull request #955: Fix MSD routing data refreshing for update methods
URL: https://github.com/apache/helix/pull/955#discussion_r408970600
##########
File path: helix-rest/src/main/java/org/apache/helix/rest/metadatastore/ZkMetadataStoreDirectory.java
##########
@@ -304,42 +314,35 @@ public boolean deleteShardingKey(String namespace, String realm, String sharding
*/
@Override
public void refreshRoutingData(String namespace) {
- // Safe to ignore the callback if any of the maps are null.
- // If routingDataMap is null, then it will be populated by the constructor anyway
- // If routingDataMap is not null, then it's safe for the callback function to update it
- if (_routingZkAddressMap == null || _realmToShardingKeysMap == null
- || _routingDataReaderMap == null || _routingDataWriterMap == null) {
- LOG.warn(
- "refreshRoutingData callback called before ZKMetadataStoreDirectory was fully initialized. Skipping refresh!");
- return;
- }
-
// Check if namespace exists; otherwise, return as a NOP and log it
if (!_routingZkAddressMap.containsKey(namespace)) {
LOG.error(
"Failed to refresh internally-cached routing data! Namespace not found: " + namespace);
return;
}
- // Remove the raw data first in case of failure on creation
- _realmToShardingKeysMap.remove(namespace);
- // Remove routing data first in case of failure on creation
- _routingDataMap.remove(namespace);
-
Map<String, List<String>> rawRoutingData;
try {
rawRoutingData = _routingDataReaderMap.get(namespace).getRoutingData();
- _realmToShardingKeysMap.put(namespace, rawRoutingData);
} catch (InvalidRoutingDataException e) {
LOG.error("Failed to refresh cached routing data for namespace {}", namespace, e);
+ _realmToShardingKeysMap.remove(namespace);
Review comment:
As you pointed out yesterday, it's better to assign empty data here.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
[GitHub] [helix] pkuwm commented on a change in pull request #955: Fix MSD
routing data refreshing for update methods
Posted by GitBox <gi...@apache.org>.
pkuwm commented on a change in pull request #955: Fix MSD routing data refreshing for update methods
URL: https://github.com/apache/helix/pull/955#discussion_r410081907
##########
File path: helix-rest/src/main/java/org/apache/helix/rest/metadatastore/ZkMetadataStoreDirectory.java
##########
@@ -304,42 +314,35 @@ public boolean deleteShardingKey(String namespace, String realm, String sharding
*/
@Override
public void refreshRoutingData(String namespace) {
- // Safe to ignore the callback if any of the maps are null.
- // If routingDataMap is null, then it will be populated by the constructor anyway
- // If routingDataMap is not null, then it's safe for the callback function to update it
- if (_routingZkAddressMap == null || _realmToShardingKeysMap == null
- || _routingDataReaderMap == null || _routingDataWriterMap == null) {
- LOG.warn(
- "refreshRoutingData callback called before ZKMetadataStoreDirectory was fully initialized. Skipping refresh!");
- return;
- }
-
// Check if namespace exists; otherwise, return as a NOP and log it
if (!_routingZkAddressMap.containsKey(namespace)) {
LOG.error(
"Failed to refresh internally-cached routing data! Namespace not found: " + namespace);
return;
}
- // Remove the raw data first in case of failure on creation
- _realmToShardingKeysMap.remove(namespace);
- // Remove routing data first in case of failure on creation
- _routingDataMap.remove(namespace);
-
Map<String, List<String>> rawRoutingData;
try {
rawRoutingData = _routingDataReaderMap.get(namespace).getRoutingData();
- _realmToShardingKeysMap.put(namespace, rawRoutingData);
} catch (InvalidRoutingDataException e) {
LOG.error("Failed to refresh cached routing data for namespace {}", namespace, e);
+ _realmToShardingKeysMap.remove(namespace);
+ _routingDataMap.remove(namespace);
return;
}
+ TrieRoutingData trieRoutingData;
try {
- _routingDataMap.put(namespace, new TrieRoutingData(rawRoutingData));
+ trieRoutingData = new TrieRoutingData(rawRoutingData);
} catch (InvalidRoutingDataException e) {
LOG.warn("TrieRoutingData is not created for namespace {}", namespace, e);
+ _realmToShardingKeysMap.remove(namespace);
+ _routingDataMap.remove(namespace);
+ return;
}
+
+ _realmToShardingKeysMap.put(namespace, rawRoutingData);
+ _routingDataMap.put(namespace, trieRoutingData);
Review comment:
I prefer to catch the exception specifically for this operation `trieRoutingData = new TrieRoutingData(rawRoutingData);` to make it clean.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
[GitHub] [helix] NealSun96 commented on a change in pull request #955: Fix
MSD routing data refreshing for update methods
Posted by GitBox <gi...@apache.org>.
NealSun96 commented on a change in pull request #955: Fix MSD routing data refreshing for update methods
URL: https://github.com/apache/helix/pull/955#discussion_r409085997
##########
File path: helix-rest/src/main/java/org/apache/helix/rest/metadatastore/ZkMetadataStoreDirectory.java
##########
@@ -304,42 +314,35 @@ public boolean deleteShardingKey(String namespace, String realm, String sharding
*/
@Override
public void refreshRoutingData(String namespace) {
- // Safe to ignore the callback if any of the maps are null.
- // If routingDataMap is null, then it will be populated by the constructor anyway
- // If routingDataMap is not null, then it's safe for the callback function to update it
- if (_routingZkAddressMap == null || _realmToShardingKeysMap == null
- || _routingDataReaderMap == null || _routingDataWriterMap == null) {
- LOG.warn(
- "refreshRoutingData callback called before ZKMetadataStoreDirectory was fully initialized. Skipping refresh!");
- return;
- }
-
// Check if namespace exists; otherwise, return as a NOP and log it
if (!_routingZkAddressMap.containsKey(namespace)) {
LOG.error(
"Failed to refresh internally-cached routing data! Namespace not found: " + namespace);
return;
}
- // Remove the raw data first in case of failure on creation
- _realmToShardingKeysMap.remove(namespace);
- // Remove routing data first in case of failure on creation
- _routingDataMap.remove(namespace);
-
Map<String, List<String>> rawRoutingData;
try {
rawRoutingData = _routingDataReaderMap.get(namespace).getRoutingData();
- _realmToShardingKeysMap.put(namespace, rawRoutingData);
} catch (InvalidRoutingDataException e) {
LOG.error("Failed to refresh cached routing data for namespace {}", namespace, e);
+ _realmToShardingKeysMap.remove(namespace);
Review comment:
My understanding is that you prefer not deleting namespaces, and I agree with your point, so it's safe to change that behavior. We can modify the test cases to reflect that.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
[GitHub] [helix] pkuwm commented on a change in pull request #955: Fix MSD
routing data refreshing for update methods
Posted by GitBox <gi...@apache.org>.
pkuwm commented on a change in pull request #955: Fix MSD routing data refreshing for update methods
URL: https://github.com/apache/helix/pull/955#discussion_r409022985
##########
File path: helix-rest/src/main/java/org/apache/helix/rest/metadatastore/ZkMetadataStoreDirectory.java
##########
@@ -304,42 +314,35 @@ public boolean deleteShardingKey(String namespace, String realm, String sharding
*/
@Override
public void refreshRoutingData(String namespace) {
- // Safe to ignore the callback if any of the maps are null.
- // If routingDataMap is null, then it will be populated by the constructor anyway
- // If routingDataMap is not null, then it's safe for the callback function to update it
- if (_routingZkAddressMap == null || _realmToShardingKeysMap == null
- || _routingDataReaderMap == null || _routingDataWriterMap == null) {
- LOG.warn(
- "refreshRoutingData callback called before ZKMetadataStoreDirectory was fully initialized. Skipping refresh!");
Review comment:
Can you help me understand in what case `the callback could be triggered before ZKMetadataStoreDirectory is fully initialized`?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
[GitHub] [helix] pkuwm commented on a change in pull request #955: Fix MSD
routing data refreshing for update methods
Posted by GitBox <gi...@apache.org>.
pkuwm commented on a change in pull request #955: Fix MSD routing data refreshing for update methods
URL: https://github.com/apache/helix/pull/955#discussion_r410080791
##########
File path: helix-rest/src/main/java/org/apache/helix/rest/metadatastore/ZkMetadataStoreDirectory.java
##########
@@ -304,42 +314,35 @@ public boolean deleteShardingKey(String namespace, String realm, String sharding
*/
@Override
public void refreshRoutingData(String namespace) {
- // Safe to ignore the callback if any of the maps are null.
- // If routingDataMap is null, then it will be populated by the constructor anyway
- // If routingDataMap is not null, then it's safe for the callback function to update it
- if (_routingZkAddressMap == null || _realmToShardingKeysMap == null
- || _routingDataReaderMap == null || _routingDataWriterMap == null) {
- LOG.warn(
- "refreshRoutingData callback called before ZKMetadataStoreDirectory was fully initialized. Skipping refresh!");
- return;
- }
-
// Check if namespace exists; otherwise, return as a NOP and log it
if (!_routingZkAddressMap.containsKey(namespace)) {
LOG.error(
"Failed to refresh internally-cached routing data! Namespace not found: " + namespace);
return;
}
- // Remove the raw data first in case of failure on creation
- _realmToShardingKeysMap.remove(namespace);
- // Remove routing data first in case of failure on creation
- _routingDataMap.remove(namespace);
-
Map<String, List<String>> rawRoutingData;
try {
rawRoutingData = _routingDataReaderMap.get(namespace).getRoutingData();
- _realmToShardingKeysMap.put(namespace, rawRoutingData);
} catch (InvalidRoutingDataException e) {
LOG.error("Failed to refresh cached routing data for namespace {}", namespace, e);
+ _realmToShardingKeysMap.remove(namespace);
Review comment:
Discussed offline. I've changed to assign an empty map to `_realmToShardingKeysMap` namespace. And the tests pass.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
[GitHub] [helix] pkuwm commented on a change in pull request #955: Fix MSD
routing data refreshing for update methods
Posted by GitBox <gi...@apache.org>.
pkuwm commented on a change in pull request #955: Fix MSD routing data refreshing for update methods
URL: https://github.com/apache/helix/pull/955#discussion_r410078265
##########
File path: helix-rest/src/main/java/org/apache/helix/rest/metadatastore/ZkMetadataStoreDirectory.java
##########
@@ -304,42 +314,35 @@ public boolean deleteShardingKey(String namespace, String realm, String sharding
*/
@Override
public void refreshRoutingData(String namespace) {
- // Safe to ignore the callback if any of the maps are null.
- // If routingDataMap is null, then it will be populated by the constructor anyway
- // If routingDataMap is not null, then it's safe for the callback function to update it
- if (_routingZkAddressMap == null || _realmToShardingKeysMap == null
- || _routingDataReaderMap == null || _routingDataWriterMap == null) {
- LOG.warn(
- "refreshRoutingData callback called before ZKMetadataStoreDirectory was fully initialized. Skipping refresh!");
Review comment:
@NealSun96 Thanks for the explanation.
After I check the code carefully, I don't see what you said is the case. `ZkRoutingDataReader` is initialized after the maps are created. So I think this check is safe to remove.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
[GitHub] [helix] NealSun96 commented on a change in pull request #955: Fix
MSD routing data refreshing for update methods
Posted by GitBox <gi...@apache.org>.
NealSun96 commented on a change in pull request #955: Fix MSD routing data refreshing for update methods
URL: https://github.com/apache/helix/pull/955#discussion_r409090040
##########
File path: helix-rest/src/test/java/org/apache/helix/rest/metadatastore/TestZkMetadataStoreDirectory.java
##########
@@ -344,6 +345,10 @@ public void testDataDeletionCallback() throws Exception {
+ " contains either empty or invalid routing data!")) {
return false;
}
+ } catch (IllegalArgumentException iae) {
+ if (!iae.getMessage().equals("Provided path is not a valid Zookeeper path: anyKey")) {
+ return false;
Review comment:
I see your point now, thanks for the explanation. Was this test case flaky because of that? This is a very good catch! ๐
I think it makes more sense to replace "anyKey" with an actual existing key in the routing data; the handling of `IllegalArgumentException` is only caused by "anyKey", and is not what this test case should focus on. My point is that using "anyKey" is a mistake. We shouldn't change the test case to accommodate that mistake; we can just correct it by using something like `TEST_SHARDING_KEYS`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
[GitHub] [helix] pkuwm commented on a change in pull request #955: Fix MSD
routing data refreshing for update methods
Posted by GitBox <gi...@apache.org>.
pkuwm commented on a change in pull request #955: Fix MSD routing data refreshing for update methods
URL: https://github.com/apache/helix/pull/955#discussion_r409031819
##########
File path: helix-rest/src/test/java/org/apache/helix/rest/metadatastore/TestZkMetadataStoreDirectory.java
##########
@@ -344,6 +345,10 @@ public void testDataDeletionCallback() throws Exception {
+ " contains either empty or invalid routing data!")) {
return false;
}
+ } catch (IllegalArgumentException iae) {
+ if (!iae.getMessage().equals("Provided path is not a valid Zookeeper path: anyKey")) {
+ return false;
Review comment:
Refreshing routing has a delay. And this is why `TestHelper.verify()` is used. Before routing is deleted in cached, namespace is still in cache, right? So the request still goes to TrieRouting, and IllegalArgumentException would be thrown because `anyKey` is invalid. In this case, we should catch the exception and retry, right?
The better I think is this. I will update PR.
```
Assert.assertTrue(TestHelper.verify(() -> {
for (String namespace : _routingZkAddrMap.keySet()) {
try {
_metadataStoreDirectory.getMetadataStoreRealm(namespace, "anyKey");
Assert.fail("Should not successfully get routing data");
} catch (IllegalStateException e) {
// If other IllegalStateException, it is unexpected and this test should fail.
if (!e.getMessage().equals("Failed to get metadata store realm: Namespace " + namespace
+ " contains either empty or invalid routing data!")) {
throw e;
}
} catch (IllegalArgumentException iae) {
// If routing data is not yet refreshed, return false and retry.
if (iae.getMessage().equals("Provided path is not a valid Zookeeper path: anyKey")) {
return false;
}
// If other IllegalArgumentException, it is not expected and this test should fail.
throw iae;
}
}
return true;
}, TestHelper.WAIT_DURATION));
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
[GitHub] [helix] NealSun96 commented on a change in pull request #955: Fix
MSD routing data refreshing for update methods
Posted by GitBox <gi...@apache.org>.
NealSun96 commented on a change in pull request #955: Fix MSD routing data refreshing for update methods
URL: https://github.com/apache/helix/pull/955#discussion_r408967822
##########
File path: helix-rest/src/main/java/org/apache/helix/rest/metadatastore/ZkMetadataStoreDirectory.java
##########
@@ -304,42 +314,35 @@ public boolean deleteShardingKey(String namespace, String realm, String sharding
*/
@Override
public void refreshRoutingData(String namespace) {
- // Safe to ignore the callback if any of the maps are null.
- // If routingDataMap is null, then it will be populated by the constructor anyway
- // If routingDataMap is not null, then it's safe for the callback function to update it
- if (_routingZkAddressMap == null || _realmToShardingKeysMap == null
- || _routingDataReaderMap == null || _routingDataWriterMap == null) {
- LOG.warn(
- "refreshRoutingData callback called before ZKMetadataStoreDirectory was fully initialized. Skipping refresh!");
Review comment:
This if statement shouldn't be deleted. The reasoning is clearly shown in the logging: the callback could be triggered before `ZKMetadataStoreDirectory` is fully initialized. Removing this if statement leaves the rest of the logic unguarded.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
[GitHub] [helix] pkuwm commented on a change in pull request #955: Fix MSD
routing data refreshing for update methods
Posted by GitBox <gi...@apache.org>.
pkuwm commented on a change in pull request #955: Fix MSD routing data refreshing for update methods
URL: https://github.com/apache/helix/pull/955#discussion_r409024277
##########
File path: helix-rest/src/main/java/org/apache/helix/rest/metadatastore/ZkMetadataStoreDirectory.java
##########
@@ -304,42 +314,35 @@ public boolean deleteShardingKey(String namespace, String realm, String sharding
*/
@Override
public void refreshRoutingData(String namespace) {
- // Safe to ignore the callback if any of the maps are null.
- // If routingDataMap is null, then it will be populated by the constructor anyway
- // If routingDataMap is not null, then it's safe for the callback function to update it
- if (_routingZkAddressMap == null || _realmToShardingKeysMap == null
- || _routingDataReaderMap == null || _routingDataWriterMap == null) {
- LOG.warn(
- "refreshRoutingData callback called before ZKMetadataStoreDirectory was fully initialized. Skipping refresh!");
- return;
- }
-
// Check if namespace exists; otherwise, return as a NOP and log it
if (!_routingZkAddressMap.containsKey(namespace)) {
LOG.error(
"Failed to refresh internally-cached routing data! Namespace not found: " + namespace);
return;
}
- // Remove the raw data first in case of failure on creation
- _realmToShardingKeysMap.remove(namespace);
- // Remove routing data first in case of failure on creation
- _routingDataMap.remove(namespace);
-
Map<String, List<String>> rawRoutingData;
try {
rawRoutingData = _routingDataReaderMap.get(namespace).getRoutingData();
- _realmToShardingKeysMap.put(namespace, rawRoutingData);
} catch (InvalidRoutingDataException e) {
LOG.error("Failed to refresh cached routing data for namespace {}", namespace, e);
+ _realmToShardingKeysMap.remove(namespace);
Review comment:
I keep it here because if it is assigned empty data, one other test `TestRoutingDataUpdate` would fail because it assumes routing namespace is deleted. Let's have more careful discussion about the behavior before we change it here.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
[GitHub] [helix] NealSun96 commented on a change in pull request #955: Fix
MSD routing data refreshing for update methods
Posted by GitBox <gi...@apache.org>.
NealSun96 commented on a change in pull request #955: Fix MSD routing data refreshing for update methods
URL: https://github.com/apache/helix/pull/955#discussion_r409085250
##########
File path: helix-rest/src/main/java/org/apache/helix/rest/metadatastore/ZkMetadataStoreDirectory.java
##########
@@ -304,42 +314,35 @@ public boolean deleteShardingKey(String namespace, String realm, String sharding
*/
@Override
public void refreshRoutingData(String namespace) {
- // Safe to ignore the callback if any of the maps are null.
- // If routingDataMap is null, then it will be populated by the constructor anyway
- // If routingDataMap is not null, then it's safe for the callback function to update it
- if (_routingZkAddressMap == null || _realmToShardingKeysMap == null
- || _routingDataReaderMap == null || _routingDataWriterMap == null) {
- LOG.warn(
- "refreshRoutingData callback called before ZKMetadataStoreDirectory was fully initialized. Skipping refresh!");
Review comment:
When the `ZkMetadataStoreDirectory` is initialized, it initializes `ZkRoutingDataReader` which subscribe to ZNode changes and calls `refreshRoutingData()` when an event happens. It's possible that an event comes in before `ZkMetadataStoreDirectory` finishes initialization, which we don't want to handle.
This is my understanding to it. This was introduced to me first by @narendly , and he was also the first to write this statement so he could provide further comments if necessary.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
[GitHub] [helix] NealSun96 commented on a change in pull request #955: Fix
MSD routing data refreshing for update methods
Posted by GitBox <gi...@apache.org>.
NealSun96 commented on a change in pull request #955: Fix MSD routing data refreshing for update methods
URL: https://github.com/apache/helix/pull/955#discussion_r408971036
##########
File path: helix-rest/src/main/java/org/apache/helix/rest/metadatastore/ZkMetadataStoreDirectory.java
##########
@@ -304,42 +314,35 @@ public boolean deleteShardingKey(String namespace, String realm, String sharding
*/
@Override
public void refreshRoutingData(String namespace) {
- // Safe to ignore the callback if any of the maps are null.
- // If routingDataMap is null, then it will be populated by the constructor anyway
- // If routingDataMap is not null, then it's safe for the callback function to update it
- if (_routingZkAddressMap == null || _realmToShardingKeysMap == null
- || _routingDataReaderMap == null || _routingDataWriterMap == null) {
- LOG.warn(
- "refreshRoutingData callback called before ZKMetadataStoreDirectory was fully initialized. Skipping refresh!");
- return;
- }
-
// Check if namespace exists; otherwise, return as a NOP and log it
if (!_routingZkAddressMap.containsKey(namespace)) {
LOG.error(
"Failed to refresh internally-cached routing data! Namespace not found: " + namespace);
return;
}
- // Remove the raw data first in case of failure on creation
- _realmToShardingKeysMap.remove(namespace);
- // Remove routing data first in case of failure on creation
- _routingDataMap.remove(namespace);
-
Map<String, List<String>> rawRoutingData;
try {
rawRoutingData = _routingDataReaderMap.get(namespace).getRoutingData();
- _realmToShardingKeysMap.put(namespace, rawRoutingData);
} catch (InvalidRoutingDataException e) {
LOG.error("Failed to refresh cached routing data for namespace {}", namespace, e);
+ _realmToShardingKeysMap.remove(namespace);
+ _routingDataMap.remove(namespace);
return;
}
+ TrieRoutingData trieRoutingData;
try {
- _routingDataMap.put(namespace, new TrieRoutingData(rawRoutingData));
+ trieRoutingData = new TrieRoutingData(rawRoutingData);
} catch (InvalidRoutingDataException e) {
LOG.warn("TrieRoutingData is not created for namespace {}", namespace, e);
+ _realmToShardingKeysMap.remove(namespace);
+ _routingDataMap.remove(namespace);
+ return;
}
+
+ _realmToShardingKeysMap.put(namespace, rawRoutingData);
+ _routingDataMap.put(namespace, trieRoutingData);
Review comment:
Overall, I think it's more clean to do `try {overwriteOldData} catch {eraseOldData}` for both blocks of the logic.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
[GitHub] [helix] narendly commented on issue #955: Fix MSD routing data
refreshing for update methods
Posted by GitBox <gi...@apache.org>.
narendly commented on issue #955: Fix MSD routing data refreshing for update methods
URL: https://github.com/apache/helix/pull/955#issuecomment-614129166
@NealSun96 Could you please review this change?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
[GitHub] [helix] NealSun96 commented on a change in pull request #955: Fix
MSD routing data refreshing for update methods
Posted by GitBox <gi...@apache.org>.
NealSun96 commented on a change in pull request #955: Fix MSD routing data refreshing for update methods
URL: https://github.com/apache/helix/pull/955#discussion_r408975123
##########
File path: helix-rest/src/test/java/org/apache/helix/rest/metadatastore/TestZkMetadataStoreDirectory.java
##########
@@ -344,6 +345,10 @@ public void testDataDeletionCallback() throws Exception {
+ " contains either empty or invalid routing data!")) {
return false;
}
+ } catch (IllegalArgumentException iae) {
+ if (!iae.getMessage().equals("Provided path is not a valid Zookeeper path: anyKey")) {
+ return false;
Review comment:
The meaning of the block on line 343 is "ensure that an `IllegalStateException` is raised and the message is 'Failed to get...'". Is this block ensuring a `IllegalArgumentException` is raised? That's not what this test case is testing.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
[GitHub] [helix] NealSun96 commented on a change in pull request #955: Fix
MSD routing data refreshing for update methods
Posted by GitBox <gi...@apache.org>.
NealSun96 commented on a change in pull request #955: Fix MSD routing data refreshing for update methods
URL: https://github.com/apache/helix/pull/955#discussion_r409086434
##########
File path: helix-rest/src/main/java/org/apache/helix/rest/metadatastore/ZkMetadataStoreDirectory.java
##########
@@ -304,42 +314,35 @@ public boolean deleteShardingKey(String namespace, String realm, String sharding
*/
@Override
public void refreshRoutingData(String namespace) {
- // Safe to ignore the callback if any of the maps are null.
- // If routingDataMap is null, then it will be populated by the constructor anyway
- // If routingDataMap is not null, then it's safe for the callback function to update it
- if (_routingZkAddressMap == null || _realmToShardingKeysMap == null
- || _routingDataReaderMap == null || _routingDataWriterMap == null) {
- LOG.warn(
- "refreshRoutingData callback called before ZKMetadataStoreDirectory was fully initialized. Skipping refresh!");
- return;
- }
-
// Check if namespace exists; otherwise, return as a NOP and log it
if (!_routingZkAddressMap.containsKey(namespace)) {
LOG.error(
"Failed to refresh internally-cached routing data! Namespace not found: " + namespace);
return;
}
- // Remove the raw data first in case of failure on creation
- _realmToShardingKeysMap.remove(namespace);
- // Remove routing data first in case of failure on creation
- _routingDataMap.remove(namespace);
-
Map<String, List<String>> rawRoutingData;
try {
rawRoutingData = _routingDataReaderMap.get(namespace).getRoutingData();
- _realmToShardingKeysMap.put(namespace, rawRoutingData);
} catch (InvalidRoutingDataException e) {
LOG.error("Failed to refresh cached routing data for namespace {}", namespace, e);
+ _realmToShardingKeysMap.remove(namespace);
+ _routingDataMap.remove(namespace);
return;
}
+ TrieRoutingData trieRoutingData;
try {
- _routingDataMap.put(namespace, new TrieRoutingData(rawRoutingData));
+ trieRoutingData = new TrieRoutingData(rawRoutingData);
} catch (InvalidRoutingDataException e) {
LOG.warn("TrieRoutingData is not created for namespace {}", namespace, e);
+ _realmToShardingKeysMap.remove(namespace);
+ _routingDataMap.remove(namespace);
+ return;
}
+
+ _realmToShardingKeysMap.put(namespace, rawRoutingData);
+ _routingDataMap.put(namespace, trieRoutingData);
Review comment:
The validation process = the exception. If something is caught it means the data is invalid. So validation will always go first before update.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org