You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by GitBox <gi...@apache.org> on 2020/07/22 21:39:19 UTC
[GitHub] [hadoop-ozone] avijayanhwx opened a new pull request #1243: HDDS-4008. Recon should fallback to ozone.om.service.ids when the internal service id is not defined.
avijayanhwx opened a new pull request #1243:
URL: https://github.com/apache/hadoop-ozone/pull/1243
## What changes were proposed in this pull request?
Recon connects to OM via RPC using the "ozone.om.internal.service.id" to get updates. If the above config is not defined, but the ozone.om.service.ids is defined, Recon should use the latter as a fallback. Currently, a single Recon instance supports only 1 OM HA cluster at a time. Hence, if multiple ids are defined, Recon will pick the first.
## What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-4008
## How was this patch tested?
Manually tested.
Added unit test.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] avijayanhwx commented on a change in pull request #1243: HDDS-4008. Recon should fallback to ozone.om.service.ids when the internal service id is not defined.
Posted by GitBox <gi...@apache.org>.
avijayanhwx commented on a change in pull request #1243:
URL: https://github.com/apache/hadoop-ozone/pull/1243#discussion_r459622263
##########
File path: hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/OmUtils.java
##########
@@ -527,4 +529,45 @@ public static void validateKeyName(String keyName)
OMException.ResultCodes.INVALID_KEY_NAME);
}
}
+
+ /**
+ * Return configured OzoneManager service id based on the following logic.
+ * Look at 'ozone.om.internal.service.id' first. If configured, return that.
+ * If the above is not configured, look at 'ozone.om.service.ids'.
+ * If count(ozone.om.service.ids) == 1, return that id.
+ * If count(ozone.om.service.ids) > 1 throw exception
+ * If 'ozone.om.service.ids' is not configured, return null. (Non HA)
+ * @param conf configuration
+ * @return OM service ID.
+ * @throws IOException on error.
+ */
+ public static String getOzoneManagerServiceId(OzoneConfiguration conf)
+ throws IOException {
+ Collection<String> omServiceIds;
+ String localOMServiceId = conf.get(OZONE_OM_INTERNAL_SERVICE_ID);
+ if (localOMServiceId == null) {
+ LOG.info("{} is not defined, falling back to {} to find serviceID for "
+ + "OzoneManager if it is HA enabled cluster",
+ OZONE_OM_INTERNAL_SERVICE_ID, OZONE_OM_SERVICE_IDS_KEY);
+ omServiceIds = conf.getTrimmedStringCollection(
+ OZONE_OM_SERVICE_IDS_KEY);
+ if (omServiceIds.size() > 1) {
+ throw new IOException(String.format(
+ "More than 1 OzoneManager ServiceID (ozone.om.service.ids) " +
+ "configured : %s, but ozone.om.internal.service.id is not " +
+ "configured.", omServiceIds.toString()));
+ }
+ } else {
+ omServiceIds = Collections.singletonList(localOMServiceId);
+ }
+
Review comment:
Thanks @bharatviswa504. Will fix this.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] codecov-commenter commented on pull request #1243: HDDS-4008. Recon should fallback to ozone.om.service.ids when the internal service id is not defined.
Posted by GitBox <gi...@apache.org>.
codecov-commenter commented on pull request #1243:
URL: https://github.com/apache/hadoop-ozone/pull/1243#issuecomment-662795727
# [Codecov](https://codecov.io/gh/apache/hadoop-ozone/pull/1243?src=pr&el=h1) Report
> Merging [#1243](https://codecov.io/gh/apache/hadoop-ozone/pull/1243?src=pr&el=desc) into [master](https://codecov.io/gh/apache/hadoop-ozone/commit/7dac140024214c2189b72fad0566a0252d63e93c&el=desc) will **increase** coverage by `0.07%`.
> The diff coverage is `100.00%`.
[](https://codecov.io/gh/apache/hadoop-ozone/pull/1243?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #1243 +/- ##
============================================
+ Coverage 73.66% 73.73% +0.07%
- Complexity 10087 10105 +18
============================================
Files 974 974
Lines 50043 50072 +29
Branches 4863 4872 +9
============================================
+ Hits 36865 36923 +58
+ Misses 10854 10823 -31
- Partials 2324 2326 +2
```
| [Impacted Files](https://codecov.io/gh/apache/hadoop-ozone/pull/1243?src=pr&el=tree) | Coverage Δ | Complexity Δ | |
|---|---|---|---|
| [...src/main/java/org/apache/hadoop/ozone/OmUtils.java](https://codecov.io/gh/apache/hadoop-ozone/pull/1243/diff?src=pr&el=tree#diff-aGFkb29wLW96b25lL2NvbW1vbi9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvaGFkb29wL296b25lL09tVXRpbHMuamF2YQ==) | `79.13% <100.00%> (+2.33%)` | `45.00 <4.00> (+4.00)` | |
| [...ache/hadoop/ozone/recon/ReconControllerModule.java](https://codecov.io/gh/apache/hadoop-ozone/pull/1243/diff?src=pr&el=tree#diff-aGFkb29wLW96b25lL3JlY29uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9oYWRvb3Avb3pvbmUvcmVjb24vUmVjb25Db250cm9sbGVyTW9kdWxlLmphdmE=) | `79.48% <100.00%> (ø)` | `6.00 <0.00> (ø)` | |
| [...hdds/scm/container/common/helpers/ExcludeList.java](https://codecov.io/gh/apache/hadoop-ozone/pull/1243/diff?src=pr&el=tree#diff-aGFkb29wLWhkZHMvY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9oYWRvb3AvaGRkcy9zY20vY29udGFpbmVyL2NvbW1vbi9oZWxwZXJzL0V4Y2x1ZGVMaXN0LmphdmE=) | `86.95% <0.00%> (-13.05%)` | `19.00% <0.00%> (-3.00%)` | |
| [...ent/algorithms/SCMContainerPlacementRackAware.java](https://codecov.io/gh/apache/hadoop-ozone/pull/1243/diff?src=pr&el=tree#diff-aGFkb29wLWhkZHMvc2VydmVyLXNjbS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvaGFkb29wL2hkZHMvc2NtL2NvbnRhaW5lci9wbGFjZW1lbnQvYWxnb3JpdGhtcy9TQ01Db250YWluZXJQbGFjZW1lbnRSYWNrQXdhcmUuamF2YQ==) | `76.69% <0.00%> (-3.01%)` | `31.00% <0.00%> (-2.00%)` | |
| [.../statemachine/background/BlockDeletingService.java](https://codecov.io/gh/apache/hadoop-ozone/pull/1243/diff?src=pr&el=tree#diff-aGFkb29wLWhkZHMvY29udGFpbmVyLXNlcnZpY2Uvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2hhZG9vcC9vem9uZS9jb250YWluZXIva2V5dmFsdWUvc3RhdGVtYWNoaW5lL2JhY2tncm91bmQvQmxvY2tEZWxldGluZ1NlcnZpY2UuamF2YQ==) | `75.73% <0.00%> (-1.48%)` | `12.00% <0.00%> (-1.00%)` | |
| [...doop/ozone/container/keyvalue/KeyValueHandler.java](https://codecov.io/gh/apache/hadoop-ozone/pull/1243/diff?src=pr&el=tree#diff-aGFkb29wLWhkZHMvY29udGFpbmVyLXNlcnZpY2Uvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2hhZG9vcC9vem9uZS9jb250YWluZXIva2V5dmFsdWUvS2V5VmFsdWVIYW5kbGVyLmphdmE=) | `63.55% <0.00%> (-1.12%)` | `66.00% <0.00%> (-1.00%)` | |
| [...g/apache/hadoop/ozone/om/helpers/OzoneFSUtils.java](https://codecov.io/gh/apache/hadoop-ozone/pull/1243/diff?src=pr&el=tree#diff-aGFkb29wLW96b25lL2NvbW1vbi9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvaGFkb29wL296b25lL29tL2hlbHBlcnMvT3pvbmVGU1V0aWxzLmphdmE=) | `84.84% <0.00%> (-0.87%)` | `18.00% <0.00%> (+8.00%)` | :arrow_down: |
| [...iner/common/statemachine/DatanodeStateMachine.java](https://codecov.io/gh/apache/hadoop-ozone/pull/1243/diff?src=pr&el=tree#diff-aGFkb29wLWhkZHMvY29udGFpbmVyLXNlcnZpY2Uvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2hhZG9vcC9vem9uZS9jb250YWluZXIvY29tbW9uL3N0YXRlbWFjaGluZS9EYXRhbm9kZVN0YXRlTWFjaGluZS5qYXZh) | `85.10% <0.00%> (-0.54%)` | `31.00% <0.00%> (ø%)` | |
| [.../ozone/container/common/volume/AbstractFuture.java](https://codecov.io/gh/apache/hadoop-ozone/pull/1243/diff?src=pr&el=tree#diff-aGFkb29wLWhkZHMvY29udGFpbmVyLXNlcnZpY2Uvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2hhZG9vcC9vem9uZS9jb250YWluZXIvY29tbW9uL3ZvbHVtZS9BYnN0cmFjdEZ1dHVyZS5qYXZh) | `30.12% <0.00%> (-0.52%)` | `20.00% <0.00%> (-1.00%)` | |
| [...apache/hadoop/ozone/client/io/KeyOutputStream.java](https://codecov.io/gh/apache/hadoop-ozone/pull/1243/diff?src=pr&el=tree#diff-aGFkb29wLW96b25lL2NsaWVudC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvaGFkb29wL296b25lL2NsaWVudC9pby9LZXlPdXRwdXRTdHJlYW0uamF2YQ==) | `80.41% <0.00%> (-0.42%)` | `47.00% <0.00%> (-1.00%)` | |
| ... and [15 more](https://codecov.io/gh/apache/hadoop-ozone/pull/1243/diff?src=pr&el=tree-more) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/hadoop-ozone/pull/1243?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/hadoop-ozone/pull/1243?src=pr&el=footer). Last update [7dac140...2a2b5c8](https://codecov.io/gh/apache/hadoop-ozone/pull/1243?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] bharatviswa504 commented on a change in pull request #1243: HDDS-4008. Recon should fallback to ozone.om.service.ids when the internal service id is not defined.
Posted by GitBox <gi...@apache.org>.
bharatviswa504 commented on a change in pull request #1243:
URL: https://github.com/apache/hadoop-ozone/pull/1243#discussion_r459165738
##########
File path: hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/OmUtils.java
##########
@@ -527,4 +529,45 @@ public static void validateKeyName(String keyName)
OMException.ResultCodes.INVALID_KEY_NAME);
}
}
+
+ /**
+ * Return configured OzoneManager service id based on the following logic.
+ * Look at 'ozone.om.internal.service.id' first. If configured, return that.
+ * If the above is not configured, look at 'ozone.om.service.ids'.
+ * If count(ozone.om.service.ids) == 1, return that id.
+ * If count(ozone.om.service.ids) > 1 throw exception
+ * If 'ozone.om.service.ids' is not configured, return null. (Non HA)
+ * @param conf configuration
+ * @return OM service ID.
+ * @throws IOException on error.
+ */
+ public static String getOzoneManagerServiceId(OzoneConfiguration conf)
+ throws IOException {
+ Collection<String> omServiceIds;
+ String localOMServiceId = conf.get(OZONE_OM_INTERNAL_SERVICE_ID);
+ if (localOMServiceId == null) {
+ LOG.info("{} is not defined, falling back to {} to find serviceID for "
+ + "OzoneManager if it is HA enabled cluster",
+ OZONE_OM_INTERNAL_SERVICE_ID, OZONE_OM_SERVICE_IDS_KEY);
+ omServiceIds = conf.getTrimmedStringCollection(
+ OZONE_OM_SERVICE_IDS_KEY);
+ if (omServiceIds.size() > 1) {
+ throw new IOException(String.format(
+ "More than 1 OzoneManager ServiceID (ozone.om.service.ids) " +
+ "configured : %s, but ozone.om.internal.service.id is not " +
+ "configured.", omServiceIds.toString()));
+ }
+ } else {
+ omServiceIds = Collections.singletonList(localOMServiceId);
+ }
+
Review comment:
I think we also need to check if localServiceID value is one of the ozone.om.service.ids value.
To find if there is a mistake in the config.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] avijayanhwx closed pull request #1243: HDDS-4008. Recon should fallback to ozone.om.service.ids when the internal service id is not defined.
Posted by GitBox <gi...@apache.org>.
avijayanhwx closed pull request #1243:
URL: https://github.com/apache/hadoop-ozone/pull/1243
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] bharatviswa504 commented on a change in pull request #1243: HDDS-4008. Recon should fallback to ozone.om.service.ids when the internal service id is not defined.
Posted by GitBox <gi...@apache.org>.
bharatviswa504 commented on a change in pull request #1243:
URL: https://github.com/apache/hadoop-ozone/pull/1243#discussion_r459660177
##########
File path: hadoop-ozone/recon/src/main/resources/webapps/recon/ozone-recon-web/src/views/overview/overview.tsx
##########
@@ -180,7 +180,7 @@ export class Overview extends React.Component<Record<string, object>, IOverviewS
<OverviewCard loading={loading} title='Buckets' data={buckets.toString()} icon='folder-open'/>
</Col>
<Col xs={24} sm={18} md={12} lg={12} xl={6}>
- <OverviewCard loading={loading} title='Keys' data={keys.toString()} icon='file-text'/>
+ <OverviewCard loading={loading} title='Keys (Estimated)' data={keys.toString()} icon='file-text'/>
Review comment:
That is fine. But a new Jira should be nicer here.
I am fine with it.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] avijayanhwx merged pull request #1243: HDDS-4008. Recon should fallback to ozone.om.service.ids when the internal service id is not defined.
Posted by GitBox <gi...@apache.org>.
avijayanhwx merged pull request #1243:
URL: https://github.com/apache/hadoop-ozone/pull/1243
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] avijayanhwx commented on a change in pull request #1243: HDDS-4008. Recon should fallback to ozone.om.service.ids when the internal service id is not defined.
Posted by GitBox <gi...@apache.org>.
avijayanhwx commented on a change in pull request #1243:
URL: https://github.com/apache/hadoop-ozone/pull/1243#discussion_r459591303
##########
File path: hadoop-ozone/recon/src/main/resources/webapps/recon/ozone-recon-web/src/views/overview/overview.tsx
##########
@@ -180,7 +180,7 @@ export class Overview extends React.Component<Record<string, object>, IOverviewS
<OverviewCard loading={loading} title='Buckets' data={buckets.toString()} icon='folder-open'/>
</Col>
<Col xs={24} sm={18} md={12} lg={12} xl={6}>
- <OverviewCard loading={loading} title='Keys' data={keys.toString()} icon='file-text'/>
+ <OverviewCard loading={loading} title='Keys (Estimated)' data={keys.toString()} icon='file-text'/>
Review comment:
I made this trivial change to cover for HDDS-4009 until that is fixed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] avijayanhwx commented on pull request #1243: HDDS-4008. Recon should fallback to ozone.om.service.ids when the internal service id is not defined.
Posted by GitBox <gi...@apache.org>.
avijayanhwx commented on pull request #1243:
URL: https://github.com/apache/hadoop-ozone/pull/1243#issuecomment-663225883
Thank you for the reviews @bharatviswa504 & @vivekratnavel.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] bharatviswa504 commented on a change in pull request #1243: HDDS-4008. Recon should fallback to ozone.om.service.ids when the internal service id is not defined.
Posted by GitBox <gi...@apache.org>.
bharatviswa504 commented on a change in pull request #1243:
URL: https://github.com/apache/hadoop-ozone/pull/1243#discussion_r459166889
##########
File path: hadoop-ozone/recon/src/main/resources/webapps/recon/ozone-recon-web/src/views/overview/overview.tsx
##########
@@ -180,7 +180,7 @@ export class Overview extends React.Component<Record<string, object>, IOverviewS
<OverviewCard loading={loading} title='Buckets' data={buckets.toString()} icon='folder-open'/>
</Col>
<Col xs={24} sm={18} md={12} lg={12} xl={6}>
- <OverviewCard loading={loading} title='Keys' data={keys.toString()} icon='file-text'/>
+ <OverviewCard loading={loading} title='Keys (Estimated)' data={keys.toString()} icon='file-text'/>
Review comment:
Not related to this change??
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org