You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@fluo.apache.org by GitBox <gi...@apache.org> on 2020/07/13 21:52:03 UTC
[GitHub] [fluo-muchos] arvindshmicrosoft opened a new pull request #356: Normalize hdfs-site.xml across HA and non-HA cases
arvindshmicrosoft opened a new pull request #356:
URL: https://github.com/apache/fluo-muchos/pull/356
* Explicitly specify the data directory for the non-HA checkpoint
(secondary namenode) to use (in a non-HA config)
* Rearrange some elements in the hdfs-site.xml file in a more logical order
* Use HDFS namespace (using nameservice_id) in non-HA cases as well (HA
config already uses it). This change allows using the nameservice_id
as a stable and simple way to reference namenodes regardless of
whether HA is used / or not
* Configure the use of ConfiguredFailoverProxyProvider in non-HA cases
as well (HA config already used it), so that namespace can be resolved
to physical namenodes in all cases
[Reference from Hadoop docs](https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/Federation.html#Federation_Configuration) - though this is a doc for HDFS federation
the specific change in this PR is orthogonal and merely aims to
normalize hdfs-site.xml across both HA / standalone configurations.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [fluo-muchos] arvindshmicrosoft edited a comment on pull request #356: Normalize hdfs-site.xml across HA and non-HA cases
Posted by GitBox <gi...@apache.org>.
arvindshmicrosoft edited a comment on pull request #356:
URL: https://github.com/apache/fluo-muchos/pull/356#issuecomment-658321483
Merging this as per above discussion
- #357 is the first of the 2 follow-up PRs, and that one fixes the fluo-env.sh so that Fluo runs with a HDFS nameservice based DFS root
- #358 simplifies hdfs_root as per the above discussion. It should be merged only after #357 is merged.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [fluo-muchos] arvindshmicrosoft edited a comment on pull request #356: Normalize hdfs-site.xml across HA and non-HA cases
Posted by GitBox <gi...@apache.org>.
arvindshmicrosoft edited a comment on pull request #356:
URL: https://github.com/apache/fluo-muchos/pull/356#issuecomment-658321483
Merging this as per above discussion
- #357 is the first of the 2 follow-up PRs; it fixes the fluo-env.sh so that Fluo runs with a HDFS nameservice based DFS root
- #358 simplifies hdfs_root as per the above discussion. It should be merged only after #357 is merged.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [fluo-muchos] arvindshmicrosoft commented on pull request #356: Normalize hdfs-site.xml across HA and non-HA cases
Posted by GitBox <gi...@apache.org>.
arvindshmicrosoft commented on pull request #356:
URL: https://github.com/apache/fluo-muchos/pull/356#issuecomment-657956693
Thank you @keith-turner, appreciate your review. Re: the 2 points you mentioned:
1. Agreed, it should now be possible to just use the nameservice_id instead of the namenode[0]. I will push up an update.
2. The secondary namenode was working fine previously, just that it would place its data into a location under /tmp (as specified in [hdfs-default.xml](http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/hdfs-default.xml). Consequently, `muchos wipe` would not clean up this folder, and I had noticed some weird problems between setup / wipe / setup cycles. Now that it is placed under the worker_data_dirs, the `wipe` command does clean its contents up.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [fluo-muchos] arvindshmicrosoft commented on pull request #356: Normalize hdfs-site.xml across HA and non-HA cases
Posted by GitBox <gi...@apache.org>.
arvindshmicrosoft commented on pull request #356:
URL: https://github.com/apache/fluo-muchos/pull/356#issuecomment-657977076
> Thank you @keith-turner, appreciate your review. Re: the 2 points you mentioned:
>
> 1. Agreed, it should now be possible to just use the nameservice_id instead of the namenode[0]. I will push up an update.
> 2. The secondary namenode was working fine previously, just that it would place its data into a location under /tmp (as specified in [hdfs-default.xml](http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/hdfs-default.xml). Consequently, `muchos wipe` would not clean up this folder, and I had noticed some weird problems between setup / wipe / setup cycles. Now that it is placed under the worker_data_dirs, the `wipe` command does clean its contents up.
@keith-turner I realized an issue which will prevent changing hdfs_root to use the nameservice_id right away: Since Fluo 1.2.0 does not have the [recent fix](https://github.com/apache/fluo/commit/957eaed4fed01cf93304b645cfcd27a42c5fb9d3) to ensure hdfs-site.xml is loaded, switching hdfs_root to the nameservice will cause problems for running Fluo using Muchos. So here is what I propose:
1. We merge this PR in as it is. There will be no breaking changes as a result of just this change.
2. In a subsequent PR, I will add a task to check if Fluo 1.x is being run, and in that case, patch fluo-env.sh to the equivalent of [the other fluo-env.sh fix](https://github.com/apache/fluo/commit/957eaed4fed01cf93304b645cfcd27a42c5fb9d3)
3. In a third PR after the above one merges, I will change hdfs_root to use the nameservice_id in all cases
I believe this will be a clean and safe way to proceed. Please let me know if you foresee any issues. If you are good, I will merge this PR and create tracking issues for the 2 follow-ups as mentioned above.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [fluo-muchos] arvindshmicrosoft edited a comment on pull request #356: Normalize hdfs-site.xml across HA and non-HA cases
Posted by GitBox <gi...@apache.org>.
arvindshmicrosoft edited a comment on pull request #356:
URL: https://github.com/apache/fluo-muchos/pull/356#issuecomment-657977076
> 1. Agreed, it should now be possible to just use the nameservice_id instead of the namenode[0]. I will push up an update.
@keith-turner I realized an issue which will prevent changing hdfs_root to use the nameservice_id right away: Since Fluo 1.2.0 does not have the [recent fix](https://github.com/apache/fluo/commit/957eaed4fed01cf93304b645cfcd27a42c5fb9d3) to ensure hdfs-site.xml is loaded, switching hdfs_root to the nameservice will cause problems for running Fluo using Muchos. So here is what I propose:
1. We merge this PR in as it is. There will be no breaking changes as a result of just this change.
2. In a subsequent PR, I will add a task to check if Fluo 1.x is being run, and in that case, patch fluo-env.sh to the equivalent of [the other fluo-env.sh fix](https://github.com/apache/fluo/commit/957eaed4fed01cf93304b645cfcd27a42c5fb9d3)
3. In a third PR after the above one merges, I will change hdfs_root to use the nameservice_id in all cases
I believe this will be a clean and safe way to proceed. Please let me know if you foresee any issues. If you are good, I will merge this PR and create tracking issues for the 2 follow-ups as mentioned above.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [fluo-muchos] keith-turner commented on pull request #356: Normalize hdfs-site.xml across HA and non-HA cases
Posted by GitBox <gi...@apache.org>.
keith-turner commented on pull request #356:
URL: https://github.com/apache/fluo-muchos/pull/356#issuecomment-657925512
> Explicitly specify the data directory for the non-HA checkpoint
(secondary namenode) to use (in a non-HA config)
Was the secondary NN not working before this change?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [fluo-muchos] arvindshmicrosoft commented on pull request #356: Normalize hdfs-site.xml across HA and non-HA cases
Posted by GitBox <gi...@apache.org>.
arvindshmicrosoft commented on pull request #356:
URL: https://github.com/apache/fluo-muchos/pull/356#issuecomment-658321483
Merging this as per above discussion. #357 is the first of the 2 follow-up PRs.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [fluo-muchos] keith-turner commented on pull request #356: Normalize hdfs-site.xml across HA and non-HA cases
Posted by GitBox <gi...@apache.org>.
keith-turner commented on pull request #356:
URL: https://github.com/apache/fluo-muchos/pull/356#issuecomment-657925204
> This change allows using the nameservice_id as a stable and simple way to reference namenodes regardless of whether HA is used / or not
@arvindshmicrosoft with this change would it be ok to always use the nameservice id for `hdfs_root`? I looked around for where it was set and found the following in `./lib/muchos/config/base.py`.
```
"hdfs_root": (
"{% if hdfs_ha %}hdfs://{{ nameservice_id }}{% else %}"
"hdfs://{{ groups['namenode'][0] }}:8020{% endif %}"
),
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [fluo-muchos] keith-turner commented on pull request #356: Normalize hdfs-site.xml across HA and non-HA cases
Posted by GitBox <gi...@apache.org>.
keith-turner commented on pull request #356:
URL: https://github.com/apache/fluo-muchos/pull/356#issuecomment-658239252
> I believe this will be a clean and safe way to proceed. Please let me know if you foresee any issues. If you are good
I like that plan and I think this is ready to merged as is.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [fluo-muchos] arvindshmicrosoft merged pull request #356: Normalize hdfs-site.xml across HA and non-HA cases
Posted by GitBox <gi...@apache.org>.
arvindshmicrosoft merged pull request #356:
URL: https://github.com/apache/fluo-muchos/pull/356
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org