You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by "virajjasani (via GitHub)" <gi...@apache.org> on 2023/02/15 04:02:00 UTC
[GitHub] [hadoop] virajjasani opened a new pull request, #5396: HDFS-16918. Optionally shut down datanode if it does not stay connected to active namenode
virajjasani opened a new pull request, #5396:
URL: https://github.com/apache/hadoop/pull/5396
While deploying Hdfs on Envoy proxy setup, depending on the socket timeout configured at envoy, the network connection issues or packet loss could be observed. All of envoys basically form a transparent communication mesh in which each app can send and receive packets to and from localhost and is unaware of the network topology.
The primary purpose of Envoy is to make the network transparent to applications, in order to identify network issues reliably. However, sometimes such proxy based setup could result into socket connection issues b/ datanode and namenode.
Many deployment frameworks provide auto-start functionality when any of the hadoop daemons are stopped. If a given datanode does not stay connected to active namenode in the cluster i.e. does not receive heartbeat response in time from active namenode (even though active namenode is not terminated), it would not be much useful. We should be able to provide configurable behavior such that if a given datanode cannot receive heartbeat response from active namenode in configurable time duration, it should terminate itself to avoid impacting the availability SLA. This is specifically helpful when the underlying deployment or observability framework (e.g. K8S) can start up the datanode automatically upon it's shutdown (unless it is being restarted as part of rolling upgrade) and help the newly brought up datanode (in case of k8s, a new pod with dynamically changing nodes) establish new socket connection to active and standby namenodes. This should be an opt-in behavior and not default o
ne.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] virajjasani commented on pull request #5396: HDFS-16918. Optionally shut down datanode if it does not stay connected to active namenode
Posted by "virajjasani (via GitHub)" <gi...@apache.org>.
virajjasani commented on PR #5396:
URL: https://github.com/apache/hadoop/pull/5396#issuecomment-1430828452
> If the datanode is connected to observer namenode, it can serve requests, why we need to shutdown
The observer namenode takes a different condition. I was actually thinking about making this include observer namenode too i.e. if datanode has not received heartbeat from observer or active namenode in the last e.g. 30s or so, then it should shutdown. This is an option, no issues with it.
> Even if it is connected to standby, a failover happens and it will be in good shape, else if you restart a bunch of datanodes, the new namenode will be flooded by block reports and just increasing problems.
This problem would occur only if we select reasonably lower number. The recommendation for this config value is high enough to include extra time duration for namenode failover.
> If something gets messed up with Active namenode, you shutdown all, the BR are already heavy, you forced all other namenodes to handle them again, making failover more difficult. and if it is some faulty datanodes which lost connection, you didn't get that alarmed, and all Standby and Observers will keep on getting flooded by BRs, so in case Active NN literally dies and tries to failover to any of the Namenode which these Datanodes were connected, will be fed with unnecessary loads of BlockReports. (BR has an option of initial delay as well, it isn't like all bombard at once and you are sorted in 5-10 mins)
The moment when active namenode becomes messy, or dies, this is exactly what can impact the availability of the hdfs service. So either we have Observer namenode take care of read requests in the meantime or the failover needs to happen. If neither of that happens, it's the datanode that is not really useful by staying the in cluster for longer duration. Let's say namenode gets bad and failover does take time, the new active one is anyways going to take time processing BRs right?
> If something got messed with the datanode, that is why it isn't able to connect to Active. If something is in Memory not persisted to disk, or some JMX parameter or N/W parameters which can be used to figure out things gets lost.
Do you mean hsync vs hflush kind of thing for in prgress files? Is that not already taken care of?
> That is the reason most cluster administrator in not so cool situations, show XYZ datanode is unhealthy or not, if in some case they don't it should be handled over there.
The response would take time from the cluster admin applications. Why not get auto healed by datanode? Also it's not that this change is going to terminate the datanode, it's going to shut down properly.
> In case of shared datanodes in a federated setup, say it is connected to Active for one Namespace and has completely lost touch with another, then? Restart to get both working? Don't restart so that at least one stays working? Both are correct in there own ways and situation and the datanode shouldn't be in a state to decide its fate for such reasons.
IMO any namespace that is not connected to active namenode is not up for serving requests from active namenode and hence it's not in good state. I got your point but the health of a datanode should be determined based on whether all BPs are connected to active in the federated setup, is that not the real factor determining the health of datanode?
> Making anything configurable doesn't justify having it in. if we are letting any user to use this via any config as well, then we should be sure enough it is necessary and good thing to do, we can not say ohh you configured it, now it is your problem...
I am not making claim only based on making this configurable feature. But it is reasonable enough to determine best course of action for given situation. The only recommendation I have is: user should be able to get the datanode to decide whether it should shutdown gracefully when it has not heard anything from active or observer namenode for the past x sec (50/60s or so).
I have tried my best to answer above questions. Please also take a look at the Jira/PR description where this idea has been taken from. We have seen issues with specific infra and until manually shutting down datanodes, we don't see any hope for improving availability, this has happened at multiple times.
Please keep in mind that cluster administrators in cloud native env do not have access to JMX metrics due to the security constraints.
Really appreciate all your points and suggestions Ayush, please take a look again.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] ayushtkn commented on pull request #5396: HDFS-16918. Optionally shut down datanode if it does not stay connected to active namenode
Posted by "ayushtkn (via GitHub)" <gi...@apache.org>.
ayushtkn commented on PR #5396:
URL: https://github.com/apache/hadoop/pull/5396#issuecomment-1433606299
Viraj I am Sorry I am totally against this. I am writing this because I don't want to ghost you and then if someone comes and agrees then shoot a vote against.
I have mentioned a bunch of reasons above, I even think if some client is connected to datanode and happily reading a file, he might get impacted, AFAIK block location can be cached as well and there are many other reasons, I don't to get you a list, I am pretty sure you would be aware of almost all of them...
A service like datanode killing itself doesn't sound something feasible to me at all. Having these hooks and all in a service which holds data, sounds just doing the same thing but opening ways to get exploited. That sound even more risky to me.
This is something a cluster Admin services should handle. A datanode going down or having troubles is something a basic use case for HDFS, that is where replication pitches in.
Ideally it should just alarm the admins and they should figure out what went wrong, may be a restart won't fix things and you would be in loop, doing a shutdown shoot BR to the ones your are still connected and then restart.
Metrics are there which can tell you which datanode is dead, so advanced cluster administrator services can leverage that. There is [JMXJsonServlet](https://github.com/apache/hadoop/blob/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/jmx/JMXJsonServlet.java) which can be leveraged. If those services can trigger a restart and do operations like shutdown, they should be allowed to fetch metrics as well. or there is an API which getDatanodeStats and which can take dead as param or so, and such a logic can be developed by a periodic check or so.
Regarding the cloud thing and metrics stuff. I got a chance to talk to some cloud Infra folks at my org and we do have ways to get metrics. I am not sharing how, because I don't know how professionally safe it is for me. But there are ways to do so.
So, This can be handled at deployment levels. Should be done there only and this auto shutdown logic based on some factors, I am just repeating myself I am totally against it....
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] virajjasani commented on pull request #5396: HDFS-16918. Optionally shut down datanode if it does not stay connected to active namenode
Posted by "virajjasani (via GitHub)" <gi...@apache.org>.
virajjasani commented on PR #5396:
URL: https://github.com/apache/hadoop/pull/5396#issuecomment-1433630564
> or there is an API which getDatanodeStats and which can take dead as param or so, and such a logic can be developed by a periodic check or so.
I think this is interesting point, indeed I should have considered this API that is already in use. Let me get back with some improvements to it and then I can suggest building K8S script around this, that should have similar impact as what we are trying to achieve within datanode itself.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] virajjasani commented on pull request #5396: HDFS-16918. Optionally shut down datanode if it does not stay connected to active namenode
Posted by "virajjasani (via GitHub)" <gi...@apache.org>.
virajjasani commented on PR #5396:
URL: https://github.com/apache/hadoop/pull/5396#issuecomment-1480735197
@ayushtkn to provide you some update, the issue for which I was thinking of having this optional behavior is already fixed by https://github.com/apache/hadoop/commit/26fba8701c97928bb2ed2e6b456ab5ba9513e0fe
We no longer see any transient connection failures after this commit. We are also trying to harmonize socket connection timeouts for all daemons to get them in sync with OS level settings.
Hence we no longer need the functionality of this PR but we did end up building some level of resilience in k8s operators to deal with transient failures for future i.e. bounce the DN pod if it doesn't stay connected consistently to active NN pod. Had to get some connection ports accessible, etc :)
Thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] virajjasani closed pull request #5396: HDFS-16918. Optionally shut down datanode if it does not stay connected to active namenode
Posted by "virajjasani (via GitHub)" <gi...@apache.org>.
virajjasani closed pull request #5396: HDFS-16918. Optionally shut down datanode if it does not stay connected to active namenode
URL: https://github.com/apache/hadoop/pull/5396
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #5396: HDFS-16918. Optionally shut down datanode if it does not stay connected to active namenode
Posted by "hadoop-yetus (via GitHub)" <gi...@apache.org>.
hadoop-yetus commented on PR #5396:
URL: https://github.com/apache/hadoop/pull/5396#issuecomment-1432445119
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 1m 22s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available. |
| +0 :ok: | xmllint | 0m 0s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 2 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 50m 49s | | trunk passed |
| +1 :green_heart: | compile | 1m 28s | | trunk passed with JDK Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | compile | 1m 24s | | trunk passed with JDK Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | checkstyle | 1m 8s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 29s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 8s | | trunk passed with JDK Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | javadoc | 1m 32s | | trunk passed with JDK Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 3m 35s | | trunk passed |
| +1 :green_heart: | shadedclient | 29m 26s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 30s | | the patch passed |
| +1 :green_heart: | compile | 1m 23s | | the patch passed with JDK Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | javac | 1m 23s | | the patch passed |
| +1 :green_heart: | compile | 1m 13s | | the patch passed with JDK Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | javac | 1m 13s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| -0 :warning: | checkstyle | 0m 54s | [/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5396/2/artifact/out/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt) | hadoop-hdfs-project/hadoop-hdfs: The patch generated 1 new + 325 unchanged - 0 fixed = 326 total (was 325) |
| +1 :green_heart: | mvnsite | 1m 23s | | the patch passed |
| -1 :x: | javadoc | 0m 53s | [/patch-javadoc-hadoop-hdfs-project_hadoop-hdfs-jdkUbuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5396/2/artifact/out/patch-javadoc-hadoop-hdfs-project_hadoop-hdfs-jdkUbuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.txt) | hadoop-hdfs in the patch failed with JDK Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04. |
| +1 :green_heart: | javadoc | 1m 26s | | the patch passed with JDK Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 3m 29s | | the patch passed |
| +1 :green_heart: | shadedclient | 29m 11s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| -1 :x: | unit | 251m 57s | [/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5396/2/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt) | hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 0m 43s | | The patch does not generate ASF License warnings. |
| | | 385m 0s | | |
| Reason | Tests |
|-------:|:------|
| Failed junit tests | hadoop.hdfs.server.namenode.TestAuditLogger |
| | hadoop.hdfs.server.namenode.TestFSNamesystemLockReport |
| | hadoop.hdfs.server.namenode.TestAuditLogs |
| | hadoop.hdfs.server.namenode.TestFsck |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.42 ServerAPI=1.42 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5396/2/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/5396 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets xmllint |
| uname | Linux 5d0f90e11c93 4.15.0-200-generic #211-Ubuntu SMP Thu Nov 24 18:16:04 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 3400be46ce4cf29409a2b031a8860a80d61313df |
| Default Java | Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5396/2/testReport/ |
| Max. process+thread count | 2431 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5396/2/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] virajjasani commented on pull request #5396: HDFS-16918. Optionally shut down datanode if it does not stay connected to active namenode
Posted by "virajjasani (via GitHub)" <gi...@apache.org>.
virajjasani commented on PR #5396:
URL: https://github.com/apache/hadoop/pull/5396#issuecomment-1433668204
Any BP service actor with "Namenode HA state" as "Active" and "Last Heartbeat Response" > 60s (configurable), should be treated as "State Active Namenode".
Maybe we can do that. Alright, sorry for adding up more and more comments, let me find the best way to expose things.
For cloud native infra, it's still not easy to let metrics be exposed to the pod where we want to but will have to go for some security approvals, will work on this in parallel.
Let me try fixing or at least normalizing the Namenode states in such a manner that we can expose "Stale Active Namenode" kind of Namenode HA state in metrics. That would be fairly easy for client to consume.
It should also not be backward incompatible given that HDFS-16902 has been a very recent change. So making changes now in it before it can make it to a release should be fine I guess.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] ayushtkn commented on pull request #5396: HDFS-16918. Optionally shut down datanode if it does not stay connected to active namenode
Posted by "ayushtkn (via GitHub)" <gi...@apache.org>.
ayushtkn commented on PR #5396:
URL: https://github.com/apache/hadoop/pull/5396#issuecomment-1430800093
From Admin I mean cluster Administrator services, they can keep a track of datanodes and decide on what needs to be done to the datanode.
If those services can shoot a restart if the datanode is shutdown, they can track in which situation the datanode needs to restarted.
Not checking the code, but comments:
- If the datanode is connected to observer namenode, it can serve requests, why we need to shutdown,
- Even if it is connected to standby, a failover happens and it will be in good shape, else if you restart a bunch of datanodes, the new namenode will be flooded by block reports and just increasing problems.
- If something gets messed up with Active namenode, you shutdown all, the BR are already heavy, you forced all other namenodes to handle them again, making failover more difficult. and if it is some faulty datanodes which lost connection, you didn't get that alarmed, and all Standby and Observers will keep on getting flooded by BRs, so in case Active NN literally dies and tries to failover to any of the Namenode which these Datanodes were connected, will be fed with unnecessary loads of BlockReports. (BR has an option of initial delay as well, it isn't like all bombard at once and you are sorted in 5-10 mins)
- If something got messed with the datanode, that is why it isn't able to connect to Active. If something is in Memory not persisted to disk, or some JMX parameter or N/W parameters which can be used to figure out things gets lost.
- That is the reason most cluster administrator in not so cool situations, show XYZ datanode is unhealthy or not, if in some case they don't it should be handled over there.
- In case of shared datanodes in a federated setup, say it is connected to Active for one Namespace and has completely lost touch with another, then? Restart to get both working? Don't restart so that at least one stays working? Both are correct in there own ways and situation and the datanode shouldn't be in a state to decide its fate for such reasons.
We do terminate Namenode is a bunch of conditions for sure, I don't want to get deep into those reasons, it is more or less preventive measure to terminate Namenode, if something serious has happened. This by architecture of HDFS itself isn't look very valid for HDFS.
PS. Making anything configurable doesn't justify having it in. if we are letting any user to use this via any config as well, then we should be sure enough it is necessary and good thing to do, we can not say ohh you configured it, now it is your problem...
I would say it is just pulling those cluster administrator things to datanode, like what Cloudera Manager or may be Ambari should do.
Not in favour of this...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] virajjasani commented on pull request #5396: HDFS-16918. Optionally shut down datanode if it does not stay connected to active namenode
Posted by "virajjasani (via GitHub)" <gi...@apache.org>.
virajjasani commented on PR #5396:
URL: https://github.com/apache/hadoop/pull/5396#issuecomment-1433612218
Thanks for the reply Ayush, appreciate it as always.
Are you saying that default implementation of "logging and/or exposing JMX metrics" for a given datanode if it doesn't stay connected is also not feasible according to you? I know we have metric that says "lastHeartbeat" and "lastHeartbeatResponseTime" but it's still difficult for user or script to apply a loop into BP service actor metrics rather than getting as simple log or metric as "this datanode has not heard from active namenode in the last 60s or so". Are you at least fine with keeping this as default implementation logic?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] virajjasani commented on pull request #5396: HDFS-16918. Optionally shut down datanode if it does not stay connected to active namenode
Posted by "virajjasani (via GitHub)" <gi...@apache.org>.
virajjasani commented on PR #5396:
URL: https://github.com/apache/hadoop/pull/5396#issuecomment-1433655322
Btw just to give you more insights, what I am worried about is cases like this:
In this case, dn is connected to active nn
<img width="1397" alt="Screenshot 2023-02-16 at 12 06 25 PM" src="https://user-images.githubusercontent.com/34790606/219476091-989e6a73-d54d-4c34-b60f-65152cf6980c.png">
However in this case, dn is live, it's TCP connection is lost to active nn and it is healthy and yet not connected to active nn
<img width="1371" alt="Screenshot 2023-02-16 at 12 06 37 PM" src="https://user-images.githubusercontent.com/34790606/219476263-9a039e14-9a0a-4c18-a103-fb448a61cf58.png">
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] ayushtkn commented on pull request #5396: HDFS-16918. Optionally shut down datanode if it does not stay connected to active namenode
Posted by "ayushtkn (via GitHub)" <gi...@apache.org>.
ayushtkn commented on PR #5396:
URL: https://github.com/apache/hadoop/pull/5396#issuecomment-1433706087
I think I have lost the flow now 😅
But I think using the getDataNodeStats is a cool thing to explore, it is under a read lock so not costly either, and would be easier to process also may be...
"Usually" around metrics, if things can be derived using the exposed ones, we don't coin new ones, generally, there are tools which can do that logics and show you fancy graphs and all also combining metrics together and doing maths on them as well...
The Dn case seems a corner case, it won't be very common and need to be careful around not getting pass a split-brain scenario. There are bunch of checks around though, but they are just to verify we don't get a false active claim acknowledged..
But just thinking about this case, it can be figured out by simple logic or scripts, if there are two claiming active, the one from which the last response time is less can be used for those decisions.
Something like
```
Variables to Store: activeNnId and LastActiveResponseTime=MAX
Fetch Metrics From DN
Iterate over all Namenodes.
Check if Active
NnLastResponseTime < LastActiveResponseTime
Store the nnId and last Response Time
else
Move Forward
if LastActiveResponseTime < configurable value
conclude dead and
do
<Whatever>
```
May be some if else or equality might have got inverted, just for idea sake...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] virajjasani commented on pull request #5396: HDFS-16918. Optionally shut down datanode if it does not stay connected to active namenode
Posted by "virajjasani (via GitHub)" <gi...@apache.org>.
virajjasani commented on PR #5396:
URL: https://github.com/apache/hadoop/pull/5396#issuecomment-1431843597
Please ignore test failures, they are not relevant. I will track them separately.
For the change, this is our real usecase. I have tested it on two clusters and deploy in prod as well. Limited infrastructure needs this kind of coverage but it is indeed required.
My only concern is that not every deployment framework has ability to perform action based on JMX metrics, specifically cloud native ones with more security falls in the criteria.
On the other hand, datanode is not much useful for long time if active namenode is not available if client is writing new blocks etc.
@jojochuang @tasanuma @tomscut could you please also take a look and provide your feedback?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #5396: HDFS-16918. Optionally shut down datanode if it does not stay connected to active namenode
Posted by "hadoop-yetus (via GitHub)" <gi...@apache.org>.
hadoop-yetus commented on PR #5396:
URL: https://github.com/apache/hadoop/pull/5396#issuecomment-1431048438
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 53s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available. |
| +0 :ok: | xmllint | 0m 1s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 2 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 46m 29s | | trunk passed |
| +1 :green_heart: | compile | 1m 29s | | trunk passed with JDK Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | compile | 1m 20s | | trunk passed with JDK Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | checkstyle | 1m 7s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 30s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 7s | | trunk passed with JDK Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | javadoc | 1m 26s | | trunk passed with JDK Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 3m 31s | | trunk passed |
| +1 :green_heart: | shadedclient | 28m 26s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 27s | | the patch passed |
| +1 :green_heart: | compile | 1m 23s | | the patch passed with JDK Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | javac | 1m 23s | | the patch passed |
| +1 :green_heart: | compile | 1m 15s | | the patch passed with JDK Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | javac | 1m 15s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| -0 :warning: | checkstyle | 0m 54s | [/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5396/1/artifact/out/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt) | hadoop-hdfs-project/hadoop-hdfs: The patch generated 1 new + 124 unchanged - 0 fixed = 125 total (was 124) |
| +1 :green_heart: | mvnsite | 1m 21s | | the patch passed |
| -1 :x: | javadoc | 0m 53s | [/patch-javadoc-hadoop-hdfs-project_hadoop-hdfs-jdkUbuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5396/1/artifact/out/patch-javadoc-hadoop-hdfs-project_hadoop-hdfs-jdkUbuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.txt) | hadoop-hdfs in the patch failed with JDK Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04. |
| +1 :green_heart: | javadoc | 1m 23s | | the patch passed with JDK Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 3m 26s | | the patch passed |
| +1 :green_heart: | shadedclient | 28m 27s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| -1 :x: | unit | 229m 42s | [/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5396/1/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt) | hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 0m 42s | | The patch does not generate ASF License warnings. |
| | | 355m 51s | | |
| Reason | Tests |
|-------:|:------|
| Failed junit tests | hadoop.hdfs.server.namenode.TestFSNamesystemLockReport |
| | hadoop.hdfs.server.namenode.TestAuditLogs |
| | hadoop.hdfs.server.namenode.TestAuditLogger |
| | hadoop.tools.TestHdfsConfigFields |
| | hadoop.hdfs.server.datanode.TestDirectoryScanner |
| | hadoop.hdfs.server.namenode.ha.TestObserverNode |
| | hadoop.hdfs.server.namenode.TestFsck |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.42 ServerAPI=1.42 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5396/1/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/5396 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets xmllint |
| uname | Linux 94bc9e51ca74 4.15.0-200-generic #211-Ubuntu SMP Thu Nov 24 18:16:04 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / afff2e429b4c2edc5423107fa1e706723883f45f |
| Default Java | Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5396/1/testReport/ |
| Max. process+thread count | 2170 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5396/1/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] virajjasani commented on pull request #5396: HDFS-16918. Optionally shut down datanode if it does not stay connected to active namenode
Posted by "virajjasani (via GitHub)" <gi...@apache.org>.
virajjasani commented on PR #5396:
URL: https://github.com/apache/hadoop/pull/5396#issuecomment-1433649342
Yes, we do have check at namenode side:
```
@Override // NameNodeMXBean
public String getDeadNodes() {
final Map<String, Map<String, Object>> info =
new HashMap<String, Map<String, Object>>();
final List<DatanodeDescriptor> dead = new ArrayList<DatanodeDescriptor>();
blockManager.getDatanodeManager().fetchDatanodes(null, dead, false);
for (DatanodeDescriptor node : dead) {
...
...
...
```
I am thinking more about `getDataNodeStats()` API
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] virajjasani commented on pull request #5396: HDFS-16918. Optionally shut down datanode if it does not stay connected to active namenode
Posted by "virajjasani (via GitHub)" <gi...@apache.org>.
virajjasani commented on PR #5396:
URL: https://github.com/apache/hadoop/pull/5396#issuecomment-1433614873
> I have mentioned a bunch of reasons above, I even think if some client is connected to datanode and happily reading a file, he might get impacted, AFAIK block location can be cached as well and there are many other reasons
Yes this is valid point but only until block locations stay cached at client :)
But I understand there is no point discussing on the same reasons as we would keep dragging the same point. Thanks!!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] virajjasani commented on pull request #5396: HDFS-16918. Optionally shut down datanode if it does not stay connected to active namenode
Posted by "virajjasani (via GitHub)" <gi...@apache.org>.
virajjasani commented on PR #5396:
URL: https://github.com/apache/hadoop/pull/5396#issuecomment-1430775234
For this change, the entire behavior is optional:
```
<property>
<name>dfs.datanode.health.activennconnect.timeout</name>
<value>0</value>
<description>
If the value is greater than 0, each datanode would try to determine if it is healthy i.e.
all block pools are correctly initialized and able to heartbeat to active namenode. At any
given time, if the datanode looses connection to active namenode for the duration of
milliseconds represented by the value of this config, it will attempt to shut down itself.
If the value is 0, datanode would not perform any such checks.
</description>
</property>
```
Without providing non-default value for this config, this behavior does not take any effect.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #5396: HDFS-16918. Optionally shut down datanode if it does not stay connected to active namenode
Posted by "hadoop-yetus (via GitHub)" <gi...@apache.org>.
hadoop-yetus commented on PR #5396:
URL: https://github.com/apache/hadoop/pull/5396#issuecomment-1432794232
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 1m 14s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available. |
| +0 :ok: | xmllint | 0m 0s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 2 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 47m 21s | | trunk passed |
| +1 :green_heart: | compile | 1m 35s | | trunk passed with JDK Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | compile | 1m 21s | | trunk passed with JDK Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | checkstyle | 1m 6s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 33s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 7s | | trunk passed with JDK Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | javadoc | 1m 31s | | trunk passed with JDK Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 3m 36s | | trunk passed |
| +1 :green_heart: | shadedclient | 29m 15s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 30s | | the patch passed |
| +1 :green_heart: | compile | 1m 22s | | the patch passed with JDK Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | javac | 1m 22s | | the patch passed |
| +1 :green_heart: | compile | 1m 16s | | the patch passed with JDK Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | javac | 1m 16s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | checkstyle | 0m 55s | | the patch passed |
| +1 :green_heart: | mvnsite | 1m 21s | | the patch passed |
| -1 :x: | javadoc | 0m 54s | [/patch-javadoc-hadoop-hdfs-project_hadoop-hdfs-jdkUbuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5396/3/artifact/out/patch-javadoc-hadoop-hdfs-project_hadoop-hdfs-jdkUbuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.txt) | hadoop-hdfs in the patch failed with JDK Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04. |
| +1 :green_heart: | javadoc | 1m 28s | | the patch passed with JDK Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 3m 30s | | the patch passed |
| +1 :green_heart: | shadedclient | 24m 14s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| -1 :x: | unit | 241m 33s | [/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5396/3/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt) | hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 0m 45s | | The patch does not generate ASF License warnings. |
| | | 366m 1s | | |
| Reason | Tests |
|-------:|:------|
| Failed junit tests | hadoop.hdfs.server.namenode.TestAuditLogger |
| | hadoop.hdfs.server.namenode.TestFSNamesystemLockReport |
| | hadoop.hdfs.server.datanode.TestDirectoryScanner |
| | hadoop.hdfs.server.namenode.TestAuditLogs |
| | hadoop.hdfs.server.namenode.TestFsck |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.42 ServerAPI=1.42 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5396/3/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/5396 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets xmllint |
| uname | Linux 3a3903bc636b 4.15.0-200-generic #211-Ubuntu SMP Thu Nov 24 18:16:04 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 91e7a726426d754ab4bfc548d1e9d51a7d4ef0e1 |
| Default Java | Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5396/3/testReport/ |
| Max. process+thread count | 2248 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5396/3/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] virajjasani commented on pull request #5396: HDFS-16918. Optionally shut down datanode if it does not stay connected to active namenode
Posted by "virajjasani (via GitHub)" <gi...@apache.org>.
virajjasani commented on PR #5396:
URL: https://github.com/apache/hadoop/pull/5396#issuecomment-1433662089
In the second case where dn is not connected to active nn, the BP offer service would still list active nn as nn-1. The only way for us to actually let a client (administrative applications in this case) know that the given dn is actually out of luck connecting to active nn is by exposing new metric which does internal check of looping through BP service actor metrics and making sure that all BPs have exactly one nn listed as "Active" and has lastHeartbeatReponseTime within few seconds.
This is the logic we somehow needs to expose for the clients (admins to take actions, for k8s, it will be some scripting that checks health of dn pods periodically).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] ayushtkn commented on pull request #5396: HDFS-16918. Optionally shut down datanode if it does not stay connected to active namenode
Posted by "ayushtkn (via GitHub)" <gi...@apache.org>.
ayushtkn commented on PR #5396:
URL: https://github.com/apache/hadoop/pull/5396#issuecomment-1433632089
The active NN knows that which datanode is dead. That is how it shows in the UI as well. There would be some param in the JMX which must be telling the state of the datanode to the active namenode. I can pull that out for you, if you want, but it is in the UI, so there would be a metric for sure, just being lazy to check the code again:
Datanode has metrics and you know post what time it is declared dead. Any service can have periodic health checks and have a check. Anyway you have a service which checks if datanode is dead and restarts, some logics here and there in that to have a periodic check to shoot a shutdown as well, should do.
<img width="1273" alt="image" src="https://user-images.githubusercontent.com/25608848/219470918-db38d602-984f-4baa-9860-aee19b2af646.png">
Code point of view implementing such a logic sounds very naive to me. or may be minimal effort thing
Not dragging the use case list either, because there ain't no end to that, client was X and he was in Y state and blah blah, datanode block reconstruction works, around block movements and it won't end....
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] virajjasani commented on pull request #5396: HDFS-16918. Optionally shut down datanode if it does not stay connected to active namenode
Posted by "virajjasani (via GitHub)" <gi...@apache.org>.
virajjasani commented on PR #5396:
URL: https://github.com/apache/hadoop/pull/5396#issuecomment-1430773119
In the large fleet of datanodes, any datanode that does not stay connected to active namenode due to any connectivity issue, can choose to get itself shutdown rather than impacting the availability and that is wise thing for datanode itself to do, of course not mandatorily but as an opt-in behavior.
While admin can do that, no human interaction can be fast enough to take action in the large scale cluster in just a matter of few seconds.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] virajjasani commented on pull request #5396: HDFS-16918. Optionally shut down datanode if it does not stay connected to active namenode
Posted by "virajjasani (via GitHub)" <gi...@apache.org>.
virajjasani commented on PR #5396:
URL: https://github.com/apache/hadoop/pull/5396#issuecomment-1432160563
In a distributed system, it is essential to have robust fail-fast mechanisms in place to prevent issues related to network partitioning. The system must be designed to prevent further degradation of availability and consistency in the event of a network partition. Several distributed systems offer fail-safe approaches, and for some, partition tolerance is critical to the extent that even a few seconds of heartbeat loss can trigger the removal of an application server instance from the cluster. For instance, a majority of zooKeeper clients utilize the ephemeral nodes for this purpose to make system reliable, fault-tolerant and strongly consistent in the event of network partition.
From the hdfs architecture viewpoint, it is crucial to understand the critical role that active and observer namenode play in file system operations. In a large-scale cluster, if the datanodes holding the same block (primary and replicas) lose connection to both active and observer namenodes for a significant amount of time, delaying the process of shutting down such datanodes and restarting it to re-establish the connection with the namenodes (assuming the active namenode is alive, assumption is important in the even of network partition to reestablish the connection) will further deteriorate the availability of the service. This scenario underscores the importance of resolving network partitioning.
This is a real use case for hdfs and it is not prudent to assume that every deployment or cluster management application must be able to restart datanodes based on JMX metrics, as this would introduce another application to resolve the network partition impact of hdfs. Besides, popular cluster management applications are not typically used in all cloud-native env. Even if these cluster management applications are deployed, certain security constraints may restrict their access to JMX metrics and prevent them from interfering with hdfs operations. The applications that can only trigger alerts for users based on set parameters (for instance, missing blocks > 0) are allowed to access JMX metrics.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] virajjasani commented on pull request #5396: HDFS-16918. Optionally shut down datanode if it does not stay connected to active namenode
Posted by "virajjasani (via GitHub)" <gi...@apache.org>.
virajjasani commented on PR #5396:
URL: https://github.com/apache/hadoop/pull/5396#issuecomment-1432167209
Created HDFS-16925 to fix regex expressions for namenode audit log tests
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] virajjasani commented on pull request #5396: HDFS-16918. Optionally shut down datanode if it does not stay connected to active namenode
Posted by "virajjasani (via GitHub)" <gi...@apache.org>.
virajjasani commented on PR #5396:
URL: https://github.com/apache/hadoop/pull/5396#issuecomment-1433513544
How about this? This change is working fine on the cluster as is and it is a real requirement as I explained in the [above comment](https://github.com/apache/hadoop/pull/5396#issuecomment-1432160563).
If we do not want to keep this change as is i.e. shutdown datanode if not connected to active namenode, how about we provide a pluggable implementation?
Let's say, by default, if the datanode does not stay connected to active namenode for 60s, in the default implementation (that we can provide with this patch) we take action of just logging (or maybe expose metric, whatever reviewers feel feasible) the fact that this datanode is not being useful for client as it has lost connected to active namenode for more than the past 60s. This is the default implementation that we can keep. On the other hand, users are allowed to have their own pluggable implementation so let's say if someone wants to shutdown datanode after 60s (default) of loosing connection, they will have to use new implementation with action as "shutdown datanode".
Hence, we have two configs for this change:
1. time duration for loosing connection (`dfs.datanode.health.activennconnect.timeout`) which we already have, but with default value as 60s
2. action to be performed by datanode when above threshold is reached (maybe something like `dfs.datanode.activennconnect.timeout.action.impl`) with default implementation that would take action of just logging or exposing metric as per consensus.
Any user can have their own implementation separately maintained and that implementation can take action of shutting down datanode, or running another script that could invoke dfsadmin action. Anything should be fine but now the code stays with users.
Thoughts?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] virajjasani commented on pull request #5396: HDFS-16918. Optionally shut down datanode if it does not stay connected to active namenode
Posted by "virajjasani (via GitHub)" <gi...@apache.org>.
virajjasani commented on PR #5396:
URL: https://github.com/apache/hadoop/pull/5396#issuecomment-1433713252
> The Dn case seems a corner case, it won't be very common and need to be careful around not getting pass a split-brain scenario. There are bunch of checks around though, but they are just to verify we don't get a false active claim acknowledged..
Oh yes, this was my first focus, I tried adding bunch of logs internally just to ensure if we are seeing some bugs here, but so far things look good. It's the TCP connection that Envoy infra is messing up (only sometimes, not often). But nvm, this is still worth spending time on.
Thanks for all good points, I am going to close this PR mostly soon (just figuring out a few more details) so that it doesn't pile up in the open PRs.
Thanks a lot for spending a lot of your time here, it's just so priceless!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org