You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by John Lilley <jo...@redpoint.net> on 2014/08/23 22:40:57 UTC
winutils and security
Up to this point, we've been able to run as a Hadoop client application (HDFS + YARN) from Windows without winutils.exe, despite always seeing messages complaining about it in the logs. However, we are now integrating with secure clusters and are having some mysterious errors. Before these errors occur, messages come from Hadoop like those below. Is it possible that this is leading to our security failures? (I posted previously about that problem but got no response). What does winutils.exe have to do with security, if anything?
Thanks
john
The relevant portions of the log seem to be:
2014-08-23 14:33:10 DEBUG org.apache.hadoop.metrics2.impl.MetricsSystemImpl: UgiMetrics, User and group related metrics
2014-08-23 14:33:10 DEBUG org.apache.hadoop.security.Groups: Creating new Groups object
2014-08-23 14:33:10 DEBUG org.apache.hadoop.util.NativeCodeLoader: Trying to load the custom-built native-hadoop library...
2014-08-23 14:33:10 DEBUG org.apache.hadoop.util.NativeCodeLoader: Failed to load native-hadoop with error: java.lang.UnsatisfiedLinkError: no hadoop in java.library.path
2014-08-23 14:33:10 DEBUG org.apache.hadoop.util.NativeCodeLoader: java.library.path=[...]
2014-08-23 14:33:10 WARN org.apache.hadoop.util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using
builtin-java classes where applicable
2014-08-23 14:33:10 DEBUG org.apache.hadoop.security.JniBasedUnixGroupsMappingWithFallback: Falling back to shell based
2014-08-23 14:33:10 DEBUG org.apache.hadoop.security.JniBasedUnixGroupsMappingWithFallback: Group mapping impl=org.apache.hadoop.security.ShellBasedUnixGroupsMapping
2014-08-23 14:33:10 DEBUG org.apache.hadoop.security.Groups: Group mapping impl=org.apache.hadoop.security.JniBasedUnixGroupsMappingWithFallback; cacheTimeout=300000
RE: winutils and security
Posted by John Lilley <jo...@redpoint.net>.
One more follow up, in case someone stumbles across this in the future. From what we can tell, the Hadoop security initialization is very sensitive to startup order, and this has been confirmed by discussions with other people. The only thing that we've been able to make work at all reliably uses the following sequence, in a single thread, preferably very close to startup.
1. Load/set Configuration that can be used by HDFS and YARN.
2. Set UserGroupInformation() and log in using either password or keytab.
3. Open the HDFS FileSystem
4. Call addDelegationTokens() to extract delegated Credentials for HDFS and keep them around.
Once this has been done, it appears tha tall is well. We can use those Credentials in the YARN application master launch context.
john
From: John Lilley [mailto:john.lilley@redpoint.net]
Sent: Sunday, August 24, 2014 11:05 AM
To: user@hadoop.apache.org
Subject: RE: winutils and security
Following up on this, I was able to extract a winutils.exe and Hadoop.dll from a Hadoop install for Windows, and set up HADDOP_HOME and PATH to find them. It makes no difference to security, apparently.
John
From: John Lilley [mailto:john.lilley@redpoint.net]
Sent: Saturday, August 23, 2014 2:41 PM
To: 'user@hadoop.apache.org'
Subject: winutils and security
Up to this point, we've been able to run as a Hadoop client application (HDFS + YARN) from Windows without winutils.exe, despite always seeing messages complaining about it in the logs. However, we are now integrating with secure clusters and are having some mysterious errors. Before these errors occur, messages come from Hadoop like those below. Is it possible that this is leading to our security failures? (I posted previously about that problem but got no response). What does winutils.exe have to do with security, if anything?
Thanks
john
The relevant portions of the log seem to be:
2014-08-23 14:33:10 DEBUG org.apache.hadoop.metrics2.impl.MetricsSystemImpl: UgiMetrics, User and group related metrics
2014-08-23 14:33:10 DEBUG org.apache.hadoop.security.Groups: Creating new Groups object
2014-08-23 14:33:10 DEBUG org.apache.hadoop.util.NativeCodeLoader: Trying to load the custom-built native-hadoop library...
2014-08-23 14:33:10 DEBUG org.apache.hadoop.util.NativeCodeLoader: Failed to load native-hadoop with error: java.lang.UnsatisfiedLinkError: no hadoop in java.library.path
2014-08-23 14:33:10 DEBUG org.apache.hadoop.util.NativeCodeLoader: java.library.path=[...]
2014-08-23 14:33:10 WARN org.apache.hadoop.util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using
builtin-java classes where applicable
2014-08-23 14:33:10 DEBUG org.apache.hadoop.security.JniBasedUnixGroupsMappingWithFallback: Falling back to shell based
2014-08-23 14:33:10 DEBUG org.apache.hadoop.security.JniBasedUnixGroupsMappingWithFallback: Group mapping impl=org.apache.hadoop.security.ShellBasedUnixGroupsMapping
2014-08-23 14:33:10 DEBUG org.apache.hadoop.security.Groups: Group mapping impl=org.apache.hadoop.security.JniBasedUnixGroupsMappingWithFallback; cacheTimeout=300000
RE: winutils and security
Posted by John Lilley <jo...@redpoint.net>.
One more follow up, in case someone stumbles across this in the future. From what we can tell, the Hadoop security initialization is very sensitive to startup order, and this has been confirmed by discussions with other people. The only thing that we've been able to make work at all reliably uses the following sequence, in a single thread, preferably very close to startup.
1. Load/set Configuration that can be used by HDFS and YARN.
2. Set UserGroupInformation() and log in using either password or keytab.
3. Open the HDFS FileSystem
4. Call addDelegationTokens() to extract delegated Credentials for HDFS and keep them around.
Once this has been done, it appears tha tall is well. We can use those Credentials in the YARN application master launch context.
john
From: John Lilley [mailto:john.lilley@redpoint.net]
Sent: Sunday, August 24, 2014 11:05 AM
To: user@hadoop.apache.org
Subject: RE: winutils and security
Following up on this, I was able to extract a winutils.exe and Hadoop.dll from a Hadoop install for Windows, and set up HADDOP_HOME and PATH to find them. It makes no difference to security, apparently.
John
From: John Lilley [mailto:john.lilley@redpoint.net]
Sent: Saturday, August 23, 2014 2:41 PM
To: 'user@hadoop.apache.org'
Subject: winutils and security
Up to this point, we've been able to run as a Hadoop client application (HDFS + YARN) from Windows without winutils.exe, despite always seeing messages complaining about it in the logs. However, we are now integrating with secure clusters and are having some mysterious errors. Before these errors occur, messages come from Hadoop like those below. Is it possible that this is leading to our security failures? (I posted previously about that problem but got no response). What does winutils.exe have to do with security, if anything?
Thanks
john
The relevant portions of the log seem to be:
2014-08-23 14:33:10 DEBUG org.apache.hadoop.metrics2.impl.MetricsSystemImpl: UgiMetrics, User and group related metrics
2014-08-23 14:33:10 DEBUG org.apache.hadoop.security.Groups: Creating new Groups object
2014-08-23 14:33:10 DEBUG org.apache.hadoop.util.NativeCodeLoader: Trying to load the custom-built native-hadoop library...
2014-08-23 14:33:10 DEBUG org.apache.hadoop.util.NativeCodeLoader: Failed to load native-hadoop with error: java.lang.UnsatisfiedLinkError: no hadoop in java.library.path
2014-08-23 14:33:10 DEBUG org.apache.hadoop.util.NativeCodeLoader: java.library.path=[...]
2014-08-23 14:33:10 WARN org.apache.hadoop.util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using
builtin-java classes where applicable
2014-08-23 14:33:10 DEBUG org.apache.hadoop.security.JniBasedUnixGroupsMappingWithFallback: Falling back to shell based
2014-08-23 14:33:10 DEBUG org.apache.hadoop.security.JniBasedUnixGroupsMappingWithFallback: Group mapping impl=org.apache.hadoop.security.ShellBasedUnixGroupsMapping
2014-08-23 14:33:10 DEBUG org.apache.hadoop.security.Groups: Group mapping impl=org.apache.hadoop.security.JniBasedUnixGroupsMappingWithFallback; cacheTimeout=300000
RE: winutils and security
Posted by John Lilley <jo...@redpoint.net>.
One more follow up, in case someone stumbles across this in the future. From what we can tell, the Hadoop security initialization is very sensitive to startup order, and this has been confirmed by discussions with other people. The only thing that we've been able to make work at all reliably uses the following sequence, in a single thread, preferably very close to startup.
1. Load/set Configuration that can be used by HDFS and YARN.
2. Set UserGroupInformation() and log in using either password or keytab.
3. Open the HDFS FileSystem
4. Call addDelegationTokens() to extract delegated Credentials for HDFS and keep them around.
Once this has been done, it appears tha tall is well. We can use those Credentials in the YARN application master launch context.
john
From: John Lilley [mailto:john.lilley@redpoint.net]
Sent: Sunday, August 24, 2014 11:05 AM
To: user@hadoop.apache.org
Subject: RE: winutils and security
Following up on this, I was able to extract a winutils.exe and Hadoop.dll from a Hadoop install for Windows, and set up HADDOP_HOME and PATH to find them. It makes no difference to security, apparently.
John
From: John Lilley [mailto:john.lilley@redpoint.net]
Sent: Saturday, August 23, 2014 2:41 PM
To: 'user@hadoop.apache.org'
Subject: winutils and security
Up to this point, we've been able to run as a Hadoop client application (HDFS + YARN) from Windows without winutils.exe, despite always seeing messages complaining about it in the logs. However, we are now integrating with secure clusters and are having some mysterious errors. Before these errors occur, messages come from Hadoop like those below. Is it possible that this is leading to our security failures? (I posted previously about that problem but got no response). What does winutils.exe have to do with security, if anything?
Thanks
john
The relevant portions of the log seem to be:
2014-08-23 14:33:10 DEBUG org.apache.hadoop.metrics2.impl.MetricsSystemImpl: UgiMetrics, User and group related metrics
2014-08-23 14:33:10 DEBUG org.apache.hadoop.security.Groups: Creating new Groups object
2014-08-23 14:33:10 DEBUG org.apache.hadoop.util.NativeCodeLoader: Trying to load the custom-built native-hadoop library...
2014-08-23 14:33:10 DEBUG org.apache.hadoop.util.NativeCodeLoader: Failed to load native-hadoop with error: java.lang.UnsatisfiedLinkError: no hadoop in java.library.path
2014-08-23 14:33:10 DEBUG org.apache.hadoop.util.NativeCodeLoader: java.library.path=[...]
2014-08-23 14:33:10 WARN org.apache.hadoop.util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using
builtin-java classes where applicable
2014-08-23 14:33:10 DEBUG org.apache.hadoop.security.JniBasedUnixGroupsMappingWithFallback: Falling back to shell based
2014-08-23 14:33:10 DEBUG org.apache.hadoop.security.JniBasedUnixGroupsMappingWithFallback: Group mapping impl=org.apache.hadoop.security.ShellBasedUnixGroupsMapping
2014-08-23 14:33:10 DEBUG org.apache.hadoop.security.Groups: Group mapping impl=org.apache.hadoop.security.JniBasedUnixGroupsMappingWithFallback; cacheTimeout=300000
RE: winutils and security
Posted by John Lilley <jo...@redpoint.net>.
One more follow up, in case someone stumbles across this in the future. From what we can tell, the Hadoop security initialization is very sensitive to startup order, and this has been confirmed by discussions with other people. The only thing that we've been able to make work at all reliably uses the following sequence, in a single thread, preferably very close to startup.
1. Load/set Configuration that can be used by HDFS and YARN.
2. Set UserGroupInformation() and log in using either password or keytab.
3. Open the HDFS FileSystem
4. Call addDelegationTokens() to extract delegated Credentials for HDFS and keep them around.
Once this has been done, it appears tha tall is well. We can use those Credentials in the YARN application master launch context.
john
From: John Lilley [mailto:john.lilley@redpoint.net]
Sent: Sunday, August 24, 2014 11:05 AM
To: user@hadoop.apache.org
Subject: RE: winutils and security
Following up on this, I was able to extract a winutils.exe and Hadoop.dll from a Hadoop install for Windows, and set up HADDOP_HOME and PATH to find them. It makes no difference to security, apparently.
John
From: John Lilley [mailto:john.lilley@redpoint.net]
Sent: Saturday, August 23, 2014 2:41 PM
To: 'user@hadoop.apache.org'
Subject: winutils and security
Up to this point, we've been able to run as a Hadoop client application (HDFS + YARN) from Windows without winutils.exe, despite always seeing messages complaining about it in the logs. However, we are now integrating with secure clusters and are having some mysterious errors. Before these errors occur, messages come from Hadoop like those below. Is it possible that this is leading to our security failures? (I posted previously about that problem but got no response). What does winutils.exe have to do with security, if anything?
Thanks
john
The relevant portions of the log seem to be:
2014-08-23 14:33:10 DEBUG org.apache.hadoop.metrics2.impl.MetricsSystemImpl: UgiMetrics, User and group related metrics
2014-08-23 14:33:10 DEBUG org.apache.hadoop.security.Groups: Creating new Groups object
2014-08-23 14:33:10 DEBUG org.apache.hadoop.util.NativeCodeLoader: Trying to load the custom-built native-hadoop library...
2014-08-23 14:33:10 DEBUG org.apache.hadoop.util.NativeCodeLoader: Failed to load native-hadoop with error: java.lang.UnsatisfiedLinkError: no hadoop in java.library.path
2014-08-23 14:33:10 DEBUG org.apache.hadoop.util.NativeCodeLoader: java.library.path=[...]
2014-08-23 14:33:10 WARN org.apache.hadoop.util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using
builtin-java classes where applicable
2014-08-23 14:33:10 DEBUG org.apache.hadoop.security.JniBasedUnixGroupsMappingWithFallback: Falling back to shell based
2014-08-23 14:33:10 DEBUG org.apache.hadoop.security.JniBasedUnixGroupsMappingWithFallback: Group mapping impl=org.apache.hadoop.security.ShellBasedUnixGroupsMapping
2014-08-23 14:33:10 DEBUG org.apache.hadoop.security.Groups: Group mapping impl=org.apache.hadoop.security.JniBasedUnixGroupsMappingWithFallback; cacheTimeout=300000
RE: winutils and security
Posted by John Lilley <jo...@redpoint.net>.
Following up on this, I was able to extract a winutils.exe and Hadoop.dll from a Hadoop install for Windows, and set up HADDOP_HOME and PATH to find them. It makes no difference to security, apparently.
John
From: John Lilley [mailto:john.lilley@redpoint.net]
Sent: Saturday, August 23, 2014 2:41 PM
To: 'user@hadoop.apache.org'
Subject: winutils and security
Up to this point, we've been able to run as a Hadoop client application (HDFS + YARN) from Windows without winutils.exe, despite always seeing messages complaining about it in the logs. However, we are now integrating with secure clusters and are having some mysterious errors. Before these errors occur, messages come from Hadoop like those below. Is it possible that this is leading to our security failures? (I posted previously about that problem but got no response). What does winutils.exe have to do with security, if anything?
Thanks
john
The relevant portions of the log seem to be:
2014-08-23 14:33:10 DEBUG org.apache.hadoop.metrics2.impl.MetricsSystemImpl: UgiMetrics, User and group related metrics
2014-08-23 14:33:10 DEBUG org.apache.hadoop.security.Groups: Creating new Groups object
2014-08-23 14:33:10 DEBUG org.apache.hadoop.util.NativeCodeLoader: Trying to load the custom-built native-hadoop library...
2014-08-23 14:33:10 DEBUG org.apache.hadoop.util.NativeCodeLoader: Failed to load native-hadoop with error: java.lang.UnsatisfiedLinkError: no hadoop in java.library.path
2014-08-23 14:33:10 DEBUG org.apache.hadoop.util.NativeCodeLoader: java.library.path=[...]
2014-08-23 14:33:10 WARN org.apache.hadoop.util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using
builtin-java classes where applicable
2014-08-23 14:33:10 DEBUG org.apache.hadoop.security.JniBasedUnixGroupsMappingWithFallback: Falling back to shell based
2014-08-23 14:33:10 DEBUG org.apache.hadoop.security.JniBasedUnixGroupsMappingWithFallback: Group mapping impl=org.apache.hadoop.security.ShellBasedUnixGroupsMapping
2014-08-23 14:33:10 DEBUG org.apache.hadoop.security.Groups: Group mapping impl=org.apache.hadoop.security.JniBasedUnixGroupsMappingWithFallback; cacheTimeout=300000
RE: winutils and security
Posted by John Lilley <jo...@redpoint.net>.
Following up on this, I was able to extract a winutils.exe and Hadoop.dll from a Hadoop install for Windows, and set up HADDOP_HOME and PATH to find them. It makes no difference to security, apparently.
John
From: John Lilley [mailto:john.lilley@redpoint.net]
Sent: Saturday, August 23, 2014 2:41 PM
To: 'user@hadoop.apache.org'
Subject: winutils and security
Up to this point, we've been able to run as a Hadoop client application (HDFS + YARN) from Windows without winutils.exe, despite always seeing messages complaining about it in the logs. However, we are now integrating with secure clusters and are having some mysterious errors. Before these errors occur, messages come from Hadoop like those below. Is it possible that this is leading to our security failures? (I posted previously about that problem but got no response). What does winutils.exe have to do with security, if anything?
Thanks
john
The relevant portions of the log seem to be:
2014-08-23 14:33:10 DEBUG org.apache.hadoop.metrics2.impl.MetricsSystemImpl: UgiMetrics, User and group related metrics
2014-08-23 14:33:10 DEBUG org.apache.hadoop.security.Groups: Creating new Groups object
2014-08-23 14:33:10 DEBUG org.apache.hadoop.util.NativeCodeLoader: Trying to load the custom-built native-hadoop library...
2014-08-23 14:33:10 DEBUG org.apache.hadoop.util.NativeCodeLoader: Failed to load native-hadoop with error: java.lang.UnsatisfiedLinkError: no hadoop in java.library.path
2014-08-23 14:33:10 DEBUG org.apache.hadoop.util.NativeCodeLoader: java.library.path=[...]
2014-08-23 14:33:10 WARN org.apache.hadoop.util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using
builtin-java classes where applicable
2014-08-23 14:33:10 DEBUG org.apache.hadoop.security.JniBasedUnixGroupsMappingWithFallback: Falling back to shell based
2014-08-23 14:33:10 DEBUG org.apache.hadoop.security.JniBasedUnixGroupsMappingWithFallback: Group mapping impl=org.apache.hadoop.security.ShellBasedUnixGroupsMapping
2014-08-23 14:33:10 DEBUG org.apache.hadoop.security.Groups: Group mapping impl=org.apache.hadoop.security.JniBasedUnixGroupsMappingWithFallback; cacheTimeout=300000
RE: winutils and security
Posted by John Lilley <jo...@redpoint.net>.
Following up on this, I was able to extract a winutils.exe and Hadoop.dll from a Hadoop install for Windows, and set up HADDOP_HOME and PATH to find them. It makes no difference to security, apparently.
John
From: John Lilley [mailto:john.lilley@redpoint.net]
Sent: Saturday, August 23, 2014 2:41 PM
To: 'user@hadoop.apache.org'
Subject: winutils and security
Up to this point, we've been able to run as a Hadoop client application (HDFS + YARN) from Windows without winutils.exe, despite always seeing messages complaining about it in the logs. However, we are now integrating with secure clusters and are having some mysterious errors. Before these errors occur, messages come from Hadoop like those below. Is it possible that this is leading to our security failures? (I posted previously about that problem but got no response). What does winutils.exe have to do with security, if anything?
Thanks
john
The relevant portions of the log seem to be:
2014-08-23 14:33:10 DEBUG org.apache.hadoop.metrics2.impl.MetricsSystemImpl: UgiMetrics, User and group related metrics
2014-08-23 14:33:10 DEBUG org.apache.hadoop.security.Groups: Creating new Groups object
2014-08-23 14:33:10 DEBUG org.apache.hadoop.util.NativeCodeLoader: Trying to load the custom-built native-hadoop library...
2014-08-23 14:33:10 DEBUG org.apache.hadoop.util.NativeCodeLoader: Failed to load native-hadoop with error: java.lang.UnsatisfiedLinkError: no hadoop in java.library.path
2014-08-23 14:33:10 DEBUG org.apache.hadoop.util.NativeCodeLoader: java.library.path=[...]
2014-08-23 14:33:10 WARN org.apache.hadoop.util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using
builtin-java classes where applicable
2014-08-23 14:33:10 DEBUG org.apache.hadoop.security.JniBasedUnixGroupsMappingWithFallback: Falling back to shell based
2014-08-23 14:33:10 DEBUG org.apache.hadoop.security.JniBasedUnixGroupsMappingWithFallback: Group mapping impl=org.apache.hadoop.security.ShellBasedUnixGroupsMapping
2014-08-23 14:33:10 DEBUG org.apache.hadoop.security.Groups: Group mapping impl=org.apache.hadoop.security.JniBasedUnixGroupsMappingWithFallback; cacheTimeout=300000
RE: winutils and security
Posted by John Lilley <jo...@redpoint.net>.
Following up on this, I was able to extract a winutils.exe and Hadoop.dll from a Hadoop install for Windows, and set up HADDOP_HOME and PATH to find them. It makes no difference to security, apparently.
John
From: John Lilley [mailto:john.lilley@redpoint.net]
Sent: Saturday, August 23, 2014 2:41 PM
To: 'user@hadoop.apache.org'
Subject: winutils and security
Up to this point, we've been able to run as a Hadoop client application (HDFS + YARN) from Windows without winutils.exe, despite always seeing messages complaining about it in the logs. However, we are now integrating with secure clusters and are having some mysterious errors. Before these errors occur, messages come from Hadoop like those below. Is it possible that this is leading to our security failures? (I posted previously about that problem but got no response). What does winutils.exe have to do with security, if anything?
Thanks
john
The relevant portions of the log seem to be:
2014-08-23 14:33:10 DEBUG org.apache.hadoop.metrics2.impl.MetricsSystemImpl: UgiMetrics, User and group related metrics
2014-08-23 14:33:10 DEBUG org.apache.hadoop.security.Groups: Creating new Groups object
2014-08-23 14:33:10 DEBUG org.apache.hadoop.util.NativeCodeLoader: Trying to load the custom-built native-hadoop library...
2014-08-23 14:33:10 DEBUG org.apache.hadoop.util.NativeCodeLoader: Failed to load native-hadoop with error: java.lang.UnsatisfiedLinkError: no hadoop in java.library.path
2014-08-23 14:33:10 DEBUG org.apache.hadoop.util.NativeCodeLoader: java.library.path=[...]
2014-08-23 14:33:10 WARN org.apache.hadoop.util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using
builtin-java classes where applicable
2014-08-23 14:33:10 DEBUG org.apache.hadoop.security.JniBasedUnixGroupsMappingWithFallback: Falling back to shell based
2014-08-23 14:33:10 DEBUG org.apache.hadoop.security.JniBasedUnixGroupsMappingWithFallback: Group mapping impl=org.apache.hadoop.security.ShellBasedUnixGroupsMapping
2014-08-23 14:33:10 DEBUG org.apache.hadoop.security.Groups: Group mapping impl=org.apache.hadoop.security.JniBasedUnixGroupsMappingWithFallback; cacheTimeout=300000