You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spark.apache.org by Pralabh Kumar <pr...@gmail.com> on 2022/04/29 16:22:55 UTC
Issue on Spark on K8s with Proxy user on Kerberized HDFS : Spark-25355
Hi dev Team
Spark-25355 added the functionality of the proxy user on K8s . However
proxy user on K8s with Kerberized HDFS is not working . It is throwing
exception and
22/04/21 17:50:30 WARN Client: Exception encountered while connecting to
the server : org.apache.hadoop.security.AccessControlException: Client
cannot authenticate via:[TOKEN, KERBEROS]
Exception in thread "main" java.net.ConnectException: Call From <driverpod>
to <namenode> failed on connection exception: java.net.ConnectException:
Connection refused; For more details see: http://
wiki.apache.org/hadoop/ConnectionRefused
at
java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native
Method)
at
java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(Unknown
Source)
at
java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown
Source)
at java.base/java.lang.reflect.Constructor.newInstance(Unknown Source)
at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:831)
at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:755)
at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1501)
at org.apache.hadoop.ipc.Client.call(Client.java:1443)
at org.apache.hadoop.ipc.Client.call(Client.java:1353)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:228)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:116)
at com.sun.proxy.$Proxy14.getFileInfo(Unknown Source)
at
On debugging deep , we found the proxy user doesn't have access to
delegation tokens in case of K8s .SparkSubmit.submit explicitly creating
the proxy user and this user doesn't have delegation token.
Please help me with the same.
Regards
Pralabh Kumar
Re: Issue on Spark on K8s with Proxy user on Kerberized HDFS : Spark-25355
Posted by Gabor Somogyi <ga...@gmail.com>.
If you take a look at the jiras then ..33 is the newly opened same issue
which is discussed on ..55.
The quality of the logs are the exact same which is useless for real
analysis...
Not going too far spark-submit has cluster mode, the logs are partial and
contains client mode (so they don't belong together).
> By having a good JIRA where the problem is discussed and the same with
the PR description helps when others try to change the code in the future.
Not sure what you mean here. The guys who made the changes understands the
feature/change which is not necessarily true for all others.
This is double true for security features which is super hairy area...
G
On Tue, May 3, 2022 at 9:28 PM Bjørn Jørgensen <bj...@gmail.com>
wrote:
> What is the JIRA ticket for this problem?
>
> SPARK-25355 <https://issues.apache.org/jira/browse/SPARK-25355> was
> marked as resolved for 2 years ago. But now there are a lot of new comments
> on whether things work and not.
>
> SPARK-39033 Support --proxy-user for Spark on K8s not working
> <https://issues.apache.org/jira/projects/SPARK/issues/SPARK-39033> is
> this yours problem?
>
> By having a good JIRA where the problem is discussed and the same with the
> PR description helps when others try to change the code in the future.
>
>
> tir. 3. mai 2022 kl. 19:20 skrev Pralabh Kumar <pr...@gmail.com>:
>
>> Hi Steve
>>
>> Thx for the input . Actually I have wrongly put the error stack
>> trace(complete error trace and other details are available in the Jira ,
>> apologies for the same. ) . The error is related to authenticatication
>> and it's happening only when the proxy user option is being used ,
>> otherwise things are working fine.
>>
>> Also my intention wasn't to point out any mistakes or
>> escalate the problem to the community. I was just looking for help and
>> direction to solve the issue.
>>
>> 22/04/26 08:54:40 DEBUG Client: closing ipc connection to <server>/<ip>:8020: org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS]
>> java.io.IOException: org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS]
>> at org.apache.hadoop.ipc.Client$Connection$1.run(Client.java:757)
>> at java.base/java.security.AccessController.doPrivileged(Native Method)
>> at java.base/javax.security.auth.Subject.doAs(Unknown Source)
>> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729)
>> at org.apache.hadoop.ipc.Client$Connection.handleSaslConnectionFailure(Client.java:720)
>> at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:813)
>> at org.apache.hadoop.ipc.Client$Connection.access$3600(Client.java:410)
>> at org.apache.hadoop.ipc.Client.getConnection(Client.java:1558)
>> at org.apache.hadoop.ipc.Client.call(Client.java:1389)
>> at org.apache.hadoop.ipc.Client.call(Client.java:1353)
>> at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:228)
>> at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:116)
>> at com.sun.proxy.$Proxy14.getFileInfo(Unknown Source)
>> at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:900)
>> at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
>> at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
>> at java.base/java.lang.reflect.Method.invoke(Unknown Source)
>> at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:422)
>> at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:165)
>> at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:157)
>> at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95)
>> at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:359)
>> at com.sun.proxy.$Proxy15.getFileInfo(Unknown Source)
>> at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1654)
>> at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1579)
>> at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1576)
>> at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
>> at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1591)
>> at org.apache.hadoop.fs.Globber.getFileStatus(Globber.java:65)
>> at org.apache.hadoop.fs.Globber.doGlob(Globber.java:270)
>> at org.apache.hadoop.fs.Globber.glob(Globber.java:149)
>> at org.apache.hadoop.fs.FileSystem.globStatus(FileSystem.java:2067)
>> at org.apache.spark.util.DependencyUtils$.resolveGlobPath(DependencyUtils.scala:318)
>> at org.apache.spark.util.DependencyUtils$.$anonfun$resolveGlobPaths$2(DependencyUtils.scala:273)
>> at org.apache.spark.util.DependencyUtils$.$anonfun$resolveGlobPaths$2$adapted(DependencyUtils.scala:271)
>> at scala.collection.TraversableLike.$anonfun$flatMap$1(TraversableLike.scala:293)
>> at scala.collection.IndexedSeqOptimized.foreach(IndexedSeqOptimized.scala:36)
>> at scala.collection.IndexedSeqOptimized.foreach$(IndexedSeqOptimized.scala:33)
>> at scala.collection.mutable.WrappedArray.foreach(WrappedArray.scala:38)
>> at scala.collection.TraversableLike.flatMap(TraversableLike.scala:293)
>> at scala.collection.TraversableLike.flatMap$(TraversableLike.scala:290)
>> at scala.collection.AbstractTraversable.flatMap(Traversable.scala:108)
>> at org.apache.spark.util.DependencyUtils$.resolveGlobPaths(DependencyUtils.scala:271)
>> at org.apache.spark.deploy.SparkSubmit.$anonfun$prepareSubmitEnvironment$4(SparkSubmit.scala:364)
>> at scala.Option.map(Option.scala:230)
>> at org.apache.spark.deploy.SparkSubmit.prepareSubmitEnvironment(SparkSubmit.scala:364)
>> at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:898)
>> at org.apache.spark.deploy.SparkSubmit$$anon$1.run(SparkSubmit.scala:165)
>> at org.apache.spark.deploy.SparkSubmit$$anon$1.run(SparkSubmit.scala:163)
>> at java.base/java.security.AccessController.doPrivileged(Native Method)
>> at java.base/javax.security.auth.Subject.doAs(Unknown Source)
>> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729)
>> at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:163)
>> at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:203)
>> at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:90)
>> at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1043)
>> at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1052)
>>
>>
>>
>> Again Thx for the input and advice regarding documentation and
>> apologies for putting the wrong error stack earlier.
>>
>> Regards
>> Pralabh Kumar
>> On Tue, May 3, 2022 at 7:39 PM Steve Loughran <st...@cloudera.com>
>> wrote:
>>
>>>
>>> Prablah, did you follow the URL provided in the exception message? i put
>>> a lot of effort in to improving the diagnostics, where the wiki articles
>>> are part of the troubleshooing process
>>> https://issues.apache.org/jira/browse/HADOOP-7469
>>>
>>> it's really disappointing when people escalate the problem to open
>>> source developers before trying to fix the problem themselves, in this
>>> case, read the error message.
>>>
>>> now, if there is some k8s related issue which makes this more common,
>>> you are encouraged to update the wiki entry with a new cause. documentation
>>> is an important contribution to open source projects, and if you have
>>> discovered a new way to recreate the failure, it would be welcome. which
>>> reminds me, i have to add something to connection reset and docker which
>>> comes down to "turn off http keepalive in maven builds"
>>>
>>> -Steve
>>>
>>>
>>>
>>>
>>>
>>> On Sat, 30 Apr 2022 at 10:45, Gabor Somogyi <ga...@gmail.com>
>>> wrote:
>>>
>>>> Hi,
>>>>
>>>> Please be aware that ConnectionRefused exception is has nothing to do
>>>> w/ authentication. See the description from Hadoop wiki:
>>>> "You get a ConnectionRefused
>>>> <https://cwiki.apache.org/confluence/display/HADOOP2/ConnectionRefused> Exception
>>>> when there is a machine at the address specified, but there is no program
>>>> listening on the specific TCP port the client is using -and there is no
>>>> firewall in the way silently dropping TCP connection requests. If you do
>>>> not know what a TCP connection request is, please consult the
>>>> specification <http://www.ietf.org/rfc/rfc793.txt>."
>>>>
>>>> This means the namenode on host:port is not reachable in the TCP layer.
>>>> Maybe there are multiple issues but I'm pretty sure that something is wrong
>>>> in the K8S net config.
>>>>
>>>> BR,
>>>> G
>>>>
>>>>
>>>> On Fri, Apr 29, 2022 at 6:23 PM Pralabh Kumar <pr...@gmail.com>
>>>> wrote:
>>>>
>>>>> Hi dev Team
>>>>>
>>>>> Spark-25355 added the functionality of the proxy user on K8s . However
>>>>> proxy user on K8s with Kerberized HDFS is not working . It is throwing
>>>>> exception and
>>>>>
>>>>> 22/04/21 17:50:30 WARN Client: Exception encountered while connecting
>>>>> to the server : org.apache.hadoop.security.AccessControlException: Client
>>>>> cannot authenticate via:[TOKEN, KERBEROS]
>>>>>
>>>>>
>>>>> Exception in thread "main" java.net.ConnectException: Call From
>>>>> <driverpod> to <namenode> failed on connection exception:
>>>>> java.net.ConnectException: Connection refused; For more details see: http:
>>>>> //wiki.apache.org/hadoop/ConnectionRefused
>>>>>
>>>>> at
>>>>> java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native
>>>>> Method)
>>>>>
>>>>> at
>>>>> java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(Unknown
>>>>> Source)
>>>>>
>>>>> at
>>>>> java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown
>>>>> Source)
>>>>>
>>>>> at java.base/java.lang.reflect.Constructor.newInstance(Unknown
>>>>> Source)
>>>>>
>>>>> at
>>>>> org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:831)
>>>>>
>>>>> at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:755)
>>>>>
>>>>> at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1501)
>>>>>
>>>>> at org.apache.hadoop.ipc.Client.call(Client.java:1443)
>>>>>
>>>>> at org.apache.hadoop.ipc.Client.call(Client.java:1353)
>>>>>
>>>>> at
>>>>> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:228)
>>>>>
>>>>> at
>>>>> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:116)
>>>>>
>>>>> at com.sun.proxy.$Proxy14.getFileInfo(Unknown Source)
>>>>>
>>>>> at
>>>>>
>>>>>
>>>>>
>>>>> On debugging deep , we found the proxy user doesn't have access to
>>>>> delegation tokens in case of K8s .SparkSubmit.submit explicitly creating
>>>>> the proxy user and this user doesn't have delegation token.
>>>>>
>>>>>
>>>>> Please help me with the same.
>>>>>
>>>>>
>>>>> Regards
>>>>>
>>>>> Pralabh Kumar
>>>>>
>>>>>
>>>>>
>>>>>
>
> --
> Bjørn Jørgensen
> Vestre Aspehaug 4, 6010 Ålesund
> Norge
>
> +47 480 94 297
>
Re: Issue on Spark on K8s with Proxy user on Kerberized HDFS : Spark-25355
Posted by Bjørn Jørgensen <bj...@gmail.com>.
What is the JIRA ticket for this problem?
SPARK-25355 <https://issues.apache.org/jira/browse/SPARK-25355> was marked
as resolved for 2 years ago. But now there are a lot of new comments on
whether things work and not.
SPARK-39033 Support --proxy-user for Spark on K8s not working
<https://issues.apache.org/jira/projects/SPARK/issues/SPARK-39033> is this
yours problem?
By having a good JIRA where the problem is discussed and the same with the
PR description helps when others try to change the code in the future.
tir. 3. mai 2022 kl. 19:20 skrev Pralabh Kumar <pr...@gmail.com>:
> Hi Steve
>
> Thx for the input . Actually I have wrongly put the error stack
> trace(complete error trace and other details are available in the Jira ,
> apologies for the same. ) . The error is related to authenticatication
> and it's happening only when the proxy user option is being used ,
> otherwise things are working fine.
>
> Also my intention wasn't to point out any mistakes or escalate the problem
> to the community. I was just looking for help and direction to solve the
> issue.
>
> 22/04/26 08:54:40 DEBUG Client: closing ipc connection to <server>/<ip>:8020: org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS]
> java.io.IOException: org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS]
> at org.apache.hadoop.ipc.Client$Connection$1.run(Client.java:757)
> at java.base/java.security.AccessController.doPrivileged(Native Method)
> at java.base/javax.security.auth.Subject.doAs(Unknown Source)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729)
> at org.apache.hadoop.ipc.Client$Connection.handleSaslConnectionFailure(Client.java:720)
> at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:813)
> at org.apache.hadoop.ipc.Client$Connection.access$3600(Client.java:410)
> at org.apache.hadoop.ipc.Client.getConnection(Client.java:1558)
> at org.apache.hadoop.ipc.Client.call(Client.java:1389)
> at org.apache.hadoop.ipc.Client.call(Client.java:1353)
> at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:228)
> at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:116)
> at com.sun.proxy.$Proxy14.getFileInfo(Unknown Source)
> at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:900)
> at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
> at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
> at java.base/java.lang.reflect.Method.invoke(Unknown Source)
> at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:422)
> at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:165)
> at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:157)
> at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95)
> at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:359)
> at com.sun.proxy.$Proxy15.getFileInfo(Unknown Source)
> at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1654)
> at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1579)
> at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1576)
> at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1591)
> at org.apache.hadoop.fs.Globber.getFileStatus(Globber.java:65)
> at org.apache.hadoop.fs.Globber.doGlob(Globber.java:270)
> at org.apache.hadoop.fs.Globber.glob(Globber.java:149)
> at org.apache.hadoop.fs.FileSystem.globStatus(FileSystem.java:2067)
> at org.apache.spark.util.DependencyUtils$.resolveGlobPath(DependencyUtils.scala:318)
> at org.apache.spark.util.DependencyUtils$.$anonfun$resolveGlobPaths$2(DependencyUtils.scala:273)
> at org.apache.spark.util.DependencyUtils$.$anonfun$resolveGlobPaths$2$adapted(DependencyUtils.scala:271)
> at scala.collection.TraversableLike.$anonfun$flatMap$1(TraversableLike.scala:293)
> at scala.collection.IndexedSeqOptimized.foreach(IndexedSeqOptimized.scala:36)
> at scala.collection.IndexedSeqOptimized.foreach$(IndexedSeqOptimized.scala:33)
> at scala.collection.mutable.WrappedArray.foreach(WrappedArray.scala:38)
> at scala.collection.TraversableLike.flatMap(TraversableLike.scala:293)
> at scala.collection.TraversableLike.flatMap$(TraversableLike.scala:290)
> at scala.collection.AbstractTraversable.flatMap(Traversable.scala:108)
> at org.apache.spark.util.DependencyUtils$.resolveGlobPaths(DependencyUtils.scala:271)
> at org.apache.spark.deploy.SparkSubmit.$anonfun$prepareSubmitEnvironment$4(SparkSubmit.scala:364)
> at scala.Option.map(Option.scala:230)
> at org.apache.spark.deploy.SparkSubmit.prepareSubmitEnvironment(SparkSubmit.scala:364)
> at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:898)
> at org.apache.spark.deploy.SparkSubmit$$anon$1.run(SparkSubmit.scala:165)
> at org.apache.spark.deploy.SparkSubmit$$anon$1.run(SparkSubmit.scala:163)
> at java.base/java.security.AccessController.doPrivileged(Native Method)
> at java.base/javax.security.auth.Subject.doAs(Unknown Source)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729)
> at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:163)
> at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:203)
> at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:90)
> at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1043)
> at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1052)
>
>
>
> Again Thx for the input and advice regarding documentation and
> apologies for putting the wrong error stack earlier.
>
> Regards
> Pralabh Kumar
> On Tue, May 3, 2022 at 7:39 PM Steve Loughran <st...@cloudera.com> wrote:
>
>>
>> Prablah, did you follow the URL provided in the exception message? i put
>> a lot of effort in to improving the diagnostics, where the wiki articles
>> are part of the troubleshooing process
>> https://issues.apache.org/jira/browse/HADOOP-7469
>>
>> it's really disappointing when people escalate the problem to open source
>> developers before trying to fix the problem themselves, in this case, read
>> the error message.
>>
>> now, if there is some k8s related issue which makes this more common, you
>> are encouraged to update the wiki entry with a new cause. documentation is
>> an important contribution to open source projects, and if you have
>> discovered a new way to recreate the failure, it would be welcome. which
>> reminds me, i have to add something to connection reset and docker which
>> comes down to "turn off http keepalive in maven builds"
>>
>> -Steve
>>
>>
>>
>>
>>
>> On Sat, 30 Apr 2022 at 10:45, Gabor Somogyi <ga...@gmail.com>
>> wrote:
>>
>>> Hi,
>>>
>>> Please be aware that ConnectionRefused exception is has nothing to do w/
>>> authentication. See the description from Hadoop wiki:
>>> "You get a ConnectionRefused
>>> <https://cwiki.apache.org/confluence/display/HADOOP2/ConnectionRefused> Exception
>>> when there is a machine at the address specified, but there is no program
>>> listening on the specific TCP port the client is using -and there is no
>>> firewall in the way silently dropping TCP connection requests. If you do
>>> not know what a TCP connection request is, please consult the
>>> specification <http://www.ietf.org/rfc/rfc793.txt>."
>>>
>>> This means the namenode on host:port is not reachable in the TCP layer.
>>> Maybe there are multiple issues but I'm pretty sure that something is wrong
>>> in the K8S net config.
>>>
>>> BR,
>>> G
>>>
>>>
>>> On Fri, Apr 29, 2022 at 6:23 PM Pralabh Kumar <pr...@gmail.com>
>>> wrote:
>>>
>>>> Hi dev Team
>>>>
>>>> Spark-25355 added the functionality of the proxy user on K8s . However
>>>> proxy user on K8s with Kerberized HDFS is not working . It is throwing
>>>> exception and
>>>>
>>>> 22/04/21 17:50:30 WARN Client: Exception encountered while connecting
>>>> to the server : org.apache.hadoop.security.AccessControlException: Client
>>>> cannot authenticate via:[TOKEN, KERBEROS]
>>>>
>>>>
>>>> Exception in thread "main" java.net.ConnectException: Call From
>>>> <driverpod> to <namenode> failed on connection exception:
>>>> java.net.ConnectException: Connection refused; For more details see: http:
>>>> //wiki.apache.org/hadoop/ConnectionRefused
>>>>
>>>> at
>>>> java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native
>>>> Method)
>>>>
>>>> at
>>>> java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(Unknown
>>>> Source)
>>>>
>>>> at
>>>> java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown
>>>> Source)
>>>>
>>>> at java.base/java.lang.reflect.Constructor.newInstance(Unknown
>>>> Source)
>>>>
>>>> at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:831)
>>>>
>>>> at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:755)
>>>>
>>>> at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1501)
>>>>
>>>> at org.apache.hadoop.ipc.Client.call(Client.java:1443)
>>>>
>>>> at org.apache.hadoop.ipc.Client.call(Client.java:1353)
>>>>
>>>> at
>>>> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:228)
>>>>
>>>> at
>>>> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:116)
>>>>
>>>> at com.sun.proxy.$Proxy14.getFileInfo(Unknown Source)
>>>>
>>>> at
>>>>
>>>>
>>>>
>>>> On debugging deep , we found the proxy user doesn't have access to
>>>> delegation tokens in case of K8s .SparkSubmit.submit explicitly creating
>>>> the proxy user and this user doesn't have delegation token.
>>>>
>>>>
>>>> Please help me with the same.
>>>>
>>>>
>>>> Regards
>>>>
>>>> Pralabh Kumar
>>>>
>>>>
>>>>
>>>>
--
Bjørn Jørgensen
Vestre Aspehaug 4, 6010 Ålesund
Norge
+47 480 94 297
Re: Issue on Spark on K8s with Proxy user on Kerberized HDFS : Spark-25355
Posted by Pralabh Kumar <pr...@gmail.com>.
Hi Steve
Thx for the input . Actually I have wrongly put the error stack
trace(complete error trace and other details are available in the Jira ,
apologies for the same. ) . The error is related to authenticatication
and it's happening only when the proxy user option is being used ,
otherwise things are working fine.
Also my intention wasn't to point out any mistakes or escalate the problem
to the community. I was just looking for help and direction to solve the
issue.
22/04/26 08:54:40 DEBUG Client: closing ipc connection to
<server>/<ip>:8020: org.apache.hadoop.security.AccessControlException:
Client cannot authenticate via:[TOKEN, KERBEROS]
java.io.IOException:
org.apache.hadoop.security.AccessControlException: Client cannot
authenticate via:[TOKEN, KERBEROS]
at org.apache.hadoop.ipc.Client$Connection$1.run(Client.java:757)
at java.base/java.security.AccessController.doPrivileged(Native Method)
at java.base/javax.security.auth.Subject.doAs(Unknown Source)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729)
at org.apache.hadoop.ipc.Client$Connection.handleSaslConnectionFailure(Client.java:720)
at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:813)
at org.apache.hadoop.ipc.Client$Connection.access$3600(Client.java:410)
at org.apache.hadoop.ipc.Client.getConnection(Client.java:1558)
at org.apache.hadoop.ipc.Client.call(Client.java:1389)
at org.apache.hadoop.ipc.Client.call(Client.java:1353)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:228)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:116)
at com.sun.proxy.$Proxy14.getFileInfo(Unknown Source)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:900)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native
Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown
Source)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown
Source)
at java.base/java.lang.reflect.Method.invoke(Unknown Source)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:422)
at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:165)
at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:157)
at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:359)
at com.sun.proxy.$Proxy15.getFileInfo(Unknown Source)
at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1654)
at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1579)
at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1576)
at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1591)
at org.apache.hadoop.fs.Globber.getFileStatus(Globber.java:65)
at org.apache.hadoop.fs.Globber.doGlob(Globber.java:270)
at org.apache.hadoop.fs.Globber.glob(Globber.java:149)
at org.apache.hadoop.fs.FileSystem.globStatus(FileSystem.java:2067)
at org.apache.spark.util.DependencyUtils$.resolveGlobPath(DependencyUtils.scala:318)
at org.apache.spark.util.DependencyUtils$.$anonfun$resolveGlobPaths$2(DependencyUtils.scala:273)
at org.apache.spark.util.DependencyUtils$.$anonfun$resolveGlobPaths$2$adapted(DependencyUtils.scala:271)
at scala.collection.TraversableLike.$anonfun$flatMap$1(TraversableLike.scala:293)
at scala.collection.IndexedSeqOptimized.foreach(IndexedSeqOptimized.scala:36)
at scala.collection.IndexedSeqOptimized.foreach$(IndexedSeqOptimized.scala:33)
at scala.collection.mutable.WrappedArray.foreach(WrappedArray.scala:38)
at scala.collection.TraversableLike.flatMap(TraversableLike.scala:293)
at scala.collection.TraversableLike.flatMap$(TraversableLike.scala:290)
at scala.collection.AbstractTraversable.flatMap(Traversable.scala:108)
at org.apache.spark.util.DependencyUtils$.resolveGlobPaths(DependencyUtils.scala:271)
at org.apache.spark.deploy.SparkSubmit.$anonfun$prepareSubmitEnvironment$4(SparkSubmit.scala:364)
at scala.Option.map(Option.scala:230)
at org.apache.spark.deploy.SparkSubmit.prepareSubmitEnvironment(SparkSubmit.scala:364)
at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:898)
at org.apache.spark.deploy.SparkSubmit$$anon$1.run(SparkSubmit.scala:165)
at org.apache.spark.deploy.SparkSubmit$$anon$1.run(SparkSubmit.scala:163)
at java.base/java.security.AccessController.doPrivileged(Native Method)
at java.base/javax.security.auth.Subject.doAs(Unknown Source)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729)
at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:163)
at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:203)
at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:90)
at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1043)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1052)
Again Thx for the input and advice regarding documentation and
apologies for putting the wrong error stack earlier.
Regards
Pralabh Kumar
On Tue, May 3, 2022 at 7:39 PM Steve Loughran <st...@cloudera.com> wrote:
>
> Prablah, did you follow the URL provided in the exception message? i put a
> lot of effort in to improving the diagnostics, where the wiki articles are
> part of the troubleshooing process
> https://issues.apache.org/jira/browse/HADOOP-7469
>
> it's really disappointing when people escalate the problem to open source
> developers before trying to fix the problem themselves, in this case, read
> the error message.
>
> now, if there is some k8s related issue which makes this more common, you
> are encouraged to update the wiki entry with a new cause. documentation is
> an important contribution to open source projects, and if you have
> discovered a new way to recreate the failure, it would be welcome. which
> reminds me, i have to add something to connection reset and docker which
> comes down to "turn off http keepalive in maven builds"
>
> -Steve
>
>
>
>
>
> On Sat, 30 Apr 2022 at 10:45, Gabor Somogyi <ga...@gmail.com>
> wrote:
>
>> Hi,
>>
>> Please be aware that ConnectionRefused exception is has nothing to do w/
>> authentication. See the description from Hadoop wiki:
>> "You get a ConnectionRefused
>> <https://cwiki.apache.org/confluence/display/HADOOP2/ConnectionRefused> Exception
>> when there is a machine at the address specified, but there is no program
>> listening on the specific TCP port the client is using -and there is no
>> firewall in the way silently dropping TCP connection requests. If you do
>> not know what a TCP connection request is, please consult the
>> specification <http://www.ietf.org/rfc/rfc793.txt>."
>>
>> This means the namenode on host:port is not reachable in the TCP layer.
>> Maybe there are multiple issues but I'm pretty sure that something is wrong
>> in the K8S net config.
>>
>> BR,
>> G
>>
>>
>> On Fri, Apr 29, 2022 at 6:23 PM Pralabh Kumar <pr...@gmail.com>
>> wrote:
>>
>>> Hi dev Team
>>>
>>> Spark-25355 added the functionality of the proxy user on K8s . However
>>> proxy user on K8s with Kerberized HDFS is not working . It is throwing
>>> exception and
>>>
>>> 22/04/21 17:50:30 WARN Client: Exception encountered while connecting to
>>> the server : org.apache.hadoop.security.AccessControlException: Client
>>> cannot authenticate via:[TOKEN, KERBEROS]
>>>
>>>
>>> Exception in thread "main" java.net.ConnectException: Call From
>>> <driverpod> to <namenode> failed on connection exception:
>>> java.net.ConnectException: Connection refused; For more details see: http:
>>> //wiki.apache.org/hadoop/ConnectionRefused
>>>
>>> at
>>> java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native
>>> Method)
>>>
>>> at
>>> java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(Unknown
>>> Source)
>>>
>>> at
>>> java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown
>>> Source)
>>>
>>> at java.base/java.lang.reflect.Constructor.newInstance(Unknown
>>> Source)
>>>
>>> at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:831)
>>>
>>> at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:755)
>>>
>>> at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1501)
>>>
>>> at org.apache.hadoop.ipc.Client.call(Client.java:1443)
>>>
>>> at org.apache.hadoop.ipc.Client.call(Client.java:1353)
>>>
>>> at
>>> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:228)
>>>
>>> at
>>> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:116)
>>>
>>> at com.sun.proxy.$Proxy14.getFileInfo(Unknown Source)
>>>
>>> at
>>>
>>>
>>>
>>> On debugging deep , we found the proxy user doesn't have access to
>>> delegation tokens in case of K8s .SparkSubmit.submit explicitly creating
>>> the proxy user and this user doesn't have delegation token.
>>>
>>>
>>> Please help me with the same.
>>>
>>>
>>> Regards
>>>
>>> Pralabh Kumar
>>>
>>>
>>>
>>>
Re: Issue on Spark on K8s with Proxy user on Kerberized HDFS : Spark-25355
Posted by Steve Loughran <st...@cloudera.com.INVALID>.
Prablah, did you follow the URL provided in the exception message? i put a
lot of effort in to improving the diagnostics, where the wiki articles are
part of the troubleshooing process
https://issues.apache.org/jira/browse/HADOOP-7469
it's really disappointing when people escalate the problem to open source
developers before trying to fix the problem themselves, in this case, read
the error message.
now, if there is some k8s related issue which makes this more common, you
are encouraged to update the wiki entry with a new cause. documentation is
an important contribution to open source projects, and if you have
discovered a new way to recreate the failure, it would be welcome. which
reminds me, i have to add something to connection reset and docker which
comes down to "turn off http keepalive in maven builds"
-Steve
On Sat, 30 Apr 2022 at 10:45, Gabor Somogyi <ga...@gmail.com>
wrote:
> Hi,
>
> Please be aware that ConnectionRefused exception is has nothing to do w/
> authentication. See the description from Hadoop wiki:
> "You get a ConnectionRefused
> <https://cwiki.apache.org/confluence/display/HADOOP2/ConnectionRefused> Exception
> when there is a machine at the address specified, but there is no program
> listening on the specific TCP port the client is using -and there is no
> firewall in the way silently dropping TCP connection requests. If you do
> not know what a TCP connection request is, please consult the
> specification <http://www.ietf.org/rfc/rfc793.txt>."
>
> This means the namenode on host:port is not reachable in the TCP layer.
> Maybe there are multiple issues but I'm pretty sure that something is wrong
> in the K8S net config.
>
> BR,
> G
>
>
> On Fri, Apr 29, 2022 at 6:23 PM Pralabh Kumar <pr...@gmail.com>
> wrote:
>
>> Hi dev Team
>>
>> Spark-25355 added the functionality of the proxy user on K8s . However
>> proxy user on K8s with Kerberized HDFS is not working . It is throwing
>> exception and
>>
>> 22/04/21 17:50:30 WARN Client: Exception encountered while connecting to
>> the server : org.apache.hadoop.security.AccessControlException: Client
>> cannot authenticate via:[TOKEN, KERBEROS]
>>
>>
>> Exception in thread "main" java.net.ConnectException: Call From
>> <driverpod> to <namenode> failed on connection exception:
>> java.net.ConnectException: Connection refused; For more details see: http:
>> //wiki.apache.org/hadoop/ConnectionRefused
>>
>> at
>> java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native
>> Method)
>>
>> at
>> java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(Unknown
>> Source)
>>
>> at
>> java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown
>> Source)
>>
>> at java.base/java.lang.reflect.Constructor.newInstance(Unknown Source)
>>
>> at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:831)
>>
>> at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:755)
>>
>> at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1501)
>>
>> at org.apache.hadoop.ipc.Client.call(Client.java:1443)
>>
>> at org.apache.hadoop.ipc.Client.call(Client.java:1353)
>>
>> at
>> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:228)
>>
>> at
>> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:116)
>>
>> at com.sun.proxy.$Proxy14.getFileInfo(Unknown Source)
>>
>> at
>>
>>
>>
>> On debugging deep , we found the proxy user doesn't have access to
>> delegation tokens in case of K8s .SparkSubmit.submit explicitly creating
>> the proxy user and this user doesn't have delegation token.
>>
>>
>> Please help me with the same.
>>
>>
>> Regards
>>
>> Pralabh Kumar
>>
>>
>>
>>
Re: Issue on Spark on K8s with Proxy user on Kerberized HDFS : Spark-25355
Posted by Gabor Somogyi <ga...@gmail.com>.
Hi,
Please be aware that ConnectionRefused exception is has nothing to do w/
authentication. See the description from Hadoop wiki:
"You get a ConnectionRefused
<https://cwiki.apache.org/confluence/display/HADOOP2/ConnectionRefused>
Exception
when there is a machine at the address specified, but there is no program
listening on the specific TCP port the client is using -and there is no
firewall in the way silently dropping TCP connection requests. If you do
not know what a TCP connection request is, please consult the specification
<http://www.ietf.org/rfc/rfc793.txt>."
This means the namenode on host:port is not reachable in the TCP layer.
Maybe there are multiple issues but I'm pretty sure that something is wrong
in the K8S net config.
BR,
G
On Fri, Apr 29, 2022 at 6:23 PM Pralabh Kumar <pr...@gmail.com>
wrote:
> Hi dev Team
>
> Spark-25355 added the functionality of the proxy user on K8s . However
> proxy user on K8s with Kerberized HDFS is not working . It is throwing
> exception and
>
> 22/04/21 17:50:30 WARN Client: Exception encountered while connecting to
> the server : org.apache.hadoop.security.AccessControlException: Client
> cannot authenticate via:[TOKEN, KERBEROS]
>
>
> Exception in thread "main" java.net.ConnectException: Call From
> <driverpod> to <namenode> failed on connection exception:
> java.net.ConnectException: Connection refused; For more details see: http:
> //wiki.apache.org/hadoop/ConnectionRefused
>
> at
> java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> Method)
>
> at
> java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(Unknown
> Source)
>
> at
> java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown
> Source)
>
> at java.base/java.lang.reflect.Constructor.newInstance(Unknown Source)
>
> at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:831)
>
> at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:755)
>
> at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1501)
>
> at org.apache.hadoop.ipc.Client.call(Client.java:1443)
>
> at org.apache.hadoop.ipc.Client.call(Client.java:1353)
>
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:228)
>
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:116)
>
> at com.sun.proxy.$Proxy14.getFileInfo(Unknown Source)
>
> at
>
>
>
> On debugging deep , we found the proxy user doesn't have access to
> delegation tokens in case of K8s .SparkSubmit.submit explicitly creating
> the proxy user and this user doesn't have delegation token.
>
>
> Please help me with the same.
>
>
> Regards
>
> Pralabh Kumar
>
>
>
>