You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@tez.apache.org by "László Bodor (Jira)" <ji...@apache.org> on 2022/01/16 08:43:00 UTC
[jira] [Resolved] (TEZ-4376) Remove javax.security from the tez-auxservices shaded jar
[ https://issues.apache.org/jira/browse/TEZ-4376?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
László Bodor resolved TEZ-4376.
-------------------------------
Resolution: Fixed
> Remove javax.security from the tez-auxservices shaded jar
> ---------------------------------------------------------
>
> Key: TEZ-4376
> URL: https://issues.apache.org/jira/browse/TEZ-4376
> Project: Apache Tez
> Issue Type: Sub-task
> Affects Versions: 0.10.1
> Reporter: László Bodor
> Assignee: László Bodor
> Priority: Major
> Labels: ShuffleHandler
> Fix For: 0.10.2
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> When I try to start yarn nodemanager with tez shufflehandler configured, I get this:
> {code:java}
> java.lang.NoClassDefFoundError: org/apache/tez/shaded/javax/security/cert/X509Certificate
> at org.apache.tez.shaded.io.netty.util.internal.EmptyArrays.<clinit>(EmptyArrays.java:38)
> at org.apache.tez.auxservices.ShuffleHandler.defaultProcessId(ShuffleHandler.java:601)
> at org.apache.tez.auxservices.ShuffleHandler.serviceStart(ShuffleHandler.java:567)
> at org.apache.hadoop.service.AbstractService.start(AbstractService.java:194)
> at org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxiliaryServiceWithCustomClassLoader.serviceStart(AuxiliaryServiceWithCustomClassLoader.java:76)
> at org.apache.hadoop.service.AbstractService.start(AbstractService.java:194)
> at org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices.serviceStart(AuxServices.java:313)
> at org.apache.hadoop.service.AbstractService.start(AbstractService.java:194)
> at org.apache.hadoop.service.CompositeService.serviceStart(CompositeService.java:121)
> at org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl.serviceStart(ContainerManagerImpl.java:654)
> at org.apache.hadoop.service.AbstractService.start(AbstractService.java:194)
> at org.apache.hadoop.service.CompositeService.serviceStart(CompositeService.java:121)
> at org.apache.hadoop.service.AbstractService.start(AbstractService.java:194)
> at org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartNodeManager(NodeManager.java:975)
> at org.apache.hadoop.yarn.server.nodemanager.NodeManager.main(NodeManager.java:1054)
> Caused by: java.lang.ClassNotFoundException: org.apache.tez.shaded.javax.security.cert.X509Certificate
> at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
> at org.apache.hadoop.util.ApplicationClassLoader.loadClass(ApplicationClassLoader.java:189)
> at org.apache.hadoop.util.ApplicationClassLoader.loadClass(ApplicationClassLoader.java:157)
> ... 15 more
> {code}
> this is because in netty 4, EmptyArrays class - which is originally referred to by DefaultChannelId, here the stacktrace shows that I simply copied it to ShuffleHandler code to insert some logging - refers to X509Certificate
> I tried to add javax.security exclude as:
> {code:java}
> <relocation>
> <pattern>javax</pattern>
> <shadedPattern>org.apache.tez.shaded.$0</shadedPattern>
> <excludes>
> <exclude>javax.crypto.*</exclude>
> <exclude>javax.security.*</exclude> <!-- HERE -->
> </excludes>
> </relocation>
> {code}
> but it didn't work, not sure why
> *when I completely removed javax relocation, it started to work*
> I'm tempted to make javax.security exclusion work, but first I need to understand why javax relocation is needed at all
> is there a chance that we want to package a custom javax to be used instead of what is provided on nodemanager's classpath?
--
This message was sent by Atlassian Jira
(v8.20.1#820001)