You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@flink.apache.org by Ken Krugler <kk...@transpac.com> on 2018/01/24 22:40:25 UTC

Problems caused by use of hadoop classpath bash command

Hi all,

With Flink 1.4 and FLINK-7477 <https://issues.apache.org/jira/browse/FLINK-7477>, I ran into a problem with jar versions for HttpCore, when using the AWS SDK to read from S3.

I believe the issue is that even when setting classloader.resolve-order to child-first in flink-conf.yaml, the change to put all jars returned by “hadoop classpath” on the classpath means that classes in these jars are found before the classes in my shaded Flink uber jar.

If I ensure that I don’t have the “hadoop” command set up on my Bash path, then I don’t run into this issue.

Does this make sense, or is there something else going on that I can fix to avoid this situation?

Regards,

— Ken

--------------------------------------------
http://about.me/kkrugler
+1 530-210-6378