[jira] [Created] (FLINK-23972) NoClassDefFoundError: Jars from yarn.ship-files are missing in the classpath of the client

["Arseniy Tashoyan (Jira)" <ji...@apache.org>]

Arseniy Tashoyan created FLINK-23972:
----------------------------------------

             Summary: NoClassDefFoundError: Jars from yarn.ship-files are missing in the classpath of the client
                 Key: FLINK-23972
                 URL: https://issues.apache.org/jira/browse/FLINK-23972
             Project: Flink
          Issue Type: Bug
          Components: Client / Job Submission
    Affects Versions: 1.13.2, 1.12.4
         Environment: Flink 1.12.4 or 1.13.2

YARN

Running the application as follows:
{code:bash}
# Find additional jars under $lib_dir
# Join jars with semicolon
jars=$(mk_string \; $(find "$lib_dir" -type f -name \*.jar))

# Provide additional jars via yarn.ship-files
/opt/flink/bin/flink \
  run \
  --target yarn-per-job \
  --detached \
  --class com.example.DummyMain \
  -Dyarn.ship-files="$jars" \
  dummy-app-1.0.jar
{code}
            Reporter: Arseniy Tashoyan


Flink provides a way to inject additional jars into the classpath of the Job Manager and Task Manager: the configuration option [yarn.ship-files|https://ci.apache.org/projects/flink/flink-docs-release-1.13/docs/deployment/config/#yarn-ship-files]. However, an applications fails to submit to YARN if it uses a class from the additional jars. The execution graph is built inside the client before the application goes to the YARN cluster. Meanwhile, the client part (the command-line tool *flink*) misses the additional jars in its classpath.

*Example with Scalaz library.*
{code:scala}
package com.example

import ALenses.aLens
import org.apache.flink.streaming.api.scala.{StreamExecutionEnvironment, createTypeInformation}
import scalaz.Lens
import scalaz.Lens.lensu

object DummyMain {

  def main(args: Array[String]): Unit = {
    val env = StreamExecutionEnvironment.getExecutionEnvironment

    /* Way 1: Using Scalaz inside the client code. */
    val inputData = Seq(1, 2, 3).map { i => aLens.set(A(0), i) }
    /* Way 2: Not using Scalaz inside the client code. */
    // val inputData = Seq(1, 2, 3).map(A)

    val input = env.fromCollection(inputData)
    val output = input
      .map { a =>
        /* Using Scalaz inside Task Manager. */
        s"DUMMY: ${aLens.get(a)}"
      }
    output.print()
    env.execute()
    ()
  }

}

case class A(
    i: Int
)

object ALenses {

  val aLens: Lens[A, Int] = lensu[A, Int](
    set = (a, i1) => a.copy(i = i1),
    get = _.i
  )

}
{code}
When using "Way 1: Using Scalaz inside the client code", the application fails immediately - inside the client, before getting to YARN:
{code:none}
java.lang.NoClassDefFoundError: scalaz/Lens$
	at com.example.ALenses$.<init>(DummyMain.scala:38)
	at com.example.ALenses$.<clinit>(DummyMain.scala)
	at com.example.DummyMain$.$anonfun$main$1(DummyMain.scala:17)
	at com.example.DummyMain$.$anonfun$main$1$adapted(DummyMain.scala:16)
	at scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:233)
	at scala.collection.immutable.List.foreach(List.scala:388)
	at scala.collection.TraversableLike.map(TraversableLike.scala:233)
	at scala.collection.TraversableLike.map$(TraversableLike.scala:226)
	at scala.collection.immutable.List.map(List.scala:294)
	at com.example.DummyMain$.main(DummyMain.scala:16)
	at com.example.DummyMain.main(DummyMain.scala)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:349)
	at org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:219)
	at org.apache.flink.client.ClientUtils.executeProgram(ClientUtils.java:114)
       ...
{code}
Remarkable: if using "Way 2: Not using Scalaz inside the client code", the application successfully runs on YARN. The subsequent usage of Scalaz inside Task Managers does not cause a failure.

This behavior is a blocker - the setting yarn.ship-files is actually non-working.

Flink provides no way to customize the classpath of the *flink* client. The function *constructFlinkClassPath()* in *FLINK_HOME/**bin/config.sh* does not provide any means for customization. Instead this function could prepend the resulting classpath with a variable *CUSTOM_CLIENT_CLASSPATH* settable by user.

 

There are 2 alternate ways to circumvent this problem - both are not suitable.

1. Pack everything in a fat-jar. Fat-jars have numerous disadvantages, like huge file size, many multiplied classes across different applications, untraceable dependencies, classpath conflicts.

2. Put application-level jars into *FLINK_HOME/lib*. We have the same Flink installation for many different applications belonging to different teams. We do not want to pollute the Flink installation with user's jars. *FLINK_HOME* is located on a filesystem not writable for ordinary users.

Discussion on StackOverflow: [Flink on yarn: how to add external jars class path in local Client|https://stackoverflow.com/questions/49423861/flink-on-yarn-how-to-add-external-jars-class-path-in-local-client].



--
This message was sent by Atlassian Jira
(v8.3.4#803005)