[jira] [Created] (BEAM-6558) Beam SQL transitive dependencies appear incomplete / broken

["Kenneth Knowles (JIRA)" <ji...@apache.org>]

Kenneth Knowles created BEAM-6558:
-------------------------------------

             Summary: Beam SQL transitive dependencies appear incomplete / broken
                 Key: BEAM-6558
                 URL: https://issues.apache.org/jira/browse/BEAM-6558
             Project: Beam
          Issue Type: Bug
          Components: dsl-sql
    Affects Versions: 2.9.0
            Reporter: Kenneth Knowles
            Assignee: Kenneth Knowles
             Fix For: 2.10.0


Reporting on the dev list:
----
Hi, all,

Newbie here trying to figure out how to use published beam-sdks-java-extensions-sql-2.9.0 in my own project.

I tried to create a gradle project to use BeamSQL sdk libraries. Here is the build.gradle I have:
{code:java}
plugins {
    id 'java'
}

group 'com.mycompany.myproject'
version '1.0-SNAPSHOT'

sourceCompatibility = 1.8

repositories {
    mavenCentral()
}

apply plugin: 'java'

sourceSets.main.java.srcDirs = [
        'src/main/java'
]

dependencies {
    compile 'org.apache.beam:beam-sdks-java-core:2.9.0'
    compile 'org.apache.beam:beam-sdks-java-extensions-sql:2.9.0'
    compile 'com.google.code.findbugs:jsr305:3.0.2'
    runtime 'org.apache.beam:beam-runners-direct-java:2.9.0'
    testCompile group: 'junit', name: 'junit', version: '4.12'
}

// Run basic SQL example
task runBasicExample(type: JavaExec) {
  description = "Run basic SQL example"
  main = "com.mycompany.myproject.streamsql.examples.BeamSQLExample"
  classpath = sourceSets.main.runtimeClasspath
  args = ["--runner=DirectRunner"]
  println classpath.getAsPath()
  println args
}
{code}
The example BeamSQLExample is just copied from [https://github.com/apache/beam/blob/master/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/example/BeamSqlExample.java].

I was able to compile the example in JDK8. However, when I tried to run it, I hit the following exception:
{code:java}
Exception in thread "main" java.util.ServiceConfigurationError: org.apache.beam.sdk.extensions.sql.impl.udf.BeamBuiltinFunctionProvider: Provider org.apache.beam.sdk.extensions.sql.impl.udf.BuiltinStringFunctions could not be instantiated
	at java.util.ServiceLoader.fail(ServiceLoader.java:232)
	at java.util.ServiceLoader.access$100(ServiceLoader.java:185)
	at java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:384)
	at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:404)
	at java.util.ServiceLoader$1.next(ServiceLoader.java:480)
	at org.apache.beam.sdk.extensions.sql.impl.BeamSqlEnv.loadBeamBuiltinFunctions(BeamSqlEnv.java:128)
	at org.apache.beam.sdk.extensions.sql.SqlTransform.expand(SqlTransform.java:94)
	at org.apache.beam.sdk.extensions.sql.SqlTransform.expand(SqlTransform.java:76)
	at org.apache.beam.sdk.Pipeline.applyInternal(Pipeline.java:537)
	at org.apache.beam.sdk.Pipeline.applyTransform(Pipeline.java:471)
	at org.apache.beam.sdk.values.PCollection.apply(PCollection.java:357)
	at com.mycompany.myprject.streamsql.examples.BeamSQLExample.main(BeamSQLExample.java:72)
Caused by: java.lang.NoClassDefFoundError: org/apache/commons/codec/DecoderException
	at java.lang.Class.getDeclaredConstructors0(Native Method)
	at java.lang.Class.privateGetDeclaredConstructors(Class.java:2671)
	at java.lang.Class.getConstructor0(Class.java:3075)
	at java.lang.Class.newInstance(Class.java:412)
	at java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:380)
	... 9 more
Caused by: java.lang.ClassNotFoundException: org.apache.commons.codec.DecoderException
	at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
	... 14 more
{code}
When I traced into the code, it turns out that the failure occurs when the classloader tries to get the default constructor w/o any parameters for BuiltinStringFunctions.class. I double checked my local gradle cache and confirmed that the jar is there:
{code:java}
SJCMAC91THJHD4:beamsql-demo ypan$ ls -l ~/.gradle/caches/modules-2/files-2.1/org.apache.beam/beam-sdks-java-extensions-sql/2.9.0/67e7675519859ff332619c4c6ea5d26a505dbd50/beam-sdks-java-extensions-sql-2.9.0.jar 
-rw-r--r--  1 ypan  192360288  12761025 Jan 29 18:02 /Users/ypan/.gradle/caches/modules-2/files-2.1/org.apache.beam/beam-sdks-java-extensions-sql/2.9.0/67e7675519859ff332619c4c6ea5d26a505dbd50/beam-sdks-java-extensions-sql-2.9.0.jar
{code}
I also have tried to compile the sql sdk libraries in Beam's source repo and just copying over the generated class files over to my runtime classpath. Apparently, that did not work well since there are many shadowed libraries now need explicit declaring dependencies. I tried to search for an example of maven or gradle dependency declaration that users will use to create their own application using the SQL SDKs but was not lucky enough to find one.

 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)