You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@beam.apache.org by "Yi Pan (Data Infrastructure) (JIRA)" <ji...@apache.org> on 2019/01/31 18:15:00 UTC
[jira] [Commented] (BEAM-6558) Beam SQL transitive dependencies
appear incomplete / broken
[ https://issues.apache.org/jira/browse/BEAM-6558?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16757579#comment-16757579 ]
Yi Pan (Data Infrastructure) commented on BEAM-6558:
----------------------------------------------------
Just one more update on this: the same build project worked when I downgrade to 2.8.0. And apparently, there is some change in the packaging method between the two versions:
For 2.9.0, the binary and source are downloaded as:
{noformat}
SJCMAC91THJHD4:2.9.0 ypan$ ls -lR ./
total 0
drwxr-xr-x 3 ypan 192360288 96 Jan 29 18:02 67e7675519859ff332619c4c6ea5d26a505dbd50
drwxr-xr-x 3 ypan 192360288 96 Jan 30 16:31 dc6378b43a42c8de5557c7985fff221e8bdcc5e6
.//67e7675519859ff332619c4c6ea5d26a505dbd50:
total 26272
-rw-r--r-- 1 ypan 192360288 12761025 Jan 29 18:02 beam-sdks-java-extensions-sql-2.9.0.jar
.//dc6378b43a42c8de5557c7985fff221e8bdcc5e6:
total 600
-rw-r--r-- 1 ypan 192360288 303908 Jan 30 16:31 beam-sdks-java-extensions-sql-2.9.0-sources.jar
{noformat}
While in 2.8.0, the downloaded packages are:
{noformat}
SJCMAC91THJHD4:2.8.0 ypan$ ls -lR ./
total 0
drwxr-xr-x 3 ypan 192360288 96 Jan 31 10:03 cc05d197a7a2397b9aef744751dc8174d7031a65
drwxr-xr-x 3 ypan 192360288 96 Jan 31 10:04 d26906da835a3def60b202d921205f2a4847da56
drwxr-xr-x 3 ypan 192360288 96 Jan 31 10:03 f4ed8fb65ae361561b2c09717a9d080fca351238
.//cc05d197a7a2397b9aef744751dc8174d7031a65:
total 25952
-rw-r--r-- 1 ypan 192360288 12742842 Jan 31 10:03 beam-sdks-java-extensions-sql-2.8.0.jar
.//d26906da835a3def60b202d921205f2a4847da56:
total 576
-rw-r--r-- 1 ypan 192360288 290878 Jan 31 10:04 beam-sdks-java-extensions-sql-2.8.0-sources.jar
.//f4ed8fb65ae361561b2c09717a9d080fca351238:
total 32
-rw-r--r-- 1 ypan 192360288 13777 Jan 31 10:03 beam-sdks-java-extensions-sql-2.8.0.pom
{noformat}
Just FYI.
> Beam SQL transitive dependencies appear incomplete / broken
> -----------------------------------------------------------
>
> Key: BEAM-6558
> URL: https://issues.apache.org/jira/browse/BEAM-6558
> Project: Beam
> Issue Type: Bug
> Components: dsl-sql
> Affects Versions: 2.9.0
> Reporter: Kenneth Knowles
> Assignee: Kenneth Knowles
> Priority: Blocker
> Fix For: 2.10.0
>
>
> Reporting on the dev list:
> ----
> Hi, all,
> Newbie here trying to figure out how to use published beam-sdks-java-extensions-sql-2.9.0 in my own project.
> I tried to create a gradle project to use BeamSQL sdk libraries. Here is the build.gradle I have:
> {code:java}
> plugins {
> id 'java'
> }
> group 'com.mycompany.myproject'
> version '1.0-SNAPSHOT'
> sourceCompatibility = 1.8
> repositories {
> mavenCentral()
> }
> apply plugin: 'java'
> sourceSets.main.java.srcDirs = [
> 'src/main/java'
> ]
> dependencies {
> compile 'org.apache.beam:beam-sdks-java-core:2.9.0'
> compile 'org.apache.beam:beam-sdks-java-extensions-sql:2.9.0'
> compile 'com.google.code.findbugs:jsr305:3.0.2'
> runtime 'org.apache.beam:beam-runners-direct-java:2.9.0'
> testCompile group: 'junit', name: 'junit', version: '4.12'
> }
> // Run basic SQL example
> task runBasicExample(type: JavaExec) {
> description = "Run basic SQL example"
> main = "com.mycompany.myproject.streamsql.examples.BeamSQLExample"
> classpath = sourceSets.main.runtimeClasspath
> args = ["--runner=DirectRunner"]
> println classpath.getAsPath()
> println args
> }
> {code}
> The example BeamSQLExample is just copied from [https://github.com/apache/beam/blob/master/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/example/BeamSqlExample.java].
> I was able to compile the example in JDK8. However, when I tried to run it, I hit the following exception:
> {code:java}
> Exception in thread "main" java.util.ServiceConfigurationError: org.apache.beam.sdk.extensions.sql.impl.udf.BeamBuiltinFunctionProvider: Provider org.apache.beam.sdk.extensions.sql.impl.udf.BuiltinStringFunctions could not be instantiated
> at java.util.ServiceLoader.fail(ServiceLoader.java:232)
> at java.util.ServiceLoader.access$100(ServiceLoader.java:185)
> at java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:384)
> at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:404)
> at java.util.ServiceLoader$1.next(ServiceLoader.java:480)
> at org.apache.beam.sdk.extensions.sql.impl.BeamSqlEnv.loadBeamBuiltinFunctions(BeamSqlEnv.java:128)
> at org.apache.beam.sdk.extensions.sql.SqlTransform.expand(SqlTransform.java:94)
> at org.apache.beam.sdk.extensions.sql.SqlTransform.expand(SqlTransform.java:76)
> at org.apache.beam.sdk.Pipeline.applyInternal(Pipeline.java:537)
> at org.apache.beam.sdk.Pipeline.applyTransform(Pipeline.java:471)
> at org.apache.beam.sdk.values.PCollection.apply(PCollection.java:357)
> at com.mycompany.myprject.streamsql.examples.BeamSQLExample.main(BeamSQLExample.java:72)
> Caused by: java.lang.NoClassDefFoundError: org/apache/commons/codec/DecoderException
> at java.lang.Class.getDeclaredConstructors0(Native Method)
> at java.lang.Class.privateGetDeclaredConstructors(Class.java:2671)
> at java.lang.Class.getConstructor0(Class.java:3075)
> at java.lang.Class.newInstance(Class.java:412)
> at java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:380)
> ... 9 more
> Caused by: java.lang.ClassNotFoundException: org.apache.commons.codec.DecoderException
> at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> ... 14 more
> {code}
> When I traced into the code, it turns out that the failure occurs when the classloader tries to get the default constructor w/o any parameters for BuiltinStringFunctions.class. I double checked my local gradle cache and confirmed that the jar is there:
> {code:java}
> SJCMAC91THJHD4:beamsql-demo ypan$ ls -l ~/.gradle/caches/modules-2/files-2.1/org.apache.beam/beam-sdks-java-extensions-sql/2.9.0/67e7675519859ff332619c4c6ea5d26a505dbd50/beam-sdks-java-extensions-sql-2.9.0.jar
> -rw-r--r-- 1 ypan 192360288 12761025 Jan 29 18:02 /Users/ypan/.gradle/caches/modules-2/files-2.1/org.apache.beam/beam-sdks-java-extensions-sql/2.9.0/67e7675519859ff332619c4c6ea5d26a505dbd50/beam-sdks-java-extensions-sql-2.9.0.jar
> {code}
> I also have tried to compile the sql sdk libraries in Beam's source repo and just copying over the generated class files over to my runtime classpath. Apparently, that did not work well since there are many shadowed libraries now need explicit declaring dependencies. I tried to search for an example of maven or gradle dependency declaration that users will use to create their own application using the SQL SDKs but was not lucky enough to find one.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)