You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@parquet.apache.org by gamaken k <ga...@gmail.com> on 2021/11/23 00:08:44 UTC
unable to get rid of NoSuchMethodError with parquet-cli
Hello,
I'm relatively new to parquet and to the community, kindly bear with me if
I'm making simple mistakes.
I cloned parquet-mr and built the repo locally on my ubuntu 20.04, When
running parquet-cli, particularly when trying to read values (cat, head)
from a parquet file, I run into NoSuchMethodError (full stack at the end of
message).
*Here is what I have tried so far to understand this problem:*
1. clean the repo and clone again and retry
2. metadata inspection commands such as `meta`, `schema`, `pages`
`dictionary` - they all work perfectly. check-stats command reports no
corruption or errors,
3. I'm running parquet-cli without Hadoop (according to this ReadMe
<https://github.com/apache/parquet-mr/tree/master/parquet-cli#running-without-hadoop>
)
4. I have tried this on two different machines (ubuntu 18.04 and
ubuntu 20.04) and gotten the same result.
5. The input file is produced by a library that I am writing to serialize
parquet data in .net (i.e, not produced by parquet-mr).
6. parquet-cpp is able to read the file correctly and deserialize the data.
parquet-mr compatibility is important to me, so I'm trying to get this
working.
I could not solve this problem, so, I would really appreciate it if someone
who understands what's wrong here could kindly share insights.
*Error stack:*
~/repos/parquet-mr/parquet-cli$ parquet cat
../../testdata/dictionaryEncodingSample.parquet
WARNING: An illegal reflective access operation has occurred
...<trimmed some more WARNs>...
Exception in thread "main" java.lang.NoSuchMethodError:
'org.apache.avro.Schema
org.apache.parquet.avro.AvroSchemaConverter.convert(org.apache.parquet.schema.MessageType)'
at org.apache.parquet.cli.util.Schemas.fromParquet(Schemas.java:89)
at org.apache.parquet.cli.BaseCommand.getAvroSchema(BaseCommand.java:405)
at org.apache.parquet.cli.commands.CatCommand.run(CatCommand.java:66)
at org.apache.parquet.cli.Main.run(Main.java:157)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
at org.apache.parquet.cli.Main.main(Main.java:187)
*mvn dependency:tree output for parquet-cli*
[INFO] -------------------< org.apache.parquet:parquet-cli
>-------------------
[INFO] Building Apache Parquet Command-line 1.13.0-SNAPSHOT
[INFO] --------------------------------[ jar
]---------------------------------
[INFO]
[INFO] --- maven-dependency-plugin:3.1.1:tree (default-cli) @ parquet-cli
---
[INFO] org.apache.parquet:parquet-cli:jar:1.13.0-SNAPSHOT
[INFO] +- org.apache.parquet:parquet-avro:jar:1.13.0-SNAPSHOT:compile
[INFO] +-
org.apache.parquet:parquet-format-structures:jar:1.13.0-SNAPSHOT:compile
[INFO] +- org.apache.parquet:parquet-common:jar:1.13.0-SNAPSHOT:compile
[INFO] +- org.apache.parquet:parquet-column:jar:1.13.0-SNAPSHOT:compile
[INFO] | +- org.apache.parquet:parquet-encoding:jar:1.13.0-SNAPSHOT:compile
[INFO] | \- org.apache.yetus:audience-annotations:jar:0.13.0:compile
[INFO] +- org.apache.parquet:parquet-hadoop:jar:1.13.0-SNAPSHOT:compile
[INFO] | +- org.xerial.snappy:snappy-java:jar:1.1.8.3:compile
[INFO] | \- commons-pool:commons-pool:jar:1.6:compile
[INFO] +- org.apache.avro:avro:jar:1.10.2:compile
[INFO] | \- org.apache.commons:commons-compress:jar:1.20:compile
[INFO] +- com.github.luben:zstd-jni:jar:1.5.0-1:runtime
[INFO] +- org.slf4j:slf4j-api:jar:1.7.22:compile
[INFO] +- net.sf.opencsv:opencsv:jar:2.3:compile
[INFO] +- org.apache.commons:commons-text:jar:1.8:compile
[INFO] | \- org.apache.commons:commons-lang3:jar:3.9:compile
[INFO] +- org.apache.parquet:parquet-jackson:jar:1.13.0-SNAPSHOT:runtime
[INFO] +- com.fasterxml.jackson.core:jackson-databind:jar:2.12.2:compile
[INFO] +- com.fasterxml.jackson.core:jackson-core:jar:2.12.2:compile
[INFO] +- com.fasterxml.jackson.core:jackson-annotations:jar:2.12.2:compile
[INFO] +- com.beust:jcommander:jar:1.72:compile
[INFO] +- org.slf4j:slf4j-log4j12:jar:1.7.22:provided
[INFO] +- com.google.guava:guava:jar:27.0.1-jre:provided
[INFO] | +- com.google.guava:failureaccess:jar:1.0.1:provided
[INFO] | +-
com.google.guava:listenablefuture:jar:9999.0-empty-to-avoid-conflict-with-guava:provided
[INFO] | +- org.checkerframework:checker-qual:jar:2.5.2:provided
[INFO] | +-
com.google.errorprone:error_prone_annotations:jar:2.2.0:provided
[INFO] | +- com.google.j2objc:j2objc-annotations:jar:1.1:provided
[INFO] | \- org.codehaus.mojo:animal-sniffer-annotations:jar:1.17:provided
[INFO] +- org.apache.hadoop:hadoop-client:jar:2.10.1:provided
[INFO] | +- org.apache.hadoop:hadoop-hdfs-client:jar:2.10.1:provided
[INFO] | | \- com.squareup.okhttp:okhttp:jar:2.7.5:provided
[INFO] | | \- com.squareup.okio:okio:jar:1.6.0:provided
[INFO] | +-
org.apache.hadoop:hadoop-mapreduce-client-app:jar:2.10.1:provided
[INFO] | | +-
org.apache.hadoop:hadoop-mapreduce-client-common:jar:2.10.1:provided
[INFO] | | \-
org.apache.hadoop:hadoop-mapreduce-client-shuffle:jar:2.10.1:provided
[INFO] | | +-
org.apache.hadoop:hadoop-yarn-server-common:jar:2.10.1:provided
[INFO] | | | +-
org.apache.hadoop:hadoop-yarn-registry:jar:2.10.1:provided
[INFO] | | | +-
org.apache.geronimo.specs:geronimo-jcache_1.0_spec:jar:1.0-alpha-1:provided
[INFO] | | | +- org.ehcache:ehcache:jar:3.3.1:provided
[INFO] | | | +- com.zaxxer:HikariCP-java7:jar:2.4.12:provided
[INFO] | | | \-
com.microsoft.sqlserver:mssql-jdbc:jar:6.2.1.jre7:provided
[INFO] | | \- org.fusesource.leveldbjni:leveldbjni-all:jar:1.8:provided
[INFO] | +- org.apache.hadoop:hadoop-yarn-api:jar:2.10.1:provided
[INFO] | | \- javax.xml.bind:jaxb-api:jar:2.2.2:provided
[INFO] | | +- javax.xml.stream:stax-api:jar:1.0-2:provided
[INFO] | | \- javax.activation:activation:jar:1.1:provided
[INFO] | +-
org.apache.hadoop:hadoop-mapreduce-client-core:jar:2.10.1:provided
[INFO] | | +- org.apache.hadoop:hadoop-yarn-client:jar:2.10.1:provided
[INFO] | | \- org.apache.hadoop:hadoop-yarn-common:jar:2.10.1:provided
[INFO] | | \- com.sun.jersey:jersey-client:jar:1.9:provided
[INFO] | +-
org.apache.hadoop:hadoop-mapreduce-client-jobclient:jar:2.10.1:provided
[INFO] | \- org.apache.hadoop:hadoop-annotations:jar:2.10.1:provided
[INFO] +- org.apache.hadoop:hadoop-common:jar:2.10.1:provided
[INFO] | +- commons-cli:commons-cli:jar:1.2:provided
[INFO] | +- org.apache.commons:commons-math3:jar:3.1.1:provided
[INFO] | +- xmlenc:xmlenc:jar:0.52:provided
[INFO] | +- org.apache.httpcomponents:httpclient:jar:4.5.2:provided
[INFO] | | \- org.apache.httpcomponents:httpcore:jar:4.4.4:provided
[INFO] | +- commons-codec:commons-codec:jar:1.4:provided
[INFO] | +- commons-net:commons-net:jar:3.1:provided
[INFO] | +- commons-collections:commons-collections:jar:3.2.2:provided
[INFO] | +- javax.servlet:servlet-api:jar:2.5:provided
[INFO] | +- org.mortbay.jetty:jetty:jar:6.1.26:provided
[INFO] | +- org.mortbay.jetty:jetty-util:jar:6.1.26:provided
[INFO] | +- org.mortbay.jetty:jetty-sslengine:jar:6.1.26:provided
[INFO] | +- javax.servlet.jsp:jsp-api:jar:2.1:provided
[INFO] | +- com.sun.jersey:jersey-core:jar:1.9:provided
[INFO] | +- com.sun.jersey:jersey-json:jar:1.9:provided
[INFO] | | +- org.codehaus.jettison:jettison:jar:1.1:provided
[INFO] | | +- com.sun.xml.bind:jaxb-impl:jar:2.2.3-1:provided
[INFO] | | +- org.codehaus.jackson:jackson-jaxrs:jar:1.8.3:provided
[INFO] | | \- org.codehaus.jackson:jackson-xc:jar:1.8.3:provided
[INFO] | +- com.sun.jersey:jersey-server:jar:1.9:provided
[INFO] | | \- asm:asm:jar:3.1:provided
[INFO] | +- net.java.dev.jets3t:jets3t:jar:0.9.0:provided
[INFO] | | \- com.jamesmurty.utils:java-xmlbuilder:jar:0.4:provided
[INFO] | +- commons-lang:commons-lang:jar:2.6:provided
[INFO] | +- commons-configuration:commons-configuration:jar:1.6:provided
[INFO] | +- commons-digester:commons-digester:jar:1.8:provided
[INFO] | +- commons-beanutils:commons-beanutils:jar:1.9.4:provided
[INFO] | +- org.codehaus.jackson:jackson-core-asl:jar:1.9.13:provided
[INFO] | +- org.codehaus.jackson:jackson-mapper-asl:jar:1.9.13:provided
[INFO] | +- com.google.protobuf:protobuf-java:jar:2.5.0:provided
[INFO] | +- com.google.code.gson:gson:jar:2.2.4:provided
[INFO] | +- org.apache.hadoop:hadoop-auth:jar:2.10.1:provided
[INFO] | | +- com.nimbusds:nimbus-jose-jwt:jar:7.9:provided
[INFO] | | | +-
com.github.stephenc.jcip:jcip-annotations:jar:1.0-1:provided
[INFO] | | | \- net.minidev:json-smart:jar:2.3:provided (version
selected from constraint [1.3.1,2.3])
[INFO] | | | \- net.minidev:accessors-smart:jar:1.2:provided
[INFO] | | | \- org.ow2.asm:asm:jar:5.0.4:provided
[INFO] | | +-
org.apache.directory.server:apacheds-kerberos-codec:jar:2.0.0-M15:provided
[INFO] | | | +-
org.apache.directory.server:apacheds-i18n:jar:2.0.0-M15:provided
[INFO] | | | +-
org.apache.directory.api:api-asn1-api:jar:1.0.0-M20:provided
[INFO] | | | \- org.apache.directory.api:api-util:jar:1.0.0-M20:provided
[INFO] | | \- org.apache.curator:curator-framework:jar:2.13.0:provided
[INFO] | +- com.jcraft:jsch:jar:0.1.55:provided
[INFO] | +- org.apache.curator:curator-client:jar:2.13.0:provided
[INFO] | +- org.apache.curator:curator-recipes:jar:2.13.0:provided
[INFO] | +- org.apache.htrace:htrace-core4:jar:4.1.0-incubating:provided
[INFO] | +- org.apache.zookeeper:zookeeper:jar:3.4.14:provided
[INFO] | | +- com.github.spotbugs:spotbugs-annotations:jar:3.1.9:provided
[INFO] | | \- io.netty:netty:jar:3.10.6.Final:provided
[INFO] | +- org.codehaus.woodstox:stax2-api:jar:3.1.4:provided
[INFO] | \- com.fasterxml.woodstox:woodstox-core:jar:5.0.3:provided
[INFO] +- com.google.code.findbugs:jsr305:jar:3.0.2:provided
[INFO] +- log4j:log4j:jar:1.2.17:provided
[INFO] +- commons-io:commons-io:jar:2.4:provided
[INFO] +- commons-logging:commons-logging:jar:1.1.3:provided
[INFO] +- junit:junit:jar:4.13.1:test
[INFO] | \- org.hamcrest:hamcrest-core:jar:1.3:test
[INFO] \- org.easymock:easymock:jar:3.4:test
[INFO] \- org.objenesis:objenesis:jar:2.2:test
[INFO]
------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO]
------------------------------------------------------------------------
Thank you
Re: unable to get rid of NoSuchMethodError with parquet-cli
Posted by gamaken k <ga...@gmail.com>.
Thanks for verifying, Gabor.
I have filed a report here:
https://issues.apache.org/jira/browse/PARQUET-2104
p.s: I assumed `master` can't possibly be broken and spent days on this
trying to find my mistake. Glad someone sees the same thing and its not
just me :D
On Tue, Nov 23, 2021 at 6:24 AM Gabor Szadovszky <ga...@apache.org> wrote:
> Hey,
>
> I can reproduce the same on master. It seems the same issue happens with
> older versions as well. I don't know how we did not find it yet. (Or I am
> making the same mistake as you :) ).
> Could you please create a jira about it and continue the discussion there?
>
> Thanks a lot,
> Gabor
>
> On Tue, Nov 23, 2021 at 1:08 AM gamaken k <ga...@gmail.com> wrote:
>
> > Hello,
> >
> > I'm relatively new to parquet and to the community, kindly bear with me
> if
> > I'm making simple mistakes.
> >
> > I cloned parquet-mr and built the repo locally on my ubuntu 20.04, When
> > running parquet-cli, particularly when trying to read values (cat, head)
> > from a parquet file, I run into NoSuchMethodError (full stack at the end
> of
> > message).
> >
> > *Here is what I have tried so far to understand this problem:*
> > 1. clean the repo and clone again and retry
> > 2. metadata inspection commands such as `meta`, `schema`, `pages`
> > `dictionary` - they all work perfectly. check-stats command reports no
> > corruption or errors,
> > 3. I'm running parquet-cli without Hadoop (according to this ReadMe
> > <
> >
> https://github.com/apache/parquet-mr/tree/master/parquet-cli#running-without-hadoop
> > >
> > )
> > 4. I have tried this on two different machines (ubuntu 18.04 and
> > ubuntu 20.04) and gotten the same result.
> > 5. The input file is produced by a library that I am writing to serialize
> > parquet data in .net (i.e, not produced by parquet-mr).
> > 6. parquet-cpp is able to read the file correctly and deserialize the
> data.
> > parquet-mr compatibility is important to me, so I'm trying to get this
> > working.
> >
> > I could not solve this problem, so, I would really appreciate it if
> someone
> > who understands what's wrong here could kindly share insights.
> >
> > *Error stack:*
> > ~/repos/parquet-mr/parquet-cli$ parquet cat
> > ../../testdata/dictionaryEncodingSample.parquet
> > WARNING: An illegal reflective access operation has occurred
> > ...<trimmed some more WARNs>...
> > Exception in thread "main" java.lang.NoSuchMethodError:
> > 'org.apache.avro.Schema
> >
> >
> org.apache.parquet.avro.AvroSchemaConverter.convert(org.apache.parquet.schema.MessageType)'
> > at org.apache.parquet.cli.util.Schemas.fromParquet(Schemas.java:89)
> > at org.apache.parquet.cli.BaseCommand.getAvroSchema(BaseCommand.java:405)
> > at org.apache.parquet.cli.commands.CatCommand.run(CatCommand.java:66)
> > at org.apache.parquet.cli.Main.run(Main.java:157)
> > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
> > at org.apache.parquet.cli.Main.main(Main.java:187)
> >
> > *mvn dependency:tree output for parquet-cli*
> > [INFO] -------------------< org.apache.parquet:parquet-cli
> > >-------------------
> > [INFO] Building Apache Parquet Command-line 1.13.0-SNAPSHOT
> > [INFO] --------------------------------[ jar
> > ]---------------------------------
> > [INFO]
> > [INFO] --- maven-dependency-plugin:3.1.1:tree (default-cli) @ parquet-cli
> > ---
> > [INFO] org.apache.parquet:parquet-cli:jar:1.13.0-SNAPSHOT
> > [INFO] +- org.apache.parquet:parquet-avro:jar:1.13.0-SNAPSHOT:compile
> > [INFO] +-
> > org.apache.parquet:parquet-format-structures:jar:1.13.0-SNAPSHOT:compile
> > [INFO] +- org.apache.parquet:parquet-common:jar:1.13.0-SNAPSHOT:compile
> > [INFO] +- org.apache.parquet:parquet-column:jar:1.13.0-SNAPSHOT:compile
> > [INFO] | +-
> > org.apache.parquet:parquet-encoding:jar:1.13.0-SNAPSHOT:compile
> > [INFO] | \- org.apache.yetus:audience-annotations:jar:0.13.0:compile
> > [INFO] +- org.apache.parquet:parquet-hadoop:jar:1.13.0-SNAPSHOT:compile
> > [INFO] | +- org.xerial.snappy:snappy-java:jar:1.1.8.3:compile
> > [INFO] | \- commons-pool:commons-pool:jar:1.6:compile
> > [INFO] +- org.apache.avro:avro:jar:1.10.2:compile
> > [INFO] | \- org.apache.commons:commons-compress:jar:1.20:compile
> > [INFO] +- com.github.luben:zstd-jni:jar:1.5.0-1:runtime
> > [INFO] +- org.slf4j:slf4j-api:jar:1.7.22:compile
> > [INFO] +- net.sf.opencsv:opencsv:jar:2.3:compile
> > [INFO] +- org.apache.commons:commons-text:jar:1.8:compile
> > [INFO] | \- org.apache.commons:commons-lang3:jar:3.9:compile
> > [INFO] +- org.apache.parquet:parquet-jackson:jar:1.13.0-SNAPSHOT:runtime
> > [INFO] +- com.fasterxml.jackson.core:jackson-databind:jar:2.12.2:compile
> > [INFO] +- com.fasterxml.jackson.core:jackson-core:jar:2.12.2:compile
> > [INFO] +-
> com.fasterxml.jackson.core:jackson-annotations:jar:2.12.2:compile
> > [INFO] +- com.beust:jcommander:jar:1.72:compile
> > [INFO] +- org.slf4j:slf4j-log4j12:jar:1.7.22:provided
> > [INFO] +- com.google.guava:guava:jar:27.0.1-jre:provided
> > [INFO] | +- com.google.guava:failureaccess:jar:1.0.1:provided
> > [INFO] | +-
> >
> >
> com.google.guava:listenablefuture:jar:9999.0-empty-to-avoid-conflict-with-guava:provided
> > [INFO] | +- org.checkerframework:checker-qual:jar:2.5.2:provided
> > [INFO] | +-
> > com.google.errorprone:error_prone_annotations:jar:2.2.0:provided
> > [INFO] | +- com.google.j2objc:j2objc-annotations:jar:1.1:provided
> > [INFO] | \-
> org.codehaus.mojo:animal-sniffer-annotations:jar:1.17:provided
> > [INFO] +- org.apache.hadoop:hadoop-client:jar:2.10.1:provided
> > [INFO] | +- org.apache.hadoop:hadoop-hdfs-client:jar:2.10.1:provided
> > [INFO] | | \- com.squareup.okhttp:okhttp:jar:2.7.5:provided
> > [INFO] | | \- com.squareup.okio:okio:jar:1.6.0:provided
> > [INFO] | +-
> > org.apache.hadoop:hadoop-mapreduce-client-app:jar:2.10.1:provided
> > [INFO] | | +-
> > org.apache.hadoop:hadoop-mapreduce-client-common:jar:2.10.1:provided
> > [INFO] | | \-
> > org.apache.hadoop:hadoop-mapreduce-client-shuffle:jar:2.10.1:provided
> > [INFO] | | +-
> > org.apache.hadoop:hadoop-yarn-server-common:jar:2.10.1:provided
> > [INFO] | | | +-
> > org.apache.hadoop:hadoop-yarn-registry:jar:2.10.1:provided
> > [INFO] | | | +-
> >
> org.apache.geronimo.specs:geronimo-jcache_1.0_spec:jar:1.0-alpha-1:provided
> > [INFO] | | | +- org.ehcache:ehcache:jar:3.3.1:provided
> > [INFO] | | | +- com.zaxxer:HikariCP-java7:jar:2.4.12:provided
> > [INFO] | | | \-
> > com.microsoft.sqlserver:mssql-jdbc:jar:6.2.1.jre7:provided
> > [INFO] | | \-
> > org.fusesource.leveldbjni:leveldbjni-all:jar:1.8:provided
> > [INFO] | +- org.apache.hadoop:hadoop-yarn-api:jar:2.10.1:provided
> > [INFO] | | \- javax.xml.bind:jaxb-api:jar:2.2.2:provided
> > [INFO] | | +- javax.xml.stream:stax-api:jar:1.0-2:provided
> > [INFO] | | \- javax.activation:activation:jar:1.1:provided
> > [INFO] | +-
> > org.apache.hadoop:hadoop-mapreduce-client-core:jar:2.10.1:provided
> > [INFO] | | +- org.apache.hadoop:hadoop-yarn-client:jar:2.10.1:provided
> > [INFO] | | \- org.apache.hadoop:hadoop-yarn-common:jar:2.10.1:provided
> > [INFO] | | \- com.sun.jersey:jersey-client:jar:1.9:provided
> > [INFO] | +-
> > org.apache.hadoop:hadoop-mapreduce-client-jobclient:jar:2.10.1:provided
> > [INFO] | \- org.apache.hadoop:hadoop-annotations:jar:2.10.1:provided
> > [INFO] +- org.apache.hadoop:hadoop-common:jar:2.10.1:provided
> > [INFO] | +- commons-cli:commons-cli:jar:1.2:provided
> > [INFO] | +- org.apache.commons:commons-math3:jar:3.1.1:provided
> > [INFO] | +- xmlenc:xmlenc:jar:0.52:provided
> > [INFO] | +- org.apache.httpcomponents:httpclient:jar:4.5.2:provided
> > [INFO] | | \- org.apache.httpcomponents:httpcore:jar:4.4.4:provided
> > [INFO] | +- commons-codec:commons-codec:jar:1.4:provided
> > [INFO] | +- commons-net:commons-net:jar:3.1:provided
> > [INFO] | +- commons-collections:commons-collections:jar:3.2.2:provided
> > [INFO] | +- javax.servlet:servlet-api:jar:2.5:provided
> > [INFO] | +- org.mortbay.jetty:jetty:jar:6.1.26:provided
> > [INFO] | +- org.mortbay.jetty:jetty-util:jar:6.1.26:provided
> > [INFO] | +- org.mortbay.jetty:jetty-sslengine:jar:6.1.26:provided
> > [INFO] | +- javax.servlet.jsp:jsp-api:jar:2.1:provided
> > [INFO] | +- com.sun.jersey:jersey-core:jar:1.9:provided
> > [INFO] | +- com.sun.jersey:jersey-json:jar:1.9:provided
> > [INFO] | | +- org.codehaus.jettison:jettison:jar:1.1:provided
> > [INFO] | | +- com.sun.xml.bind:jaxb-impl:jar:2.2.3-1:provided
> > [INFO] | | +- org.codehaus.jackson:jackson-jaxrs:jar:1.8.3:provided
> > [INFO] | | \- org.codehaus.jackson:jackson-xc:jar:1.8.3:provided
> > [INFO] | +- com.sun.jersey:jersey-server:jar:1.9:provided
> > [INFO] | | \- asm:asm:jar:3.1:provided
> > [INFO] | +- net.java.dev.jets3t:jets3t:jar:0.9.0:provided
> > [INFO] | | \- com.jamesmurty.utils:java-xmlbuilder:jar:0.4:provided
> > [INFO] | +- commons-lang:commons-lang:jar:2.6:provided
> > [INFO] | +- commons-configuration:commons-configuration:jar:1.6:provided
> > [INFO] | +- commons-digester:commons-digester:jar:1.8:provided
> > [INFO] | +- commons-beanutils:commons-beanutils:jar:1.9.4:provided
> > [INFO] | +- org.codehaus.jackson:jackson-core-asl:jar:1.9.13:provided
> > [INFO] | +- org.codehaus.jackson:jackson-mapper-asl:jar:1.9.13:provided
> > [INFO] | +- com.google.protobuf:protobuf-java:jar:2.5.0:provided
> > [INFO] | +- com.google.code.gson:gson:jar:2.2.4:provided
> > [INFO] | +- org.apache.hadoop:hadoop-auth:jar:2.10.1:provided
> > [INFO] | | +- com.nimbusds:nimbus-jose-jwt:jar:7.9:provided
> > [INFO] | | | +-
> > com.github.stephenc.jcip:jcip-annotations:jar:1.0-1:provided
> > [INFO] | | | \- net.minidev:json-smart:jar:2.3:provided (version
> > selected from constraint [1.3.1,2.3])
> > [INFO] | | | \- net.minidev:accessors-smart:jar:1.2:provided
> > [INFO] | | | \- org.ow2.asm:asm:jar:5.0.4:provided
> > [INFO] | | +-
> >
> org.apache.directory.server:apacheds-kerberos-codec:jar:2.0.0-M15:provided
> > [INFO] | | | +-
> > org.apache.directory.server:apacheds-i18n:jar:2.0.0-M15:provided
> > [INFO] | | | +-
> > org.apache.directory.api:api-asn1-api:jar:1.0.0-M20:provided
> > [INFO] | | | \-
> org.apache.directory.api:api-util:jar:1.0.0-M20:provided
> > [INFO] | | \- org.apache.curator:curator-framework:jar:2.13.0:provided
> > [INFO] | +- com.jcraft:jsch:jar:0.1.55:provided
> > [INFO] | +- org.apache.curator:curator-client:jar:2.13.0:provided
> > [INFO] | +- org.apache.curator:curator-recipes:jar:2.13.0:provided
> > [INFO] | +- org.apache.htrace:htrace-core4:jar:4.1.0-incubating:provided
> > [INFO] | +- org.apache.zookeeper:zookeeper:jar:3.4.14:provided
> > [INFO] | | +-
> com.github.spotbugs:spotbugs-annotations:jar:3.1.9:provided
> > [INFO] | | \- io.netty:netty:jar:3.10.6.Final:provided
> > [INFO] | +- org.codehaus.woodstox:stax2-api:jar:3.1.4:provided
> > [INFO] | \- com.fasterxml.woodstox:woodstox-core:jar:5.0.3:provided
> > [INFO] +- com.google.code.findbugs:jsr305:jar:3.0.2:provided
> > [INFO] +- log4j:log4j:jar:1.2.17:provided
> > [INFO] +- commons-io:commons-io:jar:2.4:provided
> > [INFO] +- commons-logging:commons-logging:jar:1.1.3:provided
> > [INFO] +- junit:junit:jar:4.13.1:test
> > [INFO] | \- org.hamcrest:hamcrest-core:jar:1.3:test
> > [INFO] \- org.easymock:easymock:jar:3.4:test
> > [INFO] \- org.objenesis:objenesis:jar:2.2:test
> > [INFO]
> > ------------------------------------------------------------------------
> > [INFO] BUILD SUCCESS
> > [INFO]
> > ------------------------------------------------------------------------
> >
> > Thank you
> >
>
Re: unable to get rid of NoSuchMethodError with parquet-cli
Posted by Gabor Szadovszky <ga...@apache.org>.
Hey,
I can reproduce the same on master. It seems the same issue happens with
older versions as well. I don't know how we did not find it yet. (Or I am
making the same mistake as you :) ).
Could you please create a jira about it and continue the discussion there?
Thanks a lot,
Gabor
On Tue, Nov 23, 2021 at 1:08 AM gamaken k <ga...@gmail.com> wrote:
> Hello,
>
> I'm relatively new to parquet and to the community, kindly bear with me if
> I'm making simple mistakes.
>
> I cloned parquet-mr and built the repo locally on my ubuntu 20.04, When
> running parquet-cli, particularly when trying to read values (cat, head)
> from a parquet file, I run into NoSuchMethodError (full stack at the end of
> message).
>
> *Here is what I have tried so far to understand this problem:*
> 1. clean the repo and clone again and retry
> 2. metadata inspection commands such as `meta`, `schema`, `pages`
> `dictionary` - they all work perfectly. check-stats command reports no
> corruption or errors,
> 3. I'm running parquet-cli without Hadoop (according to this ReadMe
> <
> https://github.com/apache/parquet-mr/tree/master/parquet-cli#running-without-hadoop
> >
> )
> 4. I have tried this on two different machines (ubuntu 18.04 and
> ubuntu 20.04) and gotten the same result.
> 5. The input file is produced by a library that I am writing to serialize
> parquet data in .net (i.e, not produced by parquet-mr).
> 6. parquet-cpp is able to read the file correctly and deserialize the data.
> parquet-mr compatibility is important to me, so I'm trying to get this
> working.
>
> I could not solve this problem, so, I would really appreciate it if someone
> who understands what's wrong here could kindly share insights.
>
> *Error stack:*
> ~/repos/parquet-mr/parquet-cli$ parquet cat
> ../../testdata/dictionaryEncodingSample.parquet
> WARNING: An illegal reflective access operation has occurred
> ...<trimmed some more WARNs>...
> Exception in thread "main" java.lang.NoSuchMethodError:
> 'org.apache.avro.Schema
>
> org.apache.parquet.avro.AvroSchemaConverter.convert(org.apache.parquet.schema.MessageType)'
> at org.apache.parquet.cli.util.Schemas.fromParquet(Schemas.java:89)
> at org.apache.parquet.cli.BaseCommand.getAvroSchema(BaseCommand.java:405)
> at org.apache.parquet.cli.commands.CatCommand.run(CatCommand.java:66)
> at org.apache.parquet.cli.Main.run(Main.java:157)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
> at org.apache.parquet.cli.Main.main(Main.java:187)
>
> *mvn dependency:tree output for parquet-cli*
> [INFO] -------------------< org.apache.parquet:parquet-cli
> >-------------------
> [INFO] Building Apache Parquet Command-line 1.13.0-SNAPSHOT
> [INFO] --------------------------------[ jar
> ]---------------------------------
> [INFO]
> [INFO] --- maven-dependency-plugin:3.1.1:tree (default-cli) @ parquet-cli
> ---
> [INFO] org.apache.parquet:parquet-cli:jar:1.13.0-SNAPSHOT
> [INFO] +- org.apache.parquet:parquet-avro:jar:1.13.0-SNAPSHOT:compile
> [INFO] +-
> org.apache.parquet:parquet-format-structures:jar:1.13.0-SNAPSHOT:compile
> [INFO] +- org.apache.parquet:parquet-common:jar:1.13.0-SNAPSHOT:compile
> [INFO] +- org.apache.parquet:parquet-column:jar:1.13.0-SNAPSHOT:compile
> [INFO] | +-
> org.apache.parquet:parquet-encoding:jar:1.13.0-SNAPSHOT:compile
> [INFO] | \- org.apache.yetus:audience-annotations:jar:0.13.0:compile
> [INFO] +- org.apache.parquet:parquet-hadoop:jar:1.13.0-SNAPSHOT:compile
> [INFO] | +- org.xerial.snappy:snappy-java:jar:1.1.8.3:compile
> [INFO] | \- commons-pool:commons-pool:jar:1.6:compile
> [INFO] +- org.apache.avro:avro:jar:1.10.2:compile
> [INFO] | \- org.apache.commons:commons-compress:jar:1.20:compile
> [INFO] +- com.github.luben:zstd-jni:jar:1.5.0-1:runtime
> [INFO] +- org.slf4j:slf4j-api:jar:1.7.22:compile
> [INFO] +- net.sf.opencsv:opencsv:jar:2.3:compile
> [INFO] +- org.apache.commons:commons-text:jar:1.8:compile
> [INFO] | \- org.apache.commons:commons-lang3:jar:3.9:compile
> [INFO] +- org.apache.parquet:parquet-jackson:jar:1.13.0-SNAPSHOT:runtime
> [INFO] +- com.fasterxml.jackson.core:jackson-databind:jar:2.12.2:compile
> [INFO] +- com.fasterxml.jackson.core:jackson-core:jar:2.12.2:compile
> [INFO] +- com.fasterxml.jackson.core:jackson-annotations:jar:2.12.2:compile
> [INFO] +- com.beust:jcommander:jar:1.72:compile
> [INFO] +- org.slf4j:slf4j-log4j12:jar:1.7.22:provided
> [INFO] +- com.google.guava:guava:jar:27.0.1-jre:provided
> [INFO] | +- com.google.guava:failureaccess:jar:1.0.1:provided
> [INFO] | +-
>
> com.google.guava:listenablefuture:jar:9999.0-empty-to-avoid-conflict-with-guava:provided
> [INFO] | +- org.checkerframework:checker-qual:jar:2.5.2:provided
> [INFO] | +-
> com.google.errorprone:error_prone_annotations:jar:2.2.0:provided
> [INFO] | +- com.google.j2objc:j2objc-annotations:jar:1.1:provided
> [INFO] | \- org.codehaus.mojo:animal-sniffer-annotations:jar:1.17:provided
> [INFO] +- org.apache.hadoop:hadoop-client:jar:2.10.1:provided
> [INFO] | +- org.apache.hadoop:hadoop-hdfs-client:jar:2.10.1:provided
> [INFO] | | \- com.squareup.okhttp:okhttp:jar:2.7.5:provided
> [INFO] | | \- com.squareup.okio:okio:jar:1.6.0:provided
> [INFO] | +-
> org.apache.hadoop:hadoop-mapreduce-client-app:jar:2.10.1:provided
> [INFO] | | +-
> org.apache.hadoop:hadoop-mapreduce-client-common:jar:2.10.1:provided
> [INFO] | | \-
> org.apache.hadoop:hadoop-mapreduce-client-shuffle:jar:2.10.1:provided
> [INFO] | | +-
> org.apache.hadoop:hadoop-yarn-server-common:jar:2.10.1:provided
> [INFO] | | | +-
> org.apache.hadoop:hadoop-yarn-registry:jar:2.10.1:provided
> [INFO] | | | +-
> org.apache.geronimo.specs:geronimo-jcache_1.0_spec:jar:1.0-alpha-1:provided
> [INFO] | | | +- org.ehcache:ehcache:jar:3.3.1:provided
> [INFO] | | | +- com.zaxxer:HikariCP-java7:jar:2.4.12:provided
> [INFO] | | | \-
> com.microsoft.sqlserver:mssql-jdbc:jar:6.2.1.jre7:provided
> [INFO] | | \-
> org.fusesource.leveldbjni:leveldbjni-all:jar:1.8:provided
> [INFO] | +- org.apache.hadoop:hadoop-yarn-api:jar:2.10.1:provided
> [INFO] | | \- javax.xml.bind:jaxb-api:jar:2.2.2:provided
> [INFO] | | +- javax.xml.stream:stax-api:jar:1.0-2:provided
> [INFO] | | \- javax.activation:activation:jar:1.1:provided
> [INFO] | +-
> org.apache.hadoop:hadoop-mapreduce-client-core:jar:2.10.1:provided
> [INFO] | | +- org.apache.hadoop:hadoop-yarn-client:jar:2.10.1:provided
> [INFO] | | \- org.apache.hadoop:hadoop-yarn-common:jar:2.10.1:provided
> [INFO] | | \- com.sun.jersey:jersey-client:jar:1.9:provided
> [INFO] | +-
> org.apache.hadoop:hadoop-mapreduce-client-jobclient:jar:2.10.1:provided
> [INFO] | \- org.apache.hadoop:hadoop-annotations:jar:2.10.1:provided
> [INFO] +- org.apache.hadoop:hadoop-common:jar:2.10.1:provided
> [INFO] | +- commons-cli:commons-cli:jar:1.2:provided
> [INFO] | +- org.apache.commons:commons-math3:jar:3.1.1:provided
> [INFO] | +- xmlenc:xmlenc:jar:0.52:provided
> [INFO] | +- org.apache.httpcomponents:httpclient:jar:4.5.2:provided
> [INFO] | | \- org.apache.httpcomponents:httpcore:jar:4.4.4:provided
> [INFO] | +- commons-codec:commons-codec:jar:1.4:provided
> [INFO] | +- commons-net:commons-net:jar:3.1:provided
> [INFO] | +- commons-collections:commons-collections:jar:3.2.2:provided
> [INFO] | +- javax.servlet:servlet-api:jar:2.5:provided
> [INFO] | +- org.mortbay.jetty:jetty:jar:6.1.26:provided
> [INFO] | +- org.mortbay.jetty:jetty-util:jar:6.1.26:provided
> [INFO] | +- org.mortbay.jetty:jetty-sslengine:jar:6.1.26:provided
> [INFO] | +- javax.servlet.jsp:jsp-api:jar:2.1:provided
> [INFO] | +- com.sun.jersey:jersey-core:jar:1.9:provided
> [INFO] | +- com.sun.jersey:jersey-json:jar:1.9:provided
> [INFO] | | +- org.codehaus.jettison:jettison:jar:1.1:provided
> [INFO] | | +- com.sun.xml.bind:jaxb-impl:jar:2.2.3-1:provided
> [INFO] | | +- org.codehaus.jackson:jackson-jaxrs:jar:1.8.3:provided
> [INFO] | | \- org.codehaus.jackson:jackson-xc:jar:1.8.3:provided
> [INFO] | +- com.sun.jersey:jersey-server:jar:1.9:provided
> [INFO] | | \- asm:asm:jar:3.1:provided
> [INFO] | +- net.java.dev.jets3t:jets3t:jar:0.9.0:provided
> [INFO] | | \- com.jamesmurty.utils:java-xmlbuilder:jar:0.4:provided
> [INFO] | +- commons-lang:commons-lang:jar:2.6:provided
> [INFO] | +- commons-configuration:commons-configuration:jar:1.6:provided
> [INFO] | +- commons-digester:commons-digester:jar:1.8:provided
> [INFO] | +- commons-beanutils:commons-beanutils:jar:1.9.4:provided
> [INFO] | +- org.codehaus.jackson:jackson-core-asl:jar:1.9.13:provided
> [INFO] | +- org.codehaus.jackson:jackson-mapper-asl:jar:1.9.13:provided
> [INFO] | +- com.google.protobuf:protobuf-java:jar:2.5.0:provided
> [INFO] | +- com.google.code.gson:gson:jar:2.2.4:provided
> [INFO] | +- org.apache.hadoop:hadoop-auth:jar:2.10.1:provided
> [INFO] | | +- com.nimbusds:nimbus-jose-jwt:jar:7.9:provided
> [INFO] | | | +-
> com.github.stephenc.jcip:jcip-annotations:jar:1.0-1:provided
> [INFO] | | | \- net.minidev:json-smart:jar:2.3:provided (version
> selected from constraint [1.3.1,2.3])
> [INFO] | | | \- net.minidev:accessors-smart:jar:1.2:provided
> [INFO] | | | \- org.ow2.asm:asm:jar:5.0.4:provided
> [INFO] | | +-
> org.apache.directory.server:apacheds-kerberos-codec:jar:2.0.0-M15:provided
> [INFO] | | | +-
> org.apache.directory.server:apacheds-i18n:jar:2.0.0-M15:provided
> [INFO] | | | +-
> org.apache.directory.api:api-asn1-api:jar:1.0.0-M20:provided
> [INFO] | | | \- org.apache.directory.api:api-util:jar:1.0.0-M20:provided
> [INFO] | | \- org.apache.curator:curator-framework:jar:2.13.0:provided
> [INFO] | +- com.jcraft:jsch:jar:0.1.55:provided
> [INFO] | +- org.apache.curator:curator-client:jar:2.13.0:provided
> [INFO] | +- org.apache.curator:curator-recipes:jar:2.13.0:provided
> [INFO] | +- org.apache.htrace:htrace-core4:jar:4.1.0-incubating:provided
> [INFO] | +- org.apache.zookeeper:zookeeper:jar:3.4.14:provided
> [INFO] | | +- com.github.spotbugs:spotbugs-annotations:jar:3.1.9:provided
> [INFO] | | \- io.netty:netty:jar:3.10.6.Final:provided
> [INFO] | +- org.codehaus.woodstox:stax2-api:jar:3.1.4:provided
> [INFO] | \- com.fasterxml.woodstox:woodstox-core:jar:5.0.3:provided
> [INFO] +- com.google.code.findbugs:jsr305:jar:3.0.2:provided
> [INFO] +- log4j:log4j:jar:1.2.17:provided
> [INFO] +- commons-io:commons-io:jar:2.4:provided
> [INFO] +- commons-logging:commons-logging:jar:1.1.3:provided
> [INFO] +- junit:junit:jar:4.13.1:test
> [INFO] | \- org.hamcrest:hamcrest-core:jar:1.3:test
> [INFO] \- org.easymock:easymock:jar:3.4:test
> [INFO] \- org.objenesis:objenesis:jar:2.2:test
> [INFO]
> ------------------------------------------------------------------------
> [INFO] BUILD SUCCESS
> [INFO]
> ------------------------------------------------------------------------
>
> Thank you
>