You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@druid.apache.org by GitBox <gi...@apache.org> on 2019/01/25 22:30:26 UTC
[GitHub] drcrallen opened a new issue #6915: Dependencies are included too
many times
drcrallen opened a new issue #6915: Dependencies are included too many times
URL: https://github.com/apache/incubator-druid/issues/6915
It is often a pain to track down dependency problems, especially transient dependencies. It is helpful if Druid only ever include a jar ONCE if possible, and not shade it so that the exact version can be tracked. Unfortunately we seem to have missed a few duplicates, or they are in extensions and might be better served to be brought into core somehow.
here is a handy little shell command I use to help find duplicates. I ran it against master at `e1033bb412244fc9dff9dea1c89d43a8e0081fd3` (today):
```
$ TARBALL="/Users/charles.allen/src/druid/distribution/target/apache-druid-0.13.0-incubating-SNAPSHOT-bin.tar.gz"; cat <(tar -t --include='*/lib/*.jar' -f "${TARBALL}" | cut -d / -f 3 ) <(tar -t --include='*/extensions/*.jar' -f "${TARBALL}" | cut -d / -f 4 ) | sed -e 's/-[0-9].*\.jar//' | sort | uniq -c | sort -n -r --stable
5 snappy-java
4 commons-beanutils
4 commons-collections
4 commons-compress
4 gson
4 objenesis
4 slf4j-api
3 commons-codec
3 commons-digester
3 commons-lang
3 commons-logging
3 commons-math3
3 curator-framework
3 curator-recipes
3 guava
3 jetty-util
3 json-smart
3 kafka-clients
3 metrics-core
3 servlet-api
2 apacheds-i18n
2 apacheds-kerberos-codec
2 api-asn1-api
2 api-util
2 aws-java-sdk-core
2 commons-beanutils-core
2 commons-configuration
2 commons-net
2 druid-lookups-cached-global
2 hadoop-auth
2 hadoop-common
2 htrace-core4
2 httpclient
2 httpcore
2 ion-java
2 jackson-annotations
2 jackson-core
2 jackson-databind
2 jackson-dataformat-cbor
2 jackson-jaxrs
2 jackson-xc
2 javax.servlet-api
2 jaxb-api
2 jcip-annotations
2 jersey-client
2 jersey-core
2 jersey-server
2 jetty
2 jetty-client
2 jetty-continuation
2 jetty-http
2 jetty-io
2 jetty-proxy
2 jetty-security
2 jetty-server
2 jetty-servlet
2 jetty-servlets
2 jetty-sslengine
2 jmespath-java
2 joda-time
2 jopt-simple
2 jsp-api
2 jsr311-api
2 kafka_2.10
2 mapdb
2 netty
2 nimbus-jose-jwt
2 okhttp
2 protobuf-java
2 scala-library
2 stax-api
2 xmlenc
2 xz
2 zkclient
1 RoaringBitmap
1 accessors-smart
1 aether-api
1 aether-connector-file
1 aether-connector-okhttp
1 aether-impl
1 aether-spi
1 aether-util
1 aggdesigner-algorithm
1 airline
1 antlr
1 antlr4-runtime
1 aopalliance
1 asm
1 asm-commons
1 asm-tree
1 async-http-client
1 async-http-client-netty-utils
1 audience-annotations
1 avatica-core
1 avatica-metrics
1 avatica-server
1 avro
1 avro-ipc
1 avro-mapred
1 aws-java-sdk-ec2
1 aws-java-sdk-kinesis
1 aws-java-sdk-kms
1 aws-java-sdk-s3
1 aws-java-sdk-sts
1 caffeine
1 calcite-core
1 calcite-linq4j
1 checker
1 classmate
1 commons-cli
1 commons-collections4
1 commons-compiler
1 commons-dbcp2
1 commons-io
1 commons-lang3
1 commons-pool
1 commons-pool2
1 commons-text
1 commons-validator
1 compress-lzf
1 config-magic
1 curator-client
1 curator-x-discovery
1 derby
1 derbyclient
1 derbynet
1 disruptor
1 druid-avro-extensions
1 druid-aws-common
1 druid-basic-security
1 druid-bloom-filter
1 druid-console
1 druid-core
1 druid-datasketches
1 druid-examples
1 druid-hdfs-storage
1 druid-histogram
1 druid-hll
1 druid-indexing-hadoop
1 druid-indexing-service
1 druid-kafka-eight
1 druid-kafka-extraction-namespace
1 druid-kafka-indexing-service
1 druid-kerberos
1 druid-kinesis-indexing-service
1 druid-lookups-cached-single
1 druid-parquet-extensions
1 druid-processing
1 druid-protobuf-extensions
1 druid-s3-extensions
1 druid-server
1 druid-services
1 druid-sql
1 druid-stats
1 error_prone_annotations
1 esri-geometry-api
1 extendedset
1 fastutil
1 geoip2
1 guice
1 guice-multibindings
1 guice-servlet
1 hadoop-client
1 hadoop-hdfs-client
1 hadoop-mapreduce-client-app
1 hadoop-mapreduce-client-common
1 hadoop-mapreduce-client-core
1 hadoop-mapreduce-client-jobclient
1 hadoop-mapreduce-client-shuffle
1 hadoop-yarn-api
1 hadoop-yarn-client
1 hadoop-yarn-common
1 hadoop-yarn-server-common
1 hibernate-validator
1 hive-storage-api
1 icu4j
1 irc-api
1 jackson-core-asl
1 jackson-dataformat-smile
1 jackson-datatype-guava
1 jackson-datatype-joda
1 jackson-jaxrs-base
1 jackson-jaxrs-json-provider
1 jackson-jaxrs-smile-provider
1 jackson-jq
1 jackson-mapper-asl
1 jackson-module-jaxb-annotations
1 janino
1 java-xmlbuilder
1 javax.activation
1 javax.el
1 javax.el-api
1 javax.inject
1 jaxb-impl
1 jboss-logging
1 jcl-over-slf4j
1 jcodings
1 jdbi
1 jersey-guice
1 jersey-json
1 jersey-servlet
1 jets3t
1 jettison
1 jline
1 jna
1 joni
1 jsch
1 json-path
1 jsr305
1 jvm-attach-api
1 kafka-schema-registry-client
1 leveldbjni-all
1 log4j
1 log4j-api
1 log4j-core
1 log4j-jul
1 log4j-slf4j-impl
1 lz4-java
1 maven-aether-provider
1 maven-artifact
1 maven-model
1 maven-model-builder
1 maven-repository-metadata
1 maven-settings
1 maven-settings-builder
1 maxminddb
1 memory
1 mysql-metadata-storage
1 netty-all
1 netty-buffer
1 netty-codec
1 netty-codec-dns
1 netty-codec-http
1 netty-codec-socks
1 netty-common
1 netty-handler
1 netty-handler-proxy
1 netty-reactive-streams
1 netty-resolver
1 netty-resolver-dns
1 netty-transport
1 netty-transport-native-epoll
1 netty-transport-native-unix-common
1 okio
1 opencsv
1 paranamer
1 parquet-avro
1 parquet-column
1 parquet-common
1 parquet-encoding
1 parquet-format
1 parquet-hadoop
1 parquet-jackson
1 plexus-interpolation
1 plexus-utils
1 postgresql
1 postgresql-metadata-storage
1 protobuf-dynamic
1 protobuf-java-util
1 reactive-streams
1 rhino
1 schema-repo-api
1 schema-repo-avro
1 schema-repo-client
1 schema-repo-common
1 shims
1 sigar
1 simple-client-sslcontext
1 sketches-core
1 spymemcached
1 stringtemplate
1 tesla-aether
1 twitter4j-async
1 twitter4j-core
1 twitter4j-stream
1 validation-api
1 velocity
1 wagon-provider-api
1 zookeeper
1 zstd-jni
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org