You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by GitBox <gi...@apache.org> on 2020/04/17 05:06:00 UTC
[GitHub] [incubator-iceberg] rdsr opened a new issue #934: We should publish
a shaded version of problematic dependencies to use in Iceberg
rdsr opened a new issue #934: We should publish a shaded version of problematic dependencies to use in Iceberg
URL: https://github.com/apache/incubator-iceberg/issues/934
As discussed in our meetup, we should have a shaded version of guava. There are two ways of doing this
- Add a Gradle module, which will be an empty module with only the problematic dependencies- guava, jackson etc. This will work fine for Gradle build, but has 2 main downsides
- Intellij does not understand it well. In my experience there will be class-path conflicts when working from within Intellij as it will fail to import the shaded jar well.
- Another issue is that we will always have to rebuild the shaded module during build, which would increase our interactive development time and can hinder in development.
- Another options is publish the problematic shaded artifact [iceberg-problematic-deps e.g] first to some public repository. It will fix both the above problems. Intellij will work and will also save us from rebuilding the shaded module every time during the build. Though I'm not sure where will we house this repo
Any thoughts on which one people prefer?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [incubator-iceberg] rdsr commented on issue #934: We should publish
a shaded version of problematic dependencies to use in Iceberg
Posted by GitBox <gi...@apache.org>.
rdsr commented on issue #934: We should publish a shaded version of problematic dependencies to use in Iceberg
URL: https://github.com/apache/incubator-iceberg/issues/934#issuecomment-615545318
Hi @aokolnychyi , @massdosage and team are working on a HiveStorageHandler which would go in `iceberg-hive` module. I think they have faced class-path issues relying on iceberg core modules together with hive artifacts in `iceberg-hive`
I too faced an issue with Guava where I was unable to run Mini HDFS to test iceberg-mr because of guava conflicts.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [incubator-iceberg] aokolnychyi commented on issue #934: We should
publish a shaded version of problematic dependencies to use in Iceberg
Posted by GitBox <gi...@apache.org>.
aokolnychyi commented on issue #934: We should publish a shaded version of problematic dependencies to use in Iceberg
URL: https://github.com/apache/incubator-iceberg/issues/934#issuecomment-615539477
@rdsr, I've missed the sync. Is this needed for people who depend on `core`?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [incubator-iceberg] massdosage commented on issue #934: We should publish a shaded version of problematic dependencies to use in Iceberg
Posted by GitBox <gi...@apache.org>.
massdosage commented on issue #934:
URL: https://github.com/apache/incubator-iceberg/issues/934#issuecomment-616439191
Yes, `iceberg-hive` depends on `iceberg-core`, `iceberg-api`, `iceberg-common` and `iceberg-mr` so all of these jar files will need to go on Hive's classpath when you use any of the classes inside iceberg-hive. Due to Guava version conflicts with the version of Guava used by Hive all usages of Guava inside Iceberg's classes need to be shaded and reolcated otherwise there are runtime version conflicts which prevent being able to integrate Iceberg and Hive. I would imagine there would be similar issues with anyone using any Iceberg modules via an API and also having their own version of Guava so would recommend applying this across the whole project if possible.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [incubator-iceberg] massdosage edited a comment on issue #934: We should publish a shaded version of problematic dependencies to use in Iceberg
Posted by GitBox <gi...@apache.org>.
massdosage edited a comment on issue #934:
URL: https://github.com/apache/incubator-iceberg/issues/934#issuecomment-634570859
Can we close this now since https://github.com/apache/incubator-iceberg/pull/935 has been merged (indirectly via https://github.com/apache/incubator-iceberg/pull/1068)? We can always open more specific tickets for other issues we find going forward but I think the main objective has been achieved for Guava.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [incubator-iceberg] rdsr closed issue #934: We should publish a shaded version of problematic dependencies to use in Iceberg
Posted by GitBox <gi...@apache.org>.
rdsr closed issue #934:
URL: https://github.com/apache/incubator-iceberg/issues/934
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [incubator-iceberg] massdosage commented on issue #934: We should publish a shaded version of problematic dependencies to use in Iceberg
Posted by GitBox <gi...@apache.org>.
massdosage commented on issue #934:
URL: https://github.com/apache/incubator-iceberg/issues/934#issuecomment-634570859
Can we close this now since https://github.com/apache/incubator-iceberg/pull/935 has been merged? We can always open more specific tickets for other issues we find going forward but I think the main objective has been achieved for Guava.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [incubator-iceberg] jerryshao edited a comment on issue #934: We should publish a shaded version of problematic dependencies to use in Iceberg
Posted by GitBox <gi...@apache.org>.
jerryshao edited a comment on issue #934:
URL: https://github.com/apache/incubator-iceberg/issues/934#issuecomment-617505128
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [incubator-iceberg] jerryshao commented on issue #934: We should publish a shaded version of problematic dependencies to use in Iceberg
Posted by GitBox <gi...@apache.org>.
jerryshao commented on issue #934:
URL: https://github.com/apache/incubator-iceberg/issues/934#issuecomment-617505128
Apache Ratis uses another repo to manage all the shaded artifacts and publish again to the maven central repo. That's the second way mentioned by @rdsr https://github.com/apache/incubator-ratis-thirdparty/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org