You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@iceberg.apache.org by Pavan Lanka <pl...@apple.com.INVALID> on 2021/05/26 16:48:43 UTC

Shaded and UnShaded?

Hi,

Currently I see that Iceberg shades <https://github.com/apache/iceberg/blob/master/build.gradle#L894> a bunch of libraries and does not offer(please correct me in case I am mistaken here) any releases that does not shade these libraries.

For some integrations (Avro, ORC, Parquet) I was wondering if we would be open to the idea of supporting both a shaded version and an unshaded version. 

The unshaded version will allow us to deal with bug fixes and compatible improvements without necessitating an Iceberg build. It is understood that this will come with potential challenges of an incompatible release that has not been whetted through the Iceberg release process.

We like to propose both a shaded and unshaded release of Iceberg to offer choice and wanted to hear from the community on this. Please share your thoughts.

Thanks,
Pavan

Re: Shaded and UnShaded?

Posted by Pavan Lanka <pl...@apple.com.INVALID>.

Thanks Russell for the clarification. I think we can make this work.

Regards,
Pavan

> On May 26, 2021, at 9:56 AM, Russell Spitzer <ru...@gmail.com> wrote:
> 
> You are looking at the "runtime" jars, the non-runtime jars are not shaded. For example
> 
> https://mvnrepository.com/artifact/org.apache.iceberg/iceberg-spark2/0.11.1 <https://mvnrepository.com/artifact/org.apache.iceberg/iceberg-spark2/0.11.1> - Unshaded
> vs
> https://mvnrepository.com/artifact/org.apache.iceberg/iceberg-spark-runtime/0.11.1 <https://mvnrepository.com/artifact/org.apache.iceberg/iceberg-spark-runtime/0.11.1> - Shaded
> 
> So as is you can actually get unshaded versions of all Iceberg libraries with every release. If you want to use unshaded libraries they are available.
> 
> Now you may ;) be using a Spark distribution which 
> includes a shaded version of Iceberg at runtime. In that case you should reach out to the providers of that runtime
> and ask them if they can figure out another solution for you.
> 
>> On May 26, 2021, at 11:48 AM, Pavan Lanka <planka@apple.com.INVALID <ma...@apple.com.INVALID>> wrote:
>> 
>> Hi,
>> 
>> Currently I see that Iceberg shades <https://github.com/apache/iceberg/blob/master/build.gradle#L894> a bunch of libraries and does not offer(please correct me in case I am mistaken here) any releases that does not shade these libraries.
>> 
>> For some integrations (Avro, ORC, Parquet) I was wondering if we would be open to the idea of supporting both a shaded version and an unshaded version. 
>> 
>> The unshaded version will allow us to deal with bug fixes and compatible improvements without necessitating an Iceberg build. It is understood that this will come with potential challenges of an incompatible release that has not been whetted through the Iceberg release process.
>> 
>> We like to propose both a shaded and unshaded release of Iceberg to offer choice and wanted to hear from the community on this. Please share your thoughts.
>> 
>> Thanks,
>> Pavan
>

Re: Shaded and UnShaded?

Posted by Russell Spitzer <ru...@gmail.com>.

You are looking at the "runtime" jars, the non-runtime jars are not shaded. For example

https://mvnrepository.com/artifact/org.apache.iceberg/iceberg-spark2/0.11.1 <https://mvnrepository.com/artifact/org.apache.iceberg/iceberg-spark2/0.11.1> - Unshaded
vs
https://mvnrepository.com/artifact/org.apache.iceberg/iceberg-spark-runtime/0.11.1 <https://mvnrepository.com/artifact/org.apache.iceberg/iceberg-spark-runtime/0.11.1> - Shaded

So as is you can actually get unshaded versions of all Iceberg libraries with every release. If you want to use unshaded libraries they are available.

Now you may ;) be using a Spark distribution which 
includes a shaded version of Iceberg at runtime. In that case you should reach out to the providers of that runtime
and ask them if they can figure out another solution for you.

> On May 26, 2021, at 11:48 AM, Pavan Lanka <pl...@apple.com.INVALID> wrote:
> 
> Hi,
> 
> Currently I see that Iceberg shades <https://github.com/apache/iceberg/blob/master/build.gradle#L894> a bunch of libraries and does not offer(please correct me in case I am mistaken here) any releases that does not shade these libraries.
> 
> For some integrations (Avro, ORC, Parquet) I was wondering if we would be open to the idea of supporting both a shaded version and an unshaded version. 
> 
> The unshaded version will allow us to deal with bug fixes and compatible improvements without necessitating an Iceberg build. It is understood that this will come with potential challenges of an incompatible release that has not been whetted through the Iceberg release process.
> 
> We like to propose both a shaded and unshaded release of Iceberg to offer choice and wanted to hear from the community on this. Please share your thoughts.
> 
> Thanks,
> Pavan