You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by GitBox <gi...@apache.org> on 2020/01/16 14:15:13 UTC
[GitHub] [incubator-iceberg] massdosage opened a new issue #740:
AvroSchemaUtil returns shaded Avro classes
massdosage opened a new issue #740: AvroSchemaUtil returns shaded Avro classes
URL: https://github.com/apache/incubator-iceberg/issues/740
When trying to use the `AvroSchemaUtil` in an external project to convert an Iceberg Schema to an Avro schema we run into a problem where the type of the Avro schema objects being returned is `org.apache.iceberg.shaded.org.apache.avro.Schema` instead of `org.apache.avro.Schema` which in turn means they can't be used in any classes which expect the original Avro type. Here is some sample code which demonstrates the problem:
```
import static org.apache.iceberg.types.Types.NestedField.required;
import org.apache.iceberg.Schema;
import org.apache.iceberg.avro.AvroSchemaUtil;
import org.apache.iceberg.types.Types;
public class HiveCatalogTest {
public void avroSchemaWrongType() {
Schema schema = new Schema(Types.StructType
.of(required(1, "foo", Types.LongType.get()), required(2, "bar", Types.LongType.get()))
.fields());
org.apache.iceberg.shaded.org.apache.avro.Schema convert = AvroSchemaUtil.convert(schema, "test");
}
}
```
This problem occurs because Avro's classes are shaded and relocated into the Iceberg jar file. One fix would be to stop doing this but it then couples Iceberg a bit more tightly to a specific Avro version. The alternative would be to make the above `AvroSchemaUtil` class less visible (e.g. private) if the intention is for it not to be used outside of Iceberg itself but it feels like this is removing a potentially useful feature.
My preference would be for the removal of the shading and relocating, I'm happy to attempt a PR to do this.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [incubator-iceberg] rdblue closed issue #740: AvroSchemaUtil
returns shaded Avro classes
Posted by GitBox <gi...@apache.org>.
rdblue closed issue #740: AvroSchemaUtil returns shaded Avro classes
URL: https://github.com/apache/incubator-iceberg/issues/740
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [incubator-iceberg] rdblue commented on issue #740: AvroSchemaUtil
returns shaded Avro classes
Posted by GitBox <gi...@apache.org>.
rdblue commented on issue #740: AvroSchemaUtil returns shaded Avro classes
URL: https://github.com/apache/incubator-iceberg/issues/740#issuecomment-577788780
I'll close this. Thanks for the good discussion, @massdosage! Can you also report the other issues you've run into?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [incubator-iceberg] massdosage commented on issue #740:
AvroSchemaUtil returns shaded Avro classes
Posted by GitBox <gi...@apache.org>.
massdosage commented on issue #740: AvroSchemaUtil returns shaded Avro classes
URL: https://github.com/apache/incubator-iceberg/issues/740#issuecomment-576301877
Thanks for the information on those options @rdsr. For now I've gone with option 1. - depending on `iceberg-core` instead of `iceberg-spark-runtime`. For this particular usage I don't need any of the Spark classes so it's fine. There are other issues with the spark-runtime module related to versions of Hive but this is probably a bigger issue than what I was reporting here. I guess we can close this?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [incubator-iceberg] massdosage commented on issue #740:
AvroSchemaUtil returns shaded Avro classes
Posted by GitBox <gi...@apache.org>.
massdosage commented on issue #740: AvroSchemaUtil returns shaded Avro classes
URL: https://github.com/apache/incubator-iceberg/issues/740#issuecomment-577790319
If/when we find more we'll be sure to report them ;) Thanks all!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [incubator-iceberg] rdsr commented on issue #740: AvroSchemaUtil
returns shaded Avro classes
Posted by GitBox <gi...@apache.org>.
rdsr commented on issue #740: AvroSchemaUtil returns shaded Avro classes
URL: https://github.com/apache/incubator-iceberg/issues/740#issuecomment-575258363
@massdosage I see it that Avro is not part of the public API of Iceberg. There are two ways around this - both of which have issues.
1. Use thin artifacts [iceberg-api, iceberg-core] instead of the shaded artifacts, where Avro will be unshaded. This can have potential class-path conflicts if you use these instead of the shaded runtime artifacts in compute engines Spark, Pig, Presto etc.
1. You can convert the Avro Schema object to a string and then parse it with the shaded Avro api in Iceberg, but then u risk these internal apis changing underneath you with future versions of Iceberg.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org