You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@iceberg.apache.org by Ryan Murray <ry...@dremio.com> on 2020/12/14 14:28:09 UTC

Breaking change in IcebergSource

Hi all,

We have a proposed PR here[1] which allows for custom Catalogs to be used
in the Spark3 Dataframe API. As discussed in the PR[2] this change breaks
support for specifying schema in the dataframe reader/writer eg:

spark.read().scheam(schema).format("iceberg").load(table)


The schema argument would no longer be supported in Spark3/Iceberg.


This feature was originally added in [3] by Xabriel and the Adobe team. Is
this feature being widely used and is the change a breaking one for a lot
of people? My understanding is that Spark3 support for these use cases is
much better.


I would appreciate any feedback and to see if we can find a way forward so
that this change can be included in the next iceberg release.


Best

Ryan Murray


[1] https://github.com/apache/iceberg/pull/1783

[2] https://github.com/apache/iceberg/pull/1783#issuecomment-742889117

[3] https://github.com/apache/iceberg/pull/590