You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@iceberg.apache.org by Chen Song <ch...@gmail.com> on 2021/04/14 14:04:08 UTC
Iceberg Python library support
Is https://iceberg.apache.org/python-feature-support/ still up to date? Are
the following statements true for Iceberg python library support?
- The python library has only limited support for read (for parquet only
it seems
https://github.com/apache/iceberg/blob/master/python/iceberg/parquet/parquet_reader.py
).
- The python library has no support for write.
- The python library doesn't have full support for metadata operations
(according to the matrix in
https://iceberg.apache.org/python-feature-support/). I am not sure about
this.
- Hive tables has support for drop table operation,
https://github.com/apache/iceberg/blob/ccb0b20831c8b07c4663ac83b0bca82acb537279/python/iceberg/hive/hive_tables.py
- FileSystem tables has support for create table,
https://github.com/apache/iceberg/blob/master/python/iceberg/core/filesystem/filesystem_tables.py
.
Best,
--
Chen Song
Re: Iceberg Python library support
Posted by Ryan Murray <ry...@gmail.com>.
Hey Chen,
No, not that I am aware of. I have some ideas/plans and I know others do as
well but we haven't made a concerted effort yet to bring everything
together. Is there something specific you are looking for or interested to
help on?
Best,
Ryan
On Wed, Apr 14, 2021 at 4:41 PM Chen Song <ch...@gmail.com> wrote:
> Thanks Rayn. A follow up question on the read path. Is there a list of
> tickets to track the on-going effort? I did a quick search and found
> https://github.com/apache/iceberg/issues?q=is%3Aissue+is%3Aopen+python+.
> There doesn't seem to be any tickets directly related with read.
>
> Chen
>
> On Wed, Apr 14, 2021 at 10:28 AM Ryan Murray <ry...@gmail.com> wrote:
>
>> Hey Chen Song,
>>
>> Answers inline below
>>
>> On Wed, Apr 14, 2021 at 4:04 PM Chen Song <ch...@gmail.com> wrote:
>>
>>> Is https://iceberg.apache.org/python-feature-support/ still up to date?
>>> Are the following statements true for Iceberg python library support?
>>>
>>> - The python library has only limited support for read (for parquet
>>> only it seems
>>> https://github.com/apache/iceberg/blob/master/python/iceberg/parquet/parquet_reader.py
>>> ).
>>>
>>> this is correct and the read path is still being worked on
>>
>>>
>>> - The python library has no support for write.
>>>
>>> correct
>>
>>>
>>> - The python library doesn't have full support for metadata
>>> operations (according to the matrix in
>>> https://iceberg.apache.org/python-feature-support/). I am not sure
>>> about this.
>>> - Hive tables has support for drop table operation,
>>> https://github.com/apache/iceberg/blob/ccb0b20831c8b07c4663ac83b0bca82acb537279/python/iceberg/hive/hive_tables.py
>>>
>>> this was added recently so yes the docs are out of date, I just raised
>> https://github.com/apache/iceberg/pull/2478 to address that
>>
>>>
>>> - FileSystem tables has support for create table,
>>> https://github.com/apache/iceberg/blob/master/python/iceberg/core/filesystem/filesystem_tables.py
>>> .
>>>
>>> both filesystem and Hive support create tables and the docs have been
>> updated but not pushed to the site yet (waiting for the next release
>> 0.12.0). The up to date support matrix:
>> https://github.com/apache/iceberg/blob/master/site/docs/python-feature-support.md
>> <https://github.com/apache/iceberg/blob/master/site/docs/python-feature-support.md>
>>
>> Best,
>>> --
>>> Chen Song
>>>
>>>
>
> --
> Chen Song
>
>
Re: Iceberg Python library support
Posted by Chen Song <ch...@gmail.com>.
Thanks Rayn. A follow up question on the read path. Is there a list of
tickets to track the on-going effort? I did a quick search and found
https://github.com/apache/iceberg/issues?q=is%3Aissue+is%3Aopen+python+.
There doesn't seem to be any tickets directly related with read.
Chen
On Wed, Apr 14, 2021 at 10:28 AM Ryan Murray <ry...@gmail.com> wrote:
> Hey Chen Song,
>
> Answers inline below
>
> On Wed, Apr 14, 2021 at 4:04 PM Chen Song <ch...@gmail.com> wrote:
>
>> Is https://iceberg.apache.org/python-feature-support/ still up to date?
>> Are the following statements true for Iceberg python library support?
>>
>> - The python library has only limited support for read (for parquet
>> only it seems
>> https://github.com/apache/iceberg/blob/master/python/iceberg/parquet/parquet_reader.py
>> ).
>>
>> this is correct and the read path is still being worked on
>
>>
>> - The python library has no support for write.
>>
>> correct
>
>>
>> - The python library doesn't have full support for metadata
>> operations (according to the matrix in
>> https://iceberg.apache.org/python-feature-support/). I am not sure
>> about this.
>> - Hive tables has support for drop table operation,
>> https://github.com/apache/iceberg/blob/ccb0b20831c8b07c4663ac83b0bca82acb537279/python/iceberg/hive/hive_tables.py
>>
>> this was added recently so yes the docs are out of date, I just raised
> https://github.com/apache/iceberg/pull/2478 to address that
>
>>
>> - FileSystem tables has support for create table,
>> https://github.com/apache/iceberg/blob/master/python/iceberg/core/filesystem/filesystem_tables.py
>> .
>>
>> both filesystem and Hive support create tables and the docs have been
> updated but not pushed to the site yet (waiting for the next release
> 0.12.0). The up to date support matrix:
> https://github.com/apache/iceberg/blob/master/site/docs/python-feature-support.md
> <https://github.com/apache/iceberg/blob/master/site/docs/python-feature-support.md>
>
> Best,
>> --
>> Chen Song
>>
>>
--
Chen Song
Re: Iceberg Python library support
Posted by Ryan Murray <ry...@gmail.com>.
Hey Chen Song,
Answers inline below
On Wed, Apr 14, 2021 at 4:04 PM Chen Song <ch...@gmail.com> wrote:
> Is https://iceberg.apache.org/python-feature-support/ still up to date?
> Are the following statements true for Iceberg python library support?
>
> - The python library has only limited support for read (for parquet
> only it seems
> https://github.com/apache/iceberg/blob/master/python/iceberg/parquet/parquet_reader.py
> ).
>
> this is correct and the read path is still being worked on
>
> - The python library has no support for write.
>
> correct
>
> - The python library doesn't have full support for metadata operations
> (according to the matrix in
> https://iceberg.apache.org/python-feature-support/). I am not sure
> about this.
> - Hive tables has support for drop table operation,
> https://github.com/apache/iceberg/blob/ccb0b20831c8b07c4663ac83b0bca82acb537279/python/iceberg/hive/hive_tables.py
>
> this was added recently so yes the docs are out of date, I just raised
https://github.com/apache/iceberg/pull/2478 to address that
>
> - FileSystem tables has support for create table,
> https://github.com/apache/iceberg/blob/master/python/iceberg/core/filesystem/filesystem_tables.py
> .
>
> both filesystem and Hive support create tables and the docs have been
updated but not pushed to the site yet (waiting for the next release
0.12.0). The up to date support matrix:
https://github.com/apache/iceberg/blob/master/site/docs/python-feature-support.md
<https://github.com/apache/iceberg/blob/master/site/docs/python-feature-support.md>
Best,
> --
> Chen Song
>
>