Impala support Apache Paimon(incubating)

["黄志洋(道五)" <da...@alibaba-inc.com>]

Hi everyone,
Apache Impala currently not support Apache Paimon(incubating).
Apache Paimon(incubating) is a streaming data lake platform that supports high-speed data ingestion, change data tracking and efficient real-time analytics.
Now, we've supported Apache Hudi and Apache Iceberg.
So I think we can consider supporting Apache Paimon(incubating).
This is Apache Paimon(incubating). document
https://paimon.apache.org/docs/master/concepts/overview/ <https://paimon.apache.org/docs/master/concepts/overview/ >
Thanks,
Huang Zhiyang

Re:Re: Impala support Apache Paimon(incubating)

[Nicholas Jiang <ni...@apache.org>]

Hi Csaba Ringhofer,


Thanks for looking into the support of Apache Paimon(incubating). The support mentioned by Huang Zhiyang is meant that using Paimon with Parquet file format. Meanwhile, the input/output format of Paimon in HMS is org.apache.paimon.hive.mapred.PaimonInputFormat/PaimonOutputFormat. What's more, Paimon notifies HMS about modifications via HiveCatalog like createTable etc.


BTW, the support of the Apache Paimon(incubating) could refer to the Iceberg implementation because the most behavior of Apache Paimon(incubating) is similar with Iceberg. Therefore, Impala supports Apache Paimon(incubating) could be token into consideration and Apache Paimon(incubating) is for streaming warehouse and lakehouse.


Regards,
Nicholas Jiang

At 2023-04-06 23:06:48, "Csaba Ringhofer" <cs...@cloudera.com> wrote:
>Hi!
>
>Looked into the documentation about Hive support:
>https://paimon.apache.org/docs/master/engines/hive/
>When reading a Paimon table as external it uses storage
>handler org.apache.paimon.hive.PaimonStorageHandler.
>Impala does not support Hive storage handlers, so that example will not
>work.
>
>>Now, we've supported Apache Hudi and Apache Iceberg.
>Does that mean using Iceberg / Hudi with Parquet file format? In that case
>Impala should be able to read the tables.
>It would be good to know how such a table looks in HMS and how Paimon
>notifies HMS about modifications in order to have
>a good guess about its behavior in Impala.
>
>
>On Mon, Apr 3, 2023 at 1:33 PM 黄志洋(道五) <da...@alibaba-inc.com> wrote:
>
>> Hi everyone,
>> Apache Impala currently not support Apache Paimon(incubating).
>> Apache Paimon(incubating) is a streaming data lake platform that supports
>> high-speed data ingestion, change data tracking and efficient real-time
>> analytics.
>> Now, we've supported Apache Hudi and Apache Iceberg.
>> So I think we can consider supporting Apache Paimon(incubating).
>> This is Apache Paimon(incubating). document
>> https://paimon.apache.org/docs/master/concepts/overview/ <
>> https://paimon.apache.org/docs/master/concepts/overview/ >
>> Thanks,
>> Huang Zhiyang
>>

Re: Impala support Apache Paimon(incubating)

[Csaba Ringhofer <cs...@cloudera.com>]

Hi!

Looked into the documentation about Hive support:
https://paimon.apache.org/docs/master/engines/hive/
When reading a Paimon table as external it uses storage
handler org.apache.paimon.hive.PaimonStorageHandler.
Impala does not support Hive storage handlers, so that example will not
work.

>Now, we've supported Apache Hudi and Apache Iceberg.
Does that mean using Iceberg / Hudi with Parquet file format? In that case
Impala should be able to read the tables.
It would be good to know how such a table looks in HMS and how Paimon
notifies HMS about modifications in order to have
a good guess about its behavior in Impala.


On Mon, Apr 3, 2023 at 1:33 PM 黄志洋(道五) <da...@alibaba-inc.com> wrote:

> Hi everyone,
> Apache Impala currently not support Apache Paimon(incubating).
> Apache Paimon(incubating) is a streaming data lake platform that supports
> high-speed data ingestion, change data tracking and efficient real-time
> analytics.
> Now, we've supported Apache Hudi and Apache Iceberg.
> So I think we can consider supporting Apache Paimon(incubating).
> This is Apache Paimon(incubating). document
> https://paimon.apache.org/docs/master/concepts/overview/ <
> https://paimon.apache.org/docs/master/concepts/overview/ >
> Thanks,
> Huang Zhiyang
>