Re: [D] Gluten Flink Pending tasks [gluten]

["KevinyhZou (via GitHub)" <gi...@apache.org>]

GitHub user KevinyhZou edited a discussion: Gluten Flink Pending tasks

Currently, we have supported nexmark sql taks from `q0` to  q22` in gluten flink, but some operator still need to be refined. We plan to prioritize support for stateless operators first , then add support for stateful operators.

### Stateless opeators
- Connector support: kafka, pulsar, hdfs 
  1. `kafka` https://github.com/bigo-sg/velox/pull/2, `hdfs` https://github.com/bigo-sg/velox/pull/6 connectors are already implementd，need more tests and refine
  2. pulsar connector not implement yet
- Format support: json, pb, parquet, orc
  1. json format is done. pb, parquet ,orc is still not implemented.
- state management: pulsar/kafka offset,  hive write part counter ...;
- checkpoint: periodic snapshot  and restore
   1. interfaces have been defined
- failover：exceptions or errors need to be catched by velox, and flink controll the failover process
- support mutiple parallelism
   currently only support single pallelism
- basic data type
   1. primitive and complex types already supported: Boolean, Int, Bigint, Double, Varchar, Char, Timestamp, Decimal, Date, Row, Array, Map;
   3. types not fully supported : Timestamp with precision or timezone, e.g timestamp(n), timestamp_ltz
- experssion support
  `FROM_UNIXTIME`, `DATE_FORMAT`, `UNIX_TIMESTAMP`, `HOUR`, `DAY`, `YEAR`, `TO_TIMESTAMP`, `PROCTIME`, `unix_timestamp`, `concat`, `Json_Value`, `TO_DATE`, `TO_TIMESTAMP_LTZ`, `SUBSTRING`, `CURRENT_DATE`, `FROM_BASE64`, `COALESCE`, `ifnull`, `TIMESTAMPADD`, `LOCALTIMESTAMP`, ``

### Stateful operators
- window aggregate：tumble / hop / session / cumulative
  some prs: https://github.com/bigo-sg/velox/pull/30 (processing time window)
                    https://github.com/KevinyhZou/velox/pull/2 (event time window)
 session / cumulative window need to be implemented

- watermark
 a basic version has been implemented
- retraction semantics
  this feature is required for group aggregate and join, which has not been implemented, maybe we should introduce the `rowkind` just as defined in `flink`, combine it with row vector.
- group aggregate
  A simple version has been implemented.
- TopN rank
  A simple version has been implemented，need more tests and refine
- join: regular join / window join / interval join / temporary join
 some prs: https://github.com/bigo-sg/velox/pull/16 (temporary join for nexmark q13)
- state: rockdb state / heap state 
  A simple version for rocksdb state backend has been implemented: https://github.com/bigo-sg/velox/pull/21, yet further refinement is still needed.
  A simple version for heap state backend has been introduced in pr: https://github.com/bigo-sg/velox/pull/30

- timers: processing time timers and event time timers
  A basic version has been implemented: https://github.com/bigo-sg/velox/pull/29 , still need some additional work：timers serialization / stateful storage 


### Feature list
- [x] Support parquet format: https://github.com/apache/gluten/issues/12202
- [x] Support orc format: https://github.com/apache/gluten/issues/12203
- [x] Support pulsar connector: https://github.com/apache/gluten/issues/12204
- [x] Kafka / Pulsar offset state management: https://github.com/apache/gluten/issues/12205
- [x] Checkpoint snapshot and restore: https://github.com/apache/gluten/issues/12206
- [x] Support multiple parallelism: https://github.com/apache/gluten/issues/12207
- [ ] Failover controll by Flink: https://github.com/apache/gluten/issues/12208
- [x] Data type support: timestamp with precision and timezone:  https://github.com/apache/gluten/issues/12209
- [ ] Expressions support: https://github.com/apache/gluten/issues/12210
- [ ] metrics collect: https://github.com/apache/gluten/issues/12429 


GitHub link: https://github.com/apache/gluten/discussions/12164

----
This is an automatically sent email for dev@gluten.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@gluten.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@gluten.apache.org
For additional commands, e-mail: dev-help@gluten.apache.org