[GitHub] [incubator-inlong] leo65535 opened a new issue #1720: [Feature] Introduce the basic framework for a workflow based on flinksql

[GitBox <gi...@apache.org>]

leo65535 opened a new issue #1720:
URL: https://github.com/apache/incubator-inlong/issues/1720


   ### Description
   
   ## overview
   Inlong-sort is used to extract data from different source systems, then transforms the data and finally loads the data into different storage systems.
   
   ## features
   ### isolation
   Each sort workflow is an independent application based on flink sql. 
   
   ### rich connectors
   Flink’s Table API & SQL programs can be connected to other external systems for reading and writing both batch and streaming tables.
   A table source provides access to data which is stored in external systems (such as a database, key-value store, message queue, or file system).
   A table sink emits a table to an external storage system. Depending on the type of source and sink, they support different formats such as CSV, Avro, Parquet, or ORC.
   
   If you want to implement your own custom table source or sink, have a look at [the user-defined sources & sinks page](https://nightlies.apache.org/flink/flink-docs-master/docs/dev/table/sourcessinks/).
   
   ### more
   We will support customized udf, dimension table lookup in the future.
   
   ## limitations
   Currently, we just support basic transform framework.
   
   ## workflow demo
   ```sql
   CREATE TABLE events (
     f_type INT,
     f_uid INT,
     ts AS localtimestamp,
     WATERMARK FOR ts AS ts
   ) WITH (
     'connector' = 'datagen',
     'rows-per-second'='5',
     'fields.f_type.min'='1',
     'fields.f_type.max'='5',
     'fields.f_uid.min'='1',
     'fields.f_uid.max'='1000'
   );
   
   CREATE TABLE print_table (
     type INT,
     uid INT,
     lstmt TIMESTAMP
   ) WITH (
     'connector' = 'print',
     'sink.parallelism' = '2'
   );
   
   INSERT INTO print_table SELECT * FROM events where f_type = 1;
   ```
   
   ### Use case
   
   _No response_
   
   ### Are you willing to submit PR?
   
   - [X] Yes, I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [incubator-inlong] leo65535 commented on issue #1720: [Feature] Introduce the basic framework for a workflow based on flinksql

[GitBox <gi...@apache.org>]

leo65535 commented on issue #1720:
URL: https://github.com/apache/incubator-inlong/issues/1720#issuecomment-956107259


   close, block for upgrading flink version first.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [incubator-inlong] leo65535 closed issue #1720: [Feature] Introduce the basic framework for a workflow based on flinksql

[GitBox <gi...@apache.org>]

leo65535 closed issue #1720:
URL: https://github.com/apache/incubator-inlong/issues/1720


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org