You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@inlong.apache.org by "dockerzhang (via GitHub)" <gi...@apache.org> on 2023/05/19 01:29:24 UTC
[GitHub] [inlong] dockerzhang commented on a diff in pull request #8055: [INLONG-8054][Sort] Update document information of Sort
dockerzhang commented on code in PR #8055:
URL: https://github.com/apache/inlong/pull/8055#discussion_r1198459229
##########
inlong-sort/quick_start.md:
##########
@@ -1,60 +1,71 @@
-#Set up flink environment
-Currently inlong-sort is based on flink, before you run an inlong-sort application,
-you need to set up flink environment.
-
-<a href="https://ci.apache.org/projects/flink/flink-docs-release-1.9/ops/deployment/cluster_setup.html" target="_blank">how to set up flink environment</a>
-
-
-#Compile inlong-sort
-`cd /your_path/Inlong/inlong-sort`
-
-`mvn clean package`
-
-After running this, you will get `core-{version}.jar` in `/your_path/Inlong/inlong-sort/core/target`
-
-#Starting an inlong-sort application
-Now you can submit job to flink with the jar compiled.
-
-<a href="https://ci.apache.org/projects/flink/flink-docs-release-1.9/ops/deployment/yarn_setup.html#submit-job-to-flink" target="_blank">how to submit job to flink</a>
-
-#Necessary configurations
-`--cluster-id ` which is used to represent a specified inlong-sort application
-
-`--zookeeper.quorum` zk quorum
-
-`--zookeeper.path.root` zk root path
-
-`--source.type` source of the application, currently only "tubemq" is supported
-
-`--sink.type` sink of the application, currently "clickhouse", "iceberg" and "hive" are supported
-
-Configurations above are necessary, you can see full configurations in
-
-`~/Inlong/inlong-sort/common/src/main/java/org/apache/inlong/sort/configuration/Constants.java`
-
-**Example**
-
-`--cluster-id my_application --zookeeper.quorum 192.127.0.1:2181 --zookeeper.path.root /zk_root --source.type tubemq --sink.type hive`
-
-#All configurations
-| name | necessary | default value |description |
-| ------------ | ------------ | ------------ | ------------ |
-|cluster-id | Y | NA | used to represent a specified inlong-sort application |
-|zookeeper.quorum | Y | NA | zk quorum |
-|zookeeper.path.root | Y | "/inlong-sort" | zk root path |
-|source.type | Y | NA | source of the application, currently only "tubemq" is supported |
-|sink.type | Y | NA | sink of the application, currently "clickhouse" and "hive" are supported |
-|source.parallelism | N | 1 | parallelism of source |
-|deserialization.parallelism | N | 1 | parallelism of deserialization |
-|sink.parallelism | N | 1 | parallelism of sink |
-|tubemq.master.address | N | NA | tubemq master address used if absent in DataFlowInfo on zk |
-|tubemq.session.key |N |"inlong-sort" | session key used when subscribing to tubemq |
-|tubemq.bootstrap.from.max | N | false | whether consume from max or not when subscribing to tubemq |
-|tubemq.message.not.found.wait.period | N | 350ms | The time of waiting period if tubemq broker return message not found |
-|tubemq.subscribe.retry.timeout | N | 300000 | The time of subscribing tubemq timeout, in millisecond |
-|zookeeper.client.session-timeout | N | 60000 | The session timeout for the ZooKeeper session in ms |
-|zookeeper.client.connection-timeout | N | 15000 | The connection timeout for ZooKeeper in ms |
-|zookeeper.client.retry-wait | N | 5000 | The pause between consecutive retries in ms |
-|zookeeper.client.max-retry-attempts | N | 3 | The number of connection retries before the client gives up |
-|zookeeper.client.acl | N | "open" | Defines the ACL (open/creator) to be configured on ZK node. The configuration value can be set to “creator” if the ZooKeeper server configuration has the “authProvider” property mapped to use SASLAuthenticationProvider and the cluster is configured to run in secure mode (Kerberos) |
-|zookeeper.sasl.disable | N | false | Whether disable zk sasl or not |
\ No newline at end of file
+---
+title: Deployment
+sidebar_position: 2
+---
Review Comment:
Please remove the --- part.
##########
inlong-sort/README.md:
##########
@@ -1,55 +1,39 @@
# Description
-# Overview
-
-InLong-Sort is used to extract data from different source systems, then transforms the data and finally loads the data
-into diffrent storage systems.
-InLong-Sort is simply a Flink Application, and relys on InLong-Manager to manage meta data(such as the source
-informations and storage informations).
-
-# Features
-
-## Supported Extract Node
-
-- Pulsar
-- MySQL
-- Kafka
-- MongoDB
-- PostgreSQL
-- HDFS
-
-## Supported Transform
-
-- String Split
-- String Regular Replace
-- String Regular Replace First Matched Value
-- Data Filter
-- Data Distinct
-- Regular Join
-
-## Supported Load Node
-
-- Hive
-- Kafka
-- HBase
-- ClickHouse
-- Iceberg
-- Hudi
-- PostgreSQL
-- HDFS
-- TDSQL Postgres
-- Redis
-
-## Future Plans
-
-### More kinds of Extract Node
-
-Oracle, SqlServer, and etc.
-
-### More kinds of Transform
-
-Time window aggregation, Content extraction, Type conversion, Time format conversion, and etc.
-
-### More kinds of Load Node
-
-Elasticsearch, and etc.
\ No newline at end of file
+## Overview
+InLong Sort is used to extract data from different source systems, then transforms the data and finally loads the data into different storage systems.
+
+InLong Sort can be used together with the Manager to manage metadata, or it can run independently in the Flink environment.
+
+## Features
+### Supports a variety of data nodes
+
+| Type | Service |
+|--------------|--------------------------------------------|
+| Extract Node | Pulsar |
+| | MySQL |
+| | Kafka |
+| | MongoDB |
+| | PostgreSQL |
+| Transform | String Split |
+| | String Regular Replace |
+| | String Regular Replace First Matched Value |
+| | Data Filter |
+| | Data Distinct |
+| | Regular Join |
+| Load Node | Hive |
+| | Kafka |
+| | HBase |
+| | ClickHouse |
+| | Iceberg |
+| | PostgreSQL |
+| | HDFS |
+| | TDSQL Postgres |
+| | Hudi |
+
+## Compile Multi-version of Flink
Review Comment:
It's better to update the guide and make it more clear, like:
## Build
```
# For Apache Flink 1.13 (default)
mvn clean install -DskipTests
# For other Apache Flink version
mvn clean install -DskipTests -P v1.15
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org