You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@seatunnel.apache.org by "devbiu (via GitHub)" <gi...@apache.org> on 2023/06/27 03:26:35 UTC
[GitHub] [seatunnel] devbiu opened a new issue, #4980: [Bug] Split transform plugin not available in Flink
devbiu opened a new issue, #4980:
URL: https://github.com/apache/seatunnel/issues/4980
### Search before asking
- [X] I had searched in the [issues](https://github.com/apache/seatunnel/issues?q=is%3Aissue+label%3A%22bug%22) and found no similar issues.
### What happened
Synchronize Kafka to MySQL using Flink
the same data as the topics of coffee
[Id":"10", "name":"test_admin", "sex":"man", "age":"26", "raw_message":"sun#wuk"]
[Id":"11", "name":"test_admin", "sex":"man", "age":"28", "raw_message":"tang#sz"]
{id":"12":"test_admin","sex":"man":"age":"28", "raw_message":"zhu#bj"}
[Id":"14", "test_admin","sex":"man","age":"28", "raw_message":"bai,gj"]
Because there is no good split source_field, so we try raw_message with the nickname of spark
The separator used is # humour, also includes, split
The sinkWe use is mysql, its structure has id, name, sex, age, raw_message and breaking parameters first_name, last_name
and console for printing
However, the result does not contain first_name to split raw_message, last_name has always been aired, the output of console does not contain first_name, last_name
The other paragraphs were also accompanied in the mysql table.
### SeaTunnel Version
seatunnel 2.3.0
### SeaTunnel Config
```conf
hocon like
env {
job.name = "seatunnel_split_8dlUQM7J"
job.mode = "STREAMING"
}
source {
Kafka {
bootstrap.servers = ""
commit_on_checkpoint = "true"
format = "json"
pattern = "false"
schema = {
fields {
id = "int"
name = "string"
sex = "string"
age = "string"
raw_message = "string"
}
}
start_mode = "earliest"
topic = "demo02"
}
}
transform {
split {
fields = ["first_name", "last_name"]
separator = "#"
}
}
sink {
Jdbc {
batch_interval_ms = 1000
batch_size = 300
connection_check_timeout_sec = 30
driver = "com.mysql.cj.jdbc.Driver"
is_exactly_once = "false"
max_commit_attempts = 3
max_retries = 3
password = ""
support_upsert_by_query_primary_key_exist = "false"
table = "xxxx"
transaction_timeout_sec = -1
url = ""
user = ""
}
Console {
}
}
```
### Running Command
```shell
/opt/seatunnel/seatunnel/bin/start-seatunnel-flink-connector-v2.sh -D env.java.opts="-Dfile.encoding=UTF-8 -Dsun.jnu.encoding=UTF-8" --config xxx.conf
```
### Error Exception
```log
not exception
But the field value output by the data split is empty
```
### Flink or Spark Version
flink 1.13.6
### Java or Scala Version
java 11
### Screenshots
_No response_
### Are you willing to submit PR?
- [X] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [seatunnel] github-actions[bot] commented on issue #4980: [Bug] Split transform plugin not available in Flink
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #4980:
URL: https://github.com/apache/seatunnel/issues/4980#issuecomment-1674080435
This issue has been closed because it has not received response for too long time. You could reopen it if you encountered similar problems in the future.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [seatunnel] github-actions[bot] closed issue #4980: [Bug] Split transform plugin not available in Flink
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #4980: [Bug] Split transform plugin not available in Flink
URL: https://github.com/apache/seatunnel/issues/4980
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [seatunnel] github-actions[bot] commented on issue #4980: [Bug] Split transform plugin not available in Flink
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #4980:
URL: https://github.com/apache/seatunnel/issues/4980#issuecomment-1656979070
This issue has been automatically marked as stale because it has not had recent activity for 30 days. It will be closed in next 7 days if no further activity occurs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [seatunnel] devbiu commented on issue #4980: [Bug] Split transform plugin not available in Flink
Posted by "devbiu (via GitHub)" <gi...@apache.org>.
devbiu commented on issue #4980:
URL: https://github.com/apache/seatunnel/issues/4980#issuecomment-1608718677
The results of synchronizing to MySQL are as follows
<img width="589" alt="image" src="https://github.com/apache/seatunnel/assets/99388822/608fafda-da73-4226-a6aa-e03288838094">
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [seatunnel] devbiu commented on issue #4980: [Bug] Split transform plugin not available in Flink
Posted by "devbiu (via GitHub)" <gi...@apache.org>.
devbiu commented on issue #4980:
URL: https://github.com/apache/seatunnel/issues/4980#issuecomment-1612311447
> The config should follow the doc of https://seatunnel.apache.org/docs/2.3.2/transform-v2/split ; add split_field = "raw_message"
Thank you for your answer. Version 2.3.2 does require the addition of split_ Field="raw_message"
But I am using seatunel version 2.3.0 and did not see any split_ field
doc of https://seatunnel.apache.org/docs/2.3.0/transform/split
<img width="839" alt="image" src="https://github.com/apache/seatunnel/assets/99388822/85d329f7-26ab-494f-ac67-3dcb4cb679dc">
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [seatunnel] john8628 commented on issue #4980: [Bug] Split transform plugin not available in Flink
Posted by "john8628 (via GitHub)" <gi...@apache.org>.
john8628 commented on issue #4980:
URL: https://github.com/apache/seatunnel/issues/4980#issuecomment-1612306134
The config should follow the doc of https://seatunnel.apache.org/docs/2.3.2/transform-v2/split ; add split_field = "raw_message"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org