You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@seatunnel.apache.org by GitBox <gi...@apache.org> on 2022/06/11 16:30:05 UTC
[GitHub] [incubator-seatunnel] wl3207601 opened a new issue, #2006: [Bug] [Module Name] The write data to doris succeed but the flink job error :"TwoPhaseCommit": "false"
wl3207601 opened a new issue, #2006:
URL: https://github.com/apache/incubator-seatunnel/issues/2006
### Search before asking
- [X] I had searched in the [issues](https://github.com/apache/incubator-seatunnel/issues?q=is%3Aissue+label%3A%22bug%22) and found no similar issues.
### What happened
source is mysql .sink is doris. write data to doris succeed but the flink job error :"TwoPhaseCommit": "false"
### SeaTunnel Version
2.1.1
### SeaTunnel Config
```conf
env {
execution.parallelism = 1
}
source {
JdbcSource {
driver = com.mysql.jdbc.Driver
url = "jdbc:mysql://localhost:3306/test"
username = test
query = "select * from authors"
}
}
transform {
}
sink {
DorisSink {
fenodes = "XX.XX.XX.XX:8030"
database = test
table = authors
user = test
password = pass
batch_size = 5
max_retries = 1
interval = 5000
}
}
```
### Running Command
```shell
./bin/start-seatunnel-flink.sh --config ./config/mysqltodoris_test.conf
```
### Error Exception
```log
Caused by: java.lang.RuntimeException: org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.exc.UnrecognizedPropertyException: Unrecognized field "TwoPhaseCommit" (class org.apache.seatunnel.flink.doris.sink.RespContent), not marked as ignorable (17 known properties: "CommitAndPublishTimeMs", "ExistingJobStatus", "TxnId", "Message", "Status", "ReadDataTimeMs", "Label", "NumberUnselectedRows", "NumberFilteredRows", "StreamLoadPutTimeMs", "LoadBytes", "NumberLoadedRows", "NumberTotalRows", "LoadTimeMs", "WriteDataTimeMs", "BeginTxnTimeMs", "ErrorURL"])
at [Source: (String)"{ "TxnId": 19600, "Label": "flink_sink_20220612_000844_f0c4b4de7a6a4931927379e102daf2cb", "TwoPhaseCommit": "false", "Status": "Success", "Message": "OK", "NumberTotalRows": 4, "NumberLoadedRows": 4, "NumberFilteredRows": 0, "NumberUnselectedRows": 0, "LoadBytes": 43, "LoadTimeMs": 75, "BeginTxnTimeMs": 1, "StreamLoadPutTimeMs": 1, "ReadDataTimeMs": 0, "WriteDataTimeMs": 38, "CommitAndPublishTimeMs": 33}"; line: 1, column: 119] (through reference chain: org.apache.seatunnel.flink.doris.sink.RespContent["TwoPhaseCommit"])
at org.apache.seatunnel.flink.doris.sink.DorisStreamLoad.load(DorisStreamLoad.java:99)
at org.apache.seatunnel.flink.doris.sink.DorisOutputFormat.flush(DorisOutputFormat.java:214)
... 5 more
```
### Flink or Spark Version
flink:1.13.6
### Java or Scala Version
java:1.8
scala:2.12
### Screenshots
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] chen0623-bak commented on issue #2006: [Bug] [Module Name] The write data to doris succeed but the flink job error :"TwoPhaseCommit": "false"
Posted by GitBox <gi...@apache.org>.
chen0623-bak commented on issue #2006:
URL: https://github.com/apache/incubator-seatunnel/issues/2006#issuecomment-1168766805
I just tried, but no error was reported. Here is my configuration file
env {
execution.parallelism = 1
}
source {
JdbcSource {
driver = "com.mysql.jdbc.Driver"
url = "jdbc:mysql://localhost/test"
username = "root"
password = "root"
result_table_name = "student_source"
}
}
transform {
}
sink {
DorisSink {
source_table_name = "student_source"
fenodes = "singlenode:8030"
database = "chen_test"
table = "student_result"
user = "root"
batch_size = 5
doris.column_separator="\t"
doris.columns="id,name,age"
}
}
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] wl3207601 commented on issue #2006: [Bug] [Module Name] The write data to doris succeed but the flink job error :"TwoPhaseCommit": "false"
Posted by GitBox <gi...@apache.org>.
wl3207601 commented on issue #2006:
URL: https://github.com/apache/incubator-seatunnel/issues/2006#issuecomment-1163907812
> which doris version does you uesed?
doris1.0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] wl3207601 commented on issue #2006: [Bug] [Module Name] The write data to doris succeed but the flink job error :"TwoPhaseCommit": "false"
Posted by GitBox <gi...@apache.org>.
wl3207601 commented on issue #2006:
URL: https://github.com/apache/incubator-seatunnel/issues/2006#issuecomment-1169433736
> env {
#2054
thanks, the doris1.0+ version have this problem
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] wl3207601 commented on issue #2006: [Bug] [Module Name] The write data to doris succeed but the flink job error :"TwoPhaseCommit": "false"
Posted by GitBox <gi...@apache.org>.
wl3207601 commented on issue #2006:
URL: https://github.com/apache/incubator-seatunnel/issues/2006#issuecomment-1163907674
doris1.0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] hk-lrzy commented on issue #2006: [Bug] [Module Name] The write data to doris succeed but the flink job error :"TwoPhaseCommit": "false"
Posted by GitBox <gi...@apache.org>.
hk-lrzy commented on issue #2006:
URL: https://github.com/apache/incubator-seatunnel/issues/2006#issuecomment-1163891101
which doris version does you uesed?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] chen0623-bak commented on issue #2006: [Bug] [Module Name] The write data to doris succeed but the flink job error :"TwoPhaseCommit": "false"
Posted by GitBox <gi...@apache.org>.
chen0623-bak commented on issue #2006:
URL: https://github.com/apache/incubator-seatunnel/issues/2006#issuecomment-1167494042
There is a problem with the version of Doris. It sends data through a request. The response has some additional attributes, resulting in JSON parsing failure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org