You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@seatunnel.apache.org by "EngonVHKxZ (via GitHub)" <gi...@apache.org> on 2023/12/28 03:21:59 UTC
[I] [Bug] [Statistic] The synchronization error count is incorrect [seatunnel]
EngonVHKxZ opened a new issue, #6102:
URL: https://github.com/apache/seatunnel/issues/6102
### Search before asking
- [X] I had searched in the [issues](https://github.com/apache/seatunnel/issues?q=is%3Aissue+label%3A%22bug%22) and found no similar issues.
### What happened
After reading data from a table in the database, converting and filtering the data, and outputting it to two tables, the data in both tables is correct, but only the data quantity in one table is counted as successful, while the data quantity in the other table is counted as failed.
### SeaTunnel Version
2.3.3
### SeaTunnel Config
```conf
env {
job.mode="BATCH"
execution.parallelism=5
}
source {
Jdbc {
url="jdbc:oracle:thin:@//xxx.xxx.xxx.xxx:1521/xxx",
driver="oracle.jdbc.OracleDriver",
user="xxx",
password="xxx",
query="select * from xxx.xxx",
fetch_size=1000
result_table_name = "jdbc_dq"
}
}
transform {
Sql {
source_table_name = "jdbc_dq"
result_table_name = "dq_result"
query = "select xxx,udf(ID) as CONDITION from jdbc_dq"
}
Sql {
source_table_name = "dq_result"
result_table_name = "success_to"
query = "select xxx from dq_result where CONDITION='true'"
}
Sql {
source_table_name = "dq_result"
result_table_name = "fail_to"
query = "select xxx from dq_result where CONDITION='false'"
}
}
sink {
Jdbc {
source_table_name = ["success_to"]
url="jdbc:oracle:thin:@//xxx.xxx.xxx.xxx:1521/HBI",
driver="oracle.jdbc.OracleDriver",
user="xxx",
password="xxx",
generate_sink_sql=true,
database="xxx",
table="xxx.xxx_1"
result_table_name="success_http"
}
Jdbc {
source_table_name = ["fail_to"]
url="jdbc:oracle:thin:@//xxx.xxx.xxx.xxx:1521/HBI",
driver="oracle.jdbc.OracleDriver",
user="xxx",
password="xxx",
generate_sink_sql=true,
database="xxx",
table="xxx.xxx_2"
result_table_name="fail_http"
}
}
```
### Running Command
```shell
run in org.apache.seatunnel.example.engine.SeaTunnelEngineExample
```
### Error Exception
```log
***********************************************
Job Statistic Information
***********************************************
Start Time : 2023-12-28 11:01:29
End Time : 2023-12-28 11:01:36
Total Time(s) : 7
Total Read Count : 8
Total Write Count : 5
Total Failed Count : 3
***********************************************
After the final task is completed, there are 5 pieces of data in the success table(xxx.xxx_1) and 3 pieces of data in the failure table(xxx.xxx_2), which do not match the statistics data
```
### Zeta or Flink or Spark Version
_No response_
### Java or Scala Version
_No response_
### Screenshots
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] [Statistic] The synchronization error count is incorrect [seatunnel]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #6102:
URL: https://github.com/apache/seatunnel/issues/6102#issuecomment-1926010812
This issue has been closed because it has not received response for too long time. You could reopen it if you encountered similar problems in the future.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] [Statistic] The synchronization error count is incorrect [seatunnel]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #6102: [Bug] [Statistic] The synchronization error count is incorrect
URL: https://github.com/apache/seatunnel/issues/6102
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] [Statistic] The synchronization error count is incorrect [seatunnel]
Posted by "Carl-Zhou-CN (via GitHub)" <gi...@apache.org>.
Carl-Zhou-CN commented on issue #6102:
URL: https://github.com/apache/seatunnel/issues/6102#issuecomment-1872112831
hi @EngonVHKxZ Can you be more specific? I can't find the anomaly?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] [Statistic] The synchronization error count is incorrect [seatunnel]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #6102:
URL: https://github.com/apache/seatunnel/issues/6102#issuecomment-1913778409
This issue has been automatically marked as stale because it has not had recent activity for 30 days. It will be closed in next 7 days if no further activity occurs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org