You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@seatunnel.apache.org by "WilliamTan778 (via GitHub)" <gi...@apache.org> on 2023/06/14 01:56:32 UTC
[GitHub] [seatunnel] WilliamTan778 opened a new issue, #4920: [Bug] [hive-sink] 2PC Task failure processing logic, class HiveSinkAggregatedCommitter method abort
WilliamTan778 opened a new issue, #4920:
URL: https://github.com/apache/seatunnel/issues/4920
### Search before asking
- [X] I had searched in the [issues](https://github.com/apache/seatunnel/issues?q=is%3Aissue+label%3A%22bug%22) and found no similar issues.
### What happened
<img width="828" alt="image" src="https://github.com/apache/seatunnel/assets/41814775/41e9c0de-d0d4-4549-b66e-0737f08812ed">
dropPartitions Although the data has not been deleted, it may cause the previous partition to no longer exist and the business cannot use it. Should the partition not be deleted
### SeaTunnel Version
2.3.1
### SeaTunnel Config
```conf
env {
execution.parallelism = 3
job.name="test_hive_source_to_hive"
}
source {
Hive {
result_table_name = "fake"
table_name = "default.test_hive_source"
metastore_uri = "thrift://172.30.1.185:9083"
}
}
transform {
Sql {
source_table_name = "fake"
result_table_name = "fake1"
query = "select test_tinyint,test_smallint,test_int,test_bigint, test_boolean,test_float,test_double,test_string, test_binary,test_timestamp,test_decimal,test_char,test_varchar,test_date, test_par1 as test_par2,test_par2 as test_par1 from fake"
}
}
sink {
Hive {
source_table_name = "fake1"
table_name = "default.test_hive_sink_text_simple1"
metastore_uri = "thrift://172.30.1.185:9083"
}
}
```
### Running Command
```shell
./bin/start-seatunnel-spark-3-connector-v2.sh \
--master "local[*]" \
--deploy-mode client \
--config test/hive.conf
```
### Error Exception
```log
no error, Logical processing problems
```
### Flink or Spark Version
spark-3.3.0-bin-hadoop2
### Java or Scala Version
Corretto-11.0.19.7.1
### Screenshots
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [seatunnel] github-actions[bot] commented on issue #4920: [Bug] [hive-sink] 2PC Task failure processing logic, class HiveSinkAggregatedCommitter method abort
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #4920:
URL: https://github.com/apache/seatunnel/issues/4920#issuecomment-1639109002
This issue has been automatically marked as stale because it has not had recent activity for 30 days. It will be closed in next 7 days if no further activity occurs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [seatunnel] EricJoy2048 commented on issue #4920: [Bug] [hive-sink] 2PC Task failure processing logic, class HiveSinkAggregatedCommitter method abort
Posted by "EricJoy2048 (via GitHub)" <gi...@apache.org>.
EricJoy2048 commented on issue #4920:
URL: https://github.com/apache/seatunnel/issues/4920#issuecomment-1593316787
Yeah, Drop partition is not a good way. Some times people may only append data to exists partitions. If we drop the partition, may delete the data file not write by this job.
Can you update the code and fix it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [seatunnel] zhilinli123 commented on issue #4920: [Bug] [hive-sink] 2PC Task failure processing logic, class HiveSinkAggregatedCommitter method abort
Posted by "zhilinli123 (via GitHub)" <gi...@apache.org>.
zhilinli123 commented on issue #4920:
URL: https://github.com/apache/seatunnel/issues/4920#issuecomment-1590991641
> +1 Delete a partition will put all the files under the partition into the recycle bin. This action is really heavy and there is a risk of data loss.
Currently, the underlying data information is not deleted, only the metadata information of the partition is deleted, but I do not understand why only the metadata is deleted, what is the purpose of it? PATL: @TyrantLucifer
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [seatunnel] lightzhao commented on issue #4920: [Bug] [hive-sink] 2PC Task failure processing logic, class HiveSinkAggregatedCommitter method abort
Posted by "lightzhao (via GitHub)" <gi...@apache.org>.
lightzhao commented on issue #4920:
URL: https://github.com/apache/seatunnel/issues/4920#issuecomment-1594029969
> Yeah, Drop partition is not a good way. Some times people may only append data to exists partitions. If we drop the partition, may delete the data file not write by this job.
ok, I try to fix it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [seatunnel] zhilinli123 commented on issue #4920: [Bug] [hive-sink] 2PC Task failure processing logic, class HiveSinkAggregatedCommitter method abort
Posted by "zhilinli123 (via GitHub)" <gi...@apache.org>.
zhilinli123 commented on issue #4920:
URL: https://github.com/apache/seatunnel/issues/4920#issuecomment-1590991222
>
Currently, the underlying data information is not deleted, only the metadata information of the partition is deleted, but I do not understand why only the metadata is deleted, what is the purpose of it? PATL: @TyrantLucifer
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [seatunnel] TyrantLucifer commented on issue #4920: [Bug] [hive-sink] 2PC Task failure processing logic, class HiveSinkAggregatedCommitter method abort
Posted by "TyrantLucifer (via GitHub)" <gi...@apache.org>.
TyrantLucifer commented on issue #4920:
URL: https://github.com/apache/seatunnel/issues/4920#issuecomment-1595787612
IMO,when user write new partitions this action is useful,2pc has all the lifecycle,add a parameter to control it is better,not remove it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [seatunnel] lightzhao commented on issue #4920: [Bug] [hive-sink] 2PC Task failure processing logic, class HiveSinkAggregatedCommitter method abort
Posted by "lightzhao (via GitHub)" <gi...@apache.org>.
lightzhao commented on issue #4920:
URL: https://github.com/apache/seatunnel/issues/4920#issuecomment-1590901882
+1
Delete a partition will put all the files under the partition into the recycle bin. This action is really heavy and there is a risk of data loss.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [seatunnel] lightzhao commented on issue #4920: [Bug] [hive-sink] 2PC Task failure processing logic, class HiveSinkAggregatedCommitter method abort
Posted by "lightzhao (via GitHub)" <gi...@apache.org>.
lightzhao commented on issue #4920:
URL: https://github.com/apache/seatunnel/issues/4920#issuecomment-1592228352
> > +1 Delete a partition will put all the files under the partition into the recycle bin. This action is really heavy and there is a risk of data loss.
>
> Currently, the underlying data information is not deleted, only the metadata information of the partition is deleted, but I do not understand why only the metadata is deleted, what is the purpose of it? PATL: @TyrantLucifer
The dropPartition method will move data to the trash, not just metadata.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [seatunnel] github-actions[bot] commented on issue #4920: [Bug] [hive-sink] 2PC Task failure processing logic, class HiveSinkAggregatedCommitter method abort
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #4920:
URL: https://github.com/apache/seatunnel/issues/4920#issuecomment-1650745006
This issue has been closed because it has not received response for too long time. You could reopen it if you encountered similar problems in the future.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [seatunnel] github-actions[bot] closed issue #4920: [Bug] [hive-sink] 2PC Task failure processing logic, class HiveSinkAggregatedCommitter method abort
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #4920: [Bug] [hive-sink] 2PC Task failure processing logic, class HiveSinkAggregatedCommitter method abort
URL: https://github.com/apache/seatunnel/issues/4920
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [seatunnel] zhilinli123 commented on issue #4920: [Bug] [hive-sink] 2PC Task failure processing logic, class HiveSinkAggregatedCommitter method abort
Posted by "zhilinli123 (via GitHub)" <gi...@apache.org>.
zhilinli123 commented on issue #4920:
URL: https://github.com/apache/seatunnel/issues/4920#issuecomment-1590711170
PTAL: @TyrantLucifer
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org