You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "Martijn Visser (Jira)" <ji...@apache.org> on 2022/06/27 11:57:00 UTC
[jira] [Created] (FLINK-28263) TPC-DS Bash e2e tests don't clean-up after completing
Martijn Visser created FLINK-28263:
--------------------------------------
Summary: TPC-DS Bash e2e tests don't clean-up after completing
Key: FLINK-28263
URL: https://issues.apache.org/jira/browse/FLINK-28263
Project: Flink
Issue Type: Bug
Components: Tests
Affects Versions: 1.16.0
Reporter: Martijn Visser
When debugging the disk space usage for the e2e tests, the top 20 folders with the largest file size are:
{code:java}
2022-06-27T09:32:59.8000587Z Jun 27 09:32:59 List top 20 directories with largest file size
2022-06-27T09:33:00.9811803Z Jun 27 09:33:00 4088524 .
2022-06-27T09:33:00.9813428Z Jun 27 09:33:00 1277080 ./flink-end-to-end-tests
2022-06-27T09:33:00.9814324Z Jun 27 09:33:00 624512 ./flink-dist
2022-06-27T09:33:00.9815152Z Jun 27 09:33:00 624124 ./flink-dist/target
2022-06-27T09:33:00.9816093Z Jun 27 09:33:00 500032 ./flink-dist/target/flink-1.16-SNAPSHOT-bin
2022-06-27T09:33:00.9817429Z Jun 27 09:33:00 500028 ./flink-dist/target/flink-1.16-SNAPSHOT-bin/flink-1.16-SNAPSHOT
2022-06-27T09:33:00.9818167Z Jun 27 09:33:00 486412 ./.git
2022-06-27T09:33:00.9819096Z Jun 27 09:33:00 479416 ./.git/objects
2022-06-27T09:33:00.9819512Z Jun 27 09:33:00 479408 ./.git/objects/pack
2022-06-27T09:33:00.9820584Z Jun 27 09:33:00 461456 ./flink-connectors
2022-06-27T09:33:00.9821403Z Jun 27 09:33:00 449832 ./.git/objects/pack/pack-0bdd9e3186d0cb404910c5843d19b5cb80b84fe0.pack
2022-06-27T09:33:00.9821992Z Jun 27 09:33:00 349236 ./flink-table
2022-06-27T09:33:00.9822631Z Jun 27 09:33:00 293008 ./flink-dist/target/flink-1.16-SNAPSHOT-bin/flink-1.16-SNAPSHOT/opt
2022-06-27T09:33:00.9823233Z Jun 27 09:33:00 251272 ./flink-filesystems
2022-06-27T09:33:00.9823818Z Jun 27 09:33:00 246588 ./flink-end-to-end-tests/flink-streaming-kinesis-test
2022-06-27T09:33:00.9824502Z Jun 27 09:33:00 246464 ./flink-end-to-end-tests/flink-streaming-kinesis-test/target
2022-06-27T09:33:00.9825210Z Jun 27 09:33:00 196656 ./flink-dist/target/flink-1.16-SNAPSHOT-bin/flink-1.16-SNAPSHOT/lib
2022-06-27T09:33:00.9825966Z Jun 27 09:33:00 184364 ./flink-end-to-end-tests/flink-streaming-kinesis-test/target/KinesisExample.jar
2022-06-27T09:33:00.9826652Z Jun 27 09:33:00 156136 ./flink-end-to-end-tests/flink-tpcds-test
2022-06-27T09:33:00.9827284Z Jun 27 09:33:00 151180 ./flink-end-to-end-tests/flink-tpcds-test/target
{code}
See https://dev.azure.com/martijn0323/Flink/_build/results?buildId=2732&view=logs&j=0e31ee24-31a6-528c-a4bf-45cde9b2a14e&t=ff03a8fa-e84e-5199-efb2-5433077ce8e2&l=5093
After running {{TPC-DS end-to-end test}} and after the clean-up, the following directories are listed in the top 20:
{code:java}
2022-06-27T09:49:51.7694429Z Jun 27 09:49:51 List top 20 directories with largest file size AFTER cleaning temorary folders and files
2022-06-27T09:49:52.9617221Z Jun 27 09:49:52 5315996 .
2022-06-27T09:49:52.9618830Z Jun 27 09:49:52 2504556 ./flink-end-to-end-tests
2022-06-27T09:49:52.9619848Z Jun 27 09:49:52 1383612 ./flink-end-to-end-tests/flink-tpcds-test
2022-06-27T09:49:52.9620796Z Jun 27 09:49:52 1378656 ./flink-end-to-end-tests/flink-tpcds-test/target
2022-06-27T09:49:52.9621730Z Jun 27 09:49:52 1223944 ./flink-end-to-end-tests/flink-tpcds-test/target/table
2022-06-27T09:49:52.9622844Z Jun 27 09:49:52 624508 ./flink-dist
2022-06-27T09:49:52.9623585Z Jun 27 09:49:52 624120 ./flink-dist/target
2022-06-27T09:49:52.9624398Z Jun 27 09:49:52 500028 ./flink-dist/target/flink-1.16-SNAPSHOT-bin
2022-06-27T09:49:52.9625366Z Jun 27 09:49:52 500024 ./flink-dist/target/flink-1.16-SNAPSHOT-bin/flink-1.16-SNAPSHOT
2022-06-27T09:49:52.9625994Z Jun 27 09:49:52 486412 ./.git
2022-06-27T09:49:52.9626514Z Jun 27 09:49:52 479416 ./.git/objects
2022-06-27T09:49:52.9631740Z Jun 27 09:49:52 479408 ./.git/objects/pack
2022-06-27T09:49:52.9632755Z Jun 27 09:49:52 461456 ./flink-connectors
2022-06-27T09:49:52.9633717Z Jun 27 09:49:52 449832 ./.git/objects/pack/pack-0bdd9e3186d0cb404910c5843d19b5cb80b84fe0.pack
2022-06-27T09:49:52.9634769Z Jun 27 09:49:52 379348 ./flink-end-to-end-tests/flink-tpcds-test/target/table/store_sales.dat
2022-06-27T09:49:52.9635596Z Jun 27 09:49:52 349236 ./flink-table
2022-06-27T09:49:52.9636489Z Jun 27 09:49:52 293008 ./flink-dist/target/flink-1.16-SNAPSHOT-bin/flink-1.16-SNAPSHOT/opt
2022-06-27T09:49:52.9637526Z Jun 27 09:49:52 288980 ./flink-end-to-end-tests/flink-tpcds-test/target/table/catalog_sales.dat
2022-06-27T09:49:52.9638378Z Jun 27 09:49:52 251272 ./flink-filesystems
2022-06-27T09:49:52.9639238Z Jun 27 09:49:52 246588 ./flink-end-to-end-tests/flink-streaming-kinesis-test
{code}
See https://dev.azure.com/martijn0323/Flink/_build/results?buildId=2732&view=logs&j=0e31ee24-31a6-528c-a4bf-45cde9b2a14e&t=ff03a8fa-e84e-5199-efb2-5433077ce8e2&l=5708
This results in not enough disk space errors during various runs further downstream. This test should also properly clean-up its files
--
This message was sent by Atlassian Jira
(v8.20.7#820007)