You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2020/10/14 04:24:52 UTC
[GitHub] [hudi] halkar opened a new issue #2174: [SUPPORT] Auto-clean doesn't work
halkar opened a new issue #2174:
URL: https://github.com/apache/hudi/issues/2174
I'm trying to use Hudi with Spark EMR. Everything is ok when I run a batch job of S3 data. But when I run it of Kinesis stream it creates tens of versions of the output file and never removes them.
**To Reproduce**
This is my code
```
val hudiOptions = Map[String,String](
DataSourceWriteOptions.TABLE_TYPE_OPT_KEY -> DataSourceWriteOptions.COW_TABLE_TYPE_OPT_VAL,
DataSourceWriteOptions.OPERATION_OPT_KEY -> DataSourceWriteOptions.UPSERT_OPERATION_OPT_VAL,
HoodieStorageConfig.PARQUET_COMPRESSION_CODEC -> "snappy",
HoodieCompactionConfig.AUTO_CLEAN_PROP -> "true",
HoodieCompactionConfig.CLEANER_FILE_VERSIONS_RETAINED_PROP -> "1",
HoodieCompactionConfig.CLEANER_COMMITS_RETAINED_PROP -> "1",
DataSourceWriteOptions.HIVE_STYLE_PARTITIONING_OPT_KEY -> "true",
"hoodie.upsert.shuffle.parallelism" -> "5",
HoodieCompactionConfig.PARQUET_SMALL_FILE_LIMIT_BYTES -> (512 * 1024 * 1024).toString,
"hoodie.combine.before.insert" -> "true",
DataSourceWriteOptions.INSERT_DROP_DUPS_OPT_KEY -> "true"
)
dataframe
.write
.format("org.apache.hudi")
.options(hudiOptions)
.option(HoodieWriteConfig.TABLE_NAME, tableName)
.option(DataSourceWriteOptions.TABLE_NAME_OPT_KEY, tableName)
.option(DataSourceWriteOptions.PARTITIONPATH_FIELD_OPT_KEY, partitionPathField)
.option(DataSourceWriteOptions.RECORDKEY_FIELD_OPT_KEY, recordKeyField)
.option(DataSourceWriteOptions.PRECOMBINE_FIELD_OPT_KEY, precombineFieldKey)
.mode(SaveMode.Append)
.save(destinationPath)
```
**Environment Description**
* Hudi version: 0.6.0
* Spark version: 2.4.4
* Hive version: 2.3.7
* Hadoop version: 2.10
* Storage (HDFS/S3/GCS..): S3
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] halkar commented on issue #2174: [SUPPORT] Auto-clean doesn't work
Posted by GitBox <gi...@apache.org>.
halkar commented on issue #2174:
URL: https://github.com/apache/hudi/issues/2174#issuecomment-709685927
Cleans:
<img width="516" alt="Screen Shot 2020-10-16 at 13 05 39" src="https://user-images.githubusercontent.com/1992396/96204557-520a4e80-0fb0-11eb-8205-753b60adea75.png">
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] bvaradar commented on issue #2174: [SUPPORT] Auto-clean doesn't work
Posted by GitBox <gi...@apache.org>.
bvaradar commented on issue #2174:
URL: https://github.com/apache/hudi/issues/2174#issuecomment-709864864
@halkar : Yes, https://issues.apache.org/jira/browse/HUDI-845 tracks it
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] bvaradar commented on issue #2174: [SUPPORT] Auto-clean doesn't work
Posted by GitBox <gi...@apache.org>.
bvaradar commented on issue #2174:
URL: https://github.com/apache/hudi/issues/2174#issuecomment-709442307
@halkar : THanks for the information. Yes, this is not expected. We would have to see whether cleaning operations is succeeding. We have a cli "cleans show" that lists the cleans that have successfully happened and "clean showpartitions". Can you run the above commands.
Can you list ".hoodie" folder in the same way you listed the partitions.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] halkar commented on issue #2174: [SUPPORT] Auto-clean doesn't work
Posted by GitBox <gi...@apache.org>.
halkar commented on issue #2174:
URL: https://github.com/apache/hudi/issues/2174#issuecomment-709655075
Is it possible that the problem is caused by two versions of the app running in parallel? I also noticed that writing hoodie files sometimes hangs.
<img width="1407" alt="Screen Shot 2020-10-15 at 21 26 25" src="https://user-images.githubusercontent.com/1992396/96198576-fedccf80-0fa0-11eb-9e9c-c347dbe274ae.png">
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] halkar edited a comment on issue #2174: [SUPPORT] Auto-clean doesn't work
Posted by GitBox <gi...@apache.org>.
halkar edited a comment on issue #2174:
URL: https://github.com/apache/hudi/issues/2174#issuecomment-709654557
This is a different run. I'll add `cleans show` results later.
All partitions:
```
staff 5984 16 Oct 12:02 .hoodie
staff 224 16 Oct 12:02 graphversion=782
staff 192 16 Oct 12:02 graphversion=783
staff 128 16 Oct 12:02 graphversion=786
staff 160 16 Oct 12:02 graphversion=788
```
.hoodie folder (not all of it)
```
staff 969 16 Oct 09:16 20201015221614.rollback
staff 0 16 Oct 09:16 20201015221614.rollback.inflight
staff 1043 16 Oct 09:17 20201015221713.rollback
staff 0 16 Oct 09:17 20201015221713.rollback.inflight
staff 1043 16 Oct 09:18 20201015221834.rollback
staff 0 16 Oct 09:18 20201015221834.rollback.inflight
staff 1206 16 Oct 09:19 20201015221941.rollback
staff 0 16 Oct 09:19 20201015221941.rollback.inflight
staff 1043 16 Oct 09:20 20201015222038.rollback
staff 0 16 Oct 09:20 20201015222038.rollback.inflight
staff 1043 16 Oct 09:21 20201015222129.rollback
staff 0 16 Oct 09:21 20201015222129.rollback.inflight
staff 969 16 Oct 09:49 20201015224949.rollback
staff 0 16 Oct 09:49 20201015224949.rollback.inflight
staff 1079 16 Oct 10:49 20201015234929.rollback
staff 0 16 Oct 10:49 20201015234929.rollback.inflight
staff 1079 16 Oct 10:50 20201015235034.rollback
staff 0 16 Oct 10:50 20201015235034.rollback.inflight
staff 969 16 Oct 10:52 20201015235220.rollback
staff 0 16 Oct 10:52 20201015235220.rollback.inflight
staff 969 16 Oct 10:53 20201015235319.rollback
staff 0 16 Oct 10:53 20201015235319.rollback.inflight
staff 1079 16 Oct 10:54 20201015235414.rollback
staff 0 16 Oct 10:54 20201015235414.rollback.inflight
staff 969 16 Oct 10:55 20201015235515.rollback
staff 0 16 Oct 10:55 20201015235515.rollback.inflight
staff 1501 16 Oct 11:06 20201016000617.clean
staff 1496 16 Oct 11:06 20201016000617.clean.inflight
staff 1496 16 Oct 11:06 20201016000617.clean.requested
staff 1501 16 Oct 11:07 20201016000716.clean
staff 1496 16 Oct 11:07 20201016000716.clean.inflight
staff 1496 16 Oct 11:07 20201016000716.clean.requested
staff 1501 16 Oct 11:08 20201016000816.clean
staff 1496 16 Oct 11:08 20201016000816.clean.inflight
staff 1496 16 Oct 11:08 20201016000816.clean.requested
staff 1501 16 Oct 11:09 20201016000915.clean
staff 1496 16 Oct 11:09 20201016000915.clean.inflight
staff 1496 16 Oct 11:09 20201016000915.clean.requested
staff 1501 16 Oct 11:10 20201016001015.clean
staff 1496 16 Oct 11:10 20201016001015.clean.inflight
staff 1496 16 Oct 11:10 20201016001015.clean.requested
staff 1501 16 Oct 11:11 20201016001117.clean
staff 1496 16 Oct 11:11 20201016001117.clean.inflight
staff 1496 16 Oct 11:11 20201016001117.clean.requested
staff 1503 16 Oct 11:12 20201016001222.clean
staff 1497 16 Oct 11:12 20201016001222.clean.inflight
staff 1497 16 Oct 11:12 20201016001222.clean.requested
staff 2318 16 Oct 11:16 20201016001619.commit
staff 0 16 Oct 11:16 20201016001619.commit.requested
staff 986 16 Oct 11:16 20201016001619.inflight
staff 1655 16 Oct 11:17 20201016001718.clean
staff 1660 16 Oct 11:17 20201016001718.clean.inflight
staff 1660 16 Oct 11:17 20201016001718.clean.requested
staff 2318 16 Oct 11:17 20201016001718.commit
staff 0 16 Oct 11:17 20201016001718.commit.requested
staff 986 16 Oct 11:17 20201016001718.inflight
staff 1503 16 Oct 11:18 20201016001818.clean
staff 1497 16 Oct 11:18 20201016001818.clean.inflight
staff 1497 16 Oct 11:18 20201016001818.clean.requested
staff 2317 16 Oct 11:18 20201016001818.commit
staff 0 16 Oct 11:18 20201016001818.commit.requested
staff 985 16 Oct 11:18 20201016001818.inflight
staff 1503 16 Oct 11:20 20201016001919.clean
staff 1497 16 Oct 11:20 20201016001919.clean.inflight
staff 1497 16 Oct 11:20 20201016001919.clean.requested
staff 2317 16 Oct 11:19 20201016001919.commit
staff 0 16 Oct 11:19 20201016001919.commit.requested
staff 985 16 Oct 11:19 20201016001919.inflight
staff 1503 16 Oct 11:21 20201016002025.clean
staff 1497 16 Oct 11:21 20201016002025.clean.inflight
staff 1497 16 Oct 11:21 20201016002025.clean.requested
staff 2318 16 Oct 11:20 20201016002025.commit
staff 0 16 Oct 11:20 20201016002025.commit.requested
staff 986 16 Oct 11:20 20201016002025.inflight
staff 1503 16 Oct 11:21 20201016002124.clean
staff 1497 16 Oct 11:21 20201016002124.clean.inflight
staff 1497 16 Oct 11:21 20201016002124.clean.requested
staff 2317 16 Oct 11:21 20201016002124.commit
staff 0 16 Oct 11:21 20201016002124.commit.requested
staff 985 16 Oct 11:21 20201016002124.inflight
staff 1503 16 Oct 11:22 20201016002215.clean
staff 1497 16 Oct 11:22 20201016002215.clean.inflight
staff 1497 16 Oct 11:22 20201016002215.clean.requested
staff 2317 16 Oct 11:22 20201016002215.commit
staff 0 16 Oct 11:22 20201016002215.commit.requested
staff 985 16 Oct 11:22 20201016002215.inflight
staff 1503 16 Oct 11:23 20201016002316.clean
staff 1497 16 Oct 11:23 20201016002316.clean.inflight
staff 1497 16 Oct 11:23 20201016002316.clean.requested
staff 2317 16 Oct 11:23 20201016002316.commit
staff 0 16 Oct 11:23 20201016002316.commit.requested
staff 985 16 Oct 11:23 20201016002316.inflight
staff 1503 16 Oct 11:24 20201016002416.clean
staff 1497 16 Oct 11:24 20201016002416.clean.inflight
staff 1497 16 Oct 11:24 20201016002416.clean.requested
staff 1413 16 Oct 11:24 20201016002416.commit
staff 0 16 Oct 11:24 20201016002416.commit.requested
staff 380 16 Oct 11:24 20201016002416.inflight
staff 2319 16 Oct 11:25 20201016002516.commit
staff 0 16 Oct 11:25 20201016002516.commit.requested
staff 985 16 Oct 11:25 20201016002516.inflight
staff 1503 16 Oct 11:26 20201016002619.clean
staff 1497 16 Oct 11:26 20201016002619.clean.inflight
staff 1497 16 Oct 11:26 20201016002619.clean.requested
staff 1413 16 Oct 11:26 20201016002619.commit
staff 0 16 Oct 11:26 20201016002619.commit.requested
staff 380 16 Oct 11:26 20201016002619.inflight
staff 1413 16 Oct 11:27 20201016002723.commit
staff 0 16 Oct 11:27 20201016002723.commit.requested
staff 380 16 Oct 11:27 20201016002723.inflight
staff 1413 16 Oct 11:28 20201016002825.commit
staff 0 16 Oct 11:28 20201016002825.commit.requested
staff 380 16 Oct 11:28 20201016002825.inflight
staff 1413 16 Oct 11:29 20201016002917.commit
staff 0 16 Oct 11:29 20201016002917.commit.requested
staff 380 16 Oct 11:29 20201016002917.inflight
staff 2319 16 Oct 11:30 20201016003004.commit
staff 0 16 Oct 11:30 20201016003004.commit.requested
staff 985 16 Oct 11:30 20201016003004.inflight
staff 1505 16 Oct 11:31 20201016003106.clean
staff 1498 16 Oct 11:31 20201016003106.clean.inflight
staff 1498 16 Oct 11:31 20201016003106.clean.requested
staff 2319 16 Oct 11:31 20201016003106.commit
staff 0 16 Oct 11:31 20201016003106.commit.requested
staff 985 16 Oct 11:31 20201016003106.inflight
staff 1505 16 Oct 11:34 20201016003338.clean
staff 1498 16 Oct 11:34 20201016003338.clean.inflight
staff 1498 16 Oct 11:34 20201016003338.clean.requested
staff 2319 16 Oct 11:34 20201016003338.commit
staff 0 16 Oct 11:33 20201016003338.commit.requested
staff 985 16 Oct 11:33 20201016003338.inflight
staff 1505 16 Oct 11:37 20201016003648.clean
staff 1498 16 Oct 11:37 20201016003648.clean.inflight
staff 1498 16 Oct 11:37 20201016003648.clean.requested
staff 1413 16 Oct 11:37 20201016003648.commit
staff 0 16 Oct 11:36 20201016003648.commit.requested
staff 380 16 Oct 11:37 20201016003648.inflight
staff 1413 16 Oct 11:37 20201016003733.commit
staff 0 16 Oct 11:37 20201016003733.commit.requested
staff 380 16 Oct 11:37 20201016003733.inflight
staff 2320 16 Oct 11:40 20201016004005.commit
staff 0 16 Oct 11:40 20201016004005.commit.requested
staff 986 16 Oct 11:40 20201016004005.inflight
staff 1505 16 Oct 11:41 20201016004106.clean
staff 1498 16 Oct 11:41 20201016004106.clean.inflight
staff 1498 16 Oct 11:41 20201016004106.clean.requested
staff 2319 16 Oct 11:41 20201016004106.commit
staff 0 16 Oct 11:41 20201016004106.commit.requested
staff 985 16 Oct 11:41 20201016004106.inflight
staff 1505 16 Oct 11:44 20201016004405.clean
staff 1498 16 Oct 11:44 20201016004405.clean.inflight
staff 1498 16 Oct 11:44 20201016004405.clean.requested
staff 2319 16 Oct 11:44 20201016004405.commit
staff 0 16 Oct 11:44 20201016004405.commit.requested
staff 985 16 Oct 11:44 20201016004405.inflight
staff 1505 16 Oct 11:50 20201016005005.clean
staff 1498 16 Oct 11:50 20201016005005.clean.inflight
staff 1498 16 Oct 11:50 20201016005005.clean.requested
staff 2319 16 Oct 11:50 20201016005005.commit
staff 0 16 Oct 11:50 20201016005005.commit.requested
staff 985 16 Oct 11:50 20201016005005.inflight
staff 1505 16 Oct 11:51 20201016005106.clean
staff 1498 16 Oct 11:51 20201016005106.clean.inflight
staff 1498 16 Oct 11:51 20201016005106.clean.requested
staff 2319 16 Oct 11:51 20201016005106.commit
staff 0 16 Oct 11:51 20201016005106.commit.requested
staff 985 16 Oct 11:51 20201016005106.inflight
staff 1505 16 Oct 11:54 20201016005342.clean
staff 1498 16 Oct 11:54 20201016005342.clean.inflight
staff 1498 16 Oct 11:54 20201016005342.clean.requested
staff 2305 16 Oct 11:54 20201016005342.commit
staff 0 16 Oct 11:53 20201016005342.commit.requested
staff 989 16 Oct 11:54 20201016005342.inflight
staff 2312 16 Oct 11:59 20201016005831.commit
staff 0 16 Oct 11:58 20201016005831.commit.requested
staff 986 16 Oct 11:58 20201016005831.inflight
staff 1497 16 Oct 12:00 20201016010004.clean
staff 1494 16 Oct 12:00 20201016010004.clean.inflight
staff 1494 16 Oct 12:00 20201016010004.clean.requested
staff 2311 16 Oct 12:00 20201016010004.commit
staff 0 16 Oct 12:00 20201016010004.commit.requested
staff 985 16 Oct 12:00 20201016010004.inflight
drwxr-xr-x 24 artur staff 768 16 Oct 12:02 archived
staff 231 16 Oct 08:31 hoodie.properties
```
Partitions:
graphversion=782:
```
staff 93 16 Oct 08:32 .hoodie_partition_metadata
staff 5447259 16 Oct 09:16 efbbb2eb-d244-463d-bf64-55d0c46c057d-0_0-10154-151923_20201015221611.parquet
staff 5442031 16 Oct 09:18 efbbb2eb-d244-463d-bf64-55d0c46c057d-0_0-10800-159936_20201015221814.parquet
staff 5442099 16 Oct 09:20 efbbb2eb-d244-463d-bf64-55d0c46c057d-0_0-11245-169414_20201015222013.parquet
staff 5442001 16 Oct 09:21 efbbb2eb-d244-463d-bf64-55d0c46c057d-0_0-11481-174108_20201015222114.parquet
```
graphversion=783:
```
staff 93 16 Oct 09:11 .hoodie_partition_metadata
staff 5519741 16 Oct 10:49 125ca82f-911b-4830-86d4-2290111cc55a-0_0-25341-397081_20201015234912.parquet
staff 5512472 16 Oct 10:50 125ca82f-911b-4830-86d4-2290111cc55a-0_0-25658-401934_20201015235010.parquet
staff 5514534 16 Oct 10:51 125ca82f-911b-4830-86d4-2290111cc55a-0_0-25898-406769_20201015235111.parquet
```
graphversion=786:
```
staff 93 16 Oct 10:44 .hoodie_partition_metadata
staff 5723412 16 Oct 11:51 bb9e869a-edab-4515-92e4-e99f098f88e1-0_0-13936-223389_20201016005106.parquet
```
graphversion=788:
```
staff 93 16 Oct 11:54 .hoodie_partition_metadata
staff 5806059 16 Oct 11:59 be99cdb1-d096-4623-9767-4bb7156229c8-0_0-304-4887_20201016005831.parquet
staff 5807720 16 Oct 12:00 be99cdb1-d096-4623-9767-4bb7156229c8-0_0-629-9669_20201016010004.parquet
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] bvaradar closed issue #2174: [SUPPORT] Auto-clean doesn't work
Posted by GitBox <gi...@apache.org>.
bvaradar closed issue #2174:
URL: https://github.com/apache/hudi/issues/2174
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] halkar commented on issue #2174: [SUPPORT] Auto-clean doesn't work
Posted by GitBox <gi...@apache.org>.
halkar commented on issue #2174:
URL: https://github.com/apache/hudi/issues/2174#issuecomment-709830826
@bvaradar thanks for confirming. Are there any plans to support concurrent writes? I'll try to change the logic not do concurrent writes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] halkar edited a comment on issue #2174: [SUPPORT] Auto-clean doesn't work
Posted by GitBox <gi...@apache.org>.
halkar edited a comment on issue #2174:
URL: https://github.com/apache/hudi/issues/2174#issuecomment-709654557
This is a different run. I'll add `cleans show` results later.
All partitions:
<img width="1315" alt="Screen Shot 2020-10-16 at 11 06 57" src="https://user-images.githubusercontent.com/1992396/96198465-9db4fc00-0fa0-11eb-80ba-c951a64fd721.png">
.hoodie folder (not all of it)
<img width="1398" alt="Screen Shot 2020-10-16 at 11 08 08" src="https://user-images.githubusercontent.com/1992396/96198484-aefe0880-0fa0-11eb-819e-b0c310fe2235.png">
<img width="1380" alt="Screen Shot 2020-10-16 at 11 08 51" src="https://user-images.githubusercontent.com/1992396/96198490-b3c2bc80-0fa0-11eb-838e-4cf0b558a7e6.png">
Three partitions (the third one is empty - deployment failed).
<img width="1308" alt="Screen Shot 2020-10-16 at 11 09 16" src="https://user-images.githubusercontent.com/1992396/96198515-c76e2300-0fa0-11eb-85e8-6602e9b4930d.png">
<img width="1301" alt="Screen Shot 2020-10-16 at 11 09 23" src="https://user-images.githubusercontent.com/1992396/96198521-ca691380-0fa0-11eb-9030-ed5a8af1df4b.png">
<img width="1304" alt="Screen Shot 2020-10-16 at 11 28 06" src="https://user-images.githubusercontent.com/1992396/96199212-baeaca00-0fa2-11eb-9565-154dacd9035f.png">
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] bvaradar edited a comment on issue #2174: [SUPPORT] Auto-clean doesn't work
Posted by GitBox <gi...@apache.org>.
bvaradar edited a comment on issue #2174:
URL: https://github.com/apache/hudi/issues/2174#issuecomment-709702300
@halkar : Do you mean you are writing to the same dataset concurrently ? This is not supported and you need to be ingesting to the dataset by one write a time. Can you rerun with only one write() at a time and see if cleaner is cleaning the old file versions.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] bvaradar commented on issue #2174: [SUPPORT] Auto-clean doesn't work
Posted by GitBox <gi...@apache.org>.
bvaradar commented on issue #2174:
URL: https://github.com/apache/hudi/issues/2174#issuecomment-709702300
@halkar : Do you mean you are writing to the same dataset concurrently ? This is not supported and you need to be ingesting to the dataset by one write a time.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] halkar edited a comment on issue #2174: [SUPPORT] Auto-clean doesn't work
Posted by GitBox <gi...@apache.org>.
halkar edited a comment on issue #2174:
URL: https://github.com/apache/hudi/issues/2174#issuecomment-709654557
This is a different run. I'll add `cleans show` results later.
All partitions:
```
drwxr-xr-x 187 artur staff 5984 16 Oct 12:02 .hoodie
drwxr-xr-x 7 artur staff 224 16 Oct 12:02 graphversion=782
drwxr-xr-x 6 artur staff 192 16 Oct 12:02 graphversion=783
drwxr-xr-x 4 artur staff 128 16 Oct 12:02 graphversion=786
drwxr-xr-x 5 artur staff 160 16 Oct 12:02 graphversion=788
```
.hoodie folder (not all of it)
```
-rw-r--r-- 1 artur staff 969 16 Oct 09:16 20201015221614.rollback
-rw-r--r-- 1 artur staff 0 16 Oct 09:16 20201015221614.rollback.inflight
-rw-r--r-- 1 artur staff 1043 16 Oct 09:17 20201015221713.rollback
-rw-r--r-- 1 artur staff 0 16 Oct 09:17 20201015221713.rollback.inflight
-rw-r--r-- 1 artur staff 1043 16 Oct 09:18 20201015221834.rollback
-rw-r--r-- 1 artur staff 0 16 Oct 09:18 20201015221834.rollback.inflight
-rw-r--r-- 1 artur staff 1206 16 Oct 09:19 20201015221941.rollback
-rw-r--r-- 1 artur staff 0 16 Oct 09:19 20201015221941.rollback.inflight
-rw-r--r-- 1 artur staff 1043 16 Oct 09:20 20201015222038.rollback
-rw-r--r-- 1 artur staff 0 16 Oct 09:20 20201015222038.rollback.inflight
-rw-r--r-- 1 artur staff 1043 16 Oct 09:21 20201015222129.rollback
-rw-r--r-- 1 artur staff 0 16 Oct 09:21 20201015222129.rollback.inflight
-rw-r--r-- 1 artur staff 969 16 Oct 09:49 20201015224949.rollback
-rw-r--r-- 1 artur staff 0 16 Oct 09:49 20201015224949.rollback.inflight
-rw-r--r-- 1 artur staff 1079 16 Oct 10:49 20201015234929.rollback
-rw-r--r-- 1 artur staff 0 16 Oct 10:49 20201015234929.rollback.inflight
-rw-r--r-- 1 artur staff 1079 16 Oct 10:50 20201015235034.rollback
-rw-r--r-- 1 artur staff 0 16 Oct 10:50 20201015235034.rollback.inflight
-rw-r--r-- 1 artur staff 969 16 Oct 10:52 20201015235220.rollback
-rw-r--r-- 1 artur staff 0 16 Oct 10:52 20201015235220.rollback.inflight
-rw-r--r-- 1 artur staff 969 16 Oct 10:53 20201015235319.rollback
-rw-r--r-- 1 artur staff 0 16 Oct 10:53 20201015235319.rollback.inflight
-rw-r--r-- 1 artur staff 1079 16 Oct 10:54 20201015235414.rollback
-rw-r--r-- 1 artur staff 0 16 Oct 10:54 20201015235414.rollback.inflight
-rw-r--r-- 1 artur staff 969 16 Oct 10:55 20201015235515.rollback
-rw-r--r-- 1 artur staff 0 16 Oct 10:55 20201015235515.rollback.inflight
-rw-r--r-- 1 artur staff 1501 16 Oct 11:06 20201016000617.clean
-rw-r--r-- 1 artur staff 1496 16 Oct 11:06 20201016000617.clean.inflight
-rw-r--r-- 1 artur staff 1496 16 Oct 11:06 20201016000617.clean.requested
-rw-r--r-- 1 artur staff 1501 16 Oct 11:07 20201016000716.clean
-rw-r--r-- 1 artur staff 1496 16 Oct 11:07 20201016000716.clean.inflight
-rw-r--r-- 1 artur staff 1496 16 Oct 11:07 20201016000716.clean.requested
-rw-r--r-- 1 artur staff 1501 16 Oct 11:08 20201016000816.clean
-rw-r--r-- 1 artur staff 1496 16 Oct 11:08 20201016000816.clean.inflight
-rw-r--r-- 1 artur staff 1496 16 Oct 11:08 20201016000816.clean.requested
-rw-r--r-- 1 artur staff 1501 16 Oct 11:09 20201016000915.clean
-rw-r--r-- 1 artur staff 1496 16 Oct 11:09 20201016000915.clean.inflight
-rw-r--r-- 1 artur staff 1496 16 Oct 11:09 20201016000915.clean.requested
-rw-r--r-- 1 artur staff 1501 16 Oct 11:10 20201016001015.clean
-rw-r--r-- 1 artur staff 1496 16 Oct 11:10 20201016001015.clean.inflight
-rw-r--r-- 1 artur staff 1496 16 Oct 11:10 20201016001015.clean.requested
-rw-r--r-- 1 artur staff 1501 16 Oct 11:11 20201016001117.clean
-rw-r--r-- 1 artur staff 1496 16 Oct 11:11 20201016001117.clean.inflight
-rw-r--r-- 1 artur staff 1496 16 Oct 11:11 20201016001117.clean.requested
-rw-r--r-- 1 artur staff 1503 16 Oct 11:12 20201016001222.clean
-rw-r--r-- 1 artur staff 1497 16 Oct 11:12 20201016001222.clean.inflight
-rw-r--r-- 1 artur staff 1497 16 Oct 11:12 20201016001222.clean.requested
-rw-r--r-- 1 artur staff 2318 16 Oct 11:16 20201016001619.commit
-rw-r--r-- 1 artur staff 0 16 Oct 11:16 20201016001619.commit.requested
-rw-r--r-- 1 artur staff 986 16 Oct 11:16 20201016001619.inflight
-rw-r--r-- 1 artur staff 1655 16 Oct 11:17 20201016001718.clean
-rw-r--r-- 1 artur staff 1660 16 Oct 11:17 20201016001718.clean.inflight
-rw-r--r-- 1 artur staff 1660 16 Oct 11:17 20201016001718.clean.requested
-rw-r--r-- 1 artur staff 2318 16 Oct 11:17 20201016001718.commit
-rw-r--r-- 1 artur staff 0 16 Oct 11:17 20201016001718.commit.requested
-rw-r--r-- 1 artur staff 986 16 Oct 11:17 20201016001718.inflight
-rw-r--r-- 1 artur staff 1503 16 Oct 11:18 20201016001818.clean
-rw-r--r-- 1 artur staff 1497 16 Oct 11:18 20201016001818.clean.inflight
-rw-r--r-- 1 artur staff 1497 16 Oct 11:18 20201016001818.clean.requested
-rw-r--r-- 1 artur staff 2317 16 Oct 11:18 20201016001818.commit
-rw-r--r-- 1 artur staff 0 16 Oct 11:18 20201016001818.commit.requested
-rw-r--r-- 1 artur staff 985 16 Oct 11:18 20201016001818.inflight
-rw-r--r-- 1 artur staff 1503 16 Oct 11:20 20201016001919.clean
-rw-r--r-- 1 artur staff 1497 16 Oct 11:20 20201016001919.clean.inflight
-rw-r--r-- 1 artur staff 1497 16 Oct 11:20 20201016001919.clean.requested
-rw-r--r-- 1 artur staff 2317 16 Oct 11:19 20201016001919.commit
-rw-r--r-- 1 artur staff 0 16 Oct 11:19 20201016001919.commit.requested
-rw-r--r-- 1 artur staff 985 16 Oct 11:19 20201016001919.inflight
-rw-r--r-- 1 artur staff 1503 16 Oct 11:21 20201016002025.clean
-rw-r--r-- 1 artur staff 1497 16 Oct 11:21 20201016002025.clean.inflight
-rw-r--r-- 1 artur staff 1497 16 Oct 11:21 20201016002025.clean.requested
-rw-r--r-- 1 artur staff 2318 16 Oct 11:20 20201016002025.commit
-rw-r--r-- 1 artur staff 0 16 Oct 11:20 20201016002025.commit.requested
-rw-r--r-- 1 artur staff 986 16 Oct 11:20 20201016002025.inflight
-rw-r--r-- 1 artur staff 1503 16 Oct 11:21 20201016002124.clean
-rw-r--r-- 1 artur staff 1497 16 Oct 11:21 20201016002124.clean.inflight
-rw-r--r-- 1 artur staff 1497 16 Oct 11:21 20201016002124.clean.requested
-rw-r--r-- 1 artur staff 2317 16 Oct 11:21 20201016002124.commit
-rw-r--r-- 1 artur staff 0 16 Oct 11:21 20201016002124.commit.requested
-rw-r--r-- 1 artur staff 985 16 Oct 11:21 20201016002124.inflight
-rw-r--r-- 1 artur staff 1503 16 Oct 11:22 20201016002215.clean
-rw-r--r-- 1 artur staff 1497 16 Oct 11:22 20201016002215.clean.inflight
-rw-r--r-- 1 artur staff 1497 16 Oct 11:22 20201016002215.clean.requested
-rw-r--r-- 1 artur staff 2317 16 Oct 11:22 20201016002215.commit
-rw-r--r-- 1 artur staff 0 16 Oct 11:22 20201016002215.commit.requested
-rw-r--r-- 1 artur staff 985 16 Oct 11:22 20201016002215.inflight
-rw-r--r-- 1 artur staff 1503 16 Oct 11:23 20201016002316.clean
-rw-r--r-- 1 artur staff 1497 16 Oct 11:23 20201016002316.clean.inflight
-rw-r--r-- 1 artur staff 1497 16 Oct 11:23 20201016002316.clean.requested
-rw-r--r-- 1 artur staff 2317 16 Oct 11:23 20201016002316.commit
-rw-r--r-- 1 artur staff 0 16 Oct 11:23 20201016002316.commit.requested
-rw-r--r-- 1 artur staff 985 16 Oct 11:23 20201016002316.inflight
-rw-r--r-- 1 artur staff 1503 16 Oct 11:24 20201016002416.clean
-rw-r--r-- 1 artur staff 1497 16 Oct 11:24 20201016002416.clean.inflight
-rw-r--r-- 1 artur staff 1497 16 Oct 11:24 20201016002416.clean.requested
-rw-r--r-- 1 artur staff 1413 16 Oct 11:24 20201016002416.commit
-rw-r--r-- 1 artur staff 0 16 Oct 11:24 20201016002416.commit.requested
-rw-r--r-- 1 artur staff 380 16 Oct 11:24 20201016002416.inflight
-rw-r--r-- 1 artur staff 2319 16 Oct 11:25 20201016002516.commit
-rw-r--r-- 1 artur staff 0 16 Oct 11:25 20201016002516.commit.requested
-rw-r--r-- 1 artur staff 985 16 Oct 11:25 20201016002516.inflight
-rw-r--r-- 1 artur staff 1503 16 Oct 11:26 20201016002619.clean
-rw-r--r-- 1 artur staff 1497 16 Oct 11:26 20201016002619.clean.inflight
-rw-r--r-- 1 artur staff 1497 16 Oct 11:26 20201016002619.clean.requested
-rw-r--r-- 1 artur staff 1413 16 Oct 11:26 20201016002619.commit
-rw-r--r-- 1 artur staff 0 16 Oct 11:26 20201016002619.commit.requested
-rw-r--r-- 1 artur staff 380 16 Oct 11:26 20201016002619.inflight
-rw-r--r-- 1 artur staff 1413 16 Oct 11:27 20201016002723.commit
-rw-r--r-- 1 artur staff 0 16 Oct 11:27 20201016002723.commit.requested
-rw-r--r-- 1 artur staff 380 16 Oct 11:27 20201016002723.inflight
-rw-r--r-- 1 artur staff 1413 16 Oct 11:28 20201016002825.commit
-rw-r--r-- 1 artur staff 0 16 Oct 11:28 20201016002825.commit.requested
-rw-r--r-- 1 artur staff 380 16 Oct 11:28 20201016002825.inflight
-rw-r--r-- 1 artur staff 1413 16 Oct 11:29 20201016002917.commit
-rw-r--r-- 1 artur staff 0 16 Oct 11:29 20201016002917.commit.requested
-rw-r--r-- 1 artur staff 380 16 Oct 11:29 20201016002917.inflight
-rw-r--r-- 1 artur staff 2319 16 Oct 11:30 20201016003004.commit
-rw-r--r-- 1 artur staff 0 16 Oct 11:30 20201016003004.commit.requested
-rw-r--r-- 1 artur staff 985 16 Oct 11:30 20201016003004.inflight
-rw-r--r-- 1 artur staff 1505 16 Oct 11:31 20201016003106.clean
-rw-r--r-- 1 artur staff 1498 16 Oct 11:31 20201016003106.clean.inflight
-rw-r--r-- 1 artur staff 1498 16 Oct 11:31 20201016003106.clean.requested
-rw-r--r-- 1 artur staff 2319 16 Oct 11:31 20201016003106.commit
-rw-r--r-- 1 artur staff 0 16 Oct 11:31 20201016003106.commit.requested
-rw-r--r-- 1 artur staff 985 16 Oct 11:31 20201016003106.inflight
-rw-r--r-- 1 artur staff 1505 16 Oct 11:34 20201016003338.clean
-rw-r--r-- 1 artur staff 1498 16 Oct 11:34 20201016003338.clean.inflight
-rw-r--r-- 1 artur staff 1498 16 Oct 11:34 20201016003338.clean.requested
-rw-r--r-- 1 artur staff 2319 16 Oct 11:34 20201016003338.commit
-rw-r--r-- 1 artur staff 0 16 Oct 11:33 20201016003338.commit.requested
-rw-r--r-- 1 artur staff 985 16 Oct 11:33 20201016003338.inflight
-rw-r--r-- 1 artur staff 1505 16 Oct 11:37 20201016003648.clean
-rw-r--r-- 1 artur staff 1498 16 Oct 11:37 20201016003648.clean.inflight
-rw-r--r-- 1 artur staff 1498 16 Oct 11:37 20201016003648.clean.requested
-rw-r--r-- 1 artur staff 1413 16 Oct 11:37 20201016003648.commit
-rw-r--r-- 1 artur staff 0 16 Oct 11:36 20201016003648.commit.requested
-rw-r--r-- 1 artur staff 380 16 Oct 11:37 20201016003648.inflight
-rw-r--r-- 1 artur staff 1413 16 Oct 11:37 20201016003733.commit
-rw-r--r-- 1 artur staff 0 16 Oct 11:37 20201016003733.commit.requested
-rw-r--r-- 1 artur staff 380 16 Oct 11:37 20201016003733.inflight
-rw-r--r-- 1 artur staff 2320 16 Oct 11:40 20201016004005.commit
-rw-r--r-- 1 artur staff 0 16 Oct 11:40 20201016004005.commit.requested
-rw-r--r-- 1 artur staff 986 16 Oct 11:40 20201016004005.inflight
-rw-r--r-- 1 artur staff 1505 16 Oct 11:41 20201016004106.clean
-rw-r--r-- 1 artur staff 1498 16 Oct 11:41 20201016004106.clean.inflight
-rw-r--r-- 1 artur staff 1498 16 Oct 11:41 20201016004106.clean.requested
-rw-r--r-- 1 artur staff 2319 16 Oct 11:41 20201016004106.commit
-rw-r--r-- 1 artur staff 0 16 Oct 11:41 20201016004106.commit.requested
-rw-r--r-- 1 artur staff 985 16 Oct 11:41 20201016004106.inflight
-rw-r--r-- 1 artur staff 1505 16 Oct 11:44 20201016004405.clean
-rw-r--r-- 1 artur staff 1498 16 Oct 11:44 20201016004405.clean.inflight
-rw-r--r-- 1 artur staff 1498 16 Oct 11:44 20201016004405.clean.requested
-rw-r--r-- 1 artur staff 2319 16 Oct 11:44 20201016004405.commit
-rw-r--r-- 1 artur staff 0 16 Oct 11:44 20201016004405.commit.requested
-rw-r--r-- 1 artur staff 985 16 Oct 11:44 20201016004405.inflight
-rw-r--r-- 1 artur staff 1505 16 Oct 11:50 20201016005005.clean
-rw-r--r-- 1 artur staff 1498 16 Oct 11:50 20201016005005.clean.inflight
-rw-r--r-- 1 artur staff 1498 16 Oct 11:50 20201016005005.clean.requested
-rw-r--r-- 1 artur staff 2319 16 Oct 11:50 20201016005005.commit
-rw-r--r-- 1 artur staff 0 16 Oct 11:50 20201016005005.commit.requested
-rw-r--r-- 1 artur staff 985 16 Oct 11:50 20201016005005.inflight
-rw-r--r-- 1 artur staff 1505 16 Oct 11:51 20201016005106.clean
-rw-r--r-- 1 artur staff 1498 16 Oct 11:51 20201016005106.clean.inflight
-rw-r--r-- 1 artur staff 1498 16 Oct 11:51 20201016005106.clean.requested
-rw-r--r-- 1 artur staff 2319 16 Oct 11:51 20201016005106.commit
-rw-r--r-- 1 artur staff 0 16 Oct 11:51 20201016005106.commit.requested
-rw-r--r-- 1 artur staff 985 16 Oct 11:51 20201016005106.inflight
-rw-r--r-- 1 artur staff 1505 16 Oct 11:54 20201016005342.clean
-rw-r--r-- 1 artur staff 1498 16 Oct 11:54 20201016005342.clean.inflight
-rw-r--r-- 1 artur staff 1498 16 Oct 11:54 20201016005342.clean.requested
-rw-r--r-- 1 artur staff 2305 16 Oct 11:54 20201016005342.commit
-rw-r--r-- 1 artur staff 0 16 Oct 11:53 20201016005342.commit.requested
-rw-r--r-- 1 artur staff 989 16 Oct 11:54 20201016005342.inflight
-rw-r--r-- 1 artur staff 2312 16 Oct 11:59 20201016005831.commit
-rw-r--r-- 1 artur staff 0 16 Oct 11:58 20201016005831.commit.requested
-rw-r--r-- 1 artur staff 986 16 Oct 11:58 20201016005831.inflight
-rw-r--r-- 1 artur staff 1497 16 Oct 12:00 20201016010004.clean
-rw-r--r-- 1 artur staff 1494 16 Oct 12:00 20201016010004.clean.inflight
-rw-r--r-- 1 artur staff 1494 16 Oct 12:00 20201016010004.clean.requested
-rw-r--r-- 1 artur staff 2311 16 Oct 12:00 20201016010004.commit
-rw-r--r-- 1 artur staff 0 16 Oct 12:00 20201016010004.commit.requested
-rw-r--r-- 1 artur staff 985 16 Oct 12:00 20201016010004.inflight
drwxr-xr-x 24 artur staff 768 16 Oct 12:02 archived
-rw-r--r-- 1 artur staff 231 16 Oct 08:31 hoodie.properties
```
Partitions:
graphversion=782:
```
-rw-r--r-- 1 artur staff 93 16 Oct 08:32 .hoodie_partition_metadata
-rw-r--r-- 1 artur staff 5447259 16 Oct 09:16 efbbb2eb-d244-463d-bf64-55d0c46c057d-0_0-10154-151923_20201015221611.parquet
-rw-r--r-- 1 artur staff 5442031 16 Oct 09:18 efbbb2eb-d244-463d-bf64-55d0c46c057d-0_0-10800-159936_20201015221814.parquet
-rw-r--r-- 1 artur staff 5442099 16 Oct 09:20 efbbb2eb-d244-463d-bf64-55d0c46c057d-0_0-11245-169414_20201015222013.parquet
-rw-r--r-- 1 artur staff 5442001 16 Oct 09:21 efbbb2eb-d244-463d-bf64-55d0c46c057d-0_0-11481-174108_20201015222114.parquet
```
graphversion=783:
```
-rw-r--r-- 1 artur staff 93 16 Oct 09:11 .hoodie_partition_metadata
-rw-r--r-- 1 artur staff 5519741 16 Oct 10:49 125ca82f-911b-4830-86d4-2290111cc55a-0_0-25341-397081_20201015234912.parquet
-rw-r--r-- 1 artur staff 5512472 16 Oct 10:50 125ca82f-911b-4830-86d4-2290111cc55a-0_0-25658-401934_20201015235010.parquet
-rw-r--r-- 1 artur staff 5514534 16 Oct 10:51 125ca82f-911b-4830-86d4-2290111cc55a-0_0-25898-406769_20201015235111.parquet
```
graphversion=786:
```
-rw-r--r-- 1 artur staff 93 16 Oct 10:44 .hoodie_partition_metadata
-rw-r--r-- 1 artur staff 5723412 16 Oct 11:51 bb9e869a-edab-4515-92e4-e99f098f88e1-0_0-13936-223389_20201016005106.parquet
```
graphversion=788:
```
-rw-r--r-- 1 artur staff 93 16 Oct 11:54 .hoodie_partition_metadata
-rw-r--r-- 1 artur staff 5806059 16 Oct 11:59 be99cdb1-d096-4623-9767-4bb7156229c8-0_0-304-4887_20201016005831.parquet
-rw-r--r-- 1 artur staff 5807720 16 Oct 12:00 be99cdb1-d096-4623-9767-4bb7156229c8-0_0-629-9669_20201016010004.parquet
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] bvaradar commented on issue #2174: [SUPPORT] Auto-clean doesn't work
Posted by GitBox <gi...@apache.org>.
bvaradar commented on issue #2174:
URL: https://github.com/apache/hudi/issues/2174#issuecomment-708261688
Can you list the partitions where you are seeing uncleaned files along with listing of .hoodie folder ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] halkar commented on issue #2174: [SUPPORT] Auto-clean doesn't work
Posted by GitBox <gi...@apache.org>.
halkar commented on issue #2174:
URL: https://github.com/apache/hudi/issues/2174#issuecomment-709697654
Commits:
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] halkar edited a comment on issue #2174: [SUPPORT] Auto-clean doesn't work
Posted by GitBox <gi...@apache.org>.
halkar edited a comment on issue #2174:
URL: https://github.com/apache/hudi/issues/2174#issuecomment-708859468
@bvaradar sorry, I'm not sure if I'm getting your question correctly.
Partitions:
<img width="1300" alt="Screen Shot 2020-10-15 at 13 39 53" src="https://user-images.githubusercontent.com/1992396/96070539-26be2b80-0eec-11eb-9d16-2a5bb36f86fb.png">
First partition:
<img width="1304" alt="Screen Shot 2020-10-15 at 13 40 00" src="https://user-images.githubusercontent.com/1992396/96070563-30e02a00-0eec-11eb-9be5-770906af2fc7.png">
Second partition:
<img width="1310" alt="Screen Shot 2020-10-15 at 13 40 08" src="https://user-images.githubusercontent.com/1992396/96070603-42293680-0eec-11eb-9043-442df642040b.png">
Third partition:
<img width="1302" alt="Screen Shot 2020-10-15 at 13 40 16" src="https://user-images.githubusercontent.com/1992396/96070619-49e8db00-0eec-11eb-93c7-096abb8bf796.png">
In reality, we never write into all at the same time. The new partition is created when the new version of the job is deployed.
My main question is why the second partition has 4 files. You can see that they are 30 minutes older than files in the third partition.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] halkar commented on issue #2174: [SUPPORT] Auto-clean doesn't work
Posted by GitBox <gi...@apache.org>.
halkar commented on issue #2174:
URL: https://github.com/apache/hudi/issues/2174#issuecomment-708859468
@bvaradar sorry, I'm not sure if I'm getting your question correctly.
Partitions:
<img width="1300" alt="Screen Shot 2020-10-15 at 13 39 53" src="https://user-images.githubusercontent.com/1992396/96070539-26be2b80-0eec-11eb-9d16-2a5bb36f86fb.png">
First partition:
<img width="1304" alt="Screen Shot 2020-10-15 at 13 40 00" src="https://user-images.githubusercontent.com/1992396/96070563-30e02a00-0eec-11eb-9be5-770906af2fc7.png">
Second partition:
<img width="1310" alt="Screen Shot 2020-10-15 at 13 40 08" src="https://user-images.githubusercontent.com/1992396/96070603-42293680-0eec-11eb-9043-442df642040b.png">
Third partition:
<img width="1302" alt="Screen Shot 2020-10-15 at 13 40 16" src="https://user-images.githubusercontent.com/1992396/96070619-49e8db00-0eec-11eb-93c7-096abb8bf796.png">
In reality, we never write into all at the same time. The new partition is created when the new version of the job is deployed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] halkar commented on issue #2174: [SUPPORT] Auto-clean doesn't work
Posted by GitBox <gi...@apache.org>.
halkar commented on issue #2174:
URL: https://github.com/apache/hudi/issues/2174#issuecomment-709654557
This is a different run. I'll add `cleans show` results later.
All partitions:
<img width="1315" alt="Screen Shot 2020-10-16 at 11 06 57" src="https://user-images.githubusercontent.com/1992396/96198465-9db4fc00-0fa0-11eb-80ba-c951a64fd721.png">
.hoodie folder (not all of it)
<img width="1398" alt="Screen Shot 2020-10-16 at 11 08 08" src="https://user-images.githubusercontent.com/1992396/96198484-aefe0880-0fa0-11eb-819e-b0c310fe2235.png">
<img width="1380" alt="Screen Shot 2020-10-16 at 11 08 51" src="https://user-images.githubusercontent.com/1992396/96198490-b3c2bc80-0fa0-11eb-838e-4cf0b558a7e6.png">
Three partitions (the third one is empty - deployment failed).
<img width="1308" alt="Screen Shot 2020-10-16 at 11 09 16" src="https://user-images.githubusercontent.com/1992396/96198515-c76e2300-0fa0-11eb-85e8-6602e9b4930d.png">
<img width="1301" alt="Screen Shot 2020-10-16 at 11 09 23" src="https://user-images.githubusercontent.com/1992396/96198521-ca691380-0fa0-11eb-9030-ed5a8af1df4b.png">
<img width="1315" alt="Screen Shot 2020-10-16 at 11 09 35" src="https://user-images.githubusercontent.com/1992396/96198524-cd640400-0fa0-11eb-9fcc-05507494926b.png">
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] halkar edited a comment on issue #2174: [SUPPORT] Auto-clean doesn't work
Posted by GitBox <gi...@apache.org>.
halkar edited a comment on issue #2174:
URL: https://github.com/apache/hudi/issues/2174#issuecomment-709654557
This is a different run. I'll add `cleans show` results later.
All partitions:
```
staff 5984 16 Oct 12:02 .hoodie
staff 224 16 Oct 12:02 graphversion=782
staff 192 16 Oct 12:02 graphversion=783
staff 128 16 Oct 12:02 graphversion=786
staff 160 16 Oct 12:02 graphversion=788
```
.hoodie folder (not all of it)
```
staff 969 16 Oct 09:16 20201015221614.rollback
staff 0 16 Oct 09:16 20201015221614.rollback.inflight
staff 1043 16 Oct 09:17 20201015221713.rollback
staff 0 16 Oct 09:17 20201015221713.rollback.inflight
staff 1043 16 Oct 09:18 20201015221834.rollback
staff 0 16 Oct 09:18 20201015221834.rollback.inflight
staff 1206 16 Oct 09:19 20201015221941.rollback
staff 0 16 Oct 09:19 20201015221941.rollback.inflight
staff 1043 16 Oct 09:20 20201015222038.rollback
staff 0 16 Oct 09:20 20201015222038.rollback.inflight
staff 1043 16 Oct 09:21 20201015222129.rollback
staff 0 16 Oct 09:21 20201015222129.rollback.inflight
staff 969 16 Oct 09:49 20201015224949.rollback
staff 0 16 Oct 09:49 20201015224949.rollback.inflight
staff 1079 16 Oct 10:49 20201015234929.rollback
staff 0 16 Oct 10:49 20201015234929.rollback.inflight
staff 1079 16 Oct 10:50 20201015235034.rollback
staff 0 16 Oct 10:50 20201015235034.rollback.inflight
staff 969 16 Oct 10:52 20201015235220.rollback
staff 0 16 Oct 10:52 20201015235220.rollback.inflight
staff 969 16 Oct 10:53 20201015235319.rollback
staff 0 16 Oct 10:53 20201015235319.rollback.inflight
staff 1079 16 Oct 10:54 20201015235414.rollback
staff 0 16 Oct 10:54 20201015235414.rollback.inflight
staff 969 16 Oct 10:55 20201015235515.rollback
staff 0 16 Oct 10:55 20201015235515.rollback.inflight
staff 1501 16 Oct 11:06 20201016000617.clean
staff 1496 16 Oct 11:06 20201016000617.clean.inflight
staff 1496 16 Oct 11:06 20201016000617.clean.requested
staff 1501 16 Oct 11:07 20201016000716.clean
staff 1496 16 Oct 11:07 20201016000716.clean.inflight
staff 1496 16 Oct 11:07 20201016000716.clean.requested
staff 1501 16 Oct 11:08 20201016000816.clean
staff 1496 16 Oct 11:08 20201016000816.clean.inflight
staff 1496 16 Oct 11:08 20201016000816.clean.requested
staff 1501 16 Oct 11:09 20201016000915.clean
staff 1496 16 Oct 11:09 20201016000915.clean.inflight
staff 1496 16 Oct 11:09 20201016000915.clean.requested
staff 1501 16 Oct 11:10 20201016001015.clean
staff 1496 16 Oct 11:10 20201016001015.clean.inflight
staff 1496 16 Oct 11:10 20201016001015.clean.requested
staff 1501 16 Oct 11:11 20201016001117.clean
staff 1496 16 Oct 11:11 20201016001117.clean.inflight
staff 1496 16 Oct 11:11 20201016001117.clean.requested
staff 1503 16 Oct 11:12 20201016001222.clean
staff 1497 16 Oct 11:12 20201016001222.clean.inflight
staff 1497 16 Oct 11:12 20201016001222.clean.requested
staff 2318 16 Oct 11:16 20201016001619.commit
staff 0 16 Oct 11:16 20201016001619.commit.requested
staff 986 16 Oct 11:16 20201016001619.inflight
staff 1655 16 Oct 11:17 20201016001718.clean
staff 1660 16 Oct 11:17 20201016001718.clean.inflight
staff 1660 16 Oct 11:17 20201016001718.clean.requested
staff 2318 16 Oct 11:17 20201016001718.commit
staff 0 16 Oct 11:17 20201016001718.commit.requested
staff 986 16 Oct 11:17 20201016001718.inflight
staff 1503 16 Oct 11:18 20201016001818.clean
staff 1497 16 Oct 11:18 20201016001818.clean.inflight
staff 1497 16 Oct 11:18 20201016001818.clean.requested
staff 2317 16 Oct 11:18 20201016001818.commit
staff 0 16 Oct 11:18 20201016001818.commit.requested
staff 985 16 Oct 11:18 20201016001818.inflight
staff 1503 16 Oct 11:20 20201016001919.clean
staff 1497 16 Oct 11:20 20201016001919.clean.inflight
staff 1497 16 Oct 11:20 20201016001919.clean.requested
staff 2317 16 Oct 11:19 20201016001919.commit
staff 0 16 Oct 11:19 20201016001919.commit.requested
staff 985 16 Oct 11:19 20201016001919.inflight
staff 1503 16 Oct 11:21 20201016002025.clean
staff 1497 16 Oct 11:21 20201016002025.clean.inflight
staff 1497 16 Oct 11:21 20201016002025.clean.requested
staff 2318 16 Oct 11:20 20201016002025.commit
staff 0 16 Oct 11:20 20201016002025.commit.requested
staff 986 16 Oct 11:20 20201016002025.inflight
staff 1503 16 Oct 11:21 20201016002124.clean
staff 1497 16 Oct 11:21 20201016002124.clean.inflight
staff 1497 16 Oct 11:21 20201016002124.clean.requested
staff 2317 16 Oct 11:21 20201016002124.commit
staff 0 16 Oct 11:21 20201016002124.commit.requested
staff 985 16 Oct 11:21 20201016002124.inflight
staff 1503 16 Oct 11:22 20201016002215.clean
staff 1497 16 Oct 11:22 20201016002215.clean.inflight
staff 1497 16 Oct 11:22 20201016002215.clean.requested
staff 2317 16 Oct 11:22 20201016002215.commit
staff 0 16 Oct 11:22 20201016002215.commit.requested
staff 985 16 Oct 11:22 20201016002215.inflight
staff 1503 16 Oct 11:23 20201016002316.clean
staff 1497 16 Oct 11:23 20201016002316.clean.inflight
staff 1497 16 Oct 11:23 20201016002316.clean.requested
staff 2317 16 Oct 11:23 20201016002316.commit
staff 0 16 Oct 11:23 20201016002316.commit.requested
staff 985 16 Oct 11:23 20201016002316.inflight
staff 1503 16 Oct 11:24 20201016002416.clean
staff 1497 16 Oct 11:24 20201016002416.clean.inflight
staff 1497 16 Oct 11:24 20201016002416.clean.requested
staff 1413 16 Oct 11:24 20201016002416.commit
staff 0 16 Oct 11:24 20201016002416.commit.requested
staff 380 16 Oct 11:24 20201016002416.inflight
staff 2319 16 Oct 11:25 20201016002516.commit
staff 0 16 Oct 11:25 20201016002516.commit.requested
staff 985 16 Oct 11:25 20201016002516.inflight
staff 1503 16 Oct 11:26 20201016002619.clean
staff 1497 16 Oct 11:26 20201016002619.clean.inflight
staff 1497 16 Oct 11:26 20201016002619.clean.requested
staff 1413 16 Oct 11:26 20201016002619.commit
staff 0 16 Oct 11:26 20201016002619.commit.requested
staff 380 16 Oct 11:26 20201016002619.inflight
staff 1413 16 Oct 11:27 20201016002723.commit
staff 0 16 Oct 11:27 20201016002723.commit.requested
staff 380 16 Oct 11:27 20201016002723.inflight
staff 1413 16 Oct 11:28 20201016002825.commit
staff 0 16 Oct 11:28 20201016002825.commit.requested
staff 380 16 Oct 11:28 20201016002825.inflight
staff 1413 16 Oct 11:29 20201016002917.commit
staff 0 16 Oct 11:29 20201016002917.commit.requested
staff 380 16 Oct 11:29 20201016002917.inflight
staff 2319 16 Oct 11:30 20201016003004.commit
staff 0 16 Oct 11:30 20201016003004.commit.requested
staff 985 16 Oct 11:30 20201016003004.inflight
staff 1505 16 Oct 11:31 20201016003106.clean
staff 1498 16 Oct 11:31 20201016003106.clean.inflight
staff 1498 16 Oct 11:31 20201016003106.clean.requested
staff 2319 16 Oct 11:31 20201016003106.commit
staff 0 16 Oct 11:31 20201016003106.commit.requested
staff 985 16 Oct 11:31 20201016003106.inflight
staff 1505 16 Oct 11:34 20201016003338.clean
staff 1498 16 Oct 11:34 20201016003338.clean.inflight
staff 1498 16 Oct 11:34 20201016003338.clean.requested
staff 2319 16 Oct 11:34 20201016003338.commit
staff 0 16 Oct 11:33 20201016003338.commit.requested
staff 985 16 Oct 11:33 20201016003338.inflight
staff 1505 16 Oct 11:37 20201016003648.clean
staff 1498 16 Oct 11:37 20201016003648.clean.inflight
staff 1498 16 Oct 11:37 20201016003648.clean.requested
staff 1413 16 Oct 11:37 20201016003648.commit
staff 0 16 Oct 11:36 20201016003648.commit.requested
staff 380 16 Oct 11:37 20201016003648.inflight
staff 1413 16 Oct 11:37 20201016003733.commit
staff 0 16 Oct 11:37 20201016003733.commit.requested
staff 380 16 Oct 11:37 20201016003733.inflight
staff 2320 16 Oct 11:40 20201016004005.commit
staff 0 16 Oct 11:40 20201016004005.commit.requested
staff 986 16 Oct 11:40 20201016004005.inflight
staff 1505 16 Oct 11:41 20201016004106.clean
staff 1498 16 Oct 11:41 20201016004106.clean.inflight
staff 1498 16 Oct 11:41 20201016004106.clean.requested
staff 2319 16 Oct 11:41 20201016004106.commit
staff 0 16 Oct 11:41 20201016004106.commit.requested
staff 985 16 Oct 11:41 20201016004106.inflight
staff 1505 16 Oct 11:44 20201016004405.clean
staff 1498 16 Oct 11:44 20201016004405.clean.inflight
staff 1498 16 Oct 11:44 20201016004405.clean.requested
staff 2319 16 Oct 11:44 20201016004405.commit
staff 0 16 Oct 11:44 20201016004405.commit.requested
staff 985 16 Oct 11:44 20201016004405.inflight
staff 1505 16 Oct 11:50 20201016005005.clean
staff 1498 16 Oct 11:50 20201016005005.clean.inflight
staff 1498 16 Oct 11:50 20201016005005.clean.requested
staff 2319 16 Oct 11:50 20201016005005.commit
staff 0 16 Oct 11:50 20201016005005.commit.requested
staff 985 16 Oct 11:50 20201016005005.inflight
staff 1505 16 Oct 11:51 20201016005106.clean
staff 1498 16 Oct 11:51 20201016005106.clean.inflight
staff 1498 16 Oct 11:51 20201016005106.clean.requested
staff 2319 16 Oct 11:51 20201016005106.commit
staff 0 16 Oct 11:51 20201016005106.commit.requested
staff 985 16 Oct 11:51 20201016005106.inflight
staff 1505 16 Oct 11:54 20201016005342.clean
staff 1498 16 Oct 11:54 20201016005342.clean.inflight
staff 1498 16 Oct 11:54 20201016005342.clean.requested
staff 2305 16 Oct 11:54 20201016005342.commit
staff 0 16 Oct 11:53 20201016005342.commit.requested
staff 989 16 Oct 11:54 20201016005342.inflight
staff 2312 16 Oct 11:59 20201016005831.commit
staff 0 16 Oct 11:58 20201016005831.commit.requested
staff 986 16 Oct 11:58 20201016005831.inflight
staff 1497 16 Oct 12:00 20201016010004.clean
staff 1494 16 Oct 12:00 20201016010004.clean.inflight
staff 1494 16 Oct 12:00 20201016010004.clean.requested
staff 2311 16 Oct 12:00 20201016010004.commit
staff 0 16 Oct 12:00 20201016010004.commit.requested
staff 985 16 Oct 12:00 20201016010004.inflight
staff 768 16 Oct 12:02 archived
staff 231 16 Oct 08:31 hoodie.properties
```
Partitions:
graphversion=782
```
staff 93 16 Oct 08:32 .hoodie_partition_metadata
staff 5447259 16 Oct 09:16 efbbb2eb-d244-463d-bf64-55d0c46c057d-0_0-10154-151923_20201015221611.parquet
staff 5442031 16 Oct 09:18 efbbb2eb-d244-463d-bf64-55d0c46c057d-0_0-10800-159936_20201015221814.parquet
staff 5442099 16 Oct 09:20 efbbb2eb-d244-463d-bf64-55d0c46c057d-0_0-11245-169414_20201015222013.parquet
staff 5442001 16 Oct 09:21 efbbb2eb-d244-463d-bf64-55d0c46c057d-0_0-11481-174108_20201015222114.parquet
```
graphversion=783
```
staff 93 16 Oct 09:11 .hoodie_partition_metadata
staff 5519741 16 Oct 10:49 125ca82f-911b-4830-86d4-2290111cc55a-0_0-25341-397081_20201015234912.parquet
staff 5512472 16 Oct 10:50 125ca82f-911b-4830-86d4-2290111cc55a-0_0-25658-401934_20201015235010.parquet
staff 5514534 16 Oct 10:51 125ca82f-911b-4830-86d4-2290111cc55a-0_0-25898-406769_20201015235111.parquet
```
graphversion=786
```
staff 93 16 Oct 10:44 .hoodie_partition_metadata
staff 5723412 16 Oct 11:51 bb9e869a-edab-4515-92e4-e99f098f88e1-0_0-13936-223389_20201016005106.parquet
```
graphversion=788
```
staff 93 16 Oct 11:54 .hoodie_partition_metadata
staff 5806059 16 Oct 11:59 be99cdb1-d096-4623-9767-4bb7156229c8-0_0-304-4887_20201016005831.parquet
staff 5807720 16 Oct 12:00 be99cdb1-d096-4623-9767-4bb7156229c8-0_0-629-9669_20201016010004.parquet
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] halkar edited a comment on issue #2174: [SUPPORT] Auto-clean doesn't work
Posted by GitBox <gi...@apache.org>.
halkar edited a comment on issue #2174:
URL: https://github.com/apache/hudi/issues/2174#issuecomment-709654557
This is a different run. I'll add `cleans show` results later.
All partitions:
```
staff 5984 16 Oct 12:02 .hoodie
staff 224 16 Oct 12:02 graphversion=782
staff 192 16 Oct 12:02 graphversion=783
staff 128 16 Oct 12:02 graphversion=786
staff 160 16 Oct 12:02 graphversion=788
```
.hoodie folder (not all of it)
```
staff 969 16 Oct 09:16 20201015221614.rollback
staff 0 16 Oct 09:16 20201015221614.rollback.inflight
staff 1043 16 Oct 09:17 20201015221713.rollback
staff 0 16 Oct 09:17 20201015221713.rollback.inflight
staff 1043 16 Oct 09:18 20201015221834.rollback
staff 0 16 Oct 09:18 20201015221834.rollback.inflight
staff 1206 16 Oct 09:19 20201015221941.rollback
staff 0 16 Oct 09:19 20201015221941.rollback.inflight
staff 1043 16 Oct 09:20 20201015222038.rollback
staff 0 16 Oct 09:20 20201015222038.rollback.inflight
staff 1043 16 Oct 09:21 20201015222129.rollback
staff 0 16 Oct 09:21 20201015222129.rollback.inflight
staff 969 16 Oct 09:49 20201015224949.rollback
staff 0 16 Oct 09:49 20201015224949.rollback.inflight
staff 1079 16 Oct 10:49 20201015234929.rollback
staff 0 16 Oct 10:49 20201015234929.rollback.inflight
staff 1079 16 Oct 10:50 20201015235034.rollback
staff 0 16 Oct 10:50 20201015235034.rollback.inflight
staff 969 16 Oct 10:52 20201015235220.rollback
staff 0 16 Oct 10:52 20201015235220.rollback.inflight
staff 969 16 Oct 10:53 20201015235319.rollback
staff 0 16 Oct 10:53 20201015235319.rollback.inflight
staff 1079 16 Oct 10:54 20201015235414.rollback
staff 0 16 Oct 10:54 20201015235414.rollback.inflight
staff 969 16 Oct 10:55 20201015235515.rollback
staff 0 16 Oct 10:55 20201015235515.rollback.inflight
staff 1501 16 Oct 11:06 20201016000617.clean
staff 1496 16 Oct 11:06 20201016000617.clean.inflight
staff 1496 16 Oct 11:06 20201016000617.clean.requested
staff 1501 16 Oct 11:07 20201016000716.clean
staff 1496 16 Oct 11:07 20201016000716.clean.inflight
staff 1496 16 Oct 11:07 20201016000716.clean.requested
staff 1501 16 Oct 11:08 20201016000816.clean
staff 1496 16 Oct 11:08 20201016000816.clean.inflight
staff 1496 16 Oct 11:08 20201016000816.clean.requested
staff 1501 16 Oct 11:09 20201016000915.clean
staff 1496 16 Oct 11:09 20201016000915.clean.inflight
staff 1496 16 Oct 11:09 20201016000915.clean.requested
staff 1501 16 Oct 11:10 20201016001015.clean
staff 1496 16 Oct 11:10 20201016001015.clean.inflight
staff 1496 16 Oct 11:10 20201016001015.clean.requested
staff 1501 16 Oct 11:11 20201016001117.clean
staff 1496 16 Oct 11:11 20201016001117.clean.inflight
staff 1496 16 Oct 11:11 20201016001117.clean.requested
staff 1503 16 Oct 11:12 20201016001222.clean
staff 1497 16 Oct 11:12 20201016001222.clean.inflight
staff 1497 16 Oct 11:12 20201016001222.clean.requested
staff 2318 16 Oct 11:16 20201016001619.commit
staff 0 16 Oct 11:16 20201016001619.commit.requested
staff 986 16 Oct 11:16 20201016001619.inflight
staff 1655 16 Oct 11:17 20201016001718.clean
staff 1660 16 Oct 11:17 20201016001718.clean.inflight
staff 1660 16 Oct 11:17 20201016001718.clean.requested
staff 2318 16 Oct 11:17 20201016001718.commit
staff 0 16 Oct 11:17 20201016001718.commit.requested
staff 986 16 Oct 11:17 20201016001718.inflight
staff 1503 16 Oct 11:18 20201016001818.clean
staff 1497 16 Oct 11:18 20201016001818.clean.inflight
staff 1497 16 Oct 11:18 20201016001818.clean.requested
staff 2317 16 Oct 11:18 20201016001818.commit
staff 0 16 Oct 11:18 20201016001818.commit.requested
staff 985 16 Oct 11:18 20201016001818.inflight
staff 1503 16 Oct 11:20 20201016001919.clean
staff 1497 16 Oct 11:20 20201016001919.clean.inflight
staff 1497 16 Oct 11:20 20201016001919.clean.requested
staff 2317 16 Oct 11:19 20201016001919.commit
staff 0 16 Oct 11:19 20201016001919.commit.requested
staff 985 16 Oct 11:19 20201016001919.inflight
staff 1503 16 Oct 11:21 20201016002025.clean
staff 1497 16 Oct 11:21 20201016002025.clean.inflight
staff 1497 16 Oct 11:21 20201016002025.clean.requested
staff 2318 16 Oct 11:20 20201016002025.commit
staff 0 16 Oct 11:20 20201016002025.commit.requested
staff 986 16 Oct 11:20 20201016002025.inflight
staff 1503 16 Oct 11:21 20201016002124.clean
staff 1497 16 Oct 11:21 20201016002124.clean.inflight
staff 1497 16 Oct 11:21 20201016002124.clean.requested
staff 2317 16 Oct 11:21 20201016002124.commit
staff 0 16 Oct 11:21 20201016002124.commit.requested
staff 985 16 Oct 11:21 20201016002124.inflight
staff 1503 16 Oct 11:22 20201016002215.clean
staff 1497 16 Oct 11:22 20201016002215.clean.inflight
staff 1497 16 Oct 11:22 20201016002215.clean.requested
staff 2317 16 Oct 11:22 20201016002215.commit
staff 0 16 Oct 11:22 20201016002215.commit.requested
staff 985 16 Oct 11:22 20201016002215.inflight
staff 1503 16 Oct 11:23 20201016002316.clean
staff 1497 16 Oct 11:23 20201016002316.clean.inflight
staff 1497 16 Oct 11:23 20201016002316.clean.requested
staff 2317 16 Oct 11:23 20201016002316.commit
staff 0 16 Oct 11:23 20201016002316.commit.requested
staff 985 16 Oct 11:23 20201016002316.inflight
staff 1503 16 Oct 11:24 20201016002416.clean
staff 1497 16 Oct 11:24 20201016002416.clean.inflight
staff 1497 16 Oct 11:24 20201016002416.clean.requested
staff 1413 16 Oct 11:24 20201016002416.commit
staff 0 16 Oct 11:24 20201016002416.commit.requested
staff 380 16 Oct 11:24 20201016002416.inflight
staff 2319 16 Oct 11:25 20201016002516.commit
staff 0 16 Oct 11:25 20201016002516.commit.requested
staff 985 16 Oct 11:25 20201016002516.inflight
staff 1503 16 Oct 11:26 20201016002619.clean
staff 1497 16 Oct 11:26 20201016002619.clean.inflight
staff 1497 16 Oct 11:26 20201016002619.clean.requested
staff 1413 16 Oct 11:26 20201016002619.commit
staff 0 16 Oct 11:26 20201016002619.commit.requested
staff 380 16 Oct 11:26 20201016002619.inflight
staff 1413 16 Oct 11:27 20201016002723.commit
staff 0 16 Oct 11:27 20201016002723.commit.requested
staff 380 16 Oct 11:27 20201016002723.inflight
staff 1413 16 Oct 11:28 20201016002825.commit
staff 0 16 Oct 11:28 20201016002825.commit.requested
staff 380 16 Oct 11:28 20201016002825.inflight
staff 1413 16 Oct 11:29 20201016002917.commit
staff 0 16 Oct 11:29 20201016002917.commit.requested
staff 380 16 Oct 11:29 20201016002917.inflight
staff 2319 16 Oct 11:30 20201016003004.commit
staff 0 16 Oct 11:30 20201016003004.commit.requested
staff 985 16 Oct 11:30 20201016003004.inflight
staff 1505 16 Oct 11:31 20201016003106.clean
staff 1498 16 Oct 11:31 20201016003106.clean.inflight
staff 1498 16 Oct 11:31 20201016003106.clean.requested
staff 2319 16 Oct 11:31 20201016003106.commit
staff 0 16 Oct 11:31 20201016003106.commit.requested
staff 985 16 Oct 11:31 20201016003106.inflight
staff 1505 16 Oct 11:34 20201016003338.clean
staff 1498 16 Oct 11:34 20201016003338.clean.inflight
staff 1498 16 Oct 11:34 20201016003338.clean.requested
staff 2319 16 Oct 11:34 20201016003338.commit
staff 0 16 Oct 11:33 20201016003338.commit.requested
staff 985 16 Oct 11:33 20201016003338.inflight
staff 1505 16 Oct 11:37 20201016003648.clean
staff 1498 16 Oct 11:37 20201016003648.clean.inflight
staff 1498 16 Oct 11:37 20201016003648.clean.requested
staff 1413 16 Oct 11:37 20201016003648.commit
staff 0 16 Oct 11:36 20201016003648.commit.requested
staff 380 16 Oct 11:37 20201016003648.inflight
staff 1413 16 Oct 11:37 20201016003733.commit
staff 0 16 Oct 11:37 20201016003733.commit.requested
staff 380 16 Oct 11:37 20201016003733.inflight
staff 2320 16 Oct 11:40 20201016004005.commit
staff 0 16 Oct 11:40 20201016004005.commit.requested
staff 986 16 Oct 11:40 20201016004005.inflight
staff 1505 16 Oct 11:41 20201016004106.clean
staff 1498 16 Oct 11:41 20201016004106.clean.inflight
staff 1498 16 Oct 11:41 20201016004106.clean.requested
staff 2319 16 Oct 11:41 20201016004106.commit
staff 0 16 Oct 11:41 20201016004106.commit.requested
staff 985 16 Oct 11:41 20201016004106.inflight
staff 1505 16 Oct 11:44 20201016004405.clean
staff 1498 16 Oct 11:44 20201016004405.clean.inflight
staff 1498 16 Oct 11:44 20201016004405.clean.requested
staff 2319 16 Oct 11:44 20201016004405.commit
staff 0 16 Oct 11:44 20201016004405.commit.requested
staff 985 16 Oct 11:44 20201016004405.inflight
staff 1505 16 Oct 11:50 20201016005005.clean
staff 1498 16 Oct 11:50 20201016005005.clean.inflight
staff 1498 16 Oct 11:50 20201016005005.clean.requested
staff 2319 16 Oct 11:50 20201016005005.commit
staff 0 16 Oct 11:50 20201016005005.commit.requested
staff 985 16 Oct 11:50 20201016005005.inflight
staff 1505 16 Oct 11:51 20201016005106.clean
staff 1498 16 Oct 11:51 20201016005106.clean.inflight
staff 1498 16 Oct 11:51 20201016005106.clean.requested
staff 2319 16 Oct 11:51 20201016005106.commit
staff 0 16 Oct 11:51 20201016005106.commit.requested
staff 985 16 Oct 11:51 20201016005106.inflight
staff 1505 16 Oct 11:54 20201016005342.clean
staff 1498 16 Oct 11:54 20201016005342.clean.inflight
staff 1498 16 Oct 11:54 20201016005342.clean.requested
staff 2305 16 Oct 11:54 20201016005342.commit
staff 0 16 Oct 11:53 20201016005342.commit.requested
staff 989 16 Oct 11:54 20201016005342.inflight
staff 2312 16 Oct 11:59 20201016005831.commit
staff 0 16 Oct 11:58 20201016005831.commit.requested
staff 986 16 Oct 11:58 20201016005831.inflight
staff 1497 16 Oct 12:00 20201016010004.clean
staff 1494 16 Oct 12:00 20201016010004.clean.inflight
staff 1494 16 Oct 12:00 20201016010004.clean.requested
staff 2311 16 Oct 12:00 20201016010004.commit
staff 0 16 Oct 12:00 20201016010004.commit.requested
staff 985 16 Oct 12:00 20201016010004.inflight
drwxr-xr-x 24 artur staff 768 16 Oct 12:02 archived
staff 231 16 Oct 08:31 hoodie.properties
```
Partitions:
graphversion=782
```
staff 93 16 Oct 08:32 .hoodie_partition_metadata
staff 5447259 16 Oct 09:16 efbbb2eb-d244-463d-bf64-55d0c46c057d-0_0-10154-151923_20201015221611.parquet
staff 5442031 16 Oct 09:18 efbbb2eb-d244-463d-bf64-55d0c46c057d-0_0-10800-159936_20201015221814.parquet
staff 5442099 16 Oct 09:20 efbbb2eb-d244-463d-bf64-55d0c46c057d-0_0-11245-169414_20201015222013.parquet
staff 5442001 16 Oct 09:21 efbbb2eb-d244-463d-bf64-55d0c46c057d-0_0-11481-174108_20201015222114.parquet
```
graphversion=783
```
staff 93 16 Oct 09:11 .hoodie_partition_metadata
staff 5519741 16 Oct 10:49 125ca82f-911b-4830-86d4-2290111cc55a-0_0-25341-397081_20201015234912.parquet
staff 5512472 16 Oct 10:50 125ca82f-911b-4830-86d4-2290111cc55a-0_0-25658-401934_20201015235010.parquet
staff 5514534 16 Oct 10:51 125ca82f-911b-4830-86d4-2290111cc55a-0_0-25898-406769_20201015235111.parquet
```
graphversion=786
```
staff 93 16 Oct 10:44 .hoodie_partition_metadata
staff 5723412 16 Oct 11:51 bb9e869a-edab-4515-92e4-e99f098f88e1-0_0-13936-223389_20201016005106.parquet
```
graphversion=788
```
staff 93 16 Oct 11:54 .hoodie_partition_metadata
staff 5806059 16 Oct 11:59 be99cdb1-d096-4623-9767-4bb7156229c8-0_0-304-4887_20201016005831.parquet
staff 5807720 16 Oct 12:00 be99cdb1-d096-4623-9767-4bb7156229c8-0_0-629-9669_20201016010004.parquet
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] halkar edited a comment on issue #2174: [SUPPORT] Auto-clean doesn't work
Posted by GitBox <gi...@apache.org>.
halkar edited a comment on issue #2174:
URL: https://github.com/apache/hudi/issues/2174#issuecomment-709685927
Cleans:
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org