You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2020/12/29 17:53:49 UTC
[GitHub] [hudi] tooptoop4 opened a new issue #2393: [SUPPORT] excessive small files created
tooptoop4 opened a new issue #2393:
URL: https://github.com/apache/hudi/issues/2393
steps taken (3 sequential spark submits):
1. 3 row insert (from a single csv)
2. 3 row update (from a single csv.gz)
3. 19967954 row update (from a single csv.gz)
there are over 23000 parquet files on s3, all around 480kb
```
2020-11-25 09:37:53 0 mytable/.hoodie/.aux/
2020-11-25 11:50:41 0 mytable/.hoodie/.temp/
2020-11-25 09:38:04 2181 mytable/.hoodie/20201125093753.commit
2020-11-25 09:37:54 0 mytable/.hoodie/20201125093753.commit.requested
2020-11-25 09:37:57 350 mytable/.hoodie/20201125093753.inflight
2020-11-25 09:52:49 1159 mytable/.hoodie/20201125095231.clean
2020-11-25 09:52:49 875 mytable/.hoodie/20201125095231.clean.inflight
2020-11-25 09:52:48 875 mytable/.hoodie/20201125095231.clean.requested
2020-11-25 09:52:44 2197 mytable/.hoodie/20201125095231.commit
2020-11-25 09:52:33 0 mytable/.hoodie/20201125095231.commit.requested
2020-11-25 09:52:39 1032 mytable/.hoodie/20201125095231.inflight
2020-11-25 11:50:53 1299 mytable/.hoodie/20201125104200.clean
2020-11-25 11:50:52 945 mytable/.hoodie/20201125104200.clean.inflight
2020-11-25 11:50:51 945 mytable/.hoodie/20201125104200.clean.requested
2020-11-25 11:50:42 26296154 mytable/.hoodie/20201125104200.commit
2020-11-25 10:42:01 0 mytable/.hoodie/20201125104200.commit.requested
2020-11-25 11:02:47 1032 mytable/.hoodie/20201125104200.inflight
2020-11-25 13:14:56 0 mytable/.hoodie/20201125131454.commit.requested
2020-11-25 09:37:53 213 mytable/.hoodie/hoodie.properties
2020-11-25 09:37:59 93 mytable/unknown/.hoodie_partition_metadata
2020-11-25 11:20:01 482269 mytable/unknown/0000c472-f607-408c-8989-66caf3a1f1ff-0_7388-25-7557_20201125104200.parquet
2020-11-25 11:39:11 480273 mytable/unknown/0002d53a-9ab6-4c31-9bd9-507b89d6ac31-0_18623-25-18792_20201125104200.parquet
2020-11-25 11:17:43 481345 mytable/unknown/00031d81-a16e-478c-91b0-fffdb1cc5f3c-0_6128-25-6297_20201125104200.parquet
2020-11-25 11:26:23 481335 mytable/unknown/0003248b-084c-4567-8d47-d20f91af8c29-0_10941-25-11110_20201125104200.parquet
2020-11-25 11:30:49 482571 mytable/unknown/00040167-e102-4777-949d-d61eb834e8d6-0_13618-25-13787_20201125104200.parquet
2020-11-25 11:13:37 483496 mytable/unknown/00050b40-f11e-40ac-a232-36a82af81e38-0_4039-25-4208_20201125104200.parquet
2020-11-25 11:46:12 483891 mytable/unknown/000a8e9c-e54b-4459-b81c-45b8dd82efb9-0_22801-25-22970_20201125104200.parquet
2020-11-25 11:28:10 478902 mytable/unknown/002041f0-d33d-4321-a88a-953f7d2be3f2-0_12040-25-12209_20201125104200.parquet
2020-11-25 11:29:46 481187 mytable/unknown/00255b66-ad0d-4987-8e81-52dd14c17dcc-0_12990-25-13159_20201125104200.parquet
2020-11-25 11:19:04 481080 mytable/unknown/00269076-3986-4014-8540-9e038069b9c0-0_6864-25-7033_20201125104200.parquet
2020-11-25 11:07:09 481899 mytable/unknown/0026d816-001f-4afb-8849-c747fb3c0fa1-0_632-25-801_20201125104200.parquet
2020-11-25 11:08:05 481763 mytable/unknown/002a820e-6e83-44ce-aa72-ae7be1ddc644-0_1116-25-1285_20201125104200.parquet
2020-11-25 11:44:22 482357 mytable/unknown/002abf16-952d-49c3-8de5-09087fff93fd-0_21703-25-21872_20201125104200.parquet
2020-11-25 11:11:20 478808 mytable/unknown/002cb443-5787-42d5-bbcc-a127dad79535-0_2843-25-3012_20201125104200.parquet
2020-11-25 11:44:17 479243 mytable/unknown/00335faa-901e-4567-9478-07b8860be876-0_21662-25-21831_20201125104200.parquet
2020-11-25 11:08:11 481899 mytable/unknown/003bf393-9181-4aa2-b02f-d125931053e0-0_1168-25-1337_20201125104200.parquet
2020-11-25 11:13:52 478852 mytable/unknown/003d948c-b7f7-4737-97a4-6ab688b958f6-0_4183-25-4352_20201125104200.parquet
2020-11-25 11:32:24 480843 mytable/unknown/0040267c-f03f-4558-bd3b-4b010681dfb3-0_14526-25-14695_20201125104200.parquet
2020-11-25 11:12:45 479753 mytable/unknown/004131d9-365d-4a57-b0a0-ec86eca595cf-0_3590-25-3759_20201125104200.parquet
2020-11-25 11:15:21 479938 mytable/unknown/0045ceef-7180-4b7f-a186-2c91a9758b91-0_4886-25-5055_20201125104200.parquet
2020-11-25 11:07:51 481140 mytable/unknown/0048ffa6-4197-48c8-bbdd-95195c5606ad-0_1002-25-1171_20201125104200.parquet
2020-11-25 11:36:49 481792 mytable/unknown/004b470e-0bfb-4122-ac6b-03f5d1333027-0_17224-25-17393_20201125104200.parquet
2020-11-25 11:30:54 480000 mytable/unknown/004c4f54-61be-4baa-9714-fb3a00f10e25-0_13668-25-13837_20201125104200.parquet
2020-11-25 11:26:30 481597 mytable/unknown/004cb4b5-3f8b-4d14-8fd1-042a9c5fa59a-0_11016-25-11185_20201125104200.parquet
2020-11-25 11:40:26 483142 mytable/unknown/004ec33a-7715-4769-8b64-e83c26b0b7e8-0_19330-25-19499_20201125104200.parquet
2020-11-25 11:38:09 481798 mytable/unknown/0050e53f-9972-47b4-96c4-e0087f3f5be0-0_18001-25-18170_20201125104200.parquet
2020-11-25 11:43:13 483521 mytable/unknown/0052dd7c-0e92-4268-8812-a6ed53dc18e6-0_21049-25-21218_20201125104200.parquet
2020-11-25 11:42:58 479095 mytable/unknown/00553bac-c633-4a57-9aec-b96d1e71d819-0_20907-25-21076_20201125104200.parquet
2020-11-25 11:43:17 479115 mytable/unknown/0058b4db-fc13-47f3-948b-67641363f10a-0_21082-25-21251_20201125104200.parquet
2020-11-25 11:07:18 481308 mytable/unknown/0059e0da-ea61-431f-81ec-3d79e981554b-0_721-25-890_20201125104200.parquet
2020-11-25 11:31:57 480673 mytable/unknown/005a4171-0405-4fc1-a336-dee2097574be-0_14272-25-14441_20201125104200.parquet
2020-11-25 11:28:47 481765 mytable/unknown/005aa617-2670-4e2a-9642-bba82e68c23b-0_12395-25-12564_20201125104200.parquet
2020-11-25 11:15:23 482319 mytable/unknown/005d7704-2d8b-4522-a1cf-8d04f9b4be36-0_4901-25-5070_20201125104200.parquet
2020-11-25 11:44:32 480161 mytable/unknown/0060462c-4736-43b9-801e-47d889149d37-0_21792-25-21961_20201125104200.parquet
2020-11-25 11:34:35 481754 mytable/unknown/006131ee-af8d-48b3-aafe-448bacd13fcc-0_15856-25-16025_20201125104200.parquet
2020-11-25 11:06:19 485536 mytable/unknown/0063330a-9682-48de-bc5d-b67fa13ec999-0_167-25-336_20201125104200.parquet
2020-11-25 11:08:17 483132 mytable/unknown/0064d620-68e8-43a6-b473-106275d47154-0_1228-25-1397_20201125104200.parquet
2020-11-25 11:19:00 481348 mytable/unknown/006639d5-5d63-4160-994b-6d966d851aff-0_6823-25-6992_20201125104200.parquet
2020-11-25 11:09:03 480887 mytable/unknown/00668c0c-a6b5-4312-93a4-6f248161b3b4-0_1641-25-1810_20201125104200.parquet
2020-11-25 11:41:54 480986 mytable/unknown/00693591-7e2b-4aef-9cf2-62dd61c414e7-0_20260-25-20429_20201125104200.parquet
2020-11-25 11:11:50 481486 mytable/unknown/0075b246-fa43-48f0-844f-6ccb6893c574-0_3105-25-3274_20201125104200.parquet
2020-11-25 11:06:46 480313 mytable/unknown/0075c608-0a47-4492-9a6f-0a9365d6719f-0_404-25-573_20201125104200.parquet
2020-11-25 11:46:12 482541 mytable/unknown/0078d5fe-340f-4ba9-a154-21caa6d74f81-0_22798-25-22967_20201125104200.parquet
2020-11-25 11:30:01 483351 mytable/unknown/0079fde5-5899-4efa-96a0-ddf1a3f8cd81-0_13166-25-13335_20201125104200.parquet
2020-11-25 11:13:19 482010 mytable/unknown/007c3edb-9d58-4236-9276-aa4afb5ff291-0_3889-25-4058_20201125104200.parquet
2020-11-25 11:15:51 481009 mytable/unknown/007d0204-1ed3-4632-a606-32905b38bcd7-0_5164-25-5333_20201125104200.parquet
2020-11-25 11:24:11 483761 mytable/unknown/007fdf3d-1ccc-42a7-b484-71bbf3bdbfe4-0_9718-25-9887_20201125104200.parquet
2020-11-25 11:34:31 481006 mytable/unknown/00892b02-98e9-4185-9904-9cee3e1ba363-0_15816-25-15985_20201125104200.parquet
2020-11-25 11:23:54 481104 mytable/unknown/008a3f28-dace-4b83-837d-31dd5a5dc752-0_9543-25-9712_20201125104200.parquet
2020-11-25 11:46:31 480697 mytable/unknown/008d05ab-ddea-41d8-9e19-b6c2759aba45-0_22969-25-23138_20201125104200.parquet
2020-11-25 11:38:18 481123 mytable/unknown/009ef3ad-7c36-4cd8-8a6c-b2a8f48f2833-0_18072-25-18241_20201125104200.parquet
```
...the list of parquet files continues (around 23000)
commits show
```
║ CommitTime │ Total Bytes Written │ Total Files Added │ Total Files Updated │ Total Partitions Written │ Total Records Written │ Total Update Records Written │ Total Errors ║
╠════════════════╪═════════════════════╪═══════════════════╪═════════════════════╪══════════════════════════╪═══════════════════════╪══════════════════════════════╪══════════════╣
║ 20201125104200 │ 10.4 GB │ 23164 │ 1 │ 1 │ 19967954 │ 3 │ 0 ║
╟────────────────┼─────────────────────┼───────────────────┼─────────────────────┼──────────────────────────┼───────────────────────┼──────────────────────────────┼──────────────╢
║ 20201125095231 │ 427.2 KB │ 0 │ 1 │ 1 │ 3 │ 3 │ 0 ║
╟────────────────┼─────────────────────┼───────────────────┼─────────────────────┼──────────────────────────┼───────────────────────┼──────────────────────────────┼──────────────╢
║ 20201125093753 │ 427.2 KB │ 1 │ 0 │ 1 │ 3 │ 0 │ 0 ║
╚════════════════════════════════╧════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════
```
commit showpartitions --commit 20201125104200 --desc true --limit 10
```
║ Partition Path │ Total Files Added │ Total Files Updated │ Total Records Inserted │ Total Records Updated │ Total Bytes Written │ Total Errors ║
╠════════════════╪═══════════════════╪═════════════════════╪════════════════════════╪═══════════════════════╪═════════════════════╪══════════════╣
║ unknown │ 23164 │ 1 │ 19967951 │ 3 │ 10.4 GB │ 0 ║
╚════════════════════════════════╧════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════
```
commit showfiles --commit 20201125104200
(there were 46350 lines returned from that cli command, just showing a few below)
```
║ Partition Path │ File ID │ Previous Commit │ Total Records Updated │ Total Records Written │ Total Bytes Written │ Total Errors │ File Size ║
╠════════════════╪════════════════════════════════════════╪═════════════════╪═══════════════════════╪═══════════════════════╪═════════════════════╪══════════════╪═══════════╣
║ unknown │ ee95426b-4708-4804-88a6-515ac04cbf66-0 │ 20201125095231 │ 3 │ 23825 │ 1545005 │ 0 │ 1545005 ║
╟────────────────┼────────────────────────────────────────┼─────────────────┼───────────────────────┼───────────────────────┼─────────────────────┼──────────────┼───────────╢
║ unknown │ 87762d9c-518e-48c7-a717-9f3d12ebf02d-0 │ null │ 0 │ 891 │ 483220 │ 0 │ 483220 ║
╟────────────────┼────────────────────────────────────────┼─────────────────┼───────────────────────┼───────────────────────┼─────────────────────┼──────────────┼───────────╢
║ unknown │ 74d4703a-f9fc-4c49-96d2-724fc1c225de-0 │ null │ 0 │ 898 │ 483704 │ 0 │ 483704 ║
╟────────────────┼────────────────────────────────────────┼─────────────────┼───────────────────────┼───────────────────────┼─────────────────────┼──────────────┼───────────╢
║ unknown │ efd84547-0899-4517-b3ce-3832f9acb2da-0 │ null │ 0 │ 892 │ 483882 │ 0 │ 483882 ║
╟────────────────┼────────────────────────────────────────┼─────────────────┼───────────────────────┼───────────────────────┼─────────────────────┼──────────────┼───────────╢
║ unknown │ 95f4567b-f94d-48b3-921a-5efefc69b42e-0 │ null │ 0 │ 870 │ 482258 │ 0 │ 482258 ║
╟────────────────┼────────────────────────────────────────┼─────────────────┼───────────────────────┼───────────────────────┼─────────────────────┼──────────────┼───────────╢
║ unknown │ 08b2c30d-a03e-4df8-9202-75a07932e337-0 │ null │ 0 │ 872 │ 481902 │ 0 │ 481902 ║
╟────────────────┼────────────────────────────────────────┼─────────────────┼───────────────────────┼───────────────────────┼─────────────────────┼──────────────┼───────────╢
║ unknown │ 161b5e44-b676-4b31-9301-ece6645cec46-0 │ null │ 0 │ 867 │ 482395 │ 0 │ 482395 ║
```
**stats wa**
```
║ CommitTime │ Total Upserted │ Total Written │ Write Amplification Factor ║
╠════════════════╪════════════════╪═══════════════╪════════════════════════════╣
║ 20201125093753 │ 0 │ 3 │ 0 ║
╟────────────────┼────────────────┼───────────────┼────────────────────────────╢
║ 20201125095231 │ 3 │ 3 │ 1.00 ║
╟────────────────┼────────────────┼───────────────┼────────────────────────────╢
║ 20201125104200 │ 3 │ 19967954 │ 6655984.50 ║
╟────────────────┼────────────────┼───────────────┼────────────────────────────╢
║ Total │ 6 │ 19967960 │ 3327993.25 ║
╚════════════════╧════════════════╧═══════════════╧════════════════════════════╝
```
**Expected behavior**
i would expect much less parquet files, each of larger size (ie around 110mb each)
**Environment Description**
Hudi version : 0.5.3
Spark version : 2.4.6
Hive version : 2.3.4
Hadoop version : 2.8.5
Storage (HDFS/S3/GCS..) : S3
Running on Docker? (yes/no) : no
COW table, spark datasource
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] bvaradar commented on issue #2393: [SUPPORT] excessive small files created
Posted by GitBox <gi...@apache.org>.
bvaradar commented on issue #2393:
URL: https://github.com/apache/hudi/issues/2393#issuecomment-752987079
@tooptoop4 : Hudi does not have access to incoming size of the files being loaded as it is managed by spark. The input dataframe is the interaction point for Hudi. With custom merge logic and payload size in memory different from that in file, it is not possible to get an accurate measure. I have opened an exploratory jira to look at this : https://issues.apache.org/jira/browse/HUDI-1501
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] bvaradar commented on issue #2393: [SUPPORT] excessive small files created
Posted by GitBox <gi...@apache.org>.
bvaradar commented on issue #2393:
URL: https://github.com/apache/hudi/issues/2393#issuecomment-752338459
Can you look at the log message "AvgRecordSize => " and see what value is getting printed. My suspicion is your second commit has comparatively very large record size (in bytes) when compared to the records in the next big batch. Hudi dynamically adapts to changing record size by leveraging record size seen in previous commits.
With 0.6.0, we have made the change to look for previous commits where number of records written per file was more than Small file limit (minimum size for a file to be not treated as small files). Is it possible to try 0.6.0 and see if you are having the same experience (this can still happen for you in 0.6.0 if the 3 records that you are writing is big enough to become greater than the small file size limit).
Added Jira to let user override this estimate : https://issues.apache.org/jira/browse/HUDI-1499
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] bvaradar closed issue #2393: [SUPPORT] excessive small files created
Posted by GitBox <gi...@apache.org>.
bvaradar closed issue #2393:
URL: https://github.com/apache/hudi/issues/2393
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] tooptoop4 commented on issue #2393: [SUPPORT] excessive small files created
Posted by GitBox <gi...@apache.org>.
tooptoop4 commented on issue #2393:
URL: https://github.com/apache/hudi/issues/2393#issuecomment-752452753
@bvaradar
1st run: INFO org.apache.hudi.table.HoodieCopyOnWriteTable - AvgRecordSize => 1024
2nd run: INFO org.apache.hudi.table.HoodieCopyOnWriteTable - AvgRecordSize => 145826
3rd run: INFO org.apache.hudi.table.HoodieCopyOnWriteTable - AvgRecordSize => 145835
rather than manually changing sizes, can hudi automatically look at incoming size of file being loaded (not caring about size in prior commits)?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org