You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2021/12/13 23:21:51 UTC
[GitHub] [hudi] nochimow opened a new issue #4299: [SUPPORT] Upsert performance severed decreased after 3 years of data loading
nochimow opened a new issue #4299:
URL: https://github.com/apache/hudi/issues/4299
Hi there,
I'm currently facing some performance gaps in one specific table after we load 3 years of data.
Our typical cenario is the following:
Ingestion of 57 avro files (stored on S3) with typical sizes of 70-128MB giving 2,75GB of input data. We create a spark data-frame loading all these files and writing into Hudi.
This data is equal 186 million of rows.
The schema of the table is composed of 7 string columns, 2 BigInt columns, partitioned by 3 String columns (Day, Month, Year).
We also know that this table may have 5 to 10% of updates. (The data comes from a CDC engine)
In the beginning, the same load duration was < 20minutes, but after loading 3 years of data, the duration increased to 1,5hours, and after we upgrade hudi from 0.8 to 0.10 the duration decreased to 1,25 hours in average.
I'm looking for any tuning tips that may help decreasing the load duration even more, and try to let this time more stable considering the data growth over time.
**Other useful information:**
**Hoodie configs:**
"hoodie.datasource.write.keygenerator.class": "org.apache.hudi.keygen.ComplexKeyGenerator",
"hoodie.datasource.write.payload.class": "org.apache.hudi.common.model.DefaultHoodieRecordPayload",
"hoodie.datasource.hive_sync.partition_extractor_class": "org.apache.hudi.hive.MultiPartKeysValueExtractor",
"hoodie.table.name": table_name,
"hoodie.datasource.write.recordkey.field": IDX_COL,
"hoodie.datasource.write.partitionpath.field": pks,
"hoodie.datasource.write.hive_style_partitioning": "true",
"hoodie.datasource.write.precombine.field": tiebreaker,
"hoodie.datasource.write.operation": operation,
"hoodie.write.concurrency.mode": "single_writer",
"hoodie.cleaner.commits.retained": 1,
"hoodie.fail.on.timeline.archiving": False,
"hoodie.keep.max.commits": 3,
"hoodie.keep.min.commits": 2,
"hoodie.bloom.index.use.caching": True,
"hoodie.parquet.compression.codec": "snappy"
**Environment Description**
AWS Glue Job
**Hudi version :**
0.10
**Spark version :**
"Spark 2.4 - Python 3 with improved job times (Glue Version 2.0)"
**Storage (HDFS/S3/GCS..) :**
S3
**Running on Docker? (yes/no) :**
No
**Additional context:**
Infrastructure: Glue Job + S3
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xushiyan commented on issue #4299: [SUPPORT] Upsert performance decreased after 3 years of data loading
Posted by GitBox <gi...@apache.org>.
xushiyan commented on issue #4299:
URL: https://github.com/apache/hudi/issues/4299#issuecomment-1015089248
@XuQianJin-Stars can you help investigate this issue please? thank you.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nochimow edited a comment on issue #4299: [SUPPORT] Upsert performance decreased after 3 years of data loading
Posted by GitBox <gi...@apache.org>.
nochimow edited a comment on issue #4299:
URL: https://github.com/apache/hudi/issues/4299#issuecomment-1006558081
Including the results of the "stats wa" command of hudi-cli for this table:
CommitTime │ Total Upserted │ Total Written │ Write Amplification Factor
20220114003510183 │ 58350 │ 2536170743 │ 43464.79
20220114063428309 │ 2431165 │ 1820888870 │ 748.98
20220114123435503 │ 1753626 │ 1486216962 │ 847.51
Total │ 4243141 │ 5843276575 │ 1377.11
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] ChiehFu commented on issue #4299: [SUPPORT] Upsert performance decreased after 3 years of data loading
Posted by GitBox <gi...@apache.org>.
ChiehFu commented on issue #4299:
URL: https://github.com/apache/hudi/issues/4299#issuecomment-1024642686
@XuQianJin-Stars I also noticed a similar performance issue on Upsert in 0.10.0, please see my post #4682.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] XuQianJin-Stars commented on issue #4299: [SUPPORT] Upsert performance decreased after 3 years of data loading
Posted by GitBox <gi...@apache.org>.
XuQianJin-Stars commented on issue #4299:
URL: https://github.com/apache/hudi/issues/4299#issuecomment-1015119793
> @XuQianJin-Stars can you help investigate this issue please? thank you.
well, I will follow this question later.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] YuweiXiao commented on issue #4299: [SUPPORT] Upsert performance decreased after 3 years of data loading
Posted by GitBox <gi...@apache.org>.
YuweiXiao commented on issue #4299:
URL: https://github.com/apache/hudi/issues/4299#issuecomment-993557836
try to limit the parquet file size.
`hoodie.parquet.small.file.limit = 10485760` 10MB
also, maybe try the hbase index, should perform better than Bloom Filter
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nochimow commented on issue #4299: [SUPPORT] Upsert performance decreased after 3 years of data loading
Posted by GitBox <gi...@apache.org>.
nochimow commented on issue #4299:
URL: https://github.com/apache/hudi/issues/4299#issuecomment-993533012
Yes, this case is a CoW table.
I really wanted to check if there was some kind of tune on this case before going to MoR, because the read performance is also important for us.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nochimow commented on issue #4299: [SUPPORT] Upsert performance decreased after 3 years of data loading
Posted by GitBox <gi...@apache.org>.
nochimow commented on issue #4299:
URL: https://github.com/apache/hudi/issues/4299#issuecomment-999547324
Hello,
Tried changing the hoodie.parquet.small.file.limit parameter, but this made no difference in our case.
@nsivabalan, got the sparkui from the most time consuming step (41min).
If you need something more specific let me know.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on issue #4299: [SUPPORT] Upsert performance decreased after 3 years of data loading
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on issue #4299:
URL: https://github.com/apache/hudi/issues/4299#issuecomment-997049895
would you mind posting spark stages UI as well. We can see if indexing or small file handling is taking lot of time and tweak some params.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nochimow edited a comment on issue #4299: [SUPPORT] Upsert performance decreased after 3 years of data loading
Posted by GitBox <gi...@apache.org>.
nochimow edited a comment on issue #4299:
URL: https://github.com/apache/hudi/issues/4299#issuecomment-1006558081
Including the results of the "stats wa" command of hudi-cli for this table:
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] YuweiXiao edited a comment on issue #4299: [SUPPORT] Upsert performance decreased after 3 years of data loading
Posted by GitBox <gi...@apache.org>.
YuweiXiao edited a comment on issue #4299:
URL: https://github.com/apache/hudi/issues/4299#issuecomment-993557836
try to limit the parquet file size.
`hoodie.parquet.small.file.limit = 10485760` 10MB
also, maybe try the hbase index, should perform better than Bloom Filter
and add a partition key if possible, which can save indexing time
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] YuweiXiao commented on issue #4299: [SUPPORT] Upsert performance decreased after 3 years of data loading
Posted by GitBox <gi...@apache.org>.
YuweiXiao commented on issue #4299:
URL: https://github.com/apache/hudi/issues/4299#issuecomment-993455521
looking at the spark ui figure, seems there is severe write amplification (38 min writing).
are you using COW table? maybe try MOR to enable log write.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nochimow commented on issue #4299: [SUPPORT] Upsert performance decreased after 3 years of data loading
Posted by GitBox <gi...@apache.org>.
nochimow commented on issue #4299:
URL: https://github.com/apache/hudi/issues/4299#issuecomment-1006558081
Hi,
Still waiting for some updates on this case.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nochimow edited a comment on issue #4299: [SUPPORT] Upsert performance decreased after 3 years of data loading
Posted by GitBox <gi...@apache.org>.
nochimow edited a comment on issue #4299:
URL: https://github.com/apache/hudi/issues/4299#issuecomment-999547324
Hello,
Tried changing the hoodie.parquet.small.file.limit parameter, but this made no difference in our case.
@nsivabalan, got the sparkui from the most time consuming step (41min).
If you need something more specific let me know.
[tasks.csv](https://github.com/apache/hudi/files/7762762/tasks.csv)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org