You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@parquet.apache.org by "mapleFU (via GitHub)" <gi...@apache.org> on 2023/04/04 05:09:25 UTC
[GitHub] [parquet-format] mapleFU commented on pull request #195: PARQUET-2256: Add BloomFilter Compression
mapleFU commented on PR #195:
URL: https://github.com/apache/parquet-format/pull/195#issuecomment-1495352894
Hi @gszadovszky @wgtmac . I was busy these weeks, so maybe a bit late here.
Though https://issues.apache.org/jira/browse/HUDI-558 says that using compression on BloomFilter works well, after take some experiments, I found that:
1. When BloomFilter is "sparse" ( which means filter is big, and filling value is small ), compression may works well
2. When BloomFilter hash size is similar with ndv, the zstd, lz4, snappy don't works well.
```
size: 100, fpp: 0.1
Hash Ndv: 100
Size before compress: 144 zstd compressed length:153
lz4 compressed length:141
snappy compressed length:143
size: 1000, fpp: 0.1
Hash Ndv: 1000
Size before compress: 1040 zstd compressed length:1050
lz4 compressed length:1040
snappy compressed length:1040
size: 10000, fpp: 0.1
Hash Ndv: 10000
Size before compress: 8209 zstd compressed length:7314
lz4 compressed length:8238
snappy compressed length:8209
size: 100000, fpp: 0.1
Hash Ndv: 100000
Size before compress: 131089 zstd compressed length:131104
lz4 compressed length:131599
snappy compressed length:131094
size: 500000, fpp: 0.1
Hash Ndv: 500000
Size before compress: 524305 zstd compressed length:506198
lz4 compressed length:526357
snappy compressed length:524328
size: 100, fpp: 0.05
Hash Ndv: 100
Size before compress: 144 zstd compressed length:153
lz4 compressed length:141
snappy compressed length:143
size: 1000, fpp: 0.05
Hash Ndv: 1000
Size before compress: 1040 zstd compressed length:1050
lz4 compressed length:1040
snappy compressed length:1040
size: 10000, fpp: 0.05
Hash Ndv: 10000
Size before compress: 16401 zstd compressed length:16411
lz4 compressed length:16462
snappy compressed length:16402
size: 100000, fpp: 0.05
Hash Ndv: 100000
Size before compress: 131089 zstd compressed length:131104
lz4 compressed length:131599
snappy compressed length:131094
size: 500000, fpp: 0.05
Hash Ndv: 500000
Size before compress: 524305 zstd compressed length:506198
lz4 compressed length:526357
snappy compressed length:524328
size: 100, fpp: 0.01
Hash Ndv: 100
Size before compress: 144 zstd compressed length:153
lz4 compressed length:141
snappy compressed length:143
size: 1000, fpp: 0.01
Hash Ndv: 1000
Size before compress: 2064 zstd compressed length:2074
lz4 compressed length:2068
snappy compressed length:2064
size: 10000, fpp: 0.01
Hash Ndv: 10000
Size before compress: 16401 zstd compressed length:16411
lz4 compressed length:16462
snappy compressed length:16402
size: 100000, fpp: 0.01
Hash Ndv: 100000
Size before compress: 131089 zstd compressed length:131104
lz4 compressed length:131599
snappy compressed length:131094
size: 500000, fpp: 0.01
Hash Ndv: 500000
Size before compress: 1048594 zstd compressed length:1007916
lz4 compressed length:1052703
snappy compressed length:1048641
size: 100, fpp: 0.005
Hash Ndv: 100
Size before compress: 272 zstd compressed length:282
lz4 compressed length:269
snappy compressed length:272
size: 1000, fpp: 0.005
Hash Ndv: 1000
Size before compress: 2064 zstd compressed length:2074
lz4 compressed length:2068
snappy compressed length:2064
size: 10000, fpp: 0.005
Hash Ndv: 10000
Size before compress: 16401 zstd compressed length:16411
lz4 compressed length:16462
snappy compressed length:16402
size: 100000, fpp: 0.005
Hash Ndv: 100000
Size before compress: 262161 zstd compressed length:238073
lz4 compressed length:263185
snappy compressed length:262172
size: 500000, fpp: 0.005
Hash Ndv: 500000
Size before compress: 1048594 zstd compressed length:1007916
lz4 compressed length:1052703
snappy compressed length:1048641
```
So I think maybe it's not a good idea for compress SPBF if ndv estimation is great. When size estimation get worse ( if estimated size is 8 times larger than real hash-value size), the compression rate will goes to 50% for zstd, lz4.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org