You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@impala.apache.org by "Csaba Ringhofer (Code Review)" <ge...@cloudera.org> on 2026/07/15 15:49:07 UTC
[Impala-ASF-CR] IMPALA-15184: Switch tpcds json from bzip to LZ4
Csaba Ringhofer has uploaded this change for review. ( http://gerrit.cloudera.org:8080/24586
Change subject: IMPALA-15184: Switch tpcds json from bzip to LZ4
......................................................................
IMPALA-15184: Switch tpcds json from bzip to LZ4
bzip is very slow, and it is enough to test it on
smaller data (already covered in functional text).
Switching to LZ4 (which was both untested and should
be more common) decreases loading time of tpcds on all
file formats from 20 min to 10 min.
This was not the longest leg so far, so may not not
show up in total dataload time, but becomes important
in combination with other optimizations (e.g. increased
parallelism).
Change-Id: Ia5e787619cc9f2b1cd30d40440aecd20fcef4117
---
M testdata/bin/generate-schema-statements.py
M testdata/workloads/tpcds/tpcds_core.csv
2 files changed, 2 insertions(+), 1 deletion(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/86/24586/1
--
To view, visit http://gerrit.cloudera.org:8080/24586
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: Ia5e787619cc9f2b1cd30d40440aecd20fcef4117
Gerrit-Change-Number: 24586
Gerrit-PatchSet: 1
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
[Impala-ASF-CR] IMPALA-15184: Switch tpcds json from bzip to zstd
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24586 )
Change subject: IMPALA-15184: Switch tpcds json from bzip to zstd
......................................................................
Patch Set 3:
Build Failed
https://jenkins.impala.io/job/gerrit-code-review-checks/23264/ : Initial code review checks failed. See linked job for details on the failure.
--
To view, visit http://gerrit.cloudera.org:8080/24586
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia5e787619cc9f2b1cd30d40440aecd20fcef4117
Gerrit-Change-Number: 24586
Gerrit-PatchSet: 3
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Comment-Date: Thu, 16 Jul 2026 08:35:57 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-15184: Switch tpcds json from bzip to zstd
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24586 )
Change subject: IMPALA-15184: Switch tpcds json from bzip to zstd
......................................................................
Patch Set 3: Verified-1
Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/13747/
--
To view, visit http://gerrit.cloudera.org:8080/24586
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia5e787619cc9f2b1cd30d40440aecd20fcef4117
Gerrit-Change-Number: 24586
Gerrit-PatchSet: 3
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Comment-Date: Thu, 16 Jul 2026 12:40:38 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-15184: Switch tpcds json from bzip to zstd
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24586 )
Change subject: IMPALA-15184: Switch tpcds json from bzip to zstd
......................................................................
Patch Set 4: Verified+1
--
To view, visit http://gerrit.cloudera.org:8080/24586
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia5e787619cc9f2b1cd30d40440aecd20fcef4117
Gerrit-Change-Number: 24586
Gerrit-PatchSet: 4
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Comment-Date: Thu, 16 Jul 2026 20:43:53 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-15184: Switch tpcds json from bzip to zstd
Posted by "Csaba Ringhofer (Code Review)" <ge...@cloudera.org>.
Hello Joe McDonnell, Michael Smith, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/24586
to look at the new patch set (#3).
Change subject: IMPALA-15184: Switch tpcds json from bzip to zstd
......................................................................
IMPALA-15184: Switch tpcds json from bzip to zstd
bzip is very slow, and it is enough to test it on
smaller data (already covered in functional text).
Switching to zstd, which did not have its own db.
From dataload logs before change:
Loading TPC-H data OK (Took: 5 min 39 sec)
Loading TPC-DS data OK (Took: 22 min 33 sec)
Loading functional-query data OK (Took: 30 min 43 sec)
After change:
Loading TPC-H data OK (Took: 5 min 22 sec)
Loading TPC-DS data OK (Took: 13 min 49 sec)
Loading functional-query data OK (Took: 27 min 30 sec)
Total dataload time is decreased from ~35m to ~30m.
The longest leg functional-query also decreased, likely
due to less contention on the max 2 parallel Hive queries
(see IMPALA-15185 for details on Hive parallelism).
Change-Id: Ia5e787619cc9f2b1cd30d40440aecd20fcef4117
---
M testdata/bin/generate-schema-statements.py
M testdata/workloads/tpcds/tpcds_core.csv
2 files changed, 2 insertions(+), 1 deletion(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/86/24586/3
--
To view, visit http://gerrit.cloudera.org:8080/24586
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ia5e787619cc9f2b1cd30d40440aecd20fcef4117
Gerrit-Change-Number: 24586
Gerrit-PatchSet: 3
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
[Impala-ASF-CR] IMPALA-15184: Switch tpcds json from bzip to LZ4
Posted by "Michael Smith (Code Review)" <ge...@cloudera.org>.
Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/24586 )
Change subject: IMPALA-15184: Switch tpcds json from bzip to LZ4
......................................................................
Patch Set 2: Code-Review+1
I'd also be interested in how disk usage changes.
--
To view, visit http://gerrit.cloudera.org:8080/24586
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia5e787619cc9f2b1cd30d40440aecd20fcef4117
Gerrit-Change-Number: 24586
Gerrit-PatchSet: 2
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Comment-Date: Wed, 15 Jul 2026 20:09:14 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-15184: Switch tpcds json from bzip to LZ4
Posted by "Joe McDonnell (Code Review)" <ge...@cloudera.org>.
Joe McDonnell has posted comments on this change. ( http://gerrit.cloudera.org:8080/24586 )
Change subject: IMPALA-15184: Switch tpcds json from bzip to LZ4
......................................................................
Patch Set 2:
> > > It makes sense to me to use a faster compression algorithm than
> > > bzip. Does the disk usage change much for this table? Another
> > > option is ZSTD which is almost as fast to compress as LZ4 with
> a
> > > higher compression ratio.
> >
> > Checked locally, writing looks even a bit faster for zstd than
> for
> > lz4.
> >
> > sizes of store_sales (1 replica):
>
> > > It makes sense to me to use a faster compression algorithm than
> > > bzip. Does the disk usage change much for this table? Another
> > > option is ZSTD which is almost as fast to compress as LZ4 with
> a
> > > higher compression ratio.
> >
> > Checked locally, writing looks even a bit faster for zstd than
> for
> > lz4.
> >
> > sizes of store_sales (1 replica):
>
> bzip: 123 MB
> zstd: 186 MB
> lz4 : 343 MB
>
> lz4 is indeed much bigger
> I vote for using zstd instead
Yeah, I've seen cases in the past where the IO is the bottleneck on these machines with EBS. I think ZSTD is fast enough that IO to EBS becomes the bottleneck, so the extra compression gives it an edge over LZ4. ZSTD sounds great to me.
--
To view, visit http://gerrit.cloudera.org:8080/24586
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia5e787619cc9f2b1cd30d40440aecd20fcef4117
Gerrit-Change-Number: 24586
Gerrit-PatchSet: 2
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Comment-Date: Wed, 15 Jul 2026 21:44:19 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-15184: Switch tpcds json from bzip to zstd
Posted by "Joe McDonnell (Code Review)" <ge...@cloudera.org>.
Joe McDonnell has posted comments on this change. ( http://gerrit.cloudera.org:8080/24586 )
Change subject: IMPALA-15184: Switch tpcds json from bzip to zstd
......................................................................
Patch Set 3: Code-Review+1
This looks good to me. The test issue doesn't look related.
--
To view, visit http://gerrit.cloudera.org:8080/24586
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia5e787619cc9f2b1cd30d40440aecd20fcef4117
Gerrit-Change-Number: 24586
Gerrit-PatchSet: 3
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Comment-Date: Thu, 16 Jul 2026 15:19:01 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-15184: Switch tpcds json from bzip to LZ4
Posted by "Joe McDonnell (Code Review)" <ge...@cloudera.org>.
Joe McDonnell has posted comments on this change. ( http://gerrit.cloudera.org:8080/24586 )
Change subject: IMPALA-15184: Switch tpcds json from bzip to LZ4
......................................................................
Patch Set 2: Code-Review+1
It makes sense to me to use a faster compression algorithm than bzip. Does the disk usage change much for this table? Another option is ZSTD which is almost as fast to compress as LZ4 with a higher compression ratio.
--
To view, visit http://gerrit.cloudera.org:8080/24586
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia5e787619cc9f2b1cd30d40440aecd20fcef4117
Gerrit-Change-Number: 24586
Gerrit-PatchSet: 2
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Comment-Date: Wed, 15 Jul 2026 19:06:39 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-15184: Switch tpcds json from bzip to zstd
Posted by "Csaba Ringhofer (Code Review)" <ge...@cloudera.org>.
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/24586 )
Change subject: IMPALA-15184: Switch tpcds json from bzip to zstd
......................................................................
Patch Set 3:
> > > > > It makes sense to me to use a faster compression algorithm
> > than
> > > > > bzip. Does the disk usage change much for this table?
> Another
> > > > > option is ZSTD which is almost as fast to compress as LZ4
> > with
> > > a
> > > > > higher compression ratio.
> > > >
> > > > Checked locally, writing looks even a bit faster for zstd
> than
> > > for
> > > > lz4.
> > > >
> > > > sizes of store_sales (1 replica):
> > >
> > > > > It makes sense to me to use a faster compression algorithm
> > than
> > > > > bzip. Does the disk usage change much for this table?
> Another
> > > > > option is ZSTD which is almost as fast to compress as LZ4
> > with
> > > a
> > > > > higher compression ratio.
> > > >
> > > > Checked locally, writing looks even a bit faster for zstd
> than
> > > for
> > > > lz4.
> > > >
> > > > sizes of store_sales (1 replica):
> > >
> > > bzip: 123 MB
> > > zstd: 186 MB
> > > lz4 : 343 MB
> > >
> > > lz4 is indeed much bigger
> > > I vote for using zstd instead
> >
> > Yeah, I've seen cases in the past where the IO is the bottleneck
> on
> > these machines with EBS. I think ZSTD is fast enough that IO to
> EBS
> > becomes the bottleneck, so the extra compression gives it an edge
> > over LZ4. ZSTD sounds great to me.
>
> realized that we actually never use non-parquet for tpcds
> this code is misleading:
> https://github.com/apache/impala/blob/371cad3e015e48a80b32de0104491b2fd4696e29/tests/query_test/test_tpcds_queries.py#L49
>
> due to workload_exploration_strategy mechanism get_workload() is
> tpcds
>
> https://github.com/apache/impala/blob/371cad3e015e48a80b32de0104491b2fd4696e29/bin/run-all-tests.sh#L189
I miss copypasted something.
get_workload() is tpcds -> if get_workload() is tpccds, exploration_strategy() is always core
--
To view, visit http://gerrit.cloudera.org:8080/24586
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia5e787619cc9f2b1cd30d40440aecd20fcef4117
Gerrit-Change-Number: 24586
Gerrit-PatchSet: 3
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Comment-Date: Thu, 16 Jul 2026 09:28:15 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-15184: Switch tpcds json from bzip to zstd
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24586 )
Change subject: IMPALA-15184: Switch tpcds json from bzip to zstd
......................................................................
Patch Set 4: Code-Review+2
--
To view, visit http://gerrit.cloudera.org:8080/24586
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia5e787619cc9f2b1cd30d40440aecd20fcef4117
Gerrit-Change-Number: 24586
Gerrit-PatchSet: 4
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Comment-Date: Thu, 16 Jul 2026 16:12:55 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-15184: Switch tpcds json from bzip to zstd
Posted by "Michael Smith (Code Review)" <ge...@cloudera.org>.
Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/24586 )
Change subject: IMPALA-15184: Switch tpcds json from bzip to zstd
......................................................................
Patch Set 3: Code-Review+2
--
To view, visit http://gerrit.cloudera.org:8080/24586
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia5e787619cc9f2b1cd30d40440aecd20fcef4117
Gerrit-Change-Number: 24586
Gerrit-PatchSet: 3
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Comment-Date: Thu, 16 Jul 2026 16:12:42 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-15184: Switch tpcds json from bzip to zstd
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24586 )
Change subject: IMPALA-15184: Switch tpcds json from bzip to zstd
......................................................................
Patch Set 4:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/13752/ DRY_RUN=false
--
To view, visit http://gerrit.cloudera.org:8080/24586
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia5e787619cc9f2b1cd30d40440aecd20fcef4117
Gerrit-Change-Number: 24586
Gerrit-PatchSet: 4
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Comment-Date: Thu, 16 Jul 2026 16:12:56 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-15184: Switch tpcds json from bzip to zstd
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/24586 )
Change subject: IMPALA-15184: Switch tpcds json from bzip to zstd
......................................................................
IMPALA-15184: Switch tpcds json from bzip to zstd
bzip is very slow, and it is enough to test it on
smaller data (already covered in functional text).
Switching to zstd, which did not have its own db.
From dataload logs before change:
Loading TPC-H data OK (Took: 5 min 39 sec)
Loading TPC-DS data OK (Took: 22 min 33 sec)
Loading functional-query data OK (Took: 30 min 43 sec)
After change:
Loading TPC-H data OK (Took: 5 min 22 sec)
Loading TPC-DS data OK (Took: 13 min 49 sec)
Loading functional-query data OK (Took: 27 min 30 sec)
Total dataload time is decreased from ~35m to ~30m.
The longest leg functional-query also decreased, likely
due to less contention on the max 2 parallel Hive queries
(see IMPALA-15185 for details on Hive parallelism).
Change-Id: Ia5e787619cc9f2b1cd30d40440aecd20fcef4117
Reviewed-on: http://gerrit.cloudera.org:8080/24586
Reviewed-by: Impala Public Jenkins <im...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
---
M testdata/bin/generate-schema-statements.py
M testdata/workloads/tpcds/tpcds_core.csv
2 files changed, 2 insertions(+), 1 deletion(-)
Approvals:
Impala Public Jenkins: Looks good to me, approved; Verified
--
To view, visit http://gerrit.cloudera.org:8080/24586
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: Ia5e787619cc9f2b1cd30d40440aecd20fcef4117
Gerrit-Change-Number: 24586
Gerrit-PatchSet: 5
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
[Impala-ASF-CR] IMPALA-15184: Switch tpcds json from bzip to zstd
Posted by "Csaba Ringhofer (Code Review)" <ge...@cloudera.org>.
Csaba Ringhofer has removed a vote on this change.
Change subject: IMPALA-15184: Switch tpcds json from bzip to zstd
......................................................................
Removed Verified-1 by Impala Public Jenkins <im...@cloudera.com>
--
To view, visit http://gerrit.cloudera.org:8080/24586
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: deleteVote
Gerrit-Change-Id: Ia5e787619cc9f2b1cd30d40440aecd20fcef4117
Gerrit-Change-Number: 24586
Gerrit-PatchSet: 3
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
[Impala-ASF-CR] IMPALA-15184: Switch tpcds json from bzip to zstd
Posted by "Laszlo Gaal (Code Review)" <ge...@cloudera.org>.
Laszlo Gaal has posted comments on this change. ( http://gerrit.cloudera.org:8080/24586 )
Change subject: IMPALA-15184: Switch tpcds json from bzip to zstd
......................................................................
Patch Set 3:
> Patch Set 2:
>
> > > > It makes sense to me to use a faster compression algorithm than
> > > > bzip. Does the disk usage change much for this table? Another
> > > > option is ZSTD which is almost as fast to compress as LZ4 with
> > a
> > > > higher compression ratio.
> > >
> > > Checked locally, writing looks even a bit faster for zstd than
> > for
> > > lz4.
> > >
> > > sizes of store_sales (1 replica):
> >
> > > > It makes sense to me to use a faster compression algorithm than
> > > > bzip. Does the disk usage change much for this table? Another
> > > > option is ZSTD which is almost as fast to compress as LZ4 with
> > a
> > > > higher compression ratio.
> > >
> > > Checked locally, writing looks even a bit faster for zstd than
> > for
> > > lz4.
> > >
> > > sizes of store_sales (1 replica):
> >
> > bzip: 123 MB
> > zstd: 186 MB
> > lz4 : 343 MB
> >
> > lz4 is indeed much bigger
> > I vote for using zstd instead
>
> Yeah, I've seen cases in the past where the IO is the bottleneck on these machines with EBS. I think ZSTD is fast enough that IO to EBS becomes the bottleneck, so the extra compression gives it an edge over LZ4. ZSTD sounds great to me.
re/IO bandwidth:
- local runs should not be a problem, as most of the developers run NVMe drives on their workstations. Even a regular SATA SSD can usually read/write at ~500 GB/sec unless you're dumping tens of gigabytes.
- EBS drives on our CI infrastructure are provisioned for 250 MB/sec bandwidth, which I found to be a cost-effective tradeoff between performance and extra cost (GP3 EBS volumes provide 125 GB/sec base bandwidth, which can be increased at extra cost). This can be increased as long as the extra bandwidth results in reduced build times, offsetting the extra cost by saving money on EC2 VM instance charges.
--
To view, visit http://gerrit.cloudera.org:8080/24586
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia5e787619cc9f2b1cd30d40440aecd20fcef4117
Gerrit-Change-Number: 24586
Gerrit-PatchSet: 3
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Comment-Date: Thu, 16 Jul 2026 10:20:32 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-15184: Switch tpcds json from bzip to LZ4
Posted by "Csaba Ringhofer (Code Review)" <ge...@cloudera.org>.
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/24586 )
Change subject: IMPALA-15184: Switch tpcds json from bzip to LZ4
......................................................................
Patch Set 2:
> It makes sense to me to use a faster compression algorithm than
> bzip. Does the disk usage change much for this table? Another
> option is ZSTD which is almost as fast to compress as LZ4 with a
> higher compression ratio.
Checked locally, writing looks even a bit faster for zstd than for lz4.
sizes of store_sales (1 replica):
--
To view, visit http://gerrit.cloudera.org:8080/24586
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia5e787619cc9f2b1cd30d40440aecd20fcef4117
Gerrit-Change-Number: 24586
Gerrit-PatchSet: 2
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Comment-Date: Wed, 15 Jul 2026 21:31:52 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-15184: Switch tpcds json from bzip to zstd
Posted by "Csaba Ringhofer (Code Review)" <ge...@cloudera.org>.
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/24586 )
Change subject: IMPALA-15184: Switch tpcds json from bzip to zstd
......................................................................
Patch Set 3:
> > > > It makes sense to me to use a faster compression algorithm
> than
> > > > bzip. Does the disk usage change much for this table? Another
> > > > option is ZSTD which is almost as fast to compress as LZ4
> with
> > a
> > > > higher compression ratio.
> > >
> > > Checked locally, writing looks even a bit faster for zstd than
> > for
> > > lz4.
> > >
> > > sizes of store_sales (1 replica):
> >
> > > > It makes sense to me to use a faster compression algorithm
> than
> > > > bzip. Does the disk usage change much for this table? Another
> > > > option is ZSTD which is almost as fast to compress as LZ4
> with
> > a
> > > > higher compression ratio.
> > >
> > > Checked locally, writing looks even a bit faster for zstd than
> > for
> > > lz4.
> > >
> > > sizes of store_sales (1 replica):
> >
> > bzip: 123 MB
> > zstd: 186 MB
> > lz4 : 343 MB
> >
> > lz4 is indeed much bigger
> > I vote for using zstd instead
>
> Yeah, I've seen cases in the past where the IO is the bottleneck on
> these machines with EBS. I think ZSTD is fast enough that IO to EBS
> becomes the bottleneck, so the extra compression gives it an edge
> over LZ4. ZSTD sounds great to me.
realized that we actually never use non-parquet for tpcds
this code is misleading:
https://github.com/apache/impala/blob/371cad3e015e48a80b32de0104491b2fd4696e29/tests/query_test/test_tpcds_queries.py#L49
due to workload_exploration_strategy mechanism get_workload() is tpcds
https://github.com/apache/impala/blob/371cad3e015e48a80b32de0104491b2fd4696e29/bin/run-all-tests.sh#L189
--
To view, visit http://gerrit.cloudera.org:8080/24586
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia5e787619cc9f2b1cd30d40440aecd20fcef4117
Gerrit-Change-Number: 24586
Gerrit-PatchSet: 3
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Comment-Date: Thu, 16 Jul 2026 09:26:04 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-15184: Switch tpcds json from bzip to LZ4
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24586 )
Change subject: IMPALA-15184: Switch tpcds json from bzip to LZ4
......................................................................
Patch Set 1:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/23249/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/24586
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia5e787619cc9f2b1cd30d40440aecd20fcef4117
Gerrit-Change-Number: 24586
Gerrit-PatchSet: 1
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Wed, 15 Jul 2026 16:12:55 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-15184: Switch tpcds json from bzip to LZ4
Posted by "Csaba Ringhofer (Code Review)" <ge...@cloudera.org>.
Hello Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/24586
to look at the new patch set (#2).
Change subject: IMPALA-15184: Switch tpcds json from bzip to LZ4
......................................................................
IMPALA-15184: Switch tpcds json from bzip to LZ4
bzip is very slow, and it is enough to test it on
smaller data (already covered in functional text).
Switching to LZ4, which did not have its own db.
From dataload logs before change:
Loading TPC-H data OK (Took: 5 min 39 sec)
Loading TPC-DS data OK (Took: 22 min 33 sec)
Loading functional-query data OK (Took: 30 min 43 sec)
After change:
Loading TPC-H data OK (Took: 5 min 32 sec)
Loading TPC-DS data OK (Took: 13 min 40 sec)
Loading functional-query data OK (Took: 27 min 10 sec)
Total dataload time is decreased from ~35m to ~30m.
The longest leg functional-query also decreased, likely
due to less contention on the max 2 parallel Hive queries
(see IMPALA-15185 for details on Hive parallelism).
Change-Id: Ia5e787619cc9f2b1cd30d40440aecd20fcef4117
---
M testdata/bin/generate-schema-statements.py
M testdata/workloads/tpcds/tpcds_core.csv
2 files changed, 2 insertions(+), 1 deletion(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/86/24586/2
--
To view, visit http://gerrit.cloudera.org:8080/24586
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ia5e787619cc9f2b1cd30d40440aecd20fcef4117
Gerrit-Change-Number: 24586
Gerrit-PatchSet: 2
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
[Impala-ASF-CR] IMPALA-15184: Switch tpcds json from bzip to zstd
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24586 )
Change subject: IMPALA-15184: Switch tpcds json from bzip to zstd
......................................................................
Patch Set 3:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/13747/ DRY_RUN=true
--
To view, visit http://gerrit.cloudera.org:8080/24586
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia5e787619cc9f2b1cd30d40440aecd20fcef4117
Gerrit-Change-Number: 24586
Gerrit-PatchSet: 3
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Comment-Date: Thu, 16 Jul 2026 08:12:46 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-15184: Switch tpcds json from bzip to LZ4
Posted by "Michael Smith (Code Review)" <ge...@cloudera.org>.
Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/24586 )
Change subject: IMPALA-15184: Switch tpcds json from bzip to LZ4
......................................................................
Patch Set 2: Code-Review+2
Works for me. zstd would also be reasonable.
--
To view, visit http://gerrit.cloudera.org:8080/24586
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia5e787619cc9f2b1cd30d40440aecd20fcef4117
Gerrit-Change-Number: 24586
Gerrit-PatchSet: 2
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Comment-Date: Wed, 15 Jul 2026 19:48:20 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-15184: Switch tpcds json from bzip to LZ4
Posted by "Csaba Ringhofer (Code Review)" <ge...@cloudera.org>.
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/24586 )
Change subject: IMPALA-15184: Switch tpcds json from bzip to LZ4
......................................................................
Patch Set 2:
> > It makes sense to me to use a faster compression algorithm than
> > bzip. Does the disk usage change much for this table? Another
> > option is ZSTD which is almost as fast to compress as LZ4 with a
> > higher compression ratio.
>
> Checked locally, writing looks even a bit faster for zstd than for
> lz4.
>
> sizes of store_sales (1 replica):
> > It makes sense to me to use a faster compression algorithm than
> > bzip. Does the disk usage change much for this table? Another
> > option is ZSTD which is almost as fast to compress as LZ4 with a
> > higher compression ratio.
>
> Checked locally, writing looks even a bit faster for zstd than for
> lz4.
>
> sizes of store_sales (1 replica):
bzip: 123 MB
zstd: 186 MB
lz4 : 343 MB
lz4 is indeed much bigger
I vote for using zstd instead
--
To view, visit http://gerrit.cloudera.org:8080/24586
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia5e787619cc9f2b1cd30d40440aecd20fcef4117
Gerrit-Change-Number: 24586
Gerrit-PatchSet: 2
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Comment-Date: Wed, 15 Jul 2026 21:34:18 +0000
Gerrit-HasComments: No