You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "zhuqi-lucas (via GitHub)" <gi...@apache.org> on 2025/07/17 03:32:24 UTC

[PR] benchmark: Add parquet h2o support [datafusion]

zhuqi-lucas opened a new pull request, #16804:
URL: https://github.com/apache/datafusion/pull/16804

   ## Which issue does this PR close?
   Currently, we only support for CSV format for h2o benchmark, but from the compare with other database result, it is using parquet, so this ticket try to support parquet format benchmark in datafusion.
   
   
   Details:
   https://github.com/apache/datafusion/issues/16710#issuecomment-3078052376
   
   
   ## Rationale for this change
   
   Currently, we only support for CSV format for h2o benchmark, but from the compare with other database result, it is using parquet, so this ticket try to support parquet format benchmark in datafusion.
   
   ## What changes are included in this PR?
   
   Currently, we only support for CSV format for h2o benchmark, but from the compare with other database result, it is using parquet, so this ticket try to support parquet format benchmark in datafusion.
   
   ## Are these changes tested?
   
   Tested group by, join has some problem.
   
   ## Are there any user-facing changes?
   
   Yes, new format support.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] benchmark: Add parquet h2o support [datafusion]

Posted by "zhuqi-lucas (via GitHub)" <gi...@apache.org>.
zhuqi-lucas commented on PR #16804:
URL: https://github.com/apache/datafusion/pull/16804#issuecomment-3082603389

   Updated, it works now, the falsa has merged the fix and released: https://github.com/mrpowers-io/falsa/pull/28
   
   ```rust
   ./bench.sh data h2o_small_join_parquet
   ***************************
   DataFusion Benchmark Runner and Data Generator
   COMMAND: data
   BENCHMARK: h2o_small_join_parquet
   DATA_DIR: /Users/zhuqi/arrow-datafusion/benchmarks/data
   CARGO_COMMAND: cargo run --release
   PREFER_HASH_JOIN: true
   ***************************
   Found Python version 3.13, which is suitable.
   Using Python command: /opt/homebrew/bin/python3
   Installing falsa...
   Generating h2o test data in /Users/zhuqi/arrow-datafusion/benchmarks/data/h2o with size=SMALL and format=PARQUET
   10 rows will be saved into: /Users/zhuqi/arrow-datafusion/benchmarks/data/h2o/J1_1e7_1e1_0.parquet
   
   10000 rows will be saved into: /Users/zhuqi/arrow-datafusion/benchmarks/data/h2o/J1_1e7_1e4_0.parquet
   
   10000000 rows will be saved into: /Users/zhuqi/arrow-datafusion/benchmarks/data/h2o/J1_1e7_1e7_NA.parquet
   
   An SMALL data schema is the following:
   id1: int64 not null
   id4: string not null
   v2: double not null
   
   An output format is PARQUET
   
   Batch mode is supported.
   In case of memory problems you can try to reduce a batch_size.
   
   
   Working... ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100% 0:00:00
   
   An MEDIUM data schema is the following:
   id1: int64 not null
   id2: int64 not null
   id4: string not null
   id5: string not null
   v2: double not null
   
   An output format is PARQUET
   
   Batch mode is supported.
   In case of memory problems you can try to reduce a batch_size.
   
   
   Working... ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100% 0:00:00
   
   An BIG data schema is the following:
   id1: int64 not null
   id2: int64 not null
   id3: int64 not null
   id4: string not null
   id5: string not null
   id6: string not null
   v2: double not null
   
   An output format is PARQUET
   
   Batch mode is supported.
   In case of memory problems you can try to reduce a batch_size.
   
   
   Working... ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100% 0:00:02
   
   An LSH data schema is the following:
   id1: int64 not null
   id2: int64 not null
   id3: int64 not null
   id4: string not null
   id5: string not null
   id6: string not null
   v1: double not null
   
   An output format is PARQUET
   
   Batch mode is supported.
   In case of memory problems you can try to reduce a batch_size.
   
   
   Working... ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100% 0:00:00
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] benchmark: Add parquet h2o support [datafusion]

Posted by "zhuqi-lucas (via GitHub)" <gi...@apache.org>.
zhuqi-lucas commented on PR #16804:
URL: https://github.com/apache/datafusion/pull/16804#issuecomment-3083126587

   > Thank you! it LGTM. I have also tested it locally.
   
   Thank you @2010YOUY01 for review!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] benchmark: Add parquet h2o support [datafusion]

Posted by "zhuqi-lucas (via GitHub)" <gi...@apache.org>.
zhuqi-lucas commented on PR #16804:
URL: https://github.com/apache/datafusion/pull/16804#issuecomment-3082332726

   Updated: error for parquet join data generate, it works for group by:
   
   ```rust
   ./bench.sh data h2o_medium_join_parquet
   ***************************
   DataFusion Benchmark Runner and Data Generator
   COMMAND: data
   BENCHMARK: h2o_medium_join_parquet
   DATA_DIR: /Users/zhuqi/arrow-datafusion/benchmarks/data
   CARGO_COMMAND: cargo run --release
   PREFER_HASH_JOIN: true
   ***************************
   Found Python version 3.13, which is suitable.
   Using Python command: /opt/homebrew/bin/python3
   Installing falsa...
   Generating h2o test data in /Users/zhuqi/arrow-datafusion/benchmarks/data/h2o with size=MEDIUM and format=PARQUET
   100 rows will be saved into: /Users/zhuqi/arrow-datafusion/benchmarks/data/h2o/J1_1e8_1e2_0.parquet
   
   100000 rows will be saved into: /Users/zhuqi/arrow-datafusion/benchmarks/data/h2o/J1_1e8_1e5_0.parquet
   
   100000000 rows will be saved into: /Users/zhuqi/arrow-datafusion/benchmarks/data/h2o/J1_1e8_1e8_NA.parquet
   
   An SMALL data schema is the following:
   id1: int64 not null
   id4: string
   v2: double
   
   An output format is PARQUET
   
   Batch mode is supported.
   In case of memory problems you can try to reduce a batch_size.
   
   
   Working... ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0% -:--:--
   ╭──────────────────────────────────────────────────────────────────────────────────────── Traceback (most recent call last) ─────────────────────────────────────────────────────────────────────────────────────────╮
   │ /Users/zhuqi/arrow-datafusion/benchmarks/venv/lib/python3.13/site-packages/falsa/app.py:144 in join                                                                                                                │
   │                                                                                                                                                                                                                    │
   │   141 │   )                                                                                                                                                                                                        │
   │   142 │                                                                                                                                                                                                            │
   │   143 │   for batch in track(join_small.iter_batches(), total=len(join_small.batches)):                                                                                                                            │
   │ ❱ 144 │   │   writer_small.write_batch(batch)                                                                                                                                                                      │
   │   145 │   writer_small.close()                                                                                                                                                                                     │
   │   146 │                                                                                                                                                                                                            │
   │   147 │   if data_format is Format.DELTA:                                                                                                                                                                          │
   │                                                                                                                                                                                                                    │
   │ ╭──────────────────────────────────────────────────────────────────────────────────────────────────── locals ────────────────────────────────────────────────────────────────────────────────────────────────────╮ │
   │ │                batch = pyarrow.RecordBatch                                                                                                                                                                     │ │
   │ │                        id1: int64 not null                                                                                                                                                                     │ │
   │ │                        id4: string not null                                                                                                                                                                    │ │
   │ │                        v2: double not null                                                                                                                                                                     │ │
   │ │                        ----                                                                                                                                                                                    │ │
   │ │                        id1: [12,77,106,10,52,105,29,64,46,51,...,110,82,72,8,1,104,69,5,44,25]                                                                                                                 │ │
   │ │                        id4: ["id12","id77","id106","id10","id52","id105","id29","id64","id46","id51",...,"id110","id82","id72","id8","id1","id104","id69","id5","id44","id25"]                                 │ │
   │ │                        v2:                                                                                                                                                                                     │ │
   │ │                        [53.0075954693085,32.410072200393316,72.68372205230826,71.61363809771296,86.99915627358179,15.539006557813716,23.840799451398684,7.23383214431385,6.366591524991982,20.222312628293857… │ │
   │ │           batch_size = 5000000                                                                                                                                                                                 │ │
   │ │    data_filename_big = 'J1_1e8_1e8_NA.parquet'                                                                                                                                                                 │ │
   │ │    data_filename_lhs = 'J1_1e8_NA_0.parquet'                                                                                                                                                                   │ │
   │ │ data_filename_medium = 'J1_1e8_1e5_0.parquet'                                                                                                                                                                  │ │
   │ │  data_filename_small = 'J1_1e8_1e2_0.parquet'                                                                                                                                                                  │ │
   │ │          data_format = <Format.PARQUET: 'PARQUET'>                                                                                                                                                             │ │
   │ │      generation_seed = 6839596180442651345                                                                                                                                                                     │ │
   │ │             join_big = <falsa.local_fs.JoinBigGenerator object at 0x105e2fe00>                                                                                                                                 │ │
   │ │             join_lhs = <falsa.local_fs.JoinLHSGenerator object at 0x105e2da90>                                                                                                                                 │ │
   │ │          join_medium = <falsa.local_fs.JoinMediumGenerator object at 0x105e2fcb0>                                                                                                                              │ │
   │ │           join_small = <falsa.local_fs.JoinSmallGenerator object at 0x105e2fb60>                                                                                                                               │ │
   │ │                    k = 10                                                                                                                                                                                      │ │
   │ │            keys_seed = 1026847926404610461                                                                                                                                                                     │ │
   │ │                n_big = 100000000                                                                                                                                                                               │ │
   │ │             n_medium = 100000                                                                                                                                                                                  │ │
   │ │              n_small = 100                                                                                                                                                                                     │ │
   │ │                  nas = 0                                                                                                                                                                                       │ │
   │ │           output_big = PosixPath('/Users/zhuqi/arrow-datafusion/benchmarks/data/h2o/J1_1e8_1e8_NA.parquet')                                                                                                    │ │
   │ │           output_dir = PosixPath('/Users/zhuqi/arrow-datafusion/benchmarks/data/h2o')                                                                                                                          │ │
   │ │           output_lhs = PosixPath('/Users/zhuqi/arrow-datafusion/benchmarks/data/h2o/J1_1e8_NA_0.parquet')                                                                                                      │ │
   │ │        output_medium = PosixPath('/Users/zhuqi/arrow-datafusion/benchmarks/data/h2o/J1_1e8_1e5_0.parquet')                                                                                                     │ │
   │ │         output_small = PosixPath('/Users/zhuqi/arrow-datafusion/benchmarks/data/h2o/J1_1e8_1e2_0.parquet')                                                                                                     │ │
   │ │          path_prefix = '/Users/zhuqi/arrow-datafusion/benchmarks/data/h2o'                                                                                                                                     │ │
   │ │                 seed = 42                                                                                                                                                                                      │ │
   │ │                 size = <Size.MEDIUM: 'MEDIUM'>                                                                                                                                                                 │ │
   │ │         writer_small = <pyarrow.parquet.core.ParquetWriter object at 0x1067d4050>                                                                                                                              │ │
   │ ╰────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯ │
   │                                                                                                                                                                                                                    │
   │ /Users/zhuqi/arrow-datafusion/benchmarks/venv/lib/python3.13/site-packages/pyarrow/parquet/core.py:1089 in write_batch                                                                                             │
   │                                                                                                                                                                                                                    │
   │   1086 │   │   │   will be used instead.                                                                                                                                                                           │
   │   1087 │   │   """                                                                                                                                                                                                 │
   │   1088 │   │   table = pa.Table.from_batches([batch], batch.schema)                                                                                                                                                │
   │ ❱ 1089 │   │   self.write_table(table, row_group_size)                                                                                                                                                             │
   │   1090 │                                                                                                                                                                                                           │
   │   1091 │   def write_table(self, table, row_group_size=None):                                                                                                                                                      │
   │   1092 │   │   """                                                                                                                                                                                                 │
   │                                                                                                                                                                                                                    │
   │ ╭──────────────────────────────────────────────────────────────────────────────────────────────────── locals ────────────────────────────────────────────────────────────────────────────────────────────────────╮ │
   │ │          batch = pyarrow.RecordBatch                                                                                                                                                                           │ │
   │ │                  id1: int64 not null                                                                                                                                                                           │ │
   │ │                  id4: string not null                                                                                                                                                                          │ │
   │ │                  v2: double not null                                                                                                                                                                           │ │
   │ │                  ----                                                                                                                                                                                          │ │
   │ │                  id1: [12,77,106,10,52,105,29,64,46,51,...,110,82,72,8,1,104,69,5,44,25]                                                                                                                       │ │
   │ │                  id4: ["id12","id77","id106","id10","id52","id105","id29","id64","id46","id51",...,"id110","id82","id72","id8","id1","id104","id69","id5","id44","id25"]                                       │ │
   │ │                  v2:                                                                                                                                                                                           │ │
   │ │                  [53.0075954693085,32.410072200393316,72.68372205230826,71.61363809771296,86.99915627358179,15.539006557813716,23.840799451398684,7.23383214431385,6.366591524991982,20.222312628293857,...,6… │ │
   │ │ row_group_size = None                                                                                                                                                                                          │ │
   │ │           self = <pyarrow.parquet.core.ParquetWriter object at 0x1067d4050>                                                                                                                                    │ │
   │ │          table = pyarrow.Table                                                                                                                                                                                 │ │
   │ │                  id1: int64 not null                                                                                                                                                                           │ │
   │ │                  id4: string not null                                                                                                                                                                          │ │
   │ │                  v2: double not null                                                                                                                                                                           │ │
   │ │                  ----                                                                                                                                                                                          │ │
   │ │                  id1: [[12,77,106,10,52,...,104,69,5,44,25]]                                                                                                                                                   │ │
   │ │                  id4: [["id12","id77","id106","id10","id52",...,"id104","id69","id5","id44","id25"]]                                                                                                           │ │
   │ │                  v2:                                                                                                                                                                                           │ │
   │ │                  [[53.0075954693085,32.410072200393316,72.68372205230826,71.61363809771296,86.99915627358179,...,26.7118533955444,73.44416011403574,93.63022604514522,51.816253173876824,78.95727980955964]]   │ │
   │ ╰────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯ │
   │                                                                                                                                                                                                                    │
   │ /Users/zhuqi/arrow-datafusion/benchmarks/venv/lib/python3.13/site-packages/pyarrow/parquet/core.py:1113 in write_table                                                                                             │
   │                                                                                                                                                                                                                    │
   │   1110 │   │   │   msg = ('Table schema does not match schema used to create file: '                                                                                                                               │
   │   1111 │   │   │   │      '\ntable:\n{!s} vs. \nfile:\n{!s}'                                                                                                                                                       │
   │   1112 │   │   │   │      .format(table.schema, self.schema))                                                                                                                                                      │
   │ ❱ 1113 │   │   │   raise ValueError(msg)                                                                                                                                                                           │
   │   1114 │   │                                                                                                                                                                                                       │
   │   1115 │   │   self.writer.write_table(table, row_group_size=row_group_size)                                                                                                                                       │
   │   1116                                                                                                                                                                                                             │
   │                                                                                                                                                                                                                    │
   │ ╭──────────────────────────────────────────────────────────────────────────────────────────────────── locals ────────────────────────────────────────────────────────────────────────────────────────────────────╮ │
   │ │            msg = 'Table schema does not match schema used to create file: \ntable:\nid1: int64 not n'+98                                                                                                       │ │
   │ │ row_group_size = None                                                                                                                                                                                          │ │
   │ │           self = <pyarrow.parquet.core.ParquetWriter object at 0x1067d4050>                                                                                                                                    │ │
   │ │          table = pyarrow.Table                                                                                                                                                                                 │ │
   │ │                  id1: int64 not null                                                                                                                                                                           │ │
   │ │                  id4: string not null                                                                                                                                                                          │ │
   │ │                  v2: double not null                                                                                                                                                                           │ │
   │ │                  ----                                                                                                                                                                                          │ │
   │ │                  id1: [[12,77,106,10,52,...,104,69,5,44,25]]                                                                                                                                                   │ │
   │ │                  id4: [["id12","id77","id106","id10","id52",...,"id104","id69","id5","id44","id25"]]                                                                                                           │ │
   │ │                  v2:                                                                                                                                                                                           │ │
   │ │                  [[53.0075954693085,32.410072200393316,72.68372205230826,71.61363809771296,86.99915627358179,...,26.7118533955444,73.44416011403574,93.63022604514522,51.816253173876824,78.95727980955964]]   │ │
   │ ╰────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯ │
   ╰────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
   ValueError: Table schema does not match schema used to create file:
   table:
   id1: int64 not null
   id4: string not null
   v2: double not null vs.
   file:
   id1: int64 not null
   id4: string
   v2: double
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] benchmark: Add parquet h2o support [datafusion]

Posted by "2010YOUY01 (via GitHub)" <gi...@apache.org>.
2010YOUY01 commented on code in PR #16804:
URL: https://github.com/apache/datafusion/pull/16804#discussion_r2212651111


##########
benchmarks/bench.sh:
##########
@@ -100,15 +100,24 @@ clickbench_pushdown:    ClickBench queries against partitioned (100 files) parqu
 clickbench_extended:    ClickBench \"inspired\" queries against a single parquet (DataFusion specific)
 
 # H2O.ai Benchmarks (Group By, Join, Window)
-h2o_small:              h2oai benchmark with small dataset (1e7 rows) for groupby,  default file format is csv
-h2o_medium:             h2oai benchmark with medium dataset (1e8 rows) for groupby, default file format is csv
-h2o_big:                h2oai benchmark with large dataset (1e9 rows) for groupby,  default file format is csv
-h2o_small_join:         h2oai benchmark with small dataset (1e7 rows) for join,  default file format is csv
-h2o_medium_join:        h2oai benchmark with medium dataset (1e8 rows) for join, default file format is csv
-h2o_big_join:           h2oai benchmark with large dataset (1e9 rows) for join,  default file format is csv
-h2o_small_window:       Extended h2oai benchmark with small dataset (1e7 rows) for window,  default file format is csv
-h2o_medium_window:      Extended h2oai benchmark with medium dataset (1e8 rows) for window, default file format is csv
-h2o_big_window:         Extended h2oai benchmark with large dataset (1e9 rows) for window,  default file format is csv
+h2o_small:                      h2oai benchmark with small dataset (1e7 rows) for groupby,  default file format is csv

Review Comment:
   Later, we can clean it up with additional size/format options
   like
   `./bench.sh run h2o_join medium parquet`



##########
benchmarks/bench.sh:
##########
@@ -775,6 +840,7 @@ data_h2o() {
 
     # Set virtual environment directory
     VIRTUAL_ENV="${PWD}/venv"
+    rm -rf "$VIRTUAL_ENV"

Review Comment:
   Could you add a comment for this line?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] benchmark: Add parquet h2o support [datafusion]

Posted by "zhuqi-lucas (via GitHub)" <gi...@apache.org>.
zhuqi-lucas commented on code in PR #16804:
URL: https://github.com/apache/datafusion/pull/16804#discussion_r2212666921


##########
benchmarks/bench.sh:
##########
@@ -775,6 +840,7 @@ data_h2o() {
 
     # Set virtual environment directory
     VIRTUAL_ENV="${PWD}/venv"
+    rm -rf "$VIRTUAL_ENV"

Review Comment:
   Thank you @2010YOUY01 , i removed this line in latest PR, i think this is tested code i added.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] benchmark: Add parquet h2o support [datafusion]

Posted by "zhuqi-lucas (via GitHub)" <gi...@apache.org>.
zhuqi-lucas commented on PR #16804:
URL: https://github.com/apache/datafusion/pull/16804#issuecomment-3082646775

   > @zhuqi-lucas Just for the case falsa generates too much noise in stdout (and runner's output), I can add a command-line argument to suppress it. Something like `--silent`.
   
   Thank you @SemyonSinchenko , it's ok for benchmark data generation, we can see the details info.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] benchmark: Add parquet h2o support [datafusion]

Posted by "zhuqi-lucas (via GitHub)" <gi...@apache.org>.
zhuqi-lucas commented on code in PR #16804:
URL: https://github.com/apache/datafusion/pull/16804#discussion_r2212668550


##########
benchmarks/bench.sh:
##########
@@ -100,15 +100,24 @@ clickbench_pushdown:    ClickBench queries against partitioned (100 files) parqu
 clickbench_extended:    ClickBench \"inspired\" queries against a single parquet (DataFusion specific)
 
 # H2O.ai Benchmarks (Group By, Join, Window)
-h2o_small:              h2oai benchmark with small dataset (1e7 rows) for groupby,  default file format is csv
-h2o_medium:             h2oai benchmark with medium dataset (1e8 rows) for groupby, default file format is csv
-h2o_big:                h2oai benchmark with large dataset (1e9 rows) for groupby,  default file format is csv
-h2o_small_join:         h2oai benchmark with small dataset (1e7 rows) for join,  default file format is csv
-h2o_medium_join:        h2oai benchmark with medium dataset (1e8 rows) for join, default file format is csv
-h2o_big_join:           h2oai benchmark with large dataset (1e9 rows) for join,  default file format is csv
-h2o_small_window:       Extended h2oai benchmark with small dataset (1e7 rows) for window,  default file format is csv
-h2o_medium_window:      Extended h2oai benchmark with medium dataset (1e8 rows) for window, default file format is csv
-h2o_big_window:         Extended h2oai benchmark with large dataset (1e9 rows) for window,  default file format is csv
+h2o_small:                      h2oai benchmark with small dataset (1e7 rows) for groupby,  default file format is csv

Review Comment:
   Good suggestion @2010YOUY01 , it's more clear.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] benchmark: Add parquet h2o support [datafusion]

Posted by "zhuqi-lucas (via GitHub)" <gi...@apache.org>.
zhuqi-lucas commented on PR #16804:
URL: https://github.com/apache/datafusion/pull/16804#issuecomment-3082341745

   Create the jira for falsa side, it fails with generate parquet data for join set, but it works well with group by.
   
   https://github.com/mrpowers-io/falsa/issues/27


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] benchmark: Add parquet h2o support [datafusion]

Posted by "SemyonSinchenko (via GitHub)" <gi...@apache.org>.
SemyonSinchenko commented on PR #16804:
URL: https://github.com/apache/datafusion/pull/16804#issuecomment-3082635479

   @zhuqi-lucas Just for the case falsa generates too much noise in stdout (and runner's output), I can add a command-line argument to suppress it. Something like `--silent`.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] benchmark: Add parquet h2o support [datafusion]

Posted by "zhuqi-lucas (via GitHub)" <gi...@apache.org>.
zhuqi-lucas commented on PR #16804:
URL: https://github.com/apache/datafusion/pull/16804#issuecomment-3086556019

   Thank you @alamb @jonathanc-n for review.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] benchmark: Add parquet h2o support [datafusion]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb merged PR #16804:
URL: https://github.com/apache/datafusion/pull/16804


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org