You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@parquet.apache.org by "corwinjoy (via GitHub)" <gi...@apache.org> on 2024/08/23 01:35:42 UTC

[PR] Synthetic float32 footers [parquet-benchmark]

corwinjoy opened a new pull request, #2:
URL: https://github.com/apache/parquet-benchmark/pull/2

   I know that we primarily want real-world footers here but I believe there is also a place for some simple synthetic footers to zero-in on problem areas with performance.  As a start, I have contributed a simple set of float32 footers (since float32 tables are often used in numerical applications). 
   This PR contains:
   Synthetic footers using all float32 columns. Contents are: 1 rowgroup | 10,000 rows | float32 data type
   with 10, 100, and 1000 columns.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] Synthetic float32 footers [parquet-benchmark]

Posted by "alkis (via GitHub)" <gi...@apache.org>.

alkis commented on PR #2:
URL: https://github.com/apache/parquet-benchmark/pull/2#issuecomment-2367232388

   If we are going to take synthetic footers we should put them in `footer/synthetic` or some other directory. When one runs benchmarks against them it should be clear these are not realistic and shouldn't be "weighted" the same way as real ones.
   
   Plus how are these generated? Do they Statistics, custom key-val data? From experience it is these auxiliary structures that mostly contribute to size and thus parsing speed.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] Synthetic float32 footers [parquet-benchmark]

Posted by "emkornfield (via GitHub)" <gi...@apache.org>.

emkornfield commented on code in PR #2:
URL: https://github.com/apache/parquet-benchmark/pull/2#discussion_r1798459684


##########
footer/synthetic-test_data_generator.cc:
##########


Review Comment:
   I'd actually prefer to keep it, having source make things unambiguous.  I think putting these in there own folder makes more sense to me then the prefix.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] Synthetic float32 footers [parquet-benchmark]

Posted by "emkornfield (via GitHub)" <gi...@apache.org>.

emkornfield commented on PR #2:
URL: https://github.com/apache/parquet-benchmark/pull/2#issuecomment-2364356049

   Thanks for the contribution it might pay to add a information file to designate how these where generated.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] Synthetic float32 footers [parquet-benchmark]

Posted by "rizaon (via GitHub)" <gi...@apache.org>.

rizaon commented on PR #2:
URL: https://github.com/apache/parquet-benchmark/pull/2#issuecomment-2375078220

   Apache Impala has some unique synthetic parquet files that we use for unit tests.
   
   Parquet files with deep nesting, used by [TestMaxNestingDepth](https://github.com/apache/impala/blob/da9400d63ce62e37ea531e7fe6564bf4ac2e0e45/tests/query_test/test_nested_types.py#L899)
   https://github.com/apache/impala/blob/master/testdata/max_nesting_depth/README
   
   [widetable_1000_cols](https://github.com/apache/impala/blob/da9400d63ce62e37ea531e7fe6564bf4ac2e0e45/testdata/datasets/functional/functional_schema_template.sql#L2451) table with 1000 columns parquet file.
   
   [lineitem_sixblocks.parquet](https://github.com/apache/impala/blob/master/testdata/LineItemMultiBlock/lineitem_sixblocks.parquet) with 400 row groups.
   https://github.com/apache/impala/blob/master/testdata/LineItemMultiBlock/README.dox
   
   Are they interesting to take in?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org