You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "AdamGS (via GitHub)" <gi...@apache.org> on 2026/03/21 19:30:10 UTC

[I] TPC-DS at different scale factors [datafusion-benchmarks]

AdamGS opened a new issue, #30:
URL: https://github.com/apache/datafusion-benchmarks/issues/30

   My understanding from #25 is that the data generation itself is somewhat random so it should be static rather than generated on the fly, so would a PR adding a fixed version of SF=10 be welcome?
   
   My main motivation is diving into [this](https://github.com/apache/datafusion/issues/20078) DataFusion issue I've filed a while ago, right now my main way to reproduce the issue is through another benchmarking setup but that is a bit convoluted and it'll be really nice to use the built-in setup in the main repo.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org

Re: [I] TPC-DS at different scale factors [datafusion-benchmarks]

Posted by "Dandandan (via GitHub)" <gi...@apache.org>.
Dandandan commented on issue #30:
URL: https://github.com/apache/datafusion-benchmarks/issues/30#issuecomment-4109651563

   Yes, this would be very welcome!
   
   I think we should start benchmarking at bigger scale factors 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org

Re: [I] TPC-DS at different scale factors [datafusion-benchmarks]

Posted by "AdamGS (via GitHub)" <gi...@apache.org>.
AdamGS commented on issue #30:
URL: https://github.com/apache/datafusion-benchmarks/issues/30#issuecomment-4110110622

   Seeing that its almost 4GB of data, I think its time to use git lfs here, any objections/alternatives you think are preferred? 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org

Re: [I] TPC-DS at different scale factors [datafusion-benchmarks]

Posted by "AdamGS (via GitHub)" <gi...@apache.org>.
AdamGS commented on issue #30:
URL: https://github.com/apache/datafusion-benchmarks/issues/30#issuecomment-4142110811

   github doesn't allow pushing LFS objects to public forks, so it might be easier to wait for `tpchgen-rs` to support TPC-DS data generation so it'll just be on the fly


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org

Re: [I] TPC-DS at different scale factors [datafusion-benchmarks]

Posted by "Dandandan (via GitHub)" <gi...@apache.org>.
Dandandan commented on issue #30:
URL: https://github.com/apache/datafusion-benchmarks/issues/30#issuecomment-4109660490

   I am not sure btw if TPC-DS is random - it should be generated on the fly but should produce identical output each time.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org