[GitHub] [carbondata] marchpure commented on pull request #3997: [CARBONDATA-4045] Add TPCDS TestCase for Spark on CarbonData Integration Test

[GitBox <gi...@apache.org>]

marchpure commented on pull request #3997:
URL: https://github.com/apache/carbondata/pull/3997#issuecomment-717045718


   > @marchpure : The reason why we don't have TPCH and TPCDS in UT is we need a huge data set, loading huge data takes time.
   > 
   > What is the reason behind adding this? we can anyways have separate TPCH or TPCDS machines that can have automation script to give performance benchmark on every release
   > 
   > Also no need to run TPCH and TPCDS on every PR builder. Running once per release is enough.
   > 
   > @QiangCai , @kunal642 : What's your opinion on this?
   
   1. The TPCDS dataset in this PR is really small(totally 33KB).  it won"t took so much time to load and query.  It may help to avoid possible issues with accepted overhead. 
   2. it also help us to debug tpcds.  explain plan ～analyse in local environment. 
   
   the inspiration to add tpcds test case is CARBONDATA 4008.  Whose issue is Spark on CarbonData will fail in TPCDS Query 83. This issue seems has been there for a log time,  which implies that our UT is not enough. 
   I believe that we can add a profile to turn on/off of TPCDS test in the future if the automatic TPCDS machine is ready.
   
   Maybe we can have a module name 'carbondata-integretion-test'?


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org