You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@impala.apache.org by "Michael Smith (Code Review)" <ge...@cloudera.org> on 2025/10/23 22:16:50 UTC

[Impala-ASF-CR] WIP IMPALA-9715: Add devdata dataload option

Michael Smith has uploaded this change for review. ( http://gerrit.cloudera.org:8080/23588


Change subject: WIP IMPALA-9715: Add devdata dataload option
......................................................................

WIP IMPALA-9715: Add devdata dataload option

Adds a lightweight dataload for development that only loads text and
parquet files. These are the most common formats used in testing, and
provide a good baseline for development while using much less storage.

Change-Id: Ia942c2df5f238c570a2fc5e547786ac62b9f3af4
---
M bin/impala-config.sh
M buildall.sh
M testdata/bin/create-load-data.sh
3 files changed, 21 insertions(+), 3 deletions(-)



  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/88/23588/1
-- 
To view, visit http://gerrit.cloudera.org:8080/23588
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: Ia942c2df5f238c570a2fc5e547786ac62b9f3af4
Gerrit-Change-Number: 23588
Gerrit-PatchSet: 1
Gerrit-Owner: Michael Smith <mi...@cloudera.com>

[Impala-ASF-CR] IMPALA-9715: Add devdata dataload option

Posted by "Jason Fehr (Code Review)" <ge...@cloudera.org>.
Jason Fehr has posted comments on this change. ( http://gerrit.cloudera.org:8080/23588 )

Change subject: IMPALA-9715: Add devdata dataload option
......................................................................


Patch Set 11:

(2 comments)

http://gerrit.cloudera.org:8080/#/c/23588/11//COMMIT_MSG
Commit Message:

http://gerrit.cloudera.org:8080/#/c/23588/11//COMMIT_MSG@13
PS11, Line 13: -devdata
> This is against how buildall.sh and load-data.py work now, but wouldn't it 
At this point, I think its best if we stick to how the rest of buildall.sh works.


http://gerrit.cloudera.org:8080/#/c/23588/11/buildall.sh
File buildall.sh:

http://gerrit.cloudera.org:8080/#/c/23588/11/buildall.sh@104
PS11, Line 104:     -devdata)
Should there be an entry in the help text for this new flag?



-- 
To view, visit http://gerrit.cloudera.org:8080/23588
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia942c2df5f238c570a2fc5e547786ac62b9f3af4
Gerrit-Change-Number: 23588
Gerrit-PatchSet: 11
Gerrit-Owner: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Pranav Lodha <pr...@cloudera.com>
Gerrit-Comment-Date: Mon, 17 Nov 2025 22:52:44 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-9715: Add devdata dataload option

Posted by "Michael Smith (Code Review)" <ge...@cloudera.org>.
Hello Pranav Lodha, Impala Public Jenkins, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/23588

to look at the new patch set (#6).

Change subject: IMPALA-9715: Add devdata dataload option
......................................................................

IMPALA-9715: Add devdata dataload option

Adds a lightweight dataload for development that only loads text and
parquet files. These are the most common formats used in testing, and
provide a good baseline for development while using much less storage.

Enable with `./buildall.sh -devdata`. Also skips loading tables that
require Hive as they're less commonly used in testing.

Data load with this option (first 3 run in parallel)

    Loading TPC-H data OK (Took: 0 min 15 sec)
    Loading TPC-DS data OK (Took: 2 min 18 sec)
    Loading functional-query data OK (Took: 3 min 43 sec)
    Loading Hive UDFs OK (Took: 0 min 12 sec)
    Running custom post-load steps OK (Took: 0 min 7 sec)
    Caching test tables OK (Took: 0 min 3 sec)
    Creating internal HBase table OK (Took: 0 min 12 sec)
    Checking HDFS health OK (Took: 0 min 2 sec)
    Logging created files OK (Took: 0 min 2 sec)
    Loading external data sources OK (Took: 0 min 19 sec)
    Computing table stats OK (Took: 0 min 14 sec)
    Creating tpcds testcase data OK (Took: 0 min 47 sec)

Reduces dataload from ~30 minutes to ~5 minutes with 16 cores.

Change-Id: Ia942c2df5f238c570a2fc5e547786ac62b9f3af4
---
M bin/impala-config.sh
M bin/load-data.py
M buildall.sh
M testdata/bin/create-load-data.sh
M testdata/bin/generate-schema-statements.py
5 files changed, 71 insertions(+), 15 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/88/23588/6
-- 
To view, visit http://gerrit.cloudera.org:8080/23588
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ia942c2df5f238c570a2fc5e547786ac62b9f3af4
Gerrit-Change-Number: 23588
Gerrit-PatchSet: 6
Gerrit-Owner: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Pranav Lodha <pr...@cloudera.com>

[Impala-ASF-CR] IMPALA-9715: Add devdata dataload option

Posted by "Michael Smith (Code Review)" <ge...@cloudera.org>.
Hello Jason Fehr, Pranav Lodha, Impala Public Jenkins, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/23588

to look at the new patch set (#11).

Change subject: IMPALA-9715: Add devdata dataload option
......................................................................

IMPALA-9715: Add devdata dataload option

Adds a lightweight dataload for development that only loads text and
parquet files. These are the most common formats used in testing, and
provide a good baseline for development while using much less storage.

Enable with `./buildall.sh -devdata`. Also skips loading tables that
require Hive as they're less commonly used in testing.

Data load with this option (first 3 run in parallel)

    Loading TPC-H data OK (Took: 0 min 23 sec)
    Loading TPC-DS data OK (Took: 1 min 58 sec)
    Loading functional-query data OK (Took: 2 min 16 sec)
    Loading Hive UDFs OK (Took: 0 min 12 sec)
    Running custom post-load steps OK (Took: 0 min 7 sec)
    Caching test tables OK (Took: 0 min 3 sec)
    Creating internal HBase table OK (Took: 0 min 12 sec)
    Checking HDFS health OK (Took: 0 min 2 sec)
    Logging created files OK (Took: 0 min 2 sec)
    Loading external data sources OK (Took: 0 min 19 sec)
    Computing table stats OK (Took: 0 min 14 sec)
    Creating tpcds testcase data OK (Took: 0 min 47 sec)

Reduces dataload from ~22 minutes to ~4 minutes with 16 cores.

Change-Id: Ia942c2df5f238c570a2fc5e547786ac62b9f3af4
---
M bin/impala-config.sh
M bin/load-data.py
M buildall.sh
M testdata/bin/create-load-data.sh
M testdata/bin/generate-schema-statements.py
5 files changed, 72 insertions(+), 15 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/88/23588/11
-- 
To view, visit http://gerrit.cloudera.org:8080/23588
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ia942c2df5f238c570a2fc5e547786ac62b9f3af4
Gerrit-Change-Number: 23588
Gerrit-PatchSet: 11
Gerrit-Owner: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Pranav Lodha <pr...@cloudera.com>

[Impala-ASF-CR] WIP IMPALA-9715: Add devdata dataload option

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/23588 )

Change subject: WIP IMPALA-9715: Add devdata dataload option
......................................................................


Patch Set 1:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/20767/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/23588
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia942c2df5f238c570a2fc5e547786ac62b9f3af4
Gerrit-Change-Number: 23588
Gerrit-PatchSet: 1
Gerrit-Owner: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Thu, 23 Oct 2025 22:40:45 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-9715: Add devdata dataload option

Posted by "Michael Smith (Code Review)" <ge...@cloudera.org>.
Hello Jason Fehr, Pranav Lodha, Impala Public Jenkins, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/23588

to look at the new patch set (#8).

Change subject: IMPALA-9715: Add devdata dataload option
......................................................................

IMPALA-9715: Add devdata dataload option

Adds a lightweight dataload for development that only loads text and
parquet files. These are the most common formats used in testing, and
provide a good baseline for development while using much less storage.

Enable with `./buildall.sh -devdata`. Also skips loading tables that
require Hive as they're less commonly used in testing.

Data load with this option (first 3 run in parallel)

    Loading TPC-H data OK (Took: 0 min 15 sec)
    Loading TPC-DS data OK (Took: 2 min 18 sec)
    Loading functional-query data OK (Took: 3 min 43 sec)
    Loading Hive UDFs OK (Took: 0 min 12 sec)
    Running custom post-load steps OK (Took: 0 min 7 sec)
    Caching test tables OK (Took: 0 min 3 sec)
    Creating internal HBase table OK (Took: 0 min 12 sec)
    Checking HDFS health OK (Took: 0 min 2 sec)
    Logging created files OK (Took: 0 min 2 sec)
    Loading external data sources OK (Took: 0 min 19 sec)
    Computing table stats OK (Took: 0 min 14 sec)
    Creating tpcds testcase data OK (Took: 0 min 47 sec)

Reduces dataload from ~30 minutes to ~5 minutes with 16 cores.

Change-Id: Ia942c2df5f238c570a2fc5e547786ac62b9f3af4
---
M bin/impala-config.sh
M bin/load-data.py
M buildall.sh
M testdata/bin/create-load-data.sh
M testdata/bin/generate-schema-statements.py
5 files changed, 72 insertions(+), 15 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/88/23588/8
-- 
To view, visit http://gerrit.cloudera.org:8080/23588
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ia942c2df5f238c570a2fc5e547786ac62b9f3af4
Gerrit-Change-Number: 23588
Gerrit-PatchSet: 8
Gerrit-Owner: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Pranav Lodha <pr...@cloudera.com>

[Impala-ASF-CR] IMPALA-9715: Add devdata dataload option

Posted by "Michael Smith (Code Review)" <ge...@cloudera.org>.
Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/23588 )

Change subject: IMPALA-9715: Add devdata dataload option
......................................................................


Patch Set 5:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/23588/5/testdata/bin/create-load-data.sh
File testdata/bin/create-load-data.sh:

http://gerrit.cloudera.org:8080/#/c/23588/5/testdata/bin/create-load-data.sh@617
PS5, Line 617:   if [[ -z "$DEFAULT_TABLE_FORMATS" ]]; then
It'd also be tempting to skip some of the less important tables in functional-query. https://github.com/apache/impala/blob/master/testdata/bin/compute-table-stats.sh#L36-L39 covers the ones that are broadly useful.



-- 
To view, visit http://gerrit.cloudera.org:8080/23588
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia942c2df5f238c570a2fc5e547786ac62b9f3af4
Gerrit-Change-Number: 23588
Gerrit-PatchSet: 5
Gerrit-Owner: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Pranav Lodha <pr...@cloudera.com>
Gerrit-Comment-Date: Wed, 05 Nov 2025 04:15:30 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-9715: Add devdata dataload option

Posted by "Michael Smith (Code Review)" <ge...@cloudera.org>.
Hello Pranav Lodha, Impala Public Jenkins, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/23588

to look at the new patch set (#7).

Change subject: IMPALA-9715: Add devdata dataload option
......................................................................

IMPALA-9715: Add devdata dataload option

Adds a lightweight dataload for development that only loads text and
parquet files. These are the most common formats used in testing, and
provide a good baseline for development while using much less storage.

Enable with `./buildall.sh -devdata`. Also skips loading tables that
require Hive as they're less commonly used in testing.

Data load with this option (first 3 run in parallel)

    Loading TPC-H data OK (Took: 0 min 15 sec)
    Loading TPC-DS data OK (Took: 2 min 18 sec)
    Loading functional-query data OK (Took: 3 min 43 sec)
    Loading Hive UDFs OK (Took: 0 min 12 sec)
    Running custom post-load steps OK (Took: 0 min 7 sec)
    Caching test tables OK (Took: 0 min 3 sec)
    Creating internal HBase table OK (Took: 0 min 12 sec)
    Checking HDFS health OK (Took: 0 min 2 sec)
    Logging created files OK (Took: 0 min 2 sec)
    Loading external data sources OK (Took: 0 min 19 sec)
    Computing table stats OK (Took: 0 min 14 sec)
    Creating tpcds testcase data OK (Took: 0 min 47 sec)

Reduces dataload from ~30 minutes to ~5 minutes with 16 cores.

Change-Id: Ia942c2df5f238c570a2fc5e547786ac62b9f3af4
---
M bin/impala-config.sh
M bin/load-data.py
M buildall.sh
M testdata/bin/create-load-data.sh
M testdata/bin/generate-schema-statements.py
5 files changed, 72 insertions(+), 15 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/88/23588/7
-- 
To view, visit http://gerrit.cloudera.org:8080/23588
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ia942c2df5f238c570a2fc5e547786ac62b9f3af4
Gerrit-Change-Number: 23588
Gerrit-PatchSet: 7
Gerrit-Owner: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Pranav Lodha <pr...@cloudera.com>

[Impala-ASF-CR] IMPALA-9715: Add devdata dataload option

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/23588 )

Change subject: IMPALA-9715: Add devdata dataload option
......................................................................


Patch Set 7:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/20888/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/23588
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia942c2df5f238c570a2fc5e547786ac62b9f3af4
Gerrit-Change-Number: 23588
Gerrit-PatchSet: 7
Gerrit-Owner: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Pranav Lodha <pr...@cloudera.com>
Gerrit-Comment-Date: Wed, 12 Nov 2025 22:37:21 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-9715: Add devdata dataload option

Posted by "Joe McDonnell (Code Review)" <ge...@cloudera.org>.
Joe McDonnell has posted comments on this change. ( http://gerrit.cloudera.org:8080/23588 )

Change subject: IMPALA-9715: Add devdata dataload option
......................................................................


Patch Set 11:

If a developer has this dev dataload, what is the right way to run tests? i.e. is there something that skips tests that need tables that aren't loaded?


-- 
To view, visit http://gerrit.cloudera.org:8080/23588
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia942c2df5f238c570a2fc5e547786ac62b9f3af4
Gerrit-Change-Number: 23588
Gerrit-PatchSet: 11
Gerrit-Owner: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Pranav Lodha <pr...@cloudera.com>
Gerrit-Comment-Date: Mon, 17 Nov 2025 18:08:14 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-9715: Add devdata dataload option

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/23588 )

Change subject: IMPALA-9715: Add devdata dataload option
......................................................................


Patch Set 9:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/20918/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/23588
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia942c2df5f238c570a2fc5e547786ac62b9f3af4
Gerrit-Change-Number: 23588
Gerrit-PatchSet: 9
Gerrit-Owner: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Pranav Lodha <pr...@cloudera.com>
Gerrit-Comment-Date: Fri, 14 Nov 2025 01:50:34 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-9715: Add devdata dataload option

Posted by "Michael Smith (Code Review)" <ge...@cloudera.org>.
Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/23588 )

Change subject: IMPALA-9715: Add devdata dataload option
......................................................................


Patch Set 11:

> Patch Set 11:
> 
> If a developer has this dev dataload, what is the right way to run tests? i.e. is there something that skips tests that need tables that aren't loaded?

I haven't gotten that far yet. buildall.sh will limit formats for test vectors, but it needs more work as a complete testing workflow.


-- 
To view, visit http://gerrit.cloudera.org:8080/23588
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia942c2df5f238c570a2fc5e547786ac62b9f3af4
Gerrit-Change-Number: 23588
Gerrit-PatchSet: 11
Gerrit-Owner: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Pranav Lodha <pr...@cloudera.com>
Gerrit-Comment-Date: Mon, 17 Nov 2025 18:18:20 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-9715: Add devdata dataload option

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/23588 )

Change subject: IMPALA-9715: Add devdata dataload option
......................................................................


Patch Set 10:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/20919/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/23588
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia942c2df5f238c570a2fc5e547786ac62b9f3af4
Gerrit-Change-Number: 23588
Gerrit-PatchSet: 10
Gerrit-Owner: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Pranav Lodha <pr...@cloudera.com>
Gerrit-Comment-Date: Fri, 14 Nov 2025 02:30:03 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-9715: Add devdata dataload option

Posted by "Michael Smith (Code Review)" <ge...@cloudera.org>.
Hello Pranav Lodha, Impala Public Jenkins, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/23588

to look at the new patch set (#5).

Change subject: IMPALA-9715: Add devdata dataload option
......................................................................

IMPALA-9715: Add devdata dataload option

Adds a lightweight dataload for development that only loads text and
parquet files. These are the most common formats used in testing, and
provide a good baseline for development while using much less storage.

Change-Id: Ia942c2df5f238c570a2fc5e547786ac62b9f3af4
---
M bin/impala-config.sh
M buildall.sh
M testdata/bin/create-load-data.sh
3 files changed, 37 insertions(+), 15 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/88/23588/5
-- 
To view, visit http://gerrit.cloudera.org:8080/23588
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ia942c2df5f238c570a2fc5e547786ac62b9f3af4
Gerrit-Change-Number: 23588
Gerrit-PatchSet: 5
Gerrit-Owner: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Pranav Lodha <pr...@cloudera.com>

[Impala-ASF-CR] IMPALA-9715: Add devdata dataload option

Posted by "Pranav Lodha (Code Review)" <ge...@cloudera.org>.
Pranav Lodha has posted comments on this change. ( http://gerrit.cloudera.org:8080/23588 )

Change subject: IMPALA-9715: Add devdata dataload option
......................................................................


Patch Set 11: Code-Review+1

(1 comment)

> Patch Set 11: Commit message was updated.

Thanks Michael for working on this, this will be super helpful while developing!

We should also reflect this option in our 'Building Impala' docs.

http://gerrit.cloudera.org:8080/#/c/23588/11/testdata/bin/generate-schema-statements.py
File testdata/bin/generate-schema-statements.py:

http://gerrit.cloudera.org:8080/#/c/23588/11/testdata/bin/generate-schema-statements.py@782
PS11, Line 782: (\w+)\b')
do we wanna add some special characters here as well?



-- 
To view, visit http://gerrit.cloudera.org:8080/23588
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia942c2df5f238c570a2fc5e547786ac62b9f3af4
Gerrit-Change-Number: 23588
Gerrit-PatchSet: 11
Gerrit-Owner: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Pranav Lodha <pr...@cloudera.com>
Gerrit-Comment-Date: Fri, 14 Nov 2025 06:33:58 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-9715: Add devdata dataload option

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/23588 )

Change subject: IMPALA-9715: Add devdata dataload option
......................................................................


Patch Set 8:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/20915/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/23588
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia942c2df5f238c570a2fc5e547786ac62b9f3af4
Gerrit-Change-Number: 23588
Gerrit-PatchSet: 8
Gerrit-Owner: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Pranav Lodha <pr...@cloudera.com>
Gerrit-Comment-Date: Fri, 14 Nov 2025 00:38:40 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-9715: Add devdata dataload option

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/23588 )

Change subject: IMPALA-9715: Add devdata dataload option
......................................................................


Patch Set 6:

(3 comments)

gerrit-auto-critic failed. You can reproduce it locally using command:

  python3 bin/jenkins/critique-gerrit-review.py --dryrun

To run it, you might need a virtual env with Python3's venv installed.

http://gerrit.cloudera.org:8080/#/c/23588/6/testdata/bin/generate-schema-statements.py
File testdata/bin/generate-schema-statements.py:

http://gerrit.cloudera.org:8080/#/c/23588/6/testdata/bin/generate-schema-statements.py@779
PS6, Line 779: d
flake8: E306 expected 1 blank line before a nested definition, found 0


http://gerrit.cloudera.org:8080/#/c/23588/6/testdata/bin/generate-schema-statements.py@822
PS6, Line 822: o
flake8: W504 line break after binary operator


http://gerrit.cloudera.org:8080/#/c/23588/6/testdata/bin/generate-schema-statements.py@823
PS6, Line 823: o
flake8: W504 line break after binary operator



-- 
To view, visit http://gerrit.cloudera.org:8080/23588
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia942c2df5f238c570a2fc5e547786ac62b9f3af4
Gerrit-Change-Number: 23588
Gerrit-PatchSet: 6
Gerrit-Owner: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Pranav Lodha <pr...@cloudera.com>
Gerrit-Comment-Date: Thu, 06 Nov 2025 23:20:15 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-9715: Add devdata dataload option

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/23588 )

Change subject: IMPALA-9715: Add devdata dataload option
......................................................................


Patch Set 6:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/20845/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/23588
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia942c2df5f238c570a2fc5e547786ac62b9f3af4
Gerrit-Change-Number: 23588
Gerrit-PatchSet: 6
Gerrit-Owner: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Pranav Lodha <pr...@cloudera.com>
Gerrit-Comment-Date: Thu, 06 Nov 2025 23:46:51 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-9715: Add devdata dataload option

Posted by "Michael Smith (Code Review)" <ge...@cloudera.org>.
Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/23588 )

Change subject: IMPALA-9715: Add devdata dataload option
......................................................................


Patch Set 11:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/23588/11/testdata/bin/generate-schema-statements.py
File testdata/bin/generate-schema-statements.py:

http://gerrit.cloudera.org:8080/#/c/23588/11/testdata/bin/generate-schema-statements.py@782
PS11, Line 782: (\w+)\b')
> do we wanna add some special characters here as well?
Do we have tables with special characters in the name? I could probably just match everything until word boundary.



-- 
To view, visit http://gerrit.cloudera.org:8080/23588
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia942c2df5f238c570a2fc5e547786ac62b9f3af4
Gerrit-Change-Number: 23588
Gerrit-PatchSet: 11
Gerrit-Owner: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Pranav Lodha <pr...@cloudera.com>
Gerrit-Comment-Date: Fri, 14 Nov 2025 18:58:05 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-9715: Add devdata dataload option

Posted by "Csaba Ringhofer (Code Review)" <ge...@cloudera.org>.
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/23588 )

Change subject: IMPALA-9715: Add devdata dataload option
......................................................................


Patch Set 11: Code-Review+1

(7 comments)

Looks very useful!
I mainly added high level comments, lot of them are probably out of scope for the change.

http://gerrit.cloudera.org:8080/#/c/23588/11//COMMIT_MSG
Commit Message:

http://gerrit.cloudera.org:8080/#/c/23588/11//COMMIT_MSG@13
PS11, Line 13: -devdata
This is against how buildall.sh and load-data.py work now, but wouldn't it be simpler to have an env var for this, e.g. IMPALA_DATALOAD_TYPE=dev? For example if you only want to run some specific tests in a Jenkins job and know that they only need dev dataload, then injecting this env var could be used to speed up the build, while forcing the call to buildall.sh to be parameterized differently is more complicated. The env var would also make it possible to have skipIfs for this.


http://gerrit.cloudera.org:8080/#/c/23588/11//COMMIT_MSG@13
PS11, Line 13:  Also skips loading tables that
             : require Hive as they're less commonly used in testing.
This is decided by having DEPENDENT_LOAD_HIVE section, right? This could be mentioned here.

The biggest gap I see and what worth mentioning is that this removes most tables with complex types.


http://gerrit.cloudera.org:8080/#/c/23588/11//COMMIT_MSG@19
PS11, Line 19:     Loading TPC-DS data OK (Took: 1 min 58 sec)
Is tpc-ds actually important for development? Or this is kept to keep more tests runnable? I practically only use it to have a table with many partitions + files (store_sales). For benchmarking a bigger scale factor is needed.


http://gerrit.cloudera.org:8080/#/c/23588/11//COMMIT_MSG@31
PS11, Line 31: Reduces dataload from ~22 minutes to ~4 minutes with 16 cores.
A few more things that could be interesting here:
- the size of the dataload, which can be a limiting factor if someone doesn't have a large hard drive drive - I keep usually 1-2 dataloads and share them between Impala checkouts to avoid using too much space, and this may be avoidable if the size is drastically smaller with devdata as most checkouts probably only need that.
- the memory on the machine and how much was used would be also nice to know, as again, this is a barrier for people with weaker machines to develep Impala - e.g. it would be great if it would turn out that 16GB is now enough for the dataload


http://gerrit.cloudera.org:8080/#/c/23588/11/buildall.sh
File buildall.sh:

http://gerrit.cloudera.org:8080/#/c/23588/11/buildall.sh@265
PS11, Line 265:       echo "[-testdata] : Loads test data. Implied as true if -snapshot_file is"\
              :            "specified. If -snapshot_file is not specified, data will be regenerated."
Can you add dev data here? It also worth explaining how it interacts with testdata


http://gerrit.cloudera.org:8080/#/c/23588/11/buildall.sh@715
PS11, Line 715: if [[ $NEED_MINICLUSTER -eq 1 ]]; then
              :   start_test_cluster_dependencies
              : fi
Probably it should be a different config, but if now Hive / Kudu / HBase is needed then we could skip starting these components. The biggest benefit I see is the lowered memory need by not running Kudu.


http://gerrit.cloudera.org:8080/#/c/23588/11/testdata/bin/create-load-data.sh
File testdata/bin/create-load-data.sh:

http://gerrit.cloudera.org:8080/#/c/23588/11/testdata/bin/create-load-data.sh@610
PS11, Line 610:   warm-up-hive
Does dev dataload actually no longer need Hive? If yes, then this could be skipped.



-- 
To view, visit http://gerrit.cloudera.org:8080/23588
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia942c2df5f238c570a2fc5e547786ac62b9f3af4
Gerrit-Change-Number: 23588
Gerrit-PatchSet: 11
Gerrit-Owner: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Pranav Lodha <pr...@cloudera.com>
Gerrit-Comment-Date: Mon, 17 Nov 2025 15:36:50 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-9715: Add devdata dataload option

Posted by "Joe McDonnell (Code Review)" <ge...@cloudera.org>.
Joe McDonnell has posted comments on this change. ( http://gerrit.cloudera.org:8080/23588 )

Change subject: IMPALA-9715: Add devdata dataload option
......................................................................


Patch Set 11:

> > Patch Set 11:
 > >
 > > If a developer has this dev dataload, what is the right way to
 > run tests? i.e. is there something that skips tests that need
 > tables that aren't loaded?
 > 
 > I haven't gotten that far yet. buildall.sh will limit formats for
 > test vectors, but it needs more work as a complete testing
 > workflow.

I'm ok with experimenting with new idioms for this. We have a couple existing idioms that don't quite do this. exploration_strategy does determine the table formats that we test and some tests skip below exhaustive (in addition to table format restrictions). It currently doesn't impact dataload (functional-query is always loaded at exhaustive and tpch/tpcds are loaded at core). One could imagine an exploration_strategy=minimal option and modifying dataload to respect exploration strategy (which should also help core tests). That wouldn't exclude the tables that are loaded with Hive, so it wouldn't be as fast or small. I do think that a bunch of tests would probably run fine. If we wanted to exclude additional tables, the tests would need some way to say that they need those tables.

workloads are mostly meaningless, because almost everything is functional-query.

I wish we knew which tests needed which tables.


-- 
To view, visit http://gerrit.cloudera.org:8080/23588
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia942c2df5f238c570a2fc5e547786ac62b9f3af4
Gerrit-Change-Number: 23588
Gerrit-PatchSet: 11
Gerrit-Owner: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Pranav Lodha <pr...@cloudera.com>
Gerrit-Comment-Date: Mon, 17 Nov 2025 19:56:21 +0000
Gerrit-HasComments: No