You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@impala.apache.org by "Paul Rogers (Code Review)" <ge...@cloudera.org> on 2019/02/17 19:55:07 UTC

[Impala-ASF-CR] IMPALA-8185: Abstract out real/mock file system operations

Paul Rogers has uploaded this change for review. ( http://gerrit.cloudera.org:8080/12437


Change subject: IMPALA-8185: Abstract out real/mock file system operations
......................................................................

IMPALA-8185: Abstract out real/mock file system operations

The recent addition of the test case builder introduced the idea of
running a query in "test case mode" on a system different from the on on
which the test case was created. A number of if-statements were used to
mock file system operations which depend on the file system to exist.

This patch abstracts out the mock vs. real operations into a set of
"facade" classes. The HDFS facade performs the real opeations, the
"mock" facade mocks the operations well enough to fool the planner.
Selection of the facade depends on the recently-added test case mode
query option.

Tests: Reran all tests: both those that depend on a real file system and
the test case builder tests that depend on the mock file system.

Change-Id: I1a385923b64c9fb59cc6e700ee7ee14919398e6d
---
M fe/src/main/java/org/apache/impala/analysis/Analyzer.java
A fe/src/main/java/org/apache/impala/analysis/FileSystemFacade.java
A fe/src/main/java/org/apache/impala/analysis/HdfsFileSystemFacade.java
M fe/src/main/java/org/apache/impala/analysis/HdfsUri.java
A fe/src/main/java/org/apache/impala/analysis/MockFileSystemFacade.java
M fe/src/main/java/org/apache/impala/planner/HdfsPartitionPruner.java
M fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java
7 files changed, 282 insertions(+), 141 deletions(-)



  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/37/12437/1
-- 
To view, visit http://gerrit.cloudera.org:8080/12437
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: I1a385923b64c9fb59cc6e700ee7ee14919398e6d
Gerrit-Change-Number: 12437
Gerrit-PatchSet: 1
Gerrit-Owner: Paul Rogers <pr...@cloudera.com>

[Impala-ASF-CR] IMPALA-8185: Abstract out real/mock file system operations

Posted by "Paul Rogers (Code Review)" <ge...@cloudera.org>.
Paul Rogers has posted comments on this change. ( http://gerrit.cloudera.org:8080/12437 )

Change subject: IMPALA-8185: Abstract out real/mock file system operations
......................................................................


Patch Set 4:

(2 comments)

http://gerrit.cloudera.org:8080/#/c/12437/4/fe/src/main/java/org/apache/impala/analysis/FileSystemFacade.java
File fe/src/main/java/org/apache/impala/analysis/FileSystemFacade.java:

http://gerrit.cloudera.org:8080/#/c/12437/4/fe/src/main/java/org/apache/impala/analysis/FileSystemFacade.java@64
PS4, Line 64:   ScanAllocation allocateScans(FeFsTable table, TScanRangeSpec scanRangeSpecs,
> Where do we document what this does?
Done


http://gerrit.cloudera.org:8080/#/c/12437/4/fe/src/main/java/org/apache/impala/analysis/HdfsFileSystemFacade.java
File fe/src/main/java/org/apache/impala/analysis/HdfsFileSystemFacade.java:

http://gerrit.cloudera.org:8080/#/c/12437/4/fe/src/main/java/org/apache/impala/analysis/HdfsFileSystemFacade.java@102
PS4, Line 102:   public ScanAllocation allocateScans(FeFsTable table, TScanRangeSpec scanRangeSpecs,
> Do you know what this is used for?
To be honest, I didn't work out what the method does. I just noticed that this code relies on having a real file system; something we don't have in "test case mode." The work here simply picks the code up out of HdfsScanNode and plops it down here in a way that we can mock for test cases.

That said, if something looks odd, please do file a JIRA to explain the issue and perhaps we can figure out a solution as separate task.



-- 
To view, visit http://gerrit.cloudera.org:8080/12437
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I1a385923b64c9fb59cc6e700ee7ee14919398e6d
Gerrit-Change-Number: 12437
Gerrit-PatchSet: 4
Gerrit-Owner: Paul Rogers <pr...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Paul Rogers <pr...@cloudera.com>
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Comment-Date: Thu, 21 Feb 2019 01:18:48 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-8185: Abstract out real/mock file system operations

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12437 )

Change subject: IMPALA-8185: Abstract out real/mock file system operations
......................................................................


Patch Set 4:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/2185/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/12437
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I1a385923b64c9fb59cc6e700ee7ee14919398e6d
Gerrit-Change-Number: 12437
Gerrit-PatchSet: 4
Gerrit-Owner: Paul Rogers <pr...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Paul Rogers <pr...@cloudera.com>
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Comment-Date: Wed, 20 Feb 2019 23:08:29 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-8185: Abstract out real/mock file system operations

Posted by "Paul Rogers (Code Review)" <ge...@cloudera.org>.
Hello Bharath Vissapragada, Philip Zeyliger, Impala Public Jenkins, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/12437

to look at the new patch set (#6).

Change subject: IMPALA-8185: Abstract out real/mock file system operations
......................................................................

IMPALA-8185: Abstract out real/mock file system operations

The recent addition of the test case builder introduced the idea of
running a query in "test case mode" on a system different from the on on
which the test case was created. A number of if-statements were used to
mock file system operations which depend on the file system to exist.

This patch abstracts out the mock vs. real operations into a set of
"facade" classes. The HDFS facade performs the real opeations, the
"mock" facade mocks the operations well enough to fool the planner.
Selection of the facade depends on the recently-added test case mode
query option.

Tests: Reran all tests: both those that depend on a real file system and
the test case builder tests that depend on the mock file system.

Change-Id: I1a385923b64c9fb59cc6e700ee7ee14919398e6d
---
M fe/src/main/java/org/apache/impala/analysis/Analyzer.java
A fe/src/main/java/org/apache/impala/analysis/FileSystemFacade.java
A fe/src/main/java/org/apache/impala/analysis/HdfsFileSystemFacade.java
M fe/src/main/java/org/apache/impala/analysis/HdfsUri.java
A fe/src/main/java/org/apache/impala/analysis/MockFileSystemFacade.java
M fe/src/main/java/org/apache/impala/planner/HdfsPartitionPruner.java
M fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java
7 files changed, 352 insertions(+), 137 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/37/12437/6
-- 
To view, visit http://gerrit.cloudera.org:8080/12437
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I1a385923b64c9fb59cc6e700ee7ee14919398e6d
Gerrit-Change-Number: 12437
Gerrit-PatchSet: 6
Gerrit-Owner: Paul Rogers <pr...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Paul Rogers <pr...@cloudera.com>
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>

[Impala-ASF-CR] IMPALA-8185: Abstract out real/mock file system operations

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12437 )

Change subject: IMPALA-8185: Abstract out real/mock file system operations
......................................................................


Patch Set 3:

Build Failed 

https://jenkins.impala.io/job/gerrit-code-review-checks/2183/ : Initial code review checks failed. See linked job for details on the failure.


-- 
To view, visit http://gerrit.cloudera.org:8080/12437
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I1a385923b64c9fb59cc6e700ee7ee14919398e6d
Gerrit-Change-Number: 12437
Gerrit-PatchSet: 3
Gerrit-Owner: Paul Rogers <pr...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Paul Rogers <pr...@cloudera.com>
Gerrit-Comment-Date: Wed, 20 Feb 2019 22:26:47 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-8185: Abstract out real/mock file system operations

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12437 )

Change subject: IMPALA-8185: Abstract out real/mock file system operations
......................................................................


Patch Set 6:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/2286/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/12437
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I1a385923b64c9fb59cc6e700ee7ee14919398e6d
Gerrit-Change-Number: 12437
Gerrit-PatchSet: 6
Gerrit-Owner: Paul Rogers <pr...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Paul Rogers <pr...@cloudera.com>
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Comment-Date: Wed, 27 Feb 2019 23:56:03 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-8185: Abstract out real/mock file system operations

Posted by "Paul Rogers (Code Review)" <ge...@cloudera.org>.
Paul Rogers has posted comments on this change. ( http://gerrit.cloudera.org:8080/12437 )

Change subject: IMPALA-8185: Abstract out real/mock file system operations
......................................................................


Patch Set 1:

Bharath, this is the code we discussed that abstracts out the file system operations into a "facade", then implements "real" and "mock" versions. It is a first step toward integrating the text-based dump parser with the framework you created in the test case builder.

Pre-review tests passed: https://jenkins.impala.io/job/pre-review-test/308/


-- 
To view, visit http://gerrit.cloudera.org:8080/12437
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I1a385923b64c9fb59cc6e700ee7ee14919398e6d
Gerrit-Change-Number: 12437
Gerrit-PatchSet: 1
Gerrit-Owner: Paul Rogers <pr...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Paul Rogers <pr...@cloudera.com>
Gerrit-Comment-Date: Sun, 17 Feb 2019 19:56:54 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-8185: Abstract out real/mock file system operations

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12437 )

Change subject: IMPALA-8185: Abstract out real/mock file system operations
......................................................................


Patch Set 1:

(5 comments)

http://gerrit.cloudera.org:8080/#/c/12437/1/fe/src/main/java/org/apache/impala/analysis/HdfsFileSystemFacade.java
File fe/src/main/java/org/apache/impala/analysis/HdfsFileSystemFacade.java:

http://gerrit.cloudera.org:8080/#/c/12437/1/fe/src/main/java/org/apache/impala/analysis/HdfsFileSystemFacade.java@119
PS1, Line 119:       alloc.totalNodes = Math.min(numLocalNodes + numRemoteNodes, alloc.cluster.numExecutors());
line too long (96 > 90)


http://gerrit.cloudera.org:8080/#/c/12437/1/fe/src/main/java/org/apache/impala/analysis/MockFileSystemFacade.java
File fe/src/main/java/org/apache/impala/analysis/MockFileSystemFacade.java:

http://gerrit.cloudera.org:8080/#/c/12437/1/fe/src/main/java/org/apache/impala/analysis/MockFileSystemFacade.java@37
PS1, Line 37:   public ScanAllocation allocateScans(FeFsTable table, TScanRangeSpec scanRangeSpecs, ListMap<TNetworkAddress> hostMap) {
line too long (121 > 90)


http://gerrit.cloudera.org:8080/#/c/12437/1/fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java
File fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java:

http://gerrit.cloudera.org:8080/#/c/12437/1/fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java@792
PS1, Line 792:       BlockSizeReport report = analyzer.fsFacade().maxBlockSize(partition.getLocationPath());
line too long (93 > 90)


http://gerrit.cloudera.org:8080/#/c/12437/1/fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java@1114
PS1, Line 1114:     if (alloc.totalNodes < alloc.cluster.numExecutors() && scanRangeSpecs_.isSetSplit_specs()) {
line too long (96 > 90)


http://gerrit.cloudera.org:8080/#/c/12437/1/fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java@1126
PS1, Line 1126:           + " localRanges=" + alloc.numLocalRanges + " remoteRanges=" + alloc.numRemoteRanges
line too long (93 > 90)



-- 
To view, visit http://gerrit.cloudera.org:8080/12437
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I1a385923b64c9fb59cc6e700ee7ee14919398e6d
Gerrit-Change-Number: 12437
Gerrit-PatchSet: 1
Gerrit-Owner: Paul Rogers <pr...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Sun, 17 Feb 2019 19:55:58 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-8185: Abstract out real/mock file system operations

Posted by "Paul Rogers (Code Review)" <ge...@cloudera.org>.
Paul Rogers has posted comments on this change. ( http://gerrit.cloudera.org:8080/12437 )

Change subject: IMPALA-8185: Abstract out real/mock file system operations
......................................................................


Patch Set 2:

Rebased on master and fix style nagger warnings.


-- 
To view, visit http://gerrit.cloudera.org:8080/12437
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I1a385923b64c9fb59cc6e700ee7ee14919398e6d
Gerrit-Change-Number: 12437
Gerrit-PatchSet: 2
Gerrit-Owner: Paul Rogers <pr...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Paul Rogers <pr...@cloudera.com>
Gerrit-Comment-Date: Wed, 20 Feb 2019 21:45:42 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-8185: Abstract out real/mock file system operations

Posted by "Paul Rogers (Code Review)" <ge...@cloudera.org>.
Hello Bharath Vissapragada, Impala Public Jenkins, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/12437

to look at the new patch set (#3).

Change subject: IMPALA-8185: Abstract out real/mock file system operations
......................................................................

IMPALA-8185: Abstract out real/mock file system operations

The recent addition of the test case builder introduced the idea of
running a query in "test case mode" on a system different from the on on
which the test case was created. A number of if-statements were used to
mock file system operations which depend on the file system to exist.

This patch abstracts out the mock vs. real operations into a set of
"facade" classes. The HDFS facade performs the real opeations, the
"mock" facade mocks the operations well enough to fool the planner.
Selection of the facade depends on the recently-added test case mode
query option.

Tests: Reran all tests: both those that depend on a real file system and
the test case builder tests that depend on the mock file system.

Change-Id: I1a385923b64c9fb59cc6e700ee7ee14919398e6d
---
M fe/src/main/java/org/apache/impala/analysis/Analyzer.java
A fe/src/main/java/org/apache/impala/analysis/FileSystemFacade.java
A fe/src/main/java/org/apache/impala/analysis/HdfsFileSystemFacade.java
M fe/src/main/java/org/apache/impala/analysis/HdfsUri.java
A fe/src/main/java/org/apache/impala/analysis/MockFileSystemFacade.java
M fe/src/main/java/org/apache/impala/planner/HdfsPartitionPruner.java
M fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java
M testdata/workloads/functional-planner/queries/PlannerTest/mt-dop-validation.test
M testdata/workloads/functional-planner/queries/PlannerTest/tpch-nested.test
9 files changed, 296 insertions(+), 150 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/37/12437/3
-- 
To view, visit http://gerrit.cloudera.org:8080/12437
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I1a385923b64c9fb59cc6e700ee7ee14919398e6d
Gerrit-Change-Number: 12437
Gerrit-PatchSet: 3
Gerrit-Owner: Paul Rogers <pr...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Paul Rogers <pr...@cloudera.com>

[Impala-ASF-CR] IMPALA-8185: Abstract out real/mock file system operations

Posted by "Bharath Vissapragada (Code Review)" <ge...@cloudera.org>.
Bharath Vissapragada has posted comments on this change. ( http://gerrit.cloudera.org:8080/12437 )

Change subject: IMPALA-8185: Abstract out real/mock file system operations
......................................................................


Patch Set 5:

(4 comments)

The general separation of FS stuff looks good to me.

http://gerrit.cloudera.org:8080/#/c/12437/4/fe/src/main/java/org/apache/impala/analysis/HdfsFileSystemFacade.java
File fe/src/main/java/org/apache/impala/analysis/HdfsFileSystemFacade.java:

http://gerrit.cloudera.org:8080/#/c/12437/4/fe/src/main/java/org/apache/impala/analysis/HdfsFileSystemFacade.java@102
PS4, Line 102:   public ScanAllocation allocateScans(FeFsTable table, TScanRangeSpec scanRangeSpecs,
> Do you know what this is used for?
Per my understanding, we use this in cost formulae. Say, if you have a join on top of the scan, the lhs scan determines the number of nodes on which the join runs and we factor in the num nodes to determine the overall join cost (for ex: is inverted join any cheaper?)

Yes this can diverge with the scheduler implementation and ideally we need to consolidate. I remember asking Alex exactly this and he confirmed the behavior.


http://gerrit.cloudera.org:8080/#/c/12437/5/fe/src/main/java/org/apache/impala/analysis/HdfsFileSystemFacade.java
File fe/src/main/java/org/apache/impala/analysis/HdfsFileSystemFacade.java:

http://gerrit.cloudera.org:8080/#/c/12437/5/fe/src/main/java/org/apache/impala/analysis/HdfsFileSystemFacade.java@42
PS5, Line 42: new Configuration();
Use FileSystemUtil.getConfiguration(). I vaguely remember that building a new Conf object was expensive and I don't think we need to do it for every query.


http://gerrit.cloudera.org:8080/#/c/12437/5/fe/src/main/java/org/apache/impala/analysis/MockFileSystemFacade.java
File fe/src/main/java/org/apache/impala/analysis/MockFileSystemFacade.java:

http://gerrit.cloudera.org:8080/#/c/12437/5/fe/src/main/java/org/apache/impala/analysis/MockFileSystemFacade.java@40
PS5, Line 40:   public Path validatePath(Analyzer analyzer, Path path,
Add comments why we pass?


http://gerrit.cloudera.org:8080/#/c/12437/5/testdata/workloads/functional-planner/queries/PlannerTest/mt-dop-validation.test
File testdata/workloads/functional-planner/queries/PlannerTest/mt-dop-validation.test:

http://gerrit.cloudera.org:8080/#/c/12437/5/testdata/workloads/functional-planner/queries/PlannerTest/mt-dop-validation.test@222
PS5, Line 222: |  Per-Host Resources: mem-estimate=88.00MB mem-reservation=88.00MB thread-reservation=1
Any idea why the mem-estimates changed with this patch?



-- 
To view, visit http://gerrit.cloudera.org:8080/12437
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I1a385923b64c9fb59cc6e700ee7ee14919398e6d
Gerrit-Change-Number: 12437
Gerrit-PatchSet: 5
Gerrit-Owner: Paul Rogers <pr...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Paul Rogers <pr...@cloudera.com>
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Comment-Date: Thu, 21 Feb 2019 21:56:03 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-8185: Abstract out real/mock file system operations

Posted by "Paul Rogers (Code Review)" <ge...@cloudera.org>.
Hello Bharath Vissapragada, Philip Zeyliger, Impala Public Jenkins, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/12437

to look at the new patch set (#5).

Change subject: IMPALA-8185: Abstract out real/mock file system operations
......................................................................

IMPALA-8185: Abstract out real/mock file system operations

The recent addition of the test case builder introduced the idea of
running a query in "test case mode" on a system different from the on on
which the test case was created. A number of if-statements were used to
mock file system operations which depend on the file system to exist.

This patch abstracts out the mock vs. real operations into a set of
"facade" classes. The HDFS facade performs the real opeations, the
"mock" facade mocks the operations well enough to fool the planner.
Selection of the facade depends on the recently-added test case mode
query option.

Tests: Reran all tests: both those that depend on a real file system and
the test case builder tests that depend on the mock file system.

Change-Id: I1a385923b64c9fb59cc6e700ee7ee14919398e6d
---
M fe/src/main/java/org/apache/impala/analysis/Analyzer.java
A fe/src/main/java/org/apache/impala/analysis/FileSystemFacade.java
A fe/src/main/java/org/apache/impala/analysis/HdfsFileSystemFacade.java
M fe/src/main/java/org/apache/impala/analysis/HdfsUri.java
A fe/src/main/java/org/apache/impala/analysis/MockFileSystemFacade.java
M fe/src/main/java/org/apache/impala/planner/HdfsPartitionPruner.java
M fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java
M testdata/workloads/functional-planner/queries/PlannerTest/mt-dop-validation.test
M testdata/workloads/functional-planner/queries/PlannerTest/tpch-nested.test
9 files changed, 376 insertions(+), 150 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/37/12437/5
-- 
To view, visit http://gerrit.cloudera.org:8080/12437
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I1a385923b64c9fb59cc6e700ee7ee14919398e6d
Gerrit-Change-Number: 12437
Gerrit-PatchSet: 5
Gerrit-Owner: Paul Rogers <pr...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Paul Rogers <pr...@cloudera.com>
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>

[Impala-ASF-CR] IMPALA-8185: Abstract out real/mock file system operations

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12437 )

Change subject: IMPALA-8185: Abstract out real/mock file system operations
......................................................................


Patch Set 2:

Build Failed 

https://jenkins.impala.io/job/gerrit-code-review-checks/2182/ : Initial code review checks failed. See linked job for details on the failure.


-- 
To view, visit http://gerrit.cloudera.org:8080/12437
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I1a385923b64c9fb59cc6e700ee7ee14919398e6d
Gerrit-Change-Number: 12437
Gerrit-PatchSet: 2
Gerrit-Owner: Paul Rogers <pr...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Paul Rogers <pr...@cloudera.com>
Gerrit-Comment-Date: Wed, 20 Feb 2019 22:14:19 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-8185: Abstract out real/mock file system operations

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12437 )

Change subject: IMPALA-8185: Abstract out real/mock file system operations
......................................................................


Patch Set 1:

Build Failed 

https://jenkins.impala.io/job/gerrit-code-review-checks/2150/ : Initial code review checks failed. See linked job for details on the failure.


-- 
To view, visit http://gerrit.cloudera.org:8080/12437
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I1a385923b64c9fb59cc6e700ee7ee14919398e6d
Gerrit-Change-Number: 12437
Gerrit-PatchSet: 1
Gerrit-Owner: Paul Rogers <pr...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Paul Rogers <pr...@cloudera.com>
Gerrit-Comment-Date: Sun, 17 Feb 2019 20:38:27 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-8185: Abstract out real/mock file system operations

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12437 )

Change subject: IMPALA-8185: Abstract out real/mock file system operations
......................................................................


Patch Set 2:

(5 comments)

http://gerrit.cloudera.org:8080/#/c/12437/2/fe/src/main/java/org/apache/impala/analysis/HdfsFileSystemFacade.java
File fe/src/main/java/org/apache/impala/analysis/HdfsFileSystemFacade.java:

http://gerrit.cloudera.org:8080/#/c/12437/2/fe/src/main/java/org/apache/impala/analysis/HdfsFileSystemFacade.java@119
PS2, Line 119:       alloc.totalNodes = Math.min(numLocalNodes + numRemoteNodes, alloc.cluster.numExecutors());
line too long (96 > 90)


http://gerrit.cloudera.org:8080/#/c/12437/2/fe/src/main/java/org/apache/impala/analysis/MockFileSystemFacade.java
File fe/src/main/java/org/apache/impala/analysis/MockFileSystemFacade.java:

http://gerrit.cloudera.org:8080/#/c/12437/2/fe/src/main/java/org/apache/impala/analysis/MockFileSystemFacade.java@37
PS2, Line 37:   public ScanAllocation allocateScans(FeFsTable table, TScanRangeSpec scanRangeSpecs, ListMap<TNetworkAddress> hostMap) {
line too long (121 > 90)


http://gerrit.cloudera.org:8080/#/c/12437/2/fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java
File fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java:

http://gerrit.cloudera.org:8080/#/c/12437/2/fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java@792
PS2, Line 792:       BlockSizeReport report = analyzer.fsFacade().maxBlockSize(partition.getLocationPath());
line too long (93 > 90)


http://gerrit.cloudera.org:8080/#/c/12437/2/fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java@1114
PS2, Line 1114:     if (alloc.totalNodes < alloc.cluster.numExecutors() && scanRangeSpecs_.isSetSplit_specs()) {
line too long (96 > 90)


http://gerrit.cloudera.org:8080/#/c/12437/2/fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java@1126
PS2, Line 1126:           + " localRanges=" + alloc.numLocalRanges + " remoteRanges=" + alloc.numRemoteRanges
line too long (93 > 90)



-- 
To view, visit http://gerrit.cloudera.org:8080/12437
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I1a385923b64c9fb59cc6e700ee7ee14919398e6d
Gerrit-Change-Number: 12437
Gerrit-PatchSet: 2
Gerrit-Owner: Paul Rogers <pr...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Paul Rogers <pr...@cloudera.com>
Gerrit-Comment-Date: Wed, 20 Feb 2019 21:42:18 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-8185: Abstract out real/mock file system operations

Posted by "Paul Rogers (Code Review)" <ge...@cloudera.org>.
Hello Bharath Vissapragada, Impala Public Jenkins, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/12437

to look at the new patch set (#2).

Change subject: IMPALA-8185: Abstract out real/mock file system operations
......................................................................

IMPALA-8185: Abstract out real/mock file system operations

The recent addition of the test case builder introduced the idea of
running a query in "test case mode" on a system different from the on on
which the test case was created. A number of if-statements were used to
mock file system operations which depend on the file system to exist.

This patch abstracts out the mock vs. real operations into a set of
"facade" classes. The HDFS facade performs the real opeations, the
"mock" facade mocks the operations well enough to fool the planner.
Selection of the facade depends on the recently-added test case mode
query option.

Tests: Reran all tests: both those that depend on a real file system and
the test case builder tests that depend on the mock file system.

Change-Id: I1a385923b64c9fb59cc6e700ee7ee14919398e6d
---
M fe/src/main/java/org/apache/impala/analysis/Analyzer.java
A fe/src/main/java/org/apache/impala/analysis/FileSystemFacade.java
A fe/src/main/java/org/apache/impala/analysis/HdfsFileSystemFacade.java
M fe/src/main/java/org/apache/impala/analysis/HdfsUri.java
A fe/src/main/java/org/apache/impala/analysis/MockFileSystemFacade.java
M fe/src/main/java/org/apache/impala/planner/HdfsPartitionPruner.java
M fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java
M testdata/workloads/functional-planner/queries/PlannerTest/mt-dop-validation.test
M testdata/workloads/functional-planner/queries/PlannerTest/tpch-nested.test
9 files changed, 291 insertions(+), 150 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/37/12437/2
-- 
To view, visit http://gerrit.cloudera.org:8080/12437
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I1a385923b64c9fb59cc6e700ee7ee14919398e6d
Gerrit-Change-Number: 12437
Gerrit-PatchSet: 2
Gerrit-Owner: Paul Rogers <pr...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Paul Rogers <pr...@cloudera.com>

[Impala-ASF-CR] IMPALA-8185: Abstract out real/mock file system operations

Posted by "Paul Rogers (Code Review)" <ge...@cloudera.org>.
Hello Bharath Vissapragada, Impala Public Jenkins, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/12437

to look at the new patch set (#4).

Change subject: IMPALA-8185: Abstract out real/mock file system operations
......................................................................

IMPALA-8185: Abstract out real/mock file system operations

The recent addition of the test case builder introduced the idea of
running a query in "test case mode" on a system different from the on on
which the test case was created. A number of if-statements were used to
mock file system operations which depend on the file system to exist.

This patch abstracts out the mock vs. real operations into a set of
"facade" classes. The HDFS facade performs the real opeations, the
"mock" facade mocks the operations well enough to fool the planner.
Selection of the facade depends on the recently-added test case mode
query option.

Tests: Reran all tests: both those that depend on a real file system and
the test case builder tests that depend on the mock file system.

Change-Id: I1a385923b64c9fb59cc6e700ee7ee14919398e6d
---
M fe/src/main/java/org/apache/impala/analysis/Analyzer.java
A fe/src/main/java/org/apache/impala/analysis/FileSystemFacade.java
A fe/src/main/java/org/apache/impala/analysis/HdfsFileSystemFacade.java
M fe/src/main/java/org/apache/impala/analysis/HdfsUri.java
A fe/src/main/java/org/apache/impala/analysis/MockFileSystemFacade.java
M fe/src/main/java/org/apache/impala/planner/HdfsPartitionPruner.java
M fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java
M testdata/workloads/functional-planner/queries/PlannerTest/mt-dop-validation.test
M testdata/workloads/functional-planner/queries/PlannerTest/tpch-nested.test
9 files changed, 347 insertions(+), 150 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/37/12437/4
-- 
To view, visit http://gerrit.cloudera.org:8080/12437
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I1a385923b64c9fb59cc6e700ee7ee14919398e6d
Gerrit-Change-Number: 12437
Gerrit-PatchSet: 4
Gerrit-Owner: Paul Rogers <pr...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Paul Rogers <pr...@cloudera.com>

[Impala-ASF-CR] IMPALA-8185: Abstract out real/mock file system operations

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12437 )

Change subject: IMPALA-8185: Abstract out real/mock file system operations
......................................................................


Patch Set 5:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/2189/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/12437
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I1a385923b64c9fb59cc6e700ee7ee14919398e6d
Gerrit-Change-Number: 12437
Gerrit-PatchSet: 5
Gerrit-Owner: Paul Rogers <pr...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Paul Rogers <pr...@cloudera.com>
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Comment-Date: Thu, 21 Feb 2019 01:47:21 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-8185: Abstract out real/mock file system operations

Posted by "Philip Zeyliger (Code Review)" <ge...@cloudera.org>.
Philip Zeyliger has posted comments on this change. ( http://gerrit.cloudera.org:8080/12437 )

Change subject: IMPALA-8185: Abstract out real/mock file system operations
......................................................................


Patch Set 4:

(2 comments)

Just kibitzing: would be good to move the comments to the interface so people can tell what's going on?

http://gerrit.cloudera.org:8080/#/c/12437/4/fe/src/main/java/org/apache/impala/analysis/FileSystemFacade.java
File fe/src/main/java/org/apache/impala/analysis/FileSystemFacade.java:

http://gerrit.cloudera.org:8080/#/c/12437/4/fe/src/main/java/org/apache/impala/analysis/FileSystemFacade.java@64
PS4, Line 64:   ScanAllocation allocateScans(FeFsTable table, TScanRangeSpec scanRangeSpecs,
Where do we document what this does?


http://gerrit.cloudera.org:8080/#/c/12437/4/fe/src/main/java/org/apache/impala/analysis/HdfsFileSystemFacade.java
File fe/src/main/java/org/apache/impala/analysis/HdfsFileSystemFacade.java:

http://gerrit.cloudera.org:8080/#/c/12437/4/fe/src/main/java/org/apache/impala/analysis/HdfsFileSystemFacade.java@102
PS4, Line 102:   public ScanAllocation allocateScans(FeFsTable table, TScanRangeSpec scanRangeSpecs,
Do you know what this is used for?

It's kind of different from the C++ version of the same (be/src/scheduling/scheduler.h). I don't think it matters, but it's weirdo.

I'd basically be surprised of computeNumNodes() isn't just:

max(1, min(numExecutors, numBlocks))

i.e., always at least 1 and at most numExecutors, but determined by the number of blocks.



-- 
To view, visit http://gerrit.cloudera.org:8080/12437
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I1a385923b64c9fb59cc6e700ee7ee14919398e6d
Gerrit-Change-Number: 12437
Gerrit-PatchSet: 4
Gerrit-Owner: Paul Rogers <pr...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Paul Rogers <pr...@cloudera.com>
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Comment-Date: Wed, 20 Feb 2019 22:43:04 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-8185: Abstract out real/mock file system operations

Posted by "Paul Rogers (Code Review)" <ge...@cloudera.org>.
Paul Rogers has abandoned this change. ( http://gerrit.cloudera.org:8080/12437 )

Change subject: IMPALA-8185: Abstract out real/mock file system operations
......................................................................


Abandoned

Can live without it.
-- 
To view, visit http://gerrit.cloudera.org:8080/12437
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: abandon
Gerrit-Change-Id: I1a385923b64c9fb59cc6e700ee7ee14919398e6d
Gerrit-Change-Number: 12437
Gerrit-PatchSet: 6
Gerrit-Owner: Paul Rogers <pr...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Paul Rogers <pr...@cloudera.com>
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>