You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@impala.apache.org by "Abhishek Rawat (Code Review)" <ge...@cloudera.org> on 2019/04/30 08:15:07 UTC

[Impala-ASF-CR] IMPALA-4658: Potential race if compiler reorders ReachedLimit() usage.

Abhishek Rawat has uploaded this change for review. ( http://gerrit.cloudera.org:8080/13178


Change subject: IMPALA-4658: Potential race if compiler reorders ReachedLimit() usage.
......................................................................

IMPALA-4658: Potential race if compiler reorders ReachedLimit() usage.

The ExecNode::num_rows_returned_ member is shared by the scan node
(thread) and the related scanner threads. There is a potential race here
since the compiler is free to reorder the load/store of
num_rows_returned_ member. Also, num_rows_returned_ is not accessed in
a thread safe manner.

This change introduces a new atomic member
ExecNode::num_rows_returned_shared_, to be used in multithreaded code
paths. Both the num_rows_returned_ and num_rows_returned_shared_
members are now private to the ExecNode class. The subclasses must
access/update them through the public getters/modifiers. The getters/
modifiers for single threaded code-path are: rows_returned(),
ReachedLimit(), SetNumRowsReturned(), IncrementNumRowsReturned(), and
DecrementNumRowsReturned(). Thread safe counterparts for these
functions are: rows_returned_shared(), ReachedLimitShared(),
IncrementNumRowsReturnedShared(), and DecrementNumRowsReturnedShared().
Debug checks were added to ensure that num_rows_returned_ is used in
single threaded code-path and num_rows_returned_shared_ is used in
multi threaded code path.

The initial attempt to address this issue by making num_rows_returned_
atomic caused performance regressions in single threaded code-paths
which update the variable inside a loop, such as
ExchangeNode::GetNext(). The regression could be attributed to extra
path length introduced by atomic operations such as read modify
write. And since the GetNext() function could be called multiple times,
with each call updating the atomic multiple times, the regression could
be huge. Some serious thoughts were given to other approaches such as
updating the atomic num_rows_returned_ variable outside of a loop, but
such a change would have been very extensive and error prone. There is
nothing stopping someone from not adding atomic updates in a tight loop.
After much deliberation it was decided that the single threaded code
path should not be penalised and as such it probably makes sense to
use separate variables for single threaded and multi-threaded code
paths, provided the interfaces are simple enough, easy to maintain and
there are checks in place to ensure proper use.

The scan nodes with task based multi-threading (MT) also use the new
thread-safe functions, even though the variable is not shared amongst
various scan node threads. This is because both the task based MT code
path and regular scan node code path calls some common functions and it
was getting very tricky and ugly to differentiate between the two. The
overhead of using atomics is minimal for MT code path since the atomic
is not accessed for every row.

Testing:
- Performance tests were run on tpch and tpcds workload. No real
  regressions were found, but both baseline and test runs showed high
  degree of variability for some queries. Multiple runs were done and
  it was concluded that the variability is not introduced by this change.

Change-Id: I4cbbfad80f7ab87dd6f192a24e2c68f7c66b047e
---
M be/src/exec/aggregation-node.cc
M be/src/exec/analytic-eval-node.cc
M be/src/exec/cardinality-check-node.cc
M be/src/exec/data-source-scan-node.cc
M be/src/exec/exchange-node.cc
M be/src/exec/exec-node.cc
M be/src/exec/exec-node.h
M be/src/exec/hbase-scan-node.cc
M be/src/exec/hdfs-avro-scanner.cc
M be/src/exec/hdfs-orc-scanner.cc
M be/src/exec/hdfs-rcfile-scanner.cc
M be/src/exec/hdfs-scan-node-base.cc
M be/src/exec/hdfs-scan-node-mt.cc
M be/src/exec/hdfs-scan-node.cc
M be/src/exec/hdfs-scanner.cc
M be/src/exec/hdfs-sequence-scanner.cc
M be/src/exec/hdfs-text-scanner.cc
M be/src/exec/kudu-scan-node-mt.cc
M be/src/exec/kudu-scan-node.cc
M be/src/exec/kudu-scanner.cc
M be/src/exec/nested-loop-join-node.cc
M be/src/exec/parquet/hdfs-parquet-scanner.cc
M be/src/exec/partial-sort-node.cc
M be/src/exec/partitioned-hash-join-node.cc
M be/src/exec/select-node-ir.cc
M be/src/exec/select-node.cc
M be/src/exec/sort-node.cc
M be/src/exec/streaming-aggregation-node.cc
M be/src/exec/subplan-node.cc
M be/src/exec/topn-node.cc
M be/src/exec/union-node.cc
M be/src/exec/unnest-node.cc
32 files changed, 168 insertions(+), 111 deletions(-)



  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/78/13178/2
-- 
To view, visit http://gerrit.cloudera.org:8080/13178
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: I4cbbfad80f7ab87dd6f192a24e2c68f7c66b047e
Gerrit-Change-Number: 13178
Gerrit-PatchSet: 2
Gerrit-Owner: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>

[Impala-ASF-CR] IMPALA-4658: Potential race if compiler reorders ReachedLimit() usage.

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13178 )

Change subject: IMPALA-4658: Potential race if compiler reorders ReachedLimit() usage.
......................................................................


Patch Set 7:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/3132/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/13178
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I4cbbfad80f7ab87dd6f192a24e2c68f7c66b047e
Gerrit-Change-Number: 13178
Gerrit-PatchSet: 7
Gerrit-Owner: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Comment-Date: Wed, 08 May 2019 21:20:15 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-4658: Potential race if compiler reorders ReachedLimit() usage.

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13178 )

Change subject: IMPALA-4658: Potential race if compiler reorders ReachedLimit() usage.
......................................................................


Patch Set 8: Code-Review+2


-- 
To view, visit http://gerrit.cloudera.org:8080/13178
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I4cbbfad80f7ab87dd6f192a24e2c68f7c66b047e
Gerrit-Change-Number: 13178
Gerrit-PatchSet: 8
Gerrit-Owner: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Comment-Date: Fri, 10 May 2019 16:32:42 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-4658: Potential race if compiler reorders ReachedLimit() usage.

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13178 )

Change subject: IMPALA-4658: Potential race if compiler reorders ReachedLimit() usage.
......................................................................


Patch Set 6:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/3073/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/13178
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I4cbbfad80f7ab87dd6f192a24e2c68f7c66b047e
Gerrit-Change-Number: 13178
Gerrit-PatchSet: 6
Gerrit-Owner: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Comment-Date: Sat, 04 May 2019 01:36:56 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-4658: Potential race if compiler reorders ReachedLimit() usage.

Posted by "Tim Armstrong (Code Review)" <ge...@cloudera.org>.
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/13178 )

Change subject: IMPALA-4658: Potential race if compiler reorders ReachedLimit() usage.
......................................................................


Patch Set 7: Code-Review+2


-- 
To view, visit http://gerrit.cloudera.org:8080/13178
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I4cbbfad80f7ab87dd6f192a24e2c68f7c66b047e
Gerrit-Change-Number: 13178
Gerrit-PatchSet: 7
Gerrit-Owner: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Comment-Date: Fri, 10 May 2019 16:32:27 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-4658: Potential race if compiler reorders ReachedLimit() usage.

Posted by "Csaba Ringhofer (Code Review)" <ge...@cloudera.org>.
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/13178 )

Change subject: IMPALA-4658: Potential race if compiler reorders ReachedLimit() usage.
......................................................................


Patch Set 7: Code-Review+1

The change looks good to me. I am unsure about the names, but I do not have ideas.


-- 
To view, visit http://gerrit.cloudera.org:8080/13178
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I4cbbfad80f7ab87dd6f192a24e2c68f7c66b047e
Gerrit-Change-Number: 13178
Gerrit-PatchSet: 7
Gerrit-Owner: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Comment-Date: Fri, 10 May 2019 16:07:18 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-4658: Potential race if compiler reorders ReachedLimit() usage.

Posted by "Abhishek Rawat (Code Review)" <ge...@cloudera.org>.
Abhishek Rawat has uploaded a new patch set (#5). ( http://gerrit.cloudera.org:8080/13178 )

Change subject: IMPALA-4658: Potential race if compiler reorders ReachedLimit() usage.
......................................................................

IMPALA-4658: Potential race if compiler reorders ReachedLimit() usage.

The ExecNode::num_rows_returned_ member is shared by the scan node
(thread) and the related scanner threads. There is a potential race here
since the compiler is free to reorder the load/store of
num_rows_returned_ member. Also, num_rows_returned_ is not accessed in
a thread safe manner.

This change introduces regular and thread safe functions for accessing
and updating the num_rows_returned_ member, which is now private to the
ExecNode class. The getters/modifiers for single threaded code-paths
are:
  rows_returned()
  ReachedLimit()
  SetNumRowsReturned()
  IncrementNumRowsReturned()
  DecrementNumRowsReturned()
  CheckLimitAndTruncateRowBatchIfNeeded()
Thread safe counterparts are:
  rows_returned_shared()
  ReachedLimitShared()
  IncrementNumRowsReturnedShared()
  DecrementNumRowsReturnedShared()
  CheckLimitAndTruncateRowBatchIfNeededShared()
Debug checks are added to ensure that regular and thread safe functions
are used properly in single-threaded and multi-threaded code paths,
respectively.

The initial attempt to address this issue by making num_rows_returned_
atomic caused performance regressions in single threaded code paths
which update the variable inside a loop, such as
ExchangeNode::GetNext(). The regression could be attributed to extra
path length introduced by atomic operations such as read modify
write. And since the GetNext() function could be called multiple times,
with each call updating the atomic multiple times, the regression could
be huge. Some serious thoughts were given to other approaches such as
updating the atomic num_rows_returned_ variable outside of a loop, but
such a change would have been very extensive and error prone. There is
nothing stopping someone from not adding atomic updates in a tight loop.
After much deliberation it was decided that the single threaded code
path should not be penalised and as such it probably makes sense to
use separate functions for single threaded and multi-threaded code
paths, provided the interfaces are simple enough, easy to maintain and
there are checks in place to ensure proper use.

The scan nodes with task based multi-threading (MT) also use the new
thread-safe functions in the code-paths shared with the non MT
scan nodes, even though the variable is not shared amongst various
threads. This is because both the task based MT code-path and the
regular scan node code path calls some common functions and it
was getting very tricky and ugly to differentiate between the two. The
overhead of using atomics is minimal for MT code path since the atomic
is not accessed for every row. The only exceptions are HBaseScanNode
and DataSourceScanNode which uses the regular functions.

Testing:
- Performance tests were run on tpch and tpcds workload. No real
  regressions were found, but both baseline and test runs showed high
  degree of variability for some queries. Multiple runs were done and
  it was concluded that the variability is not introduced by this
  patch.

Change-Id: I4cbbfad80f7ab87dd6f192a24e2c68f7c66b047e
---
M be/src/exec/aggregation-node.cc
M be/src/exec/analytic-eval-node.cc
M be/src/exec/cardinality-check-node.cc
M be/src/exec/data-source-scan-node.cc
M be/src/exec/exchange-node.cc
M be/src/exec/exec-node.cc
M be/src/exec/exec-node.h
M be/src/exec/hbase-scan-node.cc
M be/src/exec/hdfs-avro-scanner.cc
M be/src/exec/hdfs-orc-scanner.cc
M be/src/exec/hdfs-rcfile-scanner.cc
M be/src/exec/hdfs-scan-node-base.cc
M be/src/exec/hdfs-scan-node-mt.cc
M be/src/exec/hdfs-scan-node-mt.h
M be/src/exec/hdfs-scan-node.cc
M be/src/exec/hdfs-scan-node.h
M be/src/exec/hdfs-scanner.cc
M be/src/exec/hdfs-sequence-scanner.cc
M be/src/exec/hdfs-text-scanner.cc
M be/src/exec/kudu-scan-node-mt.cc
M be/src/exec/kudu-scan-node-mt.h
M be/src/exec/kudu-scan-node.cc
M be/src/exec/kudu-scan-node.h
M be/src/exec/kudu-scanner.cc
M be/src/exec/nested-loop-join-node.cc
M be/src/exec/parquet/hdfs-parquet-scanner.cc
M be/src/exec/partial-sort-node.cc
M be/src/exec/partitioned-hash-join-node.cc
M be/src/exec/select-node-ir.cc
M be/src/exec/select-node.cc
M be/src/exec/sort-node.cc
M be/src/exec/streaming-aggregation-node.cc
M be/src/exec/subplan-node.cc
M be/src/exec/topn-node.cc
M be/src/exec/union-node.cc
M be/src/exec/unnest-node.cc
36 files changed, 213 insertions(+), 133 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/78/13178/5
-- 
To view, visit http://gerrit.cloudera.org:8080/13178
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I4cbbfad80f7ab87dd6f192a24e2c68f7c66b047e
Gerrit-Change-Number: 13178
Gerrit-PatchSet: 5
Gerrit-Owner: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>

[Impala-ASF-CR] IMPALA-4658: Potential race if compiler reorders ReachedLimit() usage.

Posted by "Abhishek Rawat (Code Review)" <ge...@cloudera.org>.
Abhishek Rawat has uploaded a new patch set (#7). ( http://gerrit.cloudera.org:8080/13178 )

Change subject: IMPALA-4658: Potential race if compiler reorders ReachedLimit() usage.
......................................................................

IMPALA-4658: Potential race if compiler reorders ReachedLimit() usage.

The ExecNode::num_rows_returned_ member is shared by the scan node
(thread) and the related scanner threads. There is a potential race here
since the compiler is free to reorder the load/store of
num_rows_returned_ member. Also, num_rows_returned_ is not accessed in
a thread safe manner.

This change introduces regular and thread safe functions for accessing
and updating the num_rows_returned_ member, which is now private to the
ExecNode class. The getters/modifiers for single threaded code-paths
are:
  rows_returned()
  ReachedLimit()
  SetNumRowsReturned()
  IncrementNumRowsReturned()
  DecrementNumRowsReturned()
  CheckLimitAndTruncateRowBatchIfNeeded()
Thread safe counterparts are:
  rows_returned_shared()
  ReachedLimitShared()
  IncrementNumRowsReturnedShared()
  DecrementNumRowsReturnedShared()
  CheckLimitAndTruncateRowBatchIfNeededShared()
Debug checks are added to ensure that regular and thread safe functions
are used properly in single-threaded and multi-threaded code paths,
respectively.

The initial attempt to address this issue by making num_rows_returned_
atomic caused performance regressions in single threaded code paths
which update the variable inside a loop, such as
ExchangeNode::GetNext(). The regression could be attributed to extra
path length introduced by atomic operations such as read modify
write. And since the GetNext() function could be called multiple times,
with each call updating the atomic multiple times, the regression could
be huge. Some serious thoughts were given to other approaches such as
updating the atomic num_rows_returned_ variable outside of a loop, but
such a change would have been very extensive and error prone. There is
nothing stopping someone from not adding atomic updates in a tight loop.
After much deliberation it was decided that the single threaded code
path should not be penalised and as such it probably makes sense to
use separate functions for single threaded and multi-threaded code
paths, provided the interfaces are simple enough, easy to maintain and
there are checks in place to ensure proper use.

The scan nodes with task based multi-threading (MT) also use the new
thread-safe functions in the code-paths shared with the non MT
scan nodes, even though the variable is not shared amongst various
threads. This is because both the task based MT code-path and the
regular scan node code path calls some common functions and it
was getting very tricky and ugly to differentiate between the two. The
overhead of using atomics is minimal for MT code path since the atomic
is not accessed for every row. The only exceptions are HBaseScanNode
and DataSourceScanNode which uses the regular functions.

Testing:
- Performance tests were run on tpch and tpcds workload. No real
  regressions were found, but both baseline and test runs showed high
  degree of variability for some queries. Multiple runs were done and
  it was concluded that the variability is not introduced by this
  patch.

Change-Id: I4cbbfad80f7ab87dd6f192a24e2c68f7c66b047e
---
M be/src/exec/aggregation-node.cc
M be/src/exec/analytic-eval-node.cc
M be/src/exec/cardinality-check-node.cc
M be/src/exec/data-source-scan-node.cc
M be/src/exec/exchange-node.cc
M be/src/exec/exec-node.cc
M be/src/exec/exec-node.h
M be/src/exec/hbase-scan-node.cc
M be/src/exec/hdfs-avro-scanner.cc
M be/src/exec/hdfs-orc-scanner.cc
M be/src/exec/hdfs-rcfile-scanner.cc
M be/src/exec/hdfs-scan-node-base.cc
M be/src/exec/hdfs-scan-node-mt.cc
M be/src/exec/hdfs-scan-node-mt.h
M be/src/exec/hdfs-scan-node.cc
M be/src/exec/hdfs-scan-node.h
M be/src/exec/hdfs-scanner.cc
M be/src/exec/hdfs-sequence-scanner.cc
M be/src/exec/hdfs-text-scanner.cc
M be/src/exec/kudu-scan-node-mt.cc
M be/src/exec/kudu-scan-node-mt.h
M be/src/exec/kudu-scan-node.cc
M be/src/exec/kudu-scan-node.h
M be/src/exec/kudu-scanner.cc
M be/src/exec/nested-loop-join-node.cc
M be/src/exec/parquet/hdfs-parquet-scanner.cc
M be/src/exec/partial-sort-node.cc
M be/src/exec/partitioned-hash-join-node.cc
M be/src/exec/select-node-ir.cc
M be/src/exec/select-node.cc
M be/src/exec/sort-node.cc
M be/src/exec/streaming-aggregation-node.cc
M be/src/exec/subplan-node.cc
M be/src/exec/topn-node.cc
M be/src/exec/union-node.cc
M be/src/exec/unnest-node.cc
36 files changed, 235 insertions(+), 137 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/78/13178/7
-- 
To view, visit http://gerrit.cloudera.org:8080/13178
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I4cbbfad80f7ab87dd6f192a24e2c68f7c66b047e
Gerrit-Change-Number: 13178
Gerrit-PatchSet: 7
Gerrit-Owner: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>

[Impala-ASF-CR] IMPALA-4658: Potential race if compiler reorders ReachedLimit() usage.

Posted by "Abhishek Rawat (Code Review)" <ge...@cloudera.org>.
Abhishek Rawat has posted comments on this change. ( http://gerrit.cloudera.org:8080/13178 )

Change subject: IMPALA-4658: Potential race if compiler reorders ReachedLimit() usage.
......................................................................


Patch Set 4:

(12 comments)

http://gerrit.cloudera.org:8080/#/c/13178/4//COMMIT_MSG
Commit Message:

http://gerrit.cloudera.org:8080/#/c/13178/4//COMMIT_MSG@17
PS4, Line 17: The getters/modifiers for single threaded code-paths
            : are: rows_returned(), ReachedLimit(), SetNumRowsReturned(),
            : IncrementNumRowsReturned(), and DecrementNumRowsReturned(). Thread safe
            : counterparts are: rows_returned_shared(), ReachedLimitShared(),
            : IncrementNumRowsReturnedShared(), and DecrementNumRowsReturnedShared().
> nit: this could be more readable if the functions were written to separate 
Done


http://gerrit.cloudera.org:8080/#/c/13178/4//COMMIT_MSG@33
PS4, Line 33: Some serious thoughts were given to other approaches such as
            : updating the atomic num_rows_returned_ variable outside of a loop, but
            : such a change would have been very extensive and error prone.
> Did you think about creating a separate variable like limit_reached_shared_
The patch#2 was kind of doing something similar basically having rows_returned and rows_returned_shared.

In the latest patch#5 the MT nodes use synchronized functions only in the code path shared with non MT nodes.


http://gerrit.cloudera.org:8080/#/c/13178/4//COMMIT_MSG@56
PS4, Line 56: .
> nit: please wrap commit message at 72 chars
Done


http://gerrit.cloudera.org:8080/#/c/13178/4/be/src/exec/exec-node.h
File be/src/exec/exec-node.h:

http://gerrit.cloudera.org:8080/#/c/13178/4/be/src/exec/exec-node.h@211
PS4, Line 211: !IsScanNode() || type() == TPlanNodeType::HBASE_SCAN_NODE
             :         || type() == TPlanNodeType::DATA_SOURCE_NODE
I ended up creating following virtual functions and using them in the debug check:
LimitCheckedFromMultipleThreads() - returns true if multiple threads check the limit for a given scan node.
IsTaskBasedMultiThreadingSupport() - returns true if scan node supports Task Based MT. its generic but i think that should be ok in the current codebase since scan nodes are the only ones which have task based MT.


http://gerrit.cloudera.org:8080/#/c/13178/4/be/src/exec/exec-node.h@401
PS4, Line 401: int64_t num_rows_returned_;
> Can you add a comment about this variable's usage? It is not too typical th
Done.


http://gerrit.cloudera.org:8080/#/c/13178/4/be/src/exec/hdfs-scan-node-mt.cc
File be/src/exec/hdfs-scan-node-mt.cc:

http://gerrit.cloudera.org:8080/#/c/13178/4/be/src/exec/hdfs-scan-node-mt.cc@114
PS4, Line 114: IncrementNumRowsReturnedShared
> The comment states that we use Shared versions in MT nodes for the sake of 
Made changes such that we limit the use of atomic functions to scanner code-paths only


http://gerrit.cloudera.org:8080/#/c/13178/4/be/src/exec/hdfs-scan-node.cc
File be/src/exec/hdfs-scan-node.cc:

http://gerrit.cloudera.org:8080/#/c/13178/4/be/src/exec/hdfs-scan-node.cc@134
PS4, Line 134:     // Update the number of materialized rows now instead of when they are materialized.
             :     // This means that scanners might process and queue up more rows than are necessary
             :     // for the limit case but we want to avoid the synchronized writes to
             :     // num_rows_returned_.
             :     IncrementNumRowsReturnedShared(row_batch->num_rows());
> Yeah the comment is a little weird now in that it's justifying using the st
Deleted the comment and added new comment.


http://gerrit.cloudera.org:8080/#/c/13178/4/be/src/exec/hdfs-scan-node.cc@144
PS4, Line 144:       DecrementNumRowsReturnedShared(num_rows_over);
> Is this logic racy in that multiple threads could see that they pushed the 
Tim already responded, this particular codepath is single threaded, but just to add to that we seem to have different flavors of scan nodes:

1. HdfsScanNode/KuduScanNode are single threaded consumers (single consumer multiple producer scenario). They spawn scanner threads which poll the main scan node thread to determine if limit has been reached. The scan node consumes RowBatchs from the RowBatchQueue, which is populated by the various scanner threads. Once the scan node reaches the (scan) limit_ it shuts down the RowBatchQueue and any excess RowBatches in the q basically gets freed. Any excess rows in the final RowBatch won't be consumed since we cap the final RowBatch once we reach the limit_. So we will never undershoot or overshoot the limit_.

2. DataSourceScanNode/HBaseScanNode are single threaded scan nodes.

3. HdfsScanNodeMt/KuduScanNodeMt (Task Based multi-threading): We execute multiple instances (degree determined by optimizer) of plan fragment per node. Each scan node basically has an associated scanner object. 


The new *_shared*/*Shared* interfaces are used by the HdfsScanNode/KuduScanNode (1 above) and the related spawned scanner (producers) threads. For code simplicity, 3. above also uses the shared functions in code-paths common with non MT scan nodes. When we do move to Task Based multi-threading we can get rid of the shared interfaces.


http://gerrit.cloudera.org:8080/#/c/13178/4/be/src/exec/hdfs-scan-node.cc@144
PS4, Line 144:       DecrementNumRowsReturnedShared(num_rows_over);
> I do not understand the reason for doing this "increment, then decrement if
I added a CheckLimitAndTruncateRowBatchIfNeeded and also a CheckLimitAndTruncateRowBatchIfNeededShared version for consistency. Tried to use this new function in as many exec nodes as possible.


http://gerrit.cloudera.org:8080/#/c/13178/4/be/src/exec/hdfs-sequence-scanner.cc
File be/src/exec/hdfs-sequence-scanner.cc:

http://gerrit.cloudera.org:8080/#/c/13178/4/be/src/exec/hdfs-sequence-scanner.cc@295
PS4, Line 295:       scan_node_->limit() - scan_node_->rows_returned_shared();
> similar question here about whether this is going to get wrong results arou
There are two cases here:
- Single scan-node thread and multiple scanner threads: The scanner threads will try to write tuples using max(max_added_tuples, num_to_process) as the limit. If the scan node thread reaches the limit then it will basically just discard the unnecessary RowBatches and cap the final RowBatch as needed.
- Single scan-node thread which does the scan using a scanner object. We should not have any issues.


http://gerrit.cloudera.org:8080/#/c/13178/4/be/src/exec/kudu-scan-node.cc
File be/src/exec/kudu-scan-node.cc:

http://gerrit.cloudera.org:8080/#/c/13178/4/be/src/exec/kudu-scan-node.cc@111
PS4, Line 111:       int num_rows_over = rows_returned_shared() - limit_;
> same concern here
This is a single threaded code-path executed by the main (scan-node) thread.


http://gerrit.cloudera.org:8080/#/c/13178/4/be/src/exec/sort-node.cc
File be/src/exec/sort-node.cc:

http://gerrit.cloudera.org:8080/#/c/13178/4/be/src/exec/sort-node.cc@138
PS4, Line 138:   IncrementNumRowsReturned(row_batch->num_rows());
             :   if (ReachedLimit()) {
             :     row_batch->set_num_rows(row_batch->num_rows() - (rows_returned() - limit_));
             :     *eos = true;
             :   }
             : 
             :   COUNTER_SET(rows_returned_counter_, rows_returned());
> Is it normal that we can leave rows_returned_counter_ > limit_? This seems 
I think this should be fixed as the counter should reflect the proper number of rows consumed. I have fixed this.



-- 
To view, visit http://gerrit.cloudera.org:8080/13178
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I4cbbfad80f7ab87dd6f192a24e2c68f7c66b047e
Gerrit-Change-Number: 13178
Gerrit-PatchSet: 4
Gerrit-Owner: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Comment-Date: Fri, 03 May 2019 23:22:39 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-4658: Potential race if compiler reorders ReachedLimit() usage.

Posted by "Csaba Ringhofer (Code Review)" <ge...@cloudera.org>.
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/13178 )

Change subject: IMPALA-4658: Potential race if compiler reorders ReachedLimit() usage.
......................................................................


Patch Set 6: Code-Review+1

(4 comments)

http://gerrit.cloudera.org:8080/#/c/13178/6/be/src/exec/exec-node.h
File be/src/exec/exec-node.h:

http://gerrit.cloudera.org:8080/#/c/13178/6/be/src/exec/exec-node.h@209
PS6, Line 209:   virtual bool LimitCheckedFromMultipleThreads() const { return false; }
             :   virtual bool IsTaskBasedMultiThreadingSupport() const { return false; }
optional: maybe creating an enum like ThreadingModel would be better to express this? e.g. SINGLE_THREADED, NON_TASK_BASED_SCANNER, TASK_BASED_SCANNER.


http://gerrit.cloudera.org:8080/#/c/13178/6/be/src/exec/exec-node.h@277
PS6, Line 277:   /// Caps the input row batch to ensure that the limit is not exceeded.
             :   /// Sets the eos and returns true, if the limit is reached.
             :   bool CheckLimitAndTruncateRowBatchIfNeeded(RowBatch* row_batch, bool* eos);
             : 
             :   /// Caps the input row batch to ensure that the limit is not exceeded.
             :   /// Sets the eos and returns true, if the limit is reached.
             :   /// Uses thread safe functions.
             :   bool CheckLimitAndTruncateRowBatchIfNeededShared(RowBatch* row_batch, bool* eos);
These could be moved to "protected". Can you check other functions too and make them protected, unless other classes use them?


http://gerrit.cloudera.org:8080/#/c/13178/6/be/src/exec/exec-node.cc
File be/src/exec/exec-node.cc:

http://gerrit.cloudera.org:8080/#/c/13178/6/be/src/exec/exec-node.cc@418
PS6, Line 418: (limit_ == -1 || (rows_returned() + row_batch_size) < limit_)
Same as line 436.


http://gerrit.cloudera.org:8080/#/c/13178/6/be/src/exec/exec-node.cc@436
PS6, Line 436: (limit_ == -1 || (rows_returned_shared() + row_batch_size) < limit_)
'reached_limit' could be set to this value from the start. It could be also reused instead of calling ReachedLimitShared() - we already assume that no other thread changes num_rows_returned_.



-- 
To view, visit http://gerrit.cloudera.org:8080/13178
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I4cbbfad80f7ab87dd6f192a24e2c68f7c66b047e
Gerrit-Change-Number: 13178
Gerrit-PatchSet: 6
Gerrit-Owner: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Comment-Date: Mon, 06 May 2019 13:51:07 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-4658: Potential race if compiler reorders ReachedLimit() usage.

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13178 )

Change subject: IMPALA-4658: Potential race if compiler reorders ReachedLimit() usage.
......................................................................


Patch Set 5:

Build Failed 

https://jenkins.impala.io/job/gerrit-code-review-checks/3071/ : Initial code review checks failed. See linked job for details on the failure.


-- 
To view, visit http://gerrit.cloudera.org:8080/13178
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I4cbbfad80f7ab87dd6f192a24e2c68f7c66b047e
Gerrit-Change-Number: 13178
Gerrit-PatchSet: 5
Gerrit-Owner: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Comment-Date: Fri, 03 May 2019 23:49:05 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-4658: Potential race if compiler reorders ReachedLimit() usage.

Posted by "Csaba Ringhofer (Code Review)" <ge...@cloudera.org>.
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/13178 )

Change subject: IMPALA-4658: Potential race if compiler reorders ReachedLimit() usage.
......................................................................


Patch Set 4:

(10 comments)

http://gerrit.cloudera.org:8080/#/c/13178/4//COMMIT_MSG
Commit Message:

http://gerrit.cloudera.org:8080/#/c/13178/4//COMMIT_MSG@9
PS4, Line 9: The ExecNode::num_rows_returned_ member is shared by the scan node
           : (thread) and the related scanner threads. There is a potential race here
           : since the compiler is free to reorder the load/store of
           : num_rows_returned_ member. Also, num_rows_returned_ is not accessed in
           : a thread safe manner.
Can this actually cause problems? I think that all scanners read num_rows_returned_ in order to stop earlier if no more rows are needed, but it is not a problem if they return some extra row (the ScannerNode will ignore them), it is just wasted work.


http://gerrit.cloudera.org:8080/#/c/13178/4//COMMIT_MSG@17
PS4, Line 17: The getters/modifiers for single threaded code-paths
            : are: rows_returned(), ReachedLimit(), SetNumRowsReturned(),
            : IncrementNumRowsReturned(), and DecrementNumRowsReturned(). Thread safe
            : counterparts are: rows_returned_shared(), ReachedLimitShared(),
            : IncrementNumRowsReturnedShared(), and DecrementNumRowsReturnedShared().
nit: this could be more readable if the functions were written to separate lines


http://gerrit.cloudera.org:8080/#/c/13178/4//COMMIT_MSG@33
PS4, Line 33: Some serious thoughts were given to other approaches such as
            : updating the atomic num_rows_returned_ variable outside of a loop, but
            : such a change would have been very extensive and error prone.
Did you think about creating a separate variable like limit_reached_shared_, and using only that one in a synchronized way? Scanners are only interested about whether the limit is reached, and IncrementNumRowsShared() could update it if needed, leading to less synchronized writes.


http://gerrit.cloudera.org:8080/#/c/13178/4//COMMIT_MSG@56
PS4, Line 56: .
nit: please wrap commit message at 72 chars


http://gerrit.cloudera.org:8080/#/c/13178/4/be/src/exec/exec-node.h
File be/src/exec/exec-node.h:

http://gerrit.cloudera.org:8080/#/c/13178/4/be/src/exec/exec-node.h@211
PS4, Line 211: !IsScanNode() || type() == TPlanNodeType::HBASE_SCAN_NODE
             :         || type() == TPlanNodeType::DATA_SOURCE_NODE
I would prefer to move this logic to a virtual function like IsScanNodeNonTaskBasedMultiThreadingSupport() or LimitCheckedFromMultipleThreads(). (the _MT naming makes it hard to find a nice name) It will make it easier to add more scanners without multi-threading support in the future.


http://gerrit.cloudera.org:8080/#/c/13178/4/be/src/exec/exec-node.h@401
PS4, Line 401: int64_t num_rows_returned_;
Can you add a comment about this variable's usage? It is not too typical that a variable is used both in an atomic and a non-atomic way, and readers may think that the non-atomic access is done by mistake.

+ 1 reason for highlighting this is that it may make sense to collect "shared" variables and change layout to move them to a separate cache line to reduce the effect of atomic operations.


http://gerrit.cloudera.org:8080/#/c/13178/4/be/src/exec/hdfs-scan-node-mt.cc
File be/src/exec/hdfs-scan-node-mt.cc:

http://gerrit.cloudera.org:8080/#/c/13178/4/be/src/exec/hdfs-scan-node-mt.cc@114
PS4, Line 114: IncrementNumRowsReturnedShared
The comment states that we use Shared versions in MT nodes for the sake of simplicity. Can you add a comment here too? I am afraid about readers coming to wrong conclusions because of the unnecessary synchronization.


http://gerrit.cloudera.org:8080/#/c/13178/4/be/src/exec/hdfs-scan-node.cc
File be/src/exec/hdfs-scan-node.cc:

http://gerrit.cloudera.org:8080/#/c/13178/4/be/src/exec/hdfs-scan-node.cc@134
PS4, Line 134:     // Update the number of materialized rows now instead of when they are materialized.
             :     // This means that scanners might process and queue up more rows than are necessary
             :     // for the limit case but we want to avoid the synchronized writes to
             :     // num_rows_returned_.
             :     IncrementNumRowsReturnedShared(row_batch->num_rows());
The comment + the Shared function seem weird together - the comment says that we avoid synchronized writes, then we do a synchronized write.


http://gerrit.cloudera.org:8080/#/c/13178/4/be/src/exec/hdfs-scan-node.cc@144
PS4, Line 144:       DecrementNumRowsReturnedShared(num_rows_over);
> Is this logic racy in that multiple threads could see that they pushed the 
I do not understand the reason for doing this "increment, then decrement if it is more than the limit" thing. Couldn't we just increment with limit_ == -1 ? row_batch->num_rows() : min(row_batch->num_rows(), limit_)? Most nodes seem to use a similar logic.

Creating a function for this logic would be nice, e.g. bool ExecNode::CheckLimitAndTruncateRowBatchIfNeeded(RowBatch* row_batch, bool* eos).


http://gerrit.cloudera.org:8080/#/c/13178/4/be/src/exec/sort-node.cc
File be/src/exec/sort-node.cc:

http://gerrit.cloudera.org:8080/#/c/13178/4/be/src/exec/sort-node.cc@138
PS4, Line 138:   IncrementNumRowsReturned(row_batch->num_rows());
             :   if (ReachedLimit()) {
             :     row_batch->set_num_rows(row_batch->num_rows() - (rows_returned() - limit_));
             :     *eos = true;
             :   }
             : 
             :   COUNTER_SET(rows_returned_counter_, rows_returned());
Is it normal that we can leave rows_returned_counter_ > limit_? This seems to be the only node where it is not capped at limit_.



-- 
To view, visit http://gerrit.cloudera.org:8080/13178
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I4cbbfad80f7ab87dd6f192a24e2c68f7c66b047e
Gerrit-Change-Number: 13178
Gerrit-PatchSet: 4
Gerrit-Owner: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Comment-Date: Thu, 02 May 2019 18:32:26 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-4658: Potential race if compiler reorders ReachedLimit() usage.

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13178 )

Change subject: IMPALA-4658: Potential race if compiler reorders ReachedLimit() usage.
......................................................................


Patch Set 8: Verified+1


-- 
To view, visit http://gerrit.cloudera.org:8080/13178
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I4cbbfad80f7ab87dd6f192a24e2c68f7c66b047e
Gerrit-Change-Number: 13178
Gerrit-PatchSet: 8
Gerrit-Owner: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Comment-Date: Fri, 10 May 2019 22:07:10 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-4658: Potential race if compiler reorders ReachedLimit() usage.

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13178 )

Change subject: IMPALA-4658: Potential race if compiler reorders ReachedLimit() usage.
......................................................................


Patch Set 2:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/2999/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/13178
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I4cbbfad80f7ab87dd6f192a24e2c68f7c66b047e
Gerrit-Change-Number: 13178
Gerrit-PatchSet: 2
Gerrit-Owner: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Comment-Date: Tue, 30 Apr 2019 09:12:33 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-4658: Potential race if compiler reorders ReachedLimit() usage.

Posted by "Abhishek Rawat (Code Review)" <ge...@cloudera.org>.
Abhishek Rawat has posted comments on this change. ( http://gerrit.cloudera.org:8080/13178 )

Change subject: IMPALA-4658: Potential race if compiler reorders ReachedLimit() usage.
......................................................................


Patch Set 2:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/13178/2/be/src/exec/exec-node.h
File be/src/exec/exec-node.h:

http://gerrit.cloudera.org:8080/#/c/13178/2/be/src/exec/exec-node.h@391
PS2, Line 391:   AtomicInt64 num_rows_returned_shared_;
> I made the change so that we only have a single variable. In the multi-thre
It seems the tool did not send my response from yesterday to the reviewers. I also did some basic perf tests and I see that using LongAdder we regress the single threaded code-paths such as ExchangeNode::GetNext (evident from following result see 11:EXCHANGE stats below)

(R) Regression: TPCH(50) TPCH-Q22 [parquet / none / none] (3.78s -> 3.98s [+5.29%])
+---------------------+------------+----------+----------+------------+------------+----------+----------+------------+--------+---------+-----------+
| Operator            | % of Query | Avg      | Base Avg | Delta(Avg) | StdDev(%)  | Max      | Base Max | Delta(Max) | #Hosts | #Rows   | Est #Rows |
+---------------------+------------+----------+----------+------------+------------+----------+----------+------------+--------+---------+-----------+
| 05:HASH JOIN        | 44.01%     | 2.51s    | 2.52s    | -0.36%     |   3.56%    | 2.75s    | 2.67s    | +2.93%     | 3      | 318.22K | 750.00K   |
| 01:SCAN HDFS        | 3.62%      | 206.70ms | 198.37ms | +4.20%     |   8.68%    | 268.71ms | 286.23ms | -6.12%     | 3      | 1.91M   | 750.00K   |
| 11:EXCHANGE         | 8.45%      | 482.53ms | 275.75ms | +74.99%    |   8.77%    | 620.61ms | 355.22ms | +74.71%    | 3      | 75.00M  | 75.00M    |
| F00:EXCHANGE SENDER | 35.94%     | 2.05s    | 2.03s    | +1.01%     | * 12.67% * | 3.18s    | 2.65s    | +20.01%    | 3      | -1      | -1        |
| 03:SCAN HDFS        | 3.20%      | 183.07ms | 173.45ms | +5.55%     | * 11.75% * | 254.60ms | 235.88ms | +7.94%     | 3      | 75.00M  | 75.00M    |
+---------------------+------------+----------+----------+------------+------------+----------+----------+------------+--------+---------+-----------+

The pre-review tests on the latest code with patch #4 is clean: https://jenkins.impala.io/job/ubuntu-16.04-from-scratch/5450/testReport/



-- 
To view, visit http://gerrit.cloudera.org:8080/13178
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I4cbbfad80f7ab87dd6f192a24e2c68f7c66b047e
Gerrit-Change-Number: 13178
Gerrit-PatchSet: 2
Gerrit-Owner: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Comment-Date: Wed, 01 May 2019 22:23:28 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-4658: Potential race if compiler reorders ReachedLimit() usage.

Posted by "Abhishek Rawat (Code Review)" <ge...@cloudera.org>.
Abhishek Rawat has uploaded a new patch set (#6). ( http://gerrit.cloudera.org:8080/13178 )

Change subject: IMPALA-4658: Potential race if compiler reorders ReachedLimit() usage.
......................................................................

IMPALA-4658: Potential race if compiler reorders ReachedLimit() usage.

The ExecNode::num_rows_returned_ member is shared by the scan node
(thread) and the related scanner threads. There is a potential race here
since the compiler is free to reorder the load/store of
num_rows_returned_ member. Also, num_rows_returned_ is not accessed in
a thread safe manner.

This change introduces regular and thread safe functions for accessing
and updating the num_rows_returned_ member, which is now private to the
ExecNode class. The getters/modifiers for single threaded code-paths
are:
  rows_returned()
  ReachedLimit()
  SetNumRowsReturned()
  IncrementNumRowsReturned()
  DecrementNumRowsReturned()
  CheckLimitAndTruncateRowBatchIfNeeded()
Thread safe counterparts are:
  rows_returned_shared()
  ReachedLimitShared()
  IncrementNumRowsReturnedShared()
  DecrementNumRowsReturnedShared()
  CheckLimitAndTruncateRowBatchIfNeededShared()
Debug checks are added to ensure that regular and thread safe functions
are used properly in single-threaded and multi-threaded code paths,
respectively.

The initial attempt to address this issue by making num_rows_returned_
atomic caused performance regressions in single threaded code paths
which update the variable inside a loop, such as
ExchangeNode::GetNext(). The regression could be attributed to extra
path length introduced by atomic operations such as read modify
write. And since the GetNext() function could be called multiple times,
with each call updating the atomic multiple times, the regression could
be huge. Some serious thoughts were given to other approaches such as
updating the atomic num_rows_returned_ variable outside of a loop, but
such a change would have been very extensive and error prone. There is
nothing stopping someone from not adding atomic updates in a tight loop.
After much deliberation it was decided that the single threaded code
path should not be penalised and as such it probably makes sense to
use separate functions for single threaded and multi-threaded code
paths, provided the interfaces are simple enough, easy to maintain and
there are checks in place to ensure proper use.

The scan nodes with task based multi-threading (MT) also use the new
thread-safe functions in the code-paths shared with the non MT
scan nodes, even though the variable is not shared amongst various
threads. This is because both the task based MT code-path and the
regular scan node code path calls some common functions and it
was getting very tricky and ugly to differentiate between the two. The
overhead of using atomics is minimal for MT code path since the atomic
is not accessed for every row. The only exceptions are HBaseScanNode
and DataSourceScanNode which uses the regular functions.

Testing:
- Performance tests were run on tpch and tpcds workload. No real
  regressions were found, but both baseline and test runs showed high
  degree of variability for some queries. Multiple runs were done and
  it was concluded that the variability is not introduced by this
  patch.

Change-Id: I4cbbfad80f7ab87dd6f192a24e2c68f7c66b047e
---
M be/src/exec/aggregation-node.cc
M be/src/exec/analytic-eval-node.cc
M be/src/exec/cardinality-check-node.cc
M be/src/exec/data-source-scan-node.cc
M be/src/exec/exchange-node.cc
M be/src/exec/exec-node.cc
M be/src/exec/exec-node.h
M be/src/exec/hbase-scan-node.cc
M be/src/exec/hdfs-avro-scanner.cc
M be/src/exec/hdfs-orc-scanner.cc
M be/src/exec/hdfs-rcfile-scanner.cc
M be/src/exec/hdfs-scan-node-base.cc
M be/src/exec/hdfs-scan-node-mt.cc
M be/src/exec/hdfs-scan-node-mt.h
M be/src/exec/hdfs-scan-node.cc
M be/src/exec/hdfs-scan-node.h
M be/src/exec/hdfs-scanner.cc
M be/src/exec/hdfs-sequence-scanner.cc
M be/src/exec/hdfs-text-scanner.cc
M be/src/exec/kudu-scan-node-mt.cc
M be/src/exec/kudu-scan-node-mt.h
M be/src/exec/kudu-scan-node.cc
M be/src/exec/kudu-scan-node.h
M be/src/exec/kudu-scanner.cc
M be/src/exec/nested-loop-join-node.cc
M be/src/exec/parquet/hdfs-parquet-scanner.cc
M be/src/exec/partial-sort-node.cc
M be/src/exec/partitioned-hash-join-node.cc
M be/src/exec/select-node-ir.cc
M be/src/exec/select-node.cc
M be/src/exec/sort-node.cc
M be/src/exec/streaming-aggregation-node.cc
M be/src/exec/subplan-node.cc
M be/src/exec/topn-node.cc
M be/src/exec/union-node.cc
M be/src/exec/unnest-node.cc
36 files changed, 220 insertions(+), 137 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/78/13178/6
-- 
To view, visit http://gerrit.cloudera.org:8080/13178
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I4cbbfad80f7ab87dd6f192a24e2c68f7c66b047e
Gerrit-Change-Number: 13178
Gerrit-PatchSet: 6
Gerrit-Owner: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>

[Impala-ASF-CR] IMPALA-4658: Potential race if compiler reorders ReachedLimit() usage.

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/13178 )

Change subject: IMPALA-4658: Potential race if compiler reorders ReachedLimit() usage.
......................................................................

IMPALA-4658: Potential race if compiler reorders ReachedLimit() usage.

The ExecNode::num_rows_returned_ member is shared by the scan node
(thread) and the related scanner threads. There is a potential race here
since the compiler is free to reorder the load/store of
num_rows_returned_ member. Also, num_rows_returned_ is not accessed in
a thread safe manner.

This change introduces regular and thread safe functions for accessing
and updating the num_rows_returned_ member, which is now private to the
ExecNode class. The getters/modifiers for single threaded code-paths
are:
  rows_returned()
  ReachedLimit()
  SetNumRowsReturned()
  IncrementNumRowsReturned()
  DecrementNumRowsReturned()
  CheckLimitAndTruncateRowBatchIfNeeded()
Thread safe counterparts are:
  rows_returned_shared()
  ReachedLimitShared()
  IncrementNumRowsReturnedShared()
  DecrementNumRowsReturnedShared()
  CheckLimitAndTruncateRowBatchIfNeededShared()
Debug checks are added to ensure that regular and thread safe functions
are used properly in single-threaded and multi-threaded code paths,
respectively.

The initial attempt to address this issue by making num_rows_returned_
atomic caused performance regressions in single threaded code paths
which update the variable inside a loop, such as
ExchangeNode::GetNext(). The regression could be attributed to extra
path length introduced by atomic operations such as read modify
write. And since the GetNext() function could be called multiple times,
with each call updating the atomic multiple times, the regression could
be huge. Some serious thoughts were given to other approaches such as
updating the atomic num_rows_returned_ variable outside of a loop, but
such a change would have been very extensive and error prone. There is
nothing stopping someone from not adding atomic updates in a tight loop.
After much deliberation it was decided that the single threaded code
path should not be penalised and as such it probably makes sense to
use separate functions for single threaded and multi-threaded code
paths, provided the interfaces are simple enough, easy to maintain and
there are checks in place to ensure proper use.

The scan nodes with task based multi-threading (MT) also use the new
thread-safe functions in the code-paths shared with the non MT
scan nodes, even though the variable is not shared amongst various
threads. This is because both the task based MT code-path and the
regular scan node code path calls some common functions and it
was getting very tricky and ugly to differentiate between the two. The
overhead of using atomics is minimal for MT code path since the atomic
is not accessed for every row. The only exceptions are HBaseScanNode
and DataSourceScanNode which uses the regular functions.

Testing:
- Performance tests were run on tpch and tpcds workload. No real
  regressions were found, but both baseline and test runs showed high
  degree of variability for some queries. Multiple runs were done and
  it was concluded that the variability is not introduced by this
  patch.

Change-Id: I4cbbfad80f7ab87dd6f192a24e2c68f7c66b047e
Reviewed-on: http://gerrit.cloudera.org:8080/13178
Reviewed-by: Impala Public Jenkins <im...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
---
M be/src/exec/aggregation-node.cc
M be/src/exec/analytic-eval-node.cc
M be/src/exec/cardinality-check-node.cc
M be/src/exec/data-source-scan-node.cc
M be/src/exec/exchange-node.cc
M be/src/exec/exec-node.cc
M be/src/exec/exec-node.h
M be/src/exec/hbase-scan-node.cc
M be/src/exec/hdfs-avro-scanner.cc
M be/src/exec/hdfs-orc-scanner.cc
M be/src/exec/hdfs-rcfile-scanner.cc
M be/src/exec/hdfs-scan-node-base.cc
M be/src/exec/hdfs-scan-node-mt.cc
M be/src/exec/hdfs-scan-node-mt.h
M be/src/exec/hdfs-scan-node.cc
M be/src/exec/hdfs-scan-node.h
M be/src/exec/hdfs-scanner.cc
M be/src/exec/hdfs-sequence-scanner.cc
M be/src/exec/hdfs-text-scanner.cc
M be/src/exec/kudu-scan-node-mt.cc
M be/src/exec/kudu-scan-node-mt.h
M be/src/exec/kudu-scan-node.cc
M be/src/exec/kudu-scan-node.h
M be/src/exec/kudu-scanner.cc
M be/src/exec/nested-loop-join-node.cc
M be/src/exec/parquet/hdfs-parquet-scanner.cc
M be/src/exec/partial-sort-node.cc
M be/src/exec/partitioned-hash-join-node.cc
M be/src/exec/select-node-ir.cc
M be/src/exec/select-node.cc
M be/src/exec/sort-node.cc
M be/src/exec/streaming-aggregation-node.cc
M be/src/exec/subplan-node.cc
M be/src/exec/topn-node.cc
M be/src/exec/union-node.cc
M be/src/exec/unnest-node.cc
36 files changed, 235 insertions(+), 137 deletions(-)

Approvals:
  Impala Public Jenkins: Looks good to me, approved; Verified

-- 
To view, visit http://gerrit.cloudera.org:8080/13178
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: I4cbbfad80f7ab87dd6f192a24e2c68f7c66b047e
Gerrit-Change-Number: 13178
Gerrit-PatchSet: 9
Gerrit-Owner: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>

[Impala-ASF-CR] IMPALA-4658: Potential race if compiler reorders ReachedLimit() usage.

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13178 )

Change subject: IMPALA-4658: Potential race if compiler reorders ReachedLimit() usage.
......................................................................


Patch Set 4:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/3023/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/13178
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I4cbbfad80f7ab87dd6f192a24e2c68f7c66b047e
Gerrit-Change-Number: 13178
Gerrit-PatchSet: 4
Gerrit-Owner: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Comment-Date: Wed, 01 May 2019 06:25:09 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-4658: Potential race if compiler reorders ReachedLimit() usage.

Posted by "Tim Armstrong (Code Review)" <ge...@cloudera.org>.
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/13178 )

Change subject: IMPALA-4658: Potential race if compiler reorders ReachedLimit() usage.
......................................................................


Patch Set 4:

Only the main fragment thread should be updating num_rows_returned_ in the scan nodes - the other scanner threads poll it for early termination.


-- 
To view, visit http://gerrit.cloudera.org:8080/13178
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I4cbbfad80f7ab87dd6f192a24e2c68f7c66b047e
Gerrit-Change-Number: 13178
Gerrit-PatchSet: 4
Gerrit-Owner: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Comment-Date: Wed, 01 May 2019 22:59:00 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-4658: Potential race if compiler reorders ReachedLimit() usage.

Posted by "Tim Armstrong (Code Review)" <ge...@cloudera.org>.
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/13178 )

Change subject: IMPALA-4658: Potential race if compiler reorders ReachedLimit() usage.
......................................................................


Patch Set 2:

(6 comments)

I did a pass over this. Looks good - just had a couple of suggestions for cleanup. I wanted to do another final pass after follow-on changes just to double-check that the new version was identical to the old version.

http://gerrit.cloudera.org:8080/#/c/13178/2//COMMIT_MSG
Commit Message:

http://gerrit.cloudera.org:8080/#/c/13178/2//COMMIT_MSG@46
PS2, Line 46: The scan nodes with task based multi-threading (MT) also use the new
This seems reasonable. Ideally we could avoid this for DataSourceScanNode and HBaseScanNode though, since those don't have MT implementations. Maybe we can just make an exception in the DCHECK - we have the TPlanNodeType in the ExecNode.


http://gerrit.cloudera.org:8080/#/c/13178/2/be/src/exec/exec-node.h
File be/src/exec/exec-node.h:

http://gerrit.cloudera.org:8080/#/c/13178/2/be/src/exec/exec-node.h@209
PS2, Line 209: retrurned
typo


http://gerrit.cloudera.org:8080/#/c/13178/2/be/src/exec/exec-node.h@215
PS2, Line 215: retrurned
typo


http://gerrit.cloudera.org:8080/#/c/13178/2/be/src/exec/exec-node.h@216
PS2, Line 216:   /// rows_returned().
Can you add a short TODO that these can be removed if we remove the legacy multi-threaded scan nodes.


http://gerrit.cloudera.org:8080/#/c/13178/2/be/src/exec/exec-node.h@236
PS2, Line 236: resturned
typo: returned


http://gerrit.cloudera.org:8080/#/c/13178/2/be/src/exec/exec-node.h@391
PS2, Line 391:   AtomicInt64 num_rows_returned_shared_;
It's a bit of a matter of taste but I think it would be better to just have a single member and use the atomic Acquire_Load, Release_Store, Barrier_AtomicIncrement operations on it directly. It's well-encapsulated in these functions so should be ok



-- 
To view, visit http://gerrit.cloudera.org:8080/13178
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I4cbbfad80f7ab87dd6f192a24e2c68f7c66b047e
Gerrit-Change-Number: 13178
Gerrit-PatchSet: 2
Gerrit-Owner: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Comment-Date: Tue, 30 Apr 2019 15:47:41 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-4658: Potential race if compiler reorders ReachedLimit() usage.

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13178 )

Change subject: IMPALA-4658: Potential race if compiler reorders ReachedLimit() usage.
......................................................................


Patch Set 8:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/4225/ DRY_RUN=false


-- 
To view, visit http://gerrit.cloudera.org:8080/13178
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I4cbbfad80f7ab87dd6f192a24e2c68f7c66b047e
Gerrit-Change-Number: 13178
Gerrit-PatchSet: 8
Gerrit-Owner: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Comment-Date: Fri, 10 May 2019 16:32:43 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-4658: Potential race if compiler reorders ReachedLimit() usage.

Posted by "Tim Armstrong (Code Review)" <ge...@cloudera.org>.
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/13178 )

Change subject: IMPALA-4658: Potential race if compiler reorders ReachedLimit() usage.
......................................................................


Patch Set 5:

(2 comments)

http://gerrit.cloudera.org:8080/#/c/13178/5/be/src/exec/exec-node.h
File be/src/exec/exec-node.h:

http://gerrit.cloudera.org:8080/#/c/13178/5/be/src/exec/exec-node.h@279
PS5, Line 279:   bool CheckLimitAndTruncateRowBatchIfNeeded(RowBatch* row_batch, bool* eos) {
We can probably just move these to the .cc file to avoid the additional header dependency - these aren't like enough on the hot path that a direct function call makes a difference.


http://gerrit.cloudera.org:8080/#/c/13178/5/be/src/exec/hdfs-scan-node.cc
File be/src/exec/hdfs-scan-node.cc:

http://gerrit.cloudera.org:8080/#/c/13178/5/be/src/exec/hdfs-scan-node.cc@136
PS5, Line 136:     if (CheckLimitAndTruncateRowBatchIfNeededShared(row_batch, eos)) {
The house style is to put conditionals like this on one line if they fit, i.e. 

    if (CheckLimitAndTruncateRowBatchIfNeededShared(row_batch, eos)) SetDone();



-- 
To view, visit http://gerrit.cloudera.org:8080/13178
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I4cbbfad80f7ab87dd6f192a24e2c68f7c66b047e
Gerrit-Change-Number: 13178
Gerrit-PatchSet: 5
Gerrit-Owner: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Comment-Date: Sat, 04 May 2019 00:00:17 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-4658: Potential race if compiler reorders ReachedLimit() usage.

Posted by "Tim Armstrong (Code Review)" <ge...@cloudera.org>.
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/13178 )

Change subject: IMPALA-4658: Potential race if compiler reorders ReachedLimit() usage.
......................................................................


Patch Set 4:

(2 comments)

http://gerrit.cloudera.org:8080/#/c/13178/4//COMMIT_MSG
Commit Message:

http://gerrit.cloudera.org:8080/#/c/13178/4//COMMIT_MSG@9
PS4, Line 9: The ExecNode::num_rows_returned_ member is shared by the scan node
           : (thread) and the related scanner threads. There is a potential race here
           : since the compiler is free to reorder the load/store of
           : num_rows_returned_ member. Also, num_rows_returned_ is not accessed in
           : a thread safe manner.
> Can this actually cause problems? I think that all scanners read num_rows_r
It's hard to reason about the implications. It would be valid  for the load instruction to get pulled out of a loop and potentially have a non-terminating loop. I don't think we have code where this would happen now, but who knows - the point is more that the pattern is unsafe and you can't reason about the behaviour of the functional locally.

It also is going to generate warnings in TSAN, which we'd prefer to avoid.


http://gerrit.cloudera.org:8080/#/c/13178/4/be/src/exec/hdfs-scan-node.cc
File be/src/exec/hdfs-scan-node.cc:

http://gerrit.cloudera.org:8080/#/c/13178/4/be/src/exec/hdfs-scan-node.cc@134
PS4, Line 134:     // Update the number of materialized rows now instead of when they are materialized.
             :     // This means that scanners might process and queue up more rows than are necessary
             :     // for the limit case but we want to avoid the synchronized writes to
             :     // num_rows_returned_.
             :     IncrementNumRowsReturnedShared(row_batch->num_rows());
> The comment + the Shared function seem weird together - the comment says th
Yeah the comment is a little weird now in that it's justifying using the standard pattern for num_rows_returned_ instead of an alternative design. The alternative design would actually probably violate invariants of ExecNode by incrementing num_rows_returned_ before returning the actual nodes. So it doesn't make a lot of sense in context.

We could just delete the comment I think. Or something short describing the consequences like

  // Note that the scanner threads may have processed and queued up extra rows before this thread incremented the rows returned.



-- 
To view, visit http://gerrit.cloudera.org:8080/13178
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I4cbbfad80f7ab87dd6f192a24e2c68f7c66b047e
Gerrit-Change-Number: 13178
Gerrit-PatchSet: 4
Gerrit-Owner: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Comment-Date: Thu, 02 May 2019 20:27:39 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-4658: Potential race if compiler reorders ReachedLimit() usage.

Posted by "Todd Lipcon (Code Review)" <ge...@cloudera.org>.
Todd Lipcon has posted comments on this change. ( http://gerrit.cloudera.org:8080/13178 )

Change subject: IMPALA-4658: Potential race if compiler reorders ReachedLimit() usage.
......................................................................


Patch Set 2:

(2 comments)

http://gerrit.cloudera.org:8080/#/c/13178/2/be/src/exec/exec-node.h
File be/src/exec/exec-node.h:

http://gerrit.cloudera.org:8080/#/c/13178/2/be/src/exec/exec-node.h@390
PS2, Line 390:   int64_t num_rows_returned_;
might be worth using a DFAKE_SCOPED_LOCK_THREAD_LOCKED from gutil/threading/thread_collision_warner.h on the write path to the non-shared variant which would ensure in debug builds that only a single thread ever modifies this value


http://gerrit.cloudera.org:8080/#/c/13178/2/be/src/exec/exec-node.h@391
PS2, Line 391:   AtomicInt64 num_rows_returned_shared_;
> It's a bit of a matter of taste but I think it would be better to just have
agreed, I find it very confusing to have two separate variables here, unless the semantics are something like:

- the "num_rows_returned_" is non-atomic and can only be modified by a special thread which is bound to this execnode (used by non-multi-threaded nodes)
- the "_shared_" variant can be used by multi-threaded nodes
- all readers should operate by loading both and adding the results together.

in other words, having two separate write paths seems fine as long as there is a consolidated read path.

Another option here is to use a striped counter (see LongAdder from kudu/util/striped64.h ) which might be fast enough to use multi-threaded.



-- 
To view, visit http://gerrit.cloudera.org:8080/13178
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I4cbbfad80f7ab87dd6f192a24e2c68f7c66b047e
Gerrit-Change-Number: 13178
Gerrit-PatchSet: 2
Gerrit-Owner: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Comment-Date: Tue, 30 Apr 2019 21:18:13 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-4658: Potential race if compiler reorders ReachedLimit() usage.

Posted by "Abhishek Rawat (Code Review)" <ge...@cloudera.org>.
Abhishek Rawat has uploaded a new patch set (#4). ( http://gerrit.cloudera.org:8080/13178 )

Change subject: IMPALA-4658: Potential race if compiler reorders ReachedLimit() usage.
......................................................................

IMPALA-4658: Potential race if compiler reorders ReachedLimit() usage.

The ExecNode::num_rows_returned_ member is shared by the scan node
(thread) and the related scanner threads. There is a potential race here
since the compiler is free to reorder the load/store of
num_rows_returned_ member. Also, num_rows_returned_ is not accessed in
a thread safe manner.

This change introduces regular and thread safe functions for accessing
and updating the num_rows_returned_ member, which is now private to the
ExecNode class. The getters/modifiers for single threaded code-paths
are: rows_returned(), ReachedLimit(), SetNumRowsReturned(),
IncrementNumRowsReturned(), and DecrementNumRowsReturned(). Thread safe
counterparts are: rows_returned_shared(), ReachedLimitShared(),
IncrementNumRowsReturnedShared(), and DecrementNumRowsReturnedShared().
Debug checks are added to ensure that regular and thread safe functions
are used properly in single-threaded and multi-threaded code paths,
respectively.

The initial attempt to address this issue by making num_rows_returned_
atomic caused performance regressions in single threaded code paths
which update the variable inside a loop, such as
ExchangeNode::GetNext(). The regression could be attributed to extra
path length introduced by atomic operations such as read modify
write. And since the GetNext() function could be called multiple times,
with each call updating the atomic multiple times, the regression could
be huge. Some serious thoughts were given to other approaches such as
updating the atomic num_rows_returned_ variable outside of a loop, but
such a change would have been very extensive and error prone. There is
nothing stopping someone from not adding atomic updates in a tight loop.
After much deliberation it was decided that the single threaded code
path should not be penalised and as such it probably makes sense to
use separate functions for single threaded and multi-threaded code
paths, provided the interfaces are simple enough, easy to maintain and
there are checks in place to ensure proper use.

The scan nodes with task based multi-threading (MT) also use the new
thread-safe functions, even though the variable is not shared amongst
various scan node threads. This is because both the task based MT code
path and regular scan node code path calls some common functions and it
was getting very tricky and ugly to differentiate between the two. The
overhead of using atomics is minimal for MT code path since the atomic
is not accessed for every row. The only exceptions are HBaseScanNode
and DataSourceScanNode which uses the regular functions.

Testing:
- Performance tests were run on tpch and tpcds workload. No real
  regressions were found, but both baseline and test runs showed high
  degree of variability for some queries. Multiple runs were done and
  it was concluded that the variability is not introduced by this change.

Change-Id: I4cbbfad80f7ab87dd6f192a24e2c68f7c66b047e
---
M be/src/exec/aggregation-node.cc
M be/src/exec/analytic-eval-node.cc
M be/src/exec/cardinality-check-node.cc
M be/src/exec/data-source-scan-node.cc
M be/src/exec/exchange-node.cc
M be/src/exec/exec-node.cc
M be/src/exec/exec-node.h
M be/src/exec/hbase-scan-node.cc
M be/src/exec/hdfs-avro-scanner.cc
M be/src/exec/hdfs-orc-scanner.cc
M be/src/exec/hdfs-rcfile-scanner.cc
M be/src/exec/hdfs-scan-node-base.cc
M be/src/exec/hdfs-scan-node-mt.cc
M be/src/exec/hdfs-scan-node.cc
M be/src/exec/hdfs-scanner.cc
M be/src/exec/hdfs-sequence-scanner.cc
M be/src/exec/hdfs-text-scanner.cc
M be/src/exec/kudu-scan-node-mt.cc
M be/src/exec/kudu-scan-node.cc
M be/src/exec/kudu-scanner.cc
M be/src/exec/nested-loop-join-node.cc
M be/src/exec/parquet/hdfs-parquet-scanner.cc
M be/src/exec/partial-sort-node.cc
M be/src/exec/partitioned-hash-join-node.cc
M be/src/exec/select-node-ir.cc
M be/src/exec/select-node.cc
M be/src/exec/sort-node.cc
M be/src/exec/streaming-aggregation-node.cc
M be/src/exec/subplan-node.cc
M be/src/exec/topn-node.cc
M be/src/exec/union-node.cc
M be/src/exec/unnest-node.cc
32 files changed, 174 insertions(+), 107 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/78/13178/4
-- 
To view, visit http://gerrit.cloudera.org:8080/13178
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I4cbbfad80f7ab87dd6f192a24e2c68f7c66b047e
Gerrit-Change-Number: 13178
Gerrit-PatchSet: 4
Gerrit-Owner: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>

[Impala-ASF-CR] IMPALA-4658: Potential race if compiler reorders ReachedLimit() usage.

Posted by "Todd Lipcon (Code Review)" <ge...@cloudera.org>.
Todd Lipcon has posted comments on this change. ( http://gerrit.cloudera.org:8080/13178 )

Change subject: IMPALA-4658: Potential race if compiler reorders ReachedLimit() usage.
......................................................................


Patch Set 4:

(3 comments)

http://gerrit.cloudera.org:8080/#/c/13178/4/be/src/exec/hdfs-scan-node.cc
File be/src/exec/hdfs-scan-node.cc:

http://gerrit.cloudera.org:8080/#/c/13178/4/be/src/exec/hdfs-scan-node.cc@144
PS4, Line 144:       DecrementNumRowsReturnedShared(num_rows_over);
Is this logic racy in that multiple threads could see that they pushed the limit over, and then over-compensate?

For example, with a limit of 10, and two scanner threads producing batches of 100, they might both increment by 100, see that the total rows returned is 200, and then both want to compensate by removing 190. This particular case ends up setting the rows returned counter to a negative number, but even outside of that particular issue, it seems like we might undershoot the limit


http://gerrit.cloudera.org:8080/#/c/13178/4/be/src/exec/hdfs-sequence-scanner.cc
File be/src/exec/hdfs-sequence-scanner.cc:

http://gerrit.cloudera.org:8080/#/c/13178/4/be/src/exec/hdfs-sequence-scanner.cc@295
PS4, Line 295:       scan_node_->limit() - scan_node_->rows_returned_shared();
similar question here about whether this is going to get wrong results around hitting the limit if there are actually multiple threads in this same path (I don't really know much about the MT scanner design)


http://gerrit.cloudera.org:8080/#/c/13178/4/be/src/exec/kudu-scan-node.cc
File be/src/exec/kudu-scan-node.cc:

http://gerrit.cloudera.org:8080/#/c/13178/4/be/src/exec/kudu-scan-node.cc@111
PS4, Line 111:       int num_rows_over = rows_returned_shared() - limit_;
same concern here



-- 
To view, visit http://gerrit.cloudera.org:8080/13178
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I4cbbfad80f7ab87dd6f192a24e2c68f7c66b047e
Gerrit-Change-Number: 13178
Gerrit-PatchSet: 4
Gerrit-Owner: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Comment-Date: Wed, 01 May 2019 22:57:47 +0000
Gerrit-HasComments: Yes