[jira] [Commented] (IMPALA-8779) Add RowBatchQueue interface with an implementation backed by a std::queue

["ASF subversion and git services (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/IMPALA-8779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16898329#comment-16898329 ] 

ASF subversion and git services commented on IMPALA-8779:
---------------------------------------------------------

Commit 699450aadbf45f36617472b7c777dc2d9aad066a in impala's branch refs/heads/master from Sahil Takiar
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=699450a ]

IMPALA-8779, IMPALA-8780: RowBatchQueue re-factoring and BufferedPRS impl

Improves the encapsulation of RowBatchQueue by the doing the following
re-factoring:
* Renames RowBatchQueue to BlockingRowBatchQueue which is more
indicitive of what the queue does
* Re-factors the timers managed by the scan-node into the
BlockingRowBatchQueue implementation
* Favors composition over inheritance by re-factoring
BlockingRowBatchQueue to own a BlockingQueue rather than extending one

The re-factoring lays the groundwork for introducing a generic
RowBatchQueue that all RowBatch queues inherit from.

Adds a new DequeRowBatchQueue which is a simple wrapper around a
std::deque that (1) stores unique_ptr to queued RowBatch-es and (2)
has a maximum capacity.

Implements BufferedPlanRootSink using the new DequeRowBatchQueue.
DequeRowBatchQueue is generic enough that replacing it with a
SpillableQueue (queue backed by a BufferedTupleStream) should be
straightforward. BufferedPlanRootSink is synchronized to protect access
to DequeRowBatchQueue since the queue is not thread safe.

BufferedPlanRootSink FlushFinal blocks until the consumer thread has
processed all RowBatches. This ensures that the coordinator fragment
stays alive until all results are fetched, but allows all other
fragments to be shutdown immediately.

Testing:
* Running core tests
* Updated tests/query_test/test_result_spooling.py

Change-Id: I9b1bb4b9c6f6e92c70e8fbee6ccdf48c2f85b7be
Reviewed-on: http://gerrit.cloudera.org:8080/13883
Reviewed-by: Impala Public Jenkins <im...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>


> Add RowBatchQueue interface with an implementation backed by a std::queue
> -------------------------------------------------------------------------
>
>                 Key: IMPALA-8779
>                 URL: https://issues.apache.org/jira/browse/IMPALA-8779
>             Project: IMPALA
>          Issue Type: Sub-task
>          Components: Backend
>            Reporter: Sahil Takiar
>            Assignee: Sahil Takiar
>            Priority: Major
>
> Add a {{RowBatchQueue}} interface with an implementation backed by a {{std::queue}}. Introducing a generic queue that can buffer {{RowBatch}}-es will help with the implementation of {{BufferedPlanRootSink}}. Rather than tie the {{BufferedPlanRootSink}} to a specific method of queuing row batches, we can use an interface. In future patches, a {{RowBatchQueue}} backed by a {{BufferedTupleStream}} can easily be switched out in {{BufferedPlanRootSink}}.
> We should consider re-factoring the existing {{RowBatchQueue}} to use the new interface. The KRPC receiver does some buffering of {{RowBatch}}-es as well which might benefit from the new RowBatchQueue interface, and some more KRPC buffering might be added in IMPALA-6692.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org