You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues-all@impala.apache.org by "ASF subversion and git services (JIRA)" <ji...@apache.org> on 2019/07/19 07:45:00 UTC
[jira] [Commented] (IMPALA-8656) Support for eagerly fetching and
spooling all query result rows
[ https://issues.apache.org/jira/browse/IMPALA-8656?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16888614#comment-16888614 ]
ASF subversion and git services commented on IMPALA-8656:
---------------------------------------------------------
Commit 4ff2cd86e26876607ece8ac705e6dabc1f338c07 in impala's branch refs/heads/master from stakiar
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=4ff2cd8 ]
IMPALA-8656: Re-factor PlanRootSink into blocking and buffered implementations
Refactors PlanRootSink into a base class with two subclasses:
BlockingPlanRootSink and BufferedPlanRootSink. BlockingPlanRootSink
encapsulates the current implementation of PlanRootSink and
BufferedPlanRootSink encapsulates a new implementation that will buffer
RowBatches in memory until they are read by the client. The
implementation of BlockingPlanRootSink is left to future work.
A new query option called POOL_QUERY_RESULTS controls whether a
BlockingPlanRootSink or a BufferedPlanRootSink is used as the DataSink.
POOL_QUERY_RESULTS is false by default.
Added a few more docs to PlanRootSink and BlockingPlanRootSink to make
the implementation easier to understand.
Testing:
* Added tests/query_test/test_result_spooling.py; currently only runs a
simple select limit 10 with SPOOL_QUERY_RESULTS = true and validates
that 0 rows are returned
* Ran core tests
Change-Id: I8786b1a9af68ab0a8a094970d8f955eb20d04bca
Reviewed-on: http://gerrit.cloudera.org:8080/13873
Reviewed-by: Impala Public Jenkins <im...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
> Support for eagerly fetching and spooling all query result rows
> ---------------------------------------------------------------
>
> Key: IMPALA-8656
> URL: https://issues.apache.org/jira/browse/IMPALA-8656
> Project: IMPALA
> Issue Type: Improvement
> Components: Backend
> Affects Versions: Impala 2.12.0, Impala 3.2.0
> Reporter: Michael Ho
> Assignee: Sahil Takiar
> Priority: Critical
>
> Impala's current interaction with clients is pulled-based: it relies on clients to fetch results to trigger the generation of more result row batches until all the result rows have been produced. If a client issues a query without fetching all the results, the query fragments will continue to consume the resources until the query hits is cancelled and unregistered for whatever reasons. This is undesirable as resources are held up by misbehaving clients and other queries may wait for extended period of time in admission control due to this.
> The high level idea for this JIRA is for Impala to have a mode in which result sets of queries are eagerly fetched and spooled somewhere (preferably some persistent storage). In this way, the cluster's resources are freed up once all result rows have been fetched and stored in the spooling location. Incoming client fetches can be returned from this spooled locations.
> cc'ing [~stakiar], [~twm378], [~joemcdonnell], [~lv]
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org