[jira] [Commented] (IMPALA-9692) Model QuerySchedule as a protobuf

["ASF subversion and git services (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/IMPALA-9692?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17148982#comment-17148982 ] 

ASF subversion and git services commented on IMPALA-9692:
---------------------------------------------------------

Commit b142d0324cb155fd8dbda704ed569583eeed8af5 in impala's branch refs/heads/master from Thomas Tauber-Marshall
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=b142d03 ]

IMPALA-9692 (part 3): Model QuerySchedule as a protobuf

In order to support the new admission control service, we need to be
able to return the results of an admission attempt, i.e. the query
schedule, to the coordinator.

To enable this, this patch moves all parts of the QuerySchedule class
and related classes that are required by the coordinator into a new
message QuerySchedulePB. The main admission control interface,
SubmitForAdmission(), now returns a QuerySchedulePB.

Some notable changes:
- Previously, QuerySchedule was used by Coordinator as a way to pass
  around references to parts of the TExecRequest to places like
  Coordinator::ExecSummary and Coordinator::BackendState. This has
  been replaced with the ExecParams class, which is a container for
  references to the TExecRequest and QuerySchedulePB along with
  convenience functions for accessing them.
- Similarly, FragmentExecParams, which is part of QuerySchedule,
  contains references to the associated TPlanFragment, owned by the
  TExecRequest, which were used by the Coordinator when iterating over
  the schedule to initiate the query. Since FragmentExecParamsPB can't
  contain these references, they were replaced by a map between
  fragment idx and TPlanFragment in ExecParams.
- In order to keep payloads reasonable for the eventual RPC interface,
  AdmissionController::ReleaseQuery() and ReleaseQueryBackend() now
  take a query id as a parameter instead of a QuerySchedule. To
  facilitate this, AdmissionController now maintains a map from query
  ids of running queries to the resources that were allocated for them
  so that it can look the resources up when releasing them. This map
  will be necessary when implementing the admission control service to
  facilitate proper accounting of resouces in cases like coordinator
  failures.
- As scheduling is currently organized, we first construct the
  FragmentExecParams with the FInstanceExecParams as their children,
  then we construct the BackendExecParams which get references to
  their FInstanceExecParams. Since we can't send references like these
  through an rpc, we now instead Swap() the FInstanceExecParamsPB
  into the BackendExecParamsPB.

Testing:
- Updated related tests.
- Passed a full run of existing tests.

Change-Id: I1db64e72f84604b1d8ac24e0bdd4ad6bedd6bcd9
Reviewed-on: http://gerrit.cloudera.org:8080/15961
Reviewed-by: Impala Public Jenkins <im...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>


> Model QuerySchedule as a protobuf
> ---------------------------------
>
>                 Key: IMPALA-9692
>                 URL: https://issues.apache.org/jira/browse/IMPALA-9692
>             Project: IMPALA
>          Issue Type: Sub-task
>          Components: Distributed Exec
>    Affects Versions: Impala 4.0
>            Reporter: Thomas Tauber-Marshall
>            Assignee: Thomas Tauber-Marshall
>            Priority: Major
>
> To support the new admission control service, we'll need to be able to send query schedule info from the admission controller back to the coordinators.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org