[jira] [Updated] (CALCITE-3333) Add time-based ResultSet frame size limiting

["Gabriel Reid (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/CALCITE-3333?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Gabriel Reid updated CALCITE-3333:
----------------------------------
    Summary: Add time-based ResultSet frame size limiting  (was: Add time-based of ResultSet frame size limiting)

> Add time-based ResultSet frame size limiting
> --------------------------------------------
>
>                 Key: CALCITE-3333
>                 URL: https://issues.apache.org/jira/browse/CALCITE-3333
>             Project: Calcite
>          Issue Type: New Feature
>          Components: avatica
>            Reporter: Gabriel Reid
>            Priority: Major
>              Labels: pull-request-available
>          Time Spent: 10m
>  Remaining Estimate: 0h
>
> The size of a single JDBC ResultSet frame returned in a single {{prepareAndExecute}} or {{fetch}} invocation is currently 100, meaning that each retrieval of a portion of a ResultSet will send 100 rows over the wire. This frame size may be too big in some situations, and too small in other situations.
> If the underlying data source being queried can provide thousands of (small) records per second, then only reading them at 100 per RPC call will be unnecessarily slow.
> On the other hand, if the underlying data source is only providing records at a rate of 1 per second, then it will take 100 seconds for each RPC call to return, which can lead to timeouts (particularly if Avatica server is sitting behind a proxy that has a strict request timeout).
> The main factors to take into account when finding an ideal size of frame to return for each RPC call are:
> * make the frames small enough that they don't overload either Avatica server or the client with overly large amounts of data at one time
> * make the frames large enough so that the percentage of total query time that is spent only on RPC overhead is minimized
> The general idea of this ticket is to add a pluggable "frame size limiting" functionality so that frame size limiting can be done based on the number of rows, number of bytes, amount of time spent building a frame, or any other property or combination of properties.
> Note that CALCITE-2322 contains some work to allow configuring the size of a single frame on a Connection or Statement (via the {{setFetchSize}} method), although it's not yet merged in. That ticket would also be useful, and does not conflict with the general intent of this ticket.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)