[jira] [Commented] (TRAFODION-3183) fetch huge data from server side will have core

["ASF GitHub Bot (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/TRAFODION-3183?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16726376#comment-16726376 ] 

ASF GitHub Bot commented on TRAFODION-3183:
-------------------------------------------

Github user selvaganesang commented on a diff in the pull request:

    https://github.com/apache/trafodion/pull/1694#discussion_r243471406
  
    --- Diff: core/conn/jdbcT4/src/main/java/org/trafodion/jdbc/t4/TrafT4ResultSet.java ---
    @@ -2779,6 +2779,19 @@ public boolean next() throws SQLException {
     					maxRowCnt = maxRows - totalRowsFetched_;
     				}
     
    +        // if (row width) * (fetch rows) too large, there will have core in server side.
    +        // once fetch bytes bigger than 1GB, devide several times to fetch,
    +        // each time fetch bytes less than 1GB.
    +        if (outputDesc_ != null && outputDesc_[0] != null) {
    +          long rowLength = outputDesc_[0].rowLength_;
    +          long oneGB = 1024 * 1024 * 1024;
    +          if (rowLength * maxRowCnt >= oneGB) {
    +            double multi = (rowLength * maxRowCnt) / (double)oneGB;
    +            multi = Math.ceil(multi); // devide several times to fetch
    +            maxRowCnt = (int) (maxRowCnt / multi);
    +          }
    +        }
    +
    --- End diff --
    
    Just spent some time looking at this change. As per JDBC specification, Statement.setMaxRows  sets the upper limit on the number of rows in the result set. Statement.setFetchSize sets the hint for the driver to fetch how many rows are fetched in next(). maxRows_ and fetchSize_ in this method corresponds to these values respectively.  It is possible that the application doesn't  set this value at all.  Any case, JDBC driver can decide the fetchSize based on the hint or its limitations.  I would suggest the following:
    
    1. Move this logic to calculate the number of rows to getFetchSize
    2. Use a data  source property to get the fetch size in terms of MB rather than assuming 1GB in calculation. Assume a default value a far less than 1GB says 50 or 100 MB if the property is not set.
    3. Calculate the number of rows fetchSize_ based on the setFetchSize and this  property value which ever doesn't exceed the size in terms of MB.
    4. Use getFetchSize() in this method to get the fetchSize_
    
    This would ensure that Trafodion conforms to JDBC specification in a better way because it would  let the application know how the hint of setFetchSize works.


> fetch huge data from server side will have core 
> ------------------------------------------------
>
>                 Key: TRAFODION-3183
>                 URL: https://issues.apache.org/jira/browse/TRAFODION-3183
>             Project: Apache Trafodion
>          Issue Type: Bug
>          Components: client-jdbc-t4
>    Affects Versions: any
>            Reporter: mashengchen
>            Assignee: mashengchen
>            Priority: Major
>             Fix For: 2.4
>
>
> when search a table which row lenth is very large, and set fetch row a big size then there will have core in server side, because of mxosrvr malloc a large memory. Then client will met "Server aborted abnormally or Connection timed out"



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)