You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@ignite.apache.org by Anil <an...@gmail.com> on 2016/12/01 13:10:43 UTC

Re: Fetching large number of records

Hi Val,

Yes. results are fetched in pages only. But each execute query will return
the results of a page. correct ?

I am checking if there way that direct query parser stream is available to
client though jdbc driver. Hope this is clear.

Thanks

On 1 December 2016 at 04:34, vkulichenko <va...@gmail.com>
wrote:

> Anil,
>
> JDBC driver actually uses the same Ignite API under the hood, so the
> results
> are fetched in pages as well.
>
> As for the query parser question, I didn't quite understand what you mean.
> Can you please give more details?
>
> -Val
>
>
>
> --
> View this message in context: http://apache-ignite-users.
> 70518.x6.nabble.com/Fetching-large-number-of-records-tp9267p9313.html
> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
>

Re: Fetching large number of records

Posted by vkulichenko <va...@gmail.com>.
Anil,

You're setting 100 limit for result set rows, that's why you see only 100
rows.

statement.setMaxRows(100);

-Val



--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Fetching-large-number-of-records-tp9267p9491.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: Fetching large number of records

Posted by Anil <an...@gmail.com>.
Hi Val,

Seems ResultSet#next() not fetching next page.

i tried the following snippet -

JdbcConnection conn =
(JdbcConnection)DriverManager.getConnection("<configuriation url>");

String sql = "select count from Test_counts where id is not null limit
1000";

PreparedStatement statement = conn.prepareStatement(sql);
statement.setMaxRows(100);

int i = 0;
ResultSet rs = statement.executeQuery();
while (rs.next() && i++ < 1000) {
System.out.println(rs.getString("count"));
}

i see only 100 records and 1000 is the expected records.

Do you see any issue with above snippet ? thanks


Thanks


On 3 December 2016 at 08:49, Anil <an...@gmail.com> wrote:

> Hi Val,
>
> Thanks for clarification. I understand something and i will give a try.
>
> Thanks.
>
> On 2 December 2016 at 23:17, vkulichenko <va...@gmail.com>
> wrote:
>
>> Anil,
>>
>> The JdbcQueryTask is executed each time the next page is needed. And the
>> number of rows returned by the task is limited by fetchSize:
>>
>>     if (rows.size() == fetchSize) // If fetchSize is 0 then unlimited
>>         break;
>>
>> The cursor is cached and reused there, so is this task is executed twice
>> for
>> the same result set, it will not execute the query from scratch, but will
>> get the existing cursor and start iteration from where it finished on the
>> first invocation.
>>
>> I'm not completely sure that I correctly understand what you mean by
>> streaming here, but paging is definitely in place and that's how it works
>> now.
>>
>> -Val
>>
>>
>>
>> --
>> View this message in context: http://apache-ignite-users.705
>> 18.x6.nabble.com/Fetching-large-number-of-records-tp9267p9373.html
>> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
>>
>
>

Re: Fetching large number of records

Posted by Anil <an...@gmail.com>.
Hi Val,

Thanks for clarification. I understand something and i will give a try.

Thanks.

On 2 December 2016 at 23:17, vkulichenko <va...@gmail.com>
wrote:

> Anil,
>
> The JdbcQueryTask is executed each time the next page is needed. And the
> number of rows returned by the task is limited by fetchSize:
>
>     if (rows.size() == fetchSize) // If fetchSize is 0 then unlimited
>         break;
>
> The cursor is cached and reused there, so is this task is executed twice
> for
> the same result set, it will not execute the query from scratch, but will
> get the existing cursor and start iteration from where it finished on the
> first invocation.
>
> I'm not completely sure that I correctly understand what you mean by
> streaming here, but paging is definitely in place and that's how it works
> now.
>
> -Val
>
>
>
> --
> View this message in context: http://apache-ignite-users.
> 70518.x6.nabble.com/Fetching-large-number-of-records-tp9267p9373.html
> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
>

Re: Fetching large number of records

Posted by vkulichenko <va...@gmail.com>.
Anil,

The JdbcQueryTask is executed each time the next page is needed. And the
number of rows returned by the task is limited by fetchSize:

    if (rows.size() == fetchSize) // If fetchSize is 0 then unlimited
        break;

The cursor is cached and reused there, so is this task is executed twice for
the same result set, it will not execute the query from scratch, but will
get the existing cursor and start iteration from where it finished on the
first invocation.

I'm not completely sure that I correctly understand what you mean by
streaming here, but paging is definitely in place and that's how it works
now.

-Val



--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Fetching-large-number-of-records-tp9267p9373.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: Fetching large number of records

Posted by Anil <an...@gmail.com>.
Hi Val,

I think No. Correct me if i am wrong.

i was looking at following code pieces -

1. JdbcResultSet#next()

try {
                JdbcQueryTask.QueryResult res =
                    loc ? qryTask.call() :
ignite.compute(ignite.cluster().forNodeId(nodeId)).call(qryTask);

                finished = res.isFinished();

                it = res.getRows().iterator();

                return next();
            }

 res.getRows() returns all the results and all are in memory. correct ?

2. JdbcQueryTask#call

List<List<?>> rows = new ArrayList<>();

        for (List<?> row : cursor) {
            List<Object> row0 = new ArrayList<>(row.size());

            for (Object val : row)
                row0.add(JdbcUtils.sqlType(val) ? val : val.toString());

            rows.add(row0);

            if (rows.size() == fetchSize) // If fetchSize is 0 then
unlimited
                break;
        }

all cursor rows are fetched and sent to #1

next() method is a just iterator on available rows. agree ? this is not a
streaming from server to client. Am i wrong ?

Thanks

On 2 December 2016 at 04:11, vkulichenko <va...@gmail.com>
wrote:

> Anil,
>
> While you iterate through the ResultSet on the client, it will fetch
> results
> from server in pages. Are you looking for something else?
>
> -Val
>
>
>
> --
> View this message in context: http://apache-ignite-users.
> 70518.x6.nabble.com/Fetching-large-number-of-records-tp9267p9342.html
> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
>

Re: Fetching large number of records

Posted by vkulichenko <va...@gmail.com>.
Anil,

While you iterate through the ResultSet on the client, it will fetch results
from server in pages. Are you looking for something else?

-Val



--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Fetching-large-number-of-records-tp9267p9342.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.