You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@ignite.apache.org by ilyn <iy...@att.com> on 2019/03/28 20:02:15 UTC

JDBC Thin Client Set Streaming On/Off in node.js app

Hi all,

I currently have an ETL application that parses raw logs and builds batch
inserts that load our Oracle table.  We're trying to build out our Apache
Ignite environment that will load a subset of the data.  Using the
IgniteJdbcThinDriver I wrote a simple node.js program that connects to our
Ignite node, sets streaming on, inserts X rows of data, sets streaming off,
disconnects.

For some reason sometimes all X records are inserted, but other times 1 or 2
records are not inserted.  No matter how many records I insert the missing
record seems to always be the first, second, or third.  Initially, I thought
I was beginning insertion too quickly after SET STREAMING ON, but the same
problem occurs even if I add a minute delay.

Any help would be greatly appreciated.
TIA!
Ilyn



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: JDBC Thin Client Set Streaming On/Off in node.js app

Posted by ilyn <iy...@att.com>.
Thank you so much Ilya!

I was able to successfully test, now I need to implement into my real code.
Hopefully you won't be hearing from me for awhile :)



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: JDBC Thin Client Set Streaming On/Off in node.js app

Posted by Ilya Kasnacheev <il...@gmail.com>.
Hello!

I think the problem is that you are running parallel operations on a single
connection. This is a no-no with Ignite's connection (and some other as
well, I would guess).You should be only use a JDBC connection sequentially,
but your insertRow() would do that in parallel.

Please see how I managed to sequentialize this operation and now have 15 in
table after running:
        function(callback) {
          const cities = [
            [16,'New York', 'USA', 'New York', 8008278],
            [17,'Los Angeles', 'USA', 'California', 3694820],
            [18,'Chicago', 'USA', 'Illinois', 2896016],
            [4,'Houston', 'USA', 'Texas', 1953631],
            [5,'Philadelphia', 'USA', 'Pennsylvania', 1517550],
            [6,'Moscow', 'RUS', 'Moscow (City)', 8389200],
            [7,'St Petersburg', 'RUS', 'Pietari', 4694000],
            [8,'Novosibirsk', 'RUS', 'Novosibirsk', 1398800],
            [9,'Nizni Novgorod', 'RUS', 'Nizni Novgorod', 1357000],
            [10,'Jekaterinburg', 'RUS', 'Sverdlovsk', 1266300],
            [11,'Shanghai', 'CHN', 'Shanghai', 9696300],
            [12,'Peking', 'CHN', 'Peking', 7472000],
            [13,'Chongqing', 'CHN', 'Chongqing', 6351600],
            [14,'Tianjin', 'CHN', 'Tianjin', 5286800],
            [15,'Wuhan', 'CHN', 'Hubei', 4344600]
          ];
          inserts = [];
          for (var city of cities) {
            var sql = "INSERT INTO BUTCH_city (id, name, countrycode,
district, population)
VALUES("+city[0]+",'"+city[1]+"','"+city[2]+"','"+city[3]+"',"+city[4]+");";
            inserts.push((function() {
              var s = sql;
              return function(callback) {
                console.log((new Date()) + " insert sql : " + s);
                insertRow(conn, s, callback);
              }
            })());
          }
          asyncjs.series(inserts, callback);

        },

Sorry for my awful Node code.

Regards,
-- 
Ilya Kasnacheev


пт, 29 мар. 2019 г. в 19:48, ilyn <iy...@att.com>:

> Ilya,
>
> Please see attached code.
>
> jdbcStream.js
> <http://apache-ignite-users.70518.x6.nabble.com/file/t2367/jdbcStream.js>
>
>
> Thanks!
> Ilyn
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>

Re: JDBC Thin Client Set Streaming On/Off in node.js app

Posted by ilyn <iy...@att.com>.
Ilya,

Please see attached code.

jdbcStream.js
<http://apache-ignite-users.70518.x6.nabble.com/file/t2367/jdbcStream.js>  

Thanks!
Ilyn



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: JDBC Thin Client Set Streaming On/Off in node.js app

Posted by Ilya Kasnacheev <il...@gmail.com>.
Hello!

Is it possible to make a reproducer project for this behavior?

Regards,
-- 
Ilya Kasnacheev


чт, 28 мар. 2019 г. в 23:02, ilyn <iy...@att.com>:

> Hi all,
>
> I currently have an ETL application that parses raw logs and builds batch
> inserts that load our Oracle table.  We're trying to build out our Apache
> Ignite environment that will load a subset of the data.  Using the
> IgniteJdbcThinDriver I wrote a simple node.js program that connects to our
> Ignite node, sets streaming on, inserts X rows of data, sets streaming off,
> disconnects.
>
> For some reason sometimes all X records are inserted, but other times 1 or
> 2
> records are not inserted.  No matter how many records I insert the missing
> record seems to always be the first, second, or third.  Initially, I
> thought
> I was beginning insertion too quickly after SET STREAMING ON, but the same
> problem occurs even if I add a minute delay.
>
> Any help would be greatly appreciated.
> TIA!
> Ilyn
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>