You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@ignite.apache.org by ilyn <iy...@att.com> on 2019/03/28 20:02:15 UTC
JDBC Thin Client Set Streaming On/Off in node.js app
Hi all,
I currently have an ETL application that parses raw logs and builds batch
inserts that load our Oracle table. We're trying to build out our Apache
Ignite environment that will load a subset of the data. Using the
IgniteJdbcThinDriver I wrote a simple node.js program that connects to our
Ignite node, sets streaming on, inserts X rows of data, sets streaming off,
disconnects.
For some reason sometimes all X records are inserted, but other times 1 or 2
records are not inserted. No matter how many records I insert the missing
record seems to always be the first, second, or third. Initially, I thought
I was beginning insertion too quickly after SET STREAMING ON, but the same
problem occurs even if I add a minute delay.
Any help would be greatly appreciated.
TIA!
Ilyn
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: JDBC Thin Client Set Streaming On/Off in node.js app
Posted by ilyn <iy...@att.com>.
Thank you so much Ilya!
I was able to successfully test, now I need to implement into my real code.
Hopefully you won't be hearing from me for awhile :)
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: JDBC Thin Client Set Streaming On/Off in node.js app
Posted by Ilya Kasnacheev <il...@gmail.com>.
Hello!
I think the problem is that you are running parallel operations on a single
connection. This is a no-no with Ignite's connection (and some other as
well, I would guess).You should be only use a JDBC connection sequentially,
but your insertRow() would do that in parallel.
Please see how I managed to sequentialize this operation and now have 15 in
table after running:
function(callback) {
const cities = [
[16,'New York', 'USA', 'New York', 8008278],
[17,'Los Angeles', 'USA', 'California', 3694820],
[18,'Chicago', 'USA', 'Illinois', 2896016],
[4,'Houston', 'USA', 'Texas', 1953631],
[5,'Philadelphia', 'USA', 'Pennsylvania', 1517550],
[6,'Moscow', 'RUS', 'Moscow (City)', 8389200],
[7,'St Petersburg', 'RUS', 'Pietari', 4694000],
[8,'Novosibirsk', 'RUS', 'Novosibirsk', 1398800],
[9,'Nizni Novgorod', 'RUS', 'Nizni Novgorod', 1357000],
[10,'Jekaterinburg', 'RUS', 'Sverdlovsk', 1266300],
[11,'Shanghai', 'CHN', 'Shanghai', 9696300],
[12,'Peking', 'CHN', 'Peking', 7472000],
[13,'Chongqing', 'CHN', 'Chongqing', 6351600],
[14,'Tianjin', 'CHN', 'Tianjin', 5286800],
[15,'Wuhan', 'CHN', 'Hubei', 4344600]
];
inserts = [];
for (var city of cities) {
var sql = "INSERT INTO BUTCH_city (id, name, countrycode,
district, population)
VALUES("+city[0]+",'"+city[1]+"','"+city[2]+"','"+city[3]+"',"+city[4]+");";
inserts.push((function() {
var s = sql;
return function(callback) {
console.log((new Date()) + " insert sql : " + s);
insertRow(conn, s, callback);
}
})());
}
asyncjs.series(inserts, callback);
},
Sorry for my awful Node code.
Regards,
--
Ilya Kasnacheev
пт, 29 мар. 2019 г. в 19:48, ilyn <iy...@att.com>:
> Ilya,
>
> Please see attached code.
>
> jdbcStream.js
> <http://apache-ignite-users.70518.x6.nabble.com/file/t2367/jdbcStream.js>
>
>
> Thanks!
> Ilyn
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>
Re: JDBC Thin Client Set Streaming On/Off in node.js app
Posted by ilyn <iy...@att.com>.
Ilya,
Please see attached code.
jdbcStream.js
<http://apache-ignite-users.70518.x6.nabble.com/file/t2367/jdbcStream.js>
Thanks!
Ilyn
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: JDBC Thin Client Set Streaming On/Off in node.js app
Posted by Ilya Kasnacheev <il...@gmail.com>.
Hello!
Is it possible to make a reproducer project for this behavior?
Regards,
--
Ilya Kasnacheev
чт, 28 мар. 2019 г. в 23:02, ilyn <iy...@att.com>:
> Hi all,
>
> I currently have an ETL application that parses raw logs and builds batch
> inserts that load our Oracle table. We're trying to build out our Apache
> Ignite environment that will load a subset of the data. Using the
> IgniteJdbcThinDriver I wrote a simple node.js program that connects to our
> Ignite node, sets streaming on, inserts X rows of data, sets streaming off,
> disconnects.
>
> For some reason sometimes all X records are inserted, but other times 1 or
> 2
> records are not inserted. No matter how many records I insert the missing
> record seems to always be the first, second, or third. Initially, I
> thought
> I was beginning insertion too quickly after SET STREAMING ON, but the same
> problem occurs even if I add a minute delay.
>
> Any help would be greatly appreciated.
> TIA!
> Ilyn
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>