You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@ignite.apache.org by GitBox <gi...@apache.org> on 2023/01/03 06:42:43 UTC
[GitHub] [ignite] JDRanpariya opened a new issue, #10469: Question on loading data onto ignite
JDRanpariya opened a new issue, #10469:
URL: https://github.com/apache/ignite/issues/10469
#### I tried searching for where to ask questions but didn't found anything useful.Hence the issue.
### What are various ways though which I can load data from Postgres table apart from using spark?
- I've tried reading data directly through Psycopg2 to Pyignite(thin client) but it seems slow and have parsing errors.
- I read data streamers are best for high speed transfer but is it possible to have streamer from Postgres(RDBMS) to ignite just for loading whole table using python? Its around 100k rows.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] ptupitsyn closed issue #10469: Question on loading data onto ignite
Posted by GitBox <gi...@apache.org>.
ptupitsyn closed issue #10469: Question on loading data onto ignite
URL: https://github.com/apache/ignite/issues/10469
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] ptupitsyn commented on issue #10469: Question on loading data onto ignite
Posted by GitBox <gi...@apache.org>.
ptupitsyn commented on issue #10469:
URL: https://github.com/apache/ignite/issues/10469#issuecomment-1369453409
Python thin client does not have `DataStreamer` yet. But you should be able to achieve good performance with batched `putAll`. Instead of 100k `put` calls, try doing 100 `putAll` calls, 1000 items each. This is roughly what `DataStreamer` does.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org