You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@ignite.apache.org by Muhammed Favas <fa...@expeedsoftware.com> on 2019/10/22 10:41:30 UTC
Write python dataframe to ignite table.
Hi,
Is there a way to bulk load python dataframe values to ignite table?
Regards,
Favas
Re: Write python dataframe to ignite table.
Posted by Denis Magda <dm...@apache.org>.
Andrey, Alexey, Nikolay,
Do we have any solutions for calling Ignite + Spark implementation from
Python?
-
Denis
On Tue, Oct 22, 2019 at 3:41 AM Muhammed Favas <
favas.muhammed@expeedsoftware.com> wrote:
> Hi,
>
>
>
> Is there a way to bulk load python dataframe values to ignite table?
>
>
>
> *Regards,*
>
> *Favas *
>
>
>
RE: Write python dataframe to ignite table.
Posted by Muhammed Favas <fa...@expeedsoftware.com>.
Hi Ilya,
Thanks for the response!!
My requirement to have the flexibility to map the csv column to specific table column.
The file I may be getting will have more columns and I want to import some specific set of columns to my table. I assume that flexibility is not available currently in COPY command.
Right now I have achieved it through spark dataframe for mapping the column and bulk insert into ignite. It is moreover giving nearly same performance as ignite COPY command.
Regards,
Favas
From: Ilya Kasnacheev <il...@gmail.com>
Sent: Tuesday, October 29, 2019 8:46 PM
To: user@ignite.apache.org
Subject: Re: Write python dataframe to ignite table.
Hello!
Well, COPY command does allow you to do column mapping:
COPY FROM '/path/to/local/file.csv'
INTO tablename (columnName, columnName, ...) FORMAT CSV
If you need to do non-trivial transformations, you can use JDBC driver in SET STREAMING ON mode.
Regards,
--
Ilya Kasnacheev
вт, 29 окт. 2019 г. в 13:11, Muhammed Favas <fa...@expeedsoftware.com>>:
Hi,
I have tried simple python program without using spark. First I read whole csv into python dataframe using pandas library.
Now I want to bulk insert the whole dataframe into ignite table without looping through.
The purpose of this test is to evaluate the best way(means faster wat) to bulk load csv files into ignite.
Ignite COPY command I can not use here, because I need an option to do column mapping while import csv files.
Regards,
Favas
From: Stephen Darlington <st...@gridgain.com>>
Sent: Monday, October 28, 2019 5:05 PM
To: user@ignite.apache.org<ma...@ignite.apache.org>
Subject: Re: Write python dataframe to ignite table.
What have you tried? As long as your command-line includes the right JAR files it seems to more-or-less just work for me:
https://medium.com/@sdarlington/the-trick-to-successfully-integrating-apache-ignite-and-pyspark-890e436d09ba
Regards,
Stephen
On 22 Oct 2019, at 11:41, Muhammed Favas <fa...@expeedsoftware.com>> wrote:
Hi,
Is there a way to bulk load python dataframe values to ignite table?
Regards,
Favas
Re: Write python dataframe to ignite table.
Posted by Ilya Kasnacheev <il...@gmail.com>.
Hello!
Well, COPY command does allow you to do column mapping:
COPY FROM '/path/to/local/file.csv'INTO tablename (*columnName,
columnName, ...*) FORMAT CSV
If you need to do non-trivial transformations, you can use JDBC driver in
SET STREAMING ON mode.
Regards,
--
Ilya Kasnacheev
вт, 29 окт. 2019 г. в 13:11, Muhammed Favas <
favas.muhammed@expeedsoftware.com>:
> Hi,
>
>
>
> I have tried simple python program without using spark. First I read whole
> csv into python dataframe using pandas library.
>
> Now I want to bulk insert the whole dataframe into ignite table without
> looping through.
>
>
>
> The purpose of this test is to evaluate the best way(means faster wat) to
> bulk load csv files into ignite.
>
>
>
> Ignite COPY command I can not use here, because I need an option to do
> column mapping while import csv files.
>
>
>
>
>
> *Regards,*
>
> *Favas *
>
>
>
> *From:* Stephen Darlington <st...@gridgain.com>
> *Sent:* Monday, October 28, 2019 5:05 PM
> *To:* user@ignite.apache.org
> *Subject:* Re: Write python dataframe to ignite table.
>
>
>
> What have you tried? As long as your command-line includes the right JAR
> files it seems to more-or-less just work for me:
>
>
>
>
> https://medium.com/@sdarlington/the-trick-to-successfully-integrating-apache-ignite-and-pyspark-890e436d09ba
>
>
>
> Regards,
>
> Stephen
>
>
>
> On 22 Oct 2019, at 11:41, Muhammed Favas <
> favas.muhammed@expeedsoftware.com> wrote:
>
>
>
> Hi,
>
>
>
> Is there a way to bulk load python dataframe values to ignite table?
>
>
>
> *Regards,*
>
> *Favas *
>
>
>
Re: Write python dataframe to ignite table.
Posted by Stephen Darlington <st...@gridgain.com>.
Ah, sorry. When I see “DataFrame” I immediately think of Spark.
While that would be a great addition, I don’t think the Python thin-client currently supports DataFrames. I think your two options are to use Spark DataFrames or loop through, using put/putAll to import that data. Three options if you include creating a patch for the thin-client ;)
Regards,
Stephen
> On 29 Oct 2019, at 10:04, Muhammed Favas <fa...@expeedsoftware.com> wrote:
>
> Hi,
>
> I have tried simple python program without using spark. First I read whole csv into python dataframe using pandas library.
> Now I want to bulk insert the whole dataframe into ignite table without looping through.
>
> The purpose of this test is to evaluate the best way(means faster wat) to bulk load csv files into ignite.
>
> Ignite COPY command I can not use here, because I need an option to do column mapping while import csv files.
>
>
> Regards,
> Favas
>
> From: Stephen Darlington <st...@gridgain.com>
> Sent: Monday, October 28, 2019 5:05 PM
> To: user@ignite.apache.org
> Subject: Re: Write python dataframe to ignite table.
>
> What have you tried? As long as your command-line includes the right JAR files it seems to more-or-less just work for me:
>
> https://medium.com/@sdarlington/the-trick-to-successfully-integrating-apache-ignite-and-pyspark-890e436d09ba <https://medium.com/@sdarlington/the-trick-to-successfully-integrating-apache-ignite-and-pyspark-890e436d09ba>
>
> Regards,
> Stephen
>
>
> On 22 Oct 2019, at 11:41, Muhammed Favas <favas.muhammed@expeedsoftware.com <ma...@expeedsoftware.com>> wrote:
>
> Hi,
>
> Is there a way to bulk load python dataframe values to ignite table?
>
> Regards,
> Favas
RE: Write python dataframe to ignite table.
Posted by Muhammed Favas <fa...@expeedsoftware.com>.
Hi,
I have tried simple python program without using spark. First I read whole csv into python dataframe using pandas library.
Now I want to bulk insert the whole dataframe into ignite table without looping through.
The purpose of this test is to evaluate the best way(means faster wat) to bulk load csv files into ignite.
Ignite COPY command I can not use here, because I need an option to do column mapping while import csv files.
Regards,
Favas
From: Stephen Darlington <st...@gridgain.com>
Sent: Monday, October 28, 2019 5:05 PM
To: user@ignite.apache.org
Subject: Re: Write python dataframe to ignite table.
What have you tried? As long as your command-line includes the right JAR files it seems to more-or-less just work for me:
https://medium.com/@sdarlington/the-trick-to-successfully-integrating-apache-ignite-and-pyspark-890e436d09ba
Regards,
Stephen
On 22 Oct 2019, at 11:41, Muhammed Favas <fa...@expeedsoftware.com>> wrote:
Hi,
Is there a way to bulk load python dataframe values to ignite table?
Regards,
Favas
Re: Write python dataframe to ignite table.
Posted by Stephen Darlington <st...@gridgain.com>.
What have you tried? As long as your command-line includes the right JAR files it seems to more-or-less just work for me:
https://medium.com/@sdarlington/the-trick-to-successfully-integrating-apache-ignite-and-pyspark-890e436d09ba <https://medium.com/@sdarlington/the-trick-to-successfully-integrating-apache-ignite-and-pyspark-890e436d09ba>
Regards,
Stephen
> On 22 Oct 2019, at 11:41, Muhammed Favas <fa...@expeedsoftware.com> wrote:
>
> Hi,
>
> Is there a way to bulk load python dataframe values to ignite table?
>
> Regards,
> Favas