You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Bill Sparks <js...@cray.com> on 2015/07/06 09:30:48 UTC

data conversion between flink and "other" paradigms

Just a question if there was some prior-art here. Just say someone wanted to use flink for processing, but at some point they wanted to call another function via say JNI/C which doesn't understand DataSet's. How would one go about this ... I'm assuming the code would have to convert the data to a common format prior to calling the function.





Regards,

   Bill.

RE: data conversion between flink and "other" paradigms

Posted by Bill Sparks <js...@cray.com>.
Fabian.



Thanks for the info and pointer to python. I'll check it out.



-Bill

________________________________
From: Fabian Hueske [fhueske@gmail.com]
Sent: Monday, July 06, 2015 3:23 AM
To: user@flink.apache.org
Subject: Re: data conversion between flink and "other" paradigms

Hi Bill,

a DataSet is just a logical concept in Flink. DataSets are often not persisted and just streamed along operators. At the moment, there is no way to access an intermediate DataSet of a Flink program directly (this might change in the future).

You can process data in another function by implementing a Java user function (for example a MapPartition function) and sending the data through JNI to a C function (if you need the full data set, you must set the parallelism to 1). Flink's Python API follows a similar approach to ship data from Flink to an external Python process.

Best, Fabian



2015-07-06 9:30 GMT+02:00 Bill Sparks <js...@cray.com>>:

Just a question if there was some prior-art here. Just say someone wanted to use flink for processing, but at some point they wanted to call another function via say JNI/C which doesn't understand DataSet's. How would one go about this ... I'm assuming the code would have to convert the data to a common format prior to calling the function.





Regards,

   Bill.


Re: data conversion between flink and "other" paradigms

Posted by Fabian Hueske <fh...@gmail.com>.
Hi Bill,

a DataSet is just a logical concept in Flink. DataSets are often not
persisted and just streamed along operators. At the moment, there is no way
to access an intermediate DataSet of a Flink program directly (this might
change in the future).

You can process data in another function by implementing a Java user
function (for example a MapPartition function) and sending the data through
JNI to a C function (if you need the full data set, you must set the
parallelism to 1). Flink's Python API follows a similar approach to ship
data from Flink to an external Python process.

Best, Fabian



2015-07-06 9:30 GMT+02:00 Bill Sparks <js...@cray.com>:

>  Just a question if there was some prior-art here. Just say someone
> wanted to use flink for processing, but at some point they wanted to call
> another function via say JNI/C which doesn't understand DataSet's. How
> would one go about this ... I'm assuming the code would have to convert the
> data to a common format prior to calling the function.
>
>
>
>
>
> Regards,
>
>    Bill.
>