You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Bill Sparks <js...@cray.com> on 2015/07/06 09:30:48 UTC
data conversion between flink and "other" paradigms
Just a question if there was some prior-art here. Just say someone wanted to use flink for processing, but at some point they wanted to call another function via say JNI/C which doesn't understand DataSet's. How would one go about this ... I'm assuming the code would have to convert the data to a common format prior to calling the function.
Regards,
Bill.
RE: data conversion between flink and "other" paradigms
Posted by Bill Sparks <js...@cray.com>.
Fabian.
Thanks for the info and pointer to python. I'll check it out.
-Bill
________________________________
From: Fabian Hueske [fhueske@gmail.com]
Sent: Monday, July 06, 2015 3:23 AM
To: user@flink.apache.org
Subject: Re: data conversion between flink and "other" paradigms
Hi Bill,
a DataSet is just a logical concept in Flink. DataSets are often not persisted and just streamed along operators. At the moment, there is no way to access an intermediate DataSet of a Flink program directly (this might change in the future).
You can process data in another function by implementing a Java user function (for example a MapPartition function) and sending the data through JNI to a C function (if you need the full data set, you must set the parallelism to 1). Flink's Python API follows a similar approach to ship data from Flink to an external Python process.
Best, Fabian
2015-07-06 9:30 GMT+02:00 Bill Sparks <js...@cray.com>>:
Just a question if there was some prior-art here. Just say someone wanted to use flink for processing, but at some point they wanted to call another function via say JNI/C which doesn't understand DataSet's. How would one go about this ... I'm assuming the code would have to convert the data to a common format prior to calling the function.
Regards,
Bill.
Re: data conversion between flink and "other" paradigms
Posted by Fabian Hueske <fh...@gmail.com>.
Hi Bill,
a DataSet is just a logical concept in Flink. DataSets are often not
persisted and just streamed along operators. At the moment, there is no way
to access an intermediate DataSet of a Flink program directly (this might
change in the future).
You can process data in another function by implementing a Java user
function (for example a MapPartition function) and sending the data through
JNI to a C function (if you need the full data set, you must set the
parallelism to 1). Flink's Python API follows a similar approach to ship
data from Flink to an external Python process.
Best, Fabian
2015-07-06 9:30 GMT+02:00 Bill Sparks <js...@cray.com>:
> Just a question if there was some prior-art here. Just say someone
> wanted to use flink for processing, but at some point they wanted to call
> another function via say JNI/C which doesn't understand DataSet's. How
> would one go about this ... I'm assuming the code would have to convert the
> data to a common format prior to calling the function.
>
>
>
>
>
> Regards,
>
> Bill.
>