You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by KirstiLaurila <ki...@rovio.com> on 2016/05/23 14:56:54 UTC
Re: writeAsCSV with partitionBy
Is there any plans to implement this kind of feature (possibility to write to
data specified partitions) in the near future?
--
View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/writeAsCSV-with-partitionBy-tp4893p7099.html
Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.
Re: writeAsCSV with partitionBy
Posted by Aljoscha Krettek <al...@apache.org>.
Hi,
the RollingSink can only be used with streaming. Adding support for dynamic
paths based on element contents is certainly interesting. I imagine it can
be tricky, though, to figure out when to close/flush the buckets.
Cheers,
Aljoscha
On Wed, 25 May 2016 at 08:36 KirstiLaurila <ki...@rovio.com> wrote:
> Maybe, I don't know, but with streaming. How about batch?
>
>
> Srikanth wrote
> > Isn't this related to --
> https://issues.apache.org/jira/browse/FLINK-2672
> > ??
> >
> > This can be achieved with a RollingSink[1] & custom Bucketer probably.
> >
> > [1]
> >
> https://ci.apache.org/projects/flink/flink-docs-master/api/java/org/apache/flink/streaming/connectors/fs/RollingSink.html
>
>
>
>
>
> --
> View this message in context:
> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/writeAsCSV-with-partitionBy-tp4893p7140.html
> Sent from the Apache Flink User Mailing List archive. mailing list archive
> at Nabble.com.
>
Re: writeAsCSV with partitionBy
Posted by KirstiLaurila <ki...@rovio.com>.
Maybe, I don't know, but with streaming. How about batch?
Srikanth wrote
> Isn't this related to -- https://issues.apache.org/jira/browse/FLINK-2672
> ??
>
> This can be achieved with a RollingSink[1] & custom Bucketer probably.
>
> [1]
> https://ci.apache.org/projects/flink/flink-docs-master/api/java/org/apache/flink/streaming/connectors/fs/RollingSink.html
--
View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/writeAsCSV-with-partitionBy-tp4893p7140.html
Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.
Re: writeAsCSV with partitionBy
Posted by Juho Autio <ju...@rovio.com>.
RollingSink is part of Flink Streaming API. Can it be used in Flink Batch
jobs, too?
As implied in FLINK-2672, RollingSink doesn't support dynamic bucket paths
based on the tuple fields. The path must be given when creating the
RollingSink instance, ie. before deploying the job. Yes, a custom Bucketer
can be provided, but as the current method signature is, tuple is not
passed to Bucketer.
On Tue, May 24, 2016 at 4:45 PM, Srikanth <sr...@gmail.com> wrote:
> Isn't this related to -- https://issues.apache.org/jira/browse/FLINK-2672
> ??
>
> This can be achieved with a RollingSink[1] & custom Bucketer probably.
>
> [1]
> https://ci.apache.org/projects/flink/flink-docs-master/api/java/org/apache/flink/streaming/connectors/fs/RollingSink.html
>
> Srikanth
>
> On Tue, May 24, 2016 at 1:07 AM, KirstiLaurila <ki...@rovio.com>
> wrote:
>
>> Yeah, created this one https://issues.apache.org/jira/browse/FLINK-3961
>> <https://issues.apache.org/jira/browse/FLINK-3961>
>>
>>
>>
>>
>> --
>> View this message in context:
>> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/writeAsCSV-with-partitionBy-tp4893p7118.html
>> Sent from the Apache Flink User Mailing List archive. mailing list
>> archive at Nabble.com.
>>
>
Re: writeAsCSV with partitionBy
Posted by Srikanth <sr...@gmail.com>.
Isn't this related to -- https://issues.apache.org/jira/browse/FLINK-2672 ??
This can be achieved with a RollingSink[1] & custom Bucketer probably.
[1]
https://ci.apache.org/projects/flink/flink-docs-master/api/java/org/apache/flink/streaming/connectors/fs/RollingSink.html
Srikanth
On Tue, May 24, 2016 at 1:07 AM, KirstiLaurila <ki...@rovio.com>
wrote:
> Yeah, created this one https://issues.apache.org/jira/browse/FLINK-3961
> <https://issues.apache.org/jira/browse/FLINK-3961>
>
>
>
>
> --
> View this message in context:
> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/writeAsCSV-with-partitionBy-tp4893p7118.html
> Sent from the Apache Flink User Mailing List archive. mailing list archive
> at Nabble.com.
>
Re: writeAsCSV with partitionBy
Posted by KirstiLaurila <ki...@rovio.com>.
Yeah, created this one https://issues.apache.org/jira/browse/FLINK-3961
<https://issues.apache.org/jira/browse/FLINK-3961>
--
View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/writeAsCSV-with-partitionBy-tp4893p7118.html
Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.
Re: writeAsCSV with partitionBy
Posted by Fabian Hueske <fh...@gmail.com>.
Hi Kirsti,
I'm not aware of anybody working on this issue.
Would you like to create a JIRA issue for it?
Best, Fabian
2016-05-23 16:56 GMT+02:00 KirstiLaurila <ki...@rovio.com>:
> Is there any plans to implement this kind of feature (possibility to write
> to
> data specified partitions) in the near future?
>
>
>
> --
> View this message in context:
> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/writeAsCSV-with-partitionBy-tp4893p7099.html
> Sent from the Apache Flink User Mailing List archive. mailing list archive
> at Nabble.com.
>