You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@beam.apache.org by shankara ODL <sh...@gmail.com> on 2018/02/26 09:13:46 UTC
apache beam in spring cloud dataflow
Hi All,
I am new to apache beam and spring cloud dataflow. I am trying to
integrate
apache beam in spring cloud dataflow. How to get spring-kafka message as a
source in beam pipeline ?. How to add spring-kafka as a sink in beam
pipeline ? Wanted to run pipeline forever untilfinish. Please suggest how
can I integrate ?
example wordcount PipelineOptions options = PipelineOptionsFactory.create();
Pipeline p = Pipeline.create(options);
p.apply(TextIO.read().from("gs://apache-beam-samples/shakespeare/*")) ---->
instead of TextIO.read().from want to trigger from message channel INPUT in
spring cloud dataflow
.apply(FlatMapElements
.into(TypeDescriptors.strings())
.via((String word) -> Arrays.asList(word.split("[^\\p{L}]+"))))
.apply(Filter.by((String word) -> !word.isEmpty()))
.apply(Count.<String>perElement())
.apply(MapElements
.into(TypeDescriptors.strings())
.via((KV<String, Long> wordCount) -> wordCount.getKey() + ": " +
wordCount.getValue()))
.apply(TextIO.write().to("gs://YOUR_OUTPUT_BUCKET/AND_OUTPUT_PREFIX"));
----> send the result to message channel OUTPUT
p.run().waitUntilFinish();
Thanks,
Shankara
Re: apache beam in spring cloud dataflow
Posted by Eugene Kirpichov <ki...@google.com>.
I am not familiar with "Spring Cloud Dataflow", but you can use Kafka from
Beam with KafkaIO
https://beam.apache.org/documentation/sdks/javadoc/2.3.0/org/apache/beam/sdk/io/kafka/KafkaIO.html
.
On Mon, Feb 26, 2018 at 10:10 AM Lukasz Cwik <lc...@google.com> wrote:
>
> ---------- Forwarded message ----------
> From: shankara ODL <sh...@gmail.com>
> Date: Mon, Feb 26, 2018 at 1:13 AM
> Subject: apache beam in spring cloud dataflow
> To: dev@beam.apache.org
>
>
> Hi All,
>
> I am new to apache beam and spring cloud dataflow. I am trying to
> integrate
> apache beam in spring cloud dataflow. How to get spring-kafka message as a
>
> source in beam pipeline ?. How to add spring-kafka as a sink in beam
> pipeline ? Wanted to run pipeline forever untilfinish. Please suggest how
> can I integrate ?
>
> example wordcount PipelineOptions options =
> PipelineOptionsFactory.create();
>
> Pipeline p = Pipeline.create(options);
>
> p.apply(TextIO.read().from("gs://apache-beam-samples/shakespeare/*")) ---->
>
> instead of TextIO.read().from want to trigger from message channel INPUT in
>
> spring cloud dataflow
> .apply(FlatMapElements
> .into(TypeDescriptors.strings())
> .via((String word) -> Arrays.asList(word.split("[^\\p{L}]+"))))
> .apply(Filter.by((String word) -> !word.isEmpty()))
> .apply(Count.<String>perElement())
> .apply(MapElements
> .into(TypeDescriptors.strings())
> .via((KV<String, Long> wordCount) -> wordCount.getKey() + ": " +
> wordCount.getValue()))
> .apply(TextIO.write().to("gs://YOUR_OUTPUT_BUCKET/AND_OUTPUT_PREFIX"));
> ----> send the result to message channel OUTPUT
>
> p.run().waitUntilFinish();
>
> Thanks,
> Shankara
>
>
Fwd: apache beam in spring cloud dataflow
Posted by Lukasz Cwik <lc...@google.com>.
---------- Forwarded message ----------
From: shankara ODL <sh...@gmail.com>
Date: Mon, Feb 26, 2018 at 1:13 AM
Subject: apache beam in spring cloud dataflow
To: dev@beam.apache.org
Hi All,
I am new to apache beam and spring cloud dataflow. I am trying to
integrate
apache beam in spring cloud dataflow. How to get spring-kafka message as a
source in beam pipeline ?. How to add spring-kafka as a sink in beam
pipeline ? Wanted to run pipeline forever untilfinish. Please suggest how
can I integrate ?
example wordcount PipelineOptions options = PipelineOptionsFactory.create();
Pipeline p = Pipeline.create(options);
p.apply(TextIO.read().from("gs://apache-beam-samples/shakespeare/*")) ---->
instead of TextIO.read().from want to trigger from message channel INPUT in
spring cloud dataflow
.apply(FlatMapElements
.into(TypeDescriptors.strings())
.via((String word) -> Arrays.asList(word.split("[^\\p{L}]+"))))
.apply(Filter.by((String word) -> !word.isEmpty()))
.apply(Count.<String>perElement())
.apply(MapElements
.into(TypeDescriptors.strings())
.via((KV<String, Long> wordCount) -> wordCount.getKey() + ": " +
wordCount.getValue()))
.apply(TextIO.write().to("gs://YOUR_OUTPUT_BUCKET/AND_OUTPUT_PREFIX"));
----> send the result to message channel OUTPUT
p.run().waitUntilFinish();
Thanks,
Shankara