You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@beam.apache.org by shankara ODL <sh...@gmail.com> on 2018/02/26 09:13:46 UTC

apache beam in spring cloud dataflow

Hi All,

     I am new to apache beam and spring cloud dataflow. I am trying to
integrate
apache beam in spring cloud dataflow. How to get spring-kafka message as a
source in beam pipeline ?. How to add spring-kafka as a sink in beam
pipeline ? Wanted to run pipeline forever untilfinish. Please suggest how
can I integrate ?

example wordcount PipelineOptions options = PipelineOptionsFactory.create();


Pipeline p = Pipeline.create(options);

p.apply(TextIO.read().from("gs://apache-beam-samples/shakespeare/*")) ---->
instead of TextIO.read().from want to trigger from message channel INPUT in
spring cloud dataflow
 .apply(FlatMapElements
     .into(TypeDescriptors.strings())
     .via((String word) -> Arrays.asList(word.split("[^\\p{L}]+"))))
 .apply(Filter.by((String word) -> !word.isEmpty()))
 .apply(Count.<String>perElement())
 .apply(MapElements
     .into(TypeDescriptors.strings())
     .via((KV<String, Long> wordCount) -> wordCount.getKey() + ": " +
wordCount.getValue()))
.apply(TextIO.write().to("gs://YOUR_OUTPUT_BUCKET/AND_OUTPUT_PREFIX"));
----> send the result to message channel OUTPUT

p.run().waitUntilFinish();

Thanks,
Shankara

Re: apache beam in spring cloud dataflow

Posted by Eugene Kirpichov <ki...@google.com>.
I am not familiar with "Spring Cloud Dataflow", but you can use Kafka from
Beam with KafkaIO
https://beam.apache.org/documentation/sdks/javadoc/2.3.0/org/apache/beam/sdk/io/kafka/KafkaIO.html
 .

On Mon, Feb 26, 2018 at 10:10 AM Lukasz Cwik <lc...@google.com> wrote:

>
> ---------- Forwarded message ----------
> From: shankara ODL <sh...@gmail.com>
> Date: Mon, Feb 26, 2018 at 1:13 AM
> Subject: apache beam in spring cloud dataflow
> To: dev@beam.apache.org
>
>
> Hi All,
>
>      I am new to apache beam and spring cloud dataflow. I am trying to
> integrate
> apache beam in spring cloud dataflow. How to get spring-kafka message as a
>
> source in beam pipeline ?. How to add spring-kafka as a sink in beam
> pipeline ? Wanted to run pipeline forever untilfinish. Please suggest how
> can I integrate ?
>
> example wordcount PipelineOptions options =
> PipelineOptionsFactory.create();
>
> Pipeline p = Pipeline.create(options);
>
> p.apply(TextIO.read().from("gs://apache-beam-samples/shakespeare/*")) ---->
>
> instead of TextIO.read().from want to trigger from message channel INPUT in
>
> spring cloud dataflow
>  .apply(FlatMapElements
>      .into(TypeDescriptors.strings())
>      .via((String word) -> Arrays.asList(word.split("[^\\p{L}]+"))))
>  .apply(Filter.by((String word) -> !word.isEmpty()))
>  .apply(Count.<String>perElement())
>  .apply(MapElements
>      .into(TypeDescriptors.strings())
>      .via((KV<String, Long> wordCount) -> wordCount.getKey() + ": " +
> wordCount.getValue()))
> .apply(TextIO.write().to("gs://YOUR_OUTPUT_BUCKET/AND_OUTPUT_PREFIX"));
> ----> send the result to message channel OUTPUT
>
> p.run().waitUntilFinish();
>
> Thanks,
> Shankara
>
>

Fwd: apache beam in spring cloud dataflow

Posted by Lukasz Cwik <lc...@google.com>.
---------- Forwarded message ----------
From: shankara ODL <sh...@gmail.com>
Date: Mon, Feb 26, 2018 at 1:13 AM
Subject: apache beam in spring cloud dataflow
To: dev@beam.apache.org


Hi All,

     I am new to apache beam and spring cloud dataflow. I am trying to
integrate
apache beam in spring cloud dataflow. How to get spring-kafka message as a
source in beam pipeline ?. How to add spring-kafka as a sink in beam
pipeline ? Wanted to run pipeline forever untilfinish. Please suggest how
can I integrate ?

example wordcount PipelineOptions options = PipelineOptionsFactory.create();


Pipeline p = Pipeline.create(options);

p.apply(TextIO.read().from("gs://apache-beam-samples/shakespeare/*")) ---->
instead of TextIO.read().from want to trigger from message channel INPUT in
spring cloud dataflow
 .apply(FlatMapElements
     .into(TypeDescriptors.strings())
     .via((String word) -> Arrays.asList(word.split("[^\\p{L}]+"))))
 .apply(Filter.by((String word) -> !word.isEmpty()))
 .apply(Count.<String>perElement())
 .apply(MapElements
     .into(TypeDescriptors.strings())
     .via((KV<String, Long> wordCount) -> wordCount.getKey() + ": " +
wordCount.getValue()))
.apply(TextIO.write().to("gs://YOUR_OUTPUT_BUCKET/AND_OUTPUT_PREFIX"));
----> send the result to message channel OUTPUT

p.run().waitUntilFinish();

Thanks,
Shankara