Looking for Support with spark runner, Error : java.lang.IllegalArgumentException: requirement failed: No output operations registered, so nothing to execute

["Truebody, Kyle" <Tr...@DNB.com>]

Hi all,

We have been creating a few POC's. I have build an application based off of the beam example WordCount and have modified it to read from a kinesis stream. We are limited to using the SparkRunner at this stage. Generally this works as expected when reading from stream horizon ( Output is produced etc...). But when running the pipeline via spark-submit in streaming mode, if the stream has no more data available  exception  `java.lang.IllegalArgumentException: requirement failed: No output operations registered, so nothing to execute`  is thrown and pre-maturely shutting down the job. Looking into this further I noted that  DStreamingGraph expects an outputStream to be registered. Running native spark streaming app does not behave like this.  I would expect the streaming app to be running constantly.

Is there something I have missed? Help would be greatly appreciated. Beam version 2.16. Spark version 2.4.0

Error:
2019-10-24 11:52:08 ERROR StreamingContext:91 - Error starting the context, marking it as stopped
java.lang.IllegalArgumentException: requirement failed: No output operations registered, so nothing to execute
        at scala.Predef$.require(Predef.scala:224)
        at org.apache.spark.streaming.DStreamGraph.validate(DStreamGraph.scala:168)
        at org.apache.spark.streaming.StreamingContext.validate(StreamingContext.scala:513)
        at org.apache.spark.streaming.StreamingContext.liftedTree1$1(StreamingContext.scala:573)
        at org.apache.spark.streaming.StreamingContext.start(StreamingContext.scala:572)
        at org.apache.spark.streaming.api.java.JavaStreamingContext.start(JavaStreamingContext.scala:556)
        at org.apache.beam.runners.spark.SparkRunner.lambda$run$0(SparkRunner.java:206)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)


I know this is / can be caused by  a missing output operator so the DStream cannot do any computations. So at least one needs to be invoked.

IE:

print()
foreachRDD(func)
saveAsObjectFiles(prefix, [suffix])
saveAsTextFiles(prefix, [suffix])
saveAsHadoopFiles(prefix, [suffix])


I would imagine that  beam should handle this when some sort of write IO is used.

Same of the written pipeline:

static void readWordCountFromStream(RunningOptions options) {
    Pipeline p = Pipeline.create(options);
    Properties properties = new Properties();

    KinesisIO.Read kinesisReader = KinesisCustomConfigurator.initReader(options, properties);

    TextIO.Write textWriter = TextIO.write()
            .withWindowedWrites()
            .withNumShards(1)
            .to(options.getOutputPath());

    PCollection<String> lines = p.apply("ReadLines", kinesisReader)
            .apply(ParDo.of(new KinesisRecordToStringDetailed()))
            .apply("FixedWindows",
                    Window.<String>into(FixedWindows.of(Utilities.resolveDuration(options.getWindowDuration()))));


    lines.apply("CountWords", new CountWords())
            .apply(MapElements.via(new FormatAsTextFn()))
            .apply("WriteCounts", textWriter);

    PipelineResult result = p.run();
    result.waitUntilFinish();
  }

Thanks in advance,

Cheers,

Kyle Truebody
Software Engineer
The Chase Building, 5th Floor
Carmanhall Road, Sandyford
Dublin 18, Ireland
D18 Y3X2

http://www.dnb.co.uk/
www.workatdnb.ie<http://www.workatdnb.ie/>

[cid:image002.png@01D05044.5C2AEE60]<http://www.facebook.com/DunBradstreet>[cid:image003.png@01D05044.5C2AEE60]<http://twitter.com/dnbus>[cid:image004.png@01D05044.5C2AEE60]<http://www.linkedin.com/company/dun-&-bradstreet>[cid:image005.png@01D05044.5C2AEE60]<http://www.youtube.com/user/DunandBrad>
[cid:image007.jpg@01D41455.76918150]<https://dnb.wd1.myworkdayjobs.com/Careers/4/refreshFacet/318c8bb6f553100021d223d9780d30be>[cid:image008.png@01D41455.76918150]