[jira] [Commented] (BEAM-5488) DirectRunner not producing output on TextIO withWindowedWrites() and withNumShards(1)

["Swatilekha Kar (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/BEAM-5488?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17172891#comment-17172891 ] 

Swatilekha Kar commented on BEAM-5488:
--------------------------------------

I am facing the exact same issue. When I use DataflowRunner it's able to upload the files to GCS. However, if I switch to DirectRunner, it's not uploading the files to GCS. Please look into this issue.

> DirectRunner not producing output on TextIO withWindowedWrites() and withNumShards(1)
> -------------------------------------------------------------------------------------
>
>                 Key: BEAM-5488
>                 URL: https://issues.apache.org/jira/browse/BEAM-5488
>             Project: Beam
>          Issue Type: Bug
>          Components: runner-direct, sdk-java-core
>            Reporter: Luke Cwik
>            Priority: P3
>
> Source of bug (Slack user [https://the-asf.slack.com/team/UCVN8DK7V]) and [https://stackoverflow.com/questions/52445414/apache-beam-not-saving-unbounded-data-to-text-file].
> Example provided:
> {code:java}
> public static void main(String[] args) {
>     ExerciseOptions options = PipelineOptionsFactory.fromArgs(args).withValidation().as(ExerciseOptions.class);
>     Pipeline pipeline = Pipeline.create(options);
>     pipeline
>       .apply("Read Messages from Pubsub",
>         PubsubIO
>           .readStrings()
>           .fromTopic(options.getTopicName()))
>       .apply("Set event timestamp", ParDo.of(new DoFn<String, String>() {
>         @ProcessElement
>         public void processElement(ProcessContext context) {
>           context.outputWithTimestamp(context.element(), Instant.now());
>         }
>       }))
>       .apply("Windowing", Window.into(FixedWindows.of(Duration.standardMinutes(5))))
>       .apply("Write to File",
>         TextIO
>           .write()
>           .withWindowedWrites()
>           .withNumShards(1)
>           .to(options.getOutputPrefix()));
>     pipeline.run();
>   }
> {code}
> Produces output when executed on the DataflowRunner, does not produce output on the DirectRunner.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)