You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@beam.apache.org by Pradyumna Achar <pr...@gmail.com> on 2021/02/19 06:25:31 UTC

Avoiding the extra newline with TextIO

Hi everyone,

TextIO adds an extra newline to every element. Is there a way to avoid it?
If I write a 10-byte file, I get 11 bytes in the destination (with that
extra newline)

https://github.com/apache/beam/blob/f09f2afa25299cf0af67476e10f0deb330b7b810/sdks/java/core/src/main/java/org/apache/beam/sdk/io/TextIO.java#L1300

Thank you
Regards
Pradyumna

Re: Avoiding the extra newline with TextIO

Posted by Alexey Romanenko <ar...@gmail.com>.

Well, I don’t see that it’s a very common case (at least, I don’t recall the other requests for this before) and FileIO + custom Sink should perfectly help with this. 

I’m not sure that we need to add such functionality for TextIO.write() since it assumes that it will write data line by line separated by common line’s delimiter.

> On 20 Feb 2021, at 05:13, Pradyumna Achar <pr...@gmail.com> wrote:
> 
> Thank you, yes this is what I had done yesterday as a workaround, but wanted to know if there was a standard way to do it as it would probably be a common use case.
> Regards
> Pradyumna

Re: Avoiding the extra newline with TextIO

Posted by Pradyumna Achar <pr...@gmail.com>.

Thank you, yes this is what I had done yesterday as a workaround, but
wanted to know if there was a standard way to do it as it would probably be
a common use case.
Regards
Pradyumna

Re: Avoiding the extra newline with TextIO

Posted by Alexey Romanenko <ar...@gmail.com>.

I don’t think it’s possible with TextIO but you can create your own FileIO.Sink<String> and use it together with FileIO

Something like:

p.apply(FileIO.<String>write().to(“/path/to/output").via(new MySink()));
public static class MySink implements FileIO.Sink<String> {

  private transient @Nullable PrintWriter writer;

  @Override
  public void open(WritableByteChannel channel) throws IOException {
    writer = new PrintWriter(
        new BufferedWriter(new OutputStreamWriter(Channels.newOutputStream(channel), UTF_8)));
  }

  @Override
  public void write(String element) throws IOException {
    writer.print(element);
  }

  @Override
  public void flush() throws IOException {
    writer.flush();
  }
}


> On 19 Feb 2021, at 07:25, Pradyumna Achar <pr...@gmail.com> wrote:
> 
> Hi everyone,
> 
> TextIO adds an extra newline to every element. Is there a way to avoid it? 
> If I write a 10-byte file, I get 11 bytes in the destination (with that extra newline)
> 
> https://github.com/apache/beam/blob/f09f2afa25299cf0af67476e10f0deb330b7b810/sdks/java/core/src/main/java/org/apache/beam/sdk/io/TextIO.java#L1300 <https://github.com/apache/beam/blob/f09f2afa25299cf0af67476e10f0deb330b7b810/sdks/java/core/src/main/java/org/apache/beam/sdk/io/TextIO.java#L1300>
> 
> Thank you
> Regards
> Pradyumna