[Java] TextIO not reading file as expected

[Shannon Duncan <jo...@liveramp.com>]

I have a file where every line is a record separated by a tab. So a tab
delimited file.

However as I read this file in using TextIO.read().from(filename) and pass
the results to a pardo, the elements are random chunks of the records. I
expected the element to be the entire line of text which then I'll do
parsing on from there.

This file is processed in a python pipeline with ReadFromText perfectly
fine. Just curious what would cause this on the Java side?

Thanks,
Shannon

Re: [Java] TextIO not reading file as expected

[Kenneth Knowles <ke...@apache.org>]

Glad to hear it :-)

On Fri, Jul 12, 2019 at 6:33 AM Shannon Duncan <jo...@liveramp.com>
wrote:

> So as it turns out, it was an STDOUT issue for my logging and not a data
> read in. Beam operated just fine but the way I was debugging was causing
> the glitches.
>
> Beam is operating as expected now.
>
> On Thu, Jul 11, 2019 at 10:28 PM Kenneth Knowles <ke...@apache.org> wrote:
>
>> Doesn't sound good. TextIO has been around a long time so I'm surprised.
>> Would you mind creating a ticket in Jira (
>> https://issues.apache.org/jira/projects/BEAM/) and posting some
>> technical details, like input/output/code snippets?
>>
>> Kenn
>>
>> On Thu, Jul 11, 2019 at 9:45 AM Shannon Duncan <
>> joseph.duncan@liveramp.com> wrote:
>>
>>> I have a file where every line is a record separated by a tab. So a tab
>>> delimited file.
>>>
>>> However as I read this file in using TextIO.read().from(filename) and
>>> pass the results to a pardo, the elements are random chunks of the records.
>>> I expected the element to be the entire line of text which then I'll do
>>> parsing on from there.
>>>
>>> This file is processed in a python pipeline with ReadFromText perfectly
>>> fine. Just curious what would cause this on the Java side?
>>>
>>> Thanks,
>>> Shannon
>>>
>>

Re: [Java] TextIO not reading file as expected

[Shannon Duncan <jo...@liveramp.com>]

So as it turns out, it was an STDOUT issue for my logging and not a data
read in. Beam operated just fine but the way I was debugging was causing
the glitches.

Beam is operating as expected now.

On Thu, Jul 11, 2019 at 10:28 PM Kenneth Knowles <ke...@apache.org> wrote:

> Doesn't sound good. TextIO has been around a long time so I'm surprised.
> Would you mind creating a ticket in Jira (
> https://issues.apache.org/jira/projects/BEAM/) and posting some technical
> details, like input/output/code snippets?
>
> Kenn
>
> On Thu, Jul 11, 2019 at 9:45 AM Shannon Duncan <jo...@liveramp.com>
> wrote:
>
>> I have a file where every line is a record separated by a tab. So a tab
>> delimited file.
>>
>> However as I read this file in using TextIO.read().from(filename) and
>> pass the results to a pardo, the elements are random chunks of the records.
>> I expected the element to be the entire line of text which then I'll do
>> parsing on from there.
>>
>> This file is processed in a python pipeline with ReadFromText perfectly
>> fine. Just curious what would cause this on the Java side?
>>
>> Thanks,
>> Shannon
>>
>

Re: [Java] TextIO not reading file as expected

[Kenneth Knowles <ke...@apache.org>]

Doesn't sound good. TextIO has been around a long time so I'm surprised.
Would you mind creating a ticket in Jira (
https://issues.apache.org/jira/projects/BEAM/) and posting some technical
details, like input/output/code snippets?

Kenn

On Thu, Jul 11, 2019 at 9:45 AM Shannon Duncan <jo...@liveramp.com>
wrote:

> I have a file where every line is a record separated by a tab. So a tab
> delimited file.
>
> However as I read this file in using TextIO.read().from(filename) and pass
> the results to a pardo, the elements are random chunks of the records. I
> expected the element to be the entire line of text which then I'll do
> parsing on from there.
>
> This file is processed in a python pipeline with ReadFromText perfectly
> fine. Just curious what would cause this on the Java side?
>
> Thanks,
> Shannon
>