You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@beam.apache.org by Chengzhi Zhao <w....@gmail.com> on 2019/05/07 21:46:24 UTC
Question about unbounded in-memory PCollection
Hi Beam Team,
I am new to here and recently study the programming guide, I have a
question about the in-memory data,
https://beam.apache.org/documentation/programming-guide/#creating-a-pcollection
Is there a way to create unbounded PCollection from the in-memory
collection? I want to test the unbounded PCollection locally and don't know
what's the easiest way to get unbounded PCollection. Please let me know if
I am doing something wrong or I should use a file system to do it.
Thanks in advance!
Best,
Chengzhi
Re: Question about unbounded in-memory PCollection
Posted by Chengzhi Zhao <w....@gmail.com>.
Hi Rui,
Thanks for your reply. This is what I want!!
For TestStream.java
<https://github.com/apache/beam/blob/master/sdks/java/core/src/main/java/org/apache/beam/sdk/testing/TestStream.java>[1],
my understanding is I can attach an event time to my record and let
watermark advance to get unbounded PCollection.
Please let me know if that's wrong.
[1]
https://github.com/apache/beam/blob/master/sdks/java/core/src/main/java/org/apache/beam/sdk/testing/TestStream.java
Thank you!
Best Regards,
Chengzhi
On Tue, May 7, 2019 at 6:38 PM Rui Wang <ru...@google.com> wrote:
> Does TestStream.java
> <https://github.com/apache/beam/blob/master/sdks/java/core/src/main/java/org/apache/beam/sdk/testing/TestStream.java>[1]
> satisfy your need?
>
>
>
> -Rui
>
> [1]
> https://github.com/apache/beam/blob/master/sdks/java/core/src/main/java/org/apache/beam/sdk/testing/TestStream.java
>
> On Tue, May 7, 2019 at 2:47 PM Chengzhi Zhao <w....@gmail.com>
> wrote:
>
>> Hi Beam Team,
>>
>> I am new to here and recently study the programming guide, I have a
>> question about the in-memory data,
>> https://beam.apache.org/documentation/programming-guide/#creating-a-pcollection
>>
>> Is there a way to create unbounded PCollection from the in-memory
>> collection? I want to test the unbounded PCollection locally and don't know
>> what's the easiest way to get unbounded PCollection. Please let me know if
>> I am doing something wrong or I should use a file system to do it.
>>
>> Thanks in advance!
>>
>> Best,
>> Chengzhi
>>
>> On Tue, May 7, 2019 at 5:46 PM Chengzhi Zhao <w....@gmail.com>
>> wrote:
>>
>>> Hi Beam Team,
>>>
>>> I am new to here and recently study the programming guide, I have a
>>> question about the in-memory data,
>>> https://beam.apache.org/documentation/programming-guide/#creating-a-pcollection
>>>
>>> Is there a way to create unbounded PCollection from the in-memory
>>> collection? I want to test the unbounded PCollection locally and don't know
>>> what's the easiest way to get unbounded PCollection. Please let me know if
>>> I am doing something wrong or I should use a file system to do it.
>>>
>>> Thanks in advance!
>>>
>>> Best,
>>> Chengzhi
>>>
>>
Re: Question about unbounded in-memory PCollection
Posted by Rui Wang <ru...@google.com>.
Does TestStream.java
<https://github.com/apache/beam/blob/master/sdks/java/core/src/main/java/org/apache/beam/sdk/testing/TestStream.java>[1]
satisfy your need?
-Rui
[1]
https://github.com/apache/beam/blob/master/sdks/java/core/src/main/java/org/apache/beam/sdk/testing/TestStream.java
On Tue, May 7, 2019 at 2:47 PM Chengzhi Zhao <w....@gmail.com>
wrote:
> Hi Beam Team,
>
> I am new to here and recently study the programming guide, I have a
> question about the in-memory data,
> https://beam.apache.org/documentation/programming-guide/#creating-a-pcollection
>
> Is there a way to create unbounded PCollection from the in-memory
> collection? I want to test the unbounded PCollection locally and don't know
> what's the easiest way to get unbounded PCollection. Please let me know if
> I am doing something wrong or I should use a file system to do it.
>
> Thanks in advance!
>
> Best,
> Chengzhi
>
> On Tue, May 7, 2019 at 5:46 PM Chengzhi Zhao <w....@gmail.com>
> wrote:
>
>> Hi Beam Team,
>>
>> I am new to here and recently study the programming guide, I have a
>> question about the in-memory data,
>> https://beam.apache.org/documentation/programming-guide/#creating-a-pcollection
>>
>> Is there a way to create unbounded PCollection from the in-memory
>> collection? I want to test the unbounded PCollection locally and don't know
>> what's the easiest way to get unbounded PCollection. Please let me know if
>> I am doing something wrong or I should use a file system to do it.
>>
>> Thanks in advance!
>>
>> Best,
>> Chengzhi
>>
>
Re: Question about unbounded in-memory PCollection
Posted by Chengzhi Zhao <w....@gmail.com>.
Hi Beam Team,
I am new to here and recently study the programming guide, I have a
question about the in-memory data,
https://beam.apache.org/documentation/programming-guide/#creating-a-pcollection
Is there a way to create unbounded PCollection from the in-memory
collection? I want to test the unbounded PCollection locally and don't know
what's the easiest way to get unbounded PCollection. Please let me know if
I am doing something wrong or I should use a file system to do it.
Thanks in advance!
Best,
Chengzhi
On Tue, May 7, 2019 at 5:46 PM Chengzhi Zhao <w....@gmail.com>
wrote:
> Hi Beam Team,
>
> I am new to here and recently study the programming guide, I have a
> question about the in-memory data,
> https://beam.apache.org/documentation/programming-guide/#creating-a-pcollection
>
> Is there a way to create unbounded PCollection from the in-memory
> collection? I want to test the unbounded PCollection locally and don't know
> what's the easiest way to get unbounded PCollection. Please let me know if
> I am doing something wrong or I should use a file system to do it.
>
> Thanks in advance!
>
> Best,
> Chengzhi
>