Problem with autoscaling

[Vilhelm von Ehrenheim <vo...@gmail.com>]

Hi!
I cannot make my streaming job autoscale on Dataflow. To find an answer to
why I was looking around in the Dataflow docs and found this:

> Currently, PubSubIO is the only source that supports autoscaling on
streaming pipelines.

on the following page:
https://cloud.google.com/dataflow/service/dataflow-service-desc#autoscaling

Does anyone know if this is true? I know this is not the forum for Dataflow
questions in general but I though someone else here might have experience
that support or contradict this.

Thanks,
Vilhelm von Ehrenheim

Re: Problem with autoscaling

[Raghu Angadi <ra...@google.com>]

KafkaIO also supports autoscaling on Dataflow. In fact any unbounded source
is supported as long as it implements 'getSplitBacklogBytes()' or
'getTotalBacklogBytes()' interface. E.g. PR 3551
<https://github.com/apache/beam/pull/3551> adds backlog support for
KinesisIO.

On Tue, Sep 5, 2017 at 10:19 PM, Lukasz Cwik <lc...@google.com> wrote:

> That is correct, autoscaling for streaming is only supported in Pubsub.
> What sources were you interested in?
>
> On Mon, Sep 4, 2017 at 12:54 AM, Derek Hao Hu <ph...@gmail.com>
> wrote:
>
>> I've used PubSubIO for autoscaling on a streaming pipeline and it seems
>> to be working fine so far.
>>
>> I don't know if it is the **only** source that supports autoscaling on
>> streaming pipelines though.
>>
>> On Mon, Sep 4, 2017 at 12:52 AM, Vilhelm von Ehrenheim <
>> vonehrenheim@gmail.com> wrote:
>>
>>> Hi!
>>> I cannot make my streaming job autoscale on Dataflow. To find an answer
>>> to why I was looking around in the Dataflow docs and found this:
>>>
>>> > Currently, PubSubIO is the only source that supports autoscaling on
>>> streaming pipelines.
>>>
>>> on the following page: https://cloud.google.com
>>> /dataflow/service/dataflow-service-desc#autoscaling
>>>
>>> Does anyone know if this is true? I know this is not the forum for
>>> Dataflow questions in general but I though someone else here might have
>>> experience that support or contradict this.
>>>
>>> Thanks,
>>> Vilhelm von Ehrenheim
>>>
>>
>>
>>
>> --
>> Derek Hao Hu
>>
>> Software Engineer | Snapchat
>> Snap Inc.
>>
>
>

Re: Problem with autoscaling

[Lukasz Cwik <lc...@google.com>]

That is correct, autoscaling for streaming is only supported in Pubsub.
What sources were you interested in?

On Mon, Sep 4, 2017 at 12:54 AM, Derek Hao Hu <ph...@gmail.com>
wrote:

> I've used PubSubIO for autoscaling on a streaming pipeline and it seems to
> be working fine so far.
>
> I don't know if it is the **only** source that supports autoscaling on
> streaming pipelines though.
>
> On Mon, Sep 4, 2017 at 12:52 AM, Vilhelm von Ehrenheim <
> vonehrenheim@gmail.com> wrote:
>
>> Hi!
>> I cannot make my streaming job autoscale on Dataflow. To find an answer
>> to why I was looking around in the Dataflow docs and found this:
>>
>> > Currently, PubSubIO is the only source that supports autoscaling on
>> streaming pipelines.
>>
>> on the following page: https://cloud.google.com
>> /dataflow/service/dataflow-service-desc#autoscaling
>>
>> Does anyone know if this is true? I know this is not the forum for
>> Dataflow questions in general but I though someone else here might have
>> experience that support or contradict this.
>>
>> Thanks,
>> Vilhelm von Ehrenheim
>>
>
>
>
> --
> Derek Hao Hu
>
> Software Engineer | Snapchat
> Snap Inc.
>

Re: Problem with autoscaling

[Derek Hao Hu <ph...@gmail.com>]

I've used PubSubIO for autoscaling on a streaming pipeline and it seems to
be working fine so far.

I don't know if it is the **only** source that supports autoscaling on
streaming pipelines though.

On Mon, Sep 4, 2017 at 12:52 AM, Vilhelm von Ehrenheim <
vonehrenheim@gmail.com> wrote:

> Hi!
> I cannot make my streaming job autoscale on Dataflow. To find an answer to
> why I was looking around in the Dataflow docs and found this:
>
> > Currently, PubSubIO is the only source that supports autoscaling on
> streaming pipelines.
>
> on the following page: https://cloud.google.com/dataflow/service/dataflow-
> service-desc#autoscaling
>
> Does anyone know if this is true? I know this is not the forum for
> Dataflow questions in general but I though someone else here might have
> experience that support or contradict this.
>
> Thanks,
> Vilhelm von Ehrenheim
>



-- 
Derek Hao Hu

Software Engineer | Snapchat
Snap Inc.