You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by "Sofer, Tovi " <to...@citi.com> on 2017/12/05 15:25:03 UTC

slot group indication per operator

Hi all,
I am trying to use the slot group feature, by having 'default' group and additional 'market' group.
The purpose is to divide the resources equally between two sources and their following operators.
I've set the slotGroup on the source of the market data.
Can I assume that all following operators created from this source will use same slot group of 'market'?
(The operators created for market stream are pretty complex, with connect and split).
In Web UI I saw there are 16 slots, but didn't see indication per operator to which group it was assigned. How can I know?

Relevant Code:

env.setParallelism(8);
conf.setInteger(ConfigConstants.TASK_MANAGER_NUM_TASK_SLOTS, 16); \\ to allow Parallelism of 8 per group

// Market source and operators:

KeyedStream<SpotTickEvent, Tuple> windowedStreamA = sourceProvider.provide(env)
        .name(spotSourceProvider.getName())
        .slotSharingGroup(SourceMsgType.MARKET.slotGroup())
        .flatMap(new ParserMapper(new MarketMessageParser()))
        .name(ParserMapper.class.getSimpleName())
        .filter(new USDFilter())
        .name(USDFilter.class.getSimpleName())
        .keyBy(MarketEvent.CURRENCY_FIELD)
        .timeWindow(Time.of(windowSizeMs, TimeUnit.MILLISECONDS))
        .process(new LastInWindowPriceChangeFunction()))
        .name(LastInWindowPriceChangeFunction.class.getSimpleName())
        .keyBy(SpotTickEvent.CURRENCY_FIELD);


marketConnectedStream = windowedStreamA.connect(windowedStreamB)
            .flatMap(new MarketCoMapper()))
            .name(MarketCoMapper.class.getSimpleName())



SplitStream<MarketAWithMarketB> stocksWithSpotsStreams = marketConnectedStream
        .split( market -> ImmutableList.of("splitA"," splitB") );

DataStream< MarketAWithMarketB> splitA = stocksWithSpotsStreams.select("splitA ");


Thanks and regards,
Tovi



RE: slot group indication per operator

Posted by "Sofer, Tovi " <to...@citi.com>.
Hi.

Any update or suggestion on this?

Best regards,
Tovi
From: Timo Walther [mailto:twalthr@apache.org]
Sent: יום ג 05 דצמבר 2017 18:55
To: user@flink.apache.org
Cc: chesnay@apache.org
Subject: Re: slot group indication per operator

Hi Tovi,

you are right, it is difficult to check the correct behavior.

@Chesnay: Do you know if we can get this information? If not through the Web UI, maybe via REST? Do we have access to the full ExecutionGraph somewhere?

Otherwise it might make sense to open an issue for this.

Regards,
Timo


Am 12/5/17 um 4:25 PM schrieb Sofer, Tovi :
Hi all,
I am trying to use the slot group feature, by having ‘default’ group and additional ‘market’ group.
The purpose is to divide the resources equally between two sources and their following operators.
I’ve set the slotGroup on the source of the market data.
Can I assume that all following operators created from this source will use same slot group of ‘market’?
(The operators created for market stream are pretty complex, with connect and split).
In Web UI I saw there are 16 slots, but didn’t see indication per operator to which group it was assigned. How can I know?

Relevant Code:

env.setParallelism(8);
conf.setInteger(ConfigConstants.TASK_MANAGER_NUM_TASK_SLOTS, 16); \\ to allow Parallelism of 8 per group

// Market source and operators:

KeyedStream<SpotTickEvent, Tuple> windowedStreamA = sourceProvider.provide(env)

        .name(spotSourceProvider.getName())

        .slotSharingGroup(SourceMsgType.MARKET.slotGroup())

        .flatMap(new ParserMapper(new MarketMessageParser()))

        .name(ParserMapper.class.getSimpleName())

        .filter(new USDFilter())

        .name(USDFilter.class.getSimpleName())

        .keyBy(MarketEvent.CURRENCY_FIELD)

        .timeWindow(Time.of(windowSizeMs, TimeUnit.MILLISECONDS))

        .process(new LastInWindowPriceChangeFunction()))

        .name(LastInWindowPriceChangeFunction.class.getSimpleName())

        .keyBy(SpotTickEvent.CURRENCY_FIELD);


marketConnectedStream = windowedStreamA.connect(windowedStreamB)

            .flatMap(new MarketCoMapper()))

            .name(MarketCoMapper.class.getSimpleName())



SplitStream<MarketAWithMarketB> stocksWithSpotsStreams = marketConnectedStream

        .split( market -> ImmutableList.of("splitA"," splitB") );



DataStream< MarketAWithMarketB> splitA = stocksWithSpotsStreams.select("splitA ");


Thanks and regards,
Tovi





Re: slot group indication per operator

Posted by Timo Walther <tw...@apache.org>.
Hi Tovi,

you are right, it is difficult to check the correct behavior.

@Chesnay: Do you know if we can get this information? If not through the 
Web UI, maybe via REST? Do we have access to the full ExecutionGraph 
somewhere?

Otherwise it might make sense to open an issue for this.

Regards,
Timo


Am 12/5/17 um 4:25 PM schrieb Sofer, Tovi :
>
> Hi all,
>
> I am trying to use the slot group feature, by having ‘default’ group 
> and additional ‘market’ group.
>
> The purpose is to divide the resources equally between two sources and 
> their following operators.
>
> I’ve set the slotGroup on the source of the market data.
>
> Can I assume that all following operators created from this source 
> will use same slot group of ‘market’?
>
> (The operators created for market stream are pretty complex, with 
> connect and split).
>
> In Web UI I saw there are 16 slots, but didn’t see indication per 
> operator to which group it was assigned. How can I know?
>
> Relevant Code:
> env.setParallelism(8);
>
> conf.setInteger(ConfigConstants.*/TASK_MANAGER_NUM_TASK_SLOTS/*, 16); 
> \\ to allow Parallelism of 8 per group
>
> // Market source and operators:
>
> KeyedStream<SpotTickEvent, Tuple> windowedStreamA = *sourceProvider*.provide(env)         .name(*spotSourceProvider*.getName())         .slotSharingGroup(SourceMsgType.*/MARKET/*.slotGroup())         .flatMap(*new *ParserMapper(*new *MarketMessageParser()))         .name(ParserMapper.*class*.getSimpleName())         .filter(*new *USDFilter())         .name(USDFilter.*class*.getSimpleName())         .keyBy(MarketEvent.*/CURRENCY_FIELD/*)         .timeWindow(Time./of/(windowSizeMs, TimeUnit.*/MILLISECONDS/*))         .process(*new *LastInWindowPriceChangeFunction()))         
> .name(LastInWindowPriceChangeFunction.*class*.getSimpleName())         .keyBy(SpotTickEvent.*/CURRENCY_FIELD/*);
>
> marketConnectedStream = windowedStreamA.connect(windowedStreamB) 
>             .flatMap(*new *MarketCoMapper()))             .name(MarketCoMapper.*class*.getSimpleName())
> SplitStream<MarketAWithMarketB> stocksWithSpotsStreams = 
> marketConnectedStream         .split( market -> ImmutableList./of/(*"splitA"*,*" splitB"*) ); DataStream< MarketAWithMarketB> splitA = 
> stocksWithSpotsStreams.select(*"splitA "*);
>
> Thanks and regards,
>
> Tovi
>