You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@apex.apache.org by Christian Hellström <ps...@gmail.com> on 2016/03/18 19:49:35 UTC
Out of order processing
The doc states that the order of events in processors is preserved (based
on how they are emitted), but does that mean that there is no event-time
concept that allows out-of-order processing of events?
Moreover, can events be re-prioritized based on their contents, e.g. when
they contain special phrases (e.g. AUTH in a security context). NiFi allows
this, which is useful in situations with limited bandwidth. I was just
wondering...
Thanks!
Re: Out of order processing
Posted by Thomas Weise <th...@gmail.com>.
Currently event time can be handled on the operator level with time
bucketing (which allows for late / out of order arrival). We are planning
to generalize this in the future.
Thomas
On Fri, Mar 18, 2016 at 12:00 PM, Christian Hellström <ps...@gmail.com>
wrote:
> Ok, thanks. Are there plans to support event-time processing?
>
> That would work with the separate path in the DAG. Cool!
> On 18 Mar 2016 19:58, "Sandesh Hegde" <sa...@datatorrent.com> wrote:
>
>> Hello Christian,
>>
>> Apex doesn't have a support for event time processing. For the second
>> scenario, how about creating a separate path in the DAG based on the
>> content?
>>
>> Thanks
>> Sandesh
>>
>> On Fri, Mar 18, 2016 at 11:49 AM Christian Hellström <
>> psilonline@gmail.com> wrote:
>>
>>> The doc states that the order of events in processors is preserved
>>> (based on how they are emitted), but does that mean that there is no
>>> event-time concept that allows out-of-order processing of events?
>>>
>>> Moreover, can events be re-prioritized based on their contents, e.g.
>>> when they contain special phrases (e.g. AUTH in a security context). NiFi
>>> allows this, which is useful in situations with limited bandwidth. I was
>>> just wondering...
>>>
>>> Thanks!
>>>
>>
Re: Out of order processing
Posted by Amol Kekre <am...@datatorrent.com>.
Christian,
Currently you can do so in user code. It should work well. An operator that
emits events on different ports based on time buckets will work. A second
approach would be to use event-time as the partition key and rotate through
partitions say every N bucket.
Thks,
Amol
On Fri, Mar 18, 2016 at 12:06 PM, Sandesh Hegde <sa...@datatorrent.com>
wrote:
> We definitely want to support "event-time" and more advanced windowing
> concepts, but it also depends on the interest of the community.
>
> Our current roadmap is here
> http://apex.incubator.apache.org/roadmap.html
>
>
>
> On Fri, Mar 18, 2016 at 12:00 PM Christian Hellström <ps...@gmail.com>
> wrote:
>
>> Ok, thanks. Are there plans to support event-time processing?
>>
>> That would work with the separate path in the DAG. Cool!
>> On 18 Mar 2016 19:58, "Sandesh Hegde" <sa...@datatorrent.com> wrote:
>>
>>> Hello Christian,
>>>
>>> Apex doesn't have a support for event time processing. For the second
>>> scenario, how about creating a separate path in the DAG based on the
>>> content?
>>>
>>> Thanks
>>> Sandesh
>>>
>>> On Fri, Mar 18, 2016 at 11:49 AM Christian Hellström <
>>> psilonline@gmail.com> wrote:
>>>
>>>> The doc states that the order of events in processors is preserved
>>>> (based on how they are emitted), but does that mean that there is no
>>>> event-time concept that allows out-of-order processing of events?
>>>>
>>>> Moreover, can events be re-prioritized based on their contents, e.g.
>>>> when they contain special phrases (e.g. AUTH in a security context). NiFi
>>>> allows this, which is useful in situations with limited bandwidth. I was
>>>> just wondering...
>>>>
>>>> Thanks!
>>>>
>>>
Re: Out of order processing
Posted by Sandesh Hegde <sa...@datatorrent.com>.
We definitely want to support "event-time" and more advanced windowing
concepts, but it also depends on the interest of the community.
Our current roadmap is here
http://apex.incubator.apache.org/roadmap.html
On Fri, Mar 18, 2016 at 12:00 PM Christian Hellström <ps...@gmail.com>
wrote:
> Ok, thanks. Are there plans to support event-time processing?
>
> That would work with the separate path in the DAG. Cool!
> On 18 Mar 2016 19:58, "Sandesh Hegde" <sa...@datatorrent.com> wrote:
>
>> Hello Christian,
>>
>> Apex doesn't have a support for event time processing. For the second
>> scenario, how about creating a separate path in the DAG based on the
>> content?
>>
>> Thanks
>> Sandesh
>>
>> On Fri, Mar 18, 2016 at 11:49 AM Christian Hellström <
>> psilonline@gmail.com> wrote:
>>
>>> The doc states that the order of events in processors is preserved
>>> (based on how they are emitted), but does that mean that there is no
>>> event-time concept that allows out-of-order processing of events?
>>>
>>> Moreover, can events be re-prioritized based on their contents, e.g.
>>> when they contain special phrases (e.g. AUTH in a security context). NiFi
>>> allows this, which is useful in situations with limited bandwidth. I was
>>> just wondering...
>>>
>>> Thanks!
>>>
>>
Re: Out of order processing
Posted by Christian Hellström <ps...@gmail.com>.
Ok, thanks. Are there plans to support event-time processing?
That would work with the separate path in the DAG. Cool!
On 18 Mar 2016 19:58, "Sandesh Hegde" <sa...@datatorrent.com> wrote:
> Hello Christian,
>
> Apex doesn't have a support for event time processing. For the second
> scenario, how about creating a separate path in the DAG based on the
> content?
>
> Thanks
> Sandesh
>
> On Fri, Mar 18, 2016 at 11:49 AM Christian Hellström <ps...@gmail.com>
> wrote:
>
>> The doc states that the order of events in processors is preserved (based
>> on how they are emitted), but does that mean that there is no event-time
>> concept that allows out-of-order processing of events?
>>
>> Moreover, can events be re-prioritized based on their contents, e.g. when
>> they contain special phrases (e.g. AUTH in a security context). NiFi allows
>> this, which is useful in situations with limited bandwidth. I was just
>> wondering...
>>
>> Thanks!
>>
>
Re: Out of order processing
Posted by Sandesh Hegde <sa...@datatorrent.com>.
Hello Christian,
Apex doesn't have a support for event time processing. For the second
scenario, how about creating a separate path in the DAG based on the
content?
Thanks
Sandesh
On Fri, Mar 18, 2016 at 11:49 AM Christian Hellström <ps...@gmail.com>
wrote:
> The doc states that the order of events in processors is preserved (based
> on how they are emitted), but does that mean that there is no event-time
> concept that allows out-of-order processing of events?
>
> Moreover, can events be re-prioritized based on their contents, e.g. when
> they contain special phrases (e.g. AUTH in a security context). NiFi allows
> this, which is useful in situations with limited bandwidth. I was just
> wondering...
>
> Thanks!
>