Re: [DISCUSS] SPIP: Native support of session window

[Yuanjian Li <xy...@gmail.com>]

Cool, thanks!
Sorry for the late reply, we'll check out the UT and your design doc ASAP
when we back from National Day holiday.

Thanks,
Yuanjian Li

Jungtaek Lim <ka...@gmail.com> 于2018年9月29日周六 上午5:21写道：

> Btw, just wrote up detailed design doc on existing patch:
>
> https://docs.google.com/document/d/1tUO29BDXb9127RiivUS7Hv324dC0YHuokYvyQRpurDY/edit?usp=sharing
>
> This doc is a wall of text, since I guess we already imagine how session
> window works (and I showed a simple example in SPIP doc), so try to avoid
> drawing something which would take non-trivial efforts. New classes are
> linked to the actual source code so that we can read the code directly
> whenever curious/wonders about something.
>
> Please let me know anytime if something is unclear and need elaboration.
>
> -Jungtaek Lim (HeartSaVioR)
>
> 2018년 9월 28일 (금) 오후 10:18, Jungtaek Lim <ka...@gmail.com>님이 작성:
>
>> Thanks for sharing your proposal as well as implementation. Looks like
>> your proposal is more like focused to design details: I may be better to
>> write one more for design details and share it as well. Stay tuned!
>>
>> Btw, I'm trying out your patch to see whether it passes the tests I've
>> added, and looks like it fails on below UT:
>>
>> https://github.com/apache/spark/blob/ad0b7466ef3f79354a99bd1b95c23e4c308502d5/sql/core/src/test/scala/org/apache/spark/sql/streaming/EventTimeWatermarkSuite.scala#L475-L573
>> Could you take a look at UT and see whether I'm missing here or the UT is
>> correct?
>>
>> (Actually most of UTs I've added fail but some UTs are for update mode,
>> and the patch doesn't provide same experience with select only session
>> window, so I'm pointing only one UT which is testing basic session window.)
>>
>> -Jungtaek Lim (HeartSaVioR)
>>
>> 2018년 9월 28일 (금) 오후 9:22, Yuanjian Li <xy...@gmail.com>님이 작성:
>>
>>> Hi Jungtaek:
>>>
>>>    We also meet this problem during migration of streaming application
>>> to Structure Streaming in Baidu practice, we solved this in our folk and
>>> just steady running in product.
>>>    As the initial plan we are doing the code clean work and preparing
>>> give a SPIP in Oct, happy to see your proposal. Hope we share some spots
>>> together.
>>>    Here’s the PR and doc:
>>> https://github.com/apache/spark/pull/22583
>>>
>>> https://docs.google.com/document/d/1zeAc7QKSO7J4-Yk06kc76kvldl-QHLCDJuu04d7k2bg/edit?usp=sharing
>>>
>>> Thanks,
>>> Yuanjian Li
>>>
>>>
>>> 在 2018年9月28日，06:22，Jungtaek Lim <ka...@gmail.com> 写道：
>>>
>>> Hi all,
>>>
>>> I would like to initiate discussion thread to discuss "Native support of
>>> session window".
>>> Origin issue is filed to SPARK-10816 [1] but I can file another one for
>>> representing SPIP if necessary. WIP but working PR is available as well, so
>>> we can even test it directly or see the difference if some of us feel more
>>> convenient to go through the source code instead of doc.
>>>
>>> I've attached PDF version of SPIP in SPARK-10816, but adding Google Docs
>>> link [2] for who feel convenient to comment in doc.
>>>
>>> Please let me know if we would like to see also technical design for
>>> this. I avoid to go too deep on SPIP doc so anyone could review and see the
>>> benefit of adopting this.
>>>
>>> Looking forward to hear your feedback.
>>>
>>> Thanks,
>>> Jungtaek Lim (HeartSaVioR)
>>>
>>> 1. https://issues.apache.org/jira/browse/SPARK-10816
>>> 2.
>>> https://docs.google.com/document/d/1_rMLmUSyGzb62RnP2A3WX6D6uRxox8Q_7WcoI_HrTw4/edit?usp=sharing
>>> 3. https://github.com/apache/spark/pull/22482
>>>
>>>
>>>
>>>