You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@spark.apache.org by Jacek Laskowski <ja...@japila.pl> on 2021/05/15 14:28:49 UTC

Purpose of OffsetHolder as a LeafNode?

Hi,

Just stumbled upon OffsetHolder [1] and am curious why it's a LeafNode?
What logical plan could it be part of?

[1]
https://github.com/apache/spark/blob/1a6708918b32e821bff26a00d2d8b7236b29515f/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/MicroBatchExecution.scala#L633

Pozdrawiam,
Jacek Laskowski
----
https://about.me/JacekLaskowski
"The Internals Of" Online Books <https://books.japila.pl/>
Follow me on https://twitter.com/jaceklaskowski

<https://twitter.com/jaceklaskowski>

Re: Purpose of OffsetHolder as a LeafNode?

Posted by Wenchen Fan <cl...@gmail.com>.

It's just an immediate place holder to update the query plan in each
micro-batch.

On Sat, May 15, 2021 at 10:29 PM Jacek Laskowski <ja...@japila.pl> wrote:

> Hi,
>
> Just stumbled upon OffsetHolder [1] and am curious why it's a LeafNode?
> What logical plan could it be part of?
>
> [1]
> https://github.com/apache/spark/blob/1a6708918b32e821bff26a00d2d8b7236b29515f/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/MicroBatchExecution.scala#L633
>
> Pozdrawiam,
> Jacek Laskowski
> ----
> https://about.me/JacekLaskowski
> "The Internals Of" Online Books <https://books.japila.pl/>
> Follow me on https://twitter.com/jaceklaskowski
>
> <https://twitter.com/jaceklaskowski>
>