Documentation on org.apache.spark.sql.functions backend.

[Vipul Rajan <vi...@gmail.com>]

I am trying to create a function that reads data from Kafka, communicates
with confluent schema registry and decodes avro data with evolving schemas.
I am trying to not create hack-ish patches and to write proper code that I
could maybe even create pull requests for. looking at the code I have been
able to figure out a few things regarding how expressions are generated and
how they help to accomplish what a function does, but there is still a ton
I just cannot wrap my head around.

I am unable to find any documentation that gets into such nitty gritties of
Spark. *I am writing in hopes to find some help. Do you have any
documentation that explains how a function
(org.apache.spark.sql.function._) is turned into a logical plan?*

Re: Documentation on org.apache.spark.sql.functions backend.

[Marco Gaido <ma...@gmail.com>]

Hi Vipul,

I am afraid I cannot help you on that.

Thanks,
Marco

Il giorno lun 16 set 2019 alle ore 10:44 Vipul Rajan <vi...@gmail.com>
ha scritto:

> Hi Marco,
>
> That does help. Thanks, for taking the time. I am confused as to how that
> Expression is created. There are methods like eval, nullSafeEval,
> doGenCode. Aren't there any architectural docs that could help with what is
> exactly happening? Reverse engineering seems a bit daunting.
>
> Regards
>
> On Mon, Sep 16, 2019 at 1:36 PM Marco Gaido <ma...@gmail.com>
> wrote:
>
>> Hi Vipul,
>>
>> a function is never turned in a logical plan. A function is turned into
>> an Expression. And an Expression can be part of many Logical or Physical
>> Plans.
>> Hope this helps.
>>
>> Thanks,
>> Marco
>>
>> Il giorno lun 16 set 2019 alle ore 08:27 Vipul Rajan <
>> vipul.s.paul@gmail.com> ha scritto:
>>
>>> I am trying to create a function that reads data from Kafka,
>>> communicates with confluent schema registry and decodes avro data with
>>> evolving schemas. I am trying to not create hack-ish patches and to write
>>> proper code that I could maybe even create pull requests for. looking at
>>> the code I have been able to figure out a few things regarding how
>>> expressions are generated and how they help to accomplish what a function
>>> does, but there is still a ton I just cannot wrap my head around.
>>>
>>> I am unable to find any documentation that gets into such nitty gritties
>>> of Spark. *I am writing in hopes to find some help. Do you have any
>>> documentation that explains how a function
>>> (org.apache.spark.sql.function._) is turned into a logical plan?*
>>>
>>

Re: Documentation on org.apache.spark.sql.functions backend.

[Vipul Rajan <vi...@gmail.com>]

Hi Marco,

That does help. Thanks, for taking the time. I am confused as to how that
Expression is created. There are methods like eval, nullSafeEval,
doGenCode. Aren't there any architectural docs that could help with what is
exactly happening? Reverse engineering seems a bit daunting.

Regards

On Mon, Sep 16, 2019 at 1:36 PM Marco Gaido <ma...@gmail.com> wrote:

> Hi Vipul,
>
> a function is never turned in a logical plan. A function is turned into an
> Expression. And an Expression can be part of many Logical or Physical Plans.
> Hope this helps.
>
> Thanks,
> Marco
>
> Il giorno lun 16 set 2019 alle ore 08:27 Vipul Rajan <
> vipul.s.paul@gmail.com> ha scritto:
>
>> I am trying to create a function that reads data from Kafka, communicates
>> with confluent schema registry and decodes avro data with evolving schemas.
>> I am trying to not create hack-ish patches and to write proper code that I
>> could maybe even create pull requests for. looking at the code I have been
>> able to figure out a few things regarding how expressions are generated and
>> how they help to accomplish what a function does, but there is still a ton
>> I just cannot wrap my head around.
>>
>> I am unable to find any documentation that gets into such nitty gritties
>> of Spark. *I am writing in hopes to find some help. Do you have any
>> documentation that explains how a function
>> (org.apache.spark.sql.function._) is turned into a logical plan?*
>>
>

Re: Documentation on org.apache.spark.sql.functions backend.

[Marco Gaido <ma...@gmail.com>]

Hi Vipul,

a function is never turned in a logical plan. A function is turned into an
Expression. And an Expression can be part of many Logical or Physical Plans.
Hope this helps.

Thanks,
Marco

Il giorno lun 16 set 2019 alle ore 08:27 Vipul Rajan <vi...@gmail.com>
ha scritto:

> I am trying to create a function that reads data from Kafka, communicates
> with confluent schema registry and decodes avro data with evolving schemas.
> I am trying to not create hack-ish patches and to write proper code that I
> could maybe even create pull requests for. looking at the code I have been
> able to figure out a few things regarding how expressions are generated and
> how they help to accomplish what a function does, but there is still a ton
> I just cannot wrap my head around.
>
> I am unable to find any documentation that gets into such nitty gritties
> of Spark. *I am writing in hopes to find some help. Do you have any
> documentation that explains how a function
> (org.apache.spark.sql.function._) is turned into a logical plan?*
>