Spark_sql

[Ajay Chander <it...@gmail.com>]

Hi Everyone,

I have a use case where I have to create a DataFrame inside the map()
function. To create a DataFrame it need sqlContext or hiveContext. Now how
do I pass the context to my map function ? And I am doing it in java. I
tried creating a class "TestClass" which implements "Function<Row, String>"
and inside the call method I want to create the DataFrame, so I created a
parameterized constructor to pass context from driver program to TestClass
and use that context to create DataFrame. But it seems like it's a wrong
way of doing. Can anyone help me in this? Thanks in advance.

Regards,
Aj

Re: Spark_sql

[Ted Yu <yu...@gmail.com>]

I don't think passing sqlContext to map() is supported.

Can you describe your use case in more detail ? Why do you need to create a
DataFrame inside the map() function ?

Cheers

On Wed, Oct 21, 2015 at 6:32 PM, Ajay Chander <it...@gmail.com> wrote:

> Hi Everyone,
>
> I have a use case where I have to create a DataFrame inside the map()
> function. To create a DataFrame it need sqlContext or hiveContext. Now how
> do I pass the context to my map function ? And I am doing it in java. I
> tried creating a class "TestClass" which implements "Function<Row, String>"
> and inside the call method I want to create the DataFrame, so I created a
> parameterized constructor to pass context from driver program to TestClass
> and use that context to create DataFrame. But it seems like it's a wrong
> way of doing. Can anyone help me in this? Thanks in advance.
>
> Regards,
> Aj
>