You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-user@hadoop.apache.org by nagarjuna kanamarlapudi <na...@gmail.com> on 2013/01/18 13:59:16 UTC
Execution of udf
Hi,
Select col1,myudf(col2,col3) from table1;
In what phase if map reduce an udf is executed.
In the very beginning, I assumed that hive will be joining two tables.,
getting the required columns and then applies udf on columns specified
I.e., essentially on reducer phase . But later on I realised that I was
wrong.
Is there any specific parameter which suggests hive to call udf at reducer
phase rather than at Mapper phase.
Regards,
Nagarjuna
--
Sent from iPhone
Re: Execution of udf
Posted by Dean Wampler <de...@thinkbiganalytics.com>.
That should be an identity pass through or NOOP that does nothing, but
MapReduce requires "something".
On Fri, Jan 18, 2013 at 8:57 AM, nagarjuna kanamarlapudi <
nagarjuna.kanamarlapudi@gmail.com> wrote:
> No but the query execution shows a reducer running .. And infant I feel
> that reduce phase can be there
>
>
> On Friday, January 18, 2013, Dean Wampler wrote:
>
>> There is no reduce phase needed in this query.
>>
>> On Fri, Jan 18, 2013 at 6:59 AM, nagarjuna kanamarlapudi <
>> nagarjuna.kanamarlapudi@gmail.com> wrote:
>>
>>> Hi,
>>>
>>> Select col1,myudf(col2,col3) from table1;
>>>
>>>
>>> In what phase if map reduce an udf is executed.
>>>
>>> In the very beginning, I assumed that hive will be joining two tables.,
>>> getting the required columns and then applies udf on columns specified
>>> I.e., essentially on reducer phase . But later on I realised that I was
>>> wrong.
>>>
>>> Is there any specific parameter which suggests hive to call udf at
>>> reducer phase rather than at Mapper phase.
>>>
>>>
>>> Regards,
>>> Nagarjuna
>>>
>>>
>>> --
>>> Sent from iPhone
>>>
>>
>>
>>
>> --
>> *Dean Wampler, Ph.D.*
>> thinkbiganalytics.com
>> +1-312-339-1330
>>
>>
>
> --
> Sent from iPhone
>
--
*Dean Wampler, Ph.D.*
thinkbiganalytics.com
+1-312-339-1330
Re: Execution of udf
Posted by Dean Wampler <de...@thinkbiganalytics.com>.
That should be an identity pass through or NOOP that does nothing, but
MapReduce requires "something".
On Fri, Jan 18, 2013 at 8:57 AM, nagarjuna kanamarlapudi <
nagarjuna.kanamarlapudi@gmail.com> wrote:
> No but the query execution shows a reducer running .. And infant I feel
> that reduce phase can be there
>
>
> On Friday, January 18, 2013, Dean Wampler wrote:
>
>> There is no reduce phase needed in this query.
>>
>> On Fri, Jan 18, 2013 at 6:59 AM, nagarjuna kanamarlapudi <
>> nagarjuna.kanamarlapudi@gmail.com> wrote:
>>
>>> Hi,
>>>
>>> Select col1,myudf(col2,col3) from table1;
>>>
>>>
>>> In what phase if map reduce an udf is executed.
>>>
>>> In the very beginning, I assumed that hive will be joining two tables.,
>>> getting the required columns and then applies udf on columns specified
>>> I.e., essentially on reducer phase . But later on I realised that I was
>>> wrong.
>>>
>>> Is there any specific parameter which suggests hive to call udf at
>>> reducer phase rather than at Mapper phase.
>>>
>>>
>>> Regards,
>>> Nagarjuna
>>>
>>>
>>> --
>>> Sent from iPhone
>>>
>>
>>
>>
>> --
>> *Dean Wampler, Ph.D.*
>> thinkbiganalytics.com
>> +1-312-339-1330
>>
>>
>
> --
> Sent from iPhone
>
--
*Dean Wampler, Ph.D.*
thinkbiganalytics.com
+1-312-339-1330
Re: Execution of udf
Posted by Dean Wampler <de...@thinkbiganalytics.com>.
That should be an identity pass through or NOOP that does nothing, but
MapReduce requires "something".
On Fri, Jan 18, 2013 at 8:57 AM, nagarjuna kanamarlapudi <
nagarjuna.kanamarlapudi@gmail.com> wrote:
> No but the query execution shows a reducer running .. And infant I feel
> that reduce phase can be there
>
>
> On Friday, January 18, 2013, Dean Wampler wrote:
>
>> There is no reduce phase needed in this query.
>>
>> On Fri, Jan 18, 2013 at 6:59 AM, nagarjuna kanamarlapudi <
>> nagarjuna.kanamarlapudi@gmail.com> wrote:
>>
>>> Hi,
>>>
>>> Select col1,myudf(col2,col3) from table1;
>>>
>>>
>>> In what phase if map reduce an udf is executed.
>>>
>>> In the very beginning, I assumed that hive will be joining two tables.,
>>> getting the required columns and then applies udf on columns specified
>>> I.e., essentially on reducer phase . But later on I realised that I was
>>> wrong.
>>>
>>> Is there any specific parameter which suggests hive to call udf at
>>> reducer phase rather than at Mapper phase.
>>>
>>>
>>> Regards,
>>> Nagarjuna
>>>
>>>
>>> --
>>> Sent from iPhone
>>>
>>
>>
>>
>> --
>> *Dean Wampler, Ph.D.*
>> thinkbiganalytics.com
>> +1-312-339-1330
>>
>>
>
> --
> Sent from iPhone
>
--
*Dean Wampler, Ph.D.*
thinkbiganalytics.com
+1-312-339-1330
Re: Execution of udf
Posted by Dean Wampler <de...@thinkbiganalytics.com>.
That should be an identity pass through or NOOP that does nothing, but
MapReduce requires "something".
On Fri, Jan 18, 2013 at 8:57 AM, nagarjuna kanamarlapudi <
nagarjuna.kanamarlapudi@gmail.com> wrote:
> No but the query execution shows a reducer running .. And infant I feel
> that reduce phase can be there
>
>
> On Friday, January 18, 2013, Dean Wampler wrote:
>
>> There is no reduce phase needed in this query.
>>
>> On Fri, Jan 18, 2013 at 6:59 AM, nagarjuna kanamarlapudi <
>> nagarjuna.kanamarlapudi@gmail.com> wrote:
>>
>>> Hi,
>>>
>>> Select col1,myudf(col2,col3) from table1;
>>>
>>>
>>> In what phase if map reduce an udf is executed.
>>>
>>> In the very beginning, I assumed that hive will be joining two tables.,
>>> getting the required columns and then applies udf on columns specified
>>> I.e., essentially on reducer phase . But later on I realised that I was
>>> wrong.
>>>
>>> Is there any specific parameter which suggests hive to call udf at
>>> reducer phase rather than at Mapper phase.
>>>
>>>
>>> Regards,
>>> Nagarjuna
>>>
>>>
>>> --
>>> Sent from iPhone
>>>
>>
>>
>>
>> --
>> *Dean Wampler, Ph.D.*
>> thinkbiganalytics.com
>> +1-312-339-1330
>>
>>
>
> --
> Sent from iPhone
>
--
*Dean Wampler, Ph.D.*
thinkbiganalytics.com
+1-312-339-1330
Re: Execution of udf
Posted by Dean Wampler <de...@thinkbiganalytics.com>.
That should be an identity pass through or NOOP that does nothing, but
MapReduce requires "something".
On Fri, Jan 18, 2013 at 8:57 AM, nagarjuna kanamarlapudi <
nagarjuna.kanamarlapudi@gmail.com> wrote:
> No but the query execution shows a reducer running .. And infant I feel
> that reduce phase can be there
>
>
> On Friday, January 18, 2013, Dean Wampler wrote:
>
>> There is no reduce phase needed in this query.
>>
>> On Fri, Jan 18, 2013 at 6:59 AM, nagarjuna kanamarlapudi <
>> nagarjuna.kanamarlapudi@gmail.com> wrote:
>>
>>> Hi,
>>>
>>> Select col1,myudf(col2,col3) from table1;
>>>
>>>
>>> In what phase if map reduce an udf is executed.
>>>
>>> In the very beginning, I assumed that hive will be joining two tables.,
>>> getting the required columns and then applies udf on columns specified
>>> I.e., essentially on reducer phase . But later on I realised that I was
>>> wrong.
>>>
>>> Is there any specific parameter which suggests hive to call udf at
>>> reducer phase rather than at Mapper phase.
>>>
>>>
>>> Regards,
>>> Nagarjuna
>>>
>>>
>>> --
>>> Sent from iPhone
>>>
>>
>>
>>
>> --
>> *Dean Wampler, Ph.D.*
>> thinkbiganalytics.com
>> +1-312-339-1330
>>
>>
>
> --
> Sent from iPhone
>
--
*Dean Wampler, Ph.D.*
thinkbiganalytics.com
+1-312-339-1330
Re: Execution of udf
Posted by nagarjuna kanamarlapudi <na...@gmail.com>.
No but the query execution shows a reducer running .. And infant I feel
that reduce phase can be there
On Friday, January 18, 2013, Dean Wampler wrote:
> There is no reduce phase needed in this query.
>
> On Fri, Jan 18, 2013 at 6:59 AM, nagarjuna kanamarlapudi <
> nagarjuna.kanamarlapudi@gmail.com <javascript:_e({}, 'cvml',
> 'nagarjuna.kanamarlapudi@gmail.com');>> wrote:
>
>> Hi,
>>
>> Select col1,myudf(col2,col3) from table1;
>>
>>
>> In what phase if map reduce an udf is executed.
>>
>> In the very beginning, I assumed that hive will be joining two tables.,
>> getting the required columns and then applies udf on columns specified
>> I.e., essentially on reducer phase . But later on I realised that I was
>> wrong.
>>
>> Is there any specific parameter which suggests hive to call udf at
>> reducer phase rather than at Mapper phase.
>>
>>
>> Regards,
>> Nagarjuna
>>
>>
>> --
>> Sent from iPhone
>>
>
>
>
> --
> *Dean Wampler, Ph.D.*
> thinkbiganalytics.com
> +1-312-339-1330
>
>
--
Sent from iPhone
Re: Execution of udf
Posted by nagarjuna kanamarlapudi <na...@gmail.com>.
No but the query execution shows a reducer running .. And infant I feel
that reduce phase can be there
On Friday, January 18, 2013, Dean Wampler wrote:
> There is no reduce phase needed in this query.
>
> On Fri, Jan 18, 2013 at 6:59 AM, nagarjuna kanamarlapudi <
> nagarjuna.kanamarlapudi@gmail.com <javascript:_e({}, 'cvml',
> 'nagarjuna.kanamarlapudi@gmail.com');>> wrote:
>
>> Hi,
>>
>> Select col1,myudf(col2,col3) from table1;
>>
>>
>> In what phase if map reduce an udf is executed.
>>
>> In the very beginning, I assumed that hive will be joining two tables.,
>> getting the required columns and then applies udf on columns specified
>> I.e., essentially on reducer phase . But later on I realised that I was
>> wrong.
>>
>> Is there any specific parameter which suggests hive to call udf at
>> reducer phase rather than at Mapper phase.
>>
>>
>> Regards,
>> Nagarjuna
>>
>>
>> --
>> Sent from iPhone
>>
>
>
>
> --
> *Dean Wampler, Ph.D.*
> thinkbiganalytics.com
> +1-312-339-1330
>
>
--
Sent from iPhone
Re: Execution of udf
Posted by nagarjuna kanamarlapudi <na...@gmail.com>.
No but the query execution shows a reducer running .. And infant I feel
that reduce phase can be there
On Friday, January 18, 2013, Dean Wampler wrote:
> There is no reduce phase needed in this query.
>
> On Fri, Jan 18, 2013 at 6:59 AM, nagarjuna kanamarlapudi <
> nagarjuna.kanamarlapudi@gmail.com <javascript:_e({}, 'cvml',
> 'nagarjuna.kanamarlapudi@gmail.com');>> wrote:
>
>> Hi,
>>
>> Select col1,myudf(col2,col3) from table1;
>>
>>
>> In what phase if map reduce an udf is executed.
>>
>> In the very beginning, I assumed that hive will be joining two tables.,
>> getting the required columns and then applies udf on columns specified
>> I.e., essentially on reducer phase . But later on I realised that I was
>> wrong.
>>
>> Is there any specific parameter which suggests hive to call udf at
>> reducer phase rather than at Mapper phase.
>>
>>
>> Regards,
>> Nagarjuna
>>
>>
>> --
>> Sent from iPhone
>>
>
>
>
> --
> *Dean Wampler, Ph.D.*
> thinkbiganalytics.com
> +1-312-339-1330
>
>
--
Sent from iPhone
Re: Execution of udf
Posted by nagarjuna kanamarlapudi <na...@gmail.com>.
No but the query execution shows a reducer running .. And infant I feel
that reduce phase can be there
On Friday, January 18, 2013, Dean Wampler wrote:
> There is no reduce phase needed in this query.
>
> On Fri, Jan 18, 2013 at 6:59 AM, nagarjuna kanamarlapudi <
> nagarjuna.kanamarlapudi@gmail.com <javascript:_e({}, 'cvml',
> 'nagarjuna.kanamarlapudi@gmail.com');>> wrote:
>
>> Hi,
>>
>> Select col1,myudf(col2,col3) from table1;
>>
>>
>> In what phase if map reduce an udf is executed.
>>
>> In the very beginning, I assumed that hive will be joining two tables.,
>> getting the required columns and then applies udf on columns specified
>> I.e., essentially on reducer phase . But later on I realised that I was
>> wrong.
>>
>> Is there any specific parameter which suggests hive to call udf at
>> reducer phase rather than at Mapper phase.
>>
>>
>> Regards,
>> Nagarjuna
>>
>>
>> --
>> Sent from iPhone
>>
>
>
>
> --
> *Dean Wampler, Ph.D.*
> thinkbiganalytics.com
> +1-312-339-1330
>
>
--
Sent from iPhone
Re: Execution of udf
Posted by nagarjuna kanamarlapudi <na...@gmail.com>.
No but the query execution shows a reducer running .. And infant I feel
that reduce phase can be there
On Friday, January 18, 2013, Dean Wampler wrote:
> There is no reduce phase needed in this query.
>
> On Fri, Jan 18, 2013 at 6:59 AM, nagarjuna kanamarlapudi <
> nagarjuna.kanamarlapudi@gmail.com <javascript:_e({}, 'cvml',
> 'nagarjuna.kanamarlapudi@gmail.com');>> wrote:
>
>> Hi,
>>
>> Select col1,myudf(col2,col3) from table1;
>>
>>
>> In what phase if map reduce an udf is executed.
>>
>> In the very beginning, I assumed that hive will be joining two tables.,
>> getting the required columns and then applies udf on columns specified
>> I.e., essentially on reducer phase . But later on I realised that I was
>> wrong.
>>
>> Is there any specific parameter which suggests hive to call udf at
>> reducer phase rather than at Mapper phase.
>>
>>
>> Regards,
>> Nagarjuna
>>
>>
>> --
>> Sent from iPhone
>>
>
>
>
> --
> *Dean Wampler, Ph.D.*
> thinkbiganalytics.com
> +1-312-339-1330
>
>
--
Sent from iPhone
Re: Execution of udf
Posted by Dean Wampler <de...@thinkbiganalytics.com>.
There is no reduce phase needed in this query.
On Fri, Jan 18, 2013 at 6:59 AM, nagarjuna kanamarlapudi <
nagarjuna.kanamarlapudi@gmail.com> wrote:
> Hi,
>
> Select col1,myudf(col2,col3) from table1;
>
>
> In what phase if map reduce an udf is executed.
>
> In the very beginning, I assumed that hive will be joining two tables.,
> getting the required columns and then applies udf on columns specified
> I.e., essentially on reducer phase . But later on I realised that I was
> wrong.
>
> Is there any specific parameter which suggests hive to call udf at reducer
> phase rather than at Mapper phase.
>
>
> Regards,
> Nagarjuna
>
>
> --
> Sent from iPhone
>
--
*Dean Wampler, Ph.D.*
thinkbiganalytics.com
+1-312-339-1330
Re: Execution of udf
Posted by Dean Wampler <de...@thinkbiganalytics.com>.
There is no reduce phase needed in this query.
On Fri, Jan 18, 2013 at 6:59 AM, nagarjuna kanamarlapudi <
nagarjuna.kanamarlapudi@gmail.com> wrote:
> Hi,
>
> Select col1,myudf(col2,col3) from table1;
>
>
> In what phase if map reduce an udf is executed.
>
> In the very beginning, I assumed that hive will be joining two tables.,
> getting the required columns and then applies udf on columns specified
> I.e., essentially on reducer phase . But later on I realised that I was
> wrong.
>
> Is there any specific parameter which suggests hive to call udf at reducer
> phase rather than at Mapper phase.
>
>
> Regards,
> Nagarjuna
>
>
> --
> Sent from iPhone
>
--
*Dean Wampler, Ph.D.*
thinkbiganalytics.com
+1-312-339-1330
Re: Execution of udf
Posted by Dean Wampler <de...@thinkbiganalytics.com>.
There is no reduce phase needed in this query.
On Fri, Jan 18, 2013 at 6:59 AM, nagarjuna kanamarlapudi <
nagarjuna.kanamarlapudi@gmail.com> wrote:
> Hi,
>
> Select col1,myudf(col2,col3) from table1;
>
>
> In what phase if map reduce an udf is executed.
>
> In the very beginning, I assumed that hive will be joining two tables.,
> getting the required columns and then applies udf on columns specified
> I.e., essentially on reducer phase . But later on I realised that I was
> wrong.
>
> Is there any specific parameter which suggests hive to call udf at reducer
> phase rather than at Mapper phase.
>
>
> Regards,
> Nagarjuna
>
>
> --
> Sent from iPhone
>
--
*Dean Wampler, Ph.D.*
thinkbiganalytics.com
+1-312-339-1330
Re: Execution of udf
Posted by Dean Wampler <de...@thinkbiganalytics.com>.
There is no reduce phase needed in this query.
On Fri, Jan 18, 2013 at 6:59 AM, nagarjuna kanamarlapudi <
nagarjuna.kanamarlapudi@gmail.com> wrote:
> Hi,
>
> Select col1,myudf(col2,col3) from table1;
>
>
> In what phase if map reduce an udf is executed.
>
> In the very beginning, I assumed that hive will be joining two tables.,
> getting the required columns and then applies udf on columns specified
> I.e., essentially on reducer phase . But later on I realised that I was
> wrong.
>
> Is there any specific parameter which suggests hive to call udf at reducer
> phase rather than at Mapper phase.
>
>
> Regards,
> Nagarjuna
>
>
> --
> Sent from iPhone
>
--
*Dean Wampler, Ph.D.*
thinkbiganalytics.com
+1-312-339-1330
Re: Execution of udf
Posted by Dean Wampler <de...@thinkbiganalytics.com>.
There is no reduce phase needed in this query.
On Fri, Jan 18, 2013 at 6:59 AM, nagarjuna kanamarlapudi <
nagarjuna.kanamarlapudi@gmail.com> wrote:
> Hi,
>
> Select col1,myudf(col2,col3) from table1;
>
>
> In what phase if map reduce an udf is executed.
>
> In the very beginning, I assumed that hive will be joining two tables.,
> getting the required columns and then applies udf on columns specified
> I.e., essentially on reducer phase . But later on I realised that I was
> wrong.
>
> Is there any specific parameter which suggests hive to call udf at reducer
> phase rather than at Mapper phase.
>
>
> Regards,
> Nagarjuna
>
>
> --
> Sent from iPhone
>
--
*Dean Wampler, Ph.D.*
thinkbiganalytics.com
+1-312-339-1330