You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hadoop.apache.org by 王洪军 <wa...@gmail.com> on 2013/06/16 06:45:54 UTC
how to get the mapreduce code which was pig/hive script translated to?
hi,all
how to get the mapreduce code which was pig/hive script translated to?
Best regards,
wanghj
Re: how to get the mapreduce code which was pig/hive script
translated to?
Posted by Marcos Luis Ortiz Valmaseda <ma...@gmail.com>.
Edward is right. With log4j, you can see that. Here, you have the example:
https://github.com/apache/hadoop-common/blob/HADOOP-3628/conf/log4j.properties
The relevant info in the docs:
http://hadoop.apache.org/docs/stable/cluster_setup.html#Logging
Some working examples:
http://stackoverflow.com/questions/8975609/setting-the-logging-level-in-hadoop-to-warn
http://stackoverflow.com/questions/7801610/how-to-suppress-hadoop-logging-message-on-the-console?rq=1
2013/6/16 Edward Capriolo <ed...@gmail.com>
> Hive serializes the entire plan into an XML file if you set the log 4j
> settings to debug you should get the locations to the files itgenerates
> before launching the job.
>
>
> On Sun, Jun 16, 2013 at 11:08 AM, Harsh J <ha...@cloudera.com> wrote:
>
>> This is a question for the Hive/Pig lists to answer best.
>>
>> Note though that they only compile a plan, not the code. The code is
>> available already, the compiled plan just structures the execution
>> flow. If you take a look at the sources, you'll find the bits and
>> pieces that get linked together depending on the operation.
>>
>> On Sun, Jun 16, 2013 at 10:15 AM, 王洪军 <wa...@gmail.com> wrote:
>> > hi,all
>> >
>> > how to get the mapreduce code which was pig/hive script translated
>> to?
>> >
>> > Best regards,
>> > wanghj
>> >
>>
>>
>>
>> --
>> Harsh J
>>
>
>
--
Marcos Ortiz Valmaseda
Product Manager at PDVSA
http://about.me/marcosortiz
Re: how to get the mapreduce code which was pig/hive script translated to?
Posted by Bharati <bh...@mparallelo.com>.
If I want to access and read the map classes that have been created where do I look for them.
Thanks,
Bharati
Sent from my iPad
On Jun 16, 2013, at 4:06 PM, Lance Norskog <go...@gmail.com> wrote:
> Both Pig and Hive have an 'explain plan' command that prints a schematic version. This might make it easier to see what M/R algorithms are used. Mostly the data goes through single-threaded transforms inside a mapper or reducer.
>
> https://cwiki.apache.org/Hive/languagemanual-explain.html
>
> On 06/16/2013 09:17 AM, Edward Capriolo wrote:
>> Hive serializes the entire plan into an XML file if you set the log 4j settings to debug you should get the locations to the files itgenerates before launching the job.
>>
>>
>> On Sun, Jun 16, 2013 at 11:08 AM, Harsh J <ha...@cloudera.com> wrote:
>>> This is a question for the Hive/Pig lists to answer best.
>>>
>>> Note though that they only compile a plan, not the code. The code is
>>> available already, the compiled plan just structures the execution
>>> flow. If you take a look at the sources, you'll find the bits and
>>> pieces that get linked together depending on the operation.
>>>
>>> On Sun, Jun 16, 2013 at 10:15 AM, 王洪军 <wa...@gmail.com> wrote:
>>> > hi,all
>>> >
>>> > how to get the mapreduce code which was pig/hive script translated to?
>>> >
>>> > Best regards,
>>> > wanghj
>>> >
>>>
>>>
>>>
>>> --
>>> Harsh J
>>
>
Re: how to get the mapreduce code which was pig/hive script translated to?
Posted by Bharati <bh...@mparallelo.com>.
If I want to access and read the map classes that have been created where do I look for them.
Thanks,
Bharati
Sent from my iPad
On Jun 16, 2013, at 4:06 PM, Lance Norskog <go...@gmail.com> wrote:
> Both Pig and Hive have an 'explain plan' command that prints a schematic version. This might make it easier to see what M/R algorithms are used. Mostly the data goes through single-threaded transforms inside a mapper or reducer.
>
> https://cwiki.apache.org/Hive/languagemanual-explain.html
>
> On 06/16/2013 09:17 AM, Edward Capriolo wrote:
>> Hive serializes the entire plan into an XML file if you set the log 4j settings to debug you should get the locations to the files itgenerates before launching the job.
>>
>>
>> On Sun, Jun 16, 2013 at 11:08 AM, Harsh J <ha...@cloudera.com> wrote:
>>> This is a question for the Hive/Pig lists to answer best.
>>>
>>> Note though that they only compile a plan, not the code. The code is
>>> available already, the compiled plan just structures the execution
>>> flow. If you take a look at the sources, you'll find the bits and
>>> pieces that get linked together depending on the operation.
>>>
>>> On Sun, Jun 16, 2013 at 10:15 AM, 王洪军 <wa...@gmail.com> wrote:
>>> > hi,all
>>> >
>>> > how to get the mapreduce code which was pig/hive script translated to?
>>> >
>>> > Best regards,
>>> > wanghj
>>> >
>>>
>>>
>>>
>>> --
>>> Harsh J
>>
>
Re: how to get the mapreduce code which was pig/hive script translated to?
Posted by Bharati <bh...@mparallelo.com>.
If I want to access and read the map classes that have been created where do I look for them.
Thanks,
Bharati
Sent from my iPad
On Jun 16, 2013, at 4:06 PM, Lance Norskog <go...@gmail.com> wrote:
> Both Pig and Hive have an 'explain plan' command that prints a schematic version. This might make it easier to see what M/R algorithms are used. Mostly the data goes through single-threaded transforms inside a mapper or reducer.
>
> https://cwiki.apache.org/Hive/languagemanual-explain.html
>
> On 06/16/2013 09:17 AM, Edward Capriolo wrote:
>> Hive serializes the entire plan into an XML file if you set the log 4j settings to debug you should get the locations to the files itgenerates before launching the job.
>>
>>
>> On Sun, Jun 16, 2013 at 11:08 AM, Harsh J <ha...@cloudera.com> wrote:
>>> This is a question for the Hive/Pig lists to answer best.
>>>
>>> Note though that they only compile a plan, not the code. The code is
>>> available already, the compiled plan just structures the execution
>>> flow. If you take a look at the sources, you'll find the bits and
>>> pieces that get linked together depending on the operation.
>>>
>>> On Sun, Jun 16, 2013 at 10:15 AM, 王洪军 <wa...@gmail.com> wrote:
>>> > hi,all
>>> >
>>> > how to get the mapreduce code which was pig/hive script translated to?
>>> >
>>> > Best regards,
>>> > wanghj
>>> >
>>>
>>>
>>>
>>> --
>>> Harsh J
>>
>
Re: how to get the mapreduce code which was pig/hive script translated to?
Posted by Bharati <bh...@mparallelo.com>.
If I want to access and read the map classes that have been created where do I look for them.
Thanks,
Bharati
Sent from my iPad
On Jun 16, 2013, at 4:06 PM, Lance Norskog <go...@gmail.com> wrote:
> Both Pig and Hive have an 'explain plan' command that prints a schematic version. This might make it easier to see what M/R algorithms are used. Mostly the data goes through single-threaded transforms inside a mapper or reducer.
>
> https://cwiki.apache.org/Hive/languagemanual-explain.html
>
> On 06/16/2013 09:17 AM, Edward Capriolo wrote:
>> Hive serializes the entire plan into an XML file if you set the log 4j settings to debug you should get the locations to the files itgenerates before launching the job.
>>
>>
>> On Sun, Jun 16, 2013 at 11:08 AM, Harsh J <ha...@cloudera.com> wrote:
>>> This is a question for the Hive/Pig lists to answer best.
>>>
>>> Note though that they only compile a plan, not the code. The code is
>>> available already, the compiled plan just structures the execution
>>> flow. If you take a look at the sources, you'll find the bits and
>>> pieces that get linked together depending on the operation.
>>>
>>> On Sun, Jun 16, 2013 at 10:15 AM, 王洪军 <wa...@gmail.com> wrote:
>>> > hi,all
>>> >
>>> > how to get the mapreduce code which was pig/hive script translated to?
>>> >
>>> > Best regards,
>>> > wanghj
>>> >
>>>
>>>
>>>
>>> --
>>> Harsh J
>>
>
Re: how to get the mapreduce code which was pig/hive script translated
to?
Posted by Lance Norskog <go...@gmail.com>.
Both Pig and Hive have an 'explain plan' command that prints a schematic
version. This might make it easier to see what M/R algorithms are used.
Mostly the data goes through single-threaded transforms inside a mapper
or reducer.
https://cwiki.apache.org/Hive/languagemanual-explain.html
On 06/16/2013 09:17 AM, Edward Capriolo wrote:
> Hive serializes the entire plan into an XML file if you set the log 4j
> settings to debug you should get the locations to the files
> itgenerates before launching the job.
>
>
> On Sun, Jun 16, 2013 at 11:08 AM, Harsh J <harsh@cloudera.com
> <ma...@cloudera.com>> wrote:
>
> This is a question for the Hive/Pig lists to answer best.
>
> Note though that they only compile a plan, not the code. The code is
> available already, the compiled plan just structures the execution
> flow. If you take a look at the sources, you'll find the bits and
> pieces that get linked together depending on the operation.
>
> On Sun, Jun 16, 2013 at 10:15 AM, 王洪军 <wanghj966@gmail.com
> <ma...@gmail.com>> wrote:
> > hi,all
> >
> > how to get the mapreduce code which was pig/hive script
> translated to?
> >
> > Best regards,
> > wanghj
> >
>
>
>
> --
> Harsh J
>
>
Re: how to get the mapreduce code which was pig/hive script
translated to?
Posted by Marcos Luis Ortiz Valmaseda <ma...@gmail.com>.
Edward is right. With log4j, you can see that. Here, you have the example:
https://github.com/apache/hadoop-common/blob/HADOOP-3628/conf/log4j.properties
The relevant info in the docs:
http://hadoop.apache.org/docs/stable/cluster_setup.html#Logging
Some working examples:
http://stackoverflow.com/questions/8975609/setting-the-logging-level-in-hadoop-to-warn
http://stackoverflow.com/questions/7801610/how-to-suppress-hadoop-logging-message-on-the-console?rq=1
2013/6/16 Edward Capriolo <ed...@gmail.com>
> Hive serializes the entire plan into an XML file if you set the log 4j
> settings to debug you should get the locations to the files itgenerates
> before launching the job.
>
>
> On Sun, Jun 16, 2013 at 11:08 AM, Harsh J <ha...@cloudera.com> wrote:
>
>> This is a question for the Hive/Pig lists to answer best.
>>
>> Note though that they only compile a plan, not the code. The code is
>> available already, the compiled plan just structures the execution
>> flow. If you take a look at the sources, you'll find the bits and
>> pieces that get linked together depending on the operation.
>>
>> On Sun, Jun 16, 2013 at 10:15 AM, 王洪军 <wa...@gmail.com> wrote:
>> > hi,all
>> >
>> > how to get the mapreduce code which was pig/hive script translated
>> to?
>> >
>> > Best regards,
>> > wanghj
>> >
>>
>>
>>
>> --
>> Harsh J
>>
>
>
--
Marcos Ortiz Valmaseda
Product Manager at PDVSA
http://about.me/marcosortiz
Re: how to get the mapreduce code which was pig/hive script translated
to?
Posted by Lance Norskog <go...@gmail.com>.
Both Pig and Hive have an 'explain plan' command that prints a schematic
version. This might make it easier to see what M/R algorithms are used.
Mostly the data goes through single-threaded transforms inside a mapper
or reducer.
https://cwiki.apache.org/Hive/languagemanual-explain.html
On 06/16/2013 09:17 AM, Edward Capriolo wrote:
> Hive serializes the entire plan into an XML file if you set the log 4j
> settings to debug you should get the locations to the files
> itgenerates before launching the job.
>
>
> On Sun, Jun 16, 2013 at 11:08 AM, Harsh J <harsh@cloudera.com
> <ma...@cloudera.com>> wrote:
>
> This is a question for the Hive/Pig lists to answer best.
>
> Note though that they only compile a plan, not the code. The code is
> available already, the compiled plan just structures the execution
> flow. If you take a look at the sources, you'll find the bits and
> pieces that get linked together depending on the operation.
>
> On Sun, Jun 16, 2013 at 10:15 AM, 王洪军 <wanghj966@gmail.com
> <ma...@gmail.com>> wrote:
> > hi,all
> >
> > how to get the mapreduce code which was pig/hive script
> translated to?
> >
> > Best regards,
> > wanghj
> >
>
>
>
> --
> Harsh J
>
>
Re: how to get the mapreduce code which was pig/hive script
translated to?
Posted by Marcos Luis Ortiz Valmaseda <ma...@gmail.com>.
Edward is right. With log4j, you can see that. Here, you have the example:
https://github.com/apache/hadoop-common/blob/HADOOP-3628/conf/log4j.properties
The relevant info in the docs:
http://hadoop.apache.org/docs/stable/cluster_setup.html#Logging
Some working examples:
http://stackoverflow.com/questions/8975609/setting-the-logging-level-in-hadoop-to-warn
http://stackoverflow.com/questions/7801610/how-to-suppress-hadoop-logging-message-on-the-console?rq=1
2013/6/16 Edward Capriolo <ed...@gmail.com>
> Hive serializes the entire plan into an XML file if you set the log 4j
> settings to debug you should get the locations to the files itgenerates
> before launching the job.
>
>
> On Sun, Jun 16, 2013 at 11:08 AM, Harsh J <ha...@cloudera.com> wrote:
>
>> This is a question for the Hive/Pig lists to answer best.
>>
>> Note though that they only compile a plan, not the code. The code is
>> available already, the compiled plan just structures the execution
>> flow. If you take a look at the sources, you'll find the bits and
>> pieces that get linked together depending on the operation.
>>
>> On Sun, Jun 16, 2013 at 10:15 AM, 王洪军 <wa...@gmail.com> wrote:
>> > hi,all
>> >
>> > how to get the mapreduce code which was pig/hive script translated
>> to?
>> >
>> > Best regards,
>> > wanghj
>> >
>>
>>
>>
>> --
>> Harsh J
>>
>
>
--
Marcos Ortiz Valmaseda
Product Manager at PDVSA
http://about.me/marcosortiz
Re: how to get the mapreduce code which was pig/hive script translated
to?
Posted by Lance Norskog <go...@gmail.com>.
Both Pig and Hive have an 'explain plan' command that prints a schematic
version. This might make it easier to see what M/R algorithms are used.
Mostly the data goes through single-threaded transforms inside a mapper
or reducer.
https://cwiki.apache.org/Hive/languagemanual-explain.html
On 06/16/2013 09:17 AM, Edward Capriolo wrote:
> Hive serializes the entire plan into an XML file if you set the log 4j
> settings to debug you should get the locations to the files
> itgenerates before launching the job.
>
>
> On Sun, Jun 16, 2013 at 11:08 AM, Harsh J <harsh@cloudera.com
> <ma...@cloudera.com>> wrote:
>
> This is a question for the Hive/Pig lists to answer best.
>
> Note though that they only compile a plan, not the code. The code is
> available already, the compiled plan just structures the execution
> flow. If you take a look at the sources, you'll find the bits and
> pieces that get linked together depending on the operation.
>
> On Sun, Jun 16, 2013 at 10:15 AM, 王洪军 <wanghj966@gmail.com
> <ma...@gmail.com>> wrote:
> > hi,all
> >
> > how to get the mapreduce code which was pig/hive script
> translated to?
> >
> > Best regards,
> > wanghj
> >
>
>
>
> --
> Harsh J
>
>
Re: how to get the mapreduce code which was pig/hive script
translated to?
Posted by Marcos Luis Ortiz Valmaseda <ma...@gmail.com>.
Edward is right. With log4j, you can see that. Here, you have the example:
https://github.com/apache/hadoop-common/blob/HADOOP-3628/conf/log4j.properties
The relevant info in the docs:
http://hadoop.apache.org/docs/stable/cluster_setup.html#Logging
Some working examples:
http://stackoverflow.com/questions/8975609/setting-the-logging-level-in-hadoop-to-warn
http://stackoverflow.com/questions/7801610/how-to-suppress-hadoop-logging-message-on-the-console?rq=1
2013/6/16 Edward Capriolo <ed...@gmail.com>
> Hive serializes the entire plan into an XML file if you set the log 4j
> settings to debug you should get the locations to the files itgenerates
> before launching the job.
>
>
> On Sun, Jun 16, 2013 at 11:08 AM, Harsh J <ha...@cloudera.com> wrote:
>
>> This is a question for the Hive/Pig lists to answer best.
>>
>> Note though that they only compile a plan, not the code. The code is
>> available already, the compiled plan just structures the execution
>> flow. If you take a look at the sources, you'll find the bits and
>> pieces that get linked together depending on the operation.
>>
>> On Sun, Jun 16, 2013 at 10:15 AM, 王洪军 <wa...@gmail.com> wrote:
>> > hi,all
>> >
>> > how to get the mapreduce code which was pig/hive script translated
>> to?
>> >
>> > Best regards,
>> > wanghj
>> >
>>
>>
>>
>> --
>> Harsh J
>>
>
>
--
Marcos Ortiz Valmaseda
Product Manager at PDVSA
http://about.me/marcosortiz
Re: how to get the mapreduce code which was pig/hive script translated
to?
Posted by Lance Norskog <go...@gmail.com>.
Both Pig and Hive have an 'explain plan' command that prints a schematic
version. This might make it easier to see what M/R algorithms are used.
Mostly the data goes through single-threaded transforms inside a mapper
or reducer.
https://cwiki.apache.org/Hive/languagemanual-explain.html
On 06/16/2013 09:17 AM, Edward Capriolo wrote:
> Hive serializes the entire plan into an XML file if you set the log 4j
> settings to debug you should get the locations to the files
> itgenerates before launching the job.
>
>
> On Sun, Jun 16, 2013 at 11:08 AM, Harsh J <harsh@cloudera.com
> <ma...@cloudera.com>> wrote:
>
> This is a question for the Hive/Pig lists to answer best.
>
> Note though that they only compile a plan, not the code. The code is
> available already, the compiled plan just structures the execution
> flow. If you take a look at the sources, you'll find the bits and
> pieces that get linked together depending on the operation.
>
> On Sun, Jun 16, 2013 at 10:15 AM, 王洪军 <wanghj966@gmail.com
> <ma...@gmail.com>> wrote:
> > hi,all
> >
> > how to get the mapreduce code which was pig/hive script
> translated to?
> >
> > Best regards,
> > wanghj
> >
>
>
>
> --
> Harsh J
>
>
Re: how to get the mapreduce code which was pig/hive script
translated to?
Posted by Edward Capriolo <ed...@gmail.com>.
Hive serializes the entire plan into an XML file if you set the log 4j
settings to debug you should get the locations to the files itgenerates
before launching the job.
On Sun, Jun 16, 2013 at 11:08 AM, Harsh J <ha...@cloudera.com> wrote:
> This is a question for the Hive/Pig lists to answer best.
>
> Note though that they only compile a plan, not the code. The code is
> available already, the compiled plan just structures the execution
> flow. If you take a look at the sources, you'll find the bits and
> pieces that get linked together depending on the operation.
>
> On Sun, Jun 16, 2013 at 10:15 AM, 王洪军 <wa...@gmail.com> wrote:
> > hi,all
> >
> > how to get the mapreduce code which was pig/hive script translated
> to?
> >
> > Best regards,
> > wanghj
> >
>
>
>
> --
> Harsh J
>
Re: how to get the mapreduce code which was pig/hive script
translated to?
Posted by Edward Capriolo <ed...@gmail.com>.
Hive serializes the entire plan into an XML file if you set the log 4j
settings to debug you should get the locations to the files itgenerates
before launching the job.
On Sun, Jun 16, 2013 at 11:08 AM, Harsh J <ha...@cloudera.com> wrote:
> This is a question for the Hive/Pig lists to answer best.
>
> Note though that they only compile a plan, not the code. The code is
> available already, the compiled plan just structures the execution
> flow. If you take a look at the sources, you'll find the bits and
> pieces that get linked together depending on the operation.
>
> On Sun, Jun 16, 2013 at 10:15 AM, 王洪军 <wa...@gmail.com> wrote:
> > hi,all
> >
> > how to get the mapreduce code which was pig/hive script translated
> to?
> >
> > Best regards,
> > wanghj
> >
>
>
>
> --
> Harsh J
>
Re: how to get the mapreduce code which was pig/hive script
translated to?
Posted by Edward Capriolo <ed...@gmail.com>.
Hive serializes the entire plan into an XML file if you set the log 4j
settings to debug you should get the locations to the files itgenerates
before launching the job.
On Sun, Jun 16, 2013 at 11:08 AM, Harsh J <ha...@cloudera.com> wrote:
> This is a question for the Hive/Pig lists to answer best.
>
> Note though that they only compile a plan, not the code. The code is
> available already, the compiled plan just structures the execution
> flow. If you take a look at the sources, you'll find the bits and
> pieces that get linked together depending on the operation.
>
> On Sun, Jun 16, 2013 at 10:15 AM, 王洪军 <wa...@gmail.com> wrote:
> > hi,all
> >
> > how to get the mapreduce code which was pig/hive script translated
> to?
> >
> > Best regards,
> > wanghj
> >
>
>
>
> --
> Harsh J
>
Re: how to get the mapreduce code which was pig/hive script
translated to?
Posted by Edward Capriolo <ed...@gmail.com>.
Hive serializes the entire plan into an XML file if you set the log 4j
settings to debug you should get the locations to the files itgenerates
before launching the job.
On Sun, Jun 16, 2013 at 11:08 AM, Harsh J <ha...@cloudera.com> wrote:
> This is a question for the Hive/Pig lists to answer best.
>
> Note though that they only compile a plan, not the code. The code is
> available already, the compiled plan just structures the execution
> flow. If you take a look at the sources, you'll find the bits and
> pieces that get linked together depending on the operation.
>
> On Sun, Jun 16, 2013 at 10:15 AM, 王洪军 <wa...@gmail.com> wrote:
> > hi,all
> >
> > how to get the mapreduce code which was pig/hive script translated
> to?
> >
> > Best regards,
> > wanghj
> >
>
>
>
> --
> Harsh J
>
Re: how to get the mapreduce code which was pig/hive script
translated to?
Posted by Harsh J <ha...@cloudera.com>.
This is a question for the Hive/Pig lists to answer best.
Note though that they only compile a plan, not the code. The code is
available already, the compiled plan just structures the execution
flow. If you take a look at the sources, you'll find the bits and
pieces that get linked together depending on the operation.
On Sun, Jun 16, 2013 at 10:15 AM, 王洪军 <wa...@gmail.com> wrote:
> hi,all
>
> how to get the mapreduce code which was pig/hive script translated to?
>
> Best regards,
> wanghj
>
--
Harsh J
Re: how to get the mapreduce code which was pig/hive script
translated to?
Posted by Harsh J <ha...@cloudera.com>.
This is a question for the Hive/Pig lists to answer best.
Note though that they only compile a plan, not the code. The code is
available already, the compiled plan just structures the execution
flow. If you take a look at the sources, you'll find the bits and
pieces that get linked together depending on the operation.
On Sun, Jun 16, 2013 at 10:15 AM, 王洪军 <wa...@gmail.com> wrote:
> hi,all
>
> how to get the mapreduce code which was pig/hive script translated to?
>
> Best regards,
> wanghj
>
--
Harsh J
Re: how to get the mapreduce code which was pig/hive script
translated to?
Posted by Harsh J <ha...@cloudera.com>.
This is a question for the Hive/Pig lists to answer best.
Note though that they only compile a plan, not the code. The code is
available already, the compiled plan just structures the execution
flow. If you take a look at the sources, you'll find the bits and
pieces that get linked together depending on the operation.
On Sun, Jun 16, 2013 at 10:15 AM, 王洪军 <wa...@gmail.com> wrote:
> hi,all
>
> how to get the mapreduce code which was pig/hive script translated to?
>
> Best regards,
> wanghj
>
--
Harsh J
Re: how to get the mapreduce code which was pig/hive script
translated to?
Posted by Harsh J <ha...@cloudera.com>.
This is a question for the Hive/Pig lists to answer best.
Note though that they only compile a plan, not the code. The code is
available already, the compiled plan just structures the execution
flow. If you take a look at the sources, you'll find the bits and
pieces that get linked together depending on the operation.
On Sun, Jun 16, 2013 at 10:15 AM, 王洪军 <wa...@gmail.com> wrote:
> hi,all
>
> how to get the mapreduce code which was pig/hive script translated to?
>
> Best regards,
> wanghj
>
--
Harsh J