You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kylin.apache.org by Luke Han <lu...@gmail.com> on 2015/06/04 17:34:57 UTC
Fwd: Why kylin use MR instead of Tez ?
Forward to mailing list for further support.
Thanks.
Best Regards!
---------------------
Luke Han
---------- Forwarded message ----------
From: donald fossouo <fo...@gmail.com>
Date: 2015-05-27 17:30 GMT+08:00
Subject: Why kylin use MR instead of Tez ?
To: kylin-olap@googlegroups.com
I build my Hadoop env under HDP 2.2 with Hive 0.14 ;
i set in the hive-site.xml conf file default value for the
hive.excution.engine to tez but during some kylin step it use map-reduce :
--
You received this message because you are subscribed to the Google Groups
"Kylin OLAP" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to kylin-olap+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/kylin-olap/3414cb9e-0493-42e6-be68-e122fd5685f3%40googlegroups.com
<https://groups.google.com/d/msgid/kylin-olap/3414cb9e-0493-42e6-be68-e122fd5685f3%40googlegroups.com?utm_medium=email&utm_source=footer>
.
For more options, visit https://groups.google.com/d/optout.
Re: Why kylin use MR instead of Tez ?
Posted by Li Yang <li...@apache.org>.
Kylin will not depend on Tez. If user want to leverage Tez to speed up Hive
processing, he/she can configure Hive to use Tez as the default engine.
On Tue, Jun 9, 2015 at 8:00 AM, Luke Han <lu...@gmail.com> wrote:
> Hi Donald,
> You screenshot not works here, could you please resend it as attend? or
> link to dropbox or somewhere?
>
> My understanding about your question is you found the Job Jobs running
> with MR but few Tez job, is this right?
> And, the first attachment is showing the HFile Generate step which is
> not Hive relative one.
>
> Please let's know more detail.
>
> Thanks.
> Luke
>
>
> Best Regards!
> ---------------------
>
> Luke Han
>
> 2015-06-08 0:17 GMT-07:00 donald fossouo <fo...@gmail.com>:
>
> > Yes i see Map Reduce for Hive Jobs and little Tez jobs only and step with
> > Map Reduce doesn't use the full capacity of the cluster :
> >
> > [image: Images intégrées 1]
> >
> > Donald P. FOSSOUO
> >
> >
> >
> >
> >
> > 2015-06-06 13:35 GMT+02:00 Akash Mishra <ak...@gmail.com>:
> >
> >> Kylin uses Map-Reduce to build Cuboid and other task. In your Resource
> >> manager you can see both Tez jobs and Map Reduce jobs. All Hive job
> should
> >> be using Tez engine.
> >>
> >> Please let us know if you can see Map Reduce being used for Hive jobs.
> >>
> >> Thanks,
> >>
> >>
> >> On Thu, Jun 4, 2015 at 5:34 PM, Luke Han <lu...@gmail.com> wrote:
> >>
> >>> Forward to mailing list for further support.
> >>>
> >>> Thanks.
> >>>
> >>> Best Regards!
> >>> ---------------------
> >>>
> >>> Luke Han
> >>>
> >>> ---------- Forwarded message ----------
> >>> From: donald fossouo <fo...@gmail.com>
> >>> Date: 2015-05-27 17:30 GMT+08:00
> >>> Subject: Why kylin use MR instead of Tez ?
> >>> To: kylin-olap@googlegroups.com
> >>>
> >>>
> >>> I build my Hadoop env under HDP 2.2 with Hive 0.14 ;
> >>>
> >>> i set in the hive-site.xml conf file default value for the
> >>> hive.excution.engine to tez but during some kylin step it use
> map-reduce :
> >>>
> >>>
> >>> --
> >>> You received this message because you are subscribed to the Google
> >>> Groups "Kylin OLAP" group.
> >>> To unsubscribe from this group and stop receiving emails from it, send
> >>> an email to kylin-olap+unsubscribe@googlegroups.com.
> >>> To view this discussion on the web visit
> >>>
> https://groups.google.com/d/msgid/kylin-olap/3414cb9e-0493-42e6-be68-e122fd5685f3%40googlegroups.com
> >>> <
> https://groups.google.com/d/msgid/kylin-olap/3414cb9e-0493-42e6-be68-e122fd5685f3%40googlegroups.com?utm_medium=email&utm_source=footer
> >
> >>> .
> >>> For more options, visit https://groups.google.com/d/optout.
> >>>
> >>>
> >>
> >>
> >> --
> >>
> >> Regards,
> >> Akash Mishra.
> >>
> >>
> >> "Its not our abilities that make us, but our decisions."--Albus
> Dumbledore
> >>
> >
> >
>
Re: Why kylin use MR instead of Tez ?
Posted by Luke Han <lu...@gmail.com>.
Hi Donald,
You screenshot not works here, could you please resend it as attend? or
link to dropbox or somewhere?
My understanding about your question is you found the Job Jobs running
with MR but few Tez job, is this right?
And, the first attachment is showing the HFile Generate step which is
not Hive relative one.
Please let's know more detail.
Thanks.
Luke
Best Regards!
---------------------
Luke Han
2015-06-08 0:17 GMT-07:00 donald fossouo <fo...@gmail.com>:
> Yes i see Map Reduce for Hive Jobs and little Tez jobs only and step with
> Map Reduce doesn't use the full capacity of the cluster :
>
> [image: Images intégrées 1]
>
> Donald P. FOSSOUO
>
>
>
>
>
> 2015-06-06 13:35 GMT+02:00 Akash Mishra <ak...@gmail.com>:
>
>> Kylin uses Map-Reduce to build Cuboid and other task. In your Resource
>> manager you can see both Tez jobs and Map Reduce jobs. All Hive job should
>> be using Tez engine.
>>
>> Please let us know if you can see Map Reduce being used for Hive jobs.
>>
>> Thanks,
>>
>>
>> On Thu, Jun 4, 2015 at 5:34 PM, Luke Han <lu...@gmail.com> wrote:
>>
>>> Forward to mailing list for further support.
>>>
>>> Thanks.
>>>
>>> Best Regards!
>>> ---------------------
>>>
>>> Luke Han
>>>
>>> ---------- Forwarded message ----------
>>> From: donald fossouo <fo...@gmail.com>
>>> Date: 2015-05-27 17:30 GMT+08:00
>>> Subject: Why kylin use MR instead of Tez ?
>>> To: kylin-olap@googlegroups.com
>>>
>>>
>>> I build my Hadoop env under HDP 2.2 with Hive 0.14 ;
>>>
>>> i set in the hive-site.xml conf file default value for the
>>> hive.excution.engine to tez but during some kylin step it use map-reduce :
>>>
>>>
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "Kylin OLAP" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to kylin-olap+unsubscribe@googlegroups.com.
>>> To view this discussion on the web visit
>>> https://groups.google.com/d/msgid/kylin-olap/3414cb9e-0493-42e6-be68-e122fd5685f3%40googlegroups.com
>>> <https://groups.google.com/d/msgid/kylin-olap/3414cb9e-0493-42e6-be68-e122fd5685f3%40googlegroups.com?utm_medium=email&utm_source=footer>
>>> .
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>>
>>
>>
>> --
>>
>> Regards,
>> Akash Mishra.
>>
>>
>> "Its not our abilities that make us, but our decisions."--Albus Dumbledore
>>
>
>
Re: Why kylin use MR instead of Tez ?
Posted by donald fossouo <fo...@gmail.com>.
Yes i see Map Reduce for Hive Jobs and little Tez jobs only and step with
Map Reduce doesn't use the full capacity of the cluster :
[image: Images intégrées 1]
Donald P. FOSSOUO
2015-06-06 13:35 GMT+02:00 Akash Mishra <ak...@gmail.com>:
> Kylin uses Map-Reduce to build Cuboid and other task. In your Resource
> manager you can see both Tez jobs and Map Reduce jobs. All Hive job should
> be using Tez engine.
>
> Please let us know if you can see Map Reduce being used for Hive jobs.
>
> Thanks,
>
>
> On Thu, Jun 4, 2015 at 5:34 PM, Luke Han <lu...@gmail.com> wrote:
>
>> Forward to mailing list for further support.
>>
>> Thanks.
>>
>> Best Regards!
>> ---------------------
>>
>> Luke Han
>>
>> ---------- Forwarded message ----------
>> From: donald fossouo <fo...@gmail.com>
>> Date: 2015-05-27 17:30 GMT+08:00
>> Subject: Why kylin use MR instead of Tez ?
>> To: kylin-olap@googlegroups.com
>>
>>
>> I build my Hadoop env under HDP 2.2 with Hive 0.14 ;
>>
>> i set in the hive-site.xml conf file default value for the
>> hive.excution.engine to tez but during some kylin step it use map-reduce :
>>
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Kylin OLAP" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to kylin-olap+unsubscribe@googlegroups.com.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/kylin-olap/3414cb9e-0493-42e6-be68-e122fd5685f3%40googlegroups.com
>> <https://groups.google.com/d/msgid/kylin-olap/3414cb9e-0493-42e6-be68-e122fd5685f3%40googlegroups.com?utm_medium=email&utm_source=footer>
>> .
>> For more options, visit https://groups.google.com/d/optout.
>>
>>
>
>
> --
>
> Regards,
> Akash Mishra.
>
>
> "Its not our abilities that make us, but our decisions."--Albus Dumbledore
>
Re: Why kylin use MR instead of Tez ?
Posted by Li Yang <li...@apache.org>.
I don't fully get the question. Is it about Kylin should instruct hive to
use Tez, or Kylin should use Tez for its own MR jobs?
On Sat, Jun 6, 2015 at 7:35 PM, Akash Mishra <ak...@gmail.com>
wrote:
> Kylin uses Map-Reduce to build Cuboid and other task. In your Resource
> manager you can see both Tez jobs and Map Reduce jobs. All Hive job should
> be using Tez engine.
>
> Please let us know if you can see Map Reduce being used for Hive jobs.
>
> Thanks,
>
>
> On Thu, Jun 4, 2015 at 5:34 PM, Luke Han <lu...@gmail.com> wrote:
>
> > Forward to mailing list for further support.
> >
> > Thanks.
> >
> > Best Regards!
> > ---------------------
> >
> > Luke Han
> >
> > ---------- Forwarded message ----------
> > From: donald fossouo <fo...@gmail.com>
> > Date: 2015-05-27 17:30 GMT+08:00
> > Subject: Why kylin use MR instead of Tez ?
> > To: kylin-olap@googlegroups.com
> >
> >
> > I build my Hadoop env under HDP 2.2 with Hive 0.14 ;
> >
> > i set in the hive-site.xml conf file default value for the
> > hive.excution.engine to tez but during some kylin step it use map-reduce
> :
> >
> >
> > --
> > You received this message because you are subscribed to the Google Groups
> > "Kylin OLAP" group.
> > To unsubscribe from this group and stop receiving emails from it, send an
> > email to kylin-olap+unsubscribe@googlegroups.com.
> > To view this discussion on the web visit
> >
> https://groups.google.com/d/msgid/kylin-olap/3414cb9e-0493-42e6-be68-e122fd5685f3%40googlegroups.com
> > <
> https://groups.google.com/d/msgid/kylin-olap/3414cb9e-0493-42e6-be68-e122fd5685f3%40googlegroups.com?utm_medium=email&utm_source=footer
> >
> > .
> > For more options, visit https://groups.google.com/d/optout.
> >
> >
>
>
> --
>
> Regards,
> Akash Mishra.
>
>
> "Its not our abilities that make us, but our decisions."--Albus Dumbledore
>
Re: Why kylin use MR instead of Tez ?
Posted by Akash Mishra <ak...@gmail.com>.
Kylin uses Map-Reduce to build Cuboid and other task. In your Resource
manager you can see both Tez jobs and Map Reduce jobs. All Hive job should
be using Tez engine.
Please let us know if you can see Map Reduce being used for Hive jobs.
Thanks,
On Thu, Jun 4, 2015 at 5:34 PM, Luke Han <lu...@gmail.com> wrote:
> Forward to mailing list for further support.
>
> Thanks.
>
> Best Regards!
> ---------------------
>
> Luke Han
>
> ---------- Forwarded message ----------
> From: donald fossouo <fo...@gmail.com>
> Date: 2015-05-27 17:30 GMT+08:00
> Subject: Why kylin use MR instead of Tez ?
> To: kylin-olap@googlegroups.com
>
>
> I build my Hadoop env under HDP 2.2 with Hive 0.14 ;
>
> i set in the hive-site.xml conf file default value for the
> hive.excution.engine to tez but during some kylin step it use map-reduce :
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "Kylin OLAP" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to kylin-olap+unsubscribe@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/kylin-olap/3414cb9e-0493-42e6-be68-e122fd5685f3%40googlegroups.com
> <https://groups.google.com/d/msgid/kylin-olap/3414cb9e-0493-42e6-be68-e122fd5685f3%40googlegroups.com?utm_medium=email&utm_source=footer>
> .
> For more options, visit https://groups.google.com/d/optout.
>
>
--
Regards,
Akash Mishra.
"Its not our abilities that make us, but our decisions."--Albus Dumbledore