You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kylin.apache.org by Jakob Stengård <ja...@gmail.com> on 2015/03/02 13:51:04 UTC
Tez
Hi
Will kylin suport hive+tez in v0.7.x?
I've tried using tez with hive, but currently the cube build fails with
"Can't get cube source record size", because kylin expects certain console
output from map-reduce, and tez doesn't print this.
BG.
Jakob Stengard
Re: Tez
Posted by "Shi, Shaofeng" <sh...@ebay.com>.
From 0.7.1 Kylin support tables from non-default database, it strictly
require the table name in cube definition be <db_name>.<table_name>; If
your table is in default database, please append “default.” ahead of the
table name.
I assume you’re upgrading from 0.6.x to 0.7.1; The metadata or say cubes
created in 0.6.x couldn’t be parsed correctly in 0.7.1; We suggest you
re-create the cubes; You can drop the old metadata table
“kylin_metadata_qa" in hbase or rename it to another name, so Kylin will
not load the old data anymore; Kylin will automatically create the htable
if it doesn’t exist;
On 3/4/15, 5:07 PM, "Jakob Stengård" <ja...@gmail.com> wrote:
>I tried installing 0.7.1, but i get "The 'partition_date_column' format is
>invalid: , it should be {db}.{table}.{column}."
>when trying to set up a cube.
>
>Dunno if it's my tables that it doesn't like or if it's some deprecated
>settings from 0.6.5 that causes this.
>Tried deleting all Hbase tables, but i still get the error after
>restarting
>Kylin and re-syncing the tables.
>
>/J
>
>On Tue, Mar 3, 2015 at 9:24 AM, Li Yang <li...@apache.org> wrote:
>
>> I briefly looked into the code and seems the issue has been fixed in
>>0.7.x
>> (the current inverted-index branch) already. Basically, it does not
>>rely on
>> counters of the hive step any more.
>>
>> However I don't have the env to verify this. @Jakob, may try this latest
>> version http://kylin.incubator.apache.org/download/
>>
>> On Tue, Mar 3, 2015 at 6:59 AM, Jakob Stengård
>><ja...@gmail.com>
>> wrote:
>>
>> > Yeah, i know that.
>> > Here is the jira bug i filed:
>> > https://issues.apache.org/jira/browse/KYLIN-619
>> >
>> > On Mon, Mar 2, 2015 at 11:47 PM, Luke Han <lu...@gmail.com> wrote:
>> >
>> > > Hi Jakob, did you create JIRA for this already?
>> > >
>> > > Tez itself should not the problem but just some statistics
>>information
>> > > Kylin can't parse as you figure out already.
>> > >
>> > > Please let me know the JIRA number and we will enhance that to avoid
>> such
>> > > issue again.
>> > >
>> > > Thanks.
>> > > Luke
>> > >
>> > >
>> > > Best Regards!
>> > > ---------------------
>> > >
>> > > Luke Han
>> > >
>> > > 2015-03-03 2:38 GMT+08:00 Jakob Stengård <ja...@gmail.com>:
>> > >
>> > > > I looked at the source code, but the fix wasn't obvious to me.
>> > > >
>> > > > Maybe, if i have time, i'll look into it further.
>> > > >
>> > > > /J
>> > > >
>> > > >
>> > > > On Mon, Mar 2, 2015 at 7:13 PM, Adunuthula, Seshu <
>> > sadunuthula@ebay.com>
>> > > > wrote:
>> > > >
>> > > > > Thanks Jakob,
>> > > > >
>> > > > > Would be fantastic if you could dig in and submit a patch…
>> > > > >
>> > > > >
>> > > > > On 3/2/15, 10:08 AM, "Jakob Stengård" <ja...@gmail.com>
>> > > wrote:
>> > > > >
>> > > > > >I'll file a bug in jira.
>> > > > > >For your information i also happen to know that tez boosted
>> > map-reduce
>> > > > > >doesn't work either, but there, the problem seems to have to do
>> with
>> > > the
>> > > > > >map reduce jobs them selves failing.
>> > > > > >
>> > > > > >In this case however, from what i've read on git hub about the
>> > "Can't
>> > > > get
>> > > > > >cube source record size",
>> > > > > >error", it could possibly be a problem with that tez doesn't
>> output
>> > a
>> > > > > >string on the form:
>> > > > > >"HDFS Read: (\d+) HDFS Write"
>> > > > > >
>> > > > > >However, it could be something else, it could also be my
>>logging
>> > > > > >configuration for tez.
>> > > > > >
>> > > > > >/J
>> > > > > >
>> > > > > >
>> > > > > >
>> > > > > >On Mon, Mar 2, 2015 at 6:54 PM, Adunuthula, Seshu <
>> > > sadunuthula@ebay.com
>> > > > >
>> > > > > >wrote:
>> > > > > >
>> > > > > >> Did not read your logs fully. The culprit seems to be
>> > > > > >>
>> > > > > >>
>> > > > > >> 1. [QuartzScheduler_Worker-7]:[2015-03-02
>> > > > > >>
>> > > > >
>> > > >
>> > >
>> >
>>
>>>>16:09:24,737][ERROR][com.kylinolap.job.flow.JobFlowListener.handleExcep
>>>>ti
>> > > > > >>on
>> > > > > >> (JobFlowListener.java:312)] - Can't get cube source record
>>size.
>> > > > > >>
>> > > > > >> source_records_size may have to be computed differently when
>> using
>> > > Tez
>> > > > > >>as
>> > > > > >> the Hive engine. I will let Yang/Luke answer this when they
>>come
>> > > back
>> > > > > >> onlineŠ
>> > > > > >>
>> > > > > >>
>> > > > > >>
>> > > > > >>
>> > > > > >> On 3/2/15, 9:30 AM, "Adunuthula, Seshu"
>><sa...@ebay.com>
>> > > wrote:
>> > > > > >>
>> > > > > >> >Jakob,
>> > > > > >> >
>> > > > > >> >Could you go ahead and file a JIRA with the stack trace at
>> > > > > >> >https://issues.apache.org/jira/browse/KYLIN/
>> > > > > >> >
>> > > > > >> >Also include the App Master and Map/Reduce Task logs for the
>> > Tasks
>> > > > that
>> > > > > >> >failed.
>> > > > > >> >
>> > > > > >> >Regards
>> > > > > >> >Seshu
>> > > > > >> >
>> > > > > >> >On 3/2/15, 8:12 AM, "Jakob Stengård"
>><jakob.stengard@gmail.com
>> >
>> > > > wrote:
>> > > > > >> >
>> > > > > >> >>Correction: That, isn't the last job, it's step 9.
>> > > > > >> >>
>> > > > > >> >>/J
>> > > > > >> >>
>> > > > > >> >>On Mon, Mar 2, 2015 at 5:10 PM, Jakob Stengård
>> > > > > >><jakob.stengard@gmail.com
>> > > > > >> >
>> > > > > >> >>wrote:
>> > > > > >> >>
>> > > > > >> >>> Well, this only happens with
>> > > > > >> >>> <property>
>> > > > > >> >>> <name>hive.execution.engine</name>
>> > > > > >> >>> <value>tez</value>
>> > > > > >> >>> </property>
>> > > > > >> >>>
>> > > > > >> >>> If i use mr, it works fine.
>> > > > > >> >>>
>> > > > > >> >>>
>> > > > > >> >>> From what i can see form the job log, none of the
>>previous
>> > > > > >>map-reduce
>> > > > > >> >>>jobs
>> > > > > >> >>> fail, but the build fails at the last step.
>> > > > > >> >>> Log from the last mr job for example:
>> > > > > >> >>> http://pastebin.com/jNWSRKLi
>> > > > > >> >>>
>> > > > > >> >>>
>> > > > > >> >>> Is tez supposed to work? Because in that case, maybe it's
>> just
>> > > my
>> > > > > >> >>>setup.
>> > > > > >> >>>
>> > > > > >> >>> /Jakob
>> > > > > >> >>>
>> > > > > >> >>>
>> > > > > >> >>>
>> > > > > >> >>> On Mon, Mar 2, 2015 at 4:34 PM, Santosh Akhilesh <
>> > > > > >> >>> santoshakhilesh@gmail.com> wrote:
>> > > > > >> >>>
>> > > > > >> >>>> This doesn't seems to be complete kylin.log. Do you have
>> any
>> > > > > >>exception
>> > > > > >> >>>> related to MR counter in map reduce logs? If yes then
>>this
>> > > needs
>> > > > > >>to be
>> > > > > >> >>>> checked because kylin will need the MR counter to
>>calculate
>> > the
>> > > > > >>cube
>> > > > > >> >>>>size.
>> > > > > >> >>>> On Mon, 2 Mar 2015 at 8:49 pm, Jakob Stengård
>> > > > > >> >>>><ja...@gmail.com>
>> > > > > >> >>>> wrote:
>> > > > > >> >>>>
>> > > > > >> >>>> > It breaks on the last step: "Load HFile to HBase
>>Table":
>> > > > > >> >>>> >
>> > > > > >> >>>> > Kylin.log:
>> > > > > >> >>>> > http://pastebin.com/3kAQmJ3u
>> > > > > >> >>>> >
>> > > > > >> >>>> > On Mon, Mar 2, 2015 at 2:27 PM, Adunuthula, Seshu
>> > > > > >> >>>><sadunuthula@ebay.com
>> > > > > >> >>>> >
>> > > > > >> >>>> > wrote:
>> > > > > >> >>>> >
>> > > > > >> >>>> > > Jakob,
>> > > > > >> >>>> > >
>> > > > > >> >>>> > > Could you please send out the complete stack trace
>>with
>> > Tez
>> > > > > >> >>>>enabled?
>> > > > > >> >>>> > >
>> > > > > >> >>>> > > Regards
>> > > > > >> >>>> > > Seshu
>> > > > > >> >>>> > >
>> > > > > >> >>>> > > On 3/2/15, 4:51 AM, "Jakob Stengård" <
>> > > > jakob.stengard@gmail.com
>> > > > > >
>> > > > > >> >>>> wrote:
>> > > > > >> >>>> > >
>> > > > > >> >>>> > > >Hi
>> > > > > >> >>>> > > >Will kylin suport hive+tez in v0.7.x?
>> > > > > >> >>>> > > >
>> > > > > >> >>>> > > >I've tried using tez with hive, but currently the
>>cube
>> > > build
>> > > > > >> >>>>fails
>> > > > > >> >>>> with
>> > > > > >> >>>> > > >"Can't get cube source record size", because kylin
>> > expects
>> > > > > >> >>>>certain
>> > > > > >> >>>> > console
>> > > > > >> >>>> > > >output from map-reduce, and tez doesn't print this.
>> > > > > >> >>>> > > >
>> > > > > >> >>>> > > >BG.
>> > > > > >> >>>> > > >Jakob Stengard
>> > > > > >> >>>> > >
>> > > > > >> >>>> > >
>> > > > > >> >>>> >
>> > > > > >> >>>>
>> > > > > >> >>>
>> > > > > >> >>>
>> > > > > >> >
>> > > > > >>
>> > > > > >>
>> > > > >
>> > > > >
>> > > >
>> > >
>> >
>>
>
>
>
>--
>
>Med vänlig hälsning (Best Regards)
>*Jakob Stengård*
Re: Tez
Posted by Jakob Stengård <ja...@gmail.com>.
I tried installing 0.7.1, but i get "The 'partition_date_column' format is
invalid: , it should be {db}.{table}.{column}."
when trying to set up a cube.
Dunno if it's my tables that it doesn't like or if it's some deprecated
settings from 0.6.5 that causes this.
Tried deleting all Hbase tables, but i still get the error after restarting
Kylin and re-syncing the tables.
/J
On Tue, Mar 3, 2015 at 9:24 AM, Li Yang <li...@apache.org> wrote:
> I briefly looked into the code and seems the issue has been fixed in 0.7.x
> (the current inverted-index branch) already. Basically, it does not rely on
> counters of the hive step any more.
>
> However I don't have the env to verify this. @Jakob, may try this latest
> version http://kylin.incubator.apache.org/download/
>
> On Tue, Mar 3, 2015 at 6:59 AM, Jakob Stengård <ja...@gmail.com>
> wrote:
>
> > Yeah, i know that.
> > Here is the jira bug i filed:
> > https://issues.apache.org/jira/browse/KYLIN-619
> >
> > On Mon, Mar 2, 2015 at 11:47 PM, Luke Han <lu...@gmail.com> wrote:
> >
> > > Hi Jakob, did you create JIRA for this already?
> > >
> > > Tez itself should not the problem but just some statistics information
> > > Kylin can't parse as you figure out already.
> > >
> > > Please let me know the JIRA number and we will enhance that to avoid
> such
> > > issue again.
> > >
> > > Thanks.
> > > Luke
> > >
> > >
> > > Best Regards!
> > > ---------------------
> > >
> > > Luke Han
> > >
> > > 2015-03-03 2:38 GMT+08:00 Jakob Stengård <ja...@gmail.com>:
> > >
> > > > I looked at the source code, but the fix wasn't obvious to me.
> > > >
> > > > Maybe, if i have time, i'll look into it further.
> > > >
> > > > /J
> > > >
> > > >
> > > > On Mon, Mar 2, 2015 at 7:13 PM, Adunuthula, Seshu <
> > sadunuthula@ebay.com>
> > > > wrote:
> > > >
> > > > > Thanks Jakob,
> > > > >
> > > > > Would be fantastic if you could dig in and submit a patch…
> > > > >
> > > > >
> > > > > On 3/2/15, 10:08 AM, "Jakob Stengård" <ja...@gmail.com>
> > > wrote:
> > > > >
> > > > > >I'll file a bug in jira.
> > > > > >For your information i also happen to know that tez boosted
> > map-reduce
> > > > > >doesn't work either, but there, the problem seems to have to do
> with
> > > the
> > > > > >map reduce jobs them selves failing.
> > > > > >
> > > > > >In this case however, from what i've read on git hub about the
> > "Can't
> > > > get
> > > > > >cube source record size",
> > > > > >error", it could possibly be a problem with that tez doesn't
> output
> > a
> > > > > >string on the form:
> > > > > >"HDFS Read: (\d+) HDFS Write"
> > > > > >
> > > > > >However, it could be something else, it could also be my logging
> > > > > >configuration for tez.
> > > > > >
> > > > > >/J
> > > > > >
> > > > > >
> > > > > >
> > > > > >On Mon, Mar 2, 2015 at 6:54 PM, Adunuthula, Seshu <
> > > sadunuthula@ebay.com
> > > > >
> > > > > >wrote:
> > > > > >
> > > > > >> Did not read your logs fully. The culprit seems to be
> > > > > >>
> > > > > >>
> > > > > >> 1. [QuartzScheduler_Worker-7]:[2015-03-02
> > > > > >>
> > > > >
> > > >
> > >
> >
> >>16:09:24,737][ERROR][com.kylinolap.job.flow.JobFlowListener.handleExcepti
> > > > > >>on
> > > > > >> (JobFlowListener.java:312)] - Can't get cube source record size.
> > > > > >>
> > > > > >> source_records_size may have to be computed differently when
> using
> > > Tez
> > > > > >>as
> > > > > >> the Hive engine. I will let Yang/Luke answer this when they come
> > > back
> > > > > >> onlineŠ
> > > > > >>
> > > > > >>
> > > > > >>
> > > > > >>
> > > > > >> On 3/2/15, 9:30 AM, "Adunuthula, Seshu" <sa...@ebay.com>
> > > wrote:
> > > > > >>
> > > > > >> >Jakob,
> > > > > >> >
> > > > > >> >Could you go ahead and file a JIRA with the stack trace at
> > > > > >> >https://issues.apache.org/jira/browse/KYLIN/
> > > > > >> >
> > > > > >> >Also include the App Master and Map/Reduce Task logs for the
> > Tasks
> > > > that
> > > > > >> >failed.
> > > > > >> >
> > > > > >> >Regards
> > > > > >> >Seshu
> > > > > >> >
> > > > > >> >On 3/2/15, 8:12 AM, "Jakob Stengård" <jakob.stengard@gmail.com
> >
> > > > wrote:
> > > > > >> >
> > > > > >> >>Correction: That, isn't the last job, it's step 9.
> > > > > >> >>
> > > > > >> >>/J
> > > > > >> >>
> > > > > >> >>On Mon, Mar 2, 2015 at 5:10 PM, Jakob Stengård
> > > > > >><jakob.stengard@gmail.com
> > > > > >> >
> > > > > >> >>wrote:
> > > > > >> >>
> > > > > >> >>> Well, this only happens with
> > > > > >> >>> <property>
> > > > > >> >>> <name>hive.execution.engine</name>
> > > > > >> >>> <value>tez</value>
> > > > > >> >>> </property>
> > > > > >> >>>
> > > > > >> >>> If i use mr, it works fine.
> > > > > >> >>>
> > > > > >> >>>
> > > > > >> >>> From what i can see form the job log, none of the previous
> > > > > >>map-reduce
> > > > > >> >>>jobs
> > > > > >> >>> fail, but the build fails at the last step.
> > > > > >> >>> Log from the last mr job for example:
> > > > > >> >>> http://pastebin.com/jNWSRKLi
> > > > > >> >>>
> > > > > >> >>>
> > > > > >> >>> Is tez supposed to work? Because in that case, maybe it's
> just
> > > my
> > > > > >> >>>setup.
> > > > > >> >>>
> > > > > >> >>> /Jakob
> > > > > >> >>>
> > > > > >> >>>
> > > > > >> >>>
> > > > > >> >>> On Mon, Mar 2, 2015 at 4:34 PM, Santosh Akhilesh <
> > > > > >> >>> santoshakhilesh@gmail.com> wrote:
> > > > > >> >>>
> > > > > >> >>>> This doesn't seems to be complete kylin.log. Do you have
> any
> > > > > >>exception
> > > > > >> >>>> related to MR counter in map reduce logs? If yes then this
> > > needs
> > > > > >>to be
> > > > > >> >>>> checked because kylin will need the MR counter to calculate
> > the
> > > > > >>cube
> > > > > >> >>>>size.
> > > > > >> >>>> On Mon, 2 Mar 2015 at 8:49 pm, Jakob Stengård
> > > > > >> >>>><ja...@gmail.com>
> > > > > >> >>>> wrote:
> > > > > >> >>>>
> > > > > >> >>>> > It breaks on the last step: "Load HFile to HBase Table":
> > > > > >> >>>> >
> > > > > >> >>>> > Kylin.log:
> > > > > >> >>>> > http://pastebin.com/3kAQmJ3u
> > > > > >> >>>> >
> > > > > >> >>>> > On Mon, Mar 2, 2015 at 2:27 PM, Adunuthula, Seshu
> > > > > >> >>>><sadunuthula@ebay.com
> > > > > >> >>>> >
> > > > > >> >>>> > wrote:
> > > > > >> >>>> >
> > > > > >> >>>> > > Jakob,
> > > > > >> >>>> > >
> > > > > >> >>>> > > Could you please send out the complete stack trace with
> > Tez
> > > > > >> >>>>enabled?
> > > > > >> >>>> > >
> > > > > >> >>>> > > Regards
> > > > > >> >>>> > > Seshu
> > > > > >> >>>> > >
> > > > > >> >>>> > > On 3/2/15, 4:51 AM, "Jakob Stengård" <
> > > > jakob.stengard@gmail.com
> > > > > >
> > > > > >> >>>> wrote:
> > > > > >> >>>> > >
> > > > > >> >>>> > > >Hi
> > > > > >> >>>> > > >Will kylin suport hive+tez in v0.7.x?
> > > > > >> >>>> > > >
> > > > > >> >>>> > > >I've tried using tez with hive, but currently the cube
> > > build
> > > > > >> >>>>fails
> > > > > >> >>>> with
> > > > > >> >>>> > > >"Can't get cube source record size", because kylin
> > expects
> > > > > >> >>>>certain
> > > > > >> >>>> > console
> > > > > >> >>>> > > >output from map-reduce, and tez doesn't print this.
> > > > > >> >>>> > > >
> > > > > >> >>>> > > >BG.
> > > > > >> >>>> > > >Jakob Stengard
> > > > > >> >>>> > >
> > > > > >> >>>> > >
> > > > > >> >>>> >
> > > > > >> >>>>
> > > > > >> >>>
> > > > > >> >>>
> > > > > >> >
> > > > > >>
> > > > > >>
> > > > >
> > > > >
> > > >
> > >
> >
>
--
Med vänlig hälsning (Best Regards)
*Jakob Stengård*
Re: Tez
Posted by Li Yang <li...@apache.org>.
I briefly looked into the code and seems the issue has been fixed in 0.7.x
(the current inverted-index branch) already. Basically, it does not rely on
counters of the hive step any more.
However I don't have the env to verify this. @Jakob, may try this latest
version http://kylin.incubator.apache.org/download/
On Tue, Mar 3, 2015 at 6:59 AM, Jakob Stengård <ja...@gmail.com>
wrote:
> Yeah, i know that.
> Here is the jira bug i filed:
> https://issues.apache.org/jira/browse/KYLIN-619
>
> On Mon, Mar 2, 2015 at 11:47 PM, Luke Han <lu...@gmail.com> wrote:
>
> > Hi Jakob, did you create JIRA for this already?
> >
> > Tez itself should not the problem but just some statistics information
> > Kylin can't parse as you figure out already.
> >
> > Please let me know the JIRA number and we will enhance that to avoid such
> > issue again.
> >
> > Thanks.
> > Luke
> >
> >
> > Best Regards!
> > ---------------------
> >
> > Luke Han
> >
> > 2015-03-03 2:38 GMT+08:00 Jakob Stengård <ja...@gmail.com>:
> >
> > > I looked at the source code, but the fix wasn't obvious to me.
> > >
> > > Maybe, if i have time, i'll look into it further.
> > >
> > > /J
> > >
> > >
> > > On Mon, Mar 2, 2015 at 7:13 PM, Adunuthula, Seshu <
> sadunuthula@ebay.com>
> > > wrote:
> > >
> > > > Thanks Jakob,
> > > >
> > > > Would be fantastic if you could dig in and submit a patch…
> > > >
> > > >
> > > > On 3/2/15, 10:08 AM, "Jakob Stengård" <ja...@gmail.com>
> > wrote:
> > > >
> > > > >I'll file a bug in jira.
> > > > >For your information i also happen to know that tez boosted
> map-reduce
> > > > >doesn't work either, but there, the problem seems to have to do with
> > the
> > > > >map reduce jobs them selves failing.
> > > > >
> > > > >In this case however, from what i've read on git hub about the
> "Can't
> > > get
> > > > >cube source record size",
> > > > >error", it could possibly be a problem with that tez doesn't output
> a
> > > > >string on the form:
> > > > >"HDFS Read: (\d+) HDFS Write"
> > > > >
> > > > >However, it could be something else, it could also be my logging
> > > > >configuration for tez.
> > > > >
> > > > >/J
> > > > >
> > > > >
> > > > >
> > > > >On Mon, Mar 2, 2015 at 6:54 PM, Adunuthula, Seshu <
> > sadunuthula@ebay.com
> > > >
> > > > >wrote:
> > > > >
> > > > >> Did not read your logs fully. The culprit seems to be
> > > > >>
> > > > >>
> > > > >> 1. [QuartzScheduler_Worker-7]:[2015-03-02
> > > > >>
> > > >
> > >
> >
> >>16:09:24,737][ERROR][com.kylinolap.job.flow.JobFlowListener.handleExcepti
> > > > >>on
> > > > >> (JobFlowListener.java:312)] - Can't get cube source record size.
> > > > >>
> > > > >> source_records_size may have to be computed differently when using
> > Tez
> > > > >>as
> > > > >> the Hive engine. I will let Yang/Luke answer this when they come
> > back
> > > > >> onlineŠ
> > > > >>
> > > > >>
> > > > >>
> > > > >>
> > > > >> On 3/2/15, 9:30 AM, "Adunuthula, Seshu" <sa...@ebay.com>
> > wrote:
> > > > >>
> > > > >> >Jakob,
> > > > >> >
> > > > >> >Could you go ahead and file a JIRA with the stack trace at
> > > > >> >https://issues.apache.org/jira/browse/KYLIN/
> > > > >> >
> > > > >> >Also include the App Master and Map/Reduce Task logs for the
> Tasks
> > > that
> > > > >> >failed.
> > > > >> >
> > > > >> >Regards
> > > > >> >Seshu
> > > > >> >
> > > > >> >On 3/2/15, 8:12 AM, "Jakob Stengård" <ja...@gmail.com>
> > > wrote:
> > > > >> >
> > > > >> >>Correction: That, isn't the last job, it's step 9.
> > > > >> >>
> > > > >> >>/J
> > > > >> >>
> > > > >> >>On Mon, Mar 2, 2015 at 5:10 PM, Jakob Stengård
> > > > >><jakob.stengard@gmail.com
> > > > >> >
> > > > >> >>wrote:
> > > > >> >>
> > > > >> >>> Well, this only happens with
> > > > >> >>> <property>
> > > > >> >>> <name>hive.execution.engine</name>
> > > > >> >>> <value>tez</value>
> > > > >> >>> </property>
> > > > >> >>>
> > > > >> >>> If i use mr, it works fine.
> > > > >> >>>
> > > > >> >>>
> > > > >> >>> From what i can see form the job log, none of the previous
> > > > >>map-reduce
> > > > >> >>>jobs
> > > > >> >>> fail, but the build fails at the last step.
> > > > >> >>> Log from the last mr job for example:
> > > > >> >>> http://pastebin.com/jNWSRKLi
> > > > >> >>>
> > > > >> >>>
> > > > >> >>> Is tez supposed to work? Because in that case, maybe it's just
> > my
> > > > >> >>>setup.
> > > > >> >>>
> > > > >> >>> /Jakob
> > > > >> >>>
> > > > >> >>>
> > > > >> >>>
> > > > >> >>> On Mon, Mar 2, 2015 at 4:34 PM, Santosh Akhilesh <
> > > > >> >>> santoshakhilesh@gmail.com> wrote:
> > > > >> >>>
> > > > >> >>>> This doesn't seems to be complete kylin.log. Do you have any
> > > > >>exception
> > > > >> >>>> related to MR counter in map reduce logs? If yes then this
> > needs
> > > > >>to be
> > > > >> >>>> checked because kylin will need the MR counter to calculate
> the
> > > > >>cube
> > > > >> >>>>size.
> > > > >> >>>> On Mon, 2 Mar 2015 at 8:49 pm, Jakob Stengård
> > > > >> >>>><ja...@gmail.com>
> > > > >> >>>> wrote:
> > > > >> >>>>
> > > > >> >>>> > It breaks on the last step: "Load HFile to HBase Table":
> > > > >> >>>> >
> > > > >> >>>> > Kylin.log:
> > > > >> >>>> > http://pastebin.com/3kAQmJ3u
> > > > >> >>>> >
> > > > >> >>>> > On Mon, Mar 2, 2015 at 2:27 PM, Adunuthula, Seshu
> > > > >> >>>><sadunuthula@ebay.com
> > > > >> >>>> >
> > > > >> >>>> > wrote:
> > > > >> >>>> >
> > > > >> >>>> > > Jakob,
> > > > >> >>>> > >
> > > > >> >>>> > > Could you please send out the complete stack trace with
> Tez
> > > > >> >>>>enabled?
> > > > >> >>>> > >
> > > > >> >>>> > > Regards
> > > > >> >>>> > > Seshu
> > > > >> >>>> > >
> > > > >> >>>> > > On 3/2/15, 4:51 AM, "Jakob Stengård" <
> > > jakob.stengard@gmail.com
> > > > >
> > > > >> >>>> wrote:
> > > > >> >>>> > >
> > > > >> >>>> > > >Hi
> > > > >> >>>> > > >Will kylin suport hive+tez in v0.7.x?
> > > > >> >>>> > > >
> > > > >> >>>> > > >I've tried using tez with hive, but currently the cube
> > build
> > > > >> >>>>fails
> > > > >> >>>> with
> > > > >> >>>> > > >"Can't get cube source record size", because kylin
> expects
> > > > >> >>>>certain
> > > > >> >>>> > console
> > > > >> >>>> > > >output from map-reduce, and tez doesn't print this.
> > > > >> >>>> > > >
> > > > >> >>>> > > >BG.
> > > > >> >>>> > > >Jakob Stengard
> > > > >> >>>> > >
> > > > >> >>>> > >
> > > > >> >>>> >
> > > > >> >>>>
> > > > >> >>>
> > > > >> >>>
> > > > >> >
> > > > >>
> > > > >>
> > > >
> > > >
> > >
> >
>
Re: Tez
Posted by Jakob Stengård <ja...@gmail.com>.
Yeah, i know that.
Here is the jira bug i filed:
https://issues.apache.org/jira/browse/KYLIN-619
On Mon, Mar 2, 2015 at 11:47 PM, Luke Han <lu...@gmail.com> wrote:
> Hi Jakob, did you create JIRA for this already?
>
> Tez itself should not the problem but just some statistics information
> Kylin can't parse as you figure out already.
>
> Please let me know the JIRA number and we will enhance that to avoid such
> issue again.
>
> Thanks.
> Luke
>
>
> Best Regards!
> ---------------------
>
> Luke Han
>
> 2015-03-03 2:38 GMT+08:00 Jakob Stengård <ja...@gmail.com>:
>
> > I looked at the source code, but the fix wasn't obvious to me.
> >
> > Maybe, if i have time, i'll look into it further.
> >
> > /J
> >
> >
> > On Mon, Mar 2, 2015 at 7:13 PM, Adunuthula, Seshu <sa...@ebay.com>
> > wrote:
> >
> > > Thanks Jakob,
> > >
> > > Would be fantastic if you could dig in and submit a patch…
> > >
> > >
> > > On 3/2/15, 10:08 AM, "Jakob Stengård" <ja...@gmail.com>
> wrote:
> > >
> > > >I'll file a bug in jira.
> > > >For your information i also happen to know that tez boosted map-reduce
> > > >doesn't work either, but there, the problem seems to have to do with
> the
> > > >map reduce jobs them selves failing.
> > > >
> > > >In this case however, from what i've read on git hub about the "Can't
> > get
> > > >cube source record size",
> > > >error", it could possibly be a problem with that tez doesn't output a
> > > >string on the form:
> > > >"HDFS Read: (\d+) HDFS Write"
> > > >
> > > >However, it could be something else, it could also be my logging
> > > >configuration for tez.
> > > >
> > > >/J
> > > >
> > > >
> > > >
> > > >On Mon, Mar 2, 2015 at 6:54 PM, Adunuthula, Seshu <
> sadunuthula@ebay.com
> > >
> > > >wrote:
> > > >
> > > >> Did not read your logs fully. The culprit seems to be
> > > >>
> > > >>
> > > >> 1. [QuartzScheduler_Worker-7]:[2015-03-02
> > > >>
> > >
> >
> >>16:09:24,737][ERROR][com.kylinolap.job.flow.JobFlowListener.handleExcepti
> > > >>on
> > > >> (JobFlowListener.java:312)] - Can't get cube source record size.
> > > >>
> > > >> source_records_size may have to be computed differently when using
> Tez
> > > >>as
> > > >> the Hive engine. I will let Yang/Luke answer this when they come
> back
> > > >> onlineŠ
> > > >>
> > > >>
> > > >>
> > > >>
> > > >> On 3/2/15, 9:30 AM, "Adunuthula, Seshu" <sa...@ebay.com>
> wrote:
> > > >>
> > > >> >Jakob,
> > > >> >
> > > >> >Could you go ahead and file a JIRA with the stack trace at
> > > >> >https://issues.apache.org/jira/browse/KYLIN/
> > > >> >
> > > >> >Also include the App Master and Map/Reduce Task logs for the Tasks
> > that
> > > >> >failed.
> > > >> >
> > > >> >Regards
> > > >> >Seshu
> > > >> >
> > > >> >On 3/2/15, 8:12 AM, "Jakob Stengård" <ja...@gmail.com>
> > wrote:
> > > >> >
> > > >> >>Correction: That, isn't the last job, it's step 9.
> > > >> >>
> > > >> >>/J
> > > >> >>
> > > >> >>On Mon, Mar 2, 2015 at 5:10 PM, Jakob Stengård
> > > >><jakob.stengard@gmail.com
> > > >> >
> > > >> >>wrote:
> > > >> >>
> > > >> >>> Well, this only happens with
> > > >> >>> <property>
> > > >> >>> <name>hive.execution.engine</name>
> > > >> >>> <value>tez</value>
> > > >> >>> </property>
> > > >> >>>
> > > >> >>> If i use mr, it works fine.
> > > >> >>>
> > > >> >>>
> > > >> >>> From what i can see form the job log, none of the previous
> > > >>map-reduce
> > > >> >>>jobs
> > > >> >>> fail, but the build fails at the last step.
> > > >> >>> Log from the last mr job for example:
> > > >> >>> http://pastebin.com/jNWSRKLi
> > > >> >>>
> > > >> >>>
> > > >> >>> Is tez supposed to work? Because in that case, maybe it's just
> my
> > > >> >>>setup.
> > > >> >>>
> > > >> >>> /Jakob
> > > >> >>>
> > > >> >>>
> > > >> >>>
> > > >> >>> On Mon, Mar 2, 2015 at 4:34 PM, Santosh Akhilesh <
> > > >> >>> santoshakhilesh@gmail.com> wrote:
> > > >> >>>
> > > >> >>>> This doesn't seems to be complete kylin.log. Do you have any
> > > >>exception
> > > >> >>>> related to MR counter in map reduce logs? If yes then this
> needs
> > > >>to be
> > > >> >>>> checked because kylin will need the MR counter to calculate the
> > > >>cube
> > > >> >>>>size.
> > > >> >>>> On Mon, 2 Mar 2015 at 8:49 pm, Jakob Stengård
> > > >> >>>><ja...@gmail.com>
> > > >> >>>> wrote:
> > > >> >>>>
> > > >> >>>> > It breaks on the last step: "Load HFile to HBase Table":
> > > >> >>>> >
> > > >> >>>> > Kylin.log:
> > > >> >>>> > http://pastebin.com/3kAQmJ3u
> > > >> >>>> >
> > > >> >>>> > On Mon, Mar 2, 2015 at 2:27 PM, Adunuthula, Seshu
> > > >> >>>><sadunuthula@ebay.com
> > > >> >>>> >
> > > >> >>>> > wrote:
> > > >> >>>> >
> > > >> >>>> > > Jakob,
> > > >> >>>> > >
> > > >> >>>> > > Could you please send out the complete stack trace with Tez
> > > >> >>>>enabled?
> > > >> >>>> > >
> > > >> >>>> > > Regards
> > > >> >>>> > > Seshu
> > > >> >>>> > >
> > > >> >>>> > > On 3/2/15, 4:51 AM, "Jakob Stengård" <
> > jakob.stengard@gmail.com
> > > >
> > > >> >>>> wrote:
> > > >> >>>> > >
> > > >> >>>> > > >Hi
> > > >> >>>> > > >Will kylin suport hive+tez in v0.7.x?
> > > >> >>>> > > >
> > > >> >>>> > > >I've tried using tez with hive, but currently the cube
> build
> > > >> >>>>fails
> > > >> >>>> with
> > > >> >>>> > > >"Can't get cube source record size", because kylin expects
> > > >> >>>>certain
> > > >> >>>> > console
> > > >> >>>> > > >output from map-reduce, and tez doesn't print this.
> > > >> >>>> > > >
> > > >> >>>> > > >BG.
> > > >> >>>> > > >Jakob Stengard
> > > >> >>>> > >
> > > >> >>>> > >
> > > >> >>>> >
> > > >> >>>>
> > > >> >>>
> > > >> >>>
> > > >> >
> > > >>
> > > >>
> > >
> > >
> >
>
Re: Tez
Posted by Luke Han <lu...@gmail.com>.
Hi Jakob, did you create JIRA for this already?
Tez itself should not the problem but just some statistics information
Kylin can't parse as you figure out already.
Please let me know the JIRA number and we will enhance that to avoid such
issue again.
Thanks.
Luke
Best Regards!
---------------------
Luke Han
2015-03-03 2:38 GMT+08:00 Jakob Stengård <ja...@gmail.com>:
> I looked at the source code, but the fix wasn't obvious to me.
>
> Maybe, if i have time, i'll look into it further.
>
> /J
>
>
> On Mon, Mar 2, 2015 at 7:13 PM, Adunuthula, Seshu <sa...@ebay.com>
> wrote:
>
> > Thanks Jakob,
> >
> > Would be fantastic if you could dig in and submit a patch…
> >
> >
> > On 3/2/15, 10:08 AM, "Jakob Stengård" <ja...@gmail.com> wrote:
> >
> > >I'll file a bug in jira.
> > >For your information i also happen to know that tez boosted map-reduce
> > >doesn't work either, but there, the problem seems to have to do with the
> > >map reduce jobs them selves failing.
> > >
> > >In this case however, from what i've read on git hub about the "Can't
> get
> > >cube source record size",
> > >error", it could possibly be a problem with that tez doesn't output a
> > >string on the form:
> > >"HDFS Read: (\d+) HDFS Write"
> > >
> > >However, it could be something else, it could also be my logging
> > >configuration for tez.
> > >
> > >/J
> > >
> > >
> > >
> > >On Mon, Mar 2, 2015 at 6:54 PM, Adunuthula, Seshu <sadunuthula@ebay.com
> >
> > >wrote:
> > >
> > >> Did not read your logs fully. The culprit seems to be
> > >>
> > >>
> > >> 1. [QuartzScheduler_Worker-7]:[2015-03-02
> > >>
> >
> >>16:09:24,737][ERROR][com.kylinolap.job.flow.JobFlowListener.handleExcepti
> > >>on
> > >> (JobFlowListener.java:312)] - Can't get cube source record size.
> > >>
> > >> source_records_size may have to be computed differently when using Tez
> > >>as
> > >> the Hive engine. I will let Yang/Luke answer this when they come back
> > >> onlineŠ
> > >>
> > >>
> > >>
> > >>
> > >> On 3/2/15, 9:30 AM, "Adunuthula, Seshu" <sa...@ebay.com> wrote:
> > >>
> > >> >Jakob,
> > >> >
> > >> >Could you go ahead and file a JIRA with the stack trace at
> > >> >https://issues.apache.org/jira/browse/KYLIN/
> > >> >
> > >> >Also include the App Master and Map/Reduce Task logs for the Tasks
> that
> > >> >failed.
> > >> >
> > >> >Regards
> > >> >Seshu
> > >> >
> > >> >On 3/2/15, 8:12 AM, "Jakob Stengård" <ja...@gmail.com>
> wrote:
> > >> >
> > >> >>Correction: That, isn't the last job, it's step 9.
> > >> >>
> > >> >>/J
> > >> >>
> > >> >>On Mon, Mar 2, 2015 at 5:10 PM, Jakob Stengård
> > >><jakob.stengard@gmail.com
> > >> >
> > >> >>wrote:
> > >> >>
> > >> >>> Well, this only happens with
> > >> >>> <property>
> > >> >>> <name>hive.execution.engine</name>
> > >> >>> <value>tez</value>
> > >> >>> </property>
> > >> >>>
> > >> >>> If i use mr, it works fine.
> > >> >>>
> > >> >>>
> > >> >>> From what i can see form the job log, none of the previous
> > >>map-reduce
> > >> >>>jobs
> > >> >>> fail, but the build fails at the last step.
> > >> >>> Log from the last mr job for example:
> > >> >>> http://pastebin.com/jNWSRKLi
> > >> >>>
> > >> >>>
> > >> >>> Is tez supposed to work? Because in that case, maybe it's just my
> > >> >>>setup.
> > >> >>>
> > >> >>> /Jakob
> > >> >>>
> > >> >>>
> > >> >>>
> > >> >>> On Mon, Mar 2, 2015 at 4:34 PM, Santosh Akhilesh <
> > >> >>> santoshakhilesh@gmail.com> wrote:
> > >> >>>
> > >> >>>> This doesn't seems to be complete kylin.log. Do you have any
> > >>exception
> > >> >>>> related to MR counter in map reduce logs? If yes then this needs
> > >>to be
> > >> >>>> checked because kylin will need the MR counter to calculate the
> > >>cube
> > >> >>>>size.
> > >> >>>> On Mon, 2 Mar 2015 at 8:49 pm, Jakob Stengård
> > >> >>>><ja...@gmail.com>
> > >> >>>> wrote:
> > >> >>>>
> > >> >>>> > It breaks on the last step: "Load HFile to HBase Table":
> > >> >>>> >
> > >> >>>> > Kylin.log:
> > >> >>>> > http://pastebin.com/3kAQmJ3u
> > >> >>>> >
> > >> >>>> > On Mon, Mar 2, 2015 at 2:27 PM, Adunuthula, Seshu
> > >> >>>><sadunuthula@ebay.com
> > >> >>>> >
> > >> >>>> > wrote:
> > >> >>>> >
> > >> >>>> > > Jakob,
> > >> >>>> > >
> > >> >>>> > > Could you please send out the complete stack trace with Tez
> > >> >>>>enabled?
> > >> >>>> > >
> > >> >>>> > > Regards
> > >> >>>> > > Seshu
> > >> >>>> > >
> > >> >>>> > > On 3/2/15, 4:51 AM, "Jakob Stengård" <
> jakob.stengard@gmail.com
> > >
> > >> >>>> wrote:
> > >> >>>> > >
> > >> >>>> > > >Hi
> > >> >>>> > > >Will kylin suport hive+tez in v0.7.x?
> > >> >>>> > > >
> > >> >>>> > > >I've tried using tez with hive, but currently the cube build
> > >> >>>>fails
> > >> >>>> with
> > >> >>>> > > >"Can't get cube source record size", because kylin expects
> > >> >>>>certain
> > >> >>>> > console
> > >> >>>> > > >output from map-reduce, and tez doesn't print this.
> > >> >>>> > > >
> > >> >>>> > > >BG.
> > >> >>>> > > >Jakob Stengard
> > >> >>>> > >
> > >> >>>> > >
> > >> >>>> >
> > >> >>>>
> > >> >>>
> > >> >>>
> > >> >
> > >>
> > >>
> >
> >
>
Re: Tez
Posted by Jakob Stengård <ja...@gmail.com>.
I looked at the source code, but the fix wasn't obvious to me.
Maybe, if i have time, i'll look into it further.
/J
On Mon, Mar 2, 2015 at 7:13 PM, Adunuthula, Seshu <sa...@ebay.com>
wrote:
> Thanks Jakob,
>
> Would be fantastic if you could dig in and submit a patch…
>
>
> On 3/2/15, 10:08 AM, "Jakob Stengård" <ja...@gmail.com> wrote:
>
> >I'll file a bug in jira.
> >For your information i also happen to know that tez boosted map-reduce
> >doesn't work either, but there, the problem seems to have to do with the
> >map reduce jobs them selves failing.
> >
> >In this case however, from what i've read on git hub about the "Can't get
> >cube source record size",
> >error", it could possibly be a problem with that tez doesn't output a
> >string on the form:
> >"HDFS Read: (\d+) HDFS Write"
> >
> >However, it could be something else, it could also be my logging
> >configuration for tez.
> >
> >/J
> >
> >
> >
> >On Mon, Mar 2, 2015 at 6:54 PM, Adunuthula, Seshu <sa...@ebay.com>
> >wrote:
> >
> >> Did not read your logs fully. The culprit seems to be
> >>
> >>
> >> 1. [QuartzScheduler_Worker-7]:[2015-03-02
> >>
> >>16:09:24,737][ERROR][com.kylinolap.job.flow.JobFlowListener.handleExcepti
> >>on
> >> (JobFlowListener.java:312)] - Can't get cube source record size.
> >>
> >> source_records_size may have to be computed differently when using Tez
> >>as
> >> the Hive engine. I will let Yang/Luke answer this when they come back
> >> onlineŠ
> >>
> >>
> >>
> >>
> >> On 3/2/15, 9:30 AM, "Adunuthula, Seshu" <sa...@ebay.com> wrote:
> >>
> >> >Jakob,
> >> >
> >> >Could you go ahead and file a JIRA with the stack trace at
> >> >https://issues.apache.org/jira/browse/KYLIN/
> >> >
> >> >Also include the App Master and Map/Reduce Task logs for the Tasks that
> >> >failed.
> >> >
> >> >Regards
> >> >Seshu
> >> >
> >> >On 3/2/15, 8:12 AM, "Jakob Stengård" <ja...@gmail.com> wrote:
> >> >
> >> >>Correction: That, isn't the last job, it's step 9.
> >> >>
> >> >>/J
> >> >>
> >> >>On Mon, Mar 2, 2015 at 5:10 PM, Jakob Stengård
> >><jakob.stengard@gmail.com
> >> >
> >> >>wrote:
> >> >>
> >> >>> Well, this only happens with
> >> >>> <property>
> >> >>> <name>hive.execution.engine</name>
> >> >>> <value>tez</value>
> >> >>> </property>
> >> >>>
> >> >>> If i use mr, it works fine.
> >> >>>
> >> >>>
> >> >>> From what i can see form the job log, none of the previous
> >>map-reduce
> >> >>>jobs
> >> >>> fail, but the build fails at the last step.
> >> >>> Log from the last mr job for example:
> >> >>> http://pastebin.com/jNWSRKLi
> >> >>>
> >> >>>
> >> >>> Is tez supposed to work? Because in that case, maybe it's just my
> >> >>>setup.
> >> >>>
> >> >>> /Jakob
> >> >>>
> >> >>>
> >> >>>
> >> >>> On Mon, Mar 2, 2015 at 4:34 PM, Santosh Akhilesh <
> >> >>> santoshakhilesh@gmail.com> wrote:
> >> >>>
> >> >>>> This doesn't seems to be complete kylin.log. Do you have any
> >>exception
> >> >>>> related to MR counter in map reduce logs? If yes then this needs
> >>to be
> >> >>>> checked because kylin will need the MR counter to calculate the
> >>cube
> >> >>>>size.
> >> >>>> On Mon, 2 Mar 2015 at 8:49 pm, Jakob Stengård
> >> >>>><ja...@gmail.com>
> >> >>>> wrote:
> >> >>>>
> >> >>>> > It breaks on the last step: "Load HFile to HBase Table":
> >> >>>> >
> >> >>>> > Kylin.log:
> >> >>>> > http://pastebin.com/3kAQmJ3u
> >> >>>> >
> >> >>>> > On Mon, Mar 2, 2015 at 2:27 PM, Adunuthula, Seshu
> >> >>>><sadunuthula@ebay.com
> >> >>>> >
> >> >>>> > wrote:
> >> >>>> >
> >> >>>> > > Jakob,
> >> >>>> > >
> >> >>>> > > Could you please send out the complete stack trace with Tez
> >> >>>>enabled?
> >> >>>> > >
> >> >>>> > > Regards
> >> >>>> > > Seshu
> >> >>>> > >
> >> >>>> > > On 3/2/15, 4:51 AM, "Jakob Stengård" <jakob.stengard@gmail.com
> >
> >> >>>> wrote:
> >> >>>> > >
> >> >>>> > > >Hi
> >> >>>> > > >Will kylin suport hive+tez in v0.7.x?
> >> >>>> > > >
> >> >>>> > > >I've tried using tez with hive, but currently the cube build
> >> >>>>fails
> >> >>>> with
> >> >>>> > > >"Can't get cube source record size", because kylin expects
> >> >>>>certain
> >> >>>> > console
> >> >>>> > > >output from map-reduce, and tez doesn't print this.
> >> >>>> > > >
> >> >>>> > > >BG.
> >> >>>> > > >Jakob Stengard
> >> >>>> > >
> >> >>>> > >
> >> >>>> >
> >> >>>>
> >> >>>
> >> >>>
> >> >
> >>
> >>
>
>
Re: Tez
Posted by "Adunuthula, Seshu" <sa...@ebay.com>.
Thanks Jakob,
Would be fantastic if you could dig in and submit a patch…
On 3/2/15, 10:08 AM, "Jakob Stengård" <ja...@gmail.com> wrote:
>I'll file a bug in jira.
>For your information i also happen to know that tez boosted map-reduce
>doesn't work either, but there, the problem seems to have to do with the
>map reduce jobs them selves failing.
>
>In this case however, from what i've read on git hub about the "Can't get
>cube source record size",
>error", it could possibly be a problem with that tez doesn't output a
>string on the form:
>"HDFS Read: (\d+) HDFS Write"
>
>However, it could be something else, it could also be my logging
>configuration for tez.
>
>/J
>
>
>
>On Mon, Mar 2, 2015 at 6:54 PM, Adunuthula, Seshu <sa...@ebay.com>
>wrote:
>
>> Did not read your logs fully. The culprit seems to be
>>
>>
>> 1. [QuartzScheduler_Worker-7]:[2015-03-02
>>
>>16:09:24,737][ERROR][com.kylinolap.job.flow.JobFlowListener.handleExcepti
>>on
>> (JobFlowListener.java:312)] - Can't get cube source record size.
>>
>> source_records_size may have to be computed differently when using Tez
>>as
>> the Hive engine. I will let Yang/Luke answer this when they come back
>> onlineŠ
>>
>>
>>
>>
>> On 3/2/15, 9:30 AM, "Adunuthula, Seshu" <sa...@ebay.com> wrote:
>>
>> >Jakob,
>> >
>> >Could you go ahead and file a JIRA with the stack trace at
>> >https://issues.apache.org/jira/browse/KYLIN/
>> >
>> >Also include the App Master and Map/Reduce Task logs for the Tasks that
>> >failed.
>> >
>> >Regards
>> >Seshu
>> >
>> >On 3/2/15, 8:12 AM, "Jakob Stengård" <ja...@gmail.com> wrote:
>> >
>> >>Correction: That, isn't the last job, it's step 9.
>> >>
>> >>/J
>> >>
>> >>On Mon, Mar 2, 2015 at 5:10 PM, Jakob Stengård
>><jakob.stengard@gmail.com
>> >
>> >>wrote:
>> >>
>> >>> Well, this only happens with
>> >>> <property>
>> >>> <name>hive.execution.engine</name>
>> >>> <value>tez</value>
>> >>> </property>
>> >>>
>> >>> If i use mr, it works fine.
>> >>>
>> >>>
>> >>> From what i can see form the job log, none of the previous
>>map-reduce
>> >>>jobs
>> >>> fail, but the build fails at the last step.
>> >>> Log from the last mr job for example:
>> >>> http://pastebin.com/jNWSRKLi
>> >>>
>> >>>
>> >>> Is tez supposed to work? Because in that case, maybe it's just my
>> >>>setup.
>> >>>
>> >>> /Jakob
>> >>>
>> >>>
>> >>>
>> >>> On Mon, Mar 2, 2015 at 4:34 PM, Santosh Akhilesh <
>> >>> santoshakhilesh@gmail.com> wrote:
>> >>>
>> >>>> This doesn't seems to be complete kylin.log. Do you have any
>>exception
>> >>>> related to MR counter in map reduce logs? If yes then this needs
>>to be
>> >>>> checked because kylin will need the MR counter to calculate the
>>cube
>> >>>>size.
>> >>>> On Mon, 2 Mar 2015 at 8:49 pm, Jakob Stengård
>> >>>><ja...@gmail.com>
>> >>>> wrote:
>> >>>>
>> >>>> > It breaks on the last step: "Load HFile to HBase Table":
>> >>>> >
>> >>>> > Kylin.log:
>> >>>> > http://pastebin.com/3kAQmJ3u
>> >>>> >
>> >>>> > On Mon, Mar 2, 2015 at 2:27 PM, Adunuthula, Seshu
>> >>>><sadunuthula@ebay.com
>> >>>> >
>> >>>> > wrote:
>> >>>> >
>> >>>> > > Jakob,
>> >>>> > >
>> >>>> > > Could you please send out the complete stack trace with Tez
>> >>>>enabled?
>> >>>> > >
>> >>>> > > Regards
>> >>>> > > Seshu
>> >>>> > >
>> >>>> > > On 3/2/15, 4:51 AM, "Jakob Stengård" <ja...@gmail.com>
>> >>>> wrote:
>> >>>> > >
>> >>>> > > >Hi
>> >>>> > > >Will kylin suport hive+tez in v0.7.x?
>> >>>> > > >
>> >>>> > > >I've tried using tez with hive, but currently the cube build
>> >>>>fails
>> >>>> with
>> >>>> > > >"Can't get cube source record size", because kylin expects
>> >>>>certain
>> >>>> > console
>> >>>> > > >output from map-reduce, and tez doesn't print this.
>> >>>> > > >
>> >>>> > > >BG.
>> >>>> > > >Jakob Stengard
>> >>>> > >
>> >>>> > >
>> >>>> >
>> >>>>
>> >>>
>> >>>
>> >
>>
>>
Re: Tez
Posted by Jakob Stengård <ja...@gmail.com>.
I'll file a bug in jira.
For your information i also happen to know that tez boosted map-reduce
doesn't work either, but there, the problem seems to have to do with the
map reduce jobs them selves failing.
In this case however, from what i've read on git hub about the "Can't get
cube source record size",
error", it could possibly be a problem with that tez doesn't output a
string on the form:
"HDFS Read: (\d+) HDFS Write"
However, it could be something else, it could also be my logging
configuration for tez.
/J
On Mon, Mar 2, 2015 at 6:54 PM, Adunuthula, Seshu <sa...@ebay.com>
wrote:
> Did not read your logs fully. The culprit seems to be
>
>
> 1. [QuartzScheduler_Worker-7]:[2015-03-02
> 16:09:24,737][ERROR][com.kylinolap.job.flow.JobFlowListener.handleException
> (JobFlowListener.java:312)] - Can't get cube source record size.
>
> source_records_size may have to be computed differently when using Tez as
> the Hive engine. I will let Yang/Luke answer this when they come back
> onlineŠ
>
>
>
>
> On 3/2/15, 9:30 AM, "Adunuthula, Seshu" <sa...@ebay.com> wrote:
>
> >Jakob,
> >
> >Could you go ahead and file a JIRA with the stack trace at
> >https://issues.apache.org/jira/browse/KYLIN/
> >
> >Also include the App Master and Map/Reduce Task logs for the Tasks that
> >failed.
> >
> >Regards
> >Seshu
> >
> >On 3/2/15, 8:12 AM, "Jakob Stengård" <ja...@gmail.com> wrote:
> >
> >>Correction: That, isn't the last job, it's step 9.
> >>
> >>/J
> >>
> >>On Mon, Mar 2, 2015 at 5:10 PM, Jakob Stengård <jakob.stengard@gmail.com
> >
> >>wrote:
> >>
> >>> Well, this only happens with
> >>> <property>
> >>> <name>hive.execution.engine</name>
> >>> <value>tez</value>
> >>> </property>
> >>>
> >>> If i use mr, it works fine.
> >>>
> >>>
> >>> From what i can see form the job log, none of the previous map-reduce
> >>>jobs
> >>> fail, but the build fails at the last step.
> >>> Log from the last mr job for example:
> >>> http://pastebin.com/jNWSRKLi
> >>>
> >>>
> >>> Is tez supposed to work? Because in that case, maybe it's just my
> >>>setup.
> >>>
> >>> /Jakob
> >>>
> >>>
> >>>
> >>> On Mon, Mar 2, 2015 at 4:34 PM, Santosh Akhilesh <
> >>> santoshakhilesh@gmail.com> wrote:
> >>>
> >>>> This doesn't seems to be complete kylin.log. Do you have any exception
> >>>> related to MR counter in map reduce logs? If yes then this needs to be
> >>>> checked because kylin will need the MR counter to calculate the cube
> >>>>size.
> >>>> On Mon, 2 Mar 2015 at 8:49 pm, Jakob Stengård
> >>>><ja...@gmail.com>
> >>>> wrote:
> >>>>
> >>>> > It breaks on the last step: "Load HFile to HBase Table":
> >>>> >
> >>>> > Kylin.log:
> >>>> > http://pastebin.com/3kAQmJ3u
> >>>> >
> >>>> > On Mon, Mar 2, 2015 at 2:27 PM, Adunuthula, Seshu
> >>>><sadunuthula@ebay.com
> >>>> >
> >>>> > wrote:
> >>>> >
> >>>> > > Jakob,
> >>>> > >
> >>>> > > Could you please send out the complete stack trace with Tez
> >>>>enabled?
> >>>> > >
> >>>> > > Regards
> >>>> > > Seshu
> >>>> > >
> >>>> > > On 3/2/15, 4:51 AM, "Jakob Stengård" <ja...@gmail.com>
> >>>> wrote:
> >>>> > >
> >>>> > > >Hi
> >>>> > > >Will kylin suport hive+tez in v0.7.x?
> >>>> > > >
> >>>> > > >I've tried using tez with hive, but currently the cube build
> >>>>fails
> >>>> with
> >>>> > > >"Can't get cube source record size", because kylin expects
> >>>>certain
> >>>> > console
> >>>> > > >output from map-reduce, and tez doesn't print this.
> >>>> > > >
> >>>> > > >BG.
> >>>> > > >Jakob Stengard
> >>>> > >
> >>>> > >
> >>>> >
> >>>>
> >>>
> >>>
> >
>
>
Re: Tez
Posted by "Adunuthula, Seshu" <sa...@ebay.com>.
Did not read your logs fully. The culprit seems to be
1. [QuartzScheduler_Worker-7]:[2015-03-02
16:09:24,737][ERROR][com.kylinolap.job.flow.JobFlowListener.handleException
(JobFlowListener.java:312)] - Can't get cube source record size.
source_records_size may have to be computed differently when using Tez as
the Hive engine. I will let Yang/Luke answer this when they come back
onlineŠ
On 3/2/15, 9:30 AM, "Adunuthula, Seshu" <sa...@ebay.com> wrote:
>Jakob,
>
>Could you go ahead and file a JIRA with the stack trace at
>https://issues.apache.org/jira/browse/KYLIN/
>
>Also include the App Master and Map/Reduce Task logs for the Tasks that
>failed.
>
>Regards
>Seshu
>
>On 3/2/15, 8:12 AM, "Jakob Stengård" <ja...@gmail.com> wrote:
>
>>Correction: That, isn't the last job, it's step 9.
>>
>>/J
>>
>>On Mon, Mar 2, 2015 at 5:10 PM, Jakob Stengård <ja...@gmail.com>
>>wrote:
>>
>>> Well, this only happens with
>>> <property>
>>> <name>hive.execution.engine</name>
>>> <value>tez</value>
>>> </property>
>>>
>>> If i use mr, it works fine.
>>>
>>>
>>> From what i can see form the job log, none of the previous map-reduce
>>>jobs
>>> fail, but the build fails at the last step.
>>> Log from the last mr job for example:
>>> http://pastebin.com/jNWSRKLi
>>>
>>>
>>> Is tez supposed to work? Because in that case, maybe it's just my
>>>setup.
>>>
>>> /Jakob
>>>
>>>
>>>
>>> On Mon, Mar 2, 2015 at 4:34 PM, Santosh Akhilesh <
>>> santoshakhilesh@gmail.com> wrote:
>>>
>>>> This doesn't seems to be complete kylin.log. Do you have any exception
>>>> related to MR counter in map reduce logs? If yes then this needs to be
>>>> checked because kylin will need the MR counter to calculate the cube
>>>>size.
>>>> On Mon, 2 Mar 2015 at 8:49 pm, Jakob Stengård
>>>><ja...@gmail.com>
>>>> wrote:
>>>>
>>>> > It breaks on the last step: "Load HFile to HBase Table":
>>>> >
>>>> > Kylin.log:
>>>> > http://pastebin.com/3kAQmJ3u
>>>> >
>>>> > On Mon, Mar 2, 2015 at 2:27 PM, Adunuthula, Seshu
>>>><sadunuthula@ebay.com
>>>> >
>>>> > wrote:
>>>> >
>>>> > > Jakob,
>>>> > >
>>>> > > Could you please send out the complete stack trace with Tez
>>>>enabled?
>>>> > >
>>>> > > Regards
>>>> > > Seshu
>>>> > >
>>>> > > On 3/2/15, 4:51 AM, "Jakob Stengård" <ja...@gmail.com>
>>>> wrote:
>>>> > >
>>>> > > >Hi
>>>> > > >Will kylin suport hive+tez in v0.7.x?
>>>> > > >
>>>> > > >I've tried using tez with hive, but currently the cube build
>>>>fails
>>>> with
>>>> > > >"Can't get cube source record size", because kylin expects
>>>>certain
>>>> > console
>>>> > > >output from map-reduce, and tez doesn't print this.
>>>> > > >
>>>> > > >BG.
>>>> > > >Jakob Stengard
>>>> > >
>>>> > >
>>>> >
>>>>
>>>
>>>
>
Re: Tez
Posted by "Adunuthula, Seshu" <sa...@ebay.com>.
Jakob,
Could you go ahead and file a JIRA with the stack trace at
https://issues.apache.org/jira/browse/KYLIN/
Also include the App Master and Map/Reduce Task logs for the Tasks that
failed.
Regards
Seshu
On 3/2/15, 8:12 AM, "Jakob Stengård" <ja...@gmail.com> wrote:
>Correction: That, isn't the last job, it's step 9.
>
>/J
>
>On Mon, Mar 2, 2015 at 5:10 PM, Jakob Stengård <ja...@gmail.com>
>wrote:
>
>> Well, this only happens with
>> <property>
>> <name>hive.execution.engine</name>
>> <value>tez</value>
>> </property>
>>
>> If i use mr, it works fine.
>>
>>
>> From what i can see form the job log, none of the previous map-reduce
>>jobs
>> fail, but the build fails at the last step.
>> Log from the last mr job for example:
>> http://pastebin.com/jNWSRKLi
>>
>>
>> Is tez supposed to work? Because in that case, maybe it's just my setup.
>>
>> /Jakob
>>
>>
>>
>> On Mon, Mar 2, 2015 at 4:34 PM, Santosh Akhilesh <
>> santoshakhilesh@gmail.com> wrote:
>>
>>> This doesn't seems to be complete kylin.log. Do you have any exception
>>> related to MR counter in map reduce logs? If yes then this needs to be
>>> checked because kylin will need the MR counter to calculate the cube
>>>size.
>>> On Mon, 2 Mar 2015 at 8:49 pm, Jakob Stengård
>>><ja...@gmail.com>
>>> wrote:
>>>
>>> > It breaks on the last step: "Load HFile to HBase Table":
>>> >
>>> > Kylin.log:
>>> > http://pastebin.com/3kAQmJ3u
>>> >
>>> > On Mon, Mar 2, 2015 at 2:27 PM, Adunuthula, Seshu
>>><sadunuthula@ebay.com
>>> >
>>> > wrote:
>>> >
>>> > > Jakob,
>>> > >
>>> > > Could you please send out the complete stack trace with Tez
>>>enabled?
>>> > >
>>> > > Regards
>>> > > Seshu
>>> > >
>>> > > On 3/2/15, 4:51 AM, "Jakob Stengård" <ja...@gmail.com>
>>> wrote:
>>> > >
>>> > > >Hi
>>> > > >Will kylin suport hive+tez in v0.7.x?
>>> > > >
>>> > > >I've tried using tez with hive, but currently the cube build fails
>>> with
>>> > > >"Can't get cube source record size", because kylin expects certain
>>> > console
>>> > > >output from map-reduce, and tez doesn't print this.
>>> > > >
>>> > > >BG.
>>> > > >Jakob Stengard
>>> > >
>>> > >
>>> >
>>>
>>
>>
Re: Tez
Posted by Jakob Stengård <ja...@gmail.com>.
Correction: That, isn't the last job, it's step 9.
/J
On Mon, Mar 2, 2015 at 5:10 PM, Jakob Stengård <ja...@gmail.com>
wrote:
> Well, this only happens with
> <property>
> <name>hive.execution.engine</name>
> <value>tez</value>
> </property>
>
> If i use mr, it works fine.
>
>
> From what i can see form the job log, none of the previous map-reduce jobs
> fail, but the build fails at the last step.
> Log from the last mr job for example:
> http://pastebin.com/jNWSRKLi
>
>
> Is tez supposed to work? Because in that case, maybe it's just my setup.
>
> /Jakob
>
>
>
> On Mon, Mar 2, 2015 at 4:34 PM, Santosh Akhilesh <
> santoshakhilesh@gmail.com> wrote:
>
>> This doesn't seems to be complete kylin.log. Do you have any exception
>> related to MR counter in map reduce logs? If yes then this needs to be
>> checked because kylin will need the MR counter to calculate the cube size.
>> On Mon, 2 Mar 2015 at 8:49 pm, Jakob Stengård <ja...@gmail.com>
>> wrote:
>>
>> > It breaks on the last step: "Load HFile to HBase Table":
>> >
>> > Kylin.log:
>> > http://pastebin.com/3kAQmJ3u
>> >
>> > On Mon, Mar 2, 2015 at 2:27 PM, Adunuthula, Seshu <sadunuthula@ebay.com
>> >
>> > wrote:
>> >
>> > > Jakob,
>> > >
>> > > Could you please send out the complete stack trace with Tez enabled?
>> > >
>> > > Regards
>> > > Seshu
>> > >
>> > > On 3/2/15, 4:51 AM, "Jakob Stengård" <ja...@gmail.com>
>> wrote:
>> > >
>> > > >Hi
>> > > >Will kylin suport hive+tez in v0.7.x?
>> > > >
>> > > >I've tried using tez with hive, but currently the cube build fails
>> with
>> > > >"Can't get cube source record size", because kylin expects certain
>> > console
>> > > >output from map-reduce, and tez doesn't print this.
>> > > >
>> > > >BG.
>> > > >Jakob Stengard
>> > >
>> > >
>> >
>>
>
>
Re: Tez
Posted by Jakob Stengård <ja...@gmail.com>.
Well, this only happens with
<property>
<name>hive.execution.engine</name>
<value>tez</value>
</property>
If i use mr, it works fine.
>From what i can see form the job log, none of the previous map-reduce jobs
fail, but the build fails at the last step.
Log from the last mr job for example:
http://pastebin.com/jNWSRKLi
Is tez supposed to work? Because in that case, maybe it's just my setup.
/Jakob
On Mon, Mar 2, 2015 at 4:34 PM, Santosh Akhilesh <sa...@gmail.com>
wrote:
> This doesn't seems to be complete kylin.log. Do you have any exception
> related to MR counter in map reduce logs? If yes then this needs to be
> checked because kylin will need the MR counter to calculate the cube size.
> On Mon, 2 Mar 2015 at 8:49 pm, Jakob Stengård <ja...@gmail.com>
> wrote:
>
> > It breaks on the last step: "Load HFile to HBase Table":
> >
> > Kylin.log:
> > http://pastebin.com/3kAQmJ3u
> >
> > On Mon, Mar 2, 2015 at 2:27 PM, Adunuthula, Seshu <sa...@ebay.com>
> > wrote:
> >
> > > Jakob,
> > >
> > > Could you please send out the complete stack trace with Tez enabled?
> > >
> > > Regards
> > > Seshu
> > >
> > > On 3/2/15, 4:51 AM, "Jakob Stengård" <ja...@gmail.com> wrote:
> > >
> > > >Hi
> > > >Will kylin suport hive+tez in v0.7.x?
> > > >
> > > >I've tried using tez with hive, but currently the cube build fails
> with
> > > >"Can't get cube source record size", because kylin expects certain
> > console
> > > >output from map-reduce, and tez doesn't print this.
> > > >
> > > >BG.
> > > >Jakob Stengard
> > >
> > >
> >
>
Re: Tez
Posted by Santosh Akhilesh <sa...@gmail.com>.
This doesn't seems to be complete kylin.log. Do you have any exception
related to MR counter in map reduce logs? If yes then this needs to be
checked because kylin will need the MR counter to calculate the cube size.
On Mon, 2 Mar 2015 at 8:49 pm, Jakob Stengård <ja...@gmail.com>
wrote:
> It breaks on the last step: "Load HFile to HBase Table":
>
> Kylin.log:
> http://pastebin.com/3kAQmJ3u
>
> On Mon, Mar 2, 2015 at 2:27 PM, Adunuthula, Seshu <sa...@ebay.com>
> wrote:
>
> > Jakob,
> >
> > Could you please send out the complete stack trace with Tez enabled?
> >
> > Regards
> > Seshu
> >
> > On 3/2/15, 4:51 AM, "Jakob Stengård" <ja...@gmail.com> wrote:
> >
> > >Hi
> > >Will kylin suport hive+tez in v0.7.x?
> > >
> > >I've tried using tez with hive, but currently the cube build fails with
> > >"Can't get cube source record size", because kylin expects certain
> console
> > >output from map-reduce, and tez doesn't print this.
> > >
> > >BG.
> > >Jakob Stengard
> >
> >
>
Re: Tez
Posted by Jakob Stengård <ja...@gmail.com>.
It breaks on the last step: "Load HFile to HBase Table":
Kylin.log:
http://pastebin.com/3kAQmJ3u
On Mon, Mar 2, 2015 at 2:27 PM, Adunuthula, Seshu <sa...@ebay.com>
wrote:
> Jakob,
>
> Could you please send out the complete stack trace with Tez enabled?
>
> Regards
> Seshu
>
> On 3/2/15, 4:51 AM, "Jakob Stengård" <ja...@gmail.com> wrote:
>
> >Hi
> >Will kylin suport hive+tez in v0.7.x?
> >
> >I've tried using tez with hive, but currently the cube build fails with
> >"Can't get cube source record size", because kylin expects certain console
> >output from map-reduce, and tez doesn't print this.
> >
> >BG.
> >Jakob Stengard
>
>
Re: Tez
Posted by "Adunuthula, Seshu" <sa...@ebay.com>.
Jakob,
Could you please send out the complete stack trace with Tez enabled?
Regards
Seshu
On 3/2/15, 4:51 AM, "Jakob Stengård" <ja...@gmail.com> wrote:
>Hi
>Will kylin suport hive+tez in v0.7.x?
>
>I've tried using tez with hive, but currently the cube build fails with
>"Can't get cube source record size", because kylin expects certain console
>output from map-reduce, and tez doesn't print this.
>
>BG.
>Jakob Stengard