You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@twill.apache.org by Terence Yim <ch...@apache.org> on 2019/05/16 07:47:42 UTC
Re: Regarding Twill changes in HDP 3.1
Hi,
We haven't tested Twill compatibility with Hadoop 3, which is what used by
HDP 3.1. Do you have the yarn application logs for the workflow and/or the
map reduce program to help us debug this issue?
Thanks,
Terence
On Wed, May 15, 2019 at 10:41 PM Rishabh Budhouliya <
rishabh.budhouliya@external.thalesgroup.com> wrote:
> Hi, I have been testing CDAP v5.1.2 with a HDP 3.1 cluster and there is an
> issue in running a pipeline, particularly in the MapReduce program
> 'phase-1' with error "ClassNotFoundException :
> co.cask.cdap.etl.api.batch.BatchSource".
>
> Now, I know this is not the cdap-user group, but we think that this issue
> might somehow stem from Twill version upgrade, maybe the way class are
> being loaded by the container?
>
> Just wanted to know if there is some change in HDP 3.1 which we are
> missing, any pointers would be great !
>
> Regards,
> Rishabh Budhouliya
>
FW: Regarding Twill changes in HDP 3.1
Posted by Rishabh Budhouliya <ri...@external.thalesgroup.com>.
________________________________
From: Terence Yim <ch...@apache.org>
Sent: 16 May 2019 14:39
To: dev
Cc: Rishabh Budhouliya
Subject: Fwd: Regarding Twill changes in HDP 3.1
Forward to the dev@ for wider discussion
---------- Forwarded message ---------
From: Rishabh Budhouliya <ri...@external.thalesgroup.com>>
Date: Thu, May 16, 2019 at 1:06 AM
Subject: Re: Regarding Twill changes in HDP 3.1
To: Terence Yim <ch...@apache.org>>
Okay, sure, please find the logs attached below.
Thanks,
Rishabh
________________________________
From: Terence Yim <ch...@apache.org>>
Sent: 16 May 2019 13:17
To: Rishabh Budhouliya
Cc: dev
Subject: Re: Regarding Twill changes in HDP 3.1
Hi,
We haven't tested Twill compatibility with Hadoop 3, which is what used by HDP 3.1. Do you have the yarn application logs for the workflow and/or the map reduce program to help us debug this issue?
Thanks,
Terence
On Wed, May 15, 2019 at 10:41 PM Rishabh Budhouliya <ri...@external.thalesgroup.com>> wrote:
Hi, I have been testing CDAP v5.1.2 with a HDP 3.1 cluster and there is an issue in running a pipeline, particularly in the MapReduce program 'phase-1' with error "ClassNotFoundException : co.cask.cdap.etl.api.batch.BatchSource".
Now, I know this is not the cdap-user group, but we think that this issue might somehow stem from Twill version upgrade, maybe the way class are being loaded by the container?
Just wanted to know if there is some change in HDP 3.1 which we are missing, any pointers would be great !
Regards,
Rishabh Budhouliya
Re: Regarding Twill changes in HDP 3.1
Posted by Terence Yim <ch...@apache.org>.
Hi,
I scanned through the logs and don't see anything that YARN works
differently in Hadoop 3. We will need to create a Hadoop 3 cluster and try
to reproduce the problem, which will take some time.
Terence
On Thu, May 16, 2019 at 2:09 AM Terence Yim <ch...@apache.org> wrote:
> Forward to the dev@ for wider discussion
>
> ---------- Forwarded message ---------
> From: Rishabh Budhouliya <ri...@external.thalesgroup.com>
> Date: Thu, May 16, 2019 at 1:06 AM
> Subject: Re: Regarding Twill changes in HDP 3.1
> To: Terence Yim <ch...@apache.org>
>
>
> Okay, sure, please find the logs attached below.
>
>
> Thanks,
> Rishabh
> ------------------------------
> *From:* Terence Yim <ch...@apache.org>
> *Sent:* 16 May 2019 13:17
> *To:* Rishabh Budhouliya
> *Cc:* dev
> *Subject:* Re: Regarding Twill changes in HDP 3.1
>
> Hi,
>
> We haven't tested Twill compatibility with Hadoop 3, which is what used by
> HDP 3.1. Do you have the yarn application logs for the workflow and/or the
> map reduce program to help us debug this issue?
>
> Thanks,
> Terence
>
> On Wed, May 15, 2019 at 10:41 PM Rishabh Budhouliya <
> rishabh.budhouliya@external.thalesgroup.com> wrote:
>
> Hi, I have been testing CDAP v5.1.2 with a HDP 3.1 cluster and there is an
> issue in running a pipeline, particularly in the MapReduce program
> 'phase-1' with error "ClassNotFoundException :
> co.cask.cdap.etl.api.batch.BatchSource".
>
> Now, I know this is not the cdap-user group, but we think that this issue
> might somehow stem from Twill version upgrade, maybe the way class are
> being loaded by the container?
>
> Just wanted to know if there is some change in HDP 3.1 which we are
> missing, any pointers would be great !
>
> Regards,
> Rishabh Budhouliya
>
>
Fwd: Regarding Twill changes in HDP 3.1
Posted by Terence Yim <ch...@apache.org>.
Forward to the dev@ for wider discussion
---------- Forwarded message ---------
From: Rishabh Budhouliya <ri...@external.thalesgroup.com>
Date: Thu, May 16, 2019 at 1:06 AM
Subject: Re: Regarding Twill changes in HDP 3.1
To: Terence Yim <ch...@apache.org>
Okay, sure, please find the logs attached below.
Thanks,
Rishabh
------------------------------
*From:* Terence Yim <ch...@apache.org>
*Sent:* 16 May 2019 13:17
*To:* Rishabh Budhouliya
*Cc:* dev
*Subject:* Re: Regarding Twill changes in HDP 3.1
Hi,
We haven't tested Twill compatibility with Hadoop 3, which is what used by
HDP 3.1. Do you have the yarn application logs for the workflow and/or the
map reduce program to help us debug this issue?
Thanks,
Terence
On Wed, May 15, 2019 at 10:41 PM Rishabh Budhouliya <
rishabh.budhouliya@external.thalesgroup.com> wrote:
Hi, I have been testing CDAP v5.1.2 with a HDP 3.1 cluster and there is an
issue in running a pipeline, particularly in the MapReduce program
'phase-1' with error "ClassNotFoundException :
co.cask.cdap.etl.api.batch.BatchSource".
Now, I know this is not the cdap-user group, but we think that this issue
might somehow stem from Twill version upgrade, maybe the way class are
being loaded by the container?
Just wanted to know if there is some change in HDP 3.1 which we are
missing, any pointers would be great !
Regards,
Rishabh Budhouliya