You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@phoenix.apache.org by "Bulvik, Noam" <No...@teoco.com> on 2014/10/05 08:40:53 UTC
bulk loading using OOZIE
Hi,
We are trying to do periodic bulk loading using OOZIE as scheduler. We impalement script task that should call the bulk loading command line with all needed parameter. When we run the script directly from Linux console it works fine but when we run the OOZIE task it fail saying that jar is missing or we do not have permission to read it. We are using CDH5.1 and the jar it companies about is one of the CDh jars that we checked and it is exists on all machines of the cluster
Any idea ?
Regards,
Noam
Information in this e-mail and its attachments is confidential and privileged under the TEOCO confidentiality terms that can be reviewed here<http://www.teoco.com/email-disclaimer>.
RE: bulk loading using OOZIE
Posted by "Bulvik, Noam" <No...@teoco.com>.
In the log I see
2014-10-06 11:13:59,822 ERROR [phoenix-2-thread-0] org.apache.phoenix.mapreduce.CsvBulkLoadTool: Import job on table=CDRINFO failed due to 2014-10-06 11:13:59,822 WARN [phoenix-2-thread-0] org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException as:nobody (auth:SIMPLE) cause:java.io.FileNotFoundException: File does not exist: hdfs://dc50-dev-bd01:8020/opt/cloudera/parcels/CDH-5.1.0-1.cdh5.1.0.p0.53/lib/hbase/lib/hbase-server-0.98.1-cdh5.1.0.jar
2014-10-06 11:13:59,822 ERROR [phoenix-2-thread-0] org.apache.phoenix.mapreduce.CsvBulkLoadTool: Import job on table=CDRINFO failed due to exception:java.io.FileNotFoundException: File does not exist: hdfs://dc50-dev-bd01:8020/opt/cloudera/parcels/CDH-5.1.0-1.cdh5.1.0.p0.53/lib/hbase/lib/hbase-server-0.98.1-cdh5.1.0.jar
-----Original Message-----
From: Gabriel Reid [mailto:gabriel.reid@gmail.com]
Sent: Monday, October 6, 2014 9:39 AM
To: user@phoenix.apache.org
Cc: Bulvik, Noam
Subject: Re: bulk loading using OOZIE
Hi Noam,
Could you post the error message and/or stack trace you're getting when Oozie says that a jar is missing or you don't have permission to read it?
- Gabriel
On Sun, Oct 5, 2014 at 8:40 AM, Bulvik, Noam <No...@teoco.com> wrote:
> Hi,
>
>
>
> We are trying to do periodic bulk loading using OOZIE as scheduler.
> We impalement script task that should call the bulk loading command
> line with all needed parameter. When we run the script directly from
> Linux console it works fine but when we run the OOZIE task it fail
> saying that jar is missing or we do not have permission to read it. We
> are using CDH5.1 and the jar it companies about is one of the CDh jars
> that we checked and it is exists on all machines of the cluster
>
>
>
> Any idea ?
>
>
>
> Regards,
>
> Noam
>
> Information in this e-mail and its attachments is confidential and
> privileged under the TEOCO confidentiality terms that can be reviewed here.
Information in this e-mail and its attachments is confidential and privileged under the TEOCO confidentiality terms that can be reviewed here<http://www.teoco.com/email-disclaimer>.
Re: bulk loading using OOZIE
Posted by Gabriel Reid <ga...@gmail.com>.
Hi Noam,
Could you post the error message and/or stack trace you're getting
when Oozie says that a jar is missing or you don't have permission to
read it?
- Gabriel
On Sun, Oct 5, 2014 at 8:40 AM, Bulvik, Noam <No...@teoco.com> wrote:
> Hi,
>
>
>
> We are trying to do periodic bulk loading using OOZIE as scheduler. We
> impalement script task that should call the bulk loading command line with
> all needed parameter. When we run the script directly from Linux console it
> works fine but when we run the OOZIE task it fail saying that jar is missing
> or we do not have permission to read it. We are using CDH5.1 and the jar it
> companies about is one of the CDh jars that we checked and it is exists on
> all machines of the cluster
>
>
>
> Any idea ?
>
>
>
> Regards,
>
> Noam
>
> Information in this e-mail and its attachments is confidential and
> privileged under the TEOCO confidentiality terms that can be reviewed here.
RE: bulk loading using OOZIE
Posted by Artem Ervits <ar...@gmail.com>.
Try to run an example shell script without phoenix. There's a sample in
oozie directory. Also a good place to find issues is in oozie logs. Try
running your process manually without Oozie. That way you can single out
what the issue is. I'm not an expert but I think Oozie doesn't find HBase
jars.
On Oct 7, 2014 8:41 AM, "Bulvik, Noam" <No...@teoco.com> wrote:
> I do not understand how it is related, we run CSV bulk loader as Ozzie
> shell script tasks, from the logs I see that when the script starts all the
> needed env' variable including the classpath that points to CDH libs is ok.
> Then from the CSV bulk loading it tries to access some file and fail with
> the message that file not found
>
>
>
> org.apache.phoenix.mapreduce.CsvBulkLoadTool: Import job on table=CDRINFO
> failed due to exception:java.io.FileNotFoundException: File does not exist:
> hdfs://dc50-dev-bd01:8020/opt/cloudera/parcels/CDH-5.1.0-1.cdh5.1.0.p0.53/lib/hbase/lib/hbase-server-0.98.1-cdh5.1.0.jar
>
>
>
> do I need to add all CDH file to share lib for the script to work – it
> does not make sense
>
>
>
>
>
>
>
> *From:* Artem Ervits [mailto:artemervits@gmail.com]
> *Sent:* Tuesday, October 7, 2014 2:25 PM
> *To:* user@phoenix.apache.org
> *Subject:* RE: bulk loading using OOZIE
>
>
>
> Is your oozie sharelib setup? Read about it, it will explain what Oozie
> expects.
>
> On Oct 7, 2014 7:18 AM, "Bulvik, Noam" <No...@teoco.com> wrote:
>
> I checked the file exists on all cluster machine with full permission
> (it is part of CDH files )
>
>
>
> *From:* Artem Ervits [mailto:artemervits@gmail.com]
> *Sent:* Tuesday, October 7, 2014 1:37 PM
> *To:* user@phoenix.apache.org
> *Subject:* Re: bulk loading using OOZIE
>
>
>
> from one of your earlier log posts, it says it can't find a jar in the
> directory, please check the jar is there.
>
>
>
> On Tue, Oct 7, 2014 at 1:53 AM, Bulvik, Noam <No...@teoco.com>
> wrote:
>
> We login to ozzie using mapred user.
>
> It looks like the ozzie task runs the script as user nobody , when I
> executed the command line of the script after loing as nobody it works fine
> so I do not think we need to explicit permission on some files
>
>
>
> I also do not think that we need to upload the CDH jars to ozzie (AFAIK) –
> do you?
>
>
>
> *From:* Artem Ervits [mailto:artemervits@gmail.com]
> *Sent:* Tuesday, October 7, 2014 12:51 AM
> *To:* user@phoenix.apache.org
> *Subject:* Re: bulk loading using OOZIE
>
>
>
> Double check that oozie user has permissions to read the required files in
> hdfs, for example, user running oozie wf is oozie then you should upload
> the jar and any property files to /user/oozie.
>
> On Oct 5, 2014 2:41 AM, "Bulvik, Noam" <No...@teoco.com> wrote:
>
> Hi,
>
>
>
> We are trying to do periodic bulk loading using OOZIE as scheduler. We
> impalement script task that should call the bulk loading command line with
> all needed parameter. When we run the script directly from Linux console
> it works fine but when we run the OOZIE task it fail saying that jar is
> missing or we do not have permission to read it. We are using CDH5.1 and
> the jar it companies about is one of the CDh jars that we checked and it is
> exists on all machines of the cluster
>
>
>
> Any idea ?
>
>
>
> Regards,
>
> Noam
>
> Information in this e-mail and its attachments is confidential and
> privileged under the TEOCO confidentiality terms that can be reviewed here
> <http://www.teoco.com/email-disclaimer>.
>
>
>
>
>
> --
> Artem
>
>
RE: bulk loading using OOZIE
Posted by "Bulvik, Noam" <No...@teoco.com>.
I do not understand how it is related, we run CSV bulk loader as Ozzie shell script tasks, from the logs I see that when the script starts all the needed env' variable including the classpath that points to CDH libs is ok. Then from the CSV bulk loading it tries to access some file and fail with the message that file not found
org.apache.phoenix.mapreduce.CsvBulkLoadTool: Import job on table=CDRINFO failed due to exception:java.io.FileNotFoundException: File does not exist: hdfs://dc50-dev-bd01:8020/opt/cloudera/parcels/CDH-5.1.0-1.cdh5.1.0.p0.53/lib/hbase/lib/hbase-server-0.98.1-cdh5.1.0.jar
do I need to add all CDH file to share lib for the script to work – it does not make sense
From: Artem Ervits [mailto:artemervits@gmail.com]
Sent: Tuesday, October 7, 2014 2:25 PM
To: user@phoenix.apache.org
Subject: RE: bulk loading using OOZIE
Is your oozie sharelib setup? Read about it, it will explain what Oozie expects.
On Oct 7, 2014 7:18 AM, "Bulvik, Noam" <No...@teoco.com>> wrote:
I checked the file exists on all cluster machine with full permission (it is part of CDH files )
From: Artem Ervits [mailto:artemervits@gmail.com<ma...@gmail.com>]
Sent: Tuesday, October 7, 2014 1:37 PM
To: user@phoenix.apache.org<ma...@phoenix.apache.org>
Subject: Re: bulk loading using OOZIE
from one of your earlier log posts, it says it can't find a jar in the directory, please check the jar is there.
On Tue, Oct 7, 2014 at 1:53 AM, Bulvik, Noam <No...@teoco.com>> wrote:
We login to ozzie using mapred user.
It looks like the ozzie task runs the script as user nobody , when I executed the command line of the script after loing as nobody it works fine so I do not think we need to explicit permission on some files
I also do not think that we need to upload the CDH jars to ozzie (AFAIK) – do you?
From: Artem Ervits [mailto:artemervits@gmail.com<ma...@gmail.com>]
Sent: Tuesday, October 7, 2014 12:51 AM
To: user@phoenix.apache.org<ma...@phoenix.apache.org>
Subject: Re: bulk loading using OOZIE
Double check that oozie user has permissions to read the required files in hdfs, for example, user running oozie wf is oozie then you should upload the jar and any property files to /user/oozie.
On Oct 5, 2014 2:41 AM, "Bulvik, Noam" <No...@teoco.com>> wrote:
Hi,
We are trying to do periodic bulk loading using OOZIE as scheduler. We impalement script task that should call the bulk loading command line with all needed parameter. When we run the script directly from Linux console it works fine but when we run the OOZIE task it fail saying that jar is missing or we do not have permission to read it. We are using CDH5.1 and the jar it companies about is one of the CDh jars that we checked and it is exists on all machines of the cluster
Any idea ?
Regards,
Noam
Information in this e-mail and its attachments is confidential and privileged under the TEOCO confidentiality terms that can be reviewed here<http://www.teoco.com/email-disclaimer>.
--
Artem
RE: bulk loading using OOZIE
Posted by Artem Ervits <ar...@gmail.com>.
Is your oozie sharelib setup? Read about it, it will explain what Oozie
expects.
On Oct 7, 2014 7:18 AM, "Bulvik, Noam" <No...@teoco.com> wrote:
> I checked the file exists on all cluster machine with full permission
> (it is part of CDH files )
>
>
>
> *From:* Artem Ervits [mailto:artemervits@gmail.com]
> *Sent:* Tuesday, October 7, 2014 1:37 PM
> *To:* user@phoenix.apache.org
> *Subject:* Re: bulk loading using OOZIE
>
>
>
> from one of your earlier log posts, it says it can't find a jar in the
> directory, please check the jar is there.
>
>
>
> On Tue, Oct 7, 2014 at 1:53 AM, Bulvik, Noam <No...@teoco.com>
> wrote:
>
> We login to ozzie using mapred user.
>
> It looks like the ozzie task runs the script as user nobody , when I
> executed the command line of the script after loing as nobody it works fine
> so I do not think we need to explicit permission on some files
>
>
>
> I also do not think that we need to upload the CDH jars to ozzie (AFAIK) –
> do you?
>
>
>
> *From:* Artem Ervits [mailto:artemervits@gmail.com]
> *Sent:* Tuesday, October 7, 2014 12:51 AM
> *To:* user@phoenix.apache.org
> *Subject:* Re: bulk loading using OOZIE
>
>
>
> Double check that oozie user has permissions to read the required files in
> hdfs, for example, user running oozie wf is oozie then you should upload
> the jar and any property files to /user/oozie.
>
> On Oct 5, 2014 2:41 AM, "Bulvik, Noam" <No...@teoco.com> wrote:
>
> Hi,
>
>
>
> We are trying to do periodic bulk loading using OOZIE as scheduler. We
> impalement script task that should call the bulk loading command line with
> all needed parameter. When we run the script directly from Linux console
> it works fine but when we run the OOZIE task it fail saying that jar is
> missing or we do not have permission to read it. We are using CDH5.1 and
> the jar it companies about is one of the CDh jars that we checked and it is
> exists on all machines of the cluster
>
>
>
> Any idea ?
>
>
>
> Regards,
>
> Noam
>
> Information in this e-mail and its attachments is confidential and
> privileged under the TEOCO confidentiality terms that can be reviewed here
> <http://www.teoco.com/email-disclaimer>.
>
>
>
>
>
> --
> Artem
>
RE: bulk loading using OOZIE
Posted by "Bulvik, Noam" <No...@teoco.com>.
I checked the file exists on all cluster machine with full permission (it is part of CDH files )
From: Artem Ervits [mailto:artemervits@gmail.com]
Sent: Tuesday, October 7, 2014 1:37 PM
To: user@phoenix.apache.org
Subject: Re: bulk loading using OOZIE
from one of your earlier log posts, it says it can't find a jar in the directory, please check the jar is there.
On Tue, Oct 7, 2014 at 1:53 AM, Bulvik, Noam <No...@teoco.com>> wrote:
We login to ozzie using mapred user.
It looks like the ozzie task runs the script as user nobody , when I executed the command line of the script after loing as nobody it works fine so I do not think we need to explicit permission on some files
I also do not think that we need to upload the CDH jars to ozzie (AFAIK) – do you?
From: Artem Ervits [mailto:artemervits@gmail.com<ma...@gmail.com>]
Sent: Tuesday, October 7, 2014 12:51 AM
To: user@phoenix.apache.org<ma...@phoenix.apache.org>
Subject: Re: bulk loading using OOZIE
Double check that oozie user has permissions to read the required files in hdfs, for example, user running oozie wf is oozie then you should upload the jar and any property files to /user/oozie.
On Oct 5, 2014 2:41 AM, "Bulvik, Noam" <No...@teoco.com>> wrote:
Hi,
We are trying to do periodic bulk loading using OOZIE as scheduler. We impalement script task that should call the bulk loading command line with all needed parameter. When we run the script directly from Linux console it works fine but when we run the OOZIE task it fail saying that jar is missing or we do not have permission to read it. We are using CDH5.1 and the jar it companies about is one of the CDh jars that we checked and it is exists on all machines of the cluster
Any idea ?
Regards,
Noam
Information in this e-mail and its attachments is confidential and privileged under the TEOCO confidentiality terms that can be reviewed here<http://www.teoco.com/email-disclaimer>.
--
Artem
Re: bulk loading using OOZIE
Posted by Artem Ervits <ar...@gmail.com>.
from one of your earlier log posts, it says it can't find a jar in the
directory, please check the jar is there.
On Tue, Oct 7, 2014 at 1:53 AM, Bulvik, Noam <No...@teoco.com> wrote:
> We login to ozzie using mapred user.
>
> It looks like the ozzie task runs the script as user nobody , when I
> executed the command line of the script after loing as nobody it works fine
> so I do not think we need to explicit permission on some files
>
>
>
> I also do not think that we need to upload the CDH jars to ozzie (AFAIK) –
> do you?
>
>
>
> *From:* Artem Ervits [mailto:artemervits@gmail.com]
> *Sent:* Tuesday, October 7, 2014 12:51 AM
> *To:* user@phoenix.apache.org
> *Subject:* Re: bulk loading using OOZIE
>
>
>
> Double check that oozie user has permissions to read the required files in
> hdfs, for example, user running oozie wf is oozie then you should upload
> the jar and any property files to /user/oozie.
>
> On Oct 5, 2014 2:41 AM, "Bulvik, Noam" <No...@teoco.com> wrote:
>
> Hi,
>
>
>
> We are trying to do periodic bulk loading using OOZIE as scheduler. We
> impalement script task that should call the bulk loading command line with
> all needed parameter. When we run the script directly from Linux console
> it works fine but when we run the OOZIE task it fail saying that jar is
> missing or we do not have permission to read it. We are using CDH5.1 and
> the jar it companies about is one of the CDh jars that we checked and it is
> exists on all machines of the cluster
>
>
>
> Any idea ?
>
>
>
> Regards,
>
> Noam
>
> Information in this e-mail and its attachments is confidential and
> privileged under the TEOCO confidentiality terms that can be reviewed here
> <http://www.teoco.com/email-disclaimer>.
>
>
--
Artem
RE: bulk loading using OOZIE
Posted by "Bulvik, Noam" <No...@teoco.com>.
We login to ozzie using mapred user.
It looks like the ozzie task runs the script as user nobody , when I executed the command line of the script after loing as nobody it works fine so I do not think we need to explicit permission on some files
I also do not think that we need to upload the CDH jars to ozzie (AFAIK) – do you?
From: Artem Ervits [mailto:artemervits@gmail.com]
Sent: Tuesday, October 7, 2014 12:51 AM
To: user@phoenix.apache.org
Subject: Re: bulk loading using OOZIE
Double check that oozie user has permissions to read the required files in hdfs, for example, user running oozie wf is oozie then you should upload the jar and any property files to /user/oozie.
On Oct 5, 2014 2:41 AM, "Bulvik, Noam" <No...@teoco.com>> wrote:
Hi,
We are trying to do periodic bulk loading using OOZIE as scheduler. We impalement script task that should call the bulk loading command line with all needed parameter. When we run the script directly from Linux console it works fine but when we run the OOZIE task it fail saying that jar is missing or we do not have permission to read it. We are using CDH5.1 and the jar it companies about is one of the CDh jars that we checked and it is exists on all machines of the cluster
Any idea ?
Regards,
Noam
Information in this e-mail and its attachments is confidential and privileged under the TEOCO confidentiality terms that can be reviewed here<http://www.teoco.com/email-disclaimer>.
Re: bulk loading using OOZIE
Posted by Artem Ervits <ar...@gmail.com>.
Double check that oozie user has permissions to read the required files in
hdfs, for example, user running oozie wf is oozie then you should upload
the jar and any property files to /user/oozie.
On Oct 5, 2014 2:41 AM, "Bulvik, Noam" <No...@teoco.com> wrote:
> Hi,
>
>
>
> We are trying to do periodic bulk loading using OOZIE as scheduler. We
> impalement script task that should call the bulk loading command line with
> all needed parameter. When we run the script directly from Linux console
> it works fine but when we run the OOZIE task it fail saying that jar is
> missing or we do not have permission to read it. We are using CDH5.1 and
> the jar it companies about is one of the CDh jars that we checked and it is
> exists on all machines of the cluster
>
>
>
> Any idea ?
>
>
>
> Regards,
>
> Noam
> Information in this e-mail and its attachments is confidential and
> privileged under the TEOCO confidentiality terms that can be reviewed here
> <http://www.teoco.com/email-disclaimer>.
>