You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@oozie.apache.org by Harsh J <ha...@cloudera.com> on 2013/02/13 06:43:09 UTC
Re: Oozie - accessing files from local file system
Hi Shreehari,
I'd recommend using a list since my limited free time does not permit
me to make sure to reply to every single email sent to me personally.
Using a community mailing list also lets you get thoughts of several
experienced folks out there, than just one person :)
I'd recommend sending the question to the Oozie lists,
user@oozie.apache.org. I've added the list to this response. You can
subscribe to these lists via instructions at
http://oozie.apache.org/mail-lists.html.
>From a brief read, however, your problem is that Oozie may not
understand relative paths, given the user/location it runs that
command as. Try passing an absolute path, such as
"/user/shreehari/downloads/test/" for example, and verify?
On Wed, Feb 13, 2013 at 10:59 AM, <sh...@gmail.com> wrote:
> Hi Harsha,
>
> We are planning to use Oozie as work flow scheduler with Hadoop and Hive.
>
> But the problem we are facing is we are unable to access files stored in local file system to upload it in hive through Oozie.
>
> example: LOAD DATA LOCAL INPATH 'downloads/test/' INTO TABLE TEST;
>
> if i execute this in hive then it works fine, but same thing if run it through Oozie it throws error saying the path 'downloads/test/' as invalid path.
>
> As i understood Oozie run over hdfs, so it is expecting files in hdfs.
>
> I am new to hadoop and oozie. Is there anyway we can achieve this through Oozie?
>
> is there any better work flow scheduler we can use?
>
> Thanks
> Shreehari
--
Harsh J
Re: Oozie - accessing files from local file system
Posted by Alejandro Abdelnur <tu...@cloudera.com>.
I guess the HCAT integration (in hive and oozie, both WIP) would be the
answer for that.
On Wed, Feb 13, 2013 at 10:04 AM, Harish Krishnan <
harish.t.krishnan@gmail.com> wrote:
> I have also faced similar issue. Would appreciate a solution for this
> issue.
>
> Thanks & Regards,
> Harish.T.K
>
>
> On Wed, Feb 13, 2013 at 1:39 AM, Mohammad Islam <mi...@yahoo.com>
> wrote:
>
> > For data checking , did you consider coordinator in place of custom
> logic?
> >
> > Do you see any issue with external table in this respect?
> >
> > Regards,
> > Mohammad
> >
> >
> > ________________________________
> > From: Shreehari Padaki <sh...@gmail.com>
> > To: user@oozie.apache.org; Mohammad Islam <mi...@yahoo.com>
> > Cc: Harsh J <ha...@cloudera.com>
> > Sent: Wednesday, February 13, 2013 12:13 AM
> > Subject: Re: Oozie - accessing files from local file system
> >
> > yes, we can do that way but it adds one more layer and also requires some
> > extra time.
> >
> > Another thing is i want to schedule my hive queries once data is loaded
> in
> > hive tables. So now we are creating external table (to hive) in hdfs and
> > loading files through cron job and using some custom logic in oozie to
> > check data is uploaded in hdfs or not before it runs hive queries.
> >
> > So just wanted to check for other ways to do that.
> >
> > Thanks
> > Shreehari
> >
> >
> > On Wed, Feb 13, 2013 at 1:34 PM, Mohammad Islam <mi...@yahoo.com>
> > wrote:
> >
> > >
> > >
> > > Hi Shreehari,
> > > I think there is no way of executing this at this point.
> > > You are right, Oozie needs the file to be in hdfs.
> > > Is there a way of uploading the files in hdfs and then execute with
> LOAD
> > > command giving the hdfs path as the source.
> > >
> > > Regards,
> > > Mohammad
> > >
> > >
> > > ________________________________
> > > From: Shreehari Padaki <sh...@gmail.com>
> > > To: Harsh J <ha...@cloudera.com>
> > > Cc: user@oozie.apache.org
> > > Sent: Tuesday, February 12, 2013 11:53 PM
> > > Subject: Re: Oozie - accessing files from local file system
> > >
> > > Hi Harsh,
> > >
> > > We have tried giving absolute path but it didn't work. It always try to
> > > look for the path in hdfs.
> > >
> > > So what is the way to schedule a job to copy files from local file
> system
> > > to hdfs or hive? is this possible through oozie?
> > >
> > >
> > > Thanks
> > > Shreehari
> > >
> > > On Wed, Feb 13, 2013 at 11:13 AM, Harsh J <ha...@cloudera.com> wrote:
> > >
> > > > Hi Shreehari,
> > > >
> > > > I'd recommend using a list since my limited free time does not permit
> > > > me to make sure to reply to every single email sent to me personally.
> > > > Using a community mailing list also lets you get thoughts of several
> > > > experienced folks out there, than just one person :)
> > > >
> > > > I'd recommend sending the question to the Oozie lists,
> > > > user@oozie.apache.org. I've added the list to this response. You can
> > > > subscribe to these lists via instructions at
> > > > http://oozie.apache.org/mail-lists.html.
> > > >
> > > > From a brief read, however, your problem is that Oozie may not
> > > > understand relative paths, given the user/location it runs that
> > > > command as. Try passing an absolute path, such as
> > > > "/user/shreehari/downloads/test/" for example, and verify?
> > > >
> > > > On Wed, Feb 13, 2013 at 10:59 AM, <sh...@gmail.com> wrote:
> > > > > Hi Harsha,
> > > > >
> > > > > We are planning to use Oozie as work flow scheduler with Hadoop and
> > > Hive.
> > > > >
> > > > > But the problem we are facing is we are unable to access files
> stored
> > > in
> > > > local file system to upload it in hive through Oozie.
> > > > >
> > > > > example: LOAD DATA LOCAL INPATH 'downloads/test/' INTO TABLE TEST;
> > > > >
> > > > > if i execute this in hive then it works fine, but same thing if run
> > it
> > > > through Oozie it throws error saying the path 'downloads/test/' as
> > > invalid
> > > > path.
> > > > >
> > > > > As i understood Oozie run over hdfs, so it is expecting files in
> > hdfs.
> > > > >
> > > > > I am new to hadoop and oozie. Is there anyway we can achieve this
> > > > through Oozie?
> > > > >
> > > > > is there any better work flow scheduler we can use?
> > > > >
> > > > > Thanks
> > > > > Shreehari
> > > >
> > > >
> > > >
> > > > --
> > > > Harsh J
> > > >
> > >
> >
>
--
Alejandro
Re: Oozie - accessing files from local file system
Posted by Harish Krishnan <ha...@gmail.com>.
I have also faced similar issue. Would appreciate a solution for this issue.
Thanks & Regards,
Harish.T.K
On Wed, Feb 13, 2013 at 1:39 AM, Mohammad Islam <mi...@yahoo.com> wrote:
> For data checking , did you consider coordinator in place of custom logic?
>
> Do you see any issue with external table in this respect?
>
> Regards,
> Mohammad
>
>
> ________________________________
> From: Shreehari Padaki <sh...@gmail.com>
> To: user@oozie.apache.org; Mohammad Islam <mi...@yahoo.com>
> Cc: Harsh J <ha...@cloudera.com>
> Sent: Wednesday, February 13, 2013 12:13 AM
> Subject: Re: Oozie - accessing files from local file system
>
> yes, we can do that way but it adds one more layer and also requires some
> extra time.
>
> Another thing is i want to schedule my hive queries once data is loaded in
> hive tables. So now we are creating external table (to hive) in hdfs and
> loading files through cron job and using some custom logic in oozie to
> check data is uploaded in hdfs or not before it runs hive queries.
>
> So just wanted to check for other ways to do that.
>
> Thanks
> Shreehari
>
>
> On Wed, Feb 13, 2013 at 1:34 PM, Mohammad Islam <mi...@yahoo.com>
> wrote:
>
> >
> >
> > Hi Shreehari,
> > I think there is no way of executing this at this point.
> > You are right, Oozie needs the file to be in hdfs.
> > Is there a way of uploading the files in hdfs and then execute with LOAD
> > command giving the hdfs path as the source.
> >
> > Regards,
> > Mohammad
> >
> >
> > ________________________________
> > From: Shreehari Padaki <sh...@gmail.com>
> > To: Harsh J <ha...@cloudera.com>
> > Cc: user@oozie.apache.org
> > Sent: Tuesday, February 12, 2013 11:53 PM
> > Subject: Re: Oozie - accessing files from local file system
> >
> > Hi Harsh,
> >
> > We have tried giving absolute path but it didn't work. It always try to
> > look for the path in hdfs.
> >
> > So what is the way to schedule a job to copy files from local file system
> > to hdfs or hive? is this possible through oozie?
> >
> >
> > Thanks
> > Shreehari
> >
> > On Wed, Feb 13, 2013 at 11:13 AM, Harsh J <ha...@cloudera.com> wrote:
> >
> > > Hi Shreehari,
> > >
> > > I'd recommend using a list since my limited free time does not permit
> > > me to make sure to reply to every single email sent to me personally.
> > > Using a community mailing list also lets you get thoughts of several
> > > experienced folks out there, than just one person :)
> > >
> > > I'd recommend sending the question to the Oozie lists,
> > > user@oozie.apache.org. I've added the list to this response. You can
> > > subscribe to these lists via instructions at
> > > http://oozie.apache.org/mail-lists.html.
> > >
> > > From a brief read, however, your problem is that Oozie may not
> > > understand relative paths, given the user/location it runs that
> > > command as. Try passing an absolute path, such as
> > > "/user/shreehari/downloads/test/" for example, and verify?
> > >
> > > On Wed, Feb 13, 2013 at 10:59 AM, <sh...@gmail.com> wrote:
> > > > Hi Harsha,
> > > >
> > > > We are planning to use Oozie as work flow scheduler with Hadoop and
> > Hive.
> > > >
> > > > But the problem we are facing is we are unable to access files stored
> > in
> > > local file system to upload it in hive through Oozie.
> > > >
> > > > example: LOAD DATA LOCAL INPATH 'downloads/test/' INTO TABLE TEST;
> > > >
> > > > if i execute this in hive then it works fine, but same thing if run
> it
> > > through Oozie it throws error saying the path 'downloads/test/' as
> > invalid
> > > path.
> > > >
> > > > As i understood Oozie run over hdfs, so it is expecting files in
> hdfs.
> > > >
> > > > I am new to hadoop and oozie. Is there anyway we can achieve this
> > > through Oozie?
> > > >
> > > > is there any better work flow scheduler we can use?
> > > >
> > > > Thanks
> > > > Shreehari
> > >
> > >
> > >
> > > --
> > > Harsh J
> > >
> >
>
Re: Oozie - accessing files from local file system
Posted by Mohammad Islam <mi...@yahoo.com>.
For data checking , did you consider coordinator in place of custom logic?
Do you see any issue with external table in this respect?
Regards,
Mohammad
________________________________
From: Shreehari Padaki <sh...@gmail.com>
To: user@oozie.apache.org; Mohammad Islam <mi...@yahoo.com>
Cc: Harsh J <ha...@cloudera.com>
Sent: Wednesday, February 13, 2013 12:13 AM
Subject: Re: Oozie - accessing files from local file system
yes, we can do that way but it adds one more layer and also requires some
extra time.
Another thing is i want to schedule my hive queries once data is loaded in
hive tables. So now we are creating external table (to hive) in hdfs and
loading files through cron job and using some custom logic in oozie to
check data is uploaded in hdfs or not before it runs hive queries.
So just wanted to check for other ways to do that.
Thanks
Shreehari
On Wed, Feb 13, 2013 at 1:34 PM, Mohammad Islam <mi...@yahoo.com> wrote:
>
>
> Hi Shreehari,
> I think there is no way of executing this at this point.
> You are right, Oozie needs the file to be in hdfs.
> Is there a way of uploading the files in hdfs and then execute with LOAD
> command giving the hdfs path as the source.
>
> Regards,
> Mohammad
>
>
> ________________________________
> From: Shreehari Padaki <sh...@gmail.com>
> To: Harsh J <ha...@cloudera.com>
> Cc: user@oozie.apache.org
> Sent: Tuesday, February 12, 2013 11:53 PM
> Subject: Re: Oozie - accessing files from local file system
>
> Hi Harsh,
>
> We have tried giving absolute path but it didn't work. It always try to
> look for the path in hdfs.
>
> So what is the way to schedule a job to copy files from local file system
> to hdfs or hive? is this possible through oozie?
>
>
> Thanks
> Shreehari
>
> On Wed, Feb 13, 2013 at 11:13 AM, Harsh J <ha...@cloudera.com> wrote:
>
> > Hi Shreehari,
> >
> > I'd recommend using a list since my limited free time does not permit
> > me to make sure to reply to every single email sent to me personally.
> > Using a community mailing list also lets you get thoughts of several
> > experienced folks out there, than just one person :)
> >
> > I'd recommend sending the question to the Oozie lists,
> > user@oozie.apache.org. I've added the list to this response. You can
> > subscribe to these lists via instructions at
> > http://oozie.apache.org/mail-lists.html.
> >
> > From a brief read, however, your problem is that Oozie may not
> > understand relative paths, given the user/location it runs that
> > command as. Try passing an absolute path, such as
> > "/user/shreehari/downloads/test/" for example, and verify?
> >
> > On Wed, Feb 13, 2013 at 10:59 AM, <sh...@gmail.com> wrote:
> > > Hi Harsha,
> > >
> > > We are planning to use Oozie as work flow scheduler with Hadoop and
> Hive.
> > >
> > > But the problem we are facing is we are unable to access files stored
> in
> > local file system to upload it in hive through Oozie.
> > >
> > > example: LOAD DATA LOCAL INPATH 'downloads/test/' INTO TABLE TEST;
> > >
> > > if i execute this in hive then it works fine, but same thing if run it
> > through Oozie it throws error saying the path 'downloads/test/' as
> invalid
> > path.
> > >
> > > As i understood Oozie run over hdfs, so it is expecting files in hdfs.
> > >
> > > I am new to hadoop and oozie. Is there anyway we can achieve this
> > through Oozie?
> > >
> > > is there any better work flow scheduler we can use?
> > >
> > > Thanks
> > > Shreehari
> >
> >
> >
> > --
> > Harsh J
> >
>
Re: Oozie - accessing files from local file system
Posted by Shreehari Padaki <sh...@gmail.com>.
yes, we can do that way but it adds one more layer and also requires some
extra time.
Another thing is i want to schedule my hive queries once data is loaded in
hive tables. So now we are creating external table (to hive) in hdfs and
loading files through cron job and using some custom logic in oozie to
check data is uploaded in hdfs or not before it runs hive queries.
So just wanted to check for other ways to do that.
Thanks
Shreehari
On Wed, Feb 13, 2013 at 1:34 PM, Mohammad Islam <mi...@yahoo.com> wrote:
>
>
> Hi Shreehari,
> I think there is no way of executing this at this point.
> You are right, Oozie needs the file to be in hdfs.
> Is there a way of uploading the files in hdfs and then execute with LOAD
> command giving the hdfs path as the source.
>
> Regards,
> Mohammad
>
>
> ________________________________
> From: Shreehari Padaki <sh...@gmail.com>
> To: Harsh J <ha...@cloudera.com>
> Cc: user@oozie.apache.org
> Sent: Tuesday, February 12, 2013 11:53 PM
> Subject: Re: Oozie - accessing files from local file system
>
> Hi Harsh,
>
> We have tried giving absolute path but it didn't work. It always try to
> look for the path in hdfs.
>
> So what is the way to schedule a job to copy files from local file system
> to hdfs or hive? is this possible through oozie?
>
>
> Thanks
> Shreehari
>
> On Wed, Feb 13, 2013 at 11:13 AM, Harsh J <ha...@cloudera.com> wrote:
>
> > Hi Shreehari,
> >
> > I'd recommend using a list since my limited free time does not permit
> > me to make sure to reply to every single email sent to me personally.
> > Using a community mailing list also lets you get thoughts of several
> > experienced folks out there, than just one person :)
> >
> > I'd recommend sending the question to the Oozie lists,
> > user@oozie.apache.org. I've added the list to this response. You can
> > subscribe to these lists via instructions at
> > http://oozie.apache.org/mail-lists.html.
> >
> > From a brief read, however, your problem is that Oozie may not
> > understand relative paths, given the user/location it runs that
> > command as. Try passing an absolute path, such as
> > "/user/shreehari/downloads/test/" for example, and verify?
> >
> > On Wed, Feb 13, 2013 at 10:59 AM, <sh...@gmail.com> wrote:
> > > Hi Harsha,
> > >
> > > We are planning to use Oozie as work flow scheduler with Hadoop and
> Hive.
> > >
> > > But the problem we are facing is we are unable to access files stored
> in
> > local file system to upload it in hive through Oozie.
> > >
> > > example: LOAD DATA LOCAL INPATH 'downloads/test/' INTO TABLE TEST;
> > >
> > > if i execute this in hive then it works fine, but same thing if run it
> > through Oozie it throws error saying the path 'downloads/test/' as
> invalid
> > path.
> > >
> > > As i understood Oozie run over hdfs, so it is expecting files in hdfs.
> > >
> > > I am new to hadoop and oozie. Is there anyway we can achieve this
> > through Oozie?
> > >
> > > is there any better work flow scheduler we can use?
> > >
> > > Thanks
> > > Shreehari
> >
> >
> >
> > --
> > Harsh J
> >
>
Re: Oozie - accessing files from local file system
Posted by Mohammad Islam <mi...@yahoo.com>.
Hi Shreehari,
I think there is no way of executing this at this point.
You are right, Oozie needs the file to be in hdfs.
Is there a way of uploading the files in hdfs and then execute with LOAD command giving the hdfs path as the source.
Regards,
Mohammad
________________________________
From: Shreehari Padaki <sh...@gmail.com>
To: Harsh J <ha...@cloudera.com>
Cc: user@oozie.apache.org
Sent: Tuesday, February 12, 2013 11:53 PM
Subject: Re: Oozie - accessing files from local file system
Hi Harsh,
We have tried giving absolute path but it didn't work. It always try to
look for the path in hdfs.
So what is the way to schedule a job to copy files from local file system
to hdfs or hive? is this possible through oozie?
Thanks
Shreehari
On Wed, Feb 13, 2013 at 11:13 AM, Harsh J <ha...@cloudera.com> wrote:
> Hi Shreehari,
>
> I'd recommend using a list since my limited free time does not permit
> me to make sure to reply to every single email sent to me personally.
> Using a community mailing list also lets you get thoughts of several
> experienced folks out there, than just one person :)
>
> I'd recommend sending the question to the Oozie lists,
> user@oozie.apache.org. I've added the list to this response. You can
> subscribe to these lists via instructions at
> http://oozie.apache.org/mail-lists.html.
>
> From a brief read, however, your problem is that Oozie may not
> understand relative paths, given the user/location it runs that
> command as. Try passing an absolute path, such as
> "/user/shreehari/downloads/test/" for example, and verify?
>
> On Wed, Feb 13, 2013 at 10:59 AM, <sh...@gmail.com> wrote:
> > Hi Harsha,
> >
> > We are planning to use Oozie as work flow scheduler with Hadoop and Hive.
> >
> > But the problem we are facing is we are unable to access files stored in
> local file system to upload it in hive through Oozie.
> >
> > example: LOAD DATA LOCAL INPATH 'downloads/test/' INTO TABLE TEST;
> >
> > if i execute this in hive then it works fine, but same thing if run it
> through Oozie it throws error saying the path 'downloads/test/' as invalid
> path.
> >
> > As i understood Oozie run over hdfs, so it is expecting files in hdfs.
> >
> > I am new to hadoop and oozie. Is there anyway we can achieve this
> through Oozie?
> >
> > is there any better work flow scheduler we can use?
> >
> > Thanks
> > Shreehari
>
>
>
> --
> Harsh J
>
Re: Oozie - accessing files from local file system
Posted by Shreehari Padaki <sh...@gmail.com>.
Hi Harsh,
We have tried giving absolute path but it didn't work. It always try to
look for the path in hdfs.
So what is the way to schedule a job to copy files from local file system
to hdfs or hive? is this possible through oozie?
Thanks
Shreehari
On Wed, Feb 13, 2013 at 11:13 AM, Harsh J <ha...@cloudera.com> wrote:
> Hi Shreehari,
>
> I'd recommend using a list since my limited free time does not permit
> me to make sure to reply to every single email sent to me personally.
> Using a community mailing list also lets you get thoughts of several
> experienced folks out there, than just one person :)
>
> I'd recommend sending the question to the Oozie lists,
> user@oozie.apache.org. I've added the list to this response. You can
> subscribe to these lists via instructions at
> http://oozie.apache.org/mail-lists.html.
>
> From a brief read, however, your problem is that Oozie may not
> understand relative paths, given the user/location it runs that
> command as. Try passing an absolute path, such as
> "/user/shreehari/downloads/test/" for example, and verify?
>
> On Wed, Feb 13, 2013 at 10:59 AM, <sh...@gmail.com> wrote:
> > Hi Harsha,
> >
> > We are planning to use Oozie as work flow scheduler with Hadoop and Hive.
> >
> > But the problem we are facing is we are unable to access files stored in
> local file system to upload it in hive through Oozie.
> >
> > example: LOAD DATA LOCAL INPATH 'downloads/test/' INTO TABLE TEST;
> >
> > if i execute this in hive then it works fine, but same thing if run it
> through Oozie it throws error saying the path 'downloads/test/' as invalid
> path.
> >
> > As i understood Oozie run over hdfs, so it is expecting files in hdfs.
> >
> > I am new to hadoop and oozie. Is there anyway we can achieve this
> through Oozie?
> >
> > is there any better work flow scheduler we can use?
> >
> > Thanks
> > Shreehari
>
>
>
> --
> Harsh J
>