You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@oozie.apache.org by hitarth trivedi <t....@gmail.com> on 2015/03/05 20:36:51 UTC
Hbase table snapshot as input to mapreduce in oozie workflow
Hi,
I am trying to run a mapreduce job on oozie which takes hbase table
snapshot as its input. Currently I am able to do it (not using oozie
workflow) using org.apache.hadoop.hbase.mapreduce.TableMapReduceUtil which
provides me with a method initTableSnapshotMapperJob().
Is there a similar option in oozie, where I can specify hbase SNAPSHOT as
input to my map reduce job. If not what are the other options ?
Thanks,
Hitrix
Re: Hbase table snapshot as input to mapreduce in oozie workflow
Posted by Robert Kanter <rk...@cloudera.com>.
I forget why exactly. But looking at the code now, I think it's because we
were already using a JobConf. Plus, a JobConf is a subclass of
Configuration, so you should be able to use it in any methods that take a
Configuration, but not the other way around, so this is a little more
flexible.
If you want to use the Job object, you'll probably have to do something
like this:
Job j = new Job(jobConf);
// do your Job-specific stuff here
jobConf.clear();
for (Iterator<Map.Entry<String, String>> it =
j.getConfiguration().iterator(); it.hasNext(); ) {
Map.Entry<String, String> next = it.next();
jobConf.set(next.getKey(), next.getValue());
}
I know that's kind of ugly. There doesn't seem to be a nicer way to copy
the configs from the Job object back to the jobConf.
- Robert
On Tue, Mar 10, 2015 at 8:50 AM, hitarth trivedi <t....@gmail.com>
wrote:
> Hi Robert,
>
> I applied the patch on top of my Oozie. Now the issue with your patch is
> that all my MapReduce driver classes use the new mapreduce api and thus
> uses org.apache.hadoop.mapreduce.Job to configure my jobs. But in your
> OozieActionConfigurator interface you have a configure method which looks
> like void configure(org.apache.hadoop.mapred.JobConf jobConf), which forces
> me to either convert my Job object to JobConf, or run my Mapreduce Job
> using the old mapred api.
>
> Is there any reason why the conigure method wasn't designed in the way it
> was initially mentioned in the Apache Issue ticket (OOZIE-1954
> <https://issues.apache.org/jira/browse/OOZIE-1954>) public void
> updateOozieActionConfiguration(Configuration conf); ?
>
> Thanks,
> hitrix
>
> On Fri, Mar 6, 2015 at 1:04 PM, Robert Kanter <rk...@cloudera.com>
> wrote:
>
>> We haven't started planning the next Apache Oozie release yet, though I
>> imagine it will make it into that release, whenever that is. In the
>> meantime, you can try applying the patch at OOZIE-1954 on top of Oozie
>> 4.1.0.
>>
>> - Robert
>>
>> On Fri, Mar 6, 2015 at 7:46 AM, hitarth trivedi <t....@gmail.com>
>> wrote:
>>
>>> Also when can we expect your feature ( OOZIE-1954 <
>>> https://issues.apache.org/jira/browse/OOZIE-1954>) to be available in
>>> release.
>>>
>>> On Fri, Mar 6, 2015 at 9:34 AM, hitarth trivedi <t....@gmail.com>
>>> wrote:
>>>
>>>> Hi Robert,
>>>>
>>>> Thank you for that information. That is exactly what I was looking for.
>>>> But I am using oozie 4.1.0 release, and as you mentioned this feature is
>>>> not present in the current release. Is there a workaround for the same ?
>>>>
>>>> Thanks,
>>>> hitrix
>>>>
>>>> On Thu, Mar 5, 2015 at 5:27 PM, Robert Kanter <rk...@cloudera.com>
>>>> wrote:
>>>>
>>>>> If you're not using CDH, you'll want to make sure whichever Oozie
>>>>> you're
>>>>> using has OOZIE-1954 <https://issues.apache.org/jira/browse/OOZIE-1954
>>>>> >,
>>>>> which is what adds this feature. It's currently not in any Apache
>>>>> release
>>>>> yet.
>>>>>
>>>>> On Thu, Mar 5, 2015 at 2:24 PM, Robert Kanter <rk...@cloudera.com>
>>>>> wrote:
>>>>>
>>>>> > Hi,
>>>>> >
>>>>> > Starting with CDH 5.2.0, there is now a way to use Java to configure
>>>>> the
>>>>> > MapReduce action. So, you could simply write some Java code that
>>>>> calls the
>>>>> > initTableSnapshotMapperJob method on the action configuration. See
>>>>> the
>>>>> > docs here for more details and an example:
>>>>> >
>>>>> >
>>>>> http://archive.cloudera.com/cdh5/cdh/5/oozie/WorkflowFunctionalSpec.html#a3.2.2.2_Configuring_the_MapReduce_action_with_Java_code
>>>>> > Don't forget to include the necessary HBase jars with your workflow.
>>>>> >
>>>>> > - Robert
>>>>> >
>>>>> >
>>>>> >
>>>>> >
>>>>> > On Thu, Mar 5, 2015 at 11:36 AM, hitarth trivedi <
>>>>> t.hitarth@gmail.com>
>>>>> > wrote:
>>>>> >
>>>>> >> Hi,
>>>>> >>
>>>>> >> I am trying to run a mapreduce job on oozie which takes hbase table
>>>>> >> snapshot as its input. Currently I am able to do it (not using oozie
>>>>> >> workflow) using
>>>>> org.apache.hadoop.hbase.mapreduce.TableMapReduceUtil which
>>>>> >> provides me with a method initTableSnapshotMapperJob().
>>>>> >>
>>>>> >> Is there a similar option in oozie, where I can specify hbase
>>>>> SNAPSHOT as
>>>>> >> input to my map reduce job. If not what are the other options ?
>>>>> >>
>>>>> >> Thanks,
>>>>> >> Hitrix
>>>>> >>
>>>>> >
>>>>> >
>>>>>
>>>>
>>>>
>>>
>>
>
Re: Hbase table snapshot as input to mapreduce in oozie workflow
Posted by hitarth trivedi <t....@gmail.com>.
Hi Robert,
I applied the patch on top of my Oozie. Now the issue with your patch is
that all my MapReduce driver classes use the new mapreduce api and thus
uses org.apache.hadoop.mapreduce.Job to configure my jobs. But in your
OozieActionConfigurator interface you have a configure method which looks
like void configure(org.apache.hadoop.mapred.JobConf jobConf), which forces
me to either convert my Job object to JobConf, or run my Mapreduce Job
using the old mapred api.
Is there any reason why the conigure method wasn't designed in the way it
was initially mentioned in the Apache Issue ticket (OOZIE-1954
<https://issues.apache.org/jira/browse/OOZIE-1954>) public void
updateOozieActionConfiguration(Configuration conf); ?
Thanks,
hitrix
On Fri, Mar 6, 2015 at 1:04 PM, Robert Kanter <rk...@cloudera.com> wrote:
> We haven't started planning the next Apache Oozie release yet, though I
> imagine it will make it into that release, whenever that is. In the
> meantime, you can try applying the patch at OOZIE-1954 on top of Oozie
> 4.1.0.
>
> - Robert
>
> On Fri, Mar 6, 2015 at 7:46 AM, hitarth trivedi <t....@gmail.com>
> wrote:
>
>> Also when can we expect your feature ( OOZIE-1954 <
>> https://issues.apache.org/jira/browse/OOZIE-1954>) to be available in
>> release.
>>
>> On Fri, Mar 6, 2015 at 9:34 AM, hitarth trivedi <t....@gmail.com>
>> wrote:
>>
>>> Hi Robert,
>>>
>>> Thank you for that information. That is exactly what I was looking for.
>>> But I am using oozie 4.1.0 release, and as you mentioned this feature is
>>> not present in the current release. Is there a workaround for the same ?
>>>
>>> Thanks,
>>> hitrix
>>>
>>> On Thu, Mar 5, 2015 at 5:27 PM, Robert Kanter <rk...@cloudera.com>
>>> wrote:
>>>
>>>> If you're not using CDH, you'll want to make sure whichever Oozie you're
>>>> using has OOZIE-1954 <https://issues.apache.org/jira/browse/OOZIE-1954
>>>> >,
>>>> which is what adds this feature. It's currently not in any Apache
>>>> release
>>>> yet.
>>>>
>>>> On Thu, Mar 5, 2015 at 2:24 PM, Robert Kanter <rk...@cloudera.com>
>>>> wrote:
>>>>
>>>> > Hi,
>>>> >
>>>> > Starting with CDH 5.2.0, there is now a way to use Java to configure
>>>> the
>>>> > MapReduce action. So, you could simply write some Java code that
>>>> calls the
>>>> > initTableSnapshotMapperJob method on the action configuration. See
>>>> the
>>>> > docs here for more details and an example:
>>>> >
>>>> >
>>>> http://archive.cloudera.com/cdh5/cdh/5/oozie/WorkflowFunctionalSpec.html#a3.2.2.2_Configuring_the_MapReduce_action_with_Java_code
>>>> > Don't forget to include the necessary HBase jars with your workflow.
>>>> >
>>>> > - Robert
>>>> >
>>>> >
>>>> >
>>>> >
>>>> > On Thu, Mar 5, 2015 at 11:36 AM, hitarth trivedi <t.hitarth@gmail.com
>>>> >
>>>> > wrote:
>>>> >
>>>> >> Hi,
>>>> >>
>>>> >> I am trying to run a mapreduce job on oozie which takes hbase table
>>>> >> snapshot as its input. Currently I am able to do it (not using oozie
>>>> >> workflow) using org.apache.hadoop.hbase.mapreduce.TableMapReduceUtil
>>>> which
>>>> >> provides me with a method initTableSnapshotMapperJob().
>>>> >>
>>>> >> Is there a similar option in oozie, where I can specify hbase
>>>> SNAPSHOT as
>>>> >> input to my map reduce job. If not what are the other options ?
>>>> >>
>>>> >> Thanks,
>>>> >> Hitrix
>>>> >>
>>>> >
>>>> >
>>>>
>>>
>>>
>>
>
Re: Hbase table snapshot as input to mapreduce in oozie workflow
Posted by Robert Kanter <rk...@cloudera.com>.
We haven't started planning the next Apache Oozie release yet, though I
imagine it will make it into that release, whenever that is. In the
meantime, you can try applying the patch at OOZIE-1954 on top of Oozie
4.1.0.
- Robert
On Fri, Mar 6, 2015 at 7:46 AM, hitarth trivedi <t....@gmail.com> wrote:
> Also when can we expect your feature ( OOZIE-1954 <
> https://issues.apache.org/jira/browse/OOZIE-1954>) to be available in
> release.
>
> On Fri, Mar 6, 2015 at 9:34 AM, hitarth trivedi <t....@gmail.com>
> wrote:
>
>> Hi Robert,
>>
>> Thank you for that information. That is exactly what I was looking for.
>> But I am using oozie 4.1.0 release, and as you mentioned this feature is
>> not present in the current release. Is there a workaround for the same ?
>>
>> Thanks,
>> hitrix
>>
>> On Thu, Mar 5, 2015 at 5:27 PM, Robert Kanter <rk...@cloudera.com>
>> wrote:
>>
>>> If you're not using CDH, you'll want to make sure whichever Oozie you're
>>> using has OOZIE-1954 <https://issues.apache.org/jira/browse/OOZIE-1954>,
>>> which is what adds this feature. It's currently not in any Apache
>>> release
>>> yet.
>>>
>>> On Thu, Mar 5, 2015 at 2:24 PM, Robert Kanter <rk...@cloudera.com>
>>> wrote:
>>>
>>> > Hi,
>>> >
>>> > Starting with CDH 5.2.0, there is now a way to use Java to configure
>>> the
>>> > MapReduce action. So, you could simply write some Java code that
>>> calls the
>>> > initTableSnapshotMapperJob method on the action configuration. See the
>>> > docs here for more details and an example:
>>> >
>>> >
>>> http://archive.cloudera.com/cdh5/cdh/5/oozie/WorkflowFunctionalSpec.html#a3.2.2.2_Configuring_the_MapReduce_action_with_Java_code
>>> > Don't forget to include the necessary HBase jars with your workflow.
>>> >
>>> > - Robert
>>> >
>>> >
>>> >
>>> >
>>> > On Thu, Mar 5, 2015 at 11:36 AM, hitarth trivedi <t....@gmail.com>
>>> > wrote:
>>> >
>>> >> Hi,
>>> >>
>>> >> I am trying to run a mapreduce job on oozie which takes hbase table
>>> >> snapshot as its input. Currently I am able to do it (not using oozie
>>> >> workflow) using org.apache.hadoop.hbase.mapreduce.TableMapReduceUtil
>>> which
>>> >> provides me with a method initTableSnapshotMapperJob().
>>> >>
>>> >> Is there a similar option in oozie, where I can specify hbase
>>> SNAPSHOT as
>>> >> input to my map reduce job. If not what are the other options ?
>>> >>
>>> >> Thanks,
>>> >> Hitrix
>>> >>
>>> >
>>> >
>>>
>>
>>
>
Re: Hbase table snapshot as input to mapreduce in oozie workflow
Posted by hitarth trivedi <t....@gmail.com>.
Also when can we expect your feature ( OOZIE-1954 <
https://issues.apache.org/jira/browse/OOZIE-1954>) to be available in
release.
On Fri, Mar 6, 2015 at 9:34 AM, hitarth trivedi <t....@gmail.com> wrote:
> Hi Robert,
>
> Thank you for that information. That is exactly what I was looking for.
> But I am using oozie 4.1.0 release, and as you mentioned this feature is
> not present in the current release. Is there a workaround for the same ?
>
> Thanks,
> hitrix
>
> On Thu, Mar 5, 2015 at 5:27 PM, Robert Kanter <rk...@cloudera.com>
> wrote:
>
>> If you're not using CDH, you'll want to make sure whichever Oozie you're
>> using has OOZIE-1954 <https://issues.apache.org/jira/browse/OOZIE-1954>,
>> which is what adds this feature. It's currently not in any Apache release
>> yet.
>>
>> On Thu, Mar 5, 2015 at 2:24 PM, Robert Kanter <rk...@cloudera.com>
>> wrote:
>>
>> > Hi,
>> >
>> > Starting with CDH 5.2.0, there is now a way to use Java to configure the
>> > MapReduce action. So, you could simply write some Java code that calls
>> the
>> > initTableSnapshotMapperJob method on the action configuration. See the
>> > docs here for more details and an example:
>> >
>> >
>> http://archive.cloudera.com/cdh5/cdh/5/oozie/WorkflowFunctionalSpec.html#a3.2.2.2_Configuring_the_MapReduce_action_with_Java_code
>> > Don't forget to include the necessary HBase jars with your workflow.
>> >
>> > - Robert
>> >
>> >
>> >
>> >
>> > On Thu, Mar 5, 2015 at 11:36 AM, hitarth trivedi <t....@gmail.com>
>> > wrote:
>> >
>> >> Hi,
>> >>
>> >> I am trying to run a mapreduce job on oozie which takes hbase table
>> >> snapshot as its input. Currently I am able to do it (not using oozie
>> >> workflow) using org.apache.hadoop.hbase.mapreduce.TableMapReduceUtil
>> which
>> >> provides me with a method initTableSnapshotMapperJob().
>> >>
>> >> Is there a similar option in oozie, where I can specify hbase SNAPSHOT
>> as
>> >> input to my map reduce job. If not what are the other options ?
>> >>
>> >> Thanks,
>> >> Hitrix
>> >>
>> >
>> >
>>
>
>
Re: Hbase table snapshot as input to mapreduce in oozie workflow
Posted by hitarth trivedi <t....@gmail.com>.
Hi Robert,
Thank you for that information. That is exactly what I was looking for. But
I am using oozie 4.1.0 release, and as you mentioned this feature is not
present in the current release. Is there a workaround for the same ?
Thanks,
hitrix
On Thu, Mar 5, 2015 at 5:27 PM, Robert Kanter <rk...@cloudera.com> wrote:
> If you're not using CDH, you'll want to make sure whichever Oozie you're
> using has OOZIE-1954 <https://issues.apache.org/jira/browse/OOZIE-1954>,
> which is what adds this feature. It's currently not in any Apache release
> yet.
>
> On Thu, Mar 5, 2015 at 2:24 PM, Robert Kanter <rk...@cloudera.com>
> wrote:
>
> > Hi,
> >
> > Starting with CDH 5.2.0, there is now a way to use Java to configure the
> > MapReduce action. So, you could simply write some Java code that calls
> the
> > initTableSnapshotMapperJob method on the action configuration. See the
> > docs here for more details and an example:
> >
> >
> http://archive.cloudera.com/cdh5/cdh/5/oozie/WorkflowFunctionalSpec.html#a3.2.2.2_Configuring_the_MapReduce_action_with_Java_code
> > Don't forget to include the necessary HBase jars with your workflow.
> >
> > - Robert
> >
> >
> >
> >
> > On Thu, Mar 5, 2015 at 11:36 AM, hitarth trivedi <t....@gmail.com>
> > wrote:
> >
> >> Hi,
> >>
> >> I am trying to run a mapreduce job on oozie which takes hbase table
> >> snapshot as its input. Currently I am able to do it (not using oozie
> >> workflow) using org.apache.hadoop.hbase.mapreduce.TableMapReduceUtil
> which
> >> provides me with a method initTableSnapshotMapperJob().
> >>
> >> Is there a similar option in oozie, where I can specify hbase SNAPSHOT
> as
> >> input to my map reduce job. If not what are the other options ?
> >>
> >> Thanks,
> >> Hitrix
> >>
> >
> >
>
Re: Hbase table snapshot as input to mapreduce in oozie workflow
Posted by Robert Kanter <rk...@cloudera.com>.
If you're not using CDH, you'll want to make sure whichever Oozie you're
using has OOZIE-1954 <https://issues.apache.org/jira/browse/OOZIE-1954>,
which is what adds this feature. It's currently not in any Apache release
yet.
On Thu, Mar 5, 2015 at 2:24 PM, Robert Kanter <rk...@cloudera.com> wrote:
> Hi,
>
> Starting with CDH 5.2.0, there is now a way to use Java to configure the
> MapReduce action. So, you could simply write some Java code that calls the
> initTableSnapshotMapperJob method on the action configuration. See the
> docs here for more details and an example:
>
> http://archive.cloudera.com/cdh5/cdh/5/oozie/WorkflowFunctionalSpec.html#a3.2.2.2_Configuring_the_MapReduce_action_with_Java_code
> Don't forget to include the necessary HBase jars with your workflow.
>
> - Robert
>
>
>
>
> On Thu, Mar 5, 2015 at 11:36 AM, hitarth trivedi <t....@gmail.com>
> wrote:
>
>> Hi,
>>
>> I am trying to run a mapreduce job on oozie which takes hbase table
>> snapshot as its input. Currently I am able to do it (not using oozie
>> workflow) using org.apache.hadoop.hbase.mapreduce.TableMapReduceUtil which
>> provides me with a method initTableSnapshotMapperJob().
>>
>> Is there a similar option in oozie, where I can specify hbase SNAPSHOT as
>> input to my map reduce job. If not what are the other options ?
>>
>> Thanks,
>> Hitrix
>>
>
>
Re: Hbase table snapshot as input to mapreduce in oozie workflow
Posted by Robert Kanter <rk...@cloudera.com>.
Hi,
Starting with CDH 5.2.0, there is now a way to use Java to configure the
MapReduce action. So, you could simply write some Java code that calls the
initTableSnapshotMapperJob method on the action configuration. See the
docs here for more details and an example:
http://archive.cloudera.com/cdh5/cdh/5/oozie/WorkflowFunctionalSpec.html#a3.2.2.2_Configuring_the_MapReduce_action_with_Java_code
Don't forget to include the necessary HBase jars with your workflow.
- Robert
On Thu, Mar 5, 2015 at 11:36 AM, hitarth trivedi <t....@gmail.com>
wrote:
> Hi,
>
> I am trying to run a mapreduce job on oozie which takes hbase table
> snapshot as its input. Currently I am able to do it (not using oozie
> workflow) using org.apache.hadoop.hbase.mapreduce.TableMapReduceUtil which
> provides me with a method initTableSnapshotMapperJob().
>
> Is there a similar option in oozie, where I can specify hbase SNAPSHOT as
> input to my map reduce job. If not what are the other options ?
>
> Thanks,
> Hitrix
>