You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-user@hadoop.apache.org by Radhe Radhe <ra...@live.com> on 2014/04/14 15:19:46 UTC
Doubt regarding Binary Compatibility\Source Compatibility with old
*mapred* APIs and new *mapreduce* APIs in Hadoop
Hello People,
As per the Apache site http://hadoop.apache.org/docs/r2.3.0/hadoop-mapreduce-client/hadoop-mapreduce-client-core/MapReduce_Compatibility_Hadoop1_Hadoop2.html Binary Compatibility====================First, we ensure binary compatibility to the applications that use old mapred APIs. This means that applications which were built against MRv1 mapred APIs can run directly on YARN without recompilation, merely by pointing them to an Apache Hadoop 2.x cluster via configuration.
Source Compatibility====================We cannot ensure complete binary compatibility with the applications that use mapreduce APIs, as these APIs have evolved a lot since MRv1. However, we ensure source compatibility for mapreduce APIs that break binary compatibility. In other words, users should recompile their applications that use mapreduce APIs against MRv2 jars. One notable binary incompatibility break is Counter and CounterGroup.
For "Binary Compatibility" I understand that if I had build a MR job with old *mapred* APIs then they can be run directly on YARN without and changes.
Can anybody explain what do we mean by "Source Compatibility" here and also a use case where one will need it?
Does that mean code changes if I already have a MR job source code written with with old *mapred* APIs and I need to make some changes to it to run in then I need to use the new "mapreduce* API and generate the new binaries?
Thanks,-RR
Re: Doubt regarding Binary Compatibility\Source Compatibility with
old *mapred* APIs and new *mapreduce* APIs in Hadoop
Posted by Zhijie Shen <zs...@hortonworks.com>.
bq. Regarding #3 if I have ONLY the binaries i.e. jar file (compiled\build
against old MRv1 mapred APIS)
Which APIs are you talking about, *mapred* or *mapreduce*? In #3, I was
saying about *mapreduce*. If this is the case, you may be in the trouble
unfortunately, because MRv2 has evolved so much in *mapreduce *APIs that
it's difficult to ensure binary compatibility. Anyway, you should still try
your luck, as your binaries may not use the incompatible APIs. On the other
hand, if you meant *mapred* APIs instead, you binaries should just work.
- Zhijie
On Tue, Apr 15, 2014 at 1:35 PM, Radhe Radhe
<ra...@live.com>wrote:
> Thanks Zhijie for the explanation.
>
> Regarding #3 if I have ONLY the binaries i.e. jar file (compiled\build
> against old MRv1 *mapred* APIS) then how can I compile it since I don't
> have the source code i.e. Java files. All I can do with binaries i.e. jar
> file is execute it.
>
> -RR
> ------------------------------
> Date: Tue, 15 Apr 2014 13:03:53 -0700
>
> Subject: Re: Doubt regarding Binary Compatibility\Source Compatibility
> with old *mapred* APIs and new *mapreduce* APIs in Hadoop
> From: zshen@hortonworks.com
> To: user@hadoop.apache.org
>
>
> 1. If you have the binaries that were compiled against MRv1 *mapred*libs, it should just work with MRv2.
> 2. If you have the source code that refers to MRv1 *mapred* libs, it
> should be compilable without code changes. Of course, you're free to change
> your code.
> 3. If you have the binaries that were compiled against MRv1 *mapreduce* libs,
> it may not be executable directly with MRv2, but you should able to compile
> it against MRv2 *mapreduce* libs without code changes, and execute it.
>
> - Zhijie
>
>
> On Tue, Apr 15, 2014 at 12:44 PM, Radhe Radhe <
> radhe.krishna.radhe@live.com> wrote:
>
> Thanks John for your comments,
>
> I believe MRv2 has support for both the old *mapred* APIs and new
> *mapreduce* APIs.
>
> I see this way:
> [1.] One may have binaries i.e. jar file of the M\R program that used old
> *mapred* APIs
> This will work directly on MRv2(YARN).
>
> [2.] One may have the source code i.e. Java Programs of the M\R program
> that used old *mapred* APIs
> For this I need to recompile and generate the binaries i.e. jar file.
> Do I have to change the old *org.apache.hadoop.mapred* APIs to new *
> org.apache.hadoop.mapreduce* APIs? or No code changes are needed?
>
> -RR
>
> > Date: Mon, 14 Apr 2014 10:37:56 -0400
> > Subject: Re: Doubt regarding Binary Compatibility\Source Compatibility
> with old *mapred* APIs and new *mapreduce* APIs in Hadoop
> > From: john.meagher@gmail.com
> > To: user@hadoop.apache.org
>
> >
> > Also, "Source Compatibility" also means ONLY a recompile is needed.
> > No code changes should be needed.
> >
> > On Mon, Apr 14, 2014 at 10:37 AM, John Meagher <jo...@gmail.com>
> wrote:
> > > Source Compatibility = you need to recompile and use the new version
> > > as part of the compilation
> > >
> > > Binary Compatibility = you can take something compiled against the old
> > > version and run it on the new version
> > >
> > > On Mon, Apr 14, 2014 at 9:19 AM, Radhe Radhe
> > > <ra...@live.com> wrote:
> > >> Hello People,
> > >>
> > >> As per the Apache site
> > >>
> http://hadoop.apache.org/docs/r2.3.0/hadoop-mapreduce-client/hadoop-mapreduce-client-core/MapReduce_Compatibility_Hadoop1_Hadoop2.html
> > >>
> > >> Binary Compatibility
> > >> ====================
> > >> First, we ensure binary compatibility to the applications that use old
> > >> mapred APIs. This means that applications which were built against
> MRv1
> > >> mapred APIs can run directly on YARN without recompilation, merely by
> > >> pointing them to an Apache Hadoop 2.x cluster via configuration.
> > >>
> > >> Source Compatibility
> > >> ====================
> > >> We cannot ensure complete binary compatibility with the applications
> that
> > >> use mapreduce APIs, as these APIs have evolved a lot since MRv1.
> However, we
> > >> ensure source compatibility for mapreduce APIs that break binary
> > >> compatibility. In other words, users should recompile their
> applications
> > >> that use mapreduce APIs against MRv2 jars. One notable binary
> > >> incompatibility break is Counter and CounterGroup.
> > >>
> > >> For "Binary Compatibility" I understand that if I had build a MR job
> with
> > >> old *mapred* APIs then they can be run directly on YARN without and
> changes.
> > >>
> > >> Can anybody explain what do we mean by "Source Compatibility" here
> and also
> > >> a use case where one will need it?
> > >>
> > >> Does that mean code changes if I already have a MR job source code
> written
> > >> with with old *mapred* APIs and I need to make some changes to it to
> run in
> > >> then I need to use the new "mapreduce* API and generate the new
> binaries?
> > >>
> > >> Thanks,
> > >> -RR
> > >>
> > >>
>
>
>
>
> --
> Zhijie Shen
> Hortonworks Inc.
> http://hortonworks.com/
>
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity
> to which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>
--
Zhijie Shen
Hortonworks Inc.
http://hortonworks.com/
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: Doubt regarding Binary Compatibility\Source Compatibility with
old *mapred* APIs and new *mapreduce* APIs in Hadoop
Posted by Zhijie Shen <zs...@hortonworks.com>.
bq. Regarding #3 if I have ONLY the binaries i.e. jar file (compiled\build
against old MRv1 mapred APIS)
Which APIs are you talking about, *mapred* or *mapreduce*? In #3, I was
saying about *mapreduce*. If this is the case, you may be in the trouble
unfortunately, because MRv2 has evolved so much in *mapreduce *APIs that
it's difficult to ensure binary compatibility. Anyway, you should still try
your luck, as your binaries may not use the incompatible APIs. On the other
hand, if you meant *mapred* APIs instead, you binaries should just work.
- Zhijie
On Tue, Apr 15, 2014 at 1:35 PM, Radhe Radhe
<ra...@live.com>wrote:
> Thanks Zhijie for the explanation.
>
> Regarding #3 if I have ONLY the binaries i.e. jar file (compiled\build
> against old MRv1 *mapred* APIS) then how can I compile it since I don't
> have the source code i.e. Java files. All I can do with binaries i.e. jar
> file is execute it.
>
> -RR
> ------------------------------
> Date: Tue, 15 Apr 2014 13:03:53 -0700
>
> Subject: Re: Doubt regarding Binary Compatibility\Source Compatibility
> with old *mapred* APIs and new *mapreduce* APIs in Hadoop
> From: zshen@hortonworks.com
> To: user@hadoop.apache.org
>
>
> 1. If you have the binaries that were compiled against MRv1 *mapred*libs, it should just work with MRv2.
> 2. If you have the source code that refers to MRv1 *mapred* libs, it
> should be compilable without code changes. Of course, you're free to change
> your code.
> 3. If you have the binaries that were compiled against MRv1 *mapreduce* libs,
> it may not be executable directly with MRv2, but you should able to compile
> it against MRv2 *mapreduce* libs without code changes, and execute it.
>
> - Zhijie
>
>
> On Tue, Apr 15, 2014 at 12:44 PM, Radhe Radhe <
> radhe.krishna.radhe@live.com> wrote:
>
> Thanks John for your comments,
>
> I believe MRv2 has support for both the old *mapred* APIs and new
> *mapreduce* APIs.
>
> I see this way:
> [1.] One may have binaries i.e. jar file of the M\R program that used old
> *mapred* APIs
> This will work directly on MRv2(YARN).
>
> [2.] One may have the source code i.e. Java Programs of the M\R program
> that used old *mapred* APIs
> For this I need to recompile and generate the binaries i.e. jar file.
> Do I have to change the old *org.apache.hadoop.mapred* APIs to new *
> org.apache.hadoop.mapreduce* APIs? or No code changes are needed?
>
> -RR
>
> > Date: Mon, 14 Apr 2014 10:37:56 -0400
> > Subject: Re: Doubt regarding Binary Compatibility\Source Compatibility
> with old *mapred* APIs and new *mapreduce* APIs in Hadoop
> > From: john.meagher@gmail.com
> > To: user@hadoop.apache.org
>
> >
> > Also, "Source Compatibility" also means ONLY a recompile is needed.
> > No code changes should be needed.
> >
> > On Mon, Apr 14, 2014 at 10:37 AM, John Meagher <jo...@gmail.com>
> wrote:
> > > Source Compatibility = you need to recompile and use the new version
> > > as part of the compilation
> > >
> > > Binary Compatibility = you can take something compiled against the old
> > > version and run it on the new version
> > >
> > > On Mon, Apr 14, 2014 at 9:19 AM, Radhe Radhe
> > > <ra...@live.com> wrote:
> > >> Hello People,
> > >>
> > >> As per the Apache site
> > >>
> http://hadoop.apache.org/docs/r2.3.0/hadoop-mapreduce-client/hadoop-mapreduce-client-core/MapReduce_Compatibility_Hadoop1_Hadoop2.html
> > >>
> > >> Binary Compatibility
> > >> ====================
> > >> First, we ensure binary compatibility to the applications that use old
> > >> mapred APIs. This means that applications which were built against
> MRv1
> > >> mapred APIs can run directly on YARN without recompilation, merely by
> > >> pointing them to an Apache Hadoop 2.x cluster via configuration.
> > >>
> > >> Source Compatibility
> > >> ====================
> > >> We cannot ensure complete binary compatibility with the applications
> that
> > >> use mapreduce APIs, as these APIs have evolved a lot since MRv1.
> However, we
> > >> ensure source compatibility for mapreduce APIs that break binary
> > >> compatibility. In other words, users should recompile their
> applications
> > >> that use mapreduce APIs against MRv2 jars. One notable binary
> > >> incompatibility break is Counter and CounterGroup.
> > >>
> > >> For "Binary Compatibility" I understand that if I had build a MR job
> with
> > >> old *mapred* APIs then they can be run directly on YARN without and
> changes.
> > >>
> > >> Can anybody explain what do we mean by "Source Compatibility" here
> and also
> > >> a use case where one will need it?
> > >>
> > >> Does that mean code changes if I already have a MR job source code
> written
> > >> with with old *mapred* APIs and I need to make some changes to it to
> run in
> > >> then I need to use the new "mapreduce* API and generate the new
> binaries?
> > >>
> > >> Thanks,
> > >> -RR
> > >>
> > >>
>
>
>
>
> --
> Zhijie Shen
> Hortonworks Inc.
> http://hortonworks.com/
>
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity
> to which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>
--
Zhijie Shen
Hortonworks Inc.
http://hortonworks.com/
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: Doubt regarding Binary Compatibility\Source Compatibility with
old *mapred* APIs and new *mapreduce* APIs in Hadoop
Posted by Zhijie Shen <zs...@hortonworks.com>.
bq. Regarding #3 if I have ONLY the binaries i.e. jar file (compiled\build
against old MRv1 mapred APIS)
Which APIs are you talking about, *mapred* or *mapreduce*? In #3, I was
saying about *mapreduce*. If this is the case, you may be in the trouble
unfortunately, because MRv2 has evolved so much in *mapreduce *APIs that
it's difficult to ensure binary compatibility. Anyway, you should still try
your luck, as your binaries may not use the incompatible APIs. On the other
hand, if you meant *mapred* APIs instead, you binaries should just work.
- Zhijie
On Tue, Apr 15, 2014 at 1:35 PM, Radhe Radhe
<ra...@live.com>wrote:
> Thanks Zhijie for the explanation.
>
> Regarding #3 if I have ONLY the binaries i.e. jar file (compiled\build
> against old MRv1 *mapred* APIS) then how can I compile it since I don't
> have the source code i.e. Java files. All I can do with binaries i.e. jar
> file is execute it.
>
> -RR
> ------------------------------
> Date: Tue, 15 Apr 2014 13:03:53 -0700
>
> Subject: Re: Doubt regarding Binary Compatibility\Source Compatibility
> with old *mapred* APIs and new *mapreduce* APIs in Hadoop
> From: zshen@hortonworks.com
> To: user@hadoop.apache.org
>
>
> 1. If you have the binaries that were compiled against MRv1 *mapred*libs, it should just work with MRv2.
> 2. If you have the source code that refers to MRv1 *mapred* libs, it
> should be compilable without code changes. Of course, you're free to change
> your code.
> 3. If you have the binaries that were compiled against MRv1 *mapreduce* libs,
> it may not be executable directly with MRv2, but you should able to compile
> it against MRv2 *mapreduce* libs without code changes, and execute it.
>
> - Zhijie
>
>
> On Tue, Apr 15, 2014 at 12:44 PM, Radhe Radhe <
> radhe.krishna.radhe@live.com> wrote:
>
> Thanks John for your comments,
>
> I believe MRv2 has support for both the old *mapred* APIs and new
> *mapreduce* APIs.
>
> I see this way:
> [1.] One may have binaries i.e. jar file of the M\R program that used old
> *mapred* APIs
> This will work directly on MRv2(YARN).
>
> [2.] One may have the source code i.e. Java Programs of the M\R program
> that used old *mapred* APIs
> For this I need to recompile and generate the binaries i.e. jar file.
> Do I have to change the old *org.apache.hadoop.mapred* APIs to new *
> org.apache.hadoop.mapreduce* APIs? or No code changes are needed?
>
> -RR
>
> > Date: Mon, 14 Apr 2014 10:37:56 -0400
> > Subject: Re: Doubt regarding Binary Compatibility\Source Compatibility
> with old *mapred* APIs and new *mapreduce* APIs in Hadoop
> > From: john.meagher@gmail.com
> > To: user@hadoop.apache.org
>
> >
> > Also, "Source Compatibility" also means ONLY a recompile is needed.
> > No code changes should be needed.
> >
> > On Mon, Apr 14, 2014 at 10:37 AM, John Meagher <jo...@gmail.com>
> wrote:
> > > Source Compatibility = you need to recompile and use the new version
> > > as part of the compilation
> > >
> > > Binary Compatibility = you can take something compiled against the old
> > > version and run it on the new version
> > >
> > > On Mon, Apr 14, 2014 at 9:19 AM, Radhe Radhe
> > > <ra...@live.com> wrote:
> > >> Hello People,
> > >>
> > >> As per the Apache site
> > >>
> http://hadoop.apache.org/docs/r2.3.0/hadoop-mapreduce-client/hadoop-mapreduce-client-core/MapReduce_Compatibility_Hadoop1_Hadoop2.html
> > >>
> > >> Binary Compatibility
> > >> ====================
> > >> First, we ensure binary compatibility to the applications that use old
> > >> mapred APIs. This means that applications which were built against
> MRv1
> > >> mapred APIs can run directly on YARN without recompilation, merely by
> > >> pointing them to an Apache Hadoop 2.x cluster via configuration.
> > >>
> > >> Source Compatibility
> > >> ====================
> > >> We cannot ensure complete binary compatibility with the applications
> that
> > >> use mapreduce APIs, as these APIs have evolved a lot since MRv1.
> However, we
> > >> ensure source compatibility for mapreduce APIs that break binary
> > >> compatibility. In other words, users should recompile their
> applications
> > >> that use mapreduce APIs against MRv2 jars. One notable binary
> > >> incompatibility break is Counter and CounterGroup.
> > >>
> > >> For "Binary Compatibility" I understand that if I had build a MR job
> with
> > >> old *mapred* APIs then they can be run directly on YARN without and
> changes.
> > >>
> > >> Can anybody explain what do we mean by "Source Compatibility" here
> and also
> > >> a use case where one will need it?
> > >>
> > >> Does that mean code changes if I already have a MR job source code
> written
> > >> with with old *mapred* APIs and I need to make some changes to it to
> run in
> > >> then I need to use the new "mapreduce* API and generate the new
> binaries?
> > >>
> > >> Thanks,
> > >> -RR
> > >>
> > >>
>
>
>
>
> --
> Zhijie Shen
> Hortonworks Inc.
> http://hortonworks.com/
>
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity
> to which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>
--
Zhijie Shen
Hortonworks Inc.
http://hortonworks.com/
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: Doubt regarding Binary Compatibility\Source Compatibility with
old *mapred* APIs and new *mapreduce* APIs in Hadoop
Posted by Zhijie Shen <zs...@hortonworks.com>.
bq. Regarding #3 if I have ONLY the binaries i.e. jar file (compiled\build
against old MRv1 mapred APIS)
Which APIs are you talking about, *mapred* or *mapreduce*? In #3, I was
saying about *mapreduce*. If this is the case, you may be in the trouble
unfortunately, because MRv2 has evolved so much in *mapreduce *APIs that
it's difficult to ensure binary compatibility. Anyway, you should still try
your luck, as your binaries may not use the incompatible APIs. On the other
hand, if you meant *mapred* APIs instead, you binaries should just work.
- Zhijie
On Tue, Apr 15, 2014 at 1:35 PM, Radhe Radhe
<ra...@live.com>wrote:
> Thanks Zhijie for the explanation.
>
> Regarding #3 if I have ONLY the binaries i.e. jar file (compiled\build
> against old MRv1 *mapred* APIS) then how can I compile it since I don't
> have the source code i.e. Java files. All I can do with binaries i.e. jar
> file is execute it.
>
> -RR
> ------------------------------
> Date: Tue, 15 Apr 2014 13:03:53 -0700
>
> Subject: Re: Doubt regarding Binary Compatibility\Source Compatibility
> with old *mapred* APIs and new *mapreduce* APIs in Hadoop
> From: zshen@hortonworks.com
> To: user@hadoop.apache.org
>
>
> 1. If you have the binaries that were compiled against MRv1 *mapred*libs, it should just work with MRv2.
> 2. If you have the source code that refers to MRv1 *mapred* libs, it
> should be compilable without code changes. Of course, you're free to change
> your code.
> 3. If you have the binaries that were compiled against MRv1 *mapreduce* libs,
> it may not be executable directly with MRv2, but you should able to compile
> it against MRv2 *mapreduce* libs without code changes, and execute it.
>
> - Zhijie
>
>
> On Tue, Apr 15, 2014 at 12:44 PM, Radhe Radhe <
> radhe.krishna.radhe@live.com> wrote:
>
> Thanks John for your comments,
>
> I believe MRv2 has support for both the old *mapred* APIs and new
> *mapreduce* APIs.
>
> I see this way:
> [1.] One may have binaries i.e. jar file of the M\R program that used old
> *mapred* APIs
> This will work directly on MRv2(YARN).
>
> [2.] One may have the source code i.e. Java Programs of the M\R program
> that used old *mapred* APIs
> For this I need to recompile and generate the binaries i.e. jar file.
> Do I have to change the old *org.apache.hadoop.mapred* APIs to new *
> org.apache.hadoop.mapreduce* APIs? or No code changes are needed?
>
> -RR
>
> > Date: Mon, 14 Apr 2014 10:37:56 -0400
> > Subject: Re: Doubt regarding Binary Compatibility\Source Compatibility
> with old *mapred* APIs and new *mapreduce* APIs in Hadoop
> > From: john.meagher@gmail.com
> > To: user@hadoop.apache.org
>
> >
> > Also, "Source Compatibility" also means ONLY a recompile is needed.
> > No code changes should be needed.
> >
> > On Mon, Apr 14, 2014 at 10:37 AM, John Meagher <jo...@gmail.com>
> wrote:
> > > Source Compatibility = you need to recompile and use the new version
> > > as part of the compilation
> > >
> > > Binary Compatibility = you can take something compiled against the old
> > > version and run it on the new version
> > >
> > > On Mon, Apr 14, 2014 at 9:19 AM, Radhe Radhe
> > > <ra...@live.com> wrote:
> > >> Hello People,
> > >>
> > >> As per the Apache site
> > >>
> http://hadoop.apache.org/docs/r2.3.0/hadoop-mapreduce-client/hadoop-mapreduce-client-core/MapReduce_Compatibility_Hadoop1_Hadoop2.html
> > >>
> > >> Binary Compatibility
> > >> ====================
> > >> First, we ensure binary compatibility to the applications that use old
> > >> mapred APIs. This means that applications which were built against
> MRv1
> > >> mapred APIs can run directly on YARN without recompilation, merely by
> > >> pointing them to an Apache Hadoop 2.x cluster via configuration.
> > >>
> > >> Source Compatibility
> > >> ====================
> > >> We cannot ensure complete binary compatibility with the applications
> that
> > >> use mapreduce APIs, as these APIs have evolved a lot since MRv1.
> However, we
> > >> ensure source compatibility for mapreduce APIs that break binary
> > >> compatibility. In other words, users should recompile their
> applications
> > >> that use mapreduce APIs against MRv2 jars. One notable binary
> > >> incompatibility break is Counter and CounterGroup.
> > >>
> > >> For "Binary Compatibility" I understand that if I had build a MR job
> with
> > >> old *mapred* APIs then they can be run directly on YARN without and
> changes.
> > >>
> > >> Can anybody explain what do we mean by "Source Compatibility" here
> and also
> > >> a use case where one will need it?
> > >>
> > >> Does that mean code changes if I already have a MR job source code
> written
> > >> with with old *mapred* APIs and I need to make some changes to it to
> run in
> > >> then I need to use the new "mapreduce* API and generate the new
> binaries?
> > >>
> > >> Thanks,
> > >> -RR
> > >>
> > >>
>
>
>
>
> --
> Zhijie Shen
> Hortonworks Inc.
> http://hortonworks.com/
>
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity
> to which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>
--
Zhijie Shen
Hortonworks Inc.
http://hortonworks.com/
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
RE: Doubt regarding Binary Compatibility\Source Compatibility with
old *mapred* APIs and new *mapreduce* APIs in Hadoop
Posted by Radhe Radhe <ra...@live.com>.
Thanks Zhijie for the explanation.
Regarding #3 if I have ONLY the binaries i.e. jar file (compiled\build against old MRv1 mapred APIS) then how can I compile it since I don't have the source code i.e. Java files. All I can do with binaries i.e. jar file is execute it.
-RR
Date: Tue, 15 Apr 2014 13:03:53 -0700
Subject: Re: Doubt regarding Binary Compatibility\Source Compatibility with old *mapred* APIs and new *mapreduce* APIs in Hadoop
From: zshen@hortonworks.com
To: user@hadoop.apache.org
1. If you have the binaries that were compiled against MRv1 mapred libs, it should just work with MRv2.2. If you have the source code that refers to MRv1 mapred libs, it should be compilable without code changes. Of course, you're free to change your code.
3. If you have the binaries that were compiled against MRv1 mapreduce libs, it may not be executable directly with MRv2, but you should able to compile it against MRv2 mapreduce libs without code changes, and execute it.
- Zhijie
On Tue, Apr 15, 2014 at 12:44 PM, Radhe Radhe <ra...@live.com> wrote:
Thanks John for your comments,
I believe MRv2 has support for both the old *mapred* APIs and new *mapreduce* APIs.
I see this way:[1.] One may have binaries i.e. jar file of the M\R program that used old *mapred* APIs
This will work directly on MRv2(YARN).
[2.] One may have the source code i.e. Java Programs of the M\R program that used old *mapred* APIs
For this I need to recompile and generate the binaries i.e. jar file. Do I have to change the old *org.apache.hadoop.mapred* APIs to new *org.apache.hadoop.mapreduce* APIs? or No code changes are needed?
-RR
> Date: Mon, 14 Apr 2014 10:37:56 -0400
> Subject: Re: Doubt regarding Binary Compatibility\Source Compatibility with old *mapred* APIs and new *mapreduce* APIs in Hadoop
> From: john.meagher@gmail.com
> To: user@hadoop.apache.org
>
> Also, "Source Compatibility" also means ONLY a recompile is needed.
> No code changes should be needed.
>
> On Mon, Apr 14, 2014 at 10:37 AM, John Meagher <jo...@gmail.com> wrote:
> > Source Compatibility = you need to recompile and use the new version
> > as part of the compilation
> >
> > Binary Compatibility = you can take something compiled against the old
> > version and run it on the new version
> >
> > On Mon, Apr 14, 2014 at 9:19 AM, Radhe Radhe
> > <ra...@live.com> wrote:
> >> Hello People,
> >>
> >> As per the Apache site
> >> http://hadoop.apache.org/docs/r2.3.0/hadoop-mapreduce-client/hadoop-mapreduce-client-core/MapReduce_Compatibility_Hadoop1_Hadoop2.html
> >>
> >> Binary Compatibility
> >> ====================
> >> First, we ensure binary compatibility to the applications that use old
> >> mapred APIs. This means that applications which were built against MRv1
> >> mapred APIs can run directly on YARN without recompilation, merely by
> >> pointing them to an Apache Hadoop 2.x cluster via configuration.
> >>
> >> Source Compatibility
> >> ====================
> >> We cannot ensure complete binary compatibility with the applications that
> >> use mapreduce APIs, as these APIs have evolved a lot since MRv1. However, we
> >> ensure source compatibility for mapreduce APIs that break binary
> >> compatibility. In other words, users should recompile their applications
> >> that use mapreduce APIs against MRv2 jars. One notable binary
> >> incompatibility break is Counter and CounterGroup.
> >>
> >> For "Binary Compatibility" I understand that if I had build a MR job with
> >> old *mapred* APIs then they can be run directly on YARN without and changes.
> >>
> >> Can anybody explain what do we mean by "Source Compatibility" here and also
> >> a use case where one will need it?
> >>
> >> Does that mean code changes if I already have a MR job source code written
> >> with with old *mapred* APIs and I need to make some changes to it to run in
> >> then I need to use the new "mapreduce* API and generate the new binaries?
> >>
> >> Thanks,
> >> -RR
> >>
> >>
--
Zhijie ShenHortonworks Inc.http://hortonworks.com/
CONFIDENTIALITY NOTICENOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
RE: Doubt regarding Binary Compatibility\Source Compatibility with
old *mapred* APIs and new *mapreduce* APIs in Hadoop
Posted by Radhe Radhe <ra...@live.com>.
Thanks Zhijie for the explanation.
Regarding #3 if I have ONLY the binaries i.e. jar file (compiled\build against old MRv1 mapred APIS) then how can I compile it since I don't have the source code i.e. Java files. All I can do with binaries i.e. jar file is execute it.
-RR
Date: Tue, 15 Apr 2014 13:03:53 -0700
Subject: Re: Doubt regarding Binary Compatibility\Source Compatibility with old *mapred* APIs and new *mapreduce* APIs in Hadoop
From: zshen@hortonworks.com
To: user@hadoop.apache.org
1. If you have the binaries that were compiled against MRv1 mapred libs, it should just work with MRv2.2. If you have the source code that refers to MRv1 mapred libs, it should be compilable without code changes. Of course, you're free to change your code.
3. If you have the binaries that were compiled against MRv1 mapreduce libs, it may not be executable directly with MRv2, but you should able to compile it against MRv2 mapreduce libs without code changes, and execute it.
- Zhijie
On Tue, Apr 15, 2014 at 12:44 PM, Radhe Radhe <ra...@live.com> wrote:
Thanks John for your comments,
I believe MRv2 has support for both the old *mapred* APIs and new *mapreduce* APIs.
I see this way:[1.] One may have binaries i.e. jar file of the M\R program that used old *mapred* APIs
This will work directly on MRv2(YARN).
[2.] One may have the source code i.e. Java Programs of the M\R program that used old *mapred* APIs
For this I need to recompile and generate the binaries i.e. jar file. Do I have to change the old *org.apache.hadoop.mapred* APIs to new *org.apache.hadoop.mapreduce* APIs? or No code changes are needed?
-RR
> Date: Mon, 14 Apr 2014 10:37:56 -0400
> Subject: Re: Doubt regarding Binary Compatibility\Source Compatibility with old *mapred* APIs and new *mapreduce* APIs in Hadoop
> From: john.meagher@gmail.com
> To: user@hadoop.apache.org
>
> Also, "Source Compatibility" also means ONLY a recompile is needed.
> No code changes should be needed.
>
> On Mon, Apr 14, 2014 at 10:37 AM, John Meagher <jo...@gmail.com> wrote:
> > Source Compatibility = you need to recompile and use the new version
> > as part of the compilation
> >
> > Binary Compatibility = you can take something compiled against the old
> > version and run it on the new version
> >
> > On Mon, Apr 14, 2014 at 9:19 AM, Radhe Radhe
> > <ra...@live.com> wrote:
> >> Hello People,
> >>
> >> As per the Apache site
> >> http://hadoop.apache.org/docs/r2.3.0/hadoop-mapreduce-client/hadoop-mapreduce-client-core/MapReduce_Compatibility_Hadoop1_Hadoop2.html
> >>
> >> Binary Compatibility
> >> ====================
> >> First, we ensure binary compatibility to the applications that use old
> >> mapred APIs. This means that applications which were built against MRv1
> >> mapred APIs can run directly on YARN without recompilation, merely by
> >> pointing them to an Apache Hadoop 2.x cluster via configuration.
> >>
> >> Source Compatibility
> >> ====================
> >> We cannot ensure complete binary compatibility with the applications that
> >> use mapreduce APIs, as these APIs have evolved a lot since MRv1. However, we
> >> ensure source compatibility for mapreduce APIs that break binary
> >> compatibility. In other words, users should recompile their applications
> >> that use mapreduce APIs against MRv2 jars. One notable binary
> >> incompatibility break is Counter and CounterGroup.
> >>
> >> For "Binary Compatibility" I understand that if I had build a MR job with
> >> old *mapred* APIs then they can be run directly on YARN without and changes.
> >>
> >> Can anybody explain what do we mean by "Source Compatibility" here and also
> >> a use case where one will need it?
> >>
> >> Does that mean code changes if I already have a MR job source code written
> >> with with old *mapred* APIs and I need to make some changes to it to run in
> >> then I need to use the new "mapreduce* API and generate the new binaries?
> >>
> >> Thanks,
> >> -RR
> >>
> >>
--
Zhijie ShenHortonworks Inc.http://hortonworks.com/
CONFIDENTIALITY NOTICENOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
RE: Doubt regarding Binary Compatibility\Source Compatibility with
old *mapred* APIs and new *mapreduce* APIs in Hadoop
Posted by Radhe Radhe <ra...@live.com>.
Thanks Zhijie for the explanation.
Regarding #3 if I have ONLY the binaries i.e. jar file (compiled\build against old MRv1 mapred APIS) then how can I compile it since I don't have the source code i.e. Java files. All I can do with binaries i.e. jar file is execute it.
-RR
Date: Tue, 15 Apr 2014 13:03:53 -0700
Subject: Re: Doubt regarding Binary Compatibility\Source Compatibility with old *mapred* APIs and new *mapreduce* APIs in Hadoop
From: zshen@hortonworks.com
To: user@hadoop.apache.org
1. If you have the binaries that were compiled against MRv1 mapred libs, it should just work with MRv2.2. If you have the source code that refers to MRv1 mapred libs, it should be compilable without code changes. Of course, you're free to change your code.
3. If you have the binaries that were compiled against MRv1 mapreduce libs, it may not be executable directly with MRv2, but you should able to compile it against MRv2 mapreduce libs without code changes, and execute it.
- Zhijie
On Tue, Apr 15, 2014 at 12:44 PM, Radhe Radhe <ra...@live.com> wrote:
Thanks John for your comments,
I believe MRv2 has support for both the old *mapred* APIs and new *mapreduce* APIs.
I see this way:[1.] One may have binaries i.e. jar file of the M\R program that used old *mapred* APIs
This will work directly on MRv2(YARN).
[2.] One may have the source code i.e. Java Programs of the M\R program that used old *mapred* APIs
For this I need to recompile and generate the binaries i.e. jar file. Do I have to change the old *org.apache.hadoop.mapred* APIs to new *org.apache.hadoop.mapreduce* APIs? or No code changes are needed?
-RR
> Date: Mon, 14 Apr 2014 10:37:56 -0400
> Subject: Re: Doubt regarding Binary Compatibility\Source Compatibility with old *mapred* APIs and new *mapreduce* APIs in Hadoop
> From: john.meagher@gmail.com
> To: user@hadoop.apache.org
>
> Also, "Source Compatibility" also means ONLY a recompile is needed.
> No code changes should be needed.
>
> On Mon, Apr 14, 2014 at 10:37 AM, John Meagher <jo...@gmail.com> wrote:
> > Source Compatibility = you need to recompile and use the new version
> > as part of the compilation
> >
> > Binary Compatibility = you can take something compiled against the old
> > version and run it on the new version
> >
> > On Mon, Apr 14, 2014 at 9:19 AM, Radhe Radhe
> > <ra...@live.com> wrote:
> >> Hello People,
> >>
> >> As per the Apache site
> >> http://hadoop.apache.org/docs/r2.3.0/hadoop-mapreduce-client/hadoop-mapreduce-client-core/MapReduce_Compatibility_Hadoop1_Hadoop2.html
> >>
> >> Binary Compatibility
> >> ====================
> >> First, we ensure binary compatibility to the applications that use old
> >> mapred APIs. This means that applications which were built against MRv1
> >> mapred APIs can run directly on YARN without recompilation, merely by
> >> pointing them to an Apache Hadoop 2.x cluster via configuration.
> >>
> >> Source Compatibility
> >> ====================
> >> We cannot ensure complete binary compatibility with the applications that
> >> use mapreduce APIs, as these APIs have evolved a lot since MRv1. However, we
> >> ensure source compatibility for mapreduce APIs that break binary
> >> compatibility. In other words, users should recompile their applications
> >> that use mapreduce APIs against MRv2 jars. One notable binary
> >> incompatibility break is Counter and CounterGroup.
> >>
> >> For "Binary Compatibility" I understand that if I had build a MR job with
> >> old *mapred* APIs then they can be run directly on YARN without and changes.
> >>
> >> Can anybody explain what do we mean by "Source Compatibility" here and also
> >> a use case where one will need it?
> >>
> >> Does that mean code changes if I already have a MR job source code written
> >> with with old *mapred* APIs and I need to make some changes to it to run in
> >> then I need to use the new "mapreduce* API and generate the new binaries?
> >>
> >> Thanks,
> >> -RR
> >>
> >>
--
Zhijie ShenHortonworks Inc.http://hortonworks.com/
CONFIDENTIALITY NOTICENOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
RE: Doubt regarding Binary Compatibility\Source Compatibility with
old *mapred* APIs and new *mapreduce* APIs in Hadoop
Posted by Radhe Radhe <ra...@live.com>.
Thanks Zhijie for the explanation.
Regarding #3 if I have ONLY the binaries i.e. jar file (compiled\build against old MRv1 mapred APIS) then how can I compile it since I don't have the source code i.e. Java files. All I can do with binaries i.e. jar file is execute it.
-RR
Date: Tue, 15 Apr 2014 13:03:53 -0700
Subject: Re: Doubt regarding Binary Compatibility\Source Compatibility with old *mapred* APIs and new *mapreduce* APIs in Hadoop
From: zshen@hortonworks.com
To: user@hadoop.apache.org
1. If you have the binaries that were compiled against MRv1 mapred libs, it should just work with MRv2.2. If you have the source code that refers to MRv1 mapred libs, it should be compilable without code changes. Of course, you're free to change your code.
3. If you have the binaries that were compiled against MRv1 mapreduce libs, it may not be executable directly with MRv2, but you should able to compile it against MRv2 mapreduce libs without code changes, and execute it.
- Zhijie
On Tue, Apr 15, 2014 at 12:44 PM, Radhe Radhe <ra...@live.com> wrote:
Thanks John for your comments,
I believe MRv2 has support for both the old *mapred* APIs and new *mapreduce* APIs.
I see this way:[1.] One may have binaries i.e. jar file of the M\R program that used old *mapred* APIs
This will work directly on MRv2(YARN).
[2.] One may have the source code i.e. Java Programs of the M\R program that used old *mapred* APIs
For this I need to recompile and generate the binaries i.e. jar file. Do I have to change the old *org.apache.hadoop.mapred* APIs to new *org.apache.hadoop.mapreduce* APIs? or No code changes are needed?
-RR
> Date: Mon, 14 Apr 2014 10:37:56 -0400
> Subject: Re: Doubt regarding Binary Compatibility\Source Compatibility with old *mapred* APIs and new *mapreduce* APIs in Hadoop
> From: john.meagher@gmail.com
> To: user@hadoop.apache.org
>
> Also, "Source Compatibility" also means ONLY a recompile is needed.
> No code changes should be needed.
>
> On Mon, Apr 14, 2014 at 10:37 AM, John Meagher <jo...@gmail.com> wrote:
> > Source Compatibility = you need to recompile and use the new version
> > as part of the compilation
> >
> > Binary Compatibility = you can take something compiled against the old
> > version and run it on the new version
> >
> > On Mon, Apr 14, 2014 at 9:19 AM, Radhe Radhe
> > <ra...@live.com> wrote:
> >> Hello People,
> >>
> >> As per the Apache site
> >> http://hadoop.apache.org/docs/r2.3.0/hadoop-mapreduce-client/hadoop-mapreduce-client-core/MapReduce_Compatibility_Hadoop1_Hadoop2.html
> >>
> >> Binary Compatibility
> >> ====================
> >> First, we ensure binary compatibility to the applications that use old
> >> mapred APIs. This means that applications which were built against MRv1
> >> mapred APIs can run directly on YARN without recompilation, merely by
> >> pointing them to an Apache Hadoop 2.x cluster via configuration.
> >>
> >> Source Compatibility
> >> ====================
> >> We cannot ensure complete binary compatibility with the applications that
> >> use mapreduce APIs, as these APIs have evolved a lot since MRv1. However, we
> >> ensure source compatibility for mapreduce APIs that break binary
> >> compatibility. In other words, users should recompile their applications
> >> that use mapreduce APIs against MRv2 jars. One notable binary
> >> incompatibility break is Counter and CounterGroup.
> >>
> >> For "Binary Compatibility" I understand that if I had build a MR job with
> >> old *mapred* APIs then they can be run directly on YARN without and changes.
> >>
> >> Can anybody explain what do we mean by "Source Compatibility" here and also
> >> a use case where one will need it?
> >>
> >> Does that mean code changes if I already have a MR job source code written
> >> with with old *mapred* APIs and I need to make some changes to it to run in
> >> then I need to use the new "mapreduce* API and generate the new binaries?
> >>
> >> Thanks,
> >> -RR
> >>
> >>
--
Zhijie ShenHortonworks Inc.http://hortonworks.com/
CONFIDENTIALITY NOTICENOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
Re: Doubt regarding Binary Compatibility\Source Compatibility with
old *mapred* APIs and new *mapreduce* APIs in Hadoop
Posted by Zhijie Shen <zs...@hortonworks.com>.
1. If you have the binaries that were compiled against MRv1 *mapred* libs,
it should just work with MRv2.
2. If you have the source code that refers to MRv1 *mapred* libs, it should
be compilable without code changes. Of course, you're free to change your
code.
3. If you have the binaries that were compiled against MRv1 *mapreduce* libs,
it may not be executable directly with MRv2, but you should able to compile
it against MRv2 *mapreduce* libs without code changes, and execute it.
- Zhijie
On Tue, Apr 15, 2014 at 12:44 PM, Radhe Radhe
<ra...@live.com>wrote:
> Thanks John for your comments,
>
> I believe MRv2 has support for both the old *mapred* APIs and new
> *mapreduce* APIs.
>
> I see this way:
> [1.] One may have binaries i.e. jar file of the M\R program that used old
> *mapred* APIs
> This will work directly on MRv2(YARN).
>
> [2.] One may have the source code i.e. Java Programs of the M\R program
> that used old *mapred* APIs
> For this I need to recompile and generate the binaries i.e. jar file.
> Do I have to change the old *org.apache.hadoop.mapred* APIs to new *
> org.apache.hadoop.mapreduce* APIs? or No code changes are needed?
>
> -RR
>
> > Date: Mon, 14 Apr 2014 10:37:56 -0400
> > Subject: Re: Doubt regarding Binary Compatibility\Source Compatibility
> with old *mapred* APIs and new *mapreduce* APIs in Hadoop
> > From: john.meagher@gmail.com
> > To: user@hadoop.apache.org
>
> >
> > Also, "Source Compatibility" also means ONLY a recompile is needed.
> > No code changes should be needed.
> >
> > On Mon, Apr 14, 2014 at 10:37 AM, John Meagher <jo...@gmail.com>
> wrote:
> > > Source Compatibility = you need to recompile and use the new version
> > > as part of the compilation
> > >
> > > Binary Compatibility = you can take something compiled against the old
> > > version and run it on the new version
> > >
> > > On Mon, Apr 14, 2014 at 9:19 AM, Radhe Radhe
> > > <ra...@live.com> wrote:
> > >> Hello People,
> > >>
> > >> As per the Apache site
> > >>
> http://hadoop.apache.org/docs/r2.3.0/hadoop-mapreduce-client/hadoop-mapreduce-client-core/MapReduce_Compatibility_Hadoop1_Hadoop2.html
> > >>
> > >> Binary Compatibility
> > >> ====================
> > >> First, we ensure binary compatibility to the applications that use old
> > >> mapred APIs. This means that applications which were built against
> MRv1
> > >> mapred APIs can run directly on YARN without recompilation, merely by
> > >> pointing them to an Apache Hadoop 2.x cluster via configuration.
> > >>
> > >> Source Compatibility
> > >> ====================
> > >> We cannot ensure complete binary compatibility with the applications
> that
> > >> use mapreduce APIs, as these APIs have evolved a lot since MRv1.
> However, we
> > >> ensure source compatibility for mapreduce APIs that break binary
> > >> compatibility. In other words, users should recompile their
> applications
> > >> that use mapreduce APIs against MRv2 jars. One notable binary
> > >> incompatibility break is Counter and CounterGroup.
> > >>
> > >> For "Binary Compatibility" I understand that if I had build a MR job
> with
> > >> old *mapred* APIs then they can be run directly on YARN without and
> changes.
> > >>
> > >> Can anybody explain what do we mean by "Source Compatibility" here
> and also
> > >> a use case where one will need it?
> > >>
> > >> Does that mean code changes if I already have a MR job source code
> written
> > >> with with old *mapred* APIs and I need to make some changes to it to
> run in
> > >> then I need to use the new "mapreduce* API and generate the new
> binaries?
> > >>
> > >> Thanks,
> > >> -RR
> > >>
> > >>
>
--
Zhijie Shen
Hortonworks Inc.
http://hortonworks.com/
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: Doubt regarding Binary Compatibility\Source Compatibility with
old *mapred* APIs and new *mapreduce* APIs in Hadoop
Posted by Zhijie Shen <zs...@hortonworks.com>.
1. If you have the binaries that were compiled against MRv1 *mapred* libs,
it should just work with MRv2.
2. If you have the source code that refers to MRv1 *mapred* libs, it should
be compilable without code changes. Of course, you're free to change your
code.
3. If you have the binaries that were compiled against MRv1 *mapreduce* libs,
it may not be executable directly with MRv2, but you should able to compile
it against MRv2 *mapreduce* libs without code changes, and execute it.
- Zhijie
On Tue, Apr 15, 2014 at 12:44 PM, Radhe Radhe
<ra...@live.com>wrote:
> Thanks John for your comments,
>
> I believe MRv2 has support for both the old *mapred* APIs and new
> *mapreduce* APIs.
>
> I see this way:
> [1.] One may have binaries i.e. jar file of the M\R program that used old
> *mapred* APIs
> This will work directly on MRv2(YARN).
>
> [2.] One may have the source code i.e. Java Programs of the M\R program
> that used old *mapred* APIs
> For this I need to recompile and generate the binaries i.e. jar file.
> Do I have to change the old *org.apache.hadoop.mapred* APIs to new *
> org.apache.hadoop.mapreduce* APIs? or No code changes are needed?
>
> -RR
>
> > Date: Mon, 14 Apr 2014 10:37:56 -0400
> > Subject: Re: Doubt regarding Binary Compatibility\Source Compatibility
> with old *mapred* APIs and new *mapreduce* APIs in Hadoop
> > From: john.meagher@gmail.com
> > To: user@hadoop.apache.org
>
> >
> > Also, "Source Compatibility" also means ONLY a recompile is needed.
> > No code changes should be needed.
> >
> > On Mon, Apr 14, 2014 at 10:37 AM, John Meagher <jo...@gmail.com>
> wrote:
> > > Source Compatibility = you need to recompile and use the new version
> > > as part of the compilation
> > >
> > > Binary Compatibility = you can take something compiled against the old
> > > version and run it on the new version
> > >
> > > On Mon, Apr 14, 2014 at 9:19 AM, Radhe Radhe
> > > <ra...@live.com> wrote:
> > >> Hello People,
> > >>
> > >> As per the Apache site
> > >>
> http://hadoop.apache.org/docs/r2.3.0/hadoop-mapreduce-client/hadoop-mapreduce-client-core/MapReduce_Compatibility_Hadoop1_Hadoop2.html
> > >>
> > >> Binary Compatibility
> > >> ====================
> > >> First, we ensure binary compatibility to the applications that use old
> > >> mapred APIs. This means that applications which were built against
> MRv1
> > >> mapred APIs can run directly on YARN without recompilation, merely by
> > >> pointing them to an Apache Hadoop 2.x cluster via configuration.
> > >>
> > >> Source Compatibility
> > >> ====================
> > >> We cannot ensure complete binary compatibility with the applications
> that
> > >> use mapreduce APIs, as these APIs have evolved a lot since MRv1.
> However, we
> > >> ensure source compatibility for mapreduce APIs that break binary
> > >> compatibility. In other words, users should recompile their
> applications
> > >> that use mapreduce APIs against MRv2 jars. One notable binary
> > >> incompatibility break is Counter and CounterGroup.
> > >>
> > >> For "Binary Compatibility" I understand that if I had build a MR job
> with
> > >> old *mapred* APIs then they can be run directly on YARN without and
> changes.
> > >>
> > >> Can anybody explain what do we mean by "Source Compatibility" here
> and also
> > >> a use case where one will need it?
> > >>
> > >> Does that mean code changes if I already have a MR job source code
> written
> > >> with with old *mapred* APIs and I need to make some changes to it to
> run in
> > >> then I need to use the new "mapreduce* API and generate the new
> binaries?
> > >>
> > >> Thanks,
> > >> -RR
> > >>
> > >>
>
--
Zhijie Shen
Hortonworks Inc.
http://hortonworks.com/
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: Doubt regarding Binary Compatibility\Source Compatibility with
old *mapred* APIs and new *mapreduce* APIs in Hadoop
Posted by Zhijie Shen <zs...@hortonworks.com>.
1. If you have the binaries that were compiled against MRv1 *mapred* libs,
it should just work with MRv2.
2. If you have the source code that refers to MRv1 *mapred* libs, it should
be compilable without code changes. Of course, you're free to change your
code.
3. If you have the binaries that were compiled against MRv1 *mapreduce* libs,
it may not be executable directly with MRv2, but you should able to compile
it against MRv2 *mapreduce* libs without code changes, and execute it.
- Zhijie
On Tue, Apr 15, 2014 at 12:44 PM, Radhe Radhe
<ra...@live.com>wrote:
> Thanks John for your comments,
>
> I believe MRv2 has support for both the old *mapred* APIs and new
> *mapreduce* APIs.
>
> I see this way:
> [1.] One may have binaries i.e. jar file of the M\R program that used old
> *mapred* APIs
> This will work directly on MRv2(YARN).
>
> [2.] One may have the source code i.e. Java Programs of the M\R program
> that used old *mapred* APIs
> For this I need to recompile and generate the binaries i.e. jar file.
> Do I have to change the old *org.apache.hadoop.mapred* APIs to new *
> org.apache.hadoop.mapreduce* APIs? or No code changes are needed?
>
> -RR
>
> > Date: Mon, 14 Apr 2014 10:37:56 -0400
> > Subject: Re: Doubt regarding Binary Compatibility\Source Compatibility
> with old *mapred* APIs and new *mapreduce* APIs in Hadoop
> > From: john.meagher@gmail.com
> > To: user@hadoop.apache.org
>
> >
> > Also, "Source Compatibility" also means ONLY a recompile is needed.
> > No code changes should be needed.
> >
> > On Mon, Apr 14, 2014 at 10:37 AM, John Meagher <jo...@gmail.com>
> wrote:
> > > Source Compatibility = you need to recompile and use the new version
> > > as part of the compilation
> > >
> > > Binary Compatibility = you can take something compiled against the old
> > > version and run it on the new version
> > >
> > > On Mon, Apr 14, 2014 at 9:19 AM, Radhe Radhe
> > > <ra...@live.com> wrote:
> > >> Hello People,
> > >>
> > >> As per the Apache site
> > >>
> http://hadoop.apache.org/docs/r2.3.0/hadoop-mapreduce-client/hadoop-mapreduce-client-core/MapReduce_Compatibility_Hadoop1_Hadoop2.html
> > >>
> > >> Binary Compatibility
> > >> ====================
> > >> First, we ensure binary compatibility to the applications that use old
> > >> mapred APIs. This means that applications which were built against
> MRv1
> > >> mapred APIs can run directly on YARN without recompilation, merely by
> > >> pointing them to an Apache Hadoop 2.x cluster via configuration.
> > >>
> > >> Source Compatibility
> > >> ====================
> > >> We cannot ensure complete binary compatibility with the applications
> that
> > >> use mapreduce APIs, as these APIs have evolved a lot since MRv1.
> However, we
> > >> ensure source compatibility for mapreduce APIs that break binary
> > >> compatibility. In other words, users should recompile their
> applications
> > >> that use mapreduce APIs against MRv2 jars. One notable binary
> > >> incompatibility break is Counter and CounterGroup.
> > >>
> > >> For "Binary Compatibility" I understand that if I had build a MR job
> with
> > >> old *mapred* APIs then they can be run directly on YARN without and
> changes.
> > >>
> > >> Can anybody explain what do we mean by "Source Compatibility" here
> and also
> > >> a use case where one will need it?
> > >>
> > >> Does that mean code changes if I already have a MR job source code
> written
> > >> with with old *mapred* APIs and I need to make some changes to it to
> run in
> > >> then I need to use the new "mapreduce* API and generate the new
> binaries?
> > >>
> > >> Thanks,
> > >> -RR
> > >>
> > >>
>
--
Zhijie Shen
Hortonworks Inc.
http://hortonworks.com/
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: Doubt regarding Binary Compatibility\Source Compatibility with
old *mapred* APIs and new *mapreduce* APIs in Hadoop
Posted by Zhijie Shen <zs...@hortonworks.com>.
1. If you have the binaries that were compiled against MRv1 *mapred* libs,
it should just work with MRv2.
2. If you have the source code that refers to MRv1 *mapred* libs, it should
be compilable without code changes. Of course, you're free to change your
code.
3. If you have the binaries that were compiled against MRv1 *mapreduce* libs,
it may not be executable directly with MRv2, but you should able to compile
it against MRv2 *mapreduce* libs without code changes, and execute it.
- Zhijie
On Tue, Apr 15, 2014 at 12:44 PM, Radhe Radhe
<ra...@live.com>wrote:
> Thanks John for your comments,
>
> I believe MRv2 has support for both the old *mapred* APIs and new
> *mapreduce* APIs.
>
> I see this way:
> [1.] One may have binaries i.e. jar file of the M\R program that used old
> *mapred* APIs
> This will work directly on MRv2(YARN).
>
> [2.] One may have the source code i.e. Java Programs of the M\R program
> that used old *mapred* APIs
> For this I need to recompile and generate the binaries i.e. jar file.
> Do I have to change the old *org.apache.hadoop.mapred* APIs to new *
> org.apache.hadoop.mapreduce* APIs? or No code changes are needed?
>
> -RR
>
> > Date: Mon, 14 Apr 2014 10:37:56 -0400
> > Subject: Re: Doubt regarding Binary Compatibility\Source Compatibility
> with old *mapred* APIs and new *mapreduce* APIs in Hadoop
> > From: john.meagher@gmail.com
> > To: user@hadoop.apache.org
>
> >
> > Also, "Source Compatibility" also means ONLY a recompile is needed.
> > No code changes should be needed.
> >
> > On Mon, Apr 14, 2014 at 10:37 AM, John Meagher <jo...@gmail.com>
> wrote:
> > > Source Compatibility = you need to recompile and use the new version
> > > as part of the compilation
> > >
> > > Binary Compatibility = you can take something compiled against the old
> > > version and run it on the new version
> > >
> > > On Mon, Apr 14, 2014 at 9:19 AM, Radhe Radhe
> > > <ra...@live.com> wrote:
> > >> Hello People,
> > >>
> > >> As per the Apache site
> > >>
> http://hadoop.apache.org/docs/r2.3.0/hadoop-mapreduce-client/hadoop-mapreduce-client-core/MapReduce_Compatibility_Hadoop1_Hadoop2.html
> > >>
> > >> Binary Compatibility
> > >> ====================
> > >> First, we ensure binary compatibility to the applications that use old
> > >> mapred APIs. This means that applications which were built against
> MRv1
> > >> mapred APIs can run directly on YARN without recompilation, merely by
> > >> pointing them to an Apache Hadoop 2.x cluster via configuration.
> > >>
> > >> Source Compatibility
> > >> ====================
> > >> We cannot ensure complete binary compatibility with the applications
> that
> > >> use mapreduce APIs, as these APIs have evolved a lot since MRv1.
> However, we
> > >> ensure source compatibility for mapreduce APIs that break binary
> > >> compatibility. In other words, users should recompile their
> applications
> > >> that use mapreduce APIs against MRv2 jars. One notable binary
> > >> incompatibility break is Counter and CounterGroup.
> > >>
> > >> For "Binary Compatibility" I understand that if I had build a MR job
> with
> > >> old *mapred* APIs then they can be run directly on YARN without and
> changes.
> > >>
> > >> Can anybody explain what do we mean by "Source Compatibility" here
> and also
> > >> a use case where one will need it?
> > >>
> > >> Does that mean code changes if I already have a MR job source code
> written
> > >> with with old *mapred* APIs and I need to make some changes to it to
> run in
> > >> then I need to use the new "mapreduce* API and generate the new
> binaries?
> > >>
> > >> Thanks,
> > >> -RR
> > >>
> > >>
>
--
Zhijie Shen
Hortonworks Inc.
http://hortonworks.com/
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
RE: Doubt regarding Binary Compatibility\Source Compatibility with
old *mapred* APIs and new *mapreduce* APIs in Hadoop
Posted by Radhe Radhe <ra...@live.com>.
Thanks John for your comments,
I believe MRv2 has support for both the old *mapred* APIs and new *mapreduce* APIs.
I see this way:[1.] One may have binaries i.e. jar file of the M\R program that used old *mapred* APIsThis will work directly on MRv2(YARN).
[2.] One may have the source code i.e. Java Programs of the M\R program that used old *mapred* APIsFor this I need to recompile and generate the binaries i.e. jar file. Do I have to change the old *org.apache.hadoop.mapred* APIs to new *org.apache.hadoop.mapreduce* APIs? or No code changes are needed?
-RR
> Date: Mon, 14 Apr 2014 10:37:56 -0400
> Subject: Re: Doubt regarding Binary Compatibility\Source Compatibility with old *mapred* APIs and new *mapreduce* APIs in Hadoop
> From: john.meagher@gmail.com
> To: user@hadoop.apache.org
>
> Also, "Source Compatibility" also means ONLY a recompile is needed.
> No code changes should be needed.
>
> On Mon, Apr 14, 2014 at 10:37 AM, John Meagher <jo...@gmail.com> wrote:
> > Source Compatibility = you need to recompile and use the new version
> > as part of the compilation
> >
> > Binary Compatibility = you can take something compiled against the old
> > version and run it on the new version
> >
> > On Mon, Apr 14, 2014 at 9:19 AM, Radhe Radhe
> > <ra...@live.com> wrote:
> >> Hello People,
> >>
> >> As per the Apache site
> >> http://hadoop.apache.org/docs/r2.3.0/hadoop-mapreduce-client/hadoop-mapreduce-client-core/MapReduce_Compatibility_Hadoop1_Hadoop2.html
> >>
> >> Binary Compatibility
> >> ====================
> >> First, we ensure binary compatibility to the applications that use old
> >> mapred APIs. This means that applications which were built against MRv1
> >> mapred APIs can run directly on YARN without recompilation, merely by
> >> pointing them to an Apache Hadoop 2.x cluster via configuration.
> >>
> >> Source Compatibility
> >> ====================
> >> We cannot ensure complete binary compatibility with the applications that
> >> use mapreduce APIs, as these APIs have evolved a lot since MRv1. However, we
> >> ensure source compatibility for mapreduce APIs that break binary
> >> compatibility. In other words, users should recompile their applications
> >> that use mapreduce APIs against MRv2 jars. One notable binary
> >> incompatibility break is Counter and CounterGroup.
> >>
> >> For "Binary Compatibility" I understand that if I had build a MR job with
> >> old *mapred* APIs then they can be run directly on YARN without and changes.
> >>
> >> Can anybody explain what do we mean by "Source Compatibility" here and also
> >> a use case where one will need it?
> >>
> >> Does that mean code changes if I already have a MR job source code written
> >> with with old *mapred* APIs and I need to make some changes to it to run in
> >> then I need to use the new "mapreduce* API and generate the new binaries?
> >>
> >> Thanks,
> >> -RR
> >>
> >>
RE: Doubt regarding Binary Compatibility\Source Compatibility with
old *mapred* APIs and new *mapreduce* APIs in Hadoop
Posted by Radhe Radhe <ra...@live.com>.
Thanks John for your comments,
I believe MRv2 has support for both the old *mapred* APIs and new *mapreduce* APIs.
I see this way:[1.] One may have binaries i.e. jar file of the M\R program that used old *mapred* APIsThis will work directly on MRv2(YARN).
[2.] One may have the source code i.e. Java Programs of the M\R program that used old *mapred* APIsFor this I need to recompile and generate the binaries i.e. jar file. Do I have to change the old *org.apache.hadoop.mapred* APIs to new *org.apache.hadoop.mapreduce* APIs? or No code changes are needed?
-RR
> Date: Mon, 14 Apr 2014 10:37:56 -0400
> Subject: Re: Doubt regarding Binary Compatibility\Source Compatibility with old *mapred* APIs and new *mapreduce* APIs in Hadoop
> From: john.meagher@gmail.com
> To: user@hadoop.apache.org
>
> Also, "Source Compatibility" also means ONLY a recompile is needed.
> No code changes should be needed.
>
> On Mon, Apr 14, 2014 at 10:37 AM, John Meagher <jo...@gmail.com> wrote:
> > Source Compatibility = you need to recompile and use the new version
> > as part of the compilation
> >
> > Binary Compatibility = you can take something compiled against the old
> > version and run it on the new version
> >
> > On Mon, Apr 14, 2014 at 9:19 AM, Radhe Radhe
> > <ra...@live.com> wrote:
> >> Hello People,
> >>
> >> As per the Apache site
> >> http://hadoop.apache.org/docs/r2.3.0/hadoop-mapreduce-client/hadoop-mapreduce-client-core/MapReduce_Compatibility_Hadoop1_Hadoop2.html
> >>
> >> Binary Compatibility
> >> ====================
> >> First, we ensure binary compatibility to the applications that use old
> >> mapred APIs. This means that applications which were built against MRv1
> >> mapred APIs can run directly on YARN without recompilation, merely by
> >> pointing them to an Apache Hadoop 2.x cluster via configuration.
> >>
> >> Source Compatibility
> >> ====================
> >> We cannot ensure complete binary compatibility with the applications that
> >> use mapreduce APIs, as these APIs have evolved a lot since MRv1. However, we
> >> ensure source compatibility for mapreduce APIs that break binary
> >> compatibility. In other words, users should recompile their applications
> >> that use mapreduce APIs against MRv2 jars. One notable binary
> >> incompatibility break is Counter and CounterGroup.
> >>
> >> For "Binary Compatibility" I understand that if I had build a MR job with
> >> old *mapred* APIs then they can be run directly on YARN without and changes.
> >>
> >> Can anybody explain what do we mean by "Source Compatibility" here and also
> >> a use case where one will need it?
> >>
> >> Does that mean code changes if I already have a MR job source code written
> >> with with old *mapred* APIs and I need to make some changes to it to run in
> >> then I need to use the new "mapreduce* API and generate the new binaries?
> >>
> >> Thanks,
> >> -RR
> >>
> >>
RE: Doubt regarding Binary Compatibility\Source Compatibility with
old *mapred* APIs and new *mapreduce* APIs in Hadoop
Posted by Radhe Radhe <ra...@live.com>.
Thanks John for your comments,
I believe MRv2 has support for both the old *mapred* APIs and new *mapreduce* APIs.
I see this way:[1.] One may have binaries i.e. jar file of the M\R program that used old *mapred* APIsThis will work directly on MRv2(YARN).
[2.] One may have the source code i.e. Java Programs of the M\R program that used old *mapred* APIsFor this I need to recompile and generate the binaries i.e. jar file. Do I have to change the old *org.apache.hadoop.mapred* APIs to new *org.apache.hadoop.mapreduce* APIs? or No code changes are needed?
-RR
> Date: Mon, 14 Apr 2014 10:37:56 -0400
> Subject: Re: Doubt regarding Binary Compatibility\Source Compatibility with old *mapred* APIs and new *mapreduce* APIs in Hadoop
> From: john.meagher@gmail.com
> To: user@hadoop.apache.org
>
> Also, "Source Compatibility" also means ONLY a recompile is needed.
> No code changes should be needed.
>
> On Mon, Apr 14, 2014 at 10:37 AM, John Meagher <jo...@gmail.com> wrote:
> > Source Compatibility = you need to recompile and use the new version
> > as part of the compilation
> >
> > Binary Compatibility = you can take something compiled against the old
> > version and run it on the new version
> >
> > On Mon, Apr 14, 2014 at 9:19 AM, Radhe Radhe
> > <ra...@live.com> wrote:
> >> Hello People,
> >>
> >> As per the Apache site
> >> http://hadoop.apache.org/docs/r2.3.0/hadoop-mapreduce-client/hadoop-mapreduce-client-core/MapReduce_Compatibility_Hadoop1_Hadoop2.html
> >>
> >> Binary Compatibility
> >> ====================
> >> First, we ensure binary compatibility to the applications that use old
> >> mapred APIs. This means that applications which were built against MRv1
> >> mapred APIs can run directly on YARN without recompilation, merely by
> >> pointing them to an Apache Hadoop 2.x cluster via configuration.
> >>
> >> Source Compatibility
> >> ====================
> >> We cannot ensure complete binary compatibility with the applications that
> >> use mapreduce APIs, as these APIs have evolved a lot since MRv1. However, we
> >> ensure source compatibility for mapreduce APIs that break binary
> >> compatibility. In other words, users should recompile their applications
> >> that use mapreduce APIs against MRv2 jars. One notable binary
> >> incompatibility break is Counter and CounterGroup.
> >>
> >> For "Binary Compatibility" I understand that if I had build a MR job with
> >> old *mapred* APIs then they can be run directly on YARN without and changes.
> >>
> >> Can anybody explain what do we mean by "Source Compatibility" here and also
> >> a use case where one will need it?
> >>
> >> Does that mean code changes if I already have a MR job source code written
> >> with with old *mapred* APIs and I need to make some changes to it to run in
> >> then I need to use the new "mapreduce* API and generate the new binaries?
> >>
> >> Thanks,
> >> -RR
> >>
> >>
RE: Doubt regarding Binary Compatibility\Source Compatibility with
old *mapred* APIs and new *mapreduce* APIs in Hadoop
Posted by Radhe Radhe <ra...@live.com>.
Thanks John for your comments,
I believe MRv2 has support for both the old *mapred* APIs and new *mapreduce* APIs.
I see this way:[1.] One may have binaries i.e. jar file of the M\R program that used old *mapred* APIsThis will work directly on MRv2(YARN).
[2.] One may have the source code i.e. Java Programs of the M\R program that used old *mapred* APIsFor this I need to recompile and generate the binaries i.e. jar file. Do I have to change the old *org.apache.hadoop.mapred* APIs to new *org.apache.hadoop.mapreduce* APIs? or No code changes are needed?
-RR
> Date: Mon, 14 Apr 2014 10:37:56 -0400
> Subject: Re: Doubt regarding Binary Compatibility\Source Compatibility with old *mapred* APIs and new *mapreduce* APIs in Hadoop
> From: john.meagher@gmail.com
> To: user@hadoop.apache.org
>
> Also, "Source Compatibility" also means ONLY a recompile is needed.
> No code changes should be needed.
>
> On Mon, Apr 14, 2014 at 10:37 AM, John Meagher <jo...@gmail.com> wrote:
> > Source Compatibility = you need to recompile and use the new version
> > as part of the compilation
> >
> > Binary Compatibility = you can take something compiled against the old
> > version and run it on the new version
> >
> > On Mon, Apr 14, 2014 at 9:19 AM, Radhe Radhe
> > <ra...@live.com> wrote:
> >> Hello People,
> >>
> >> As per the Apache site
> >> http://hadoop.apache.org/docs/r2.3.0/hadoop-mapreduce-client/hadoop-mapreduce-client-core/MapReduce_Compatibility_Hadoop1_Hadoop2.html
> >>
> >> Binary Compatibility
> >> ====================
> >> First, we ensure binary compatibility to the applications that use old
> >> mapred APIs. This means that applications which were built against MRv1
> >> mapred APIs can run directly on YARN without recompilation, merely by
> >> pointing them to an Apache Hadoop 2.x cluster via configuration.
> >>
> >> Source Compatibility
> >> ====================
> >> We cannot ensure complete binary compatibility with the applications that
> >> use mapreduce APIs, as these APIs have evolved a lot since MRv1. However, we
> >> ensure source compatibility for mapreduce APIs that break binary
> >> compatibility. In other words, users should recompile their applications
> >> that use mapreduce APIs against MRv2 jars. One notable binary
> >> incompatibility break is Counter and CounterGroup.
> >>
> >> For "Binary Compatibility" I understand that if I had build a MR job with
> >> old *mapred* APIs then they can be run directly on YARN without and changes.
> >>
> >> Can anybody explain what do we mean by "Source Compatibility" here and also
> >> a use case where one will need it?
> >>
> >> Does that mean code changes if I already have a MR job source code written
> >> with with old *mapred* APIs and I need to make some changes to it to run in
> >> then I need to use the new "mapreduce* API and generate the new binaries?
> >>
> >> Thanks,
> >> -RR
> >>
> >>
Re: Doubt regarding Binary Compatibility\Source Compatibility with
old *mapred* APIs and new *mapreduce* APIs in Hadoop
Posted by John Meagher <jo...@gmail.com>.
Also, "Source Compatibility" also means ONLY a recompile is needed.
No code changes should be needed.
On Mon, Apr 14, 2014 at 10:37 AM, John Meagher <jo...@gmail.com> wrote:
> Source Compatibility = you need to recompile and use the new version
> as part of the compilation
>
> Binary Compatibility = you can take something compiled against the old
> version and run it on the new version
>
> On Mon, Apr 14, 2014 at 9:19 AM, Radhe Radhe
> <ra...@live.com> wrote:
>> Hello People,
>>
>> As per the Apache site
>> http://hadoop.apache.org/docs/r2.3.0/hadoop-mapreduce-client/hadoop-mapreduce-client-core/MapReduce_Compatibility_Hadoop1_Hadoop2.html
>>
>> Binary Compatibility
>> ====================
>> First, we ensure binary compatibility to the applications that use old
>> mapred APIs. This means that applications which were built against MRv1
>> mapred APIs can run directly on YARN without recompilation, merely by
>> pointing them to an Apache Hadoop 2.x cluster via configuration.
>>
>> Source Compatibility
>> ====================
>> We cannot ensure complete binary compatibility with the applications that
>> use mapreduce APIs, as these APIs have evolved a lot since MRv1. However, we
>> ensure source compatibility for mapreduce APIs that break binary
>> compatibility. In other words, users should recompile their applications
>> that use mapreduce APIs against MRv2 jars. One notable binary
>> incompatibility break is Counter and CounterGroup.
>>
>> For "Binary Compatibility" I understand that if I had build a MR job with
>> old *mapred* APIs then they can be run directly on YARN without and changes.
>>
>> Can anybody explain what do we mean by "Source Compatibility" here and also
>> a use case where one will need it?
>>
>> Does that mean code changes if I already have a MR job source code written
>> with with old *mapred* APIs and I need to make some changes to it to run in
>> then I need to use the new "mapreduce* API and generate the new binaries?
>>
>> Thanks,
>> -RR
>>
>>
Re: Doubt regarding Binary Compatibility\Source Compatibility with
old *mapred* APIs and new *mapreduce* APIs in Hadoop
Posted by John Meagher <jo...@gmail.com>.
Also, "Source Compatibility" also means ONLY a recompile is needed.
No code changes should be needed.
On Mon, Apr 14, 2014 at 10:37 AM, John Meagher <jo...@gmail.com> wrote:
> Source Compatibility = you need to recompile and use the new version
> as part of the compilation
>
> Binary Compatibility = you can take something compiled against the old
> version and run it on the new version
>
> On Mon, Apr 14, 2014 at 9:19 AM, Radhe Radhe
> <ra...@live.com> wrote:
>> Hello People,
>>
>> As per the Apache site
>> http://hadoop.apache.org/docs/r2.3.0/hadoop-mapreduce-client/hadoop-mapreduce-client-core/MapReduce_Compatibility_Hadoop1_Hadoop2.html
>>
>> Binary Compatibility
>> ====================
>> First, we ensure binary compatibility to the applications that use old
>> mapred APIs. This means that applications which were built against MRv1
>> mapred APIs can run directly on YARN without recompilation, merely by
>> pointing them to an Apache Hadoop 2.x cluster via configuration.
>>
>> Source Compatibility
>> ====================
>> We cannot ensure complete binary compatibility with the applications that
>> use mapreduce APIs, as these APIs have evolved a lot since MRv1. However, we
>> ensure source compatibility for mapreduce APIs that break binary
>> compatibility. In other words, users should recompile their applications
>> that use mapreduce APIs against MRv2 jars. One notable binary
>> incompatibility break is Counter and CounterGroup.
>>
>> For "Binary Compatibility" I understand that if I had build a MR job with
>> old *mapred* APIs then they can be run directly on YARN without and changes.
>>
>> Can anybody explain what do we mean by "Source Compatibility" here and also
>> a use case where one will need it?
>>
>> Does that mean code changes if I already have a MR job source code written
>> with with old *mapred* APIs and I need to make some changes to it to run in
>> then I need to use the new "mapreduce* API and generate the new binaries?
>>
>> Thanks,
>> -RR
>>
>>
Re: Doubt regarding Binary Compatibility\Source Compatibility with
old *mapred* APIs and new *mapreduce* APIs in Hadoop
Posted by John Meagher <jo...@gmail.com>.
Also, "Source Compatibility" also means ONLY a recompile is needed.
No code changes should be needed.
On Mon, Apr 14, 2014 at 10:37 AM, John Meagher <jo...@gmail.com> wrote:
> Source Compatibility = you need to recompile and use the new version
> as part of the compilation
>
> Binary Compatibility = you can take something compiled against the old
> version and run it on the new version
>
> On Mon, Apr 14, 2014 at 9:19 AM, Radhe Radhe
> <ra...@live.com> wrote:
>> Hello People,
>>
>> As per the Apache site
>> http://hadoop.apache.org/docs/r2.3.0/hadoop-mapreduce-client/hadoop-mapreduce-client-core/MapReduce_Compatibility_Hadoop1_Hadoop2.html
>>
>> Binary Compatibility
>> ====================
>> First, we ensure binary compatibility to the applications that use old
>> mapred APIs. This means that applications which were built against MRv1
>> mapred APIs can run directly on YARN without recompilation, merely by
>> pointing them to an Apache Hadoop 2.x cluster via configuration.
>>
>> Source Compatibility
>> ====================
>> We cannot ensure complete binary compatibility with the applications that
>> use mapreduce APIs, as these APIs have evolved a lot since MRv1. However, we
>> ensure source compatibility for mapreduce APIs that break binary
>> compatibility. In other words, users should recompile their applications
>> that use mapreduce APIs against MRv2 jars. One notable binary
>> incompatibility break is Counter and CounterGroup.
>>
>> For "Binary Compatibility" I understand that if I had build a MR job with
>> old *mapred* APIs then they can be run directly on YARN without and changes.
>>
>> Can anybody explain what do we mean by "Source Compatibility" here and also
>> a use case where one will need it?
>>
>> Does that mean code changes if I already have a MR job source code written
>> with with old *mapred* APIs and I need to make some changes to it to run in
>> then I need to use the new "mapreduce* API and generate the new binaries?
>>
>> Thanks,
>> -RR
>>
>>
Re: Doubt regarding Binary Compatibility\Source Compatibility with
old *mapred* APIs and new *mapreduce* APIs in Hadoop
Posted by John Meagher <jo...@gmail.com>.
Also, "Source Compatibility" also means ONLY a recompile is needed.
No code changes should be needed.
On Mon, Apr 14, 2014 at 10:37 AM, John Meagher <jo...@gmail.com> wrote:
> Source Compatibility = you need to recompile and use the new version
> as part of the compilation
>
> Binary Compatibility = you can take something compiled against the old
> version and run it on the new version
>
> On Mon, Apr 14, 2014 at 9:19 AM, Radhe Radhe
> <ra...@live.com> wrote:
>> Hello People,
>>
>> As per the Apache site
>> http://hadoop.apache.org/docs/r2.3.0/hadoop-mapreduce-client/hadoop-mapreduce-client-core/MapReduce_Compatibility_Hadoop1_Hadoop2.html
>>
>> Binary Compatibility
>> ====================
>> First, we ensure binary compatibility to the applications that use old
>> mapred APIs. This means that applications which were built against MRv1
>> mapred APIs can run directly on YARN without recompilation, merely by
>> pointing them to an Apache Hadoop 2.x cluster via configuration.
>>
>> Source Compatibility
>> ====================
>> We cannot ensure complete binary compatibility with the applications that
>> use mapreduce APIs, as these APIs have evolved a lot since MRv1. However, we
>> ensure source compatibility for mapreduce APIs that break binary
>> compatibility. In other words, users should recompile their applications
>> that use mapreduce APIs against MRv2 jars. One notable binary
>> incompatibility break is Counter and CounterGroup.
>>
>> For "Binary Compatibility" I understand that if I had build a MR job with
>> old *mapred* APIs then they can be run directly on YARN without and changes.
>>
>> Can anybody explain what do we mean by "Source Compatibility" here and also
>> a use case where one will need it?
>>
>> Does that mean code changes if I already have a MR job source code written
>> with with old *mapred* APIs and I need to make some changes to it to run in
>> then I need to use the new "mapreduce* API and generate the new binaries?
>>
>> Thanks,
>> -RR
>>
>>
Re: Doubt regarding Binary Compatibility\Source Compatibility with
old *mapred* APIs and new *mapreduce* APIs in Hadoop
Posted by John Meagher <jo...@gmail.com>.
Source Compatibility = you need to recompile and use the new version
as part of the compilation
Binary Compatibility = you can take something compiled against the old
version and run it on the new version
On Mon, Apr 14, 2014 at 9:19 AM, Radhe Radhe
<ra...@live.com> wrote:
> Hello People,
>
> As per the Apache site
> http://hadoop.apache.org/docs/r2.3.0/hadoop-mapreduce-client/hadoop-mapreduce-client-core/MapReduce_Compatibility_Hadoop1_Hadoop2.html
>
> Binary Compatibility
> ====================
> First, we ensure binary compatibility to the applications that use old
> mapred APIs. This means that applications which were built against MRv1
> mapred APIs can run directly on YARN without recompilation, merely by
> pointing them to an Apache Hadoop 2.x cluster via configuration.
>
> Source Compatibility
> ====================
> We cannot ensure complete binary compatibility with the applications that
> use mapreduce APIs, as these APIs have evolved a lot since MRv1. However, we
> ensure source compatibility for mapreduce APIs that break binary
> compatibility. In other words, users should recompile their applications
> that use mapreduce APIs against MRv2 jars. One notable binary
> incompatibility break is Counter and CounterGroup.
>
> For "Binary Compatibility" I understand that if I had build a MR job with
> old *mapred* APIs then they can be run directly on YARN without and changes.
>
> Can anybody explain what do we mean by "Source Compatibility" here and also
> a use case where one will need it?
>
> Does that mean code changes if I already have a MR job source code written
> with with old *mapred* APIs and I need to make some changes to it to run in
> then I need to use the new "mapreduce* API and generate the new binaries?
>
> Thanks,
> -RR
>
>
Re: Doubt regarding Binary Compatibility\Source Compatibility with
old *mapred* APIs and new *mapreduce* APIs in Hadoop
Posted by John Meagher <jo...@gmail.com>.
Source Compatibility = you need to recompile and use the new version
as part of the compilation
Binary Compatibility = you can take something compiled against the old
version and run it on the new version
On Mon, Apr 14, 2014 at 9:19 AM, Radhe Radhe
<ra...@live.com> wrote:
> Hello People,
>
> As per the Apache site
> http://hadoop.apache.org/docs/r2.3.0/hadoop-mapreduce-client/hadoop-mapreduce-client-core/MapReduce_Compatibility_Hadoop1_Hadoop2.html
>
> Binary Compatibility
> ====================
> First, we ensure binary compatibility to the applications that use old
> mapred APIs. This means that applications which were built against MRv1
> mapred APIs can run directly on YARN without recompilation, merely by
> pointing them to an Apache Hadoop 2.x cluster via configuration.
>
> Source Compatibility
> ====================
> We cannot ensure complete binary compatibility with the applications that
> use mapreduce APIs, as these APIs have evolved a lot since MRv1. However, we
> ensure source compatibility for mapreduce APIs that break binary
> compatibility. In other words, users should recompile their applications
> that use mapreduce APIs against MRv2 jars. One notable binary
> incompatibility break is Counter and CounterGroup.
>
> For "Binary Compatibility" I understand that if I had build a MR job with
> old *mapred* APIs then they can be run directly on YARN without and changes.
>
> Can anybody explain what do we mean by "Source Compatibility" here and also
> a use case where one will need it?
>
> Does that mean code changes if I already have a MR job source code written
> with with old *mapred* APIs and I need to make some changes to it to run in
> then I need to use the new "mapreduce* API and generate the new binaries?
>
> Thanks,
> -RR
>
>
Re: Doubt regarding Binary Compatibility\Source Compatibility with
old *mapred* APIs and new *mapreduce* APIs in Hadoop
Posted by John Meagher <jo...@gmail.com>.
Source Compatibility = you need to recompile and use the new version
as part of the compilation
Binary Compatibility = you can take something compiled against the old
version and run it on the new version
On Mon, Apr 14, 2014 at 9:19 AM, Radhe Radhe
<ra...@live.com> wrote:
> Hello People,
>
> As per the Apache site
> http://hadoop.apache.org/docs/r2.3.0/hadoop-mapreduce-client/hadoop-mapreduce-client-core/MapReduce_Compatibility_Hadoop1_Hadoop2.html
>
> Binary Compatibility
> ====================
> First, we ensure binary compatibility to the applications that use old
> mapred APIs. This means that applications which were built against MRv1
> mapred APIs can run directly on YARN without recompilation, merely by
> pointing them to an Apache Hadoop 2.x cluster via configuration.
>
> Source Compatibility
> ====================
> We cannot ensure complete binary compatibility with the applications that
> use mapreduce APIs, as these APIs have evolved a lot since MRv1. However, we
> ensure source compatibility for mapreduce APIs that break binary
> compatibility. In other words, users should recompile their applications
> that use mapreduce APIs against MRv2 jars. One notable binary
> incompatibility break is Counter and CounterGroup.
>
> For "Binary Compatibility" I understand that if I had build a MR job with
> old *mapred* APIs then they can be run directly on YARN without and changes.
>
> Can anybody explain what do we mean by "Source Compatibility" here and also
> a use case where one will need it?
>
> Does that mean code changes if I already have a MR job source code written
> with with old *mapred* APIs and I need to make some changes to it to run in
> then I need to use the new "mapreduce* API and generate the new binaries?
>
> Thanks,
> -RR
>
>
Re: Doubt regarding Binary Compatibility\Source Compatibility with
old *mapred* APIs and new *mapreduce* APIs in Hadoop
Posted by John Meagher <jo...@gmail.com>.
Source Compatibility = you need to recompile and use the new version
as part of the compilation
Binary Compatibility = you can take something compiled against the old
version and run it on the new version
On Mon, Apr 14, 2014 at 9:19 AM, Radhe Radhe
<ra...@live.com> wrote:
> Hello People,
>
> As per the Apache site
> http://hadoop.apache.org/docs/r2.3.0/hadoop-mapreduce-client/hadoop-mapreduce-client-core/MapReduce_Compatibility_Hadoop1_Hadoop2.html
>
> Binary Compatibility
> ====================
> First, we ensure binary compatibility to the applications that use old
> mapred APIs. This means that applications which were built against MRv1
> mapred APIs can run directly on YARN without recompilation, merely by
> pointing them to an Apache Hadoop 2.x cluster via configuration.
>
> Source Compatibility
> ====================
> We cannot ensure complete binary compatibility with the applications that
> use mapreduce APIs, as these APIs have evolved a lot since MRv1. However, we
> ensure source compatibility for mapreduce APIs that break binary
> compatibility. In other words, users should recompile their applications
> that use mapreduce APIs against MRv2 jars. One notable binary
> incompatibility break is Counter and CounterGroup.
>
> For "Binary Compatibility" I understand that if I had build a MR job with
> old *mapred* APIs then they can be run directly on YARN without and changes.
>
> Can anybody explain what do we mean by "Source Compatibility" here and also
> a use case where one will need it?
>
> Does that mean code changes if I already have a MR job source code written
> with with old *mapred* APIs and I need to make some changes to it to run in
> then I need to use the new "mapreduce* API and generate the new binaries?
>
> Thanks,
> -RR
>
>