You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@tez.apache.org by Robert Grandl <rg...@yahoo.com> on 2017/02/28 18:36:00 UTC
Compile Tez
Hi guys,
I am doing some changes in Tez-0.7.1.
The way I am debugging as of now is:
1) Compile the entire project2) Copy the tez-0.7.1.tar.gz to hdfs and set "tez.lib.uris" to that location3) Untar tez-0.7.1-minimal.tar.gz in a local directory and then run the examples.
However, this process is tedious as recompiling the entire project takes some time.
It is possible to compile only certain subprojects (e.g. tez-runtime-library) and copy the corresponding jar only?
Right now, I am not able to do this as I keep the entire tez-0.7.1.tar.gz in the hdfs.
I was trying to put the untarred version in hdfs instead and point it to "tez.lib.uris" in tez-site.xml but that does not work.
Any thoughts?
- Robert
Re: Compile Tez
Posted by Robert Grandl <rg...@yahoo.com>.
Thanks a lot.
Scripting to rebuild the tar makes the most sense.
On Tuesday, February 28, 2017 12:48 PM, Siddharth Seth <ss...@apache.org> wrote:
How big is the cluster that you are working with. If it's a single node - one option is to use files already existing on the node. TezConfiguratin.TEZ_IGNORE_LIB_URIS set to true (this is in branch-0.7 only afaik, not 0.7.1), and setup TEZ_LIB_URIS_CLASSPATH to include the relevant files.Otherwise, I'd just script building out the tar / replacing relevant files in the existing one.
On Tue, Feb 28, 2017 at 10:36 AM, Robert Grandl <rg...@yahoo.com> wrote:
Hi guys,
I am doing some changes in Tez-0.7.1.
The way I am debugging as of now is:
1) Compile the entire project2) Copy the tez-0.7.1.tar.gz to hdfs and set "tez.lib.uris" to that location3) Untar tez-0.7.1-minimal.tar.gz in a local directory and then run the examples.
However, this process is tedious as recompiling the entire project takes some time.
It is possible to compile only certain subprojects (e.g. tez-runtime-library) and copy the corresponding jar only?
Right now, I am not able to do this as I keep the entire tez-0.7.1.tar.gz in the hdfs.
I was trying to put the untarred version in hdfs instead and point it to "tez.lib.uris" in tez-site.xml but that does not work.
Any thoughts?
- Robert
Re: Compile Tez
Posted by Siddharth Seth <ss...@apache.org>.
How big is the cluster that you are working with. If it's a single node -
one option is to use files already existing on the node.
TezConfiguratin.TEZ_IGNORE_LIB_URIS set to true (this is in branch-0.7 only
afaik, not 0.7.1), and setup TEZ_LIB_URIS_CLASSPATH to include the relevant
files.
Otherwise, I'd just script building out the tar / replacing relevant files
in the existing one.
On Tue, Feb 28, 2017 at 10:36 AM, Robert Grandl <rg...@yahoo.com> wrote:
> Hi guys,
>
> I am doing some changes in Tez-0.7.1.
>
> The way I am debugging as of now is:
> 1) Compile the entire project
> 2) Copy the tez-0.7.1.tar.gz to hdfs and set "tez.lib.uris" to that
> location
> 3) Untar tez-0.7.1-minimal.tar.gz in a local directory and then run the
> examples.
>
> However, this process is tedious as recompiling the entire project takes
> some time.
>
> It is possible to compile only certain subprojects (e.g.
> tez-runtime-library) and copy the corresponding jar only?
>
> Right now, I am not able to do this as I keep the entire tez-0.7.1.tar.gz
> in the hdfs.
> I was trying to put the untarred version in hdfs instead and point it to
> "tez.lib.uris" in tez-site.xml but that does not work.
>
> Any thoughts?
>
> - Robert
>