You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pig.apache.org by Lorand Bendig <lb...@gmail.com> on 2014/06/19 15:50:02 UTC
Issue when resolving Tez dependencies
Hi,
After updating to the latest revision from trunk I couldn't build Pig
due to the issue described in PIG-4019.
The parameter list of
org.apache.tez.dag.api.VertexManagerPluginContext#setVertexParallelism
has changed and however Daniel's patch fixed it, my build still failed.
I found, that it's because of the Tez SNAPSHOT dependencies. The ivy
resolver for apache-snapshots sets checkModified="true" which updates
the metadata of Tez dependencies if there are modifications, but since
the intermediary Tez builds are published under the same revision
(0.5.0-incubating-SNAPSHOT) the local cache will only be populated for
the first time with the jars but won't get updated afterwards. So API
changes at Tez side which affects Pig, will result in build error.
Am I the only one with this issue? Or am I missing something?
A solution would be to set 'changing="true"' for the Tez dependencies in
ivy.xml.
Thus ivy would always look up the repository to check whether there are
changes and would download the most recent timestamped 0.5.0 snapshot
jars. There are, however couple of issues with this:
- Resolution of Tez jars will fail during offline builds, when one sets
usecacheonly="true" for ivy:resolve. Since by
default it's not the case, I wouldn't worry about this.
- This attribute definitely needs to be removed once Tez dependencies
evolve from SNAPSHOT
Please let me know what you think and I'll file a JIRA if necessary.
Thanks,
Lorand
Re: Issue when resolving Tez dependencies
Posted by Lorand Bendig <lb...@gmail.com>.
Hi Daniel and Cheolsoo,
No problem, thanks for the clarification. I filed PIG-4027 to fix this
issue.
Thanks,
Lorand
On 06/19/2014 08:39 PM, Daniel Dai wrote:
> Sorry to break your build, Lorand. We will try to stick to a tez
> release (0.5.0 maybe) once tez becomes stable. Now we don't have a
> choice since we need tez bug fixes which only exist in tez trunk.
> Before that, we can do whatever to make other developer's life easier.
> I don't think the issues you listed is a problem, just leave a Jira
> open, so by the time we release Pig 0.14.0, we don't forget to remove
> the flag.
>
> Thanks,
> Daniel
>
> On Thu, Jun 19, 2014 at 11:01 AM, Cheolsoo Park <pi...@gmail.com> wrote:
>> Hi Lorand,
>>
>> Sorry for the inconvenience. True that Tez snapshot jars are updated
>> frequently. I haven't run into this problem only because I closely follow
>> the Pig on Tez development.
>>
>> +1 to set 'changing="true"' until we pin Tez to a stable release.
>>
>> Thanks,
>> Cheolsoo
>>
>>
>> On Thu, Jun 19, 2014 at 6:50 AM, Lorand Bendig <lb...@gmail.com> wrote:
>>
>>> Hi,
>>>
>>> After updating to the latest revision from trunk I couldn't build Pig due
>>> to the issue described in PIG-4019.
>>> The parameter list of org.apache.tez.dag.api.VertexManagerPluginContext#setVertexParallelism
>>> has changed and however Daniel's patch fixed it, my build still failed.
>>> I found, that it's because of the Tez SNAPSHOT dependencies. The ivy
>>> resolver for apache-snapshots sets checkModified="true" which updates the
>>> metadata of Tez dependencies if there are modifications, but since the
>>> intermediary Tez builds are published under the same revision
>>> (0.5.0-incubating-SNAPSHOT) the local cache will only be populated for the
>>> first time with the jars but won't get updated afterwards. So API changes
>>> at Tez side which affects Pig, will result in build error.
>>> Am I the only one with this issue? Or am I missing something?
>>>
>>> A solution would be to set 'changing="true"' for the Tez dependencies in
>>> ivy.xml.
>>> Thus ivy would always look up the repository to check whether there are
>>> changes and would download the most recent timestamped 0.5.0 snapshot jars.
>>> There are, however couple of issues with this:
>>>
>>> - Resolution of Tez jars will fail during offline builds, when one sets
>>> usecacheonly="true" for ivy:resolve. Since by
>>> default it's not the case, I wouldn't worry about this.
>>> - This attribute definitely needs to be removed once Tez dependencies
>>> evolve from SNAPSHOT
>>>
>>> Please let me know what you think and I'll file a JIRA if necessary.
>>>
>>> Thanks,
>>> Lorand
>>>
Re: Issue when resolving Tez dependencies
Posted by Daniel Dai <da...@hortonworks.com>.
Sorry to break your build, Lorand. We will try to stick to a tez
release (0.5.0 maybe) once tez becomes stable. Now we don't have a
choice since we need tez bug fixes which only exist in tez trunk.
Before that, we can do whatever to make other developer's life easier.
I don't think the issues you listed is a problem, just leave a Jira
open, so by the time we release Pig 0.14.0, we don't forget to remove
the flag.
Thanks,
Daniel
On Thu, Jun 19, 2014 at 11:01 AM, Cheolsoo Park <pi...@gmail.com> wrote:
> Hi Lorand,
>
> Sorry for the inconvenience. True that Tez snapshot jars are updated
> frequently. I haven't run into this problem only because I closely follow
> the Pig on Tez development.
>
> +1 to set 'changing="true"' until we pin Tez to a stable release.
>
> Thanks,
> Cheolsoo
>
>
> On Thu, Jun 19, 2014 at 6:50 AM, Lorand Bendig <lb...@gmail.com> wrote:
>
>> Hi,
>>
>> After updating to the latest revision from trunk I couldn't build Pig due
>> to the issue described in PIG-4019.
>> The parameter list of org.apache.tez.dag.api.VertexManagerPluginContext#setVertexParallelism
>> has changed and however Daniel's patch fixed it, my build still failed.
>> I found, that it's because of the Tez SNAPSHOT dependencies. The ivy
>> resolver for apache-snapshots sets checkModified="true" which updates the
>> metadata of Tez dependencies if there are modifications, but since the
>> intermediary Tez builds are published under the same revision
>> (0.5.0-incubating-SNAPSHOT) the local cache will only be populated for the
>> first time with the jars but won't get updated afterwards. So API changes
>> at Tez side which affects Pig, will result in build error.
>> Am I the only one with this issue? Or am I missing something?
>>
>> A solution would be to set 'changing="true"' for the Tez dependencies in
>> ivy.xml.
>> Thus ivy would always look up the repository to check whether there are
>> changes and would download the most recent timestamped 0.5.0 snapshot jars.
>> There are, however couple of issues with this:
>>
>> - Resolution of Tez jars will fail during offline builds, when one sets
>> usecacheonly="true" for ivy:resolve. Since by
>> default it's not the case, I wouldn't worry about this.
>> - This attribute definitely needs to be removed once Tez dependencies
>> evolve from SNAPSHOT
>>
>> Please let me know what you think and I'll file a JIRA if necessary.
>>
>> Thanks,
>> Lorand
>>
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: Issue when resolving Tez dependencies
Posted by Cheolsoo Park <pi...@gmail.com>.
Hi Lorand,
Sorry for the inconvenience. True that Tez snapshot jars are updated
frequently. I haven't run into this problem only because I closely follow
the Pig on Tez development.
+1 to set 'changing="true"' until we pin Tez to a stable release.
Thanks,
Cheolsoo
On Thu, Jun 19, 2014 at 6:50 AM, Lorand Bendig <lb...@gmail.com> wrote:
> Hi,
>
> After updating to the latest revision from trunk I couldn't build Pig due
> to the issue described in PIG-4019.
> The parameter list of org.apache.tez.dag.api.VertexManagerPluginContext#setVertexParallelism
> has changed and however Daniel's patch fixed it, my build still failed.
> I found, that it's because of the Tez SNAPSHOT dependencies. The ivy
> resolver for apache-snapshots sets checkModified="true" which updates the
> metadata of Tez dependencies if there are modifications, but since the
> intermediary Tez builds are published under the same revision
> (0.5.0-incubating-SNAPSHOT) the local cache will only be populated for the
> first time with the jars but won't get updated afterwards. So API changes
> at Tez side which affects Pig, will result in build error.
> Am I the only one with this issue? Or am I missing something?
>
> A solution would be to set 'changing="true"' for the Tez dependencies in
> ivy.xml.
> Thus ivy would always look up the repository to check whether there are
> changes and would download the most recent timestamped 0.5.0 snapshot jars.
> There are, however couple of issues with this:
>
> - Resolution of Tez jars will fail during offline builds, when one sets
> usecacheonly="true" for ivy:resolve. Since by
> default it's not the case, I wouldn't worry about this.
> - This attribute definitely needs to be removed once Tez dependencies
> evolve from SNAPSHOT
>
> Please let me know what you think and I'll file a JIRA if necessary.
>
> Thanks,
> Lorand
>