You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@calcite.apache.org by Julian Hyde <jh...@apache.org> on 2017/03/11 16:14:31 UTC
Pig tests very slow
The Pig tests seem to be running very slowly (several minutes), and CPU is idle. Is anyone else seeing this?
A thread dump reveals a wait in java/net/Inet6AddressImpl.lookupAllHostAddr. If so, maybe we’re hitting [1].
Julian
[1] https://developer.ibm.com/answers/questions/284205/how-can-we-improve-performance-in-javanetinet6addr.html <https://developer.ibm.com/answers/questions/284205/how-can-we-improve-performance-in-javanetinet6addr.html>
Re: Pig tests very slow
Posted by Maryann Xue <ma...@gmail.com>.
I did. Please see my comment on CALCITE-1693.
On Mon, Mar 13, 2017 at 4:00 PM, Julian Hyde <jh...@apache.org> wrote:
> I have logged https://issues.apache.org/jira/browse/CALCITE-1693 <
> https://issues.apache.org/jira/browse/CALCITE-1693>.
>
> Maryann, Can you please try the suggested fix?
>
> > On Mar 13, 2017, at 12:46 PM, Josh Elser <jo...@gmail.com> wrote:
> >
> > +1 to that suggestion. Have run into that one so many times in the past
> (can cause up to 30s delay in initial connections).
> >
> > We can easily throw that into the surefire/failsafe plugin configs for
> the future as well.
> >
> > Eli Levine wrote:
> >> Based on info in [1], using -Djava.net.preferIPv4Stack=true should
> speed
> >> things up if issues are indeed related to IPV6. Maryanne, Julian, can
> you
> >> try running Pig tests with the -Djava.net.preferIPv4Stack=true JVM
> param?
> >> If that helps, maybe we could just add that parameter to pom.xml.
> >>
> >> Alternatively, if Pig tests are slow for some people, I can certainly
> break
> >> them out into unit and integration tests and only have integration tests
> >> actually execute Pig. This seems like a less desirable option IMHO, as
> >> integration tests are not run on 'mvn install.'
> >>
> >> Thanks,
> >>
> >> Eli
> >>
> >> [1] https://developer.ibm.com/answers/questions/284205/how-
> >> can-we-improve-performance-in-javanetinet6addr.html
> >>
> >>
> >> On Mon, Mar 13, 2017 at 10:38 AM, Maryann Xue<ma...@gmail.com>
> wrote:
> >>
> >>> Here a Mac Pro with Comcast network connection (not much better than
> what
> >>> Julian has in a ski cabin maybe). Core build took 3:22 min, and Pig
> took
> >>> 07:38 min.
> >>>
> >>> On Mon, Mar 13, 2017 at 9:08 AM, Julian Hyde<jh...@apache.org> wrote:
> >>>
> >>>> The env was a ski cabin. And as you know, ski cabins often have blunt
> >>>> knives, poor quality coffee machines, and routers that don't support
> >>> ipV6.
> >>>> The last of these might be a factor.
> >>>>
> >>>> Julian
> >>>>
> >>>>
> >>>>
> >>>>> On Mar 11, 2017, at 7:14 PM, Eli Levine<el...@gmail.com> wrote:
> >>>>>
> >>>>> Not seeing this on my Linux or Mac machines. What env are you seeing
> >>>> this in?
> >>>>> Thanks,
> >>>>>
> >>>>> Eli
> >>>>>
> >>>>>> On Mar 11, 2017, at 8:14 AM, Julian Hyde<jh...@apache.org> wrote:
> >>>>>>
> >>>>>> The Pig tests seem to be running very slowly (several minutes), and
> >>> CPU
> >>>> is idle. Is anyone else seeing this?
> >>>>>> A thread dump reveals a wait in java/net/Inet6AddressImpl.
> >>> lookupAllHostAddr.
> >>>> If so, maybe we’re hitting [1].
> >>>>>> Julian
> >>>>>>
> >>>>>> [1] https://developer.ibm.com/answers/questions/284205/how-
> >>>> can-we-improve-performance-in-javanetinet6addr.html<
> >>>> https://developer.ibm.com/answers/questions/284205/how-
> >>>> can-we-improve-performance-in-javanetinet6addr.html>
> >>>>
> >>>>
> >>
>
>
Re: Pig tests very slow
Posted by Julian Hyde <jh...@apache.org>.
I have logged https://issues.apache.org/jira/browse/CALCITE-1693 <https://issues.apache.org/jira/browse/CALCITE-1693>.
Maryann, Can you please try the suggested fix?
> On Mar 13, 2017, at 12:46 PM, Josh Elser <jo...@gmail.com> wrote:
>
> +1 to that suggestion. Have run into that one so many times in the past (can cause up to 30s delay in initial connections).
>
> We can easily throw that into the surefire/failsafe plugin configs for the future as well.
>
> Eli Levine wrote:
>> Based on info in [1], using -Djava.net.preferIPv4Stack=true should speed
>> things up if issues are indeed related to IPV6. Maryanne, Julian, can you
>> try running Pig tests with the -Djava.net.preferIPv4Stack=true JVM param?
>> If that helps, maybe we could just add that parameter to pom.xml.
>>
>> Alternatively, if Pig tests are slow for some people, I can certainly break
>> them out into unit and integration tests and only have integration tests
>> actually execute Pig. This seems like a less desirable option IMHO, as
>> integration tests are not run on 'mvn install.'
>>
>> Thanks,
>>
>> Eli
>>
>> [1] https://developer.ibm.com/answers/questions/284205/how-
>> can-we-improve-performance-in-javanetinet6addr.html
>>
>>
>> On Mon, Mar 13, 2017 at 10:38 AM, Maryann Xue<ma...@gmail.com> wrote:
>>
>>> Here a Mac Pro with Comcast network connection (not much better than what
>>> Julian has in a ski cabin maybe). Core build took 3:22 min, and Pig took
>>> 07:38 min.
>>>
>>> On Mon, Mar 13, 2017 at 9:08 AM, Julian Hyde<jh...@apache.org> wrote:
>>>
>>>> The env was a ski cabin. And as you know, ski cabins often have blunt
>>>> knives, poor quality coffee machines, and routers that don't support
>>> ipV6.
>>>> The last of these might be a factor.
>>>>
>>>> Julian
>>>>
>>>>
>>>>
>>>>> On Mar 11, 2017, at 7:14 PM, Eli Levine<el...@gmail.com> wrote:
>>>>>
>>>>> Not seeing this on my Linux or Mac machines. What env are you seeing
>>>> this in?
>>>>> Thanks,
>>>>>
>>>>> Eli
>>>>>
>>>>>> On Mar 11, 2017, at 8:14 AM, Julian Hyde<jh...@apache.org> wrote:
>>>>>>
>>>>>> The Pig tests seem to be running very slowly (several minutes), and
>>> CPU
>>>> is idle. Is anyone else seeing this?
>>>>>> A thread dump reveals a wait in java/net/Inet6AddressImpl.
>>> lookupAllHostAddr.
>>>> If so, maybe we’re hitting [1].
>>>>>> Julian
>>>>>>
>>>>>> [1] https://developer.ibm.com/answers/questions/284205/how-
>>>> can-we-improve-performance-in-javanetinet6addr.html<
>>>> https://developer.ibm.com/answers/questions/284205/how-
>>>> can-we-improve-performance-in-javanetinet6addr.html>
>>>>
>>>>
>>
Re: Pig tests very slow
Posted by Josh Elser <jo...@gmail.com>.
+1 to that suggestion. Have run into that one so many times in the past
(can cause up to 30s delay in initial connections).
We can easily throw that into the surefire/failsafe plugin configs for
the future as well.
Eli Levine wrote:
> Based on info in [1], using -Djava.net.preferIPv4Stack=true should speed
> things up if issues are indeed related to IPV6. Maryanne, Julian, can you
> try running Pig tests with the -Djava.net.preferIPv4Stack=true JVM param?
> If that helps, maybe we could just add that parameter to pom.xml.
>
> Alternatively, if Pig tests are slow for some people, I can certainly break
> them out into unit and integration tests and only have integration tests
> actually execute Pig. This seems like a less desirable option IMHO, as
> integration tests are not run on 'mvn install.'
>
> Thanks,
>
> Eli
>
> [1] https://developer.ibm.com/answers/questions/284205/how-
> can-we-improve-performance-in-javanetinet6addr.html
>
>
> On Mon, Mar 13, 2017 at 10:38 AM, Maryann Xue<ma...@gmail.com> wrote:
>
>> Here a Mac Pro with Comcast network connection (not much better than what
>> Julian has in a ski cabin maybe). Core build took 3:22 min, and Pig took
>> 07:38 min.
>>
>> On Mon, Mar 13, 2017 at 9:08 AM, Julian Hyde<jh...@apache.org> wrote:
>>
>>> The env was a ski cabin. And as you know, ski cabins often have blunt
>>> knives, poor quality coffee machines, and routers that don't support
>> ipV6.
>>> The last of these might be a factor.
>>>
>>> Julian
>>>
>>>
>>>
>>>> On Mar 11, 2017, at 7:14 PM, Eli Levine<el...@gmail.com> wrote:
>>>>
>>>> Not seeing this on my Linux or Mac machines. What env are you seeing
>>> this in?
>>>> Thanks,
>>>>
>>>> Eli
>>>>
>>>>> On Mar 11, 2017, at 8:14 AM, Julian Hyde<jh...@apache.org> wrote:
>>>>>
>>>>> The Pig tests seem to be running very slowly (several minutes), and
>> CPU
>>> is idle. Is anyone else seeing this?
>>>>> A thread dump reveals a wait in java/net/Inet6AddressImpl.
>> lookupAllHostAddr.
>>> If so, maybe we\u2019re hitting [1].
>>>>> Julian
>>>>>
>>>>> [1] https://developer.ibm.com/answers/questions/284205/how-
>>> can-we-improve-performance-in-javanetinet6addr.html<
>>> https://developer.ibm.com/answers/questions/284205/how-
>>> can-we-improve-performance-in-javanetinet6addr.html>
>>>
>>>
>
Re: Pig tests very slow
Posted by Eli Levine <el...@gmail.com>.
Based on info in [1], using -Djava.net.preferIPv4Stack=true should speed
things up if issues are indeed related to IPV6. Maryanne, Julian, can you
try running Pig tests with the -Djava.net.preferIPv4Stack=true JVM param?
If that helps, maybe we could just add that parameter to pom.xml.
Alternatively, if Pig tests are slow for some people, I can certainly break
them out into unit and integration tests and only have integration tests
actually execute Pig. This seems like a less desirable option IMHO, as
integration tests are not run on 'mvn install.'
Thanks,
Eli
[1] https://developer.ibm.com/answers/questions/284205/how-
can-we-improve-performance-in-javanetinet6addr.html
On Mon, Mar 13, 2017 at 10:38 AM, Maryann Xue <ma...@gmail.com> wrote:
> Here a Mac Pro with Comcast network connection (not much better than what
> Julian has in a ski cabin maybe). Core build took 3:22 min, and Pig took
> 07:38 min.
>
> On Mon, Mar 13, 2017 at 9:08 AM, Julian Hyde <jh...@apache.org> wrote:
>
> > The env was a ski cabin. And as you know, ski cabins often have blunt
> > knives, poor quality coffee machines, and routers that don't support
> ipV6.
> > The last of these might be a factor.
> >
> > Julian
> >
> >
> >
> > > On Mar 11, 2017, at 7:14 PM, Eli Levine <el...@gmail.com> wrote:
> > >
> > > Not seeing this on my Linux or Mac machines. What env are you seeing
> > this in?
> > >
> > > Thanks,
> > >
> > > Eli
> > >
> > >> On Mar 11, 2017, at 8:14 AM, Julian Hyde <jh...@apache.org> wrote:
> > >>
> > >> The Pig tests seem to be running very slowly (several minutes), and
> CPU
> > is idle. Is anyone else seeing this?
> > >>
> > >> A thread dump reveals a wait in java/net/Inet6AddressImpl.
> lookupAllHostAddr.
> > If so, maybe we’re hitting [1].
> > >>
> > >> Julian
> > >>
> > >> [1] https://developer.ibm.com/answers/questions/284205/how-
> > can-we-improve-performance-in-javanetinet6addr.html <
> > https://developer.ibm.com/answers/questions/284205/how-
> > can-we-improve-performance-in-javanetinet6addr.html>
> >
> >
>
Re: Pig tests very slow
Posted by Maryann Xue <ma...@gmail.com>.
Here a Mac Pro with Comcast network connection (not much better than what
Julian has in a ski cabin maybe). Core build took 3:22 min, and Pig took
07:38 min.
On Mon, Mar 13, 2017 at 9:08 AM, Julian Hyde <jh...@apache.org> wrote:
> The env was a ski cabin. And as you know, ski cabins often have blunt
> knives, poor quality coffee machines, and routers that don't support ipV6.
> The last of these might be a factor.
>
> Julian
>
>
>
> > On Mar 11, 2017, at 7:14 PM, Eli Levine <el...@gmail.com> wrote:
> >
> > Not seeing this on my Linux or Mac machines. What env are you seeing
> this in?
> >
> > Thanks,
> >
> > Eli
> >
> >> On Mar 11, 2017, at 8:14 AM, Julian Hyde <jh...@apache.org> wrote:
> >>
> >> The Pig tests seem to be running very slowly (several minutes), and CPU
> is idle. Is anyone else seeing this?
> >>
> >> A thread dump reveals a wait in java/net/Inet6AddressImpl.lookupAllHostAddr.
> If so, maybe we’re hitting [1].
> >>
> >> Julian
> >>
> >> [1] https://developer.ibm.com/answers/questions/284205/how-
> can-we-improve-performance-in-javanetinet6addr.html <
> https://developer.ibm.com/answers/questions/284205/how-
> can-we-improve-performance-in-javanetinet6addr.html>
>
>
Re: Pig tests very slow
Posted by Julian Hyde <jh...@apache.org>.
The env was a ski cabin. And as you know, ski cabins often have blunt knives, poor quality coffee machines, and routers that don't support ipV6. The last of these might be a factor.
Julian
> On Mar 11, 2017, at 7:14 PM, Eli Levine <el...@gmail.com> wrote:
>
> Not seeing this on my Linux or Mac machines. What env are you seeing this in?
>
> Thanks,
>
> Eli
>
>> On Mar 11, 2017, at 8:14 AM, Julian Hyde <jh...@apache.org> wrote:
>>
>> The Pig tests seem to be running very slowly (several minutes), and CPU is idle. Is anyone else seeing this?
>>
>> A thread dump reveals a wait in java/net/Inet6AddressImpl.lookupAllHostAddr. If so, maybe we’re hitting [1].
>>
>> Julian
>>
>> [1] https://developer.ibm.com/answers/questions/284205/how-can-we-improve-performance-in-javanetinet6addr.html <https://developer.ibm.com/answers/questions/284205/how-can-we-improve-performance-in-javanetinet6addr.html>
Re: Pig tests very slow
Posted by Eli Levine <el...@gmail.com>.
Not seeing this on my Linux or Mac machines. What env are you seeing this in?
Thanks,
Eli
> On Mar 11, 2017, at 8:14 AM, Julian Hyde <jh...@apache.org> wrote:
>
> The Pig tests seem to be running very slowly (several minutes), and CPU is idle. Is anyone else seeing this?
>
> A thread dump reveals a wait in java/net/Inet6AddressImpl.lookupAllHostAddr. If so, maybe we’re hitting [1].
>
> Julian
>
> [1] https://developer.ibm.com/answers/questions/284205/how-can-we-improve-performance-in-javanetinet6addr.html <https://developer.ibm.com/answers/questions/284205/how-can-we-improve-performance-in-javanetinet6addr.html>