You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@pig.apache.org by Mike Sukmanowsky <mi...@parsely.com> on 2013/03/14 22:05:05 UTC
EvalFunc finish() closing connections prematurely
Hi there,
I have an EvalFunc which uses an internal class that opens up connections
to a Redis and MongoDB server. This class has a close() method which
closes connections to both Redis and MongoDB.
In my EvalFunc implementation, I placed the close() method
@Override
public void finish() {
meta.close();
}
But this is leading to a
java.lang.IllegalStateException: this Mongo has been closed
indicating that Pig attempted to read from Mongo after a call to finish().
Is this an error specific to running a Pig script locally?
Mike
--
Mike Sukmanowsky
Product Lead, http://parse.ly
989 Avenue of the Americas, 3rd Floor
New York, NY 10018
p: +1 (416) 953-4248
e: mike@parsely.com
Re: EvalFunc finish() closing connections prematurely
Posted by Jonathan Coveney <jc...@gmail.com>.
Sorry about not getting to this, Mike!
Dmitry's idea is a good one. I still think that something is axing the
connections, possible between different map slots on the same mapper. I
would log very liberally and see what comes of it. And I'll try and carve
out some time.
2013/3/26 Dmitriy Ryaboy <dv...@gmail.com>
> Mike, have you tried adding logging to any EvalFunc methods that
> communicate with Mongo to see which of them is calling it after finish() ?
> Are you sure something else doesn't close Mongo connection for you?
>
>
> On Fri, Mar 22, 2013 at 8:28 AM, Mike Sukmanowsky <mi...@parsely.com>
> wrote:
>
> > Bump - any thoughts?
> >
> >
> > On Mon, Mar 18, 2013 at 4:53 PM, Mike Sukmanowsky <mi...@parsely.com>
> > wrote:
> >
> > > Sure thing - here you go
> > > https://gist.github.com/msukmanowsky/198c878e1deaf4e470a9
> > >
> > > When initialized, the ParselyMetadataService creates a new Mongo and
> > Jedis
> > > instance which the EvalFunc queries using a public method fetch().
> > > Instance of ParselyMetadataService also have a close() function which
> > > simply calls:
> > >
> > > this.mongoClient.close();
> > > this.redisClient.disconnect();
> > >
> > >
> > > On Thu, Mar 14, 2013 at 5:24 PM, Jonathan Coveney <jcoveney@gmail.com
> > >wrote:
> > >
> > >> Can you perhaps share more of your implementation? I can imagine a
> > couple
> > >> of things which would cause errors like this. Are you making sure that
> > >> each
> > >> instance of EvalFunc is dealing with a different connection?
> > >>
> > >> That's what I'd take a look at first...if that isn't the issue, I can
> > look
> > >> into how final() is implemented.
> > >>
> > >>
> > >> 2013/3/14 Mike Sukmanowsky <mi...@parsely.com>
> > >>
> > >> > Hi there,
> > >> >
> > >> > I have an EvalFunc which uses an internal class that opens up
> > >> connections
> > >> > to a Redis and MongoDB server. This class has a close() method
> which
> > >> > closes connections to both Redis and MongoDB.
> > >> >
> > >> > In my EvalFunc implementation, I placed the close() method
> > >> >
> > >> > @Override
> > >> > public void finish() {
> > >> > meta.close();
> > >> > }
> > >> >
> > >> > But this is leading to a
> > >> >
> > >> > java.lang.IllegalStateException: this Mongo has been closed
> > >> >
> > >> > indicating that Pig attempted to read from Mongo after a call to
> > >> finish().
> > >> > Is this an error specific to running a Pig script locally?
> > >> >
> > >> > Mike
> > >> >
> > >> > --
> > >> > Mike Sukmanowsky
> > >> >
> > >> > Product Lead, http://parse.ly
> > >> > 989 Avenue of the Americas, 3rd Floor
> > >> > New York, NY 10018
> > >> > p: +1 (416) 953-4248
> > >> > e: mike@parsely.com
> > >> >
> > >>
> > >
> > >
> > >
> > > --
> > > Mike Sukmanowsky
> > >
> > > Product Lead, http://parse.ly
> > > 989 Avenue of the Americas, 3rd Floor
> > > New York, NY 10018
> > > p: +1 (416) 953-4248
> > > e: mike@parsely.com
> > >
> >
> >
> >
> > --
> > Mike Sukmanowsky
> >
> > Product Lead, http://parse.ly
> > 989 Avenue of the Americas, 3rd Floor
> > New York, NY 10018
> > p: +1 (416) 953-4248
> > e: mike@parsely.com
> >
>
Re: EvalFunc finish() closing connections prematurely
Posted by Dmitriy Ryaboy <dv...@gmail.com>.
Mike, have you tried adding logging to any EvalFunc methods that
communicate with Mongo to see which of them is calling it after finish() ?
Are you sure something else doesn't close Mongo connection for you?
On Fri, Mar 22, 2013 at 8:28 AM, Mike Sukmanowsky <mi...@parsely.com> wrote:
> Bump - any thoughts?
>
>
> On Mon, Mar 18, 2013 at 4:53 PM, Mike Sukmanowsky <mi...@parsely.com>
> wrote:
>
> > Sure thing - here you go
> > https://gist.github.com/msukmanowsky/198c878e1deaf4e470a9
> >
> > When initialized, the ParselyMetadataService creates a new Mongo and
> Jedis
> > instance which the EvalFunc queries using a public method fetch().
> > Instance of ParselyMetadataService also have a close() function which
> > simply calls:
> >
> > this.mongoClient.close();
> > this.redisClient.disconnect();
> >
> >
> > On Thu, Mar 14, 2013 at 5:24 PM, Jonathan Coveney <jcoveney@gmail.com
> >wrote:
> >
> >> Can you perhaps share more of your implementation? I can imagine a
> couple
> >> of things which would cause errors like this. Are you making sure that
> >> each
> >> instance of EvalFunc is dealing with a different connection?
> >>
> >> That's what I'd take a look at first...if that isn't the issue, I can
> look
> >> into how final() is implemented.
> >>
> >>
> >> 2013/3/14 Mike Sukmanowsky <mi...@parsely.com>
> >>
> >> > Hi there,
> >> >
> >> > I have an EvalFunc which uses an internal class that opens up
> >> connections
> >> > to a Redis and MongoDB server. This class has a close() method which
> >> > closes connections to both Redis and MongoDB.
> >> >
> >> > In my EvalFunc implementation, I placed the close() method
> >> >
> >> > @Override
> >> > public void finish() {
> >> > meta.close();
> >> > }
> >> >
> >> > But this is leading to a
> >> >
> >> > java.lang.IllegalStateException: this Mongo has been closed
> >> >
> >> > indicating that Pig attempted to read from Mongo after a call to
> >> finish().
> >> > Is this an error specific to running a Pig script locally?
> >> >
> >> > Mike
> >> >
> >> > --
> >> > Mike Sukmanowsky
> >> >
> >> > Product Lead, http://parse.ly
> >> > 989 Avenue of the Americas, 3rd Floor
> >> > New York, NY 10018
> >> > p: +1 (416) 953-4248
> >> > e: mike@parsely.com
> >> >
> >>
> >
> >
> >
> > --
> > Mike Sukmanowsky
> >
> > Product Lead, http://parse.ly
> > 989 Avenue of the Americas, 3rd Floor
> > New York, NY 10018
> > p: +1 (416) 953-4248
> > e: mike@parsely.com
> >
>
>
>
> --
> Mike Sukmanowsky
>
> Product Lead, http://parse.ly
> 989 Avenue of the Americas, 3rd Floor
> New York, NY 10018
> p: +1 (416) 953-4248
> e: mike@parsely.com
>
Re: EvalFunc finish() closing connections prematurely
Posted by Mike Sukmanowsky <mi...@parsely.com>.
Bump - any thoughts?
On Mon, Mar 18, 2013 at 4:53 PM, Mike Sukmanowsky <mi...@parsely.com> wrote:
> Sure thing - here you go
> https://gist.github.com/msukmanowsky/198c878e1deaf4e470a9
>
> When initialized, the ParselyMetadataService creates a new Mongo and Jedis
> instance which the EvalFunc queries using a public method fetch().
> Instance of ParselyMetadataService also have a close() function which
> simply calls:
>
> this.mongoClient.close();
> this.redisClient.disconnect();
>
>
> On Thu, Mar 14, 2013 at 5:24 PM, Jonathan Coveney <jc...@gmail.com>wrote:
>
>> Can you perhaps share more of your implementation? I can imagine a couple
>> of things which would cause errors like this. Are you making sure that
>> each
>> instance of EvalFunc is dealing with a different connection?
>>
>> That's what I'd take a look at first...if that isn't the issue, I can look
>> into how final() is implemented.
>>
>>
>> 2013/3/14 Mike Sukmanowsky <mi...@parsely.com>
>>
>> > Hi there,
>> >
>> > I have an EvalFunc which uses an internal class that opens up
>> connections
>> > to a Redis and MongoDB server. This class has a close() method which
>> > closes connections to both Redis and MongoDB.
>> >
>> > In my EvalFunc implementation, I placed the close() method
>> >
>> > @Override
>> > public void finish() {
>> > meta.close();
>> > }
>> >
>> > But this is leading to a
>> >
>> > java.lang.IllegalStateException: this Mongo has been closed
>> >
>> > indicating that Pig attempted to read from Mongo after a call to
>> finish().
>> > Is this an error specific to running a Pig script locally?
>> >
>> > Mike
>> >
>> > --
>> > Mike Sukmanowsky
>> >
>> > Product Lead, http://parse.ly
>> > 989 Avenue of the Americas, 3rd Floor
>> > New York, NY 10018
>> > p: +1 (416) 953-4248
>> > e: mike@parsely.com
>> >
>>
>
>
>
> --
> Mike Sukmanowsky
>
> Product Lead, http://parse.ly
> 989 Avenue of the Americas, 3rd Floor
> New York, NY 10018
> p: +1 (416) 953-4248
> e: mike@parsely.com
>
--
Mike Sukmanowsky
Product Lead, http://parse.ly
989 Avenue of the Americas, 3rd Floor
New York, NY 10018
p: +1 (416) 953-4248
e: mike@parsely.com
Re: EvalFunc finish() closing connections prematurely
Posted by Mike Sukmanowsky <mi...@parsely.com>.
Sure thing - here you go
https://gist.github.com/msukmanowsky/198c878e1deaf4e470a9
When initialized, the ParselyMetadataService creates a new Mongo and Jedis
instance which the EvalFunc queries using a public method fetch().
Instance of ParselyMetadataService also have a close() function which
simply calls:
this.mongoClient.close();
this.redisClient.disconnect();
On Thu, Mar 14, 2013 at 5:24 PM, Jonathan Coveney <jc...@gmail.com>wrote:
> Can you perhaps share more of your implementation? I can imagine a couple
> of things which would cause errors like this. Are you making sure that each
> instance of EvalFunc is dealing with a different connection?
>
> That's what I'd take a look at first...if that isn't the issue, I can look
> into how final() is implemented.
>
>
> 2013/3/14 Mike Sukmanowsky <mi...@parsely.com>
>
> > Hi there,
> >
> > I have an EvalFunc which uses an internal class that opens up connections
> > to a Redis and MongoDB server. This class has a close() method which
> > closes connections to both Redis and MongoDB.
> >
> > In my EvalFunc implementation, I placed the close() method
> >
> > @Override
> > public void finish() {
> > meta.close();
> > }
> >
> > But this is leading to a
> >
> > java.lang.IllegalStateException: this Mongo has been closed
> >
> > indicating that Pig attempted to read from Mongo after a call to
> finish().
> > Is this an error specific to running a Pig script locally?
> >
> > Mike
> >
> > --
> > Mike Sukmanowsky
> >
> > Product Lead, http://parse.ly
> > 989 Avenue of the Americas, 3rd Floor
> > New York, NY 10018
> > p: +1 (416) 953-4248
> > e: mike@parsely.com
> >
>
--
Mike Sukmanowsky
Product Lead, http://parse.ly
989 Avenue of the Americas, 3rd Floor
New York, NY 10018
p: +1 (416) 953-4248
e: mike@parsely.com
Re: EvalFunc finish() closing connections prematurely
Posted by Jonathan Coveney <jc...@gmail.com>.
Can you perhaps share more of your implementation? I can imagine a couple
of things which would cause errors like this. Are you making sure that each
instance of EvalFunc is dealing with a different connection?
That's what I'd take a look at first...if that isn't the issue, I can look
into how final() is implemented.
2013/3/14 Mike Sukmanowsky <mi...@parsely.com>
> Hi there,
>
> I have an EvalFunc which uses an internal class that opens up connections
> to a Redis and MongoDB server. This class has a close() method which
> closes connections to both Redis and MongoDB.
>
> In my EvalFunc implementation, I placed the close() method
>
> @Override
> public void finish() {
> meta.close();
> }
>
> But this is leading to a
>
> java.lang.IllegalStateException: this Mongo has been closed
>
> indicating that Pig attempted to read from Mongo after a call to finish().
> Is this an error specific to running a Pig script locally?
>
> Mike
>
> --
> Mike Sukmanowsky
>
> Product Lead, http://parse.ly
> 989 Avenue of the Americas, 3rd Floor
> New York, NY 10018
> p: +1 (416) 953-4248
> e: mike@parsely.com
>