cTAKES CPE MySQL Exception

[Clayton Turner <ca...@g.cofc.edu>]

Hi, everyone.

First off, I'd like to say "awesome and thank you" for the cTAKES 3.2
release and information. I've been following those pages and it's been
really helpful for helping me move along in my own progress. Really cool
stuff.

So I'm using the Collection Processing Engine (with ytex and umls) and I'm
trying to process ~1 million notes (as opposed to the about 30 in the given
demo).

I've tried this the past 2 days and when I come back in to check the
progress I see that I've received an error about 14000 notes into the
process:

org.apache.uima.analysis_engine.AnalysisEngineProcessException: Annotator
processing failed.
CausedBy: org.springframework.transaction.CannotCreateTransactionException:
Could not open Hibernate Session for transaction; nested exception is
com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: The last packet
successfully received from the server was 53,888,249 milliseconds ago. The
last packet sent successfully to the server was 53,888,249 milliseconds
ago. is longer than the server configured value of 'wait_timeout'. You
should consider either expiring and/or testing connection validity before
use in your application, increasing the server configured values for client
timeouts, or using the Connector/J connection property 'autoReconnect=true'
to avoid this problem.

So, in my own debugging, I have ensured that autoReconnect true was on (it
always has been).

I looked at my CPE output in the command prompt and noticed a
"PacketTooBigException" so I increased the packet max size to 1G (the max
for sql server).

I increased the time allowed for timeouts.

I'm really unsure of what to do here. Should I find a way to see if there
is a problematic note that is giving me issues (though I can't understand
how 1 note would make a packet too large)? Should I try to do some
horizontal sharding and break the problem into smaller chunks (though I
would think this program could handle large datasets since it's using a
query language)? I'm just at a loss with this error, especially since it
takes so long to actually spit the error out at me.

Thanks in advance everyone,
Clayton

-- 
--
Clayton Turner
email: caturner3@g.cofc.edu
phone: (843)-424-3784
web: claytonturner.blogspot.com
-------------------------------------------------------------------------------------------------
“When scientifically investigating the natural world, the only thing worse
than a blind believer is a seeing denier.”
- Neil deGrasse Tyson

Re: cTAKES CPE MySQL Exception

[vijay garla <vn...@gmail.com>]

My guess is that this exception is coming out of the DictionaryLookup (it
creates a connection and holds on to it for the life of the AE).

If it is coming out of the DBCollectionReader/DBConsumer you're in luck, as
those use a connection pool, and you can configure it to check the
connection upon pulling from the pool

The file is: resources\org\apache\ctakes\ytex\beans-datasource.xml
see
http://commons.apache.org/proper/commons-dbcp/api-1.4/org/apache/commons/dbcp/BasicDataSource.html
- you want to set testOnBorrow to true, and set the validtionQuery to
something like "select 1"

You should also set the errorRateThreshold in the CPE config (you can't do
this via the gui - you have to do this in the xml) - that way the cpe
doesn't bomb on the first error it sees - a few bad apples shouldn't kill
the processing.

HTH,

VJ



On Thu, Jul 24, 2014 at 4:32 PM, Clayton Turner <ca...@g.cofc.edu>
wrote:

> Hi, everyone.
>
> First off, I'd like to say "awesome and thank you" for the cTAKES 3.2
> release and information. I've been following those pages and it's been
> really helpful for helping me move along in my own progress. Really cool
> stuff.
>
> So I'm using the Collection Processing Engine (with ytex and umls) and I'm
> trying to process ~1 million notes (as opposed to the about 30 in the given
> demo).
>
> I've tried this the past 2 days and when I come back in to check the
> progress I see that I've received an error about 14000 notes into the
> process:
>
> org.apache.uima.analysis_engine.AnalysisEngineProcessException: Annotator
> processing failed.
> CausedBy: org.springframework.transaction.CannotCreateTransactionException:
> Could not open Hibernate Session for transaction; nested exception is
> com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: The last packet
> successfully received from the server was 53,888,249 milliseconds ago. The
> last packet sent successfully to the server was 53,888,249 milliseconds
> ago. is longer than the server configured value of 'wait_timeout'. You
> should consider either expiring and/or testing connection validity before
> use in your application, increasing the server configured values for client
> timeouts, or using the Connector/J connection property 'autoReconnect=true'
> to avoid this problem.
>
> So, in my own debugging, I have ensured that autoReconnect true was on (it
> always has been).
>
> I looked at my CPE output in the command prompt and noticed a
> "PacketTooBigException" so I increased the packet max size to 1G (the max
> for sql server).
>
> I increased the time allowed for timeouts.
>
> I'm really unsure of what to do here. Should I find a way to see if there
> is a problematic note that is giving me issues (though I can't understand
> how 1 note would make a packet too large)? Should I try to do some
> horizontal sharding and break the problem into smaller chunks (though I
> would think this program could handle large datasets since it's using a
> query language)? I'm just at a loss with this error, especially since it
> takes so long to actually spit the error out at me.
>
> Thanks in advance everyone,
> Clayton
>
> --
> --
> Clayton Turner
> email: caturner3@g.cofc.edu
> phone: (843)-424-3784
> web: claytonturner.blogspot.com
>
> -------------------------------------------------------------------------------------------------
> “When scientifically investigating the natural world, the only thing worse
> than a blind believer is a seeing denier.”
> - Neil deGrasse Tyson
>