You are viewing a plain text version of this content. The canonical link for it is here.
Posted to server-dev@james.apache.org by Stuart Roebuck <st...@adolos.co.uk> on 2002/05/08 17:28:36 UTC
Packet is larger than max_allowed_packet from server configuration of 1048576 bytes
I've got problems with James which mean that, from time to time, the
error directory suddenly grows with multiple copies of a large incoming
email and before I know it our server has run out of disk space.
Here is the corresponding stack error. Does it mean anything to anyone?
> ava.lang.RuntimeException: Exception caught while storing mail
> Container: java.lang.IllegalArgumentException: Packet is larger than
> max_allowed_packet from server configuration of 1048576 bytes
> at
> org.apache.james.mailrepository.JDBCMailRepository.store
> (JDBCMailRepository.java:422)
> at org.apache.james.James.storeMail(James.java:452)
> at java.lang.reflect.Method.invoke(Native Method)
> at
> org.apache.avalon.phoenix.components.application.BlockInvocationHandler.invoke(
> BlockInvocationHandler.java:91)
> at $Proxy9.storeMail(Unknown Source)
> at
> org.apache.james.transport.mailets.LocalDelivery.service(LocalDelivery.java:
> 34)
> at
> org.apache.james.transport.LinearProcessor.service(LinearProcessor.java:183)
> at
> org.apache.james.transport.JamesSpoolManager.process(JamesSpoolManager.java:
> 241)
> at
> org.apache.james.transport.JamesSpoolManager.run(JamesSpoolManager.java:208)
> at
> org.apache.avalon.excalibur.thread.impl.ExecutableRunnable.execute
> (ExecutableRunnable.java:47)
> at
> org.apache.avalon.excalibur.thread.impl.WorkerThread.run(WorkerThread.java:
> 86)
> java.lang.IllegalArgumentException: Packet is larger than
> max_allowed_packet from server configuration of 1048576 bytes
> at org.gjt.mm.mysql.Buffer.ensureCapacity(Buffer.java:398)
> at org.gjt.mm.mysql.Buffer.writeBytesNoNull(Buffer.java:386)
> at
> org.gjt.mm.mysql.PreparedStatement.execute(PreparedStatement.java:1036)
> at
> org.apache.james.mailrepository.JDBCMailRepository.store
> (JDBCMailRepository.java:410)
> at org.apache.james.James.storeMail(James.java:452)
> at java.lang.reflect.Method.invoke(Native Method)
> at
> org.apache.avalon.phoenix.components.application.BlockInvocationHandler.invoke(
> BlockInvocationHandler.java:91)
> at $Proxy9.storeMail(Unknown Source)
> at
> org.apache.james.transport.mailets.LocalDelivery.service(LocalDelivery.java:
> 34)
> at
> org.apache.james.transport.LinearProcessor.service(LinearProcessor.java:183)
> at
> org.apache.james.transport.JamesSpoolManager.process(JamesSpoolManager.java:
> 241)
> at
> org.apache.james.transport.JamesSpoolManager.run(JamesSpoolManager.java:208)
> at
> org.apache.avalon.excalibur.thread.impl.ExecutableRunnable.execute
> (ExecutableRunnable.java:47)
> at
> org.apache.avalon.excalibur.thread.impl.WorkerThread.run(WorkerThread.java:
> 86)
Public Key - 1024D/88DD65AF 2001-11-23 Stuart Roebuck (Adolos)
Key fingerprint = 89D9 E405 F8B1 9B22 0FA2 F2C1 9E57 5AB1 88DD 65AF
-------------------------------------------------------------------------
Stuart Roebuck stuart.roebuck@adolos.com
Systems Architect Java, XML, MacOS X, XP,
etc.
ADOLOS <http://www.adolos.com/>
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
RE: PLEASE UPDATE /james/lib/mysql jar file!
Posted by Danny Angus <da...@thought.co.uk>.
> and I suspect that Mark would be willing
> to provide
> it under the Apache license, too.
I gather not.
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
RE: PLEASE UPDATE /james/lib/mysql jar file!
Posted by "Noel J. Bergman" <no...@devtech.com>.
What? Why is that? The driver is LGPL, so it doesn't have the viral
consequences of the GPL, and I suspect that Mark would be willing to provide
it under the Apache license, too.
--- Noel
-----Original Message-----
From: Danny Angus [mailto:danny@thought.co.uk]
Sent: Thursday, May 09, 2002 9:12
To: James Developers List
Subject: RE: PLEASE UPDATE /james/lib/mysql jar file!
Actually we may have to pull the driver from the distro altogether for legal
reasons.
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
RE: PLEASE UPDATE /james/lib/mysql jar file!
Posted by Danny Angus <da...@thought.co.uk>.
Actually we may have to pull the driver from the distro altogether for legal
reasons.
-----Original Message-----
From: Noel J. Bergman [mailto:noel@devtech.com]
Sent: 08 May 2002 17:46
To: James Developers List
Subject: PLEASE UPDATE /james/lib/mysql jar file!
> More overly, I'm pretty sure the problem is with the JDBC driver... I'd
> bet it's mysql, and while I'm not really expert about that db, there
> seems to be a meg limit on the binary field by default.
PLEASE FIX THIS: James is including an ANCIENT version of the mm.mysql
driver. The current release is v2.0.13; James is shipping version 2.0.4
(25JAN01). Amongst the missing fixes:
06-13-01 - Fixed ResultSet.getBlob() ArrayIndex out-of-bounds
- Fixed ResultSetMetaData.getColumnTypeName for TEXT/BLOB
- Fixed ArrayIndexOutOfBounds when sending large BLOB queries
(Max size packet was not being set)
The hard limit on the store is effectively set by the client server
protocol. According to the MySQL 3.X manual, "the server/client protocol
and MyISAM tables has currently a limit of
16M per communication packet / table row," so that would be our limit.
The driver is available from http://mmmysql.sourceforge.net/.
--- Noel
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
PLEASE UPDATE /james/lib/mysql jar file!
Posted by "Noel J. Bergman" <no...@devtech.com>.
> More overly, I'm pretty sure the problem is with the JDBC driver... I'd
> bet it's mysql, and while I'm not really expert about that db, there
> seems to be a meg limit on the binary field by default.
PLEASE FIX THIS: James is including an ANCIENT version of the mm.mysql
driver. The current release is v2.0.13; James is shipping version 2.0.4
(25JAN01). Amongst the missing fixes:
06-13-01 - Fixed ResultSet.getBlob() ArrayIndex out-of-bounds
- Fixed ResultSetMetaData.getColumnTypeName for TEXT/BLOB
- Fixed ArrayIndexOutOfBounds when sending large BLOB queries
(Max size packet was not being set)
The hard limit on the store is effectively set by the client server
protocol. According to the MySQL 3.X manual, "the server/client protocol
and MyISAM tables has currently a limit of
16M per communication packet / table row," so that would be our limit.
The driver is available from http://mmmysql.sourceforge.net/.
--- Noel
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
Re: Packet is larger than max_allowed_packet from server configurationof 1048576 bytes
Posted by Stuart Roebuck <st...@adolos.co.uk>.
Thank you again for all the useful suggestions.
I haven't updated James to the latest CVS, but I've updated my mysql
driver and relaunched mysql with the command line:
safe_mysqld -O max_allowed_packet=8M --user=mysql
I've since received an email that previously killed the system.
Thanks again,
Stuart.
On Thursday, May 9, 2002, at 03:11 PM, Noel J. Bergman wrote:
>> its not the driver, its actually the default MySQL configuration (in
>> /etc/my.cnf)
>
> It is both. There are limiting bugs in the version of the mmmysql
> driver
> bundled with James (e.g., "Fixed ArrayIndexOutOfBounds when sending
> large
> BLOB queries -- Max size packet was not being set"), and depending upon
> your
> mysql build, it is likely that you also need to add:
>
> set-variable = max_allowed_packet=16M
>
> to the [mysqld] section of the global configuration file, e.g.,
> /etc/my.cnf,
> or change the startup statement for mysqld to set max_allowed_packet.
>
> For those who care, the documentation for max_allowed_packet says: "This
> value by default is small, to catch big (possibly wrong) packets. You
> must
> increase this value if you are using big BLOB columns. It should be as
> big
> as the biggest BLOB you want to use. The protocol limits for
> max_allowed_packet is 16M in MySQL 3.23 and 4G in MySQL 4.0."
>
> If you expect a LOT of large packets, you might want to be aware of the
> following: "For example, if you have BLOB values that contain up to 16M
> of
> data, you must have a communication buffer limit of at least 16M (in
> both
> server and client). The client's default maximum is 16M, but the
> default
> maximum in the server is 1M. You can increase this by changing the
> value of
> the max_allowed_packet parameter when the server is started. The MySQL
> server shrinks each communication buffer to net_buffer_length bytes
> after
> each query. For clients, the size of the buffer associated with a
> connection
> is not decreased until the connection is closed, at which time client
> memory
> is reclaimed."
>
> Caveat: "You can also get strange problems with large packets if you are
> using big blobs, but you haven't given mysqld access to enough memory to
> handle the query. If you suspect this is the case, try adding ulimit -d
> 256000 to the beginning of the safe_mysqld script and restart mysqld."
>
> --- Noel
>
>
> --
> To unsubscribe, e-mail: <mailto:james-dev-
> unsubscribe@jakarta.apache.org>
> For additional commands, e-mail: <mailto:james-dev-
> help@jakarta.apache.org>
>
>
Public Key - 1024D/88DD65AF 2001-11-23 Stuart Roebuck (Adolos)
Key fingerprint = 89D9 E405 F8B1 9B22 0FA2 F2C1 9E57 5AB1 88DD 65AF
-------------------------------------------------------------------------
Stuart Roebuck stuart.roebuck@adolos.com
Systems Architect Java, XML, MacOS X, XP,
etc.
ADOLOS <http://www.adolos.com/>
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
RE: Packet is larger than max_allowed_packet from server configurationof 1048576 bytes
Posted by "Noel J. Bergman" <no...@devtech.com>.
> its not the driver, its actually the default MySQL configuration (in
> /etc/my.cnf)
It is both. There are limiting bugs in the version of the mmmysql driver
bundled with James (e.g., "Fixed ArrayIndexOutOfBounds when sending large
BLOB queries -- Max size packet was not being set"), and depending upon your
mysql build, it is likely that you also need to add:
set-variable = max_allowed_packet=16M
to the [mysqld] section of the global configuration file, e.g., /etc/my.cnf,
or change the startup statement for mysqld to set max_allowed_packet.
For those who care, the documentation for max_allowed_packet says: "This
value by default is small, to catch big (possibly wrong) packets. You must
increase this value if you are using big BLOB columns. It should be as big
as the biggest BLOB you want to use. The protocol limits for
max_allowed_packet is 16M in MySQL 3.23 and 4G in MySQL 4.0."
If you expect a LOT of large packets, you might want to be aware of the
following: "For example, if you have BLOB values that contain up to 16M of
data, you must have a communication buffer limit of at least 16M (in both
server and client). The client's default maximum is 16M, but the default
maximum in the server is 1M. You can increase this by changing the value of
the max_allowed_packet parameter when the server is started. The MySQL
server shrinks each communication buffer to net_buffer_length bytes after
each query. For clients, the size of the buffer associated with a connection
is not decreased until the connection is closed, at which time client memory
is reclaimed."
Caveat: "You can also get strange problems with large packets if you are
using big blobs, but you haven't given mysqld access to enough memory to
handle the query. If you suspect this is the case, try adding ulimit -d
256000 to the beginning of the safe_mysqld script and restart mysqld."
--- Noel
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
RE: Packet is larger than max_allowed_packet from server configurationof 1048576 bytes
Posted by Danny Angus <da...@thought.co.uk>.
>More overly, I'm pretty sure the problem is with the JDBC driver... I'd
>bet it's mysql, and while I'm not really expert about that db, there
>seems to be a meg limit on the binary field by default.
its not the driver, its actually the default MySQL configuration (in
/etc/my.cnf)
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
RE: Packet is larger than max_allowed_packet from server configuration of 1048576 bytes
Posted by "Noel J. Bergman" <no...@devtech.com>.
Stuart,
(1) The patch is to replace:
if (errors.isEmpty()) {
mail.setState(Mail.GHOST);
} else {
getMailetContext().sendMail(mail.getSender(), errors,
mail.getMessage(), Mail.ERROR);
//mail.setRecipients(errors);
//mail.setState(Mail.ERROR);
//mail.setErrorMessage("Unable to delivery locally message");
}
with:
if (!errors.isEmpty()) {
getMailetContext().sendMail(mail.getSender(),
errors, mail.getMessage(), Mail.ERROR);
}
mail.setState(Mail.GHOST);
in LocalDelivery.java.
(2) The problem is being caused, I believe, by an old version of the mysql
driver (see my separate message on that subject). After verifying that the
patch is effective, you might want to eliminate the problem by upgrading the
mysql driver in james/lib to the current release
(http://mmmysql.sourceforge.net)
I don't know if Serge has affected either change into the CVS, yet, but I'm
sure he'll post a reply shortly.
--- Noel
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
Re: Packet is larger than max_allowed_packet from server configuration of 1048576 bytes
Posted by Stuart Roebuck <st...@adolos.co.uk>.
Thank you for these really helpful replies.
I can confirm that I'm using MySQL and that only the inbox is stored
there.
In addition I have a nice example that doubly confirms your thinking.
One of the mail addresses handled by James is both forwarded to an
external email address and also stored in the inbox. When one of the
'deadly' emails arrives it gets forwarded correctly but never reaches
the inbox.
I'll try the latest CVS - could you clarify which patch needs to go on
top of that, or whether the patch is already in the lastest CVS?
I'll also look into MySQL configuration and see if I can spot something
there about maximum binary field limits.
Thanks,
Stuart.
On Wednesday, May 8, 2002, at 04:55 PM, Serge Knystautas wrote:
> Noel J. Bergman wrote:
>
>> I wonder if that is one of those cases that "shouldn't happen", as was
>> just
>> discussed in the thread on configuring the SMTP server?
>> Serge, wouldn't this hit the bad code that we just discussed? Is this
>> exception going to be tossed back and handled by
>> LocalDelivery.service(),
>> resulting in exactly the consequence we were just talking about?
>> Should
>> Stuart pull the current 2.0a3 release apply the patch to
>> LocalDelivery.service() that you just posted to the mailing list, and
>> rebuild James?
>> --- Noel
>
> Yes, this is Murphy's law at work... within an hour of pronouncing that
> an error was never reported, it's reported. Stuart can apply that
> patch and use the latest version to halve the number of messages going
> into error.
>
> More overly, I'm pretty sure the problem is with the JDBC driver... I'd
> bet it's mysql, and while I'm not really expert about that db, there
> seems to be a meg limit on the binary field by default.
>
> What's interesting though is (I'd guess that) Stuart is using the db
> for the inboxes but file system for spooling. This would explain why
> we rarely see this error in LocalDelivery because if there was a
> problem storing the message in a repository, it normally would have
> been caught when storing it in the spool first (because inboxes and
> spool would use the same repository type. certainly not a requirement
> but just in practice seems to be the case).
> -- Serge Knystautas
> Loki Technologies - Unstoppable Websites
> http://www.lokitech.com/
>
>
> --
> To unsubscribe, e-mail: <mailto:james-dev-
> unsubscribe@jakarta.apache.org>
> For additional commands, e-mail: <mailto:james-dev-
> help@jakarta.apache.org>
>
>
Public Key - 1024D/88DD65AF 2001-11-23 Stuart Roebuck (Adolos)
Key fingerprint = 89D9 E405 F8B1 9B22 0FA2 F2C1 9E57 5AB1 88DD 65AF
-------------------------------------------------------------------------
Stuart Roebuck stuart.roebuck@adolos.com
Systems Architect Java, XML, MacOS X, XP,
etc.
ADOLOS <http://www.adolos.com/>
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
Re: Packet is larger than max_allowed_packet from server configuration
of 1048576 bytes
Posted by Serge Knystautas <se...@lokitech.com>.
Noel J. Bergman wrote:
> I wonder if that is one of those cases that "shouldn't happen", as was just
> discussed in the thread on configuring the SMTP server?
>
> Serge, wouldn't this hit the bad code that we just discussed? Is this
> exception going to be tossed back and handled by LocalDelivery.service(),
> resulting in exactly the consequence we were just talking about? Should
> Stuart pull the current 2.0a3 release apply the patch to
> LocalDelivery.service() that you just posted to the mailing list, and
> rebuild James?
>
> --- Noel
Yes, this is Murphy's law at work... within an hour of pronouncing that
an error was never reported, it's reported. Stuart can apply that patch
and use the latest version to halve the number of messages going into error.
More overly, I'm pretty sure the problem is with the JDBC driver... I'd
bet it's mysql, and while I'm not really expert about that db, there
seems to be a meg limit on the binary field by default.
What's interesting though is (I'd guess that) Stuart is using the db for
the inboxes but file system for spooling. This would explain why we
rarely see this error in LocalDelivery because if there was a problem
storing the message in a repository, it normally would have been caught
when storing it in the spool first (because inboxes and spool would use
the same repository type. certainly not a requirement but just in
practice seems to be the case).
--
Serge Knystautas
Loki Technologies - Unstoppable Websites
http://www.lokitech.com/
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
RE: Packet is larger than max_allowed_packet from server configuration of 1048576 bytes
Posted by "Noel J. Bergman" <no...@devtech.com>.
> Here is the corresponding stack error. Does it mean anything to anyone?
> java.lang.RuntimeException: Exception caught while storing mail
> Container: java.lang.IllegalArgumentException: Packet is larger than
> max_allowed_packet from server configuration of 1048576 bytes
> at
> org.apache.james.mailrepository.JDBCMailRepository.store
> (JDBCMailRepository.java:422)
> at org.apache.james.James.storeMail(James.java:452)
> at java.lang.reflect.Method.invoke(Native Method)
> at
>
org.apache.avalon.phoenix.components.application.BlockInvocationHandler.invo
ke(
> BlockInvocationHandler.java:91)
> at $Proxy9.storeMail(Unknown Source)
> at
>
org.apache.james.transport.mailets.LocalDelivery.service(LocalDelivery.java:
> 34)
...
I wonder if that is one of those cases that "shouldn't happen", as was just
discussed in the thread on configuring the SMTP server?
Serge, wouldn't this hit the bad code that we just discussed? Is this
exception going to be tossed back and handled by LocalDelivery.service(),
resulting in exactly the consequence we were just talking about? Should
Stuart pull the current 2.0a3 release apply the patch to
LocalDelivery.service() that you just posted to the mailing list, and
rebuild James?
--- Noel
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>