You are viewing a plain text version of this content. The canonical link for it is here.
Posted to derby-dev@db.apache.org by "Kathey Marsden (JIRA)" <ji...@apache.org> on 2009/04/16 23:18:14 UTC
[jira] Created: (DERBY-4166) improvements to the mailjdbc test
improvements to the mailjdbc test
---------------------------------
Key: DERBY-4166
URL: https://issues.apache.org/jira/browse/DERBY-4166
Project: Derby
Issue Type: Improvement
Components: Test
Affects Versions: 10.6.0.0
Reporter: Kathey Marsden
Priority: Minor
When recently working with the mailjdbc system test org.apache.derbyTesting.system.mailjdbc on DERBY-4152 I noticed some potential improvements that might be good for the test. We should probably hold off on these improvements however until the root cause of DERBY-4152 is established, however, so we don't muddy the waters with that issue by changing the test.
1) DbTasks.moveToFolders may throw an IllegalArgumentException.
There is a line: message_id = Rn.nextInt(count - 1);
if count is 1 the argument to nextInt() might be 0 which is not allowed. I hit this once but lost the stack trace, but it is apparent that when there is only one row in the table this can occur.
2) Allow/implement multiple attachments per message and cleanup DbTasks.insertMail() logic.
- Remove the attach_id column from INBOX to allow multiple attachments.
-Make the attachment insert part of the message for loop in insertMail.
Use getGeneratedKeys() to get the id of the inserted message.
When attachments are inserted, insert (1-4) attachments and give them a corresponding attach_id from 1-4.
This will allow for removal of the select statements used to determine id and attach_id. I'll file another issue for these improvements if folks agree that they are sensible.
A detailed description of the current implementation of insertMail is described at https://issues.apache.org/jira/secure/attachment/12405685/insertMailSummary.txt
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (DERBY-4166) improvements to the mailjdbc test
Posted by "Lily Wei (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/DERBY-4166?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Lily Wei updated DERBY-4166:
----------------------------
Attachment: DERBY-4166-schemachange.diff
Thanks Kathey for the helping me review the change and submit the changes.
The attach file DERBY-4166-schemachange.diff is to attempt to fix the insert logic to avoid primary key exception and continue using delete cascade and improve the insert logic for REFRESH.ATTACH with new schema. There are a lot of logging changes. When reviewing the test performance, these log info help me to understand what is going on.
> improvements to the mailjdbc test
> ---------------------------------
>
> Key: DERBY-4166
> URL: https://issues.apache.org/jira/browse/DERBY-4166
> Project: Derby
> Issue Type: Improvement
> Components: Test
> Affects Versions: 10.6.0.0
> Reporter: Kathey Marsden
> Priority: Minor
> Attachments: DERBY-4166-databasesize.diff, Derby-4166-samedb.diff, DERBY-4166-schemachange.diff, Derby-4166.diff
>
>
> When recently working with the mailjdbc system test org.apache.derbyTesting.system.mailjdbc on DERBY-4152 I noticed some potential improvements that might be good for the test. We should probably hold off on these improvements however until the root cause of DERBY-4152 is established, however, so we don't muddy the waters with that issue by changing the test.
> 1) DbTasks.moveToFolders may throw an IllegalArgumentException.
> There is a line: message_id = Rn.nextInt(count - 1);
> if count is 1 the argument to nextInt() might be 0 which is not allowed. I hit this once but lost the stack trace, but it is apparent that when there is only one row in the table this can occur.
>
> 2) Allow/implement multiple attachments per message and cleanup DbTasks.insertMail() logic.
> - Remove the attach_id column from INBOX to allow multiple attachments.
> -Make the attachment insert part of the message for loop in insertMail.
> Use getGeneratedKeys() to get the id of the inserted message.
> When attachments are inserted, insert (1-4) attachments and give them a corresponding attach_id from 1-4.
> This will allow for removal of the select statements used to determine id and attach_id. I'll file another issue for these improvements if folks agree that they are sensible.
> A detailed description of the current implementation of insertMail is described at https://issues.apache.org/jira/secure/attachment/12405685/insertMailSummary.txt
> 3) DbTasks.databaseSize calculation is wrong. It doesn't match du -sk. The method does not recurse into subdirectories and includes the length() on directory files which is undefined accourding to the file.length() javadoc.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (DERBY-4166) improvements to the mailjdbc test
Posted by "Lily Wei (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/DERBY-4166?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Lily Wei updated DERBY-4166:
----------------------------
Attachment: derby.log
DERBY-4166-schemachange3.diff
Hi:
After take out select from Refresh thread, rearrange reading mail message then attachment from Browse thread, add sleep time for Refresh thread, Browse thread and Purge thread.
With long sleep time for Purge thread, we no long has to delete row from REFRESH.INBOX using
"delete from REFRESH.INBOX where (values {fn TIMESTAMPDIFF(SQL_TSI_DAY, date,CURRENT_TIMESTAMP)})>1" Hence, no deadlock issue introduce by mailjdbc test.
I was able to run mailjdbc test for more than 11 hours now without any issues.
I am attaching the patch for review. I believe the original intent for interrupt is for the potential deadlock issue. If we truly want to stress test the system for its scale rather than handle deadlock situation, I believe this fix should be okay. I am open for any further approach.
I also attach derby.log just for reference.
Thanks,
Lily
> improvements to the mailjdbc test
> ---------------------------------
>
> Key: DERBY-4166
> URL: https://issues.apache.org/jira/browse/DERBY-4166
> Project: Derby
> Issue Type: Improvement
> Components: Test
> Affects Versions: 10.6.0.0
> Reporter: Kathey Marsden
> Priority: Minor
> Attachments: DERBY-4166-databasesize.diff, Derby-4166-samedb.diff, DERBY-4166-schemachange.diff, DERBY-4166-schemachange2.diff, DERBY-4166-schemachange3.diff, DERBY-4166-schemachange3.diff, DERBY-4166-schemachange4.diff, Derby-4166.diff, derby.log
>
>
> When recently working with the mailjdbc system test org.apache.derbyTesting.system.mailjdbc on DERBY-4152 I noticed some potential improvements that might be good for the test. We should probably hold off on these improvements however until the root cause of DERBY-4152 is established, however, so we don't muddy the waters with that issue by changing the test.
> 1) DbTasks.moveToFolders may throw an IllegalArgumentException.
> There is a line: message_id = Rn.nextInt(count - 1);
> if count is 1 the argument to nextInt() might be 0 which is not allowed. I hit this once but lost the stack trace, but it is apparent that when there is only one row in the table this can occur.
>
> 2) Allow/implement multiple attachments per message and cleanup DbTasks.insertMail() logic.
> - Remove the attach_id column from INBOX to allow multiple attachments.
> -Make the attachment insert part of the message for loop in insertMail.
> Use getGeneratedKeys() to get the id of the inserted message.
> When attachments are inserted, insert (1-4) attachments and give them a corresponding attach_id from 1-4.
> This will allow for removal of the select statements used to determine id and attach_id. I'll file another issue for these improvements if folks agree that they are sensible.
> A detailed description of the current implementation of insertMail is described at https://issues.apache.org/jira/secure/attachment/12405685/insertMailSummary.txt
> 3) DbTasks.databaseSize calculation is wrong. It doesn't match du -sk. The method does not recurse into subdirectories and includes the length() on directory files which is undefined accourding to the file.length() javadoc.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (DERBY-4166) improvements to the mailjdbc test
Posted by "Lily Wei (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/DERBY-4166?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Lily Wei updated DERBY-4166:
----------------------------
Attachment: DERBY-4166-schemachange2.diff
Thanks Kathey for looking at the patch.
-the space/tab issue in schema.sql has been taking care of
-count_att is delete fom DbTasks.deleteMailByThread
-change the logic in Insert mail so
it will
nt num = Rn.nextInt(10 - 1);
for (int i = 0; i < num; i++) {
//insert mail to INBOX
// get the id from select id from REFRESH.INBOX (stmt1)
// insert attachments for approximately 1 in 10 mails
while (i == Rn.nextInt(10 -1)) {
int num_attatch = Rn.nextInt(5-1);
for (int j = 0; j < num_attach;j++) {
// insert the attachment using the id from stmt1
}
Test run fine for a day. However, I am seeing deadlock error now.
Within the day, delete thread hold lock on table inbox which will cascade delete table attach. Also, insert thread will hold lock on inbox for getting the id and holding lock for attach for inserting row to attach. The time the test takes to delete row from tables is usually less than a second. However, the time it takes to insert to inbox and attach can be varying from 0.x second to 16 sec.
To avoid the deadlock, I can:
1. Sleep more before delete thread gets call
2. Look at while inserting less than 10 rows to each table on total of two tables can takes up to 16 seconds.
3. Put id in a array so insert into attach does not hold lock for table inbox and table attach
4. Open for more suggestion.
2009-07-25 07:43:56.494 GMT Thread[Refresh Thread,5,main] (XID = 131869), (SESSI
ONID = 3), (DATABASE = mailsdb), (DRDAID = null), Cleanup action starting^M
2009-07-25 07:43:56.494 GMT Thread[Refresh Thread,5,main] (XID = 131869), (SESSI
ONID = 3), (DATABASE = mailsdb), (DRDAID = null), Failed Statement is: delete fr
om REFRESH.INBOX where to_delete = 1^M
ERROR 40001: A lock could not be obtained due to a deadlock, cycle of locks and
waiters is:
Lock : ROW, INBOX, (1,623)
Waiting XID : {131869, X} , REFRESH, delete from REFRESH.INBOX where to_delete
= 1
Granted XID : {131737, X}
Lock : ROW, INBOX, (1,613)
Waiting XID : {131737, X} , BROWSE, update REFRESH.INBOX set to_delete = ? whe
re id = ?
Granted XID : {131869, X}
. The selected victim is XID : 131869.^M
at org.apache.derby.iapi.error.StandardException.newException(StandardEx
ception.java:303)^M
at org.apache.derby.impl.services.locks.Deadlock.buildException(Deadlock
.java:390)^M
at org.apache.derby.impl.services.locks.ConcurrentLockSet.lockObject(Con
currentLockSet.java:613)^M
at org.apache.derby.impl.services.locks.AbstractPool.lockObject(Abstract
Pool.java:119)^M
at org.apache.derby.impl.services.locks.ConcurrentPool.lockObject(Concur
rentPool.java:28)^M
at org.apache.derby.impl.store.raw.xact.RowLocking3.lockRecordForWrite(R
owLocking3.java:248)^M
at org.apache.derby.impl.store.access.heap.HeapController.lockRow(HeapCo
ntroller.java:504)^M
at org.apache.derby.impl.store.access.heap.HeapController.lockRow(HeapCo
ntroller.java:638)^M
at org.apache.derby.impl.store.access.btree.index.B2IRowLocking3.lockRow
OnPage(B2IRowLocking3.java:309)^M
at org.apache.derby.impl.store.access.btree.index.B2IRowLocking3._lockSc
anRow(B2IRowLocking3.java:599)^M
at org.apache.derby.impl.store.access.btree.index.B2IRowLockingRR.lockSc
anRow(B2IRowLockingRR.java:105)^M
TreeForwardScan.java:305)^M
at org.apache.derby.impl.store.access.btree.BTreeScan.fetchNext(BTreeSca
n.java:1503)^M
at org.apache.derby.impl.sql.execute.TableScanResultSet.getNextRowCore(T
ableScanResultSet.java:680)^M
at org.apache.derby.impl.sql.execute.IndexRowToBaseRowResultSet.getNextR
owCore(IndexRowToBaseRowResultSet.java:373)^M
at org.apache.derby.impl.sql.execute.ProjectRestrictResultSet.getNextRow
Core(ProjectRestrictResultSet.java:256)^M
at org.apache.derby.impl.sql.execute.DMLWriteResultSet.getNextRowCore(DM
LWriteResultSet.java:127)^M
at org.apache.derby.impl.sql.execute.DeleteResultSet.collectAffectedRows
(DeleteResultSet.java:335)^M
at org.apache.derby.impl.sql.execute.DeleteCascadeResultSet.collectAffec
tedRows(DeleteCascadeResultSet.java:185)^M
at org.apache.derby.impl.sql.execute.DeleteCascadeResultSet.open(DeleteC
ascadeResultSet.java:120)^M
at org.apache.derby.impl.sql.GenericPreparedStatement.executeStmt(Generi
cPreparedStatement.java:416)^M
at org.apache.derby.impl.sql.GenericPreparedStatement.execute(GenericPre
paredStatement.java:297)^M
at org.apache.derby.impl.jdbc.EmbedStatement.executeStatement(EmbedState
ment.java:1235)^M
at org.apache.derby.impl.jdbc.EmbedPreparedStatement.executeStatement(Em
bedPreparedStatement.java:1675)^M
at org.apache.derby.impl.jdbc.EmbedPreparedStatement.executeUpdate(Embed
PreparedStatement.java:302)^M
at org.apache.derbyTesting.system.mailjdbc.utils.DbTasks.deleteMailByThr
ead(DbTasks.java:330)^M
at org.apache.derbyTesting.system.mailjdbc.tasks.Refresh.deleteMailByRef
resh(Refresh.java:105)^M
at org.apache.derbyTesting.system.mailjdbc.tasks.Refresh.doWork(Refresh.
java:83)^M
at org.apache.derbyTesting.system.mailjdbc.tasks.Refresh.run(Refresh.jav
a:51)^M
I got 4 deadlock error from one day run.
> improvements to the mailjdbc test
> ---------------------------------
>
> Key: DERBY-4166
> URL: https://issues.apache.org/jira/browse/DERBY-4166
> Project: Derby
> Issue Type: Improvement
> Components: Test
> Affects Versions: 10.6.0.0
> Reporter: Kathey Marsden
> Priority: Minor
> Attachments: DERBY-4166-databasesize.diff, Derby-4166-samedb.diff, DERBY-4166-schemachange.diff, DERBY-4166-schemachange2.diff, Derby-4166.diff
>
>
> When recently working with the mailjdbc system test org.apache.derbyTesting.system.mailjdbc on DERBY-4152 I noticed some potential improvements that might be good for the test. We should probably hold off on these improvements however until the root cause of DERBY-4152 is established, however, so we don't muddy the waters with that issue by changing the test.
> 1) DbTasks.moveToFolders may throw an IllegalArgumentException.
> There is a line: message_id = Rn.nextInt(count - 1);
> if count is 1 the argument to nextInt() might be 0 which is not allowed. I hit this once but lost the stack trace, but it is apparent that when there is only one row in the table this can occur.
>
> 2) Allow/implement multiple attachments per message and cleanup DbTasks.insertMail() logic.
> - Remove the attach_id column from INBOX to allow multiple attachments.
> -Make the attachment insert part of the message for loop in insertMail.
> Use getGeneratedKeys() to get the id of the inserted message.
> When attachments are inserted, insert (1-4) attachments and give them a corresponding attach_id from 1-4.
> This will allow for removal of the select statements used to determine id and attach_id. I'll file another issue for these improvements if folks agree that they are sensible.
> A detailed description of the current implementation of insertMail is described at https://issues.apache.org/jira/secure/attachment/12405685/insertMailSummary.txt
> 3) DbTasks.databaseSize calculation is wrong. It doesn't match du -sk. The method does not recurse into subdirectories and includes the length() on directory files which is undefined accourding to the file.length() javadoc.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (DERBY-4166) improvements to the mailjdbc test
Posted by "Lily Wei (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/DERBY-4166?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Lily Wei updated DERBY-4166:
----------------------------
Attachment: Derby-4166.diff
Change the test to reflect the following points:
1. Take out attach_id column from INBOX table. Add to_delete column to ATTACH table. Change the logic for insert into attach table. Right now, it will try to insert a attachment for each unique id in INBOX. We will mark rows that will be deleted by DeleteMailByUser(). After all the mails have been deleted from one of the thread, SYSCS_UTILS.SYSCS_INPLACE_COMPRESS_TABLE should reclaim space.
2. Change calculation of database size.
3. Add some queries to Statements.java
4. Change log info to achieve easier to read goal
5. Use user REFRESH instead of BACKUP on backup thread to avoid Derby-4295 issue
6. line: message_id = Rn.nextInt(count - 1) had been changed.
Potential issues:
The number of blob insert does not reflect the rows had been deleted from ATTACH table.
The log to track space has been return to O.S. is not very intuitive from the log information.
> improvements to the mailjdbc test
> ---------------------------------
>
> Key: DERBY-4166
> URL: https://issues.apache.org/jira/browse/DERBY-4166
> Project: Derby
> Issue Type: Improvement
> Components: Test
> Affects Versions: 10.6.0.0
> Reporter: Kathey Marsden
> Priority: Minor
> Attachments: Derby-4166.diff
>
>
> When recently working with the mailjdbc system test org.apache.derbyTesting.system.mailjdbc on DERBY-4152 I noticed some potential improvements that might be good for the test. We should probably hold off on these improvements however until the root cause of DERBY-4152 is established, however, so we don't muddy the waters with that issue by changing the test.
> 1) DbTasks.moveToFolders may throw an IllegalArgumentException.
> There is a line: message_id = Rn.nextInt(count - 1);
> if count is 1 the argument to nextInt() might be 0 which is not allowed. I hit this once but lost the stack trace, but it is apparent that when there is only one row in the table this can occur.
>
> 2) Allow/implement multiple attachments per message and cleanup DbTasks.insertMail() logic.
> - Remove the attach_id column from INBOX to allow multiple attachments.
> -Make the attachment insert part of the message for loop in insertMail.
> Use getGeneratedKeys() to get the id of the inserted message.
> When attachments are inserted, insert (1-4) attachments and give them a corresponding attach_id from 1-4.
> This will allow for removal of the select statements used to determine id and attach_id. I'll file another issue for these improvements if folks agree that they are sensible.
> A detailed description of the current implementation of insertMail is described at https://issues.apache.org/jira/secure/attachment/12405685/insertMailSummary.txt
> 3) DbTasks.databaseSize calculation is wrong. It doesn't match du -sk. The method does not recurse into subdirectories and includes the length() on directory files which is undefined accourding to the file.length() javadoc.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (DERBY-4166) improvements to the mailjdbc test
Posted by "Kathey Marsden (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/DERBY-4166?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12732840#action_12732840 ]
Kathey Marsden commented on DERBY-4166:
---------------------------------------
Thanks Lily. I committed DERBY-4166-databasesize.diff to trunk and 10.5.
> improvements to the mailjdbc test
> ---------------------------------
>
> Key: DERBY-4166
> URL: https://issues.apache.org/jira/browse/DERBY-4166
> Project: Derby
> Issue Type: Improvement
> Components: Test
> Affects Versions: 10.6.0.0
> Reporter: Kathey Marsden
> Priority: Minor
> Attachments: DERBY-4166-databasesize.diff, Derby-4166-samedb.diff, Derby-4166.diff
>
>
> When recently working with the mailjdbc system test org.apache.derbyTesting.system.mailjdbc on DERBY-4152 I noticed some potential improvements that might be good for the test. We should probably hold off on these improvements however until the root cause of DERBY-4152 is established, however, so we don't muddy the waters with that issue by changing the test.
> 1) DbTasks.moveToFolders may throw an IllegalArgumentException.
> There is a line: message_id = Rn.nextInt(count - 1);
> if count is 1 the argument to nextInt() might be 0 which is not allowed. I hit this once but lost the stack trace, but it is apparent that when there is only one row in the table this can occur.
>
> 2) Allow/implement multiple attachments per message and cleanup DbTasks.insertMail() logic.
> - Remove the attach_id column from INBOX to allow multiple attachments.
> -Make the attachment insert part of the message for loop in insertMail.
> Use getGeneratedKeys() to get the id of the inserted message.
> When attachments are inserted, insert (1-4) attachments and give them a corresponding attach_id from 1-4.
> This will allow for removal of the select statements used to determine id and attach_id. I'll file another issue for these improvements if folks agree that they are sensible.
> A detailed description of the current implementation of insertMail is described at https://issues.apache.org/jira/secure/attachment/12405685/insertMailSummary.txt
> 3) DbTasks.databaseSize calculation is wrong. It doesn't match du -sk. The method does not recurse into subdirectories and includes the length() on directory files which is undefined accourding to the file.length() javadoc.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (DERBY-4166) improvements to the mailjdbc test
Posted by "Lily Wei (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/DERBY-4166?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Lily Wei updated DERBY-4166:
----------------------------
Attachment: DERBY-4166-schemachange3.diff
Thanks Kathey for helping me with the patch.
I change insertMail() to use the getGeneratedKeys() to get the id as suggested. Also, insert the attachements should be included in the loop to insert the mail in the INBOX. The test should have 1/10 percentage to insert attach emails. I was expecting I don't see any deadlock error after this change. However, I still get deadlock on Refresh thread while deleteMailbyRefresh() query and Browse thread deleteMailbyUSer() are waiting for lock by Refresh thread insertMail().
I close and open connection at two places in the Refresh thread. First, when insertMail() get interrupted in Refresh thread. Second, when DeleteMailbyRefresh() get Exception from test run. After running the tests for more than 10 hours, this patch is running without a problem on embedded server.
> improvements to the mailjdbc test
> ---------------------------------
>
> Key: DERBY-4166
> URL: https://issues.apache.org/jira/browse/DERBY-4166
> Project: Derby
> Issue Type: Improvement
> Components: Test
> Affects Versions: 10.6.0.0
> Reporter: Kathey Marsden
> Priority: Minor
> Attachments: DERBY-4166-databasesize.diff, Derby-4166-samedb.diff, DERBY-4166-schemachange.diff, DERBY-4166-schemachange2.diff, DERBY-4166-schemachange3.diff, Derby-4166.diff
>
>
> When recently working with the mailjdbc system test org.apache.derbyTesting.system.mailjdbc on DERBY-4152 I noticed some potential improvements that might be good for the test. We should probably hold off on these improvements however until the root cause of DERBY-4152 is established, however, so we don't muddy the waters with that issue by changing the test.
> 1) DbTasks.moveToFolders may throw an IllegalArgumentException.
> There is a line: message_id = Rn.nextInt(count - 1);
> if count is 1 the argument to nextInt() might be 0 which is not allowed. I hit this once but lost the stack trace, but it is apparent that when there is only one row in the table this can occur.
>
> 2) Allow/implement multiple attachments per message and cleanup DbTasks.insertMail() logic.
> - Remove the attach_id column from INBOX to allow multiple attachments.
> -Make the attachment insert part of the message for loop in insertMail.
> Use getGeneratedKeys() to get the id of the inserted message.
> When attachments are inserted, insert (1-4) attachments and give them a corresponding attach_id from 1-4.
> This will allow for removal of the select statements used to determine id and attach_id. I'll file another issue for these improvements if folks agree that they are sensible.
> A detailed description of the current implementation of insertMail is described at https://issues.apache.org/jira/secure/attachment/12405685/insertMailSummary.txt
> 3) DbTasks.databaseSize calculation is wrong. It doesn't match du -sk. The method does not recurse into subdirectories and includes the length() on directory files which is undefined accourding to the file.length() javadoc.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (DERBY-4166) improvements to the mailjdbc test
Posted by "Kathey Marsden (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/DERBY-4166?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12735148#action_12735148 ]
Kathey Marsden commented on DERBY-4166:
---------------------------------------
Thanks Lily for the patch.
- In schema.sql there is a small space/tab issue at:
attach_id bigint generated always as identity (start with 1, increment by 1),
- in DbTasks.deleteMailByThread there is a count_att which I think is not used.
- In insertMail I think the code to insert the attachements should be included in the loop to insert the mail in the INBOX and should not require a select from INBOX after the . In general we want.
int num = Rn.nextInt(10 - 1);
for (int i = 0; i < num; i++) {
//insert mail to INBOX
... code looks fine
// get the id. with insertFirst.getGeneratedKeys()
.. need to add
// insert attachments for approximately 1 in 10 mails
if (i == Rn.nextInt(10 -1)) {
// insert from 1 to 5 attachments
num_attatch - Rn.nextInt(5-1);
for (int j = 0; j < num_attach;j++) {
// insert the attachment using the id you got from getGeneratedKeys()
}
}
> improvements to the mailjdbc test
> ---------------------------------
>
> Key: DERBY-4166
> URL: https://issues.apache.org/jira/browse/DERBY-4166
> Project: Derby
> Issue Type: Improvement
> Components: Test
> Affects Versions: 10.6.0.0
> Reporter: Kathey Marsden
> Priority: Minor
> Attachments: DERBY-4166-databasesize.diff, Derby-4166-samedb.diff, DERBY-4166-schemachange.diff, Derby-4166.diff
>
>
> When recently working with the mailjdbc system test org.apache.derbyTesting.system.mailjdbc on DERBY-4152 I noticed some potential improvements that might be good for the test. We should probably hold off on these improvements however until the root cause of DERBY-4152 is established, however, so we don't muddy the waters with that issue by changing the test.
> 1) DbTasks.moveToFolders may throw an IllegalArgumentException.
> There is a line: message_id = Rn.nextInt(count - 1);
> if count is 1 the argument to nextInt() might be 0 which is not allowed. I hit this once but lost the stack trace, but it is apparent that when there is only one row in the table this can occur.
>
> 2) Allow/implement multiple attachments per message and cleanup DbTasks.insertMail() logic.
> - Remove the attach_id column from INBOX to allow multiple attachments.
> -Make the attachment insert part of the message for loop in insertMail.
> Use getGeneratedKeys() to get the id of the inserted message.
> When attachments are inserted, insert (1-4) attachments and give them a corresponding attach_id from 1-4.
> This will allow for removal of the select statements used to determine id and attach_id. I'll file another issue for these improvements if folks agree that they are sensible.
> A detailed description of the current implementation of insertMail is described at https://issues.apache.org/jira/secure/attachment/12405685/insertMailSummary.txt
> 3) DbTasks.databaseSize calculation is wrong. It doesn't match du -sk. The method does not recurse into subdirectories and includes the length() on directory files which is undefined accourding to the file.length() javadoc.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (DERBY-4166) improvements to the mailjdbc test
Posted by "Kathey Marsden (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/DERBY-4166?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12700282#action_12700282 ]
Kathey Marsden commented on DERBY-4166:
---------------------------------------
Here is the stack trace from the IllegalArgumentException
java.lang.IllegalArgumentException
at java.util.Random.nextInt(Random.java:231)
at org.apache.derbyTesting.system.mailjdbc.utils.DbTasks.moveToFolders(DbTasks.java:363)
at org.apache.derbyTesting.system.mailjdbc.tasks.Browse.moveToFolders(Browse.java:82)
at org.apache.derbyTesting.system.mailjdbc.tasks.Browse.run(Browse.java:50)
> improvements to the mailjdbc test
> ---------------------------------
>
> Key: DERBY-4166
> URL: https://issues.apache.org/jira/browse/DERBY-4166
> Project: Derby
> Issue Type: Improvement
> Components: Test
> Affects Versions: 10.6.0.0
> Reporter: Kathey Marsden
> Priority: Minor
>
> When recently working with the mailjdbc system test org.apache.derbyTesting.system.mailjdbc on DERBY-4152 I noticed some potential improvements that might be good for the test. We should probably hold off on these improvements however until the root cause of DERBY-4152 is established, however, so we don't muddy the waters with that issue by changing the test.
> 1) DbTasks.moveToFolders may throw an IllegalArgumentException.
> There is a line: message_id = Rn.nextInt(count - 1);
> if count is 1 the argument to nextInt() might be 0 which is not allowed. I hit this once but lost the stack trace, but it is apparent that when there is only one row in the table this can occur.
>
> 2) Allow/implement multiple attachments per message and cleanup DbTasks.insertMail() logic.
> - Remove the attach_id column from INBOX to allow multiple attachments.
> -Make the attachment insert part of the message for loop in insertMail.
> Use getGeneratedKeys() to get the id of the inserted message.
> When attachments are inserted, insert (1-4) attachments and give them a corresponding attach_id from 1-4.
> This will allow for removal of the select statements used to determine id and attach_id. I'll file another issue for these improvements if folks agree that they are sensible.
> A detailed description of the current implementation of insertMail is described at https://issues.apache.org/jira/secure/attachment/12405685/insertMailSummary.txt
> 3) DbTasks.databaseSize calculation is wrong. It doesn't match du -sk. The method does not recurse into subdirectories and includes the length() on directory files which is undefined accourding to the file.length() javadoc.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (DERBY-4166) improvements to the mailjdbc test
Posted by "Kathey Marsden (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/DERBY-4166?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12732593#action_12732593 ]
Kathey Marsden commented on DERBY-4166:
---------------------------------------
Hi Lily.
Visual inspection of the patch looks fine except a typo in the instructions which has "the same database as twice:
If you want to use the same database as the same database as previous run.
Can you make a new patch with this small change and
1) Do svn diff from the top of the tree. If you want to include only specific files in your work space you can just specify those on the command line.
2) Attach the patch to DERBY-4203 instead. I think it makes more sense to put it with that issue.
> improvements to the mailjdbc test
> ---------------------------------
>
> Key: DERBY-4166
> URL: https://issues.apache.org/jira/browse/DERBY-4166
> Project: Derby
> Issue Type: Improvement
> Components: Test
> Affects Versions: 10.6.0.0
> Reporter: Kathey Marsden
> Priority: Minor
> Attachments: Derby-4166-samedb.diff, Derby-4166.diff
>
>
> When recently working with the mailjdbc system test org.apache.derbyTesting.system.mailjdbc on DERBY-4152 I noticed some potential improvements that might be good for the test. We should probably hold off on these improvements however until the root cause of DERBY-4152 is established, however, so we don't muddy the waters with that issue by changing the test.
> 1) DbTasks.moveToFolders may throw an IllegalArgumentException.
> There is a line: message_id = Rn.nextInt(count - 1);
> if count is 1 the argument to nextInt() might be 0 which is not allowed. I hit this once but lost the stack trace, but it is apparent that when there is only one row in the table this can occur.
>
> 2) Allow/implement multiple attachments per message and cleanup DbTasks.insertMail() logic.
> - Remove the attach_id column from INBOX to allow multiple attachments.
> -Make the attachment insert part of the message for loop in insertMail.
> Use getGeneratedKeys() to get the id of the inserted message.
> When attachments are inserted, insert (1-4) attachments and give them a corresponding attach_id from 1-4.
> This will allow for removal of the select statements used to determine id and attach_id. I'll file another issue for these improvements if folks agree that they are sensible.
> A detailed description of the current implementation of insertMail is described at https://issues.apache.org/jira/secure/attachment/12405685/insertMailSummary.txt
> 3) DbTasks.databaseSize calculation is wrong. It doesn't match du -sk. The method does not recurse into subdirectories and includes the length() on directory files which is undefined accourding to the file.length() javadoc.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (DERBY-4166) improvements to the mailjdbc test
Posted by "Lily Wei (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/DERBY-4166?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Lily Wei updated DERBY-4166:
----------------------------
Attachment: DERBY-4166-schemachange6.diff
1. Change the number of row delete by Purge thread. Now, the test usually maintain between 10 - 56 rows during test duration. I had run DERBY-4166-schemachange6.diff for more than 11 hours and everything is running smooth. I run it with embedded mode and network server mode.
2. The test is ready for review and suggestion for changes.
> improvements to the mailjdbc test
> ---------------------------------
>
> Key: DERBY-4166
> URL: https://issues.apache.org/jira/browse/DERBY-4166
> Project: Derby
> Issue Type: Improvement
> Components: Test
> Affects Versions: 10.6.0.0
> Reporter: Kathey Marsden
> Priority: Minor
> Attachments: DERBY-4166-databasesize.diff, Derby-4166-samedb.diff, DERBY-4166-schemachange.diff, DERBY-4166-schemachange2.diff, DERBY-4166-schemachange3.diff, DERBY-4166-schemachange3.diff, DERBY-4166-schemachange4.diff, DERBY-4166-schemachange5.diff, DERBY-4166-schemachange6.diff, Derby-4166.diff, derby.log
>
>
> When recently working with the mailjdbc system test org.apache.derbyTesting.system.mailjdbc on DERBY-4152 I noticed some potential improvements that might be good for the test. We should probably hold off on these improvements however until the root cause of DERBY-4152 is established, however, so we don't muddy the waters with that issue by changing the test.
> 1) DbTasks.moveToFolders may throw an IllegalArgumentException.
> There is a line: message_id = Rn.nextInt(count - 1);
> if count is 1 the argument to nextInt() might be 0 which is not allowed. I hit this once but lost the stack trace, but it is apparent that when there is only one row in the table this can occur.
>
> 2) Allow/implement multiple attachments per message and cleanup DbTasks.insertMail() logic.
> - Remove the attach_id column from INBOX to allow multiple attachments.
> -Make the attachment insert part of the message for loop in insertMail.
> Use getGeneratedKeys() to get the id of the inserted message.
> When attachments are inserted, insert (1-4) attachments and give them a corresponding attach_id from 1-4.
> This will allow for removal of the select statements used to determine id and attach_id. I'll file another issue for these improvements if folks agree that they are sensible.
> A detailed description of the current implementation of insertMail is described at https://issues.apache.org/jira/secure/attachment/12405685/insertMailSummary.txt
> 3) DbTasks.databaseSize calculation is wrong. It doesn't match du -sk. The method does not recurse into subdirectories and includes the length() on directory files which is undefined accourding to the file.length() javadoc.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Assigned: (DERBY-4166) improvements to the mailjdbc test
Posted by "Lily Wei (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/DERBY-4166?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Lily Wei reassigned DERBY-4166:
-------------------------------
Assignee: Lily Wei
> improvements to the mailjdbc test
> ---------------------------------
>
> Key: DERBY-4166
> URL: https://issues.apache.org/jira/browse/DERBY-4166
> Project: Derby
> Issue Type: Improvement
> Components: Test
> Affects Versions: 10.6.0.0
> Reporter: Kathey Marsden
> Assignee: Lily Wei
> Priority: Minor
> Attachments: DERBY-4166-databasesize.diff, Derby-4166-samedb.diff, DERBY-4166-schemachange.diff, DERBY-4166-schemachange2.diff, DERBY-4166-schemachange3.diff, DERBY-4166-schemachange3.diff, DERBY-4166-schemachange4.diff, DERBY-4166-schemachange5.diff, DERBY-4166-schemachange6.diff, DERBY-4166-schemachange7.diff, Derby-4166.diff, derby.log
>
>
> When recently working with the mailjdbc system test org.apache.derbyTesting.system.mailjdbc on DERBY-4152 I noticed some potential improvements that might be good for the test. We should probably hold off on these improvements however until the root cause of DERBY-4152 is established, however, so we don't muddy the waters with that issue by changing the test.
> 1) DbTasks.moveToFolders may throw an IllegalArgumentException.
> There is a line: message_id = Rn.nextInt(count - 1);
> if count is 1 the argument to nextInt() might be 0 which is not allowed. I hit this once but lost the stack trace, but it is apparent that when there is only one row in the table this can occur.
>
> 2) Allow/implement multiple attachments per message and cleanup DbTasks.insertMail() logic.
> - Remove the attach_id column from INBOX to allow multiple attachments.
> -Make the attachment insert part of the message for loop in insertMail.
> Use getGeneratedKeys() to get the id of the inserted message.
> When attachments are inserted, insert (1-4) attachments and give them a corresponding attach_id from 1-4.
> This will allow for removal of the select statements used to determine id and attach_id. I'll file another issue for these improvements if folks agree that they are sensible.
> A detailed description of the current implementation of insertMail is described at https://issues.apache.org/jira/secure/attachment/12405685/insertMailSummary.txt
> 3) DbTasks.databaseSize calculation is wrong. It doesn't match du -sk. The method does not recurse into subdirectories and includes the length() on directory files which is undefined accourding to the file.length() javadoc.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (DERBY-4166) improvements to the mailjdbc test
Posted by "Kathey Marsden (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/DERBY-4166?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Kathey Marsden updated DERBY-4166:
----------------------------------
Description:
When recently working with the mailjdbc system test org.apache.derbyTesting.system.mailjdbc on DERBY-4152 I noticed some potential improvements that might be good for the test. We should probably hold off on these improvements however until the root cause of DERBY-4152 is established, however, so we don't muddy the waters with that issue by changing the test.
1) DbTasks.moveToFolders may throw an IllegalArgumentException.
There is a line: message_id = Rn.nextInt(count - 1);
if count is 1 the argument to nextInt() might be 0 which is not allowed. I hit this once but lost the stack trace, but it is apparent that when there is only one row in the table this can occur.
2) Allow/implement multiple attachments per message and cleanup DbTasks.insertMail() logic.
- Remove the attach_id column from INBOX to allow multiple attachments.
-Make the attachment insert part of the message for loop in insertMail.
Use getGeneratedKeys() to get the id of the inserted message.
When attachments are inserted, insert (1-4) attachments and give them a corresponding attach_id from 1-4.
This will allow for removal of the select statements used to determine id and attach_id. I'll file another issue for these improvements if folks agree that they are sensible.
A detailed description of the current implementation of insertMail is described at https://issues.apache.org/jira/secure/attachment/12405685/insertMailSummary.txt
3) DbTasks.databaseSize calculation is wrong. It doesn't match du -sk. The method does not recurse into subdirectories and includes the length() on directory files which is undefined accourding to the file.length() javadoc.
was:
When recently working with the mailjdbc system test org.apache.derbyTesting.system.mailjdbc on DERBY-4152 I noticed some potential improvements that might be good for the test. We should probably hold off on these improvements however until the root cause of DERBY-4152 is established, however, so we don't muddy the waters with that issue by changing the test.
1) DbTasks.moveToFolders may throw an IllegalArgumentException.
There is a line: message_id = Rn.nextInt(count - 1);
if count is 1 the argument to nextInt() might be 0 which is not allowed. I hit this once but lost the stack trace, but it is apparent that when there is only one row in the table this can occur.
2) Allow/implement multiple attachments per message and cleanup DbTasks.insertMail() logic.
- Remove the attach_id column from INBOX to allow multiple attachments.
-Make the attachment insert part of the message for loop in insertMail.
Use getGeneratedKeys() to get the id of the inserted message.
When attachments are inserted, insert (1-4) attachments and give them a corresponding attach_id from 1-4.
This will allow for removal of the select statements used to determine id and attach_id. I'll file another issue for these improvements if folks agree that they are sensible.
A detailed description of the current implementation of insertMail is described at https://issues.apache.org/jira/secure/attachment/12405685/insertMailSummary.txt
> improvements to the mailjdbc test
> ---------------------------------
>
> Key: DERBY-4166
> URL: https://issues.apache.org/jira/browse/DERBY-4166
> Project: Derby
> Issue Type: Improvement
> Components: Test
> Affects Versions: 10.6.0.0
> Reporter: Kathey Marsden
> Priority: Minor
>
> When recently working with the mailjdbc system test org.apache.derbyTesting.system.mailjdbc on DERBY-4152 I noticed some potential improvements that might be good for the test. We should probably hold off on these improvements however until the root cause of DERBY-4152 is established, however, so we don't muddy the waters with that issue by changing the test.
> 1) DbTasks.moveToFolders may throw an IllegalArgumentException.
> There is a line: message_id = Rn.nextInt(count - 1);
> if count is 1 the argument to nextInt() might be 0 which is not allowed. I hit this once but lost the stack trace, but it is apparent that when there is only one row in the table this can occur.
>
> 2) Allow/implement multiple attachments per message and cleanup DbTasks.insertMail() logic.
> - Remove the attach_id column from INBOX to allow multiple attachments.
> -Make the attachment insert part of the message for loop in insertMail.
> Use getGeneratedKeys() to get the id of the inserted message.
> When attachments are inserted, insert (1-4) attachments and give them a corresponding attach_id from 1-4.
> This will allow for removal of the select statements used to determine id and attach_id. I'll file another issue for these improvements if folks agree that they are sensible.
> A detailed description of the current implementation of insertMail is described at https://issues.apache.org/jira/secure/attachment/12405685/insertMailSummary.txt
> 3) DbTasks.databaseSize calculation is wrong. It doesn't match du -sk. The method does not recurse into subdirectories and includes the length() on directory files which is undefined accourding to the file.length() javadoc.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (DERBY-4166) improvements to the mailjdbc test
Posted by "Kathey Marsden (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/DERBY-4166?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Kathey Marsden updated DERBY-4166:
----------------------------------
Attachment: (was: derby-4152_quick_change_to_stop_pk_violation_diff.txt)
> improvements to the mailjdbc test
> ---------------------------------
>
> Key: DERBY-4166
> URL: https://issues.apache.org/jira/browse/DERBY-4166
> Project: Derby
> Issue Type: Improvement
> Components: Test
> Affects Versions: 10.6.0.0
> Reporter: Kathey Marsden
> Priority: Minor
>
> When recently working with the mailjdbc system test org.apache.derbyTesting.system.mailjdbc on DERBY-4152 I noticed some potential improvements that might be good for the test. We should probably hold off on these improvements however until the root cause of DERBY-4152 is established, however, so we don't muddy the waters with that issue by changing the test.
> 1) DbTasks.moveToFolders may throw an IllegalArgumentException.
> There is a line: message_id = Rn.nextInt(count - 1);
> if count is 1 the argument to nextInt() might be 0 which is not allowed. I hit this once but lost the stack trace, but it is apparent that when there is only one row in the table this can occur.
>
> 2) Allow/implement multiple attachments per message and cleanup DbTasks.insertMail() logic.
> - Remove the attach_id column from INBOX to allow multiple attachments.
> -Make the attachment insert part of the message for loop in insertMail.
> Use getGeneratedKeys() to get the id of the inserted message.
> When attachments are inserted, insert (1-4) attachments and give them a corresponding attach_id from 1-4.
> This will allow for removal of the select statements used to determine id and attach_id. I'll file another issue for these improvements if folks agree that they are sensible.
> A detailed description of the current implementation of insertMail is described at https://issues.apache.org/jira/secure/attachment/12405685/insertMailSummary.txt
> 3) DbTasks.databaseSize calculation is wrong. It doesn't match du -sk. The method does not recurse into subdirectories and includes the length() on directory files which is undefined accourding to the file.length() javadoc.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (DERBY-4166) improvements to the mailjdbc test
Posted by "Kathey Marsden (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/DERBY-4166?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12735780#action_12735780 ]
Kathey Marsden commented on DERBY-4166:
---------------------------------------
I took a quick look at the patch and I think we are getting very close. I have to go over the insert logic again in some detail to post some comments but was wondering about this:
>I close and open connection at two places in the Refresh thread. First, when insertMail() get interrupted in Refresh thread. Second, when DeleteMailbyRefresh() get Exception from test run.
I think that it should suffice to just rollback() at these points instead of closing and reopening the connection. Does just a rollback() eliminate the deadlocks and timeouts?
> improvements to the mailjdbc test
> ---------------------------------
>
> Key: DERBY-4166
> URL: https://issues.apache.org/jira/browse/DERBY-4166
> Project: Derby
> Issue Type: Improvement
> Components: Test
> Affects Versions: 10.6.0.0
> Reporter: Kathey Marsden
> Priority: Minor
> Attachments: DERBY-4166-databasesize.diff, Derby-4166-samedb.diff, DERBY-4166-schemachange.diff, DERBY-4166-schemachange2.diff, DERBY-4166-schemachange3.diff, Derby-4166.diff
>
>
> When recently working with the mailjdbc system test org.apache.derbyTesting.system.mailjdbc on DERBY-4152 I noticed some potential improvements that might be good for the test. We should probably hold off on these improvements however until the root cause of DERBY-4152 is established, however, so we don't muddy the waters with that issue by changing the test.
> 1) DbTasks.moveToFolders may throw an IllegalArgumentException.
> There is a line: message_id = Rn.nextInt(count - 1);
> if count is 1 the argument to nextInt() might be 0 which is not allowed. I hit this once but lost the stack trace, but it is apparent that when there is only one row in the table this can occur.
>
> 2) Allow/implement multiple attachments per message and cleanup DbTasks.insertMail() logic.
> - Remove the attach_id column from INBOX to allow multiple attachments.
> -Make the attachment insert part of the message for loop in insertMail.
> Use getGeneratedKeys() to get the id of the inserted message.
> When attachments are inserted, insert (1-4) attachments and give them a corresponding attach_id from 1-4.
> This will allow for removal of the select statements used to determine id and attach_id. I'll file another issue for these improvements if folks agree that they are sensible.
> A detailed description of the current implementation of insertMail is described at https://issues.apache.org/jira/secure/attachment/12405685/insertMailSummary.txt
> 3) DbTasks.databaseSize calculation is wrong. It doesn't match du -sk. The method does not recurse into subdirectories and includes the length() on directory files which is undefined accourding to the file.length() javadoc.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (DERBY-4166) improvements to the mailjdbc test
Posted by "Lily Wei (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/DERBY-4166?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12732620#action_12732620 ]
Lily Wei commented on DERBY-4166:
---------------------------------
Thanks Kathey for reviewing this. I attach another patch on DERBY-4203. The next step should be change the database size method.
> improvements to the mailjdbc test
> ---------------------------------
>
> Key: DERBY-4166
> URL: https://issues.apache.org/jira/browse/DERBY-4166
> Project: Derby
> Issue Type: Improvement
> Components: Test
> Affects Versions: 10.6.0.0
> Reporter: Kathey Marsden
> Priority: Minor
> Attachments: Derby-4166-samedb.diff, Derby-4166.diff
>
>
> When recently working with the mailjdbc system test org.apache.derbyTesting.system.mailjdbc on DERBY-4152 I noticed some potential improvements that might be good for the test. We should probably hold off on these improvements however until the root cause of DERBY-4152 is established, however, so we don't muddy the waters with that issue by changing the test.
> 1) DbTasks.moveToFolders may throw an IllegalArgumentException.
> There is a line: message_id = Rn.nextInt(count - 1);
> if count is 1 the argument to nextInt() might be 0 which is not allowed. I hit this once but lost the stack trace, but it is apparent that when there is only one row in the table this can occur.
>
> 2) Allow/implement multiple attachments per message and cleanup DbTasks.insertMail() logic.
> - Remove the attach_id column from INBOX to allow multiple attachments.
> -Make the attachment insert part of the message for loop in insertMail.
> Use getGeneratedKeys() to get the id of the inserted message.
> When attachments are inserted, insert (1-4) attachments and give them a corresponding attach_id from 1-4.
> This will allow for removal of the select statements used to determine id and attach_id. I'll file another issue for these improvements if folks agree that they are sensible.
> A detailed description of the current implementation of insertMail is described at https://issues.apache.org/jira/secure/attachment/12405685/insertMailSummary.txt
> 3) DbTasks.databaseSize calculation is wrong. It doesn't match du -sk. The method does not recurse into subdirectories and includes the length() on directory files which is undefined accourding to the file.length() javadoc.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (DERBY-4166) improvements to the mailjdbc test
Posted by "Kathey Marsden (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/DERBY-4166?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12735320#action_12735320 ]
Kathey Marsden commented on DERBY-4166:
---------------------------------------
Lily said in an email
> Yes, it is schemachange2. I need the same Id in attach table because it is a foreign key of inbox. I don't know how getGeneratedKeys work. Does it get the same key in inbox?
Yes. getGeneratedKeys() will get the id that was just inserted into the inbox identity column, so you don't have to query the table to get it. See
http://java.sun.com/javase/6/docs/api/java/sql/Statement.html#getGeneratedKeys()
and
http://db.apache.org/derby/docs/dev/ref/crefjavstateautogen.html
for information
and the test AugoGenJDBC30Test for examples.
Lily also said:
>Since delete thread will delete data in inbox table, the id in inbox table will keep changing as test run.
I have not studied the delete logic carefully, but will take a look. It should be able to accommodate the approach I described for the inserts without deadlocks.
Because we will be inserting each email content and attachments as a single transaction, the delete thread should not consider it for deletion until the insert is complete and the id we get with getGeneratedKeys() should still be valid when we insert the attachments.
> improvements to the mailjdbc test
> ---------------------------------
>
> Key: DERBY-4166
> URL: https://issues.apache.org/jira/browse/DERBY-4166
> Project: Derby
> Issue Type: Improvement
> Components: Test
> Affects Versions: 10.6.0.0
> Reporter: Kathey Marsden
> Priority: Minor
> Attachments: DERBY-4166-databasesize.diff, Derby-4166-samedb.diff, DERBY-4166-schemachange.diff, DERBY-4166-schemachange2.diff, Derby-4166.diff
>
>
> When recently working with the mailjdbc system test org.apache.derbyTesting.system.mailjdbc on DERBY-4152 I noticed some potential improvements that might be good for the test. We should probably hold off on these improvements however until the root cause of DERBY-4152 is established, however, so we don't muddy the waters with that issue by changing the test.
> 1) DbTasks.moveToFolders may throw an IllegalArgumentException.
> There is a line: message_id = Rn.nextInt(count - 1);
> if count is 1 the argument to nextInt() might be 0 which is not allowed. I hit this once but lost the stack trace, but it is apparent that when there is only one row in the table this can occur.
>
> 2) Allow/implement multiple attachments per message and cleanup DbTasks.insertMail() logic.
> - Remove the attach_id column from INBOX to allow multiple attachments.
> -Make the attachment insert part of the message for loop in insertMail.
> Use getGeneratedKeys() to get the id of the inserted message.
> When attachments are inserted, insert (1-4) attachments and give them a corresponding attach_id from 1-4.
> This will allow for removal of the select statements used to determine id and attach_id. I'll file another issue for these improvements if folks agree that they are sensible.
> A detailed description of the current implementation of insertMail is described at https://issues.apache.org/jira/secure/attachment/12405685/insertMailSummary.txt
> 3) DbTasks.databaseSize calculation is wrong. It doesn't match du -sk. The method does not recurse into subdirectories and includes the length() on directory files which is undefined accourding to the file.length() javadoc.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (DERBY-4166) improvements to the mailjdbc test
Posted by "Kathey Marsden (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/DERBY-4166?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12729393#action_12729393 ]
Kathey Marsden commented on DERBY-4166:
---------------------------------------
- I think an argument like "samedb" would be better than upgrade, since sometimes you would want to resume with the same database without upgrade.
- In DbTasks.java jdbcLoad we should add javadoc describing the parameters including the new one . I think a boolean useExistingdb would be better than upgrade, since it might be used on the same branch without upgrade.
- I think for the useExistingdb option I think we should not attempt to run the schema.sql script.
- I think instead of adding to_delete to the attach table for deletion we should maybe try just having a trigger that deletes from the attach table when we delete the mail from the inbox.
- I think your database size method looks good but take out the commented old code.
> improvements to the mailjdbc test
> ---------------------------------
>
> Key: DERBY-4166
> URL: https://issues.apache.org/jira/browse/DERBY-4166
> Project: Derby
> Issue Type: Improvement
> Components: Test
> Affects Versions: 10.6.0.0
> Reporter: Kathey Marsden
> Priority: Minor
> Attachments: Derby-4166.diff
>
>
> When recently working with the mailjdbc system test org.apache.derbyTesting.system.mailjdbc on DERBY-4152 I noticed some potential improvements that might be good for the test. We should probably hold off on these improvements however until the root cause of DERBY-4152 is established, however, so we don't muddy the waters with that issue by changing the test.
> 1) DbTasks.moveToFolders may throw an IllegalArgumentException.
> There is a line: message_id = Rn.nextInt(count - 1);
> if count is 1 the argument to nextInt() might be 0 which is not allowed. I hit this once but lost the stack trace, but it is apparent that when there is only one row in the table this can occur.
>
> 2) Allow/implement multiple attachments per message and cleanup DbTasks.insertMail() logic.
> - Remove the attach_id column from INBOX to allow multiple attachments.
> -Make the attachment insert part of the message for loop in insertMail.
> Use getGeneratedKeys() to get the id of the inserted message.
> When attachments are inserted, insert (1-4) attachments and give them a corresponding attach_id from 1-4.
> This will allow for removal of the select statements used to determine id and attach_id. I'll file another issue for these improvements if folks agree that they are sensible.
> A detailed description of the current implementation of insertMail is described at https://issues.apache.org/jira/secure/attachment/12405685/insertMailSummary.txt
> 3) DbTasks.databaseSize calculation is wrong. It doesn't match du -sk. The method does not recurse into subdirectories and includes the length() on directory files which is undefined accourding to the file.length() javadoc.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (DERBY-4166) improvements to the mailjdbc test
Posted by "Kathey Marsden (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/DERBY-4166?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Kathey Marsden updated DERBY-4166:
----------------------------------
Attachment: derby-4152_quick_change_to_stop_pk_violation_diff.txt
My trunk run with the small change to eliminate the pk contraint violations has been running more than 24 hours and I have seen no significant growth with my database size ranging from 32MB - 80MB and the size going up and down as it runs.
I've attached the test diff for reference (derby-4152_quick_change_to_stop_pk_violation_diff.txt). This is not for commit, but just provides context for the test run.
So since it seems that the pk constraint violations are the only cause of growth in the test on trunk, and I guess it follows that the 10.5 run was either seeing more pk constraint violations or was growing more on the same event, although that was not apparent from my small test case.
Next I'll go back to the original behavior and change the test to record more information on the pk constraint violations, including size of the blob and space table printout and then compare the trunk and 10.4 runs.
> improvements to the mailjdbc test
> ---------------------------------
>
> Key: DERBY-4166
> URL: https://issues.apache.org/jira/browse/DERBY-4166
> Project: Derby
> Issue Type: Improvement
> Components: Test
> Affects Versions: 10.6.0.0
> Reporter: Kathey Marsden
> Priority: Minor
> Attachments: derby-4152_quick_change_to_stop_pk_violation_diff.txt
>
>
> When recently working with the mailjdbc system test org.apache.derbyTesting.system.mailjdbc on DERBY-4152 I noticed some potential improvements that might be good for the test. We should probably hold off on these improvements however until the root cause of DERBY-4152 is established, however, so we don't muddy the waters with that issue by changing the test.
> 1) DbTasks.moveToFolders may throw an IllegalArgumentException.
> There is a line: message_id = Rn.nextInt(count - 1);
> if count is 1 the argument to nextInt() might be 0 which is not allowed. I hit this once but lost the stack trace, but it is apparent that when there is only one row in the table this can occur.
>
> 2) Allow/implement multiple attachments per message and cleanup DbTasks.insertMail() logic.
> - Remove the attach_id column from INBOX to allow multiple attachments.
> -Make the attachment insert part of the message for loop in insertMail.
> Use getGeneratedKeys() to get the id of the inserted message.
> When attachments are inserted, insert (1-4) attachments and give them a corresponding attach_id from 1-4.
> This will allow for removal of the select statements used to determine id and attach_id. I'll file another issue for these improvements if folks agree that they are sensible.
> A detailed description of the current implementation of insertMail is described at https://issues.apache.org/jira/secure/attachment/12405685/insertMailSummary.txt
> 3) DbTasks.databaseSize calculation is wrong. It doesn't match du -sk. The method does not recurse into subdirectories and includes the length() on directory files which is undefined accourding to the file.length() javadoc.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (DERBY-4166) improvements to the mailjdbc test
Posted by "Lily Wei (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/DERBY-4166?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Lily Wei updated DERBY-4166:
----------------------------
Attachment: DERBY-4166-databasesize.diff
The is purpose database size calculate function change. Instead of only calculate database size base on <database dir>/seg0. We calculate every files under <database dir>. I am putting the change to 10.5 in case we can use it in 10.5.2.0 testing.
> improvements to the mailjdbc test
> ---------------------------------
>
> Key: DERBY-4166
> URL: https://issues.apache.org/jira/browse/DERBY-4166
> Project: Derby
> Issue Type: Improvement
> Components: Test
> Affects Versions: 10.6.0.0
> Reporter: Kathey Marsden
> Priority: Minor
> Attachments: DERBY-4166-databasesize.diff, Derby-4166-samedb.diff, Derby-4166.diff
>
>
> When recently working with the mailjdbc system test org.apache.derbyTesting.system.mailjdbc on DERBY-4152 I noticed some potential improvements that might be good for the test. We should probably hold off on these improvements however until the root cause of DERBY-4152 is established, however, so we don't muddy the waters with that issue by changing the test.
> 1) DbTasks.moveToFolders may throw an IllegalArgumentException.
> There is a line: message_id = Rn.nextInt(count - 1);
> if count is 1 the argument to nextInt() might be 0 which is not allowed. I hit this once but lost the stack trace, but it is apparent that when there is only one row in the table this can occur.
>
> 2) Allow/implement multiple attachments per message and cleanup DbTasks.insertMail() logic.
> - Remove the attach_id column from INBOX to allow multiple attachments.
> -Make the attachment insert part of the message for loop in insertMail.
> Use getGeneratedKeys() to get the id of the inserted message.
> When attachments are inserted, insert (1-4) attachments and give them a corresponding attach_id from 1-4.
> This will allow for removal of the select statements used to determine id and attach_id. I'll file another issue for these improvements if folks agree that they are sensible.
> A detailed description of the current implementation of insertMail is described at https://issues.apache.org/jira/secure/attachment/12405685/insertMailSummary.txt
> 3) DbTasks.databaseSize calculation is wrong. It doesn't match du -sk. The method does not recurse into subdirectories and includes the length() on directory files which is undefined accourding to the file.length() javadoc.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (DERBY-4166) improvements to the mailjdbc test
Posted by "Kathey Marsden (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/DERBY-4166?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12843313#action_12843313 ]
Kathey Marsden commented on DERBY-4166:
---------------------------------------
Hi Lily,
I committed to the 7 patch and think it is a good incremental improvement. I am a bit concerned about extending the timeouts as a way to avoid the deadlocks. I think there is still some problem with the test logic that is causing the deadlocks and hope we can reduce the sleeps later. For a stress test, too much sleep time just means less testing each day in the stress test.
I think we can close this one, but I think we should make a new issue to reduce the timeouts and eliminate the deadlocks and also track down why the compress is needed to reclaim space used by the test, since it seemed to be something different than DERBY-4055.
> improvements to the mailjdbc test
> ---------------------------------
>
> Key: DERBY-4166
> URL: https://issues.apache.org/jira/browse/DERBY-4166
> Project: Derby
> Issue Type: Improvement
> Components: Test
> Affects Versions: 10.6.0.0
> Reporter: Kathey Marsden
> Assignee: Lily Wei
> Priority: Minor
> Attachments: DERBY-4166-databasesize.diff, Derby-4166-samedb.diff, DERBY-4166-schemachange.diff, DERBY-4166-schemachange2.diff, DERBY-4166-schemachange3.diff, DERBY-4166-schemachange3.diff, DERBY-4166-schemachange4.diff, DERBY-4166-schemachange5.diff, DERBY-4166-schemachange6.diff, DERBY-4166-schemachange7.diff, Derby-4166.diff, derby.log
>
>
> When recently working with the mailjdbc system test org.apache.derbyTesting.system.mailjdbc on DERBY-4152 I noticed some potential improvements that might be good for the test. We should probably hold off on these improvements however until the root cause of DERBY-4152 is established, however, so we don't muddy the waters with that issue by changing the test.
> 1) DbTasks.moveToFolders may throw an IllegalArgumentException.
> There is a line: message_id = Rn.nextInt(count - 1);
> if count is 1 the argument to nextInt() might be 0 which is not allowed. I hit this once but lost the stack trace, but it is apparent that when there is only one row in the table this can occur.
>
> 2) Allow/implement multiple attachments per message and cleanup DbTasks.insertMail() logic.
> - Remove the attach_id column from INBOX to allow multiple attachments.
> -Make the attachment insert part of the message for loop in insertMail.
> Use getGeneratedKeys() to get the id of the inserted message.
> When attachments are inserted, insert (1-4) attachments and give them a corresponding attach_id from 1-4.
> This will allow for removal of the select statements used to determine id and attach_id. I'll file another issue for these improvements if folks agree that they are sensible.
> A detailed description of the current implementation of insertMail is described at https://issues.apache.org/jira/secure/attachment/12405685/insertMailSummary.txt
> 3) DbTasks.databaseSize calculation is wrong. It doesn't match du -sk. The method does not recurse into subdirectories and includes the length() on directory files which is undefined accourding to the file.length() javadoc.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (DERBY-4166) improvements to the mailjdbc test
Posted by "Lily Wei (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/DERBY-4166?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Lily Wei updated DERBY-4166:
----------------------------
Attachment: DERBY-4166-schemachange4.diff
I was hoping rollback will take care of the deadlock problem. However, with rollback, close connection and reopen connection, the mailjdbc embedded test still has deadlock error after 16 hours. I fix some Rn.nextInt issue for attach_id on line 200 for DbTasks.java. for DERBY-4166-schemachange4.diff. When insert is taking longer than 10 seconds, the tests will run into the situation of deadlock. I think we are very close too. I am still thinking doing more on mailjdbc side to avoid deadlock situation. Interrupt Refresh thread still happens, InsertMail method did not fail.
> improvements to the mailjdbc test
> ---------------------------------
>
> Key: DERBY-4166
> URL: https://issues.apache.org/jira/browse/DERBY-4166
> Project: Derby
> Issue Type: Improvement
> Components: Test
> Affects Versions: 10.6.0.0
> Reporter: Kathey Marsden
> Priority: Minor
> Attachments: DERBY-4166-databasesize.diff, Derby-4166-samedb.diff, DERBY-4166-schemachange.diff, DERBY-4166-schemachange2.diff, DERBY-4166-schemachange3.diff, DERBY-4166-schemachange4.diff, Derby-4166.diff
>
>
> When recently working with the mailjdbc system test org.apache.derbyTesting.system.mailjdbc on DERBY-4152 I noticed some potential improvements that might be good for the test. We should probably hold off on these improvements however until the root cause of DERBY-4152 is established, however, so we don't muddy the waters with that issue by changing the test.
> 1) DbTasks.moveToFolders may throw an IllegalArgumentException.
> There is a line: message_id = Rn.nextInt(count - 1);
> if count is 1 the argument to nextInt() might be 0 which is not allowed. I hit this once but lost the stack trace, but it is apparent that when there is only one row in the table this can occur.
>
> 2) Allow/implement multiple attachments per message and cleanup DbTasks.insertMail() logic.
> - Remove the attach_id column from INBOX to allow multiple attachments.
> -Make the attachment insert part of the message for loop in insertMail.
> Use getGeneratedKeys() to get the id of the inserted message.
> When attachments are inserted, insert (1-4) attachments and give them a corresponding attach_id from 1-4.
> This will allow for removal of the select statements used to determine id and attach_id. I'll file another issue for these improvements if folks agree that they are sensible.
> A detailed description of the current implementation of insertMail is described at https://issues.apache.org/jira/secure/attachment/12405685/insertMailSummary.txt
> 3) DbTasks.databaseSize calculation is wrong. It doesn't match du -sk. The method does not recurse into subdirectories and includes the length() on directory files which is undefined accourding to the file.length() javadoc.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (DERBY-4166) improvements to the mailjdbc test
Posted by "Kathey Marsden (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/DERBY-4166?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12735310#action_12735310 ]
Kathey Marsden commented on DERBY-4166:
---------------------------------------
Lily in the schemachange2 patch I still see the attachments in a separate loop from the mail and we still have the SELECT ID from REFRESH.INBOX which may be causing the deadlock. Did getGeneratedKeys() not work to get the id if you keep the attachment insert in the loop? I think if we do that and commit after completing the full insert of each mail into INBOX and ATTACH we shouldn't see timeouts or deadlocks.
> improvements to the mailjdbc test
> ---------------------------------
>
> Key: DERBY-4166
> URL: https://issues.apache.org/jira/browse/DERBY-4166
> Project: Derby
> Issue Type: Improvement
> Components: Test
> Affects Versions: 10.6.0.0
> Reporter: Kathey Marsden
> Priority: Minor
> Attachments: DERBY-4166-databasesize.diff, Derby-4166-samedb.diff, DERBY-4166-schemachange.diff, DERBY-4166-schemachange2.diff, Derby-4166.diff
>
>
> When recently working with the mailjdbc system test org.apache.derbyTesting.system.mailjdbc on DERBY-4152 I noticed some potential improvements that might be good for the test. We should probably hold off on these improvements however until the root cause of DERBY-4152 is established, however, so we don't muddy the waters with that issue by changing the test.
> 1) DbTasks.moveToFolders may throw an IllegalArgumentException.
> There is a line: message_id = Rn.nextInt(count - 1);
> if count is 1 the argument to nextInt() might be 0 which is not allowed. I hit this once but lost the stack trace, but it is apparent that when there is only one row in the table this can occur.
>
> 2) Allow/implement multiple attachments per message and cleanup DbTasks.insertMail() logic.
> - Remove the attach_id column from INBOX to allow multiple attachments.
> -Make the attachment insert part of the message for loop in insertMail.
> Use getGeneratedKeys() to get the id of the inserted message.
> When attachments are inserted, insert (1-4) attachments and give them a corresponding attach_id from 1-4.
> This will allow for removal of the select statements used to determine id and attach_id. I'll file another issue for these improvements if folks agree that they are sensible.
> A detailed description of the current implementation of insertMail is described at https://issues.apache.org/jira/secure/attachment/12405685/insertMailSummary.txt
> 3) DbTasks.databaseSize calculation is wrong. It doesn't match du -sk. The method does not recurse into subdirectories and includes the length() on directory files which is undefined accourding to the file.length() javadoc.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (DERBY-4166) improvements to the mailjdbc test
Posted by "Lily Wei (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/DERBY-4166?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Lily Wei updated DERBY-4166:
----------------------------
Attachment: Derby-4166-samedb.diff
Thanks Kathey for commit the patch. This is the part of patch that change to allow resumption with the same database. This patch also relate to Derby-4203. The new syntax is in the README.txt. The diff is from directory 10.5<directory>/org/apache/derbyTesting/system/mailjdbc.
> improvements to the mailjdbc test
> ---------------------------------
>
> Key: DERBY-4166
> URL: https://issues.apache.org/jira/browse/DERBY-4166
> Project: Derby
> Issue Type: Improvement
> Components: Test
> Affects Versions: 10.6.0.0
> Reporter: Kathey Marsden
> Priority: Minor
> Attachments: Derby-4166-samedb.diff, Derby-4166.diff
>
>
> When recently working with the mailjdbc system test org.apache.derbyTesting.system.mailjdbc on DERBY-4152 I noticed some potential improvements that might be good for the test. We should probably hold off on these improvements however until the root cause of DERBY-4152 is established, however, so we don't muddy the waters with that issue by changing the test.
> 1) DbTasks.moveToFolders may throw an IllegalArgumentException.
> There is a line: message_id = Rn.nextInt(count - 1);
> if count is 1 the argument to nextInt() might be 0 which is not allowed. I hit this once but lost the stack trace, but it is apparent that when there is only one row in the table this can occur.
>
> 2) Allow/implement multiple attachments per message and cleanup DbTasks.insertMail() logic.
> - Remove the attach_id column from INBOX to allow multiple attachments.
> -Make the attachment insert part of the message for loop in insertMail.
> Use getGeneratedKeys() to get the id of the inserted message.
> When attachments are inserted, insert (1-4) attachments and give them a corresponding attach_id from 1-4.
> This will allow for removal of the select statements used to determine id and attach_id. I'll file another issue for these improvements if folks agree that they are sensible.
> A detailed description of the current implementation of insertMail is described at https://issues.apache.org/jira/secure/attachment/12405685/insertMailSummary.txt
> 3) DbTasks.databaseSize calculation is wrong. It doesn't match du -sk. The method does not recurse into subdirectories and includes the length() on directory files which is undefined accourding to the file.length() javadoc.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Closed: (DERBY-4166) improvements to the mailjdbc test
Posted by "Lily Wei (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/DERBY-4166?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Lily Wei closed DERBY-4166.
---------------------------
Issue & fix info: [Newcomer, Patch Available] (was: [Newcomer])
Fix Version/s: 10.6.1.0
Resolution: Fixed
The test is good improvement from before. I am closing this bug and tracking down the issue to reduce the timeouts and eliminate the deadlocks. Also, track down why the compress is needed to reclaim space used by the test.
> improvements to the mailjdbc test
> ---------------------------------
>
> Key: DERBY-4166
> URL: https://issues.apache.org/jira/browse/DERBY-4166
> Project: Derby
> Issue Type: Improvement
> Components: Test
> Affects Versions: 10.6.1.0
> Reporter: Kathey Marsden
> Assignee: Lily Wei
> Priority: Minor
> Fix For: 10.6.1.0
>
> Attachments: DERBY-4166-databasesize.diff, Derby-4166-samedb.diff, DERBY-4166-schemachange.diff, DERBY-4166-schemachange2.diff, DERBY-4166-schemachange3.diff, DERBY-4166-schemachange3.diff, DERBY-4166-schemachange4.diff, DERBY-4166-schemachange5.diff, DERBY-4166-schemachange6.diff, DERBY-4166-schemachange7.diff, Derby-4166.diff, derby.log
>
>
> When recently working with the mailjdbc system test org.apache.derbyTesting.system.mailjdbc on DERBY-4152 I noticed some potential improvements that might be good for the test. We should probably hold off on these improvements however until the root cause of DERBY-4152 is established, however, so we don't muddy the waters with that issue by changing the test.
> 1) DbTasks.moveToFolders may throw an IllegalArgumentException.
> There is a line: message_id = Rn.nextInt(count - 1);
> if count is 1 the argument to nextInt() might be 0 which is not allowed. I hit this once but lost the stack trace, but it is apparent that when there is only one row in the table this can occur.
>
> 2) Allow/implement multiple attachments per message and cleanup DbTasks.insertMail() logic.
> - Remove the attach_id column from INBOX to allow multiple attachments.
> -Make the attachment insert part of the message for loop in insertMail.
> Use getGeneratedKeys() to get the id of the inserted message.
> When attachments are inserted, insert (1-4) attachments and give them a corresponding attach_id from 1-4.
> This will allow for removal of the select statements used to determine id and attach_id. I'll file another issue for these improvements if folks agree that they are sensible.
> A detailed description of the current implementation of insertMail is described at https://issues.apache.org/jira/secure/attachment/12405685/insertMailSummary.txt
> 3) DbTasks.databaseSize calculation is wrong. It doesn't match du -sk. The method does not recurse into subdirectories and includes the length() on directory files which is undefined accourding to the file.length() javadoc.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (DERBY-4166) improvements to the mailjdbc test
Posted by "Kathey Marsden (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/DERBY-4166?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Kathey Marsden updated DERBY-4166:
----------------------------------
Comment: was deleted
(was: My trunk run with the small change to eliminate the pk contraint violations has been running more than 24 hours and I have seen no significant growth with my database size ranging from 32MB - 80MB and the size going up and down as it runs.
I've attached the test diff for reference (derby-4152_quick_change_to_stop_pk_violation_diff.txt). This is not for commit, but just provides context for the test run.
So since it seems that the pk constraint violations are the only cause of growth in the test on trunk, and I guess it follows that the 10.5 run was either seeing more pk constraint violations or was growing more on the same event, although that was not apparent from my small test case.
Next I'll go back to the original behavior and change the test to record more information on the pk constraint violations, including size of the blob and space table printout and then compare the trunk and 10.4 runs.
)
> improvements to the mailjdbc test
> ---------------------------------
>
> Key: DERBY-4166
> URL: https://issues.apache.org/jira/browse/DERBY-4166
> Project: Derby
> Issue Type: Improvement
> Components: Test
> Affects Versions: 10.6.0.0
> Reporter: Kathey Marsden
> Priority: Minor
>
> When recently working with the mailjdbc system test org.apache.derbyTesting.system.mailjdbc on DERBY-4152 I noticed some potential improvements that might be good for the test. We should probably hold off on these improvements however until the root cause of DERBY-4152 is established, however, so we don't muddy the waters with that issue by changing the test.
> 1) DbTasks.moveToFolders may throw an IllegalArgumentException.
> There is a line: message_id = Rn.nextInt(count - 1);
> if count is 1 the argument to nextInt() might be 0 which is not allowed. I hit this once but lost the stack trace, but it is apparent that when there is only one row in the table this can occur.
>
> 2) Allow/implement multiple attachments per message and cleanup DbTasks.insertMail() logic.
> - Remove the attach_id column from INBOX to allow multiple attachments.
> -Make the attachment insert part of the message for loop in insertMail.
> Use getGeneratedKeys() to get the id of the inserted message.
> When attachments are inserted, insert (1-4) attachments and give them a corresponding attach_id from 1-4.
> This will allow for removal of the select statements used to determine id and attach_id. I'll file another issue for these improvements if folks agree that they are sensible.
> A detailed description of the current implementation of insertMail is described at https://issues.apache.org/jira/secure/attachment/12405685/insertMailSummary.txt
> 3) DbTasks.databaseSize calculation is wrong. It doesn't match du -sk. The method does not recurse into subdirectories and includes the length() on directory files which is undefined accourding to the file.length() javadoc.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (DERBY-4166) improvements to the mailjdbc test
Posted by "Lily Wei (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/DERBY-4166?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Lily Wei updated DERBY-4166:
----------------------------
Attachment: DERBY-4166-schemachange7.diff
Compare to the previous patch. There is a issue on movetoFolder method on DbTasks.java.
Where we can get 0 from select count(*) from REFRESH.INBOX and cause an Exception. This new patch fix that minor issue. The patch is ready for review.
> improvements to the mailjdbc test
> ---------------------------------
>
> Key: DERBY-4166
> URL: https://issues.apache.org/jira/browse/DERBY-4166
> Project: Derby
> Issue Type: Improvement
> Components: Test
> Affects Versions: 10.6.0.0
> Reporter: Kathey Marsden
> Priority: Minor
> Attachments: DERBY-4166-databasesize.diff, Derby-4166-samedb.diff, DERBY-4166-schemachange.diff, DERBY-4166-schemachange2.diff, DERBY-4166-schemachange3.diff, DERBY-4166-schemachange3.diff, DERBY-4166-schemachange4.diff, DERBY-4166-schemachange5.diff, DERBY-4166-schemachange6.diff, DERBY-4166-schemachange7.diff, Derby-4166.diff, derby.log
>
>
> When recently working with the mailjdbc system test org.apache.derbyTesting.system.mailjdbc on DERBY-4152 I noticed some potential improvements that might be good for the test. We should probably hold off on these improvements however until the root cause of DERBY-4152 is established, however, so we don't muddy the waters with that issue by changing the test.
> 1) DbTasks.moveToFolders may throw an IllegalArgumentException.
> There is a line: message_id = Rn.nextInt(count - 1);
> if count is 1 the argument to nextInt() might be 0 which is not allowed. I hit this once but lost the stack trace, but it is apparent that when there is only one row in the table this can occur.
>
> 2) Allow/implement multiple attachments per message and cleanup DbTasks.insertMail() logic.
> - Remove the attach_id column from INBOX to allow multiple attachments.
> -Make the attachment insert part of the message for loop in insertMail.
> Use getGeneratedKeys() to get the id of the inserted message.
> When attachments are inserted, insert (1-4) attachments and give them a corresponding attach_id from 1-4.
> This will allow for removal of the select statements used to determine id and attach_id. I'll file another issue for these improvements if folks agree that they are sensible.
> A detailed description of the current implementation of insertMail is described at https://issues.apache.org/jira/secure/attachment/12405685/insertMailSummary.txt
> 3) DbTasks.databaseSize calculation is wrong. It doesn't match du -sk. The method does not recurse into subdirectories and includes the length() on directory files which is undefined accourding to the file.length() javadoc.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (DERBY-4166) improvements to the mailjdbc test
Posted by "Lily Wei (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/DERBY-4166?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Lily Wei updated DERBY-4166:
----------------------------
Attachment: DERBY-4166-schemachange5.diff
DERBY-4166-schemachange3.diff was a used name. Attach DERBY-4166-schemachange5.diff for the version that does not have deadlock and it can run more than 14 hours.
> improvements to the mailjdbc test
> ---------------------------------
>
> Key: DERBY-4166
> URL: https://issues.apache.org/jira/browse/DERBY-4166
> Project: Derby
> Issue Type: Improvement
> Components: Test
> Affects Versions: 10.6.0.0
> Reporter: Kathey Marsden
> Priority: Minor
> Attachments: DERBY-4166-databasesize.diff, Derby-4166-samedb.diff, DERBY-4166-schemachange.diff, DERBY-4166-schemachange2.diff, DERBY-4166-schemachange3.diff, DERBY-4166-schemachange3.diff, DERBY-4166-schemachange4.diff, DERBY-4166-schemachange5.diff, Derby-4166.diff, derby.log
>
>
> When recently working with the mailjdbc system test org.apache.derbyTesting.system.mailjdbc on DERBY-4152 I noticed some potential improvements that might be good for the test. We should probably hold off on these improvements however until the root cause of DERBY-4152 is established, however, so we don't muddy the waters with that issue by changing the test.
> 1) DbTasks.moveToFolders may throw an IllegalArgumentException.
> There is a line: message_id = Rn.nextInt(count - 1);
> if count is 1 the argument to nextInt() might be 0 which is not allowed. I hit this once but lost the stack trace, but it is apparent that when there is only one row in the table this can occur.
>
> 2) Allow/implement multiple attachments per message and cleanup DbTasks.insertMail() logic.
> - Remove the attach_id column from INBOX to allow multiple attachments.
> -Make the attachment insert part of the message for loop in insertMail.
> Use getGeneratedKeys() to get the id of the inserted message.
> When attachments are inserted, insert (1-4) attachments and give them a corresponding attach_id from 1-4.
> This will allow for removal of the select statements used to determine id and attach_id. I'll file another issue for these improvements if folks agree that they are sensible.
> A detailed description of the current implementation of insertMail is described at https://issues.apache.org/jira/secure/attachment/12405685/insertMailSummary.txt
> 3) DbTasks.databaseSize calculation is wrong. It doesn't match du -sk. The method does not recurse into subdirectories and includes the length() on directory files which is undefined accourding to the file.length() javadoc.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (DERBY-4166) improvements to the mailjdbc test
Posted by "Kathey Marsden (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/DERBY-4166?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12732341#action_12732341 ]
Kathey Marsden commented on DERBY-4166:
---------------------------------------
I committed the part of Lily's patch that changes the backup user to the database owner to work around DERBY-4295. I think since we are changing a system test we should break up the patch into as small as possible increments so it will be easy to track any behavior changes. Next could be the change to allow resumption with the same database, then the database size method, then fixing the insert logic to avoid primary key exception, then adding delete cascade for the attachments. I am sure some changes will need to be bundled but as much as possible we should break them up. I don't mind multiple patches.
Thanks
Kathey
> improvements to the mailjdbc test
> ---------------------------------
>
> Key: DERBY-4166
> URL: https://issues.apache.org/jira/browse/DERBY-4166
> Project: Derby
> Issue Type: Improvement
> Components: Test
> Affects Versions: 10.6.0.0
> Reporter: Kathey Marsden
> Priority: Minor
> Attachments: Derby-4166.diff
>
>
> When recently working with the mailjdbc system test org.apache.derbyTesting.system.mailjdbc on DERBY-4152 I noticed some potential improvements that might be good for the test. We should probably hold off on these improvements however until the root cause of DERBY-4152 is established, however, so we don't muddy the waters with that issue by changing the test.
> 1) DbTasks.moveToFolders may throw an IllegalArgumentException.
> There is a line: message_id = Rn.nextInt(count - 1);
> if count is 1 the argument to nextInt() might be 0 which is not allowed. I hit this once but lost the stack trace, but it is apparent that when there is only one row in the table this can occur.
>
> 2) Allow/implement multiple attachments per message and cleanup DbTasks.insertMail() logic.
> - Remove the attach_id column from INBOX to allow multiple attachments.
> -Make the attachment insert part of the message for loop in insertMail.
> Use getGeneratedKeys() to get the id of the inserted message.
> When attachments are inserted, insert (1-4) attachments and give them a corresponding attach_id from 1-4.
> This will allow for removal of the select statements used to determine id and attach_id. I'll file another issue for these improvements if folks agree that they are sensible.
> A detailed description of the current implementation of insertMail is described at https://issues.apache.org/jira/secure/attachment/12405685/insertMailSummary.txt
> 3) DbTasks.databaseSize calculation is wrong. It doesn't match du -sk. The method does not recurse into subdirectories and includes the length() on directory files which is undefined accourding to the file.length() javadoc.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.