You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@avro.apache.org by "Thiruvalluvan M. G. (JIRA)" <ji...@apache.org> on 2010/04/24 04:33:49 UTC
[jira] Created: (AVRO-524) DataFileWriter.appendTo leads to
intermittent IOException during write()
DataFileWriter.appendTo leads to intermittent IOException during write()
------------------------------------------------------------------------
Key: AVRO-524
URL: https://issues.apache.org/jira/browse/AVRO-524
Project: Avro
Issue Type: Bug
Components: java
Reporter: Thiruvalluvan M. G.
Assignee: Thiruvalluvan M. G.
Fix For: 1.4.0
To append to a data file, we first open the file as RandomAccessFile in read-write mode, read some information such sync, seek to the end of the file and then use its FileDescriptor to create a FileOutputStream. Sharing a FileDescriptor this way could lead to problem if one of its containers is garbage-collected while the other is still in use. Please see: http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6322678. The bug got fixed in Java 7 (b06). In our case, RandomAccessFile sometimes gets garbage-collected leading to write errors. If the Java unit-tests are run multiple times, this occurs about 25% of the time on my Windows machine and about 50% on my Ubuntu Linux box.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (AVRO-524) DataFileWriter.appendTo leads to
intermittent IOException during write()
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-524?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12861987#action_12861987 ]
Doug Cutting commented on AVRO-524:
-----------------------------------
Do we even still need RandomAccessFile and a FileHandle at all anymore? These were only needed before because it's the only way I could find to get a single file descriptor that's open for both read and write. If we're willing to close and re-open the file then the header can be read with a FileInputStream, then the appends can be made to a FileOutputStream.
Note this is a minor, probably insignificant, change in semantics. If the file were to be renamed between the time its header reader is opened and the appender is opened then it would now fail, where before it would succeed. This seems very unlikely and not worth protecting against, but, for the record, it was fears of issues like this that led me to use RandomAccessFile, so that the file was only opened once, that read and write permission, file existence, etc. would all be only checked once. In general, re-opening files is risky, but, in this specific case, it probably isn't.
Finally, should we perhaps close the reader in a finally clause?
> DataFileWriter.appendTo leads to intermittent IOException during write()
> ------------------------------------------------------------------------
>
> Key: AVRO-524
> URL: https://issues.apache.org/jira/browse/AVRO-524
> Project: Avro
> Issue Type: Bug
> Components: java
> Reporter: Thiruvalluvan M. G.
> Assignee: Thiruvalluvan M. G.
> Fix For: 1.4.0
>
> Attachments: AVRO-524.patch
>
>
> To append to a data file, we first open the file as RandomAccessFile in read-write mode, read some information such sync, seek to the end of the file and then use its FileDescriptor to create a FileOutputStream. Sharing a FileDescriptor this way could lead to problem if one of its containers is garbage-collected while the other is still in use. Please see: http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6322678. The bug got fixed in Java 7 (b06). In our case, RandomAccessFile sometimes gets garbage-collected leading to write errors. If the Java unit-tests are run multiple times, this occurs about 25% of the time on my Windows machine and about 50% on my Ubuntu Linux box.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (AVRO-524) DataFileWriter.appendTo leads to
intermittent IOException during write()
Posted by "Thiruvalluvan M. G. (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-524?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Thiruvalluvan M. G. updated AVRO-524:
-------------------------------------
Attachment: AVRO-524.patch
> DataFileWriter.appendTo leads to intermittent IOException during write()
> ------------------------------------------------------------------------
>
> Key: AVRO-524
> URL: https://issues.apache.org/jira/browse/AVRO-524
> Project: Avro
> Issue Type: Bug
> Components: java
> Reporter: Thiruvalluvan M. G.
> Assignee: Thiruvalluvan M. G.
> Fix For: 1.4.0
>
> Attachments: AVRO-524.patch, AVRO-524.patch
>
>
> To append to a data file, we first open the file as RandomAccessFile in read-write mode, read some information such sync, seek to the end of the file and then use its FileDescriptor to create a FileOutputStream. Sharing a FileDescriptor this way could lead to problem if one of its containers is garbage-collected while the other is still in use. Please see: http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6322678. The bug got fixed in Java 7 (b06). In our case, RandomAccessFile sometimes gets garbage-collected leading to write errors. If the Java unit-tests are run multiple times, this occurs about 25% of the time on my Windows machine and about 50% on my Ubuntu Linux box.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (AVRO-524) DataFileWriter.appendTo leads to
intermittent IOException during write()
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-524?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12861988#action_12861988 ]
Doug Cutting commented on AVRO-524:
-----------------------------------
Also, for the record, another way to fix the bug would be to simply retain a pointer to the RandomAccessFile, no?
> DataFileWriter.appendTo leads to intermittent IOException during write()
> ------------------------------------------------------------------------
>
> Key: AVRO-524
> URL: https://issues.apache.org/jira/browse/AVRO-524
> Project: Avro
> Issue Type: Bug
> Components: java
> Reporter: Thiruvalluvan M. G.
> Assignee: Thiruvalluvan M. G.
> Fix For: 1.4.0
>
> Attachments: AVRO-524.patch
>
>
> To append to a data file, we first open the file as RandomAccessFile in read-write mode, read some information such sync, seek to the end of the file and then use its FileDescriptor to create a FileOutputStream. Sharing a FileDescriptor this way could lead to problem if one of its containers is garbage-collected while the other is still in use. Please see: http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6322678. The bug got fixed in Java 7 (b06). In our case, RandomAccessFile sometimes gets garbage-collected leading to write errors. If the Java unit-tests are run multiple times, this occurs about 25% of the time on my Windows machine and about 50% on my Ubuntu Linux box.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (AVRO-524) DataFileWriter.appendTo leads to
intermittent IOException during write()
Posted by "Thiruvalluvan M. G. (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-524?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Thiruvalluvan M. G. updated AVRO-524:
-------------------------------------
Status: Patch Available (was: Open)
> DataFileWriter.appendTo leads to intermittent IOException during write()
> ------------------------------------------------------------------------
>
> Key: AVRO-524
> URL: https://issues.apache.org/jira/browse/AVRO-524
> Project: Avro
> Issue Type: Bug
> Components: java
> Reporter: Thiruvalluvan M. G.
> Assignee: Thiruvalluvan M. G.
> Fix For: 1.4.0
>
> Attachments: AVRO-524.patch
>
>
> To append to a data file, we first open the file as RandomAccessFile in read-write mode, read some information such sync, seek to the end of the file and then use its FileDescriptor to create a FileOutputStream. Sharing a FileDescriptor this way could lead to problem if one of its containers is garbage-collected while the other is still in use. Please see: http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6322678. The bug got fixed in Java 7 (b06). In our case, RandomAccessFile sometimes gets garbage-collected leading to write errors. If the Java unit-tests are run multiple times, this occurs about 25% of the time on my Windows machine and about 50% on my Ubuntu Linux box.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (AVRO-524) DataFileWriter.appendTo leads to
intermittent IOException during write()
Posted by "Thiruvalluvan M. G. (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-524?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Thiruvalluvan M. G. updated AVRO-524:
-------------------------------------
Attachment: AVRO-524.patch
The simple solution is to open the FileOutputStream in append mode from the filename rather than the FileDescriptor. Since its FileDescriptor is not going to be used for writing anymore, the RandomAccessFile can now be opened in "r" mode instead of "rw".
> DataFileWriter.appendTo leads to intermittent IOException during write()
> ------------------------------------------------------------------------
>
> Key: AVRO-524
> URL: https://issues.apache.org/jira/browse/AVRO-524
> Project: Avro
> Issue Type: Bug
> Components: java
> Reporter: Thiruvalluvan M. G.
> Assignee: Thiruvalluvan M. G.
> Fix For: 1.4.0
>
> Attachments: AVRO-524.patch
>
>
> To append to a data file, we first open the file as RandomAccessFile in read-write mode, read some information such sync, seek to the end of the file and then use its FileDescriptor to create a FileOutputStream. Sharing a FileDescriptor this way could lead to problem if one of its containers is garbage-collected while the other is still in use. Please see: http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6322678. The bug got fixed in Java 7 (b06). In our case, RandomAccessFile sometimes gets garbage-collected leading to write errors. If the Java unit-tests are run multiple times, this occurs about 25% of the time on my Windows machine and about 50% on my Ubuntu Linux box.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (AVRO-524) DataFileWriter.appendTo leads to
intermittent IOException during write()
Posted by "Thiruvalluvan M. G. (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-524?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Thiruvalluvan M. G. updated AVRO-524:
-------------------------------------
Status: Resolved (was: Patch Available)
Resolution: Fixed
Committed revision 938347.
> DataFileWriter.appendTo leads to intermittent IOException during write()
> ------------------------------------------------------------------------
>
> Key: AVRO-524
> URL: https://issues.apache.org/jira/browse/AVRO-524
> Project: Avro
> Issue Type: Bug
> Components: java
> Reporter: Thiruvalluvan M. G.
> Assignee: Thiruvalluvan M. G.
> Fix For: 1.4.0
>
> Attachments: AVRO-524.patch
>
>
> To append to a data file, we first open the file as RandomAccessFile in read-write mode, read some information such sync, seek to the end of the file and then use its FileDescriptor to create a FileOutputStream. Sharing a FileDescriptor this way could lead to problem if one of its containers is garbage-collected while the other is still in use. Please see: http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6322678. The bug got fixed in Java 7 (b06). In our case, RandomAccessFile sometimes gets garbage-collected leading to write errors. If the Java unit-tests are run multiple times, this occurs about 25% of the time on my Windows machine and about 50% on my Ubuntu Linux box.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (AVRO-524) DataFileWriter.appendTo leads to
intermittent IOException during write()
Posted by "Thiruvalluvan M. G. (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-524?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Thiruvalluvan M. G. updated AVRO-524:
-------------------------------------
Attachment: (was: AVRO-524.patch)
> DataFileWriter.appendTo leads to intermittent IOException during write()
> ------------------------------------------------------------------------
>
> Key: AVRO-524
> URL: https://issues.apache.org/jira/browse/AVRO-524
> Project: Avro
> Issue Type: Bug
> Components: java
> Reporter: Thiruvalluvan M. G.
> Assignee: Thiruvalluvan M. G.
> Fix For: 1.4.0
>
> Attachments: AVRO-524.patch
>
>
> To append to a data file, we first open the file as RandomAccessFile in read-write mode, read some information such sync, seek to the end of the file and then use its FileDescriptor to create a FileOutputStream. Sharing a FileDescriptor this way could lead to problem if one of its containers is garbage-collected while the other is still in use. Please see: http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6322678. The bug got fixed in Java 7 (b06). In our case, RandomAccessFile sometimes gets garbage-collected leading to write errors. If the Java unit-tests are run multiple times, this occurs about 25% of the time on my Windows machine and about 50% on my Ubuntu Linux box.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.