You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "Owen O'Malley (JIRA)" <ji...@apache.org> on 2007/10/11 18:06:50 UTC
[jira] Created: (HADOOP-2033) In SequenceFile sync doesn't work
unless the file is compressed (block or record)
In SequenceFile sync doesn't work unless the file is compressed (block or record)
---------------------------------------------------------------------------------
Key: HADOOP-2033
URL: https://issues.apache.org/jira/browse/HADOOP-2033
Project: Hadoop
Issue Type: Bug
Reporter: Owen O'Malley
Assignee: Owen O'Malley
Fix For: 0.15.0
DistCp counts on the fact that SequenceFile sync actually puts a sync marker in and it currently doesn't unless the file is being compressed.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2033) In SequenceFile sync doesn't work
unless the file is compressed (block or record)
Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12535666 ]
Arun C Murthy commented on HADOOP-2033:
---------------------------------------
+1
> In SequenceFile sync doesn't work unless the file is compressed (block or record)
> ---------------------------------------------------------------------------------
>
> Key: HADOOP-2033
> URL: https://issues.apache.org/jira/browse/HADOOP-2033
> Project: Hadoop
> Issue Type: Bug
> Reporter: Owen O'Malley
> Assignee: Owen O'Malley
> Priority: Blocker
> Fix For: 0.15.0
>
> Attachments: sync-bytes-3.patch, sync-bytes-4.patch, sync-bytes.patch
>
>
> DistCp counts on the fact that SequenceFile sync actually puts a sync marker in and it currently doesn't unless the file is being compressed.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2033) In SequenceFile sync doesn't work
unless the file is compressed (block or record)
Posted by "Owen O'Malley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2033?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Owen O'Malley updated HADOOP-2033:
----------------------------------
Attachment: sync-bytes.patch
This patch makes the sync method actually write sync bytes instead of nothing. It also changes the reader so that it can handle sync bytes at the end of the file.
> In SequenceFile sync doesn't work unless the file is compressed (block or record)
> ---------------------------------------------------------------------------------
>
> Key: HADOOP-2033
> URL: https://issues.apache.org/jira/browse/HADOOP-2033
> Project: Hadoop
> Issue Type: Bug
> Reporter: Owen O'Malley
> Assignee: Owen O'Malley
> Priority: Blocker
> Fix For: 0.15.0
>
> Attachments: sync-bytes.patch
>
>
> DistCp counts on the fact that SequenceFile sync actually puts a sync marker in and it currently doesn't unless the file is being compressed.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2033) In SequenceFile sync doesn't work
unless the file is compressed (block or record)
Posted by "Owen O'Malley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2033?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Owen O'Malley updated HADOOP-2033:
----------------------------------
Resolution: Fixed
Status: Resolved (was: Patch Available)
I just committed this.
> In SequenceFile sync doesn't work unless the file is compressed (block or record)
> ---------------------------------------------------------------------------------
>
> Key: HADOOP-2033
> URL: https://issues.apache.org/jira/browse/HADOOP-2033
> Project: Hadoop
> Issue Type: Bug
> Reporter: Owen O'Malley
> Assignee: Owen O'Malley
> Priority: Blocker
> Fix For: 0.15.0
>
> Attachments: sync-bytes-3.patch, sync-bytes-4.patch, sync-bytes.patch
>
>
> DistCp counts on the fact that SequenceFile sync actually puts a sync marker in and it currently doesn't unless the file is being compressed.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2033) In SequenceFile sync doesn't work
unless the file is compressed (block or record)
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12535189 ]
Hadoop QA commented on HADOOP-2033:
-----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12367798/sync-bytes.patch
against trunk revision r584939.
@author +1. The patch does not contain any @author tags.
javadoc +1. The javadoc tool did not generate any warning messages.
javac +1. The applied patch does not generate any new compiler warnings.
findbugs +1. The patch does not introduce any new Findbugs warnings.
core tests -1. The patch failed core unit tests.
contrib tests -1. The patch failed contrib unit tests.
Test results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/953/testReport/
Findbugs warnings: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/953/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/953/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/953/console
This message is automatically generated.
> In SequenceFile sync doesn't work unless the file is compressed (block or record)
> ---------------------------------------------------------------------------------
>
> Key: HADOOP-2033
> URL: https://issues.apache.org/jira/browse/HADOOP-2033
> Project: Hadoop
> Issue Type: Bug
> Reporter: Owen O'Malley
> Assignee: Owen O'Malley
> Priority: Blocker
> Fix For: 0.15.0
>
> Attachments: sync-bytes.patch
>
>
> DistCp counts on the fact that SequenceFile sync actually puts a sync marker in and it currently doesn't unless the file is being compressed.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2033) In SequenceFile sync doesn't work
unless the file is compressed (block or record)
Posted by "Owen O'Malley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2033?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Owen O'Malley updated HADOOP-2033:
----------------------------------
Status: Patch Available (was: Open)
> In SequenceFile sync doesn't work unless the file is compressed (block or record)
> ---------------------------------------------------------------------------------
>
> Key: HADOOP-2033
> URL: https://issues.apache.org/jira/browse/HADOOP-2033
> Project: Hadoop
> Issue Type: Bug
> Reporter: Owen O'Malley
> Assignee: Owen O'Malley
> Priority: Blocker
> Fix For: 0.15.0
>
> Attachments: sync-bytes-3.patch, sync-bytes.patch
>
>
> DistCp counts on the fact that SequenceFile sync actually puts a sync marker in and it currently doesn't unless the file is being compressed.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2033) In SequenceFile sync doesn't work
unless the file is compressed (block or record)
Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12535667 ]
Arun C Murthy commented on HADOOP-2033:
---------------------------------------
Should we target this for 0.14.3 too?
> In SequenceFile sync doesn't work unless the file is compressed (block or record)
> ---------------------------------------------------------------------------------
>
> Key: HADOOP-2033
> URL: https://issues.apache.org/jira/browse/HADOOP-2033
> Project: Hadoop
> Issue Type: Bug
> Reporter: Owen O'Malley
> Assignee: Owen O'Malley
> Priority: Blocker
> Fix For: 0.15.0
>
> Attachments: sync-bytes-3.patch, sync-bytes-4.patch, sync-bytes.patch
>
>
> DistCp counts on the fact that SequenceFile sync actually puts a sync marker in and it currently doesn't unless the file is being compressed.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2033) In SequenceFile sync doesn't work
unless the file is compressed (block or record)
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12535901 ]
Hudson commented on HADOOP-2033:
--------------------------------
Integrated in Hadoop-Nightly #276 (See [http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Nightly/276/])
> In SequenceFile sync doesn't work unless the file is compressed (block or record)
> ---------------------------------------------------------------------------------
>
> Key: HADOOP-2033
> URL: https://issues.apache.org/jira/browse/HADOOP-2033
> Project: Hadoop
> Issue Type: Bug
> Reporter: Owen O'Malley
> Assignee: Owen O'Malley
> Priority: Blocker
> Fix For: 0.15.0
>
> Attachments: sync-bytes-3.patch, sync-bytes-4.patch, sync-bytes.patch
>
>
> DistCp counts on the fact that SequenceFile sync actually puts a sync marker in and it currently doesn't unless the file is being compressed.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2033) In SequenceFile sync doesn't work
unless the file is compressed (block or record)
Posted by "Chris Douglas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12535686 ]
Chris Douglas commented on HADOOP-2033:
---------------------------------------
bq. Should we target this for 0.14.3 too?
Probably not; it's not a regression and it doesn't affect the version of distcp running in 0.14.
> In SequenceFile sync doesn't work unless the file is compressed (block or record)
> ---------------------------------------------------------------------------------
>
> Key: HADOOP-2033
> URL: https://issues.apache.org/jira/browse/HADOOP-2033
> Project: Hadoop
> Issue Type: Bug
> Reporter: Owen O'Malley
> Assignee: Owen O'Malley
> Priority: Blocker
> Fix For: 0.15.0
>
> Attachments: sync-bytes-3.patch, sync-bytes-4.patch, sync-bytes.patch
>
>
> DistCp counts on the fact that SequenceFile sync actually puts a sync marker in and it currently doesn't unless the file is being compressed.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2033) In SequenceFile sync doesn't work
unless the file is compressed (block or record)
Posted by "Owen O'Malley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2033?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Owen O'Malley updated HADOOP-2033:
----------------------------------
Attachment: sync-bytes-4.patch
Some minor cleanups of comments and a minor refactoring.
> In SequenceFile sync doesn't work unless the file is compressed (block or record)
> ---------------------------------------------------------------------------------
>
> Key: HADOOP-2033
> URL: https://issues.apache.org/jira/browse/HADOOP-2033
> Project: Hadoop
> Issue Type: Bug
> Reporter: Owen O'Malley
> Assignee: Owen O'Malley
> Priority: Blocker
> Fix For: 0.15.0
>
> Attachments: sync-bytes-3.patch, sync-bytes-4.patch, sync-bytes.patch
>
>
> DistCp counts on the fact that SequenceFile sync actually puts a sync marker in and it currently doesn't unless the file is being compressed.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2033) In SequenceFile sync doesn't work
unless the file is compressed (block or record)
Posted by "Owen O'Malley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2033?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Owen O'Malley updated HADOOP-2033:
----------------------------------
Priority: Blocker (was: Major)
> In SequenceFile sync doesn't work unless the file is compressed (block or record)
> ---------------------------------------------------------------------------------
>
> Key: HADOOP-2033
> URL: https://issues.apache.org/jira/browse/HADOOP-2033
> Project: Hadoop
> Issue Type: Bug
> Reporter: Owen O'Malley
> Assignee: Owen O'Malley
> Priority: Blocker
> Fix For: 0.15.0
>
>
> DistCp counts on the fact that SequenceFile sync actually puts a sync marker in and it currently doesn't unless the file is being compressed.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2033) In SequenceFile sync doesn't work
unless the file is compressed (block or record)
Posted by "Owen O'Malley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2033?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Owen O'Malley updated HADOOP-2033:
----------------------------------
Status: Patch Available (was: Open)
> In SequenceFile sync doesn't work unless the file is compressed (block or record)
> ---------------------------------------------------------------------------------
>
> Key: HADOOP-2033
> URL: https://issues.apache.org/jira/browse/HADOOP-2033
> Project: Hadoop
> Issue Type: Bug
> Reporter: Owen O'Malley
> Assignee: Owen O'Malley
> Priority: Blocker
> Fix For: 0.15.0
>
> Attachments: sync-bytes.patch
>
>
> DistCp counts on the fact that SequenceFile sync actually puts a sync marker in and it currently doesn't unless the file is being compressed.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2033) In SequenceFile sync doesn't work
unless the file is compressed (block or record)
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12535623 ]
Hadoop QA commented on HADOOP-2033:
-----------------------------------
+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12367881/sync-bytes-3.patch
against trunk revision r585409.
@author +1. The patch does not contain any @author tags.
javadoc +1. The javadoc tool did not generate any warning messages.
javac +1. The applied patch does not generate any new compiler warnings.
findbugs +1. The patch does not introduce any new Findbugs warnings.
core tests +1. The patch passed core unit tests.
contrib tests +1. The patch passed contrib unit tests.
Test results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/964/testReport/
Findbugs warnings: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/964/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/964/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/964/console
This message is automatically generated.
> In SequenceFile sync doesn't work unless the file is compressed (block or record)
> ---------------------------------------------------------------------------------
>
> Key: HADOOP-2033
> URL: https://issues.apache.org/jira/browse/HADOOP-2033
> Project: Hadoop
> Issue Type: Bug
> Reporter: Owen O'Malley
> Assignee: Owen O'Malley
> Priority: Blocker
> Fix For: 0.15.0
>
> Attachments: sync-bytes-3.patch, sync-bytes-4.patch, sync-bytes.patch
>
>
> DistCp counts on the fact that SequenceFile sync actually puts a sync marker in and it currently doesn't unless the file is being compressed.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2033) In SequenceFile sync doesn't work
unless the file is compressed (block or record)
Posted by "Owen O'Malley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2033?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Owen O'Malley updated HADOOP-2033:
----------------------------------
Attachment: sync-bytes-3.patch
I forgot to reset the lastSyncPos when we sync and also prohibit back to back sync marks.
> In SequenceFile sync doesn't work unless the file is compressed (block or record)
> ---------------------------------------------------------------------------------
>
> Key: HADOOP-2033
> URL: https://issues.apache.org/jira/browse/HADOOP-2033
> Project: Hadoop
> Issue Type: Bug
> Reporter: Owen O'Malley
> Assignee: Owen O'Malley
> Priority: Blocker
> Fix For: 0.15.0
>
> Attachments: sync-bytes-3.patch, sync-bytes.patch
>
>
> DistCp counts on the fact that SequenceFile sync actually puts a sync marker in and it currently doesn't unless the file is being compressed.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2033) In SequenceFile sync doesn't work
unless the file is compressed (block or record)
Posted by "Owen O'Malley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2033?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Owen O'Malley updated HADOOP-2033:
----------------------------------
Status: Open (was: Patch Available)
> In SequenceFile sync doesn't work unless the file is compressed (block or record)
> ---------------------------------------------------------------------------------
>
> Key: HADOOP-2033
> URL: https://issues.apache.org/jira/browse/HADOOP-2033
> Project: Hadoop
> Issue Type: Bug
> Reporter: Owen O'Malley
> Assignee: Owen O'Malley
> Priority: Blocker
> Fix For: 0.15.0
>
> Attachments: sync-bytes.patch
>
>
> DistCp counts on the fact that SequenceFile sync actually puts a sync marker in and it currently doesn't unless the file is being compressed.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.