You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "Raghu Angadi (JIRA)" <ji...@apache.org> on 2008/09/26 19:53:44 UTC
[jira] Created: (HADOOP-4292) append() does not work for
LocalFileSystem
append() does not work for LocalFileSystem
------------------------------------------
Key: HADOOP-4292
URL: https://issues.apache.org/jira/browse/HADOOP-4292
Project: Hadoop Core
Issue Type: Bug
Components: fs
Affects Versions: 0.19.0
Reporter: Raghu Angadi
Assignee: Tsz Wo (Nicholas), SZE
Priority: Blocker
Fix For: 0.19.0
append is supported by LocalFileSystem but it does not update crc when a file is appended.
When you enable checksum verification {{TestLocalFileSystem.testAppend}} fails. Since HADOOP-4277 is a blocker for 0.17 I am planning to disable this test in HADOOP-4277.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4292) append() does not work for
LocalFileSystem
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4292?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12637229#action_12637229 ]
dhruba borthakur commented on HADOOP-4292:
------------------------------------------
The "append" design (HADOOP-1700) allows for the fact that we coudl support multiple concurrent appenders in future. This means that handling the last partial chunk is best handled in the server and *not* in the client.
> append() does not work for LocalFileSystem
> ------------------------------------------
>
> Key: HADOOP-4292
> URL: https://issues.apache.org/jira/browse/HADOOP-4292
> Project: Hadoop Core
> Issue Type: Bug
> Components: fs
> Affects Versions: 0.18.0, 0.19.0
> Reporter: Raghu Angadi
> Assignee: Hairong Kuang
> Priority: Blocker
> Fix For: 0.18.2, 0.19.0
>
>
> append is supported by LocalFileSystem but it does not update crc when a file is appended.
> When you enable checksum verification {{TestLocalFileSystem.testAppend}} fails. Since HADOOP-4277 is a blocker for 0.17 I am planning to disable this test in HADOOP-4277.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4292) append() does not work for
LocalFileSystem
Posted by "Hairong Kuang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4292?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12637260#action_12637260 ]
Hairong Kuang commented on HADOOP-4292:
---------------------------------------
The reason that I did not propose adding truncate() to FileSystem.java is that I did not make dramatic change to FileSystem APIs. But if this is OK with everybody, it is fine with me too. :-) If truncate returns an FSDataOutput, next write does not need to open it for appending first.
> append() does not work for LocalFileSystem
> ------------------------------------------
>
> Key: HADOOP-4292
> URL: https://issues.apache.org/jira/browse/HADOOP-4292
> Project: Hadoop Core
> Issue Type: Bug
> Components: fs
> Affects Versions: 0.18.0, 0.19.0
> Reporter: Raghu Angadi
> Assignee: Hairong Kuang
> Priority: Blocker
> Fix For: 0.18.2, 0.19.0
>
>
> append is supported by LocalFileSystem but it does not update crc when a file is appended.
> When you enable checksum verification {{TestLocalFileSystem.testAppend}} fails. Since HADOOP-4277 is a blocker for 0.17 I am planning to disable this test in HADOOP-4277.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4292) append() does not work for
LocalFileSystem
Posted by "Raghu Angadi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4292?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12637245#action_12637245 ]
Raghu Angadi commented on HADOOP-4292:
--------------------------------------
> The "append" design (HADOOP-1700) allows for the fact that we coudl support multiple concurrent appenders in future. This means that handling the last partial chunk is best handled in the server and not in the client.
ChecksumFileSystem is managed only on the client. It won't work for multiple appenders, just like it does not work for multiple writers now. Note that DFSOutputStream is _not_ a ChecksumFileSystem. Not sure if this is related to HADOOP-1700 at all.
> append() does not work for LocalFileSystem
> ------------------------------------------
>
> Key: HADOOP-4292
> URL: https://issues.apache.org/jira/browse/HADOOP-4292
> Project: Hadoop Core
> Issue Type: Bug
> Components: fs
> Affects Versions: 0.18.0, 0.19.0
> Reporter: Raghu Angadi
> Assignee: Hairong Kuang
> Priority: Blocker
> Fix For: 0.18.2, 0.19.0
>
>
> append is supported by LocalFileSystem but it does not update crc when a file is appended.
> When you enable checksum verification {{TestLocalFileSystem.testAppend}} fails. Since HADOOP-4277 is a blocker for 0.17 I am planning to disable this test in HADOOP-4277.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
Re: [jira] Commented: (HADOOP-4292) append() does not work for
LocalFileSystem
Posted by Hairong Kuang <ha...@yahoo-inc.com>.
Raghu's comment makes sense. We can get around it by not truncating the data
file but only the crc file. The last 4 bytes need to be overwritten anyway.
On 10/6/08 11:25 AM, "Raghu Angadi (JIRA)" <ji...@apache.org> wrote:
>
> [
> https://issues.apache.org/jira/browse/HADOOP-4292?page=com.atlassian.jira.plug
> in.system.issuetabpanels:comment-tabpanel&focusedCommentId=12637168#action_126
> 37168 ]
>
> Raghu Angadi commented on HADOOP-4292:
> --------------------------------------
>
> truncate implies that if the first append() fails, user can actually lose data
> that existed in the file before append().
>
>> append() does not work for LocalFileSystem
>> ------------------------------------------
>>
>> Key: HADOOP-4292
>> URL: https://issues.apache.org/jira/browse/HADOOP-4292
>> Project: Hadoop Core
>> Issue Type: Bug
>> Components: fs
>> Affects Versions: 0.18.0, 0.19.0
>> Reporter: Raghu Angadi
>> Assignee: Hairong Kuang
>> Priority: Blocker
>> Fix For: 0.18.2, 0.19.0
>>
>>
>> append is supported by LocalFileSystem but it does not update crc when a file
>> is appended.
>> When you enable checksum verification {{TestLocalFileSystem.testAppend}}
>> fails. Since HADOOP-4277 is a blocker for 0.17 I am planning to disable this
>> test in HADOOP-4277.
[jira] Commented: (HADOOP-4292) append() does not work for
LocalFileSystem
Posted by "Raghu Angadi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4292?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12637168#action_12637168 ]
Raghu Angadi commented on HADOOP-4292:
--------------------------------------
truncate implies that if the first append() fails, user can actually lose data that existed in the file before append().
> append() does not work for LocalFileSystem
> ------------------------------------------
>
> Key: HADOOP-4292
> URL: https://issues.apache.org/jira/browse/HADOOP-4292
> Project: Hadoop Core
> Issue Type: Bug
> Components: fs
> Affects Versions: 0.18.0, 0.19.0
> Reporter: Raghu Angadi
> Assignee: Hairong Kuang
> Priority: Blocker
> Fix For: 0.18.2, 0.19.0
>
>
> append is supported by LocalFileSystem but it does not update crc when a file is appended.
> When you enable checksum verification {{TestLocalFileSystem.testAppend}} fails. Since HADOOP-4277 is a blocker for 0.17 I am planning to disable this test in HADOOP-4277.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-4292) append() does not work for
LocalFileSystem
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4292?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tsz Wo (Nicholas), SZE updated HADOOP-4292:
-------------------------------------------
Hadoop Flags: [Reviewed]
+1 patch looks good.
> append() does not work for LocalFileSystem
> ------------------------------------------
>
> Key: HADOOP-4292
> URL: https://issues.apache.org/jira/browse/HADOOP-4292
> Project: Hadoop Core
> Issue Type: Bug
> Components: fs
> Affects Versions: 0.18.0, 0.19.0
> Reporter: Raghu Angadi
> Assignee: Hairong Kuang
> Priority: Blocker
> Fix For: 0.18.2, 0.19.0
>
> Attachments: LFSAppend.patch
>
>
> append is supported by LocalFileSystem but it does not update crc when a file is appended.
> When you enable checksum verification {{TestLocalFileSystem.testAppend}} fails. Since HADOOP-4277 is a blocker for 0.17 I am planning to disable this test in HADOOP-4277.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4292) append() does not work for
LocalFileSystem
Posted by "Hairong Kuang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4292?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12636962#action_12636962 ]
Hairong Kuang commented on HADOOP-4292:
---------------------------------------
The complexity of append exists when the last checksum chunk is partial. In this case, I plan to first read the partial chunk out, then truncate the data file to exclude the last chunk and truncate the crc file to exclude the last 4 bytes. Afterwards write the partial chunk again and data to be append.
Since LocalFileSystem is a ChecksumFileSystem, I plan to implement the above algorithm in ChecksumFileSystem so it is general to any checksum file system. For any checksum file system to support append, its raw file system needs to implement a TruncatableFileSystem interface that supports a truncate method:
FSDataOutputStream truncate(Path file, long len);
This method sets the give file's length be the given len and returns a FSDataOutputStream, whose current position is equal to the given len.
> append() does not work for LocalFileSystem
> ------------------------------------------
>
> Key: HADOOP-4292
> URL: https://issues.apache.org/jira/browse/HADOOP-4292
> Project: Hadoop Core
> Issue Type: Bug
> Components: fs
> Affects Versions: 0.18.0, 0.19.0
> Reporter: Raghu Angadi
> Assignee: Hairong Kuang
> Priority: Blocker
> Fix For: 0.18.2, 0.19.0
>
>
> append is supported by LocalFileSystem but it does not update crc when a file is appended.
> When you enable checksum verification {{TestLocalFileSystem.testAppend}} fails. Since HADOOP-4277 is a blocker for 0.17 I am planning to disable this test in HADOOP-4277.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4292) append() does not work for
LocalFileSystem
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4292?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12637234#action_12637234 ]
Doug Cutting commented on HADOOP-4292:
--------------------------------------
As Raghu points out and you acknowledged on the list, only the CRC file needs to be truncated.
An alternative would be to re-write the entire CRC file when truncate is not possible.
Should we add truncate() to FileSystem.java instead of to a new interface?
Why does the truncate method return an FSDataOutputStream? Perhaps it could return a boolean, indicating whether it was successful. Then the default implementation could simply return false. Or it could return void and the default would throw UnsupportedOperationException.
> append() does not work for LocalFileSystem
> ------------------------------------------
>
> Key: HADOOP-4292
> URL: https://issues.apache.org/jira/browse/HADOOP-4292
> Project: Hadoop Core
> Issue Type: Bug
> Components: fs
> Affects Versions: 0.18.0, 0.19.0
> Reporter: Raghu Angadi
> Assignee: Hairong Kuang
> Priority: Blocker
> Fix For: 0.18.2, 0.19.0
>
>
> append is supported by LocalFileSystem but it does not update crc when a file is appended.
> When you enable checksum verification {{TestLocalFileSystem.testAppend}} fails. Since HADOOP-4277 is a blocker for 0.17 I am planning to disable this test in HADOOP-4277.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4292) append() does not work for
LocalFileSystem
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4292?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12636165#action_12636165 ]
Tsz Wo (Nicholas), SZE commented on HADOOP-4292:
------------------------------------------------
There is another bug in LocalFileSystem:
When creating a file with LocalFileSystem.create(Path f, FsPermission permission, boolean overwrite, int bufferSize, short replication, long blockSize, Progressable progress), no checksum will be created.
> append() does not work for LocalFileSystem
> ------------------------------------------
>
> Key: HADOOP-4292
> URL: https://issues.apache.org/jira/browse/HADOOP-4292
> Project: Hadoop Core
> Issue Type: Bug
> Components: fs
> Affects Versions: 0.18.0, 0.19.0
> Reporter: Raghu Angadi
> Assignee: Tsz Wo (Nicholas), SZE
> Priority: Blocker
> Fix For: 0.18.2, 0.19.0
>
>
> append is supported by LocalFileSystem but it does not update crc when a file is appended.
> When you enable checksum verification {{TestLocalFileSystem.testAppend}} fails. Since HADOOP-4277 is a blocker for 0.17 I am planning to disable this test in HADOOP-4277.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-4292) append() does not work for
LocalFileSystem
Posted by "Hairong Kuang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4292?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hairong Kuang updated HADOOP-4292:
----------------------------------
Attachment: LFSAppend.patch
This patch marks "append" as "not supported" in ChecksumFileSystem. As a result, append is not supported in LocalFileSystem.
> append() does not work for LocalFileSystem
> ------------------------------------------
>
> Key: HADOOP-4292
> URL: https://issues.apache.org/jira/browse/HADOOP-4292
> Project: Hadoop Core
> Issue Type: Bug
> Components: fs
> Affects Versions: 0.18.0, 0.19.0
> Reporter: Raghu Angadi
> Assignee: Hairong Kuang
> Priority: Blocker
> Fix For: 0.18.2, 0.19.0
>
> Attachments: LFSAppend.patch
>
>
> append is supported by LocalFileSystem but it does not update crc when a file is appended.
> When you enable checksum verification {{TestLocalFileSystem.testAppend}} fails. Since HADOOP-4277 is a blocker for 0.17 I am planning to disable this test in HADOOP-4277.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4292) append() does not work for
LocalFileSystem
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4292?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12638670#action_12638670 ]
Doug Cutting commented on HADOOP-4292:
--------------------------------------
Owen> -1 on introducing FileSystem.truncate in 0.19.
I agree. I didn't notice earlier that this was targeted for 0.19. Append for the local fs is a new feature, and should go in 0.20. Truncate for any fs is clearly a new feature.
> append() does not work for LocalFileSystem
> ------------------------------------------
>
> Key: HADOOP-4292
> URL: https://issues.apache.org/jira/browse/HADOOP-4292
> Project: Hadoop Core
> Issue Type: Bug
> Components: fs
> Affects Versions: 0.18.0, 0.19.0
> Reporter: Raghu Angadi
> Assignee: Hairong Kuang
> Priority: Blocker
> Fix For: 0.18.2, 0.19.0
>
>
> append is supported by LocalFileSystem but it does not update crc when a file is appended.
> When you enable checksum verification {{TestLocalFileSystem.testAppend}} fails. Since HADOOP-4277 is a blocker for 0.17 I am planning to disable this test in HADOOP-4277.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4292) append() does not work for
LocalFileSystem
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4292?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12639285#action_12639285 ]
Hadoop QA commented on HADOOP-4292:
-----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12392037/LFSAppend.patch
against trunk revision 704261.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 3 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 Eclipse classpath. The patch retains Eclipse classpath integrity.
-1 core tests. The patch failed core unit tests.
+1 contrib tests. The patch passed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3453/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3453/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3453/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3453/console
This message is automatically generated.
> append() does not work for LocalFileSystem
> ------------------------------------------
>
> Key: HADOOP-4292
> URL: https://issues.apache.org/jira/browse/HADOOP-4292
> Project: Hadoop Core
> Issue Type: Bug
> Components: fs
> Affects Versions: 0.18.0, 0.19.0
> Reporter: Raghu Angadi
> Assignee: Hairong Kuang
> Priority: Blocker
> Fix For: 0.18.2, 0.19.0
>
> Attachments: LFSAppend.patch
>
>
> append is supported by LocalFileSystem but it does not update crc when a file is appended.
> When you enable checksum verification {{TestLocalFileSystem.testAppend}} fails. Since HADOOP-4277 is a blocker for 0.17 I am planning to disable this test in HADOOP-4277.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4292) append() does not work for
LocalFileSystem
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4292?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12637289#action_12637289 ]
dhruba borthakur commented on HADOOP-4292:
------------------------------------------
+1 on introducing FileSystem.truncate(Path, offset)
> append() does not work for LocalFileSystem
> ------------------------------------------
>
> Key: HADOOP-4292
> URL: https://issues.apache.org/jira/browse/HADOOP-4292
> Project: Hadoop Core
> Issue Type: Bug
> Components: fs
> Affects Versions: 0.18.0, 0.19.0
> Reporter: Raghu Angadi
> Assignee: Hairong Kuang
> Priority: Blocker
> Fix For: 0.18.2, 0.19.0
>
>
> append is supported by LocalFileSystem but it does not update crc when a file is appended.
> When you enable checksum verification {{TestLocalFileSystem.testAppend}} fails. Since HADOOP-4277 is a blocker for 0.17 I am planning to disable this test in HADOOP-4277.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4292) append() does not work for
LocalFileSystem
Posted by "Hairong Kuang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4292?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12637164#action_12637164 ]
Hairong Kuang commented on HADOOP-4292:
---------------------------------------
Specifically what's in my mind is to introduce the following interface:
{code:title=TruncatableFileSystem.java|borderStyle=solid}
interface TruncatableFileSystem {
// set the file length to be the given len and the current position to be the given len
FSDataOutputStream truncate(Path file, long len);
{code}
RawLocalFileSystem will implement TruncatableFileSystem.
In ChecksumFileSystem.append, if the raw file system is not a TruncatableFileSystem, throw an exception indicating that the append is not supported. Otherwise, read the last partial trunk, truncate both the data file and crc file, rewrite the last partial trunk.
> append() does not work for LocalFileSystem
> ------------------------------------------
>
> Key: HADOOP-4292
> URL: https://issues.apache.org/jira/browse/HADOOP-4292
> Project: Hadoop Core
> Issue Type: Bug
> Components: fs
> Affects Versions: 0.18.0, 0.19.0
> Reporter: Raghu Angadi
> Assignee: Hairong Kuang
> Priority: Blocker
> Fix For: 0.18.2, 0.19.0
>
>
> append is supported by LocalFileSystem but it does not update crc when a file is appended.
> When you enable checksum verification {{TestLocalFileSystem.testAppend}} fails. Since HADOOP-4277 is a blocker for 0.17 I am planning to disable this test in HADOOP-4277.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Assigned: (HADOOP-4292) append() does not work for
LocalFileSystem
Posted by "Hairong Kuang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4292?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hairong Kuang reassigned HADOOP-4292:
-------------------------------------
Assignee: Hairong Kuang (was: Tsz Wo (Nicholas), SZE)
> append() does not work for LocalFileSystem
> ------------------------------------------
>
> Key: HADOOP-4292
> URL: https://issues.apache.org/jira/browse/HADOOP-4292
> Project: Hadoop Core
> Issue Type: Bug
> Components: fs
> Affects Versions: 0.18.0, 0.19.0
> Reporter: Raghu Angadi
> Assignee: Hairong Kuang
> Priority: Blocker
> Fix For: 0.18.2, 0.19.0
>
>
> append is supported by LocalFileSystem but it does not update crc when a file is appended.
> When you enable checksum verification {{TestLocalFileSystem.testAppend}} fails. Since HADOOP-4277 is a blocker for 0.17 I am planning to disable this test in HADOOP-4277.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4292) append() does not work for
LocalFileSystem
Posted by "Raghu Angadi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4292?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12637278#action_12637278 ]
Raghu Angadi commented on HADOOP-4292:
--------------------------------------
> Even only the CRC file is truncated, fail in append could result in data-checksum inconsistency. How could we fix this?
We could seek to the right position and start writing from there. If that is not allowed, we could copy the crc file.
For this jira, we can implement this for LocalFS (and any simpler Filesystems), the default implementation at ChecksumFS could be to throw unsupported exception.
> append() does not work for LocalFileSystem
> ------------------------------------------
>
> Key: HADOOP-4292
> URL: https://issues.apache.org/jira/browse/HADOOP-4292
> Project: Hadoop Core
> Issue Type: Bug
> Components: fs
> Affects Versions: 0.18.0, 0.19.0
> Reporter: Raghu Angadi
> Assignee: Hairong Kuang
> Priority: Blocker
> Fix For: 0.18.2, 0.19.0
>
>
> append is supported by LocalFileSystem but it does not update crc when a file is appended.
> When you enable checksum verification {{TestLocalFileSystem.testAppend}} fails. Since HADOOP-4277 is a blocker for 0.17 I am planning to disable this test in HADOOP-4277.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4292) append() does not work for
LocalFileSystem
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4292?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12637284#action_12637284 ]
Doug Cutting commented on HADOOP-4292:
--------------------------------------
> Even only the CRC file is truncated, fail in append could result in data-checksum inconsistency. How could we fix this?
With ChecksumFileSystem, lots of failures can result in data-checksum inconsistency. I don't think this makes the situation particularly worse. Perhaps we could change the checksum file's format so that it includes a trailing element that includes the file's length or somesuch, to avoid such false-positives.
> There will be a lot of changes [ ... ]
More than the addition of a single method to FileSystem.java?
> append() does not work for LocalFileSystem
> ------------------------------------------
>
> Key: HADOOP-4292
> URL: https://issues.apache.org/jira/browse/HADOOP-4292
> Project: Hadoop Core
> Issue Type: Bug
> Components: fs
> Affects Versions: 0.18.0, 0.19.0
> Reporter: Raghu Angadi
> Assignee: Hairong Kuang
> Priority: Blocker
> Fix For: 0.18.2, 0.19.0
>
>
> append is supported by LocalFileSystem but it does not update crc when a file is appended.
> When you enable checksum verification {{TestLocalFileSystem.testAppend}} fails. Since HADOOP-4277 is a blocker for 0.17 I am planning to disable this test in HADOOP-4277.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4292) append() does not work for
LocalFileSystem
Posted by "Raghu Angadi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4292?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12637248#action_12637248 ]
Raghu Angadi commented on HADOOP-4292:
--------------------------------------
> Note that DFSOutputStream is not a ChecksumFileSystem.
I meant to say DFS is not a ChecksumFileSystem.
> append() does not work for LocalFileSystem
> ------------------------------------------
>
> Key: HADOOP-4292
> URL: https://issues.apache.org/jira/browse/HADOOP-4292
> Project: Hadoop Core
> Issue Type: Bug
> Components: fs
> Affects Versions: 0.18.0, 0.19.0
> Reporter: Raghu Angadi
> Assignee: Hairong Kuang
> Priority: Blocker
> Fix For: 0.18.2, 0.19.0
>
>
> append is supported by LocalFileSystem but it does not update crc when a file is appended.
> When you enable checksum verification {{TestLocalFileSystem.testAppend}} fails. Since HADOOP-4277 is a blocker for 0.17 I am planning to disable this test in HADOOP-4277.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-4292) append() does not work for
LocalFileSystem
Posted by "Raghu Angadi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4292?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Raghu Angadi updated HADOOP-4292:
---------------------------------
Affects Version/s: 0.18.0
Fix Version/s: 0.18.2
This affects 0.18 as well. For 18, the fix could be as simple as just throwing a 'not supported' exception.
> append() does not work for LocalFileSystem
> ------------------------------------------
>
> Key: HADOOP-4292
> URL: https://issues.apache.org/jira/browse/HADOOP-4292
> Project: Hadoop Core
> Issue Type: Bug
> Components: fs
> Affects Versions: 0.18.0, 0.19.0
> Reporter: Raghu Angadi
> Assignee: Tsz Wo (Nicholas), SZE
> Priority: Blocker
> Fix For: 0.18.2, 0.19.0
>
>
> append is supported by LocalFileSystem but it does not update crc when a file is appended.
> When you enable checksum verification {{TestLocalFileSystem.testAppend}} fails. Since HADOOP-4277 is a blocker for 0.17 I am planning to disable this test in HADOOP-4277.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4292) append() does not work for
LocalFileSystem
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4292?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12637273#action_12637273 ]
Tsz Wo (Nicholas), SZE commented on HADOOP-4292:
------------------------------------------------
> As Raghu points out and you acknowledged on the list, only the CRC file needs to be truncated.
Even only the CRC file is truncated, fail in append could result in data-checksum inconsistency. How could we fix this?
> The reason that I did not propose adding truncate() to FileSystem.java is that I did not make dramatic change to FileSystem APIs.
There will be a lot of changes. Are we going to apply all these changes to 0.18 and onwards?
> append() does not work for LocalFileSystem
> ------------------------------------------
>
> Key: HADOOP-4292
> URL: https://issues.apache.org/jira/browse/HADOOP-4292
> Project: Hadoop Core
> Issue Type: Bug
> Components: fs
> Affects Versions: 0.18.0, 0.19.0
> Reporter: Raghu Angadi
> Assignee: Hairong Kuang
> Priority: Blocker
> Fix For: 0.18.2, 0.19.0
>
>
> append is supported by LocalFileSystem but it does not update crc when a file is appended.
> When you enable checksum verification {{TestLocalFileSystem.testAppend}} fails. Since HADOOP-4277 is a blocker for 0.17 I am planning to disable this test in HADOOP-4277.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-4292) append() does not work for
LocalFileSystem
Posted by "Hairong Kuang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4292?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hairong Kuang updated HADOOP-4292:
----------------------------------
Resolution: Fixed
Status: Resolved (was: Patch Available)
I just committed this.
> append() does not work for LocalFileSystem
> ------------------------------------------
>
> Key: HADOOP-4292
> URL: https://issues.apache.org/jira/browse/HADOOP-4292
> Project: Hadoop Core
> Issue Type: Bug
> Components: fs
> Affects Versions: 0.18.0, 0.19.0
> Reporter: Raghu Angadi
> Assignee: Hairong Kuang
> Priority: Blocker
> Fix For: 0.18.2, 0.19.0
>
> Attachments: LFSAppend.patch
>
>
> append is supported by LocalFileSystem but it does not update crc when a file is appended.
> When you enable checksum verification {{TestLocalFileSystem.testAppend}} fails. Since HADOOP-4277 is a blocker for 0.17 I am planning to disable this test in HADOOP-4277.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4292) append() does not work for
LocalFileSystem
Posted by "Owen O'Malley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4292?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12638667#action_12638667 ]
Owen O'Malley commented on HADOOP-4292:
---------------------------------------
-1 on introducing FileSystem.truncate in 0.19.
I think that this patch should make ChecksumFileSystem.append throw UnsupportedOperationException.
If someone wants to make it work in 0.20 that is great. But I don't think we should push it into 0.19.
> append() does not work for LocalFileSystem
> ------------------------------------------
>
> Key: HADOOP-4292
> URL: https://issues.apache.org/jira/browse/HADOOP-4292
> Project: Hadoop Core
> Issue Type: Bug
> Components: fs
> Affects Versions: 0.18.0, 0.19.0
> Reporter: Raghu Angadi
> Assignee: Hairong Kuang
> Priority: Blocker
> Fix For: 0.18.2, 0.19.0
>
>
> append is supported by LocalFileSystem but it does not update crc when a file is appended.
> When you enable checksum verification {{TestLocalFileSystem.testAppend}} fails. Since HADOOP-4277 is a blocker for 0.17 I am planning to disable this test in HADOOP-4277.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4292) append() does not work for
LocalFileSystem
Posted by "Hairong Kuang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4292?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12637185#action_12637185 ]
Hairong Kuang commented on HADOOP-4292:
---------------------------------------
Another option is that we could have a positional write interface instead of having a truncate interface. Something like:
{code}
interface PositionalWritableFileSystem {
// Return an output stream with the given pos as its current position; next write will write to the current position
FSOutputStream append(File file, long pos);
}
{code}
> append() does not work for LocalFileSystem
> ------------------------------------------
>
> Key: HADOOP-4292
> URL: https://issues.apache.org/jira/browse/HADOOP-4292
> Project: Hadoop Core
> Issue Type: Bug
> Components: fs
> Affects Versions: 0.18.0, 0.19.0
> Reporter: Raghu Angadi
> Assignee: Hairong Kuang
> Priority: Blocker
> Fix For: 0.18.2, 0.19.0
>
>
> append is supported by LocalFileSystem but it does not update crc when a file is appended.
> When you enable checksum verification {{TestLocalFileSystem.testAppend}} fails. Since HADOOP-4277 is a blocker for 0.17 I am planning to disable this test in HADOOP-4277.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4292) append() does not work for
LocalFileSystem
Posted by "Hairong Kuang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4292?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12637285#action_12637285 ]
Hairong Kuang commented on HADOOP-4292:
---------------------------------------
> Even only the CRC file is truncated, fail in append could result in data-checksum inconsistency. How could we fix this?
Append is an operation that changes the content of file. So it has a potential risk that may lead to data-crc inconsistency. This is a problem for append & write. I do not plan to get it solved in this issue.
> My intuition is to make this two separate operations: truncate, and open-for-append.
This is fine for me.
Does the community feel comfortable for adding the method truncate to FileSystem?
> append() does not work for LocalFileSystem
> ------------------------------------------
>
> Key: HADOOP-4292
> URL: https://issues.apache.org/jira/browse/HADOOP-4292
> Project: Hadoop Core
> Issue Type: Bug
> Components: fs
> Affects Versions: 0.18.0, 0.19.0
> Reporter: Raghu Angadi
> Assignee: Hairong Kuang
> Priority: Blocker
> Fix For: 0.18.2, 0.19.0
>
>
> append is supported by LocalFileSystem but it does not update crc when a file is appended.
> When you enable checksum verification {{TestLocalFileSystem.testAppend}} fails. Since HADOOP-4277 is a blocker for 0.17 I am planning to disable this test in HADOOP-4277.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4292) append() does not work for
LocalFileSystem
Posted by "Hairong Kuang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4292?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12637172#action_12637172 ]
Hairong Kuang commented on HADOOP-4292:
---------------------------------------
Raghu's comment makes sense. How about we truncating only the crc file not the data file? So data is not at the risk of losing.
> append() does not work for LocalFileSystem
> ------------------------------------------
>
> Key: HADOOP-4292
> URL: https://issues.apache.org/jira/browse/HADOOP-4292
> Project: Hadoop Core
> Issue Type: Bug
> Components: fs
> Affects Versions: 0.18.0, 0.19.0
> Reporter: Raghu Angadi
> Assignee: Hairong Kuang
> Priority: Blocker
> Fix For: 0.18.2, 0.19.0
>
>
> append is supported by LocalFileSystem but it does not update crc when a file is appended.
> When you enable checksum verification {{TestLocalFileSystem.testAppend}} fails. Since HADOOP-4277 is a blocker for 0.17 I am planning to disable this test in HADOOP-4277.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-4292) append() does not work for
LocalFileSystem
Posted by "Hairong Kuang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4292?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hairong Kuang updated HADOOP-4292:
----------------------------------
Status: Patch Available (was: Open)
> append() does not work for LocalFileSystem
> ------------------------------------------
>
> Key: HADOOP-4292
> URL: https://issues.apache.org/jira/browse/HADOOP-4292
> Project: Hadoop Core
> Issue Type: Bug
> Components: fs
> Affects Versions: 0.18.0, 0.19.0
> Reporter: Raghu Angadi
> Assignee: Hairong Kuang
> Priority: Blocker
> Fix For: 0.18.2, 0.19.0
>
> Attachments: LFSAppend.patch
>
>
> append is supported by LocalFileSystem but it does not update crc when a file is appended.
> When you enable checksum verification {{TestLocalFileSystem.testAppend}} fails. Since HADOOP-4277 is a blocker for 0.17 I am planning to disable this test in HADOOP-4277.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4292) append() does not work for
LocalFileSystem
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4292?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12637270#action_12637270 ]
Doug Cutting commented on HADOOP-4292:
--------------------------------------
truncate does seem like a common-enough file operation to put in FileSystem.java.
> If truncate returns an FSDataOutput, next write does not need to open it for appending first.
My intuition is to make this two separate operations: truncate, and open-for-append. Do you have a strong reason to connect them?
> append() does not work for LocalFileSystem
> ------------------------------------------
>
> Key: HADOOP-4292
> URL: https://issues.apache.org/jira/browse/HADOOP-4292
> Project: Hadoop Core
> Issue Type: Bug
> Components: fs
> Affects Versions: 0.18.0, 0.19.0
> Reporter: Raghu Angadi
> Assignee: Hairong Kuang
> Priority: Blocker
> Fix For: 0.18.2, 0.19.0
>
>
> append is supported by LocalFileSystem but it does not update crc when a file is appended.
> When you enable checksum verification {{TestLocalFileSystem.testAppend}} fails. Since HADOOP-4277 is a blocker for 0.17 I am planning to disable this test in HADOOP-4277.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.