You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@storm.apache.org by hmcl <gi...@git.apache.org> on 2016/11/19 17:00:50 UTC
[GitHub] storm pull request #1787: STORM-2208: HDFS State Throws FileNotFoundExceptio...
GitHub user hmcl opened a pull request:
https://github.com/apache/storm/pull/1787
STORM-2208: HDFS State Throws FileNotFoundException in Azure Data Lake Store file system (adl://)
- Close OutputStream before file gets deleted
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/hmcl/storm-apache Apache_master_STORM-2208
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/storm/pull/1787.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1787
----
commit 0ff9a3ef4800a64471d911c5c1498e39f55f4867
Author: Hugo Louro <hm...@gmail.com>
Date: 2016-11-19T16:58:33Z
STORM-2208: HDFS State Throws FileNotFoundException in Azure Data Lake Store file system (adl://)
- Close OutputStream before file gets deleted
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request #1787: STORM-2208: HDFS State Throws FileNotFoundExceptio...
Posted by revans2 <gi...@git.apache.org>.
Github user revans2 commented on a diff in the pull request:
https://github.com/apache/storm/pull/1787#discussion_r88930878
--- Diff: external/storm-hdfs/src/main/java/org/apache/storm/hdfs/trident/HdfsState.java ---
@@ -500,6 +500,7 @@ private void updateIndex(long txId) {
bw.write(txnRecord.toString());
bw.newLine();
bw.flush();
+ out.close(); // Closing the output stream in finally block causes FileNotFoundException in Azure Data Lake Store file system (adl://)
--- End diff --
Why not just wrap the output stream in a try block instead?
```
Path tmpPath = tmpFilePath(indexFilePath.toString());
TxnRecord txnRecord = new TxnRecord(txId, options.currentFile.toString(), this.options.getCurrentOffset());
try (FSDataOutputStream out = this.options.fs.create(tmpPath, true);
BufferedWriter bw = new BufferedWriter(new OutputStreamWriter(out));) {
bw.write(txnRecord.toString());
bw.newLine();
bw.flush();
}
```
Then you don't need to worry about the close happening before the delete, even in error cases.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request #1787: STORM-2208: HDFS State Throws FileNotFoundExceptio...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/storm/pull/1787
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm issue #1787: STORM-2208: HDFS State Throws FileNotFoundException in Az...
Posted by hmcl <gi...@git.apache.org>.
Github user hmcl commented on the issue:
https://github.com/apache/storm/pull/1787
@revans2 I just tested in the Azure cluster running AFS and the fix works. Please see attached screenshots.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm issue #1787: STORM-2208: HDFS State Throws FileNotFoundException in Az...
Posted by hmcl <gi...@git.apache.org>.
Github user hmcl commented on the issue:
https://github.com/apache/storm/pull/1787
Please backport to 1.x and 1.0.x branch. It should be a trivial merge.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm issue #1787: STORM-2208: HDFS State Throws FileNotFoundException in Az...
Posted by revans2 <gi...@git.apache.org>.
Github user revans2 commented on the issue:
https://github.com/apache/storm/pull/1787
+1 looks god to me, although I am not an expert on azure, so I assume you have tested the new code and it works.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm issue #1787: STORM-2208: HDFS State Throws FileNotFoundException in Az...
Posted by hmcl <gi...@git.apache.org>.
Github user hmcl commented on the issue:
https://github.com/apache/storm/pull/1787
@revans2 agree && Done!
I followed your suggestion but most calls to fs will throw IOException. To keep only one try catch block the cleanest way to do it is to close the OutputStream inside the try block. Otherwise, we need 2 try / catch blocks.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm issue #1787: STORM-2208: HDFS State Throws FileNotFoundException in Az...
Posted by HeartSaVioR <gi...@git.apache.org>.
Github user HeartSaVioR commented on the issue:
https://github.com/apache/storm/pull/1787
+1
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm issue #1787: STORM-2208: HDFS State Throws FileNotFoundException in Az...
Posted by arunmahadevan <gi...@git.apache.org>.
Github user arunmahadevan commented on the issue:
https://github.com/apache/storm/pull/1787
+1, the changes look good. Not being able to close the stream after renaming / deleting the file is deviation from the hdfs/java fs standard which should also be supported by the azure implementation.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm issue #1787: STORM-2208: HDFS State Throws FileNotFoundException in Az...
Posted by hmcl <gi...@git.apache.org>.
Github user hmcl commented on the issue:
https://github.com/apache/storm/pull/1787
@arunmahadevan I agree! I have already filed a but to the Azure team to fix that case.
However, it does no harm that we account for this scenario.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm issue #1787: STORM-2208: HDFS State Throws FileNotFoundException in Az...
Posted by harshach <gi...@git.apache.org>.
Github user harshach commented on the issue:
https://github.com/apache/storm/pull/1787
+1
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---