storm-hdfs JIRAs

["Aaron.Dossett" <Aa...@target.com>]

Hi Storm Team,

I am have created a few JIRAs related to storm-pdfs and wanted to summarize their current state here.  The first, STORM-969, contains some key changes that I would like to get feedback on.

STORM-969 — this re-writes the execute() method of HdfsBolt to avoid ending up in unrecoverable state.  I’ve made several changes based on feedback in the JIRA.  Here at Target we are running this in production and have already benefited enormously from it, HdfsBolt is much more resilient to errors in the underlying Hadoop cluster.  I believe this is a valuable contribution to the Storm project and Target wants to share it widely.  If there are more issues with it, or if the project doesn’t believe it’s ready to be contributed, I would appreciate that feedback as well.

STORM-1073 - SequenceFileBolt could use essentially the same enhancements that HdfsBolt got in the first JIRA.   While implementing this, the large amount of duplicate code that I added convinced me that AbstractHdfsBolt could be productively refactored so that implementing classes can focus on what makes them unique.  I have a PR ready for this that I can submit if STORM-969 is accepted.

STORM-1074 — After I refactored AbstractHdfsBolt we had a request to support writing Avro binary files directly to HDFS with Storm.  Doing that was really quite easy after the refactoring.  It seems generic enough to be worth contributing back to the Storm project.  If and when 1073 and 969 are accepted I can submit a PR for this change.

Target is excited to be using Storm and we’ve already made some minor contributions back to the project, we hope these more significant contributions will also be valuable to the project.

Thank you and I look forward to any feedback on these tickets.

-Aaron

Re: storm-hdfs JIRAs

["Aaron.Dossett" <Aa...@target.com>]

Immediately after sending this I saw that Harsha had given a +1 to STORM-969 about two minutes earlier.  Thank you!

From: Aaron Dossett <Aa...@target.com>>
Date: Monday, September 28, 2015 at 4:56 PM
To: "dev@storm.apache.org<ma...@storm.apache.org>" <de...@storm.apache.org>>
Cc: Aaron Dossett <Aa...@target.com>>
Subject: storm-hdfs JIRAs

Hi Storm Team,

I am have created a few JIRAs related to storm-pdfs and wanted to summarize their current state here.  The first, STORM-969, contains some key changes that I would like to get feedback on.

STORM-969 — this re-writes the execute() method of HdfsBolt to avoid ending up in unrecoverable state.  I’ve made several changes based on feedback in the JIRA.  Here at Target we are running this in production and have already benefited enormously from it, HdfsBolt is much more resilient to errors in the underlying Hadoop cluster.  I believe this is a valuable contribution to the Storm project and Target wants to share it widely.  If there are more issues with it, or if the project doesn’t believe it’s ready to be contributed, I would appreciate that feedback as well.

STORM-1073 - SequenceFileBolt could use essentially the same enhancements that HdfsBolt got in the first JIRA.   While implementing this, the large amount of duplicate code that I added convinced me that AbstractHdfsBolt could be productively refactored so that implementing classes can focus on what makes them unique.  I have a PR ready for this that I can submit if STORM-969 is accepted.

STORM-1074 — After I refactored AbstractHdfsBolt we had a request to support writing Avro binary files directly to HDFS with Storm.  Doing that was really quite easy after the refactoring.  It seems generic enough to be worth contributing back to the Storm project.  If and when 1073 and 969 are accepted I can submit a PR for this change.

Target is excited to be using Storm and we’ve already made some minor contributions back to the project, we hope these more significant contributions will also be valuable to the project.

Thank you and I look forward to any feedback on these tickets.

-Aaron