You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@nifi.apache.org by simonellistonball <gi...@git.apache.org> on 2016/05/02 01:43:21 UTC
[GitHub] nifi pull request: First cut at NIFI-1833 - Adding Azure Storage p...
GitHub user simonellistonball opened a pull request:
https://github.com/apache/nifi/pull/399
First cut at NIFI-1833 - Adding Azure Storage processors
A little progress on a quick patch to add storage processors. Still requires some unit and integration testing. May need a little more on the documentation as well. Better supporting for Azure batching commands would be a good next step, but probably lives as a further enhancement. Comments
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/simonellistonball/nifi NIFI-1833
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/nifi/pull/399.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #399
----
commit ab5e28403f6e0dc8fc62bdb1d7e6282eda4a29f8
Author: Simon Elliston Ball <si...@simonellistonball.com>
Date: 2016-05-01T23:35:34Z
First cut at NIFI-1833 - Adding Azure Storage processors
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi issue #399: NIFI-1833 - Adding Azure Storage processors
Posted by slachterman <gi...@git.apache.org>.
Github user slachterman commented on the issue:
https://github.com/apache/nifi/pull/399
Quick question: is there overlap with [NIFI-1922](https://issues.apache.org/jira/browse/NIFI-1922)?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi issue #399: NIFI-1833 - Adding Azure Storage processors
Posted by trixpan <gi...@git.apache.org>.
Github user trixpan commented on the issue:
https://github.com/apache/nifi/pull/399
@simonellistonball - Seems like your PR has been adopted by @jtstorck and @aperepel so closing this PR as we currently have 3 PRs for the same JIRA.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi issue #399: NIFI-1833 - Adding Azure Storage processors
Posted by trixpan <gi...@git.apache.org>.
Github user trixpan commented on the issue:
https://github.com/apache/nifi/pull/399
@slachterman - thank you for the handy link
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi issue #399: First cut at NIFI-1833 - Adding Azure Storage processors
Posted by simonellistonball <gi...@git.apache.org>.
Github user simonellistonball commented on the issue:
https://github.com/apache/nifi/pull/399
I'm still working on this, yes. Apologies, I've not had the time I'd like to finish it off, but will be doing so very soon. The issue is very much the duplicate nar dependencies, which requires some refactoring to avoid the dependency this PR has on standard-processors.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi issue #399: NIFI-1833 - Adding Azure Storage processors
Posted by simonellistonball <gi...@git.apache.org>.
Github user simonellistonball commented on the issue:
https://github.com/apache/nifi/pull/399
Latest commit should remove some of the duplicate nar circular dependency issues. It's not the cleanest way to implement this, but that really wants a common cloud util library around the way cases like listing are handles. That feels like the scope for a much greater refactoring.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi issue #399: First cut at NIFI-1833 - Adding Azure Storage processors
Posted by atilcock <gi...@git.apache.org>.
Github user atilcock commented on the issue:
https://github.com/apache/nifi/pull/399
Hi Simon - are you still working on this? Not sure if it's related to the duplicate nar issue you mentioned but I've found an issue after a restart of NIFI where the processor stops working. i.e. implement a workflow with a putAzureBlob call. Stop and restart NIFI and the processor no longer responds to data in the queue.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi issue #399: NIFI-1833 - Adding Azure Storage processors
Posted by trixpan <gi...@git.apache.org>.
Github user trixpan commented on the issue:
https://github.com/apache/nifi/pull/399
@slachterman I would say so
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi pull request: First cut at NIFI-1833 - Adding Azure Storage p...
Posted by simonellistonball <gi...@git.apache.org>.
Github user simonellistonball commented on the pull request:
https://github.com/apache/nifi/pull/399#issuecomment-217411669
This needs a little bit more work. The goal here is to use a common pattern from standard processors, however, while tests work fine, loading into NiFi causes a duplicate of the standard processor nar and therefore unpredicatable ordering. One solution to this would be to factor the base AbstractListProcessor classes out of standard processors into processor-utils. However, they have a legacy dependency on the distributed cache service which would cause a circular dependency for this refactoring.
Short term solution would be to reimplement the trigger and state logic direct in the Azure processors, which violates DRY.
Longer term it would be great to have clean Abstracts for some of these common custom patterns in processor-utils, but that's for another issue and another PR.
In the meantime I'll re-implement and update the PR, so please don't worry about reviews right now.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi issue #399: NIFI-1833 - Adding Azure Storage processors
Posted by slachterman <gi...@git.apache.org>.
Github user slachterman commented on the issue:
https://github.com/apache/nifi/pull/399
Example for Azure Data Lake Storage using additional classpath functionality: https://community.hortonworks.com/articles/58793/hdf-20-use-ambari-to-enable-kerberos-for-hdf-clust.html
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi issue #399: NIFI-1833 - Adding Azure Storage processors
Posted by joewitt <gi...@git.apache.org>.
Github user joewitt commented on the issue:
https://github.com/apache/nifi/pull/399
@simonellistonball can you consider closing this in favor of the approach outlined in NIFI-1922 which takes advantage of a new runtime classpath manipulation option which has been proven to work.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi pull request #399: NIFI-1833 - Adding Azure Storage processors
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/nifi/pull/399
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi issue #399: NIFI-1833 - Adding Azure Storage processors
Posted by joaocc <gi...@git.apache.org>.
Github user joaocc commented on the issue:
https://github.com/apache/nifi/pull/399
@simonellistonball Hi. Was the work on Azure Storage Table dropped? I tried to find it in the current (1.8) version but only Storage Blob seems to be there. Thx
---