[jira] [Created] (NIFI-9892) Update Azure storage related processors to align with NiFi best practices

["Mark Payne (Jira)" <ji...@apache.org>]

Mark Payne created NIFI-9892:
--------------------------------

             Summary: Update Azure storage related processors to align with NiFi best practices
                 Key: NIFI-9892
                 URL: https://issues.apache.org/jira/browse/NIFI-9892
             Project: Apache NiFi
          Issue Type: Improvement
          Components: Extensions
            Reporter: Mark Payne
            Assignee: Mark Payne


The Azure Storage related processors have been through many iterations and updates. At this point, they are not following best practices for NiFi processors and, as a result, make configuring the processors difficult for users. Specifically, I have found the following problems in using them:
 * Properties are ordered inconsistently. Some processors have Credentials Service as the first property, some as 4th property. The ordering of 'directory' vs 'blob name' vs 'file system' are random. Processors should take great care when ordering properties in order to expose the most important properties first, in order to make configuration as simple as possible for the user.
 * Default values are inconsistent. Some processors use a default value for "Blob", some don't. The same sort of inconsistency exists for many properties.
 * Default values are largely missing. It does not make sense to have a default value for the "Container" property for "ListAzureBlobStorage" but the default value should absolutely be set for "FetchAzureBlobStorage", as well as move, delete, put, etc.
 * Poor default values. Some property values do have defaults. But they default to values like "${azure.blob}" for the filename when the NiFI Core Attribute of "filename" makes more sense.
 * Inconsistent property names. Some properties use the property name "Blob" while others use "Blob Name" to mean the same thing. There may be other, similar examples.
 * List processors do not populate core attributes. Listing processors should always populate attributes such as "filename" and "path", rather than just more specific attributes such as "azure.blob"
 * Abstract processors exist and implement `getSupportedPropertyDescriptors()`.  This is an anti-pattern. While it makes sense in many cases to implement `Set<Relationship> getRelationships()`, it should NOT implement `List<PropertyDescriptor> getSupportedPropertyDescriptors()`. This is because the point of the abstract class is for others to extend it. Others that extend it will have more customized Lists of Property Descriptors. This often leads to a pattern of calling `super.getSupportedPropertyDescriptors()` and then adding specific properties to the end of the List. This is bad because, as noted above, properties should be ordered in a way that makes the most sense for that particular processor, adding the most important properties first and keeping like properties together.
 * Directory Name property is awkward and confusing. The Directory Name property is required for several properties. A value of "/" is invalid and no value can have a leading "/". To write to the root directory, the user must go in and click the checkbox for "Empty Value". The fact that this needed to be explicitly called out in the documentation is a key indicator that it violates user expectations. While the Azure client may not allow a leading `/`, the property should absolutely allow it. And the property should not be required, allowing an unset value to default to the root directory.
 * Code cleanup
 ** Processors use old API for session read/write callbacks. Then create `AtomicReference<Exception>` to hold Exceptions so that they can be inspected later, etc. This can be cleaned up by using newer API methods that return `InputStream` / `OutputStream`.
 ** Code should mark variables `final` when possible.
 * Integration tests no longer work
 ** Some of the Integration tests do not work. Some work sometimes and fail intermittently. They need to either be fixed or deleted
 * FetchAzureDatalakeStorage emits both CONTENT_MODIFIED and FETCH provenance events. These two should not be emitted by the same processor, as FETCH implies that the content of the FlowFile was modified to match that of the data that was FETCHed.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)