[jira] [Created] (NIFI-13930) PutAzureDataLakeStorage does not cause Azure to emit a FlushWithClose event on file write

["Mark Ward (Jira)" <ji...@apache.org>]

Mark Ward created NIFI-13930:
--------------------------------

             Summary: PutAzureDataLakeStorage does not cause Azure to emit a FlushWithClose event on file write
                 Key: NIFI-13930
                 URL: https://issues.apache.org/jira/browse/NIFI-13930
             Project: Apache NiFi
          Issue Type: Bug
    Affects Versions: 1.23.2
         Environment: amd64, Windows Server 2022, Java 21.0.3
            Reporter: Mark Ward


Hi,

Note: I am raising this issue on Databrick's behalf, they've requested Jira access and are currently awaiting approval.

We use NiFi to write files to an Azure Storage account where out Databricks workspace can ingest the files using a Queue in Azure and Databrick's File Notification feature.

However, files are not being picked up in a timely manner by Databricks.

We raised with Databricks and they've investigated, with the conclusion being that NiFi's behaviour when completing the file write, and then subsequent rename, does not emit and event type that would normally be expected.

Please see Databricks's summary below for information:
{quote} # Customer uses Apache Nifi 1.23.2, which performs the following operations via the Azure API ([src|https://github.com/apache/nifi/blob/rel/nifi-1.23.2/nifi-nar-bundles/nifi-azure-bundle/nifi-azure-processors/src/main/java/org/apache/nifi/processors/azure/storage/PutAzureDataLakeStorage.java#L149-L166])
 ## Create a temp file using the [Path Create|https://learn.microsoft.com/en-us/rest/api/storageservices/datalakestoragegen2/path/create?view=rest-storageservices-datalakestoragegen2-2019-12-12] API. In this case Azure emits a [BlobCreated|https://learn.microsoft.com/en-us/azure/event-grid/event-schema-blob-storage?toc=%2Fazure%2Fstorage%2Fblobs%2Ftoc.json&tabs=event-grid-event-schema#microsoftstorageblobcreated-event-data-lake-storage-gen2-1] event with the {{api}} field set to {{CreateFile}} (which is purposefully not processed by CSMS - [source|https://github.com/databricks/universe/blob/fc4a34b61abe58fbe363cc55cdb67edd480f985e/jobs-cloud-storage-meta/src/cloud/azure/resourcemanagement/AqsEventGridResourceManagementClient.scala#L366]).
 ## [Append|https://github.com/Azure/azure-sdk-for-java/blob/main/sdk/storage/azure-storage-file-datalake/src/main/java/com/azure/storage/file/datalake/DataLakeFileClient.java#L656] content to the file, there is no file event for this operation.
 ## [Flush|https://github.com/Azure/azure-sdk-for-java/blob/main/sdk/storage/azure-storage-file-datalake/src/main/java/com/azure/storage/file/datalake/DataLakeFileClient.java#L930] the appended content file content. Nifi’s implementation flushes the file without closing it ([source|https://github.com/Azure/azure-sdk-for-java/blob/main/sdk/storage/azure-storage-file-datalake/src/main/java/com/azure/storage/file/datalake/DataLakeFileClient.java#L935]), as a result Azure +doesn’t+ emit a {{FlushWithClose}} event (CSMS processes this event type).
 ## [Rename|https://learn.microsoft.com/en-us/dotnet/api/azure.storage.files.datalake.datalakefileclient.rename?view=azure-dotnet] the temp file to its final name, this generates a [BlobRenamed|https://learn.microsoft.com/en-us/azure/event-grid/event-schema-blob-storage?toc=%2Fazure%2Fstorage%2Fblobs%2Ftoc.json&tabs=event-grid-event-schema#microsoftstorageblobrenamed-event-data-lake-storage-gen2-1] that’s processed by CSMS (but ignored, see below).
 # The behaviour observed by the customer is explained by:
 ## When (1d) happens, CSMS gets the BlobRenamed event for a file for which it doesn’t know the metadata about the source file. The BlobRenamed event doesn’t contain all the information needed by CSMS (it misses the {{etag}} and the {{{}blob size{}}}) and therefore [CSMS ignores the event|https://github.com/databricks/universe/blob/master/jobs-cloud-storage-meta/src/storage/StorageHelper.scala#L600] and no object is created.
 ## When CSMS performs the daily full scan, it finds the renamed file and creates an object for it in its database. This causes the file arrival trigger to find the file and trigger a (delayed) run.{quote}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)