You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@nifi.apache.org by "tanmaya-panda1 (via GitHub)" <gi...@apache.org> on 2023/08/18 06:19:59 UTC
[GitHub] [nifi] tanmaya-panda1 opened a new pull request, #7624: NIFI-11958 sink commits
tanmaya-panda1 opened a new pull request, #7624:
URL: https://github.com/apache/nifi/pull/7624
<!-- Licensed to the Apache Software Foundation (ASF) under one or more -->
<!-- contributor license agreements. See the NOTICE file distributed with -->
<!-- this work for additional information regarding copyright ownership. -->
<!-- The ASF licenses this file to You under the Apache License, Version 2.0 -->
<!-- (the "License"); you may not use this file except in compliance with -->
<!-- the License. You may obtain a copy of the License at -->
<!-- http://www.apache.org/licenses/LICENSE-2.0 -->
<!-- Unless required by applicable law or agreed to in writing, software -->
<!-- distributed under the License is distributed on an "AS IS" BASIS, -->
<!-- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -->
<!-- See the License for the specific language governing permissions and -->
<!-- limitations under the License. -->
# Summary
[NIFI-11958](https://issues.apache.org/jira/browse/NIFI-11958)
# Tracking
Please complete the following tracking steps prior to pull request creation.
### Issue Tracking
- [ ] [Apache NiFi Jira](https://issues.apache.org/jira/browse/NIFI) issue created
### Pull Request Tracking
- [ ] Pull Request title starts with Apache NiFi Jira issue number, such as `NIFI-00000`
- [ ] Pull Request commit message starts with Apache NiFi Jira issue number, as such `NIFI-00000`
### Pull Request Formatting
- [ ] Pull Request based on current revision of the `main` branch
- [ ] Pull Request refers to a feature branch with one commit containing changes
# Verification
Please indicate the verification steps performed prior to pull request creation.
### Build
- [ ] Build completed using `mvn clean install -P contrib-check`
- [ ] JDK 17
### Licensing
- [ ] New dependencies are compatible with the [Apache License 2.0](https://apache.org/licenses/LICENSE-2.0) according to the [License Policy](https://www.apache.org/legal/resolved.html)
- [ ] New dependencies are documented in applicable `LICENSE` and `NOTICE` files
### Documentation
- [ ] Documentation formatting appears as expected in rendered files
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] NIFI-11958 Azure Data Explorer as sink [nifi]
Posted by "exceptionfactory (via GitHub)" <gi...@apache.org>.
exceptionfactory commented on code in PR #7624:
URL: https://github.com/apache/nifi/pull/7624#discussion_r1362045237
##########
nifi-nar-bundles/nifi-azure-bundle/nifi-azure-services-api/src/main/java/org/apache/nifi/services/azure/data/explorer/KustoIngestionRequest.java:
##########
@@ -0,0 +1,57 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.services.azure.data.explorer;
+
+import com.microsoft.azure.kusto.ingest.IngestionProperties;
+
+import java.io.InputStream;
+
+public class KustoIngestionRequest {
+
+ private boolean isStreamingEnabled;
Review Comment:
Although method names should start with `is`, the corresponding field names should not begin with `is` by general convention.
```suggestion
private boolean streamingEnabled;
```
##########
nifi-nar-bundles/nifi-azure-bundle/nifi-azure-services-api/src/main/java/org/apache/nifi/services/azure/data/explorer/KustoIngestService.java:
##########
@@ -0,0 +1,28 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.services.azure.data.explorer;
+
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+
+import java.net.URISyntaxException;
+
+@Tags({"azure", "adx", "explorer", "kusto", "ingest"})
+@CapabilityDescription("Connection-Service for Azure ADX (Kusto) ingestion cluster to ingest data.")
+public interface KustoIngestService extends KustoQueryService {
Review Comment:
Is it necessary to extend the `KustoQueryService`?
##########
nifi-nar-bundles/nifi-azure-bundle/nifi-azure-processors/src/main/java/org/apache/nifi/services/azure/data/explorer/StandardKustoIngestService.java:
##########
@@ -0,0 +1,293 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.services.azure.data.explorer;
+
+import com.microsoft.azure.kusto.data.Client;
+import com.microsoft.azure.kusto.data.ClientFactory;
+import com.microsoft.azure.kusto.data.KustoResultSetTable;
+import com.microsoft.azure.kusto.data.auth.ConnectionStringBuilder;
+import com.microsoft.azure.kusto.data.exceptions.DataClientException;
+import com.microsoft.azure.kusto.data.exceptions.DataServiceException;
+import com.microsoft.azure.kusto.ingest.IngestClientFactory;
+import com.microsoft.azure.kusto.ingest.ManagedStreamingIngestClient;
+import com.microsoft.azure.kusto.ingest.QueuedIngestClient;
+import com.microsoft.azure.kusto.ingest.exceptions.IngestionClientException;
+import com.microsoft.azure.kusto.ingest.exceptions.IngestionServiceException;
+import com.microsoft.azure.kusto.ingest.result.IngestionResult;
+import com.microsoft.azure.kusto.ingest.result.IngestionStatus;
+import com.microsoft.azure.kusto.ingest.result.OperationStatus;
+import com.microsoft.azure.kusto.ingest.source.StreamSourceInfo;
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.nifi.annotation.behavior.ReadsAttribute;
+import org.apache.nifi.annotation.behavior.ReadsAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnEnabled;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.controller.AbstractControllerService;
+import org.apache.nifi.controller.ConfigurationContext;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.reporting.InitializationException;
+
+import java.io.IOException;
+import java.net.URISyntaxException;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.List;
+import java.util.Objects;
+
+@Tags({"Azure", "ADX", "Kusto", "ingest", "azure"})
+@CapabilityDescription("Sends batches of flowfile content or stream flowfile content to an Azure ADX cluster.")
+@ReadsAttributes({
+ @ReadsAttribute(attribute = "AUTH_STRATEGY", description = "The strategy/method to authenticate against Azure Active Directory, either 'application' or 'managed_identity'."),
+ @ReadsAttribute(attribute = "APP_ID", description = "Specifies Azure application id for accessing the ADX-Cluster."),
+ @ReadsAttribute(attribute = "APP_KEY", description = "Specifies Azure application key for accessing the ADX-Cluster."),
+ @ReadsAttribute(attribute = "APP_TENANT", description = "Azure application tenant for accessing the ADX-Cluster."),
+ @ReadsAttribute(attribute = "CLUSTER_URL", description = "Endpoint of ADX cluster. This is required only when streaming data to ADX cluster is enabled."),
+})
+public class StandardKustoIngestService extends AbstractControllerService implements KustoIngestService {
+
+ public static final PropertyDescriptor AUTHENTICATION_STRATEGY = new PropertyDescriptor
+ .Builder().name("Authentication Strategy")
+ .displayName("Authentication Strategy")
+ .description("Authentication method for access to Azure Data Explorer")
+ .required(true)
+ .defaultValue(KustoAuthenticationStrategy.MANAGED_IDENTITY.getValue())
+ .allowableValues(KustoAuthenticationStrategy.class)
+ .build();
+
+ public static final PropertyDescriptor APPLICATION_CLIENT_ID = new PropertyDescriptor
+ .Builder().name("Application Client ID")
+ .displayName("Application Client ID")
+ .description("Azure Data Explorer Application Client Identifier for Authentication")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+
+ public static final PropertyDescriptor APPLICATION_KEY = new PropertyDescriptor
+ .Builder().name("Application Key")
+ .displayName("Application Key")
+ .description("Azure Data Explorer Application Key for Authentication")
+ .required(true)
+ .sensitive(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .dependsOn(AUTHENTICATION_STRATEGY, KustoAuthenticationStrategy.APPLICATION_CREDENTIALS.getValue())
+ .build();
+
+ public static final PropertyDescriptor APPLICATION_TENANT_ID = new PropertyDescriptor.Builder()
+ .name("Application Tenant ID")
+ .displayName("Application Tenant ID")
+ .description("Azure Data Explorer Application Tenant Identifier for Authentication")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .dependsOn(AUTHENTICATION_STRATEGY, KustoAuthenticationStrategy.APPLICATION_CREDENTIALS.getValue())
+ .build();
+
+ public static final PropertyDescriptor CLUSTER_URI = new PropertyDescriptor
+ .Builder().name("Cluster URI")
+ .displayName("Cluster URI")
+ .description("Azure Data Explorer Cluster URI")
+ .required(true)
+ .addValidator(StandardValidators.URL_VALIDATOR)
+ .build();
+
+ private static final List<PropertyDescriptor> PROPERTY_DESCRIPTORS = Collections.unmodifiableList(Arrays.asList(
+ AUTHENTICATION_STRATEGY,
+ APPLICATION_CLIENT_ID,
+ APPLICATION_KEY,
+ APPLICATION_TENANT_ID,
+ CLUSTER_URI));
+
+ public static final Pair<String, String> NIFI_SINK = Pair.of("processor", "nifi-sink");
+
+ private volatile QueuedIngestClient queuedIngestClient;
+
+ private volatile ManagedStreamingIngestClient managedStreamingIngestClient;
+
+ private volatile Client executionClient;
+
+ @Override
+ public List<PropertyDescriptor> getSupportedPropertyDescriptors() {
+ return PROPERTY_DESCRIPTORS;
+ }
+
+ /**
+ * @param context the configuration context
+ * @throws InitializationException if unable to create a database connection
+ */
+ @OnEnabled
+ public void onEnabled(final ConfigurationContext context) throws ProcessException, URISyntaxException {
+
+ getLogger().info("Starting Azure ADX Connection Service...");
Review Comment:
This log message is not necessary because the framework logs when it is enabling Controller Services.
```suggestion
```
##########
nifi-nar-bundles/nifi-azure-bundle/nifi-azure-processors/src/main/java/org/apache/nifi/services/azure/data/explorer/StandardKustoIngestService.java:
##########
@@ -0,0 +1,293 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.services.azure.data.explorer;
+
+import com.microsoft.azure.kusto.data.Client;
+import com.microsoft.azure.kusto.data.ClientFactory;
+import com.microsoft.azure.kusto.data.KustoResultSetTable;
+import com.microsoft.azure.kusto.data.auth.ConnectionStringBuilder;
+import com.microsoft.azure.kusto.data.exceptions.DataClientException;
+import com.microsoft.azure.kusto.data.exceptions.DataServiceException;
+import com.microsoft.azure.kusto.ingest.IngestClientFactory;
+import com.microsoft.azure.kusto.ingest.ManagedStreamingIngestClient;
+import com.microsoft.azure.kusto.ingest.QueuedIngestClient;
+import com.microsoft.azure.kusto.ingest.exceptions.IngestionClientException;
+import com.microsoft.azure.kusto.ingest.exceptions.IngestionServiceException;
+import com.microsoft.azure.kusto.ingest.result.IngestionResult;
+import com.microsoft.azure.kusto.ingest.result.IngestionStatus;
+import com.microsoft.azure.kusto.ingest.result.OperationStatus;
+import com.microsoft.azure.kusto.ingest.source.StreamSourceInfo;
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.nifi.annotation.behavior.ReadsAttribute;
+import org.apache.nifi.annotation.behavior.ReadsAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnEnabled;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.controller.AbstractControllerService;
+import org.apache.nifi.controller.ConfigurationContext;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.reporting.InitializationException;
+
+import java.io.IOException;
+import java.net.URISyntaxException;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.List;
+import java.util.Objects;
+
+@Tags({"Azure", "ADX", "Kusto", "ingest", "azure"})
+@CapabilityDescription("Sends batches of flowfile content or stream flowfile content to an Azure ADX cluster.")
+@ReadsAttributes({
+ @ReadsAttribute(attribute = "AUTH_STRATEGY", description = "The strategy/method to authenticate against Azure Active Directory, either 'application' or 'managed_identity'."),
+ @ReadsAttribute(attribute = "APP_ID", description = "Specifies Azure application id for accessing the ADX-Cluster."),
+ @ReadsAttribute(attribute = "APP_KEY", description = "Specifies Azure application key for accessing the ADX-Cluster."),
+ @ReadsAttribute(attribute = "APP_TENANT", description = "Azure application tenant for accessing the ADX-Cluster."),
+ @ReadsAttribute(attribute = "CLUSTER_URL", description = "Endpoint of ADX cluster. This is required only when streaming data to ADX cluster is enabled."),
+})
+public class StandardKustoIngestService extends AbstractControllerService implements KustoIngestService {
+
+ public static final PropertyDescriptor AUTHENTICATION_STRATEGY = new PropertyDescriptor
+ .Builder().name("Authentication Strategy")
+ .displayName("Authentication Strategy")
+ .description("Authentication method for access to Azure Data Explorer")
+ .required(true)
+ .defaultValue(KustoAuthenticationStrategy.MANAGED_IDENTITY.getValue())
+ .allowableValues(KustoAuthenticationStrategy.class)
+ .build();
+
+ public static final PropertyDescriptor APPLICATION_CLIENT_ID = new PropertyDescriptor
+ .Builder().name("Application Client ID")
+ .displayName("Application Client ID")
+ .description("Azure Data Explorer Application Client Identifier for Authentication")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+
+ public static final PropertyDescriptor APPLICATION_KEY = new PropertyDescriptor
+ .Builder().name("Application Key")
+ .displayName("Application Key")
+ .description("Azure Data Explorer Application Key for Authentication")
+ .required(true)
+ .sensitive(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .dependsOn(AUTHENTICATION_STRATEGY, KustoAuthenticationStrategy.APPLICATION_CREDENTIALS.getValue())
+ .build();
+
+ public static final PropertyDescriptor APPLICATION_TENANT_ID = new PropertyDescriptor.Builder()
+ .name("Application Tenant ID")
+ .displayName("Application Tenant ID")
+ .description("Azure Data Explorer Application Tenant Identifier for Authentication")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .dependsOn(AUTHENTICATION_STRATEGY, KustoAuthenticationStrategy.APPLICATION_CREDENTIALS.getValue())
+ .build();
+
+ public static final PropertyDescriptor CLUSTER_URI = new PropertyDescriptor
+ .Builder().name("Cluster URI")
+ .displayName("Cluster URI")
+ .description("Azure Data Explorer Cluster URI")
+ .required(true)
+ .addValidator(StandardValidators.URL_VALIDATOR)
+ .build();
+
+ private static final List<PropertyDescriptor> PROPERTY_DESCRIPTORS = Collections.unmodifiableList(Arrays.asList(
+ AUTHENTICATION_STRATEGY,
+ APPLICATION_CLIENT_ID,
+ APPLICATION_KEY,
+ APPLICATION_TENANT_ID,
+ CLUSTER_URI));
+
+ public static final Pair<String, String> NIFI_SINK = Pair.of("processor", "nifi-sink");
+
+ private volatile QueuedIngestClient queuedIngestClient;
+
+ private volatile ManagedStreamingIngestClient managedStreamingIngestClient;
+
+ private volatile Client executionClient;
+
+ @Override
+ public List<PropertyDescriptor> getSupportedPropertyDescriptors() {
+ return PROPERTY_DESCRIPTORS;
+ }
+
+ /**
+ * @param context the configuration context
+ * @throws InitializationException if unable to create a database connection
+ */
+ @OnEnabled
+ public void onEnabled(final ConfigurationContext context) throws ProcessException, URISyntaxException {
+
+ getLogger().info("Starting Azure ADX Connection Service...");
+ final String applicationClientId = context.getProperty(APPLICATION_CLIENT_ID).getValue();
+ final String applicationKey = context.getProperty(APPLICATION_KEY).getValue();
+ final String applicationTenantId = context.getProperty(APPLICATION_TENANT_ID).getValue();
+ final String clusterUri = context.getProperty(CLUSTER_URI).getValue();
+ final KustoAuthenticationStrategy kustoAuthenticationStrategy = KustoAuthenticationStrategy.valueOf(context.getProperty(AUTHENTICATION_STRATEGY).getValue());
+
+ if (this.queuedIngestClient == null) {
+ this.queuedIngestClient = createKustoQueuedIngestClient(clusterUri, applicationClientId, applicationKey, applicationTenantId, kustoAuthenticationStrategy);
+ }
+
+ if (this.managedStreamingIngestClient == null) {
+ this.managedStreamingIngestClient = createKustoStreamingIngestClient(clusterUri, applicationClientId, applicationKey, applicationTenantId, kustoAuthenticationStrategy);
+ }
+
+ if (this.executionClient == null) {
+ this.executionClient = createKustoExecutionClient(clusterUri, applicationClientId, applicationKey, applicationTenantId, kustoAuthenticationStrategy);
+ }
+
+ }
+
+ @OnStopped
+ public final void onStopped() {
+ if (this.queuedIngestClient != null) {
+ try {
+ this.queuedIngestClient.close();
+ } catch (IOException e) {
+ getLogger().error("Closing Azure ADX Queued Ingest Client failed ", e);
+ } finally {
+ this.queuedIngestClient = null;
+ }
+ }
+ if (this.managedStreamingIngestClient != null) {
+ try {
+ this.managedStreamingIngestClient.close();
+ } catch (IOException e) {
+ getLogger().error("Closing Azure ADX Managed Streaming Ingest Client failed", e);
+ } finally {
+ this.managedStreamingIngestClient = null;
+ }
+ }
+ if (this.executionClient != null) {
+ try {
+ this.executionClient.close();
+ } catch (IOException e) {
+ getLogger().error("Closing Azure ADX Execution Client failed", e);
+ } finally {
+ this.executionClient = null;
+ }
+ }
+ }
+
+
+ protected QueuedIngestClient createKustoQueuedIngestClient(final String clusterUrl,
+ final String appId,
+ final String appKey,
+ final String appTenant,
+ final KustoAuthenticationStrategy kustoAuthStrategy) throws URISyntaxException {
+ ConnectionStringBuilder ingestConnectionStringBuilder = createKustoEngineConnectionString(clusterUrl, appId, appKey, appTenant, kustoAuthStrategy);
+ return IngestClientFactory.createClient(ingestConnectionStringBuilder);
+ }
+
+ protected ManagedStreamingIngestClient createKustoStreamingIngestClient(final String clusterUrl,
+ final String appId,
+ final String appKey,
+ final String appTenant,
+ final KustoAuthenticationStrategy kustoAuthStrategy) throws URISyntaxException {
+ ConnectionStringBuilder ingestConnectionStringBuilder = createKustoEngineConnectionString(clusterUrl, appId, appKey, appTenant, kustoAuthStrategy);
+ ConnectionStringBuilder streamingConnectionStringBuilder = createKustoEngineConnectionString(clusterUrl, appId, appKey, appTenant, kustoAuthStrategy);
+ return IngestClientFactory.createManagedStreamingIngestClient(ingestConnectionStringBuilder, streamingConnectionStringBuilder);
+ }
+
+ public KustoIngestionResult ingestData(KustoIngestionRequest kustoIngestionRequest) throws URISyntaxException {
+ StreamSourceInfo info = new StreamSourceInfo(kustoIngestionRequest.getInputStream());
+ //ingest data
+ IngestionResult ingestionResult;
+ try {
+ if (kustoIngestionRequest.isStreamingEnabled()) {
+ ingestionResult = managedStreamingIngestClient.ingestFromStream(info, kustoIngestionRequest.getIngestionProperties());
+ } else {
+ ingestionResult = queuedIngestClient.ingestFromStream(info, kustoIngestionRequest.getIngestionProperties());
+ }
+ if (kustoIngestionRequest.pollOnIngestionStatus()) {
+ List<IngestionStatus> statuses;
+ long startTime = System.currentTimeMillis();
+ long timeoutMillis = 600 * 1800; // 10 Minutes timeout
Review Comment:
This value should be changed to a configurable property instead of hard-coded within this method. Recommend naming the property something along the lines of `Processing Timeout`.
##########
nifi-nar-bundles/nifi-azure-bundle/nifi-azure-processors/src/main/java/org/apache/nifi/services/azure/data/explorer/StandardKustoIngestService.java:
##########
@@ -0,0 +1,293 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.services.azure.data.explorer;
+
+import com.microsoft.azure.kusto.data.Client;
+import com.microsoft.azure.kusto.data.ClientFactory;
+import com.microsoft.azure.kusto.data.KustoResultSetTable;
+import com.microsoft.azure.kusto.data.auth.ConnectionStringBuilder;
+import com.microsoft.azure.kusto.data.exceptions.DataClientException;
+import com.microsoft.azure.kusto.data.exceptions.DataServiceException;
+import com.microsoft.azure.kusto.ingest.IngestClientFactory;
+import com.microsoft.azure.kusto.ingest.ManagedStreamingIngestClient;
+import com.microsoft.azure.kusto.ingest.QueuedIngestClient;
+import com.microsoft.azure.kusto.ingest.exceptions.IngestionClientException;
+import com.microsoft.azure.kusto.ingest.exceptions.IngestionServiceException;
+import com.microsoft.azure.kusto.ingest.result.IngestionResult;
+import com.microsoft.azure.kusto.ingest.result.IngestionStatus;
+import com.microsoft.azure.kusto.ingest.result.OperationStatus;
+import com.microsoft.azure.kusto.ingest.source.StreamSourceInfo;
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.nifi.annotation.behavior.ReadsAttribute;
+import org.apache.nifi.annotation.behavior.ReadsAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnEnabled;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.controller.AbstractControllerService;
+import org.apache.nifi.controller.ConfigurationContext;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.reporting.InitializationException;
+
+import java.io.IOException;
+import java.net.URISyntaxException;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.List;
+import java.util.Objects;
+
+@Tags({"Azure", "ADX", "Kusto", "ingest", "azure"})
+@CapabilityDescription("Sends batches of flowfile content or stream flowfile content to an Azure ADX cluster.")
+@ReadsAttributes({
+ @ReadsAttribute(attribute = "AUTH_STRATEGY", description = "The strategy/method to authenticate against Azure Active Directory, either 'application' or 'managed_identity'."),
+ @ReadsAttribute(attribute = "APP_ID", description = "Specifies Azure application id for accessing the ADX-Cluster."),
+ @ReadsAttribute(attribute = "APP_KEY", description = "Specifies Azure application key for accessing the ADX-Cluster."),
+ @ReadsAttribute(attribute = "APP_TENANT", description = "Azure application tenant for accessing the ADX-Cluster."),
+ @ReadsAttribute(attribute = "CLUSTER_URL", description = "Endpoint of ADX cluster. This is required only when streaming data to ADX cluster is enabled."),
+})
+public class StandardKustoIngestService extends AbstractControllerService implements KustoIngestService {
+
+ public static final PropertyDescriptor AUTHENTICATION_STRATEGY = new PropertyDescriptor
+ .Builder().name("Authentication Strategy")
+ .displayName("Authentication Strategy")
+ .description("Authentication method for access to Azure Data Explorer")
+ .required(true)
+ .defaultValue(KustoAuthenticationStrategy.MANAGED_IDENTITY.getValue())
+ .allowableValues(KustoAuthenticationStrategy.class)
+ .build();
+
+ public static final PropertyDescriptor APPLICATION_CLIENT_ID = new PropertyDescriptor
+ .Builder().name("Application Client ID")
+ .displayName("Application Client ID")
+ .description("Azure Data Explorer Application Client Identifier for Authentication")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+
+ public static final PropertyDescriptor APPLICATION_KEY = new PropertyDescriptor
+ .Builder().name("Application Key")
+ .displayName("Application Key")
+ .description("Azure Data Explorer Application Key for Authentication")
+ .required(true)
+ .sensitive(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .dependsOn(AUTHENTICATION_STRATEGY, KustoAuthenticationStrategy.APPLICATION_CREDENTIALS.getValue())
+ .build();
+
+ public static final PropertyDescriptor APPLICATION_TENANT_ID = new PropertyDescriptor.Builder()
+ .name("Application Tenant ID")
+ .displayName("Application Tenant ID")
+ .description("Azure Data Explorer Application Tenant Identifier for Authentication")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .dependsOn(AUTHENTICATION_STRATEGY, KustoAuthenticationStrategy.APPLICATION_CREDENTIALS.getValue())
+ .build();
+
+ public static final PropertyDescriptor CLUSTER_URI = new PropertyDescriptor
+ .Builder().name("Cluster URI")
+ .displayName("Cluster URI")
+ .description("Azure Data Explorer Cluster URI")
+ .required(true)
+ .addValidator(StandardValidators.URL_VALIDATOR)
+ .build();
+
+ private static final List<PropertyDescriptor> PROPERTY_DESCRIPTORS = Collections.unmodifiableList(Arrays.asList(
+ AUTHENTICATION_STRATEGY,
+ APPLICATION_CLIENT_ID,
+ APPLICATION_KEY,
+ APPLICATION_TENANT_ID,
+ CLUSTER_URI));
+
+ public static final Pair<String, String> NIFI_SINK = Pair.of("processor", "nifi-sink");
+
+ private volatile QueuedIngestClient queuedIngestClient;
+
+ private volatile ManagedStreamingIngestClient managedStreamingIngestClient;
+
+ private volatile Client executionClient;
+
+ @Override
+ public List<PropertyDescriptor> getSupportedPropertyDescriptors() {
+ return PROPERTY_DESCRIPTORS;
+ }
+
+ /**
+ * @param context the configuration context
+ * @throws InitializationException if unable to create a database connection
+ */
+ @OnEnabled
+ public void onEnabled(final ConfigurationContext context) throws ProcessException, URISyntaxException {
+
+ getLogger().info("Starting Azure ADX Connection Service...");
+ final String applicationClientId = context.getProperty(APPLICATION_CLIENT_ID).getValue();
+ final String applicationKey = context.getProperty(APPLICATION_KEY).getValue();
+ final String applicationTenantId = context.getProperty(APPLICATION_TENANT_ID).getValue();
+ final String clusterUri = context.getProperty(CLUSTER_URI).getValue();
+ final KustoAuthenticationStrategy kustoAuthenticationStrategy = KustoAuthenticationStrategy.valueOf(context.getProperty(AUTHENTICATION_STRATEGY).getValue());
+
+ if (this.queuedIngestClient == null) {
+ this.queuedIngestClient = createKustoQueuedIngestClient(clusterUri, applicationClientId, applicationKey, applicationTenantId, kustoAuthenticationStrategy);
+ }
+
+ if (this.managedStreamingIngestClient == null) {
+ this.managedStreamingIngestClient = createKustoStreamingIngestClient(clusterUri, applicationClientId, applicationKey, applicationTenantId, kustoAuthenticationStrategy);
+ }
+
+ if (this.executionClient == null) {
+ this.executionClient = createKustoExecutionClient(clusterUri, applicationClientId, applicationKey, applicationTenantId, kustoAuthenticationStrategy);
+ }
+
+ }
+
+ @OnStopped
+ public final void onStopped() {
+ if (this.queuedIngestClient != null) {
+ try {
+ this.queuedIngestClient.close();
+ } catch (IOException e) {
+ getLogger().error("Closing Azure ADX Queued Ingest Client failed ", e);
+ } finally {
+ this.queuedIngestClient = null;
+ }
+ }
+ if (this.managedStreamingIngestClient != null) {
+ try {
+ this.managedStreamingIngestClient.close();
+ } catch (IOException e) {
+ getLogger().error("Closing Azure ADX Managed Streaming Ingest Client failed", e);
+ } finally {
+ this.managedStreamingIngestClient = null;
+ }
+ }
+ if (this.executionClient != null) {
+ try {
+ this.executionClient.close();
+ } catch (IOException e) {
+ getLogger().error("Closing Azure ADX Execution Client failed", e);
+ } finally {
+ this.executionClient = null;
+ }
+ }
+ }
+
+
+ protected QueuedIngestClient createKustoQueuedIngestClient(final String clusterUrl,
+ final String appId,
+ final String appKey,
+ final String appTenant,
+ final KustoAuthenticationStrategy kustoAuthStrategy) throws URISyntaxException {
+ ConnectionStringBuilder ingestConnectionStringBuilder = createKustoEngineConnectionString(clusterUrl, appId, appKey, appTenant, kustoAuthStrategy);
+ return IngestClientFactory.createClient(ingestConnectionStringBuilder);
+ }
+
+ protected ManagedStreamingIngestClient createKustoStreamingIngestClient(final String clusterUrl,
+ final String appId,
+ final String appKey,
+ final String appTenant,
+ final KustoAuthenticationStrategy kustoAuthStrategy) throws URISyntaxException {
+ ConnectionStringBuilder ingestConnectionStringBuilder = createKustoEngineConnectionString(clusterUrl, appId, appKey, appTenant, kustoAuthStrategy);
+ ConnectionStringBuilder streamingConnectionStringBuilder = createKustoEngineConnectionString(clusterUrl, appId, appKey, appTenant, kustoAuthStrategy);
+ return IngestClientFactory.createManagedStreamingIngestClient(ingestConnectionStringBuilder, streamingConnectionStringBuilder);
+ }
+
+ public KustoIngestionResult ingestData(KustoIngestionRequest kustoIngestionRequest) throws URISyntaxException {
+ StreamSourceInfo info = new StreamSourceInfo(kustoIngestionRequest.getInputStream());
+ //ingest data
+ IngestionResult ingestionResult;
+ try {
+ if (kustoIngestionRequest.isStreamingEnabled()) {
+ ingestionResult = managedStreamingIngestClient.ingestFromStream(info, kustoIngestionRequest.getIngestionProperties());
+ } else {
+ ingestionResult = queuedIngestClient.ingestFromStream(info, kustoIngestionRequest.getIngestionProperties());
+ }
+ if (kustoIngestionRequest.pollOnIngestionStatus()) {
+ List<IngestionStatus> statuses;
+ long startTime = System.currentTimeMillis();
+ long timeoutMillis = 600 * 1800; // 10 Minutes timeout
+ while (true) {
Review Comment:
Instead of defining an infinite loop, recommend changing the condition to evaluate the processing timeout.
##########
nifi-nar-bundles/nifi-azure-bundle/nifi-azure-processors/src/main/java/org/apache/nifi/processors/azure/data/explorer/PutAzureDataExplorer.java:
##########
@@ -0,0 +1,399 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.azure.data.explorer;
+
+import com.microsoft.azure.kusto.data.KustoResultSetTable;
+import com.microsoft.azure.kusto.ingest.IngestionMapping;
+import com.microsoft.azure.kusto.ingest.IngestionProperties;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.nifi.annotation.behavior.ReadsAttribute;
+import org.apache.nifi.annotation.behavior.ReadsAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.expression.ExpressionLanguageScope;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.services.azure.data.explorer.KustoIngestService;
+import org.apache.nifi.services.azure.data.explorer.KustoIngestionRequest;
+import org.apache.nifi.services.azure.data.explorer.KustoIngestionResult;
+import org.apache.nifi.services.azure.data.explorer.KustoQueryResponse;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.net.URISyntaxException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+
+@Tags({"Azure", "Kusto", "ADX", "Explorer", "Data"})
+@CapabilityDescription("The PutAzureDataExplorer acts as a ADX sink connector which sends flowFiles using the ADX-Service to the provided Azure Data" +
+ "Explorer Ingest Endpoint. The data can be sent through queued ingestion or streaming ingestion to the Azure Data Explorer cluster.")
+@ReadsAttributes({
+ @ReadsAttribute(attribute = "DB_NAME", description = "Specifies the name of the ADX database where the data needs to be stored."),
+ @ReadsAttribute(attribute = "TABLE_NAME", description = "Specifies the name of the ADX table where the data needs to be stored."),
+ @ReadsAttribute(attribute = "MAPPING_NAME", description = "Specifies the name of the mapping responsible for storing the data in appropriate columns."),
+ @ReadsAttribute(attribute = "FLUSH_IMMEDIATE", description = "In case of queued ingestion, this property determines whether the data should be flushed immediately to the ingest endpoint."),
+ @ReadsAttribute(attribute = "DATA_FORMAT", description = "Specifies the format of data that is send to Azure Data Explorer."),
+ @ReadsAttribute(attribute = "IGNORE_FIRST_RECORD", description = "Specifies whether we want to ignore ingestion of first record. " +
+ "This is primarily applicable for csv files. Default is set to NO"),
+ @ReadsAttribute(attribute = "POLL_ON_INGESTION_STATUS", description = "Specifies whether we want to poll on ingestion result during ingestion into ADX." +
+ "In case of applications that need high throughput it is recommended to keep the default value as false. Default is set to false." +
+ "This property should be set to true during Queued Ingestion for near realtime micro-batches of data that require acknowledgement of ingestion status.")
+})
+public class PutAzureDataExplorer extends AbstractProcessor {
+
+ public static final String FETCH_TABLE_COMMAND = "%s | count";
+ public static final String STREAMING_POLICY_SHOW_COMMAND = ".show %s %s policy streamingingestion";
+ public static final String DATABASE = "database";
+
+ private List<PropertyDescriptor> descriptors;
+ private Set<Relationship> relationships;
+ private transient KustoIngestService service;
+ private boolean isStreamingEnabled;
+ private boolean pollOnIngestionStatus;
+
+ public static final AllowableValue AVRO = new AllowableValue(
+ IngestionProperties.DataFormat.AVRO.getKustoValue(), IngestionProperties.DataFormat.AVRO.getKustoValue(),
+ "An Avro format with support for logical types and for the snappy compression codec");
+
+ public static final AllowableValue APACHEAVRO = new AllowableValue(
+ IngestionProperties.DataFormat.APACHEAVRO.getKustoValue(), IngestionProperties.DataFormat.APACHEAVRO.getKustoValue(),
+ "An Avro format with support for logical types and for the snappy compression codec.");
+
+ public static final AllowableValue CSV = new AllowableValue(
+ IngestionProperties.DataFormat.CSV.name(), IngestionProperties.DataFormat.CSV.getKustoValue(),
+ "A text file with comma-separated values (,). For more information, see RFC 4180: Common Format " +
+ "and MIME Type for Comma-Separated Values (CSV) Files.");
+
+ public static final AllowableValue JSON = new AllowableValue(
+ IngestionProperties.DataFormat.JSON.name(), IngestionProperties.DataFormat.JSON.getKustoValue(),
+ "A text file containing JSON objects separated by \\n or \\r\\n. For more information, " +
+ "see JSON Lines (JSONL).");
+
+ public static final AllowableValue MULTIJSON = new AllowableValue(
+ IngestionProperties.DataFormat.MULTIJSON.name(), IngestionProperties.DataFormat.MULTIJSON.getKustoValue(),
+ "A text file containing a JSON array of property containers (each representing a record) or any " +
+ "number of property containers separated by spaces, \\n or \\r\\n. Each property container may be " +
+ "spread across multiple lines. This format is preferable to JSON unless the data is not property " +
+ "containers.");
+
+ public static final AllowableValue ORC = new AllowableValue(
+ IngestionProperties.DataFormat.ORC.name(), IngestionProperties.DataFormat.ORC.getKustoValue(), "An ORC file.");
+
+ public static final AllowableValue PARQUET = new AllowableValue(
+ IngestionProperties.DataFormat.PARQUET.name(), IngestionProperties.DataFormat.PARQUET.getKustoValue(), "A parquet file.");
+
+ public static final AllowableValue PSV = new AllowableValue(
+ IngestionProperties.DataFormat.PSV.name(), IngestionProperties.DataFormat.PSV.getKustoValue(), "A text file with values separated by vertical bars (|).");
+
+ public static final AllowableValue SCSV = new AllowableValue(
+ IngestionProperties.DataFormat.SCSV.name(), IngestionProperties.DataFormat.SCSV.getKustoValue(), "A text file with values separated by semicolons (;).");
+
+ public static final AllowableValue SOHSV = new AllowableValue(
+ IngestionProperties.DataFormat.SOHSV.name(), IngestionProperties.DataFormat.SOHSV.getKustoValue(),
+ "A text file with SOH-separated values. (SOH is the ASCII code point 1. " +
+ "This format is used by Hive in HDInsight).");
+
+ public static final AllowableValue TSV = new AllowableValue(
+ IngestionProperties.DataFormat.TSV.name(), IngestionProperties.DataFormat.TSV.getKustoValue(), "A text file with tab delimited values (\\t).");
+
+ public static final AllowableValue TSVE = new AllowableValue(
+ IngestionProperties.DataFormat.TSVE.name(), IngestionProperties.DataFormat.TSVE.getKustoValue(),
+ "A text file with tab-delimited values (\\t). A backslash (\\) is used as escape character.");
+
+ public static final AllowableValue TXT = new AllowableValue(
+ IngestionProperties.DataFormat.TXT.name(), IngestionProperties.DataFormat.TXT.getKustoValue(),
+ "A text file with lines separated by \\n. Empty lines are skipped.");
+
+ public static final AllowableValue IGNORE_FIRST_RECORD_YES = new AllowableValue(
+ "YES", "YES",
+ "Ignore first record during ingestion");
+
+ public static final AllowableValue IGNORE_FIRST_RECORD_NO = new AllowableValue(
+ "NO", "NO",
+ "Do not ignore first record during ingestion");
+
+ public static final PropertyDescriptor DATABASE_NAME = new PropertyDescriptor.Builder()
+ .name("Database Name")
+ .displayName("Database Name")
+ .description("Azure Data Explorer Database Name for querying")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+ .build();
+
+ public static final PropertyDescriptor TABLE_NAME = new PropertyDescriptor.Builder()
+ .name("Table Name")
+ .displayName("Table Name")
+ .description("Azure Data Explorer Table Name for ingesting data")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+ .build();
+
+ public static final PropertyDescriptor MAPPING_NAME = new PropertyDescriptor
+ .Builder().name("Mapping name")
+ .displayName("Mapping name")
+ .description("The name of the mapping responsible for storing the data in the appropriate columns.")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+
+ public static final PropertyDescriptor IS_STREAMING_ENABLED = new PropertyDescriptor
+ .Builder().name("Is Streaming enabled")
+ .displayName("Is Streaming enabled")
+ .description("This property determines whether we want to stream data to ADX.")
+ .required(false)
+ .allowableValues("true", "false")
+ .addValidator(StandardValidators.BOOLEAN_VALIDATOR)
+ .defaultValue("false")
+ .build();
+
+ public static final PropertyDescriptor POLL_ON_INGESTION_STATUS = new PropertyDescriptor
+ .Builder().name("Poll on ingestion status")
+ .displayName("Whether to Poll on ingestion status")
+ .description("This property determines whether we want to poll on ingestion status after an ingestion to ADX is completed")
+ .required(false)
+ .allowableValues("true", "false")
+ .addValidator(StandardValidators.BOOLEAN_VALIDATOR)
+ .defaultValue("false")
+ .build();
+ public static final PropertyDescriptor ADX_SERVICE = new PropertyDescriptor
+ .Builder().name("Kusto Ingest Service")
+ .displayName("Kusto Ingest Service")
+ .description("Azure Data Explorer Kusto Ingest Service")
+ .required(true)
+ .identifiesControllerService(KustoIngestService.class)
+ .build();
+
+ public static final Relationship SUCCESS = new Relationship.Builder()
+ .name("success")
+ .description("Relationship for success")
+ .build();
+ public static final Relationship FAILURE = new Relationship.Builder()
+ .name("failure")
+ .description("Relationship for failure")
+ .build();
+
+ static final PropertyDescriptor DATA_FORMAT = new PropertyDescriptor.Builder()
+ .name("Data Format")
+ .displayName("Data Format")
+ .description("The format of the data that is sent to ADX.")
+ .required(true)
+ .allowableValues(AVRO, APACHEAVRO, CSV, JSON, MULTIJSON, ORC, PARQUET, PSV, SCSV, SOHSV, TSV, TSVE, TXT)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+
+ static final PropertyDescriptor IGNORE_FIRST_RECORD = new PropertyDescriptor.Builder()
+ .name("Ingestion Ignore First Record")
+ .displayName("Ingestion Ignore First Record")
+ .description("Defines whether ignore first record while ingestion.")
+ .required(false)
+ .allowableValues(IGNORE_FIRST_RECORD_YES, IGNORE_FIRST_RECORD_NO)
+ .defaultValue(IGNORE_FIRST_RECORD_NO.getValue())
+ .build();
+
+ @Override
+ protected void init(final ProcessorInitializationContext context) {
+ final List<PropertyDescriptor> descriptorList = new ArrayList<>();
+ descriptorList.add(ADX_SERVICE);
+ descriptorList.add(DATABASE_NAME);
+ descriptorList.add(TABLE_NAME);
+ descriptorList.add(MAPPING_NAME);
+ descriptorList.add(DATA_FORMAT);
+ descriptorList.add(IGNORE_FIRST_RECORD);
+ descriptorList.add(IS_STREAMING_ENABLED);
+ descriptorList.add(POLL_ON_INGESTION_STATUS);
+ this.descriptors = Collections.unmodifiableList(descriptorList);
+
+ final Set<Relationship> relationshipSet = new HashSet<>();
+ relationshipSet.add(SUCCESS);
+ relationshipSet.add(FAILURE);
+ this.relationships = Collections.unmodifiableSet(relationshipSet);
+ }
+
+ @Override
+ public Set<Relationship> getRelationships() {
+ return this.relationships;
+ }
+
+ @Override
+ public final List<PropertyDescriptor> getSupportedPropertyDescriptors() {
+ return descriptors;
+ }
+
+ @OnScheduled
+ public void onScheduled(final ProcessContext context) {
+ service = context.getProperty(ADX_SERVICE).asControllerService(KustoIngestService.class);
+ if (checkIfIngestorRoleDoesntExist(context.getProperty(DATABASE_NAME).getValue(), context.getProperty(TABLE_NAME).getValue())) {
+ getLogger().error("User might not have ingestor privileges, table validation will be skipped for all table mappings.");
+ throw new ProcessException("User might not have ingestor privileges, table validation will be skipped for all table mappings. ");
+ }
+ isStreamingEnabled = context.getProperty(IS_STREAMING_ENABLED).evaluateAttributeExpressions().asBoolean();
+ if (isStreamingEnabled) {
+ checkIfStreamingPolicyIsEnabledInADX(context.getProperty(DATABASE_NAME).getValue(), context.getProperty(DATABASE_NAME).getValue());
+ }
+ }
+
+ @Override
+ public void onTrigger(final ProcessContext context, final ProcessSession session) throws ProcessException {
+ FlowFile flowFile = session.get();
+ if (flowFile == null) {
+ context.yield();
+ return;
+ }
+
+ IngestionProperties ingestionProperties = new IngestionProperties(context.getProperty(DATABASE_NAME).getValue(),
+ context.getProperty(TABLE_NAME).getValue());
+
+ IngestionMapping.IngestionMappingKind ingestionMappingKind = null;
+
+ isStreamingEnabled = context.getProperty(IS_STREAMING_ENABLED).evaluateAttributeExpressions().asBoolean();
+ pollOnIngestionStatus = context.getProperty(POLL_ON_INGESTION_STATUS).evaluateAttributeExpressions().asBoolean();
+
+ switch (IngestionProperties.DataFormat.valueOf(context.getProperty(DATA_FORMAT).getValue())) {
+ case AVRO:
+ ingestionProperties.setDataFormat(IngestionProperties.DataFormat.AVRO);
+ ingestionMappingKind = IngestionProperties.DataFormat.AVRO.getIngestionMappingKind();
+ break;
+ case APACHEAVRO:
+ ingestionProperties.setDataFormat(IngestionProperties.DataFormat.APACHEAVRO);
+ ingestionMappingKind = IngestionProperties.DataFormat.APACHEAVRO.getIngestionMappingKind();
+ break;
+ case CSV:
+ ingestionProperties.setDataFormat(IngestionProperties.DataFormat.CSV);
+ ingestionMappingKind = IngestionProperties.DataFormat.CSV.getIngestionMappingKind();
+ break;
+ case JSON:
+ ingestionProperties.setDataFormat(IngestionProperties.DataFormat.JSON);
+ ingestionMappingKind = IngestionProperties.DataFormat.JSON.getIngestionMappingKind();
+ break;
+ case MULTIJSON:
+ ingestionProperties.setDataFormat(IngestionProperties.DataFormat.MULTIJSON);
+ ingestionMappingKind = IngestionProperties.DataFormat.MULTIJSON.getIngestionMappingKind();
+ break;
+ case ORC:
+ ingestionProperties.setDataFormat(IngestionProperties.DataFormat.ORC);
+ ingestionMappingKind = IngestionProperties.DataFormat.ORC.getIngestionMappingKind();
+ break;
+ case PARQUET:
+ ingestionProperties.setDataFormat(IngestionProperties.DataFormat.PARQUET);
+ ingestionMappingKind = IngestionProperties.DataFormat.PARQUET.getIngestionMappingKind();
+ break;
+ case PSV:
+ ingestionProperties.setDataFormat(IngestionProperties.DataFormat.PSV);
+ ingestionMappingKind = IngestionProperties.DataFormat.PSV.getIngestionMappingKind();
+ break;
+ case SCSV:
+ ingestionProperties.setDataFormat(IngestionProperties.DataFormat.SCSV);
+ ingestionMappingKind = IngestionProperties.DataFormat.SCSV.getIngestionMappingKind();
+ break;
+ case SOHSV:
+ ingestionProperties.setDataFormat(IngestionProperties.DataFormat.SOHSV);
+ ingestionMappingKind = IngestionProperties.DataFormat.SOHSV.getIngestionMappingKind();
+ break;
+ case TSV:
+ ingestionProperties.setDataFormat(IngestionProperties.DataFormat.TSV);
+ ingestionMappingKind = IngestionProperties.DataFormat.TSV.getIngestionMappingKind();
+ break;
+ case TSVE:
+ ingestionProperties.setDataFormat(IngestionProperties.DataFormat.TSVE);
+ ingestionMappingKind = IngestionProperties.DataFormat.TSVE.getIngestionMappingKind();
+ break;
+ case TXT:
+ ingestionProperties.setDataFormat(IngestionProperties.DataFormat.TXT);
+ ingestionMappingKind = IngestionProperties.DataFormat.TXT.getIngestionMappingKind();
+ break;
+ }
+
+ if (StringUtils.isNotEmpty(context.getProperty(MAPPING_NAME).getValue()) && ingestionMappingKind != null) {
+ ingestionProperties.setIngestionMapping(context.getProperty(MAPPING_NAME).getValue(), ingestionMappingKind);
+ }
+
+ ingestionProperties.setReportLevel(IngestionProperties.IngestionReportLevel.FAILURES_AND_SUCCESSES);
+ ingestionProperties.setReportMethod(IngestionProperties.IngestionReportMethod.TABLE);
+ ingestionProperties.setFlushImmediately(false);
+ ingestionProperties.setIgnoreFirstRecord(StringUtils.equalsIgnoreCase(context.getProperty(IGNORE_FIRST_RECORD).getValue(), IGNORE_FIRST_RECORD_YES.getValue()));
+
+ boolean isError = false;
+
+ // ingestion into ADX
+ try (final InputStream inputStream = session.read(flowFile)) {
+ StringBuilder ingestLogString = new StringBuilder().append("Ingesting with: ")
+ .append("dataFormat - ").append(ingestionProperties.getDataFormat()).append("|")
+ .append("ingestionMapping - ").append(ingestionProperties.getIngestionMapping().getIngestionMappingReference()).append("|")
+ .append("reportLevel - ").append(ingestionProperties.getReportLevel()).append("|")
+ .append("reportMethod - ").append(ingestionProperties.getReportMethod()).append("|")
+ .append("databaseName - ").append(ingestionProperties.getDatabaseName()).append("|")
+ .append("tableName - ").append(ingestionProperties.getTableName()).append("|")
+ .append("flushImmediately - ").append(ingestionProperties.getFlushImmediately());
+ getLogger().info(ingestLogString.toString());
+
+ KustoIngestionResult result = service.ingestData(new KustoIngestionRequest(isStreamingEnabled, pollOnIngestionStatus, inputStream, ingestionProperties));
+ getLogger().info("Poll on Ingestion Status {} and Operation status: {} ", pollOnIngestionStatus, result.toString());
+
+ if (result == KustoIngestionResult.SUCCEEDED) {
+ getLogger().info("Operation status Succeeded - {}", result.toString());
+ } else if (result == KustoIngestionResult.FAILED) {
+ getLogger().error("Operation status Error - {}", result.toString());
+ isError = true;
+ } else if (result == KustoIngestionResult.PARTIALLY_SUCCEEDED) {
+ getLogger().error("Operation status Partially succeeded - {}", result.toString());
+ isError = true;
+ }
+
+ } catch (IOException | URISyntaxException e) {
+ getLogger().error("Exception occurred while ingesting data into ADX ", e);
+ isError = true;
+ }
+
+ if (isError) {
+ getLogger().error("Process failed - {}");
+ session.transfer(flowFile, FAILURE);
+ } else {
+ getLogger().info("Process succeeded - {}");
Review Comment:
The FlowFile is missing from this log:
```suggestion
getLogger().info("Ingest processing completed {}", flowFile);
```
##########
nifi-nar-bundles/nifi-azure-bundle/nifi-azure-processors/src/main/java/org/apache/nifi/processors/azure/data/explorer/PutAzureDataExplorer.java:
##########
@@ -0,0 +1,399 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.azure.data.explorer;
+
+import com.microsoft.azure.kusto.data.KustoResultSetTable;
+import com.microsoft.azure.kusto.ingest.IngestionMapping;
+import com.microsoft.azure.kusto.ingest.IngestionProperties;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.nifi.annotation.behavior.ReadsAttribute;
+import org.apache.nifi.annotation.behavior.ReadsAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.expression.ExpressionLanguageScope;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.services.azure.data.explorer.KustoIngestService;
+import org.apache.nifi.services.azure.data.explorer.KustoIngestionRequest;
+import org.apache.nifi.services.azure.data.explorer.KustoIngestionResult;
+import org.apache.nifi.services.azure.data.explorer.KustoQueryResponse;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.net.URISyntaxException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+
+@Tags({"Azure", "Kusto", "ADX", "Explorer", "Data"})
+@CapabilityDescription("The PutAzureDataExplorer acts as a ADX sink connector which sends flowFiles using the ADX-Service to the provided Azure Data" +
+ "Explorer Ingest Endpoint. The data can be sent through queued ingestion or streaming ingestion to the Azure Data Explorer cluster.")
+@ReadsAttributes({
+ @ReadsAttribute(attribute = "DB_NAME", description = "Specifies the name of the ADX database where the data needs to be stored."),
+ @ReadsAttribute(attribute = "TABLE_NAME", description = "Specifies the name of the ADX table where the data needs to be stored."),
+ @ReadsAttribute(attribute = "MAPPING_NAME", description = "Specifies the name of the mapping responsible for storing the data in appropriate columns."),
+ @ReadsAttribute(attribute = "FLUSH_IMMEDIATE", description = "In case of queued ingestion, this property determines whether the data should be flushed immediately to the ingest endpoint."),
+ @ReadsAttribute(attribute = "DATA_FORMAT", description = "Specifies the format of data that is send to Azure Data Explorer."),
+ @ReadsAttribute(attribute = "IGNORE_FIRST_RECORD", description = "Specifies whether we want to ignore ingestion of first record. " +
+ "This is primarily applicable for csv files. Default is set to NO"),
+ @ReadsAttribute(attribute = "POLL_ON_INGESTION_STATUS", description = "Specifies whether we want to poll on ingestion result during ingestion into ADX." +
+ "In case of applications that need high throughput it is recommended to keep the default value as false. Default is set to false." +
+ "This property should be set to true during Queued Ingestion for near realtime micro-batches of data that require acknowledgement of ingestion status.")
+})
+public class PutAzureDataExplorer extends AbstractProcessor {
+
+ public static final String FETCH_TABLE_COMMAND = "%s | count";
+ public static final String STREAMING_POLICY_SHOW_COMMAND = ".show %s %s policy streamingingestion";
+ public static final String DATABASE = "database";
+
+ private List<PropertyDescriptor> descriptors;
+ private Set<Relationship> relationships;
+ private transient KustoIngestService service;
+ private boolean isStreamingEnabled;
+ private boolean pollOnIngestionStatus;
+
+ public static final AllowableValue AVRO = new AllowableValue(
+ IngestionProperties.DataFormat.AVRO.getKustoValue(), IngestionProperties.DataFormat.AVRO.getKustoValue(),
+ "An Avro format with support for logical types and for the snappy compression codec");
+
+ public static final AllowableValue APACHEAVRO = new AllowableValue(
+ IngestionProperties.DataFormat.APACHEAVRO.getKustoValue(), IngestionProperties.DataFormat.APACHEAVRO.getKustoValue(),
+ "An Avro format with support for logical types and for the snappy compression codec.");
+
+ public static final AllowableValue CSV = new AllowableValue(
+ IngestionProperties.DataFormat.CSV.name(), IngestionProperties.DataFormat.CSV.getKustoValue(),
+ "A text file with comma-separated values (,). For more information, see RFC 4180: Common Format " +
+ "and MIME Type for Comma-Separated Values (CSV) Files.");
+
+ public static final AllowableValue JSON = new AllowableValue(
+ IngestionProperties.DataFormat.JSON.name(), IngestionProperties.DataFormat.JSON.getKustoValue(),
+ "A text file containing JSON objects separated by \\n or \\r\\n. For more information, " +
+ "see JSON Lines (JSONL).");
+
+ public static final AllowableValue MULTIJSON = new AllowableValue(
+ IngestionProperties.DataFormat.MULTIJSON.name(), IngestionProperties.DataFormat.MULTIJSON.getKustoValue(),
+ "A text file containing a JSON array of property containers (each representing a record) or any " +
+ "number of property containers separated by spaces, \\n or \\r\\n. Each property container may be " +
+ "spread across multiple lines. This format is preferable to JSON unless the data is not property " +
+ "containers.");
+
+ public static final AllowableValue ORC = new AllowableValue(
+ IngestionProperties.DataFormat.ORC.name(), IngestionProperties.DataFormat.ORC.getKustoValue(), "An ORC file.");
+
+ public static final AllowableValue PARQUET = new AllowableValue(
+ IngestionProperties.DataFormat.PARQUET.name(), IngestionProperties.DataFormat.PARQUET.getKustoValue(), "A parquet file.");
+
+ public static final AllowableValue PSV = new AllowableValue(
+ IngestionProperties.DataFormat.PSV.name(), IngestionProperties.DataFormat.PSV.getKustoValue(), "A text file with values separated by vertical bars (|).");
+
+ public static final AllowableValue SCSV = new AllowableValue(
+ IngestionProperties.DataFormat.SCSV.name(), IngestionProperties.DataFormat.SCSV.getKustoValue(), "A text file with values separated by semicolons (;).");
+
+ public static final AllowableValue SOHSV = new AllowableValue(
+ IngestionProperties.DataFormat.SOHSV.name(), IngestionProperties.DataFormat.SOHSV.getKustoValue(),
+ "A text file with SOH-separated values. (SOH is the ASCII code point 1. " +
+ "This format is used by Hive in HDInsight).");
+
+ public static final AllowableValue TSV = new AllowableValue(
+ IngestionProperties.DataFormat.TSV.name(), IngestionProperties.DataFormat.TSV.getKustoValue(), "A text file with tab delimited values (\\t).");
+
+ public static final AllowableValue TSVE = new AllowableValue(
+ IngestionProperties.DataFormat.TSVE.name(), IngestionProperties.DataFormat.TSVE.getKustoValue(),
+ "A text file with tab-delimited values (\\t). A backslash (\\) is used as escape character.");
+
+ public static final AllowableValue TXT = new AllowableValue(
+ IngestionProperties.DataFormat.TXT.name(), IngestionProperties.DataFormat.TXT.getKustoValue(),
+ "A text file with lines separated by \\n. Empty lines are skipped.");
+
+ public static final AllowableValue IGNORE_FIRST_RECORD_YES = new AllowableValue(
+ "YES", "YES",
+ "Ignore first record during ingestion");
+
+ public static final AllowableValue IGNORE_FIRST_RECORD_NO = new AllowableValue(
+ "NO", "NO",
+ "Do not ignore first record during ingestion");
+
+ public static final PropertyDescriptor DATABASE_NAME = new PropertyDescriptor.Builder()
+ .name("Database Name")
+ .displayName("Database Name")
+ .description("Azure Data Explorer Database Name for querying")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+ .build();
+
+ public static final PropertyDescriptor TABLE_NAME = new PropertyDescriptor.Builder()
+ .name("Table Name")
+ .displayName("Table Name")
+ .description("Azure Data Explorer Table Name for ingesting data")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+ .build();
+
+ public static final PropertyDescriptor MAPPING_NAME = new PropertyDescriptor
+ .Builder().name("Mapping name")
+ .displayName("Mapping name")
+ .description("The name of the mapping responsible for storing the data in the appropriate columns.")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+
+ public static final PropertyDescriptor IS_STREAMING_ENABLED = new PropertyDescriptor
+ .Builder().name("Is Streaming enabled")
+ .displayName("Is Streaming enabled")
+ .description("This property determines whether we want to stream data to ADX.")
+ .required(false)
+ .allowableValues("true", "false")
+ .addValidator(StandardValidators.BOOLEAN_VALIDATOR)
+ .defaultValue("false")
+ .build();
+
+ public static final PropertyDescriptor POLL_ON_INGESTION_STATUS = new PropertyDescriptor
+ .Builder().name("Poll on ingestion status")
+ .displayName("Whether to Poll on ingestion status")
+ .description("This property determines whether we want to poll on ingestion status after an ingestion to ADX is completed")
+ .required(false)
+ .allowableValues("true", "false")
+ .addValidator(StandardValidators.BOOLEAN_VALIDATOR)
+ .defaultValue("false")
+ .build();
+ public static final PropertyDescriptor ADX_SERVICE = new PropertyDescriptor
+ .Builder().name("Kusto Ingest Service")
+ .displayName("Kusto Ingest Service")
+ .description("Azure Data Explorer Kusto Ingest Service")
+ .required(true)
+ .identifiesControllerService(KustoIngestService.class)
+ .build();
+
+ public static final Relationship SUCCESS = new Relationship.Builder()
+ .name("success")
+ .description("Relationship for success")
+ .build();
+ public static final Relationship FAILURE = new Relationship.Builder()
+ .name("failure")
+ .description("Relationship for failure")
+ .build();
+
+ static final PropertyDescriptor DATA_FORMAT = new PropertyDescriptor.Builder()
+ .name("Data Format")
+ .displayName("Data Format")
+ .description("The format of the data that is sent to ADX.")
+ .required(true)
+ .allowableValues(AVRO, APACHEAVRO, CSV, JSON, MULTIJSON, ORC, PARQUET, PSV, SCSV, SOHSV, TSV, TSVE, TXT)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+
+ static final PropertyDescriptor IGNORE_FIRST_RECORD = new PropertyDescriptor.Builder()
+ .name("Ingestion Ignore First Record")
+ .displayName("Ingestion Ignore First Record")
+ .description("Defines whether ignore first record while ingestion.")
+ .required(false)
+ .allowableValues(IGNORE_FIRST_RECORD_YES, IGNORE_FIRST_RECORD_NO)
+ .defaultValue(IGNORE_FIRST_RECORD_NO.getValue())
+ .build();
+
+ @Override
+ protected void init(final ProcessorInitializationContext context) {
+ final List<PropertyDescriptor> descriptorList = new ArrayList<>();
+ descriptorList.add(ADX_SERVICE);
+ descriptorList.add(DATABASE_NAME);
+ descriptorList.add(TABLE_NAME);
+ descriptorList.add(MAPPING_NAME);
+ descriptorList.add(DATA_FORMAT);
+ descriptorList.add(IGNORE_FIRST_RECORD);
+ descriptorList.add(IS_STREAMING_ENABLED);
+ descriptorList.add(POLL_ON_INGESTION_STATUS);
+ this.descriptors = Collections.unmodifiableList(descriptorList);
+
+ final Set<Relationship> relationshipSet = new HashSet<>();
+ relationshipSet.add(SUCCESS);
+ relationshipSet.add(FAILURE);
+ this.relationships = Collections.unmodifiableSet(relationshipSet);
+ }
+
+ @Override
+ public Set<Relationship> getRelationships() {
+ return this.relationships;
+ }
+
+ @Override
+ public final List<PropertyDescriptor> getSupportedPropertyDescriptors() {
+ return descriptors;
+ }
+
+ @OnScheduled
+ public void onScheduled(final ProcessContext context) {
+ service = context.getProperty(ADX_SERVICE).asControllerService(KustoIngestService.class);
+ if (checkIfIngestorRoleDoesntExist(context.getProperty(DATABASE_NAME).getValue(), context.getProperty(TABLE_NAME).getValue())) {
+ getLogger().error("User might not have ingestor privileges, table validation will be skipped for all table mappings.");
+ throw new ProcessException("User might not have ingestor privileges, table validation will be skipped for all table mappings. ");
+ }
+ isStreamingEnabled = context.getProperty(IS_STREAMING_ENABLED).evaluateAttributeExpressions().asBoolean();
+ if (isStreamingEnabled) {
+ checkIfStreamingPolicyIsEnabledInADX(context.getProperty(DATABASE_NAME).getValue(), context.getProperty(DATABASE_NAME).getValue());
+ }
+ }
+
+ @Override
+ public void onTrigger(final ProcessContext context, final ProcessSession session) throws ProcessException {
+ FlowFile flowFile = session.get();
+ if (flowFile == null) {
+ context.yield();
+ return;
+ }
+
+ IngestionProperties ingestionProperties = new IngestionProperties(context.getProperty(DATABASE_NAME).getValue(),
+ context.getProperty(TABLE_NAME).getValue());
+
+ IngestionMapping.IngestionMappingKind ingestionMappingKind = null;
+
+ isStreamingEnabled = context.getProperty(IS_STREAMING_ENABLED).evaluateAttributeExpressions().asBoolean();
+ pollOnIngestionStatus = context.getProperty(POLL_ON_INGESTION_STATUS).evaluateAttributeExpressions().asBoolean();
+
+ switch (IngestionProperties.DataFormat.valueOf(context.getProperty(DATA_FORMAT).getValue())) {
+ case AVRO:
+ ingestionProperties.setDataFormat(IngestionProperties.DataFormat.AVRO);
+ ingestionMappingKind = IngestionProperties.DataFormat.AVRO.getIngestionMappingKind();
+ break;
+ case APACHEAVRO:
+ ingestionProperties.setDataFormat(IngestionProperties.DataFormat.APACHEAVRO);
+ ingestionMappingKind = IngestionProperties.DataFormat.APACHEAVRO.getIngestionMappingKind();
+ break;
+ case CSV:
+ ingestionProperties.setDataFormat(IngestionProperties.DataFormat.CSV);
+ ingestionMappingKind = IngestionProperties.DataFormat.CSV.getIngestionMappingKind();
+ break;
+ case JSON:
+ ingestionProperties.setDataFormat(IngestionProperties.DataFormat.JSON);
+ ingestionMappingKind = IngestionProperties.DataFormat.JSON.getIngestionMappingKind();
+ break;
+ case MULTIJSON:
+ ingestionProperties.setDataFormat(IngestionProperties.DataFormat.MULTIJSON);
+ ingestionMappingKind = IngestionProperties.DataFormat.MULTIJSON.getIngestionMappingKind();
+ break;
+ case ORC:
+ ingestionProperties.setDataFormat(IngestionProperties.DataFormat.ORC);
+ ingestionMappingKind = IngestionProperties.DataFormat.ORC.getIngestionMappingKind();
+ break;
+ case PARQUET:
+ ingestionProperties.setDataFormat(IngestionProperties.DataFormat.PARQUET);
+ ingestionMappingKind = IngestionProperties.DataFormat.PARQUET.getIngestionMappingKind();
+ break;
+ case PSV:
+ ingestionProperties.setDataFormat(IngestionProperties.DataFormat.PSV);
+ ingestionMappingKind = IngestionProperties.DataFormat.PSV.getIngestionMappingKind();
+ break;
+ case SCSV:
+ ingestionProperties.setDataFormat(IngestionProperties.DataFormat.SCSV);
+ ingestionMappingKind = IngestionProperties.DataFormat.SCSV.getIngestionMappingKind();
+ break;
+ case SOHSV:
+ ingestionProperties.setDataFormat(IngestionProperties.DataFormat.SOHSV);
+ ingestionMappingKind = IngestionProperties.DataFormat.SOHSV.getIngestionMappingKind();
+ break;
+ case TSV:
+ ingestionProperties.setDataFormat(IngestionProperties.DataFormat.TSV);
+ ingestionMappingKind = IngestionProperties.DataFormat.TSV.getIngestionMappingKind();
+ break;
+ case TSVE:
+ ingestionProperties.setDataFormat(IngestionProperties.DataFormat.TSVE);
+ ingestionMappingKind = IngestionProperties.DataFormat.TSVE.getIngestionMappingKind();
+ break;
+ case TXT:
+ ingestionProperties.setDataFormat(IngestionProperties.DataFormat.TXT);
+ ingestionMappingKind = IngestionProperties.DataFormat.TXT.getIngestionMappingKind();
+ break;
+ }
+
+ if (StringUtils.isNotEmpty(context.getProperty(MAPPING_NAME).getValue()) && ingestionMappingKind != null) {
+ ingestionProperties.setIngestionMapping(context.getProperty(MAPPING_NAME).getValue(), ingestionMappingKind);
+ }
+
+ ingestionProperties.setReportLevel(IngestionProperties.IngestionReportLevel.FAILURES_AND_SUCCESSES);
+ ingestionProperties.setReportMethod(IngestionProperties.IngestionReportMethod.TABLE);
+ ingestionProperties.setFlushImmediately(false);
+ ingestionProperties.setIgnoreFirstRecord(StringUtils.equalsIgnoreCase(context.getProperty(IGNORE_FIRST_RECORD).getValue(), IGNORE_FIRST_RECORD_YES.getValue()));
+
+ boolean isError = false;
+
+ // ingestion into ADX
+ try (final InputStream inputStream = session.read(flowFile)) {
+ StringBuilder ingestLogString = new StringBuilder().append("Ingesting with: ")
+ .append("dataFormat - ").append(ingestionProperties.getDataFormat()).append("|")
+ .append("ingestionMapping - ").append(ingestionProperties.getIngestionMapping().getIngestionMappingReference()).append("|")
+ .append("reportLevel - ").append(ingestionProperties.getReportLevel()).append("|")
+ .append("reportMethod - ").append(ingestionProperties.getReportMethod()).append("|")
+ .append("databaseName - ").append(ingestionProperties.getDatabaseName()).append("|")
+ .append("tableName - ").append(ingestionProperties.getTableName()).append("|")
+ .append("flushImmediately - ").append(ingestionProperties.getFlushImmediately());
+ getLogger().info(ingestLogString.toString());
+
+ KustoIngestionResult result = service.ingestData(new KustoIngestionRequest(isStreamingEnabled, pollOnIngestionStatus, inputStream, ingestionProperties));
+ getLogger().info("Poll on Ingestion Status {} and Operation status: {} ", pollOnIngestionStatus, result.toString());
+
+ if (result == KustoIngestionResult.SUCCEEDED) {
+ getLogger().info("Operation status Succeeded - {}", result.toString());
+ } else if (result == KustoIngestionResult.FAILED) {
+ getLogger().error("Operation status Error - {}", result.toString());
+ isError = true;
+ } else if (result == KustoIngestionResult.PARTIALLY_SUCCEEDED) {
+ getLogger().error("Operation status Partially succeeded - {}", result.toString());
+ isError = true;
+ }
+
+ } catch (IOException | URISyntaxException e) {
+ getLogger().error("Exception occurred while ingesting data into ADX ", e);
+ isError = true;
+ }
+
+ if (isError) {
+ getLogger().error("Process failed - {}");
+ session.transfer(flowFile, FAILURE);
+ } else {
+ getLogger().info("Process succeeded - {}");
+ session.transfer(flowFile, SUCCESS);
+ }
+ }
+
+ protected void checkIfStreamingPolicyIsEnabledInADX(String entityName, String database) {
+ KustoQueryResponse kustoQueryResponse = service.executeQuery(database, String.format(STREAMING_POLICY_SHOW_COMMAND, PutAzureDataExplorer.DATABASE, entityName));
+ if (kustoQueryResponse.isError()) {
+ throw new ProcessException("Error occurred while checking if streaming policy is enabled for the table");
Review Comment:
It would be helpful to indicate additional information, including the `entityName` and `database` as part of this message.
##########
nifi-nar-bundles/nifi-azure-bundle/nifi-azure-processors/src/main/java/org/apache/nifi/processors/azure/data/explorer/PutAzureDataExplorer.java:
##########
@@ -0,0 +1,399 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.azure.data.explorer;
+
+import com.microsoft.azure.kusto.data.KustoResultSetTable;
+import com.microsoft.azure.kusto.ingest.IngestionMapping;
+import com.microsoft.azure.kusto.ingest.IngestionProperties;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.nifi.annotation.behavior.ReadsAttribute;
+import org.apache.nifi.annotation.behavior.ReadsAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.expression.ExpressionLanguageScope;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.services.azure.data.explorer.KustoIngestService;
+import org.apache.nifi.services.azure.data.explorer.KustoIngestionRequest;
+import org.apache.nifi.services.azure.data.explorer.KustoIngestionResult;
+import org.apache.nifi.services.azure.data.explorer.KustoQueryResponse;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.net.URISyntaxException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+
+@Tags({"Azure", "Kusto", "ADX", "Explorer", "Data"})
+@CapabilityDescription("The PutAzureDataExplorer acts as a ADX sink connector which sends flowFiles using the ADX-Service to the provided Azure Data" +
+ "Explorer Ingest Endpoint. The data can be sent through queued ingestion or streaming ingestion to the Azure Data Explorer cluster.")
+@ReadsAttributes({
+ @ReadsAttribute(attribute = "DB_NAME", description = "Specifies the name of the ADX database where the data needs to be stored."),
+ @ReadsAttribute(attribute = "TABLE_NAME", description = "Specifies the name of the ADX table where the data needs to be stored."),
+ @ReadsAttribute(attribute = "MAPPING_NAME", description = "Specifies the name of the mapping responsible for storing the data in appropriate columns."),
+ @ReadsAttribute(attribute = "FLUSH_IMMEDIATE", description = "In case of queued ingestion, this property determines whether the data should be flushed immediately to the ingest endpoint."),
+ @ReadsAttribute(attribute = "DATA_FORMAT", description = "Specifies the format of data that is send to Azure Data Explorer."),
+ @ReadsAttribute(attribute = "IGNORE_FIRST_RECORD", description = "Specifies whether we want to ignore ingestion of first record. " +
+ "This is primarily applicable for csv files. Default is set to NO"),
+ @ReadsAttribute(attribute = "POLL_ON_INGESTION_STATUS", description = "Specifies whether we want to poll on ingestion result during ingestion into ADX." +
+ "In case of applications that need high throughput it is recommended to keep the default value as false. Default is set to false." +
+ "This property should be set to true during Queued Ingestion for near realtime micro-batches of data that require acknowledgement of ingestion status.")
+})
+public class PutAzureDataExplorer extends AbstractProcessor {
+
+ public static final String FETCH_TABLE_COMMAND = "%s | count";
+ public static final String STREAMING_POLICY_SHOW_COMMAND = ".show %s %s policy streamingingestion";
+ public static final String DATABASE = "database";
+
+ private List<PropertyDescriptor> descriptors;
+ private Set<Relationship> relationships;
+ private transient KustoIngestService service;
+ private boolean isStreamingEnabled;
+ private boolean pollOnIngestionStatus;
+
+ public static final AllowableValue AVRO = new AllowableValue(
+ IngestionProperties.DataFormat.AVRO.getKustoValue(), IngestionProperties.DataFormat.AVRO.getKustoValue(),
+ "An Avro format with support for logical types and for the snappy compression codec");
+
+ public static final AllowableValue APACHEAVRO = new AllowableValue(
+ IngestionProperties.DataFormat.APACHEAVRO.getKustoValue(), IngestionProperties.DataFormat.APACHEAVRO.getKustoValue(),
+ "An Avro format with support for logical types and for the snappy compression codec.");
+
+ public static final AllowableValue CSV = new AllowableValue(
+ IngestionProperties.DataFormat.CSV.name(), IngestionProperties.DataFormat.CSV.getKustoValue(),
+ "A text file with comma-separated values (,). For more information, see RFC 4180: Common Format " +
+ "and MIME Type for Comma-Separated Values (CSV) Files.");
+
+ public static final AllowableValue JSON = new AllowableValue(
+ IngestionProperties.DataFormat.JSON.name(), IngestionProperties.DataFormat.JSON.getKustoValue(),
+ "A text file containing JSON objects separated by \\n or \\r\\n. For more information, " +
+ "see JSON Lines (JSONL).");
+
+ public static final AllowableValue MULTIJSON = new AllowableValue(
+ IngestionProperties.DataFormat.MULTIJSON.name(), IngestionProperties.DataFormat.MULTIJSON.getKustoValue(),
+ "A text file containing a JSON array of property containers (each representing a record) or any " +
+ "number of property containers separated by spaces, \\n or \\r\\n. Each property container may be " +
+ "spread across multiple lines. This format is preferable to JSON unless the data is not property " +
+ "containers.");
+
+ public static final AllowableValue ORC = new AllowableValue(
+ IngestionProperties.DataFormat.ORC.name(), IngestionProperties.DataFormat.ORC.getKustoValue(), "An ORC file.");
+
+ public static final AllowableValue PARQUET = new AllowableValue(
+ IngestionProperties.DataFormat.PARQUET.name(), IngestionProperties.DataFormat.PARQUET.getKustoValue(), "A parquet file.");
+
+ public static final AllowableValue PSV = new AllowableValue(
+ IngestionProperties.DataFormat.PSV.name(), IngestionProperties.DataFormat.PSV.getKustoValue(), "A text file with values separated by vertical bars (|).");
+
+ public static final AllowableValue SCSV = new AllowableValue(
+ IngestionProperties.DataFormat.SCSV.name(), IngestionProperties.DataFormat.SCSV.getKustoValue(), "A text file with values separated by semicolons (;).");
+
+ public static final AllowableValue SOHSV = new AllowableValue(
+ IngestionProperties.DataFormat.SOHSV.name(), IngestionProperties.DataFormat.SOHSV.getKustoValue(),
+ "A text file with SOH-separated values. (SOH is the ASCII code point 1. " +
+ "This format is used by Hive in HDInsight).");
+
+ public static final AllowableValue TSV = new AllowableValue(
+ IngestionProperties.DataFormat.TSV.name(), IngestionProperties.DataFormat.TSV.getKustoValue(), "A text file with tab delimited values (\\t).");
+
+ public static final AllowableValue TSVE = new AllowableValue(
+ IngestionProperties.DataFormat.TSVE.name(), IngestionProperties.DataFormat.TSVE.getKustoValue(),
+ "A text file with tab-delimited values (\\t). A backslash (\\) is used as escape character.");
+
+ public static final AllowableValue TXT = new AllowableValue(
+ IngestionProperties.DataFormat.TXT.name(), IngestionProperties.DataFormat.TXT.getKustoValue(),
+ "A text file with lines separated by \\n. Empty lines are skipped.");
+
+ public static final AllowableValue IGNORE_FIRST_RECORD_YES = new AllowableValue(
+ "YES", "YES",
+ "Ignore first record during ingestion");
+
+ public static final AllowableValue IGNORE_FIRST_RECORD_NO = new AllowableValue(
+ "NO", "NO",
+ "Do not ignore first record during ingestion");
+
+ public static final PropertyDescriptor DATABASE_NAME = new PropertyDescriptor.Builder()
+ .name("Database Name")
+ .displayName("Database Name")
+ .description("Azure Data Explorer Database Name for querying")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+ .build();
+
+ public static final PropertyDescriptor TABLE_NAME = new PropertyDescriptor.Builder()
+ .name("Table Name")
+ .displayName("Table Name")
+ .description("Azure Data Explorer Table Name for ingesting data")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+ .build();
+
+ public static final PropertyDescriptor MAPPING_NAME = new PropertyDescriptor
+ .Builder().name("Mapping name")
+ .displayName("Mapping name")
+ .description("The name of the mapping responsible for storing the data in the appropriate columns.")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+
+ public static final PropertyDescriptor IS_STREAMING_ENABLED = new PropertyDescriptor
+ .Builder().name("Is Streaming enabled")
+ .displayName("Is Streaming enabled")
+ .description("This property determines whether we want to stream data to ADX.")
+ .required(false)
+ .allowableValues("true", "false")
+ .addValidator(StandardValidators.BOOLEAN_VALIDATOR)
+ .defaultValue("false")
+ .build();
+
+ public static final PropertyDescriptor POLL_ON_INGESTION_STATUS = new PropertyDescriptor
+ .Builder().name("Poll on ingestion status")
+ .displayName("Whether to Poll on ingestion status")
+ .description("This property determines whether we want to poll on ingestion status after an ingestion to ADX is completed")
+ .required(false)
+ .allowableValues("true", "false")
+ .addValidator(StandardValidators.BOOLEAN_VALIDATOR)
+ .defaultValue("false")
+ .build();
+ public static final PropertyDescriptor ADX_SERVICE = new PropertyDescriptor
+ .Builder().name("Kusto Ingest Service")
+ .displayName("Kusto Ingest Service")
+ .description("Azure Data Explorer Kusto Ingest Service")
+ .required(true)
+ .identifiesControllerService(KustoIngestService.class)
+ .build();
+
+ public static final Relationship SUCCESS = new Relationship.Builder()
+ .name("success")
+ .description("Relationship for success")
+ .build();
+ public static final Relationship FAILURE = new Relationship.Builder()
+ .name("failure")
+ .description("Relationship for failure")
+ .build();
+
+ static final PropertyDescriptor DATA_FORMAT = new PropertyDescriptor.Builder()
+ .name("Data Format")
+ .displayName("Data Format")
+ .description("The format of the data that is sent to ADX.")
+ .required(true)
+ .allowableValues(AVRO, APACHEAVRO, CSV, JSON, MULTIJSON, ORC, PARQUET, PSV, SCSV, SOHSV, TSV, TSVE, TXT)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+
+ static final PropertyDescriptor IGNORE_FIRST_RECORD = new PropertyDescriptor.Builder()
+ .name("Ingestion Ignore First Record")
+ .displayName("Ingestion Ignore First Record")
+ .description("Defines whether ignore first record while ingestion.")
+ .required(false)
+ .allowableValues(IGNORE_FIRST_RECORD_YES, IGNORE_FIRST_RECORD_NO)
+ .defaultValue(IGNORE_FIRST_RECORD_NO.getValue())
+ .build();
+
+ @Override
+ protected void init(final ProcessorInitializationContext context) {
+ final List<PropertyDescriptor> descriptorList = new ArrayList<>();
+ descriptorList.add(ADX_SERVICE);
+ descriptorList.add(DATABASE_NAME);
+ descriptorList.add(TABLE_NAME);
+ descriptorList.add(MAPPING_NAME);
+ descriptorList.add(DATA_FORMAT);
+ descriptorList.add(IGNORE_FIRST_RECORD);
+ descriptorList.add(IS_STREAMING_ENABLED);
+ descriptorList.add(POLL_ON_INGESTION_STATUS);
+ this.descriptors = Collections.unmodifiableList(descriptorList);
+
+ final Set<Relationship> relationshipSet = new HashSet<>();
+ relationshipSet.add(SUCCESS);
+ relationshipSet.add(FAILURE);
+ this.relationships = Collections.unmodifiableSet(relationshipSet);
+ }
+
+ @Override
+ public Set<Relationship> getRelationships() {
+ return this.relationships;
+ }
+
+ @Override
+ public final List<PropertyDescriptor> getSupportedPropertyDescriptors() {
+ return descriptors;
+ }
+
+ @OnScheduled
+ public void onScheduled(final ProcessContext context) {
+ service = context.getProperty(ADX_SERVICE).asControllerService(KustoIngestService.class);
+ if (checkIfIngestorRoleDoesntExist(context.getProperty(DATABASE_NAME).getValue(), context.getProperty(TABLE_NAME).getValue())) {
+ getLogger().error("User might not have ingestor privileges, table validation will be skipped for all table mappings.");
+ throw new ProcessException("User might not have ingestor privileges, table validation will be skipped for all table mappings. ");
+ }
+ isStreamingEnabled = context.getProperty(IS_STREAMING_ENABLED).evaluateAttributeExpressions().asBoolean();
+ if (isStreamingEnabled) {
+ checkIfStreamingPolicyIsEnabledInADX(context.getProperty(DATABASE_NAME).getValue(), context.getProperty(DATABASE_NAME).getValue());
+ }
+ }
+
+ @Override
+ public void onTrigger(final ProcessContext context, final ProcessSession session) throws ProcessException {
+ FlowFile flowFile = session.get();
+ if (flowFile == null) {
+ context.yield();
+ return;
+ }
+
+ IngestionProperties ingestionProperties = new IngestionProperties(context.getProperty(DATABASE_NAME).getValue(),
+ context.getProperty(TABLE_NAME).getValue());
+
+ IngestionMapping.IngestionMappingKind ingestionMappingKind = null;
+
+ isStreamingEnabled = context.getProperty(IS_STREAMING_ENABLED).evaluateAttributeExpressions().asBoolean();
+ pollOnIngestionStatus = context.getProperty(POLL_ON_INGESTION_STATUS).evaluateAttributeExpressions().asBoolean();
+
+ switch (IngestionProperties.DataFormat.valueOf(context.getProperty(DATA_FORMAT).getValue())) {
+ case AVRO:
+ ingestionProperties.setDataFormat(IngestionProperties.DataFormat.AVRO);
+ ingestionMappingKind = IngestionProperties.DataFormat.AVRO.getIngestionMappingKind();
+ break;
+ case APACHEAVRO:
+ ingestionProperties.setDataFormat(IngestionProperties.DataFormat.APACHEAVRO);
+ ingestionMappingKind = IngestionProperties.DataFormat.APACHEAVRO.getIngestionMappingKind();
+ break;
+ case CSV:
+ ingestionProperties.setDataFormat(IngestionProperties.DataFormat.CSV);
+ ingestionMappingKind = IngestionProperties.DataFormat.CSV.getIngestionMappingKind();
+ break;
+ case JSON:
+ ingestionProperties.setDataFormat(IngestionProperties.DataFormat.JSON);
+ ingestionMappingKind = IngestionProperties.DataFormat.JSON.getIngestionMappingKind();
+ break;
+ case MULTIJSON:
+ ingestionProperties.setDataFormat(IngestionProperties.DataFormat.MULTIJSON);
+ ingestionMappingKind = IngestionProperties.DataFormat.MULTIJSON.getIngestionMappingKind();
+ break;
+ case ORC:
+ ingestionProperties.setDataFormat(IngestionProperties.DataFormat.ORC);
+ ingestionMappingKind = IngestionProperties.DataFormat.ORC.getIngestionMappingKind();
+ break;
+ case PARQUET:
+ ingestionProperties.setDataFormat(IngestionProperties.DataFormat.PARQUET);
+ ingestionMappingKind = IngestionProperties.DataFormat.PARQUET.getIngestionMappingKind();
+ break;
+ case PSV:
+ ingestionProperties.setDataFormat(IngestionProperties.DataFormat.PSV);
+ ingestionMappingKind = IngestionProperties.DataFormat.PSV.getIngestionMappingKind();
+ break;
+ case SCSV:
+ ingestionProperties.setDataFormat(IngestionProperties.DataFormat.SCSV);
+ ingestionMappingKind = IngestionProperties.DataFormat.SCSV.getIngestionMappingKind();
+ break;
+ case SOHSV:
+ ingestionProperties.setDataFormat(IngestionProperties.DataFormat.SOHSV);
+ ingestionMappingKind = IngestionProperties.DataFormat.SOHSV.getIngestionMappingKind();
+ break;
+ case TSV:
+ ingestionProperties.setDataFormat(IngestionProperties.DataFormat.TSV);
+ ingestionMappingKind = IngestionProperties.DataFormat.TSV.getIngestionMappingKind();
+ break;
+ case TSVE:
+ ingestionProperties.setDataFormat(IngestionProperties.DataFormat.TSVE);
+ ingestionMappingKind = IngestionProperties.DataFormat.TSVE.getIngestionMappingKind();
+ break;
+ case TXT:
+ ingestionProperties.setDataFormat(IngestionProperties.DataFormat.TXT);
+ ingestionMappingKind = IngestionProperties.DataFormat.TXT.getIngestionMappingKind();
+ break;
+ }
Review Comment:
Recommend moving this switch statement to a separate method that returns the `DataFormat`.
##########
nifi-nar-bundles/nifi-azure-bundle/nifi-azure-processors/src/main/java/org/apache/nifi/processors/azure/data/explorer/PutAzureDataExplorer.java:
##########
@@ -0,0 +1,399 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.azure.data.explorer;
+
+import com.microsoft.azure.kusto.data.KustoResultSetTable;
+import com.microsoft.azure.kusto.ingest.IngestionMapping;
+import com.microsoft.azure.kusto.ingest.IngestionProperties;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.nifi.annotation.behavior.ReadsAttribute;
+import org.apache.nifi.annotation.behavior.ReadsAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.expression.ExpressionLanguageScope;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.services.azure.data.explorer.KustoIngestService;
+import org.apache.nifi.services.azure.data.explorer.KustoIngestionRequest;
+import org.apache.nifi.services.azure.data.explorer.KustoIngestionResult;
+import org.apache.nifi.services.azure.data.explorer.KustoQueryResponse;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.net.URISyntaxException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+
+@Tags({"Azure", "Kusto", "ADX", "Explorer", "Data"})
+@CapabilityDescription("The PutAzureDataExplorer acts as a ADX sink connector which sends flowFiles using the ADX-Service to the provided Azure Data" +
+ "Explorer Ingest Endpoint. The data can be sent through queued ingestion or streaming ingestion to the Azure Data Explorer cluster.")
+@ReadsAttributes({
+ @ReadsAttribute(attribute = "DB_NAME", description = "Specifies the name of the ADX database where the data needs to be stored."),
+ @ReadsAttribute(attribute = "TABLE_NAME", description = "Specifies the name of the ADX table where the data needs to be stored."),
+ @ReadsAttribute(attribute = "MAPPING_NAME", description = "Specifies the name of the mapping responsible for storing the data in appropriate columns."),
+ @ReadsAttribute(attribute = "FLUSH_IMMEDIATE", description = "In case of queued ingestion, this property determines whether the data should be flushed immediately to the ingest endpoint."),
+ @ReadsAttribute(attribute = "DATA_FORMAT", description = "Specifies the format of data that is send to Azure Data Explorer."),
+ @ReadsAttribute(attribute = "IGNORE_FIRST_RECORD", description = "Specifies whether we want to ignore ingestion of first record. " +
+ "This is primarily applicable for csv files. Default is set to NO"),
+ @ReadsAttribute(attribute = "POLL_ON_INGESTION_STATUS", description = "Specifies whether we want to poll on ingestion result during ingestion into ADX." +
+ "In case of applications that need high throughput it is recommended to keep the default value as false. Default is set to false." +
+ "This property should be set to true during Queued Ingestion for near realtime micro-batches of data that require acknowledgement of ingestion status.")
+})
+public class PutAzureDataExplorer extends AbstractProcessor {
+
+ public static final String FETCH_TABLE_COMMAND = "%s | count";
+ public static final String STREAMING_POLICY_SHOW_COMMAND = ".show %s %s policy streamingingestion";
+ public static final String DATABASE = "database";
+
+ private List<PropertyDescriptor> descriptors;
+ private Set<Relationship> relationships;
+ private transient KustoIngestService service;
+ private boolean isStreamingEnabled;
+ private boolean pollOnIngestionStatus;
+
+ public static final AllowableValue AVRO = new AllowableValue(
+ IngestionProperties.DataFormat.AVRO.getKustoValue(), IngestionProperties.DataFormat.AVRO.getKustoValue(),
+ "An Avro format with support for logical types and for the snappy compression codec");
+
+ public static final AllowableValue APACHEAVRO = new AllowableValue(
+ IngestionProperties.DataFormat.APACHEAVRO.getKustoValue(), IngestionProperties.DataFormat.APACHEAVRO.getKustoValue(),
+ "An Avro format with support for logical types and for the snappy compression codec.");
+
+ public static final AllowableValue CSV = new AllowableValue(
+ IngestionProperties.DataFormat.CSV.name(), IngestionProperties.DataFormat.CSV.getKustoValue(),
+ "A text file with comma-separated values (,). For more information, see RFC 4180: Common Format " +
+ "and MIME Type for Comma-Separated Values (CSV) Files.");
+
+ public static final AllowableValue JSON = new AllowableValue(
+ IngestionProperties.DataFormat.JSON.name(), IngestionProperties.DataFormat.JSON.getKustoValue(),
+ "A text file containing JSON objects separated by \\n or \\r\\n. For more information, " +
+ "see JSON Lines (JSONL).");
+
+ public static final AllowableValue MULTIJSON = new AllowableValue(
+ IngestionProperties.DataFormat.MULTIJSON.name(), IngestionProperties.DataFormat.MULTIJSON.getKustoValue(),
+ "A text file containing a JSON array of property containers (each representing a record) or any " +
+ "number of property containers separated by spaces, \\n or \\r\\n. Each property container may be " +
+ "spread across multiple lines. This format is preferable to JSON unless the data is not property " +
+ "containers.");
+
+ public static final AllowableValue ORC = new AllowableValue(
+ IngestionProperties.DataFormat.ORC.name(), IngestionProperties.DataFormat.ORC.getKustoValue(), "An ORC file.");
+
+ public static final AllowableValue PARQUET = new AllowableValue(
+ IngestionProperties.DataFormat.PARQUET.name(), IngestionProperties.DataFormat.PARQUET.getKustoValue(), "A parquet file.");
+
+ public static final AllowableValue PSV = new AllowableValue(
+ IngestionProperties.DataFormat.PSV.name(), IngestionProperties.DataFormat.PSV.getKustoValue(), "A text file with values separated by vertical bars (|).");
+
+ public static final AllowableValue SCSV = new AllowableValue(
+ IngestionProperties.DataFormat.SCSV.name(), IngestionProperties.DataFormat.SCSV.getKustoValue(), "A text file with values separated by semicolons (;).");
+
+ public static final AllowableValue SOHSV = new AllowableValue(
+ IngestionProperties.DataFormat.SOHSV.name(), IngestionProperties.DataFormat.SOHSV.getKustoValue(),
+ "A text file with SOH-separated values. (SOH is the ASCII code point 1. " +
+ "This format is used by Hive in HDInsight).");
+
+ public static final AllowableValue TSV = new AllowableValue(
+ IngestionProperties.DataFormat.TSV.name(), IngestionProperties.DataFormat.TSV.getKustoValue(), "A text file with tab delimited values (\\t).");
+
+ public static final AllowableValue TSVE = new AllowableValue(
+ IngestionProperties.DataFormat.TSVE.name(), IngestionProperties.DataFormat.TSVE.getKustoValue(),
+ "A text file with tab-delimited values (\\t). A backslash (\\) is used as escape character.");
+
+ public static final AllowableValue TXT = new AllowableValue(
+ IngestionProperties.DataFormat.TXT.name(), IngestionProperties.DataFormat.TXT.getKustoValue(),
+ "A text file with lines separated by \\n. Empty lines are skipped.");
+
+ public static final AllowableValue IGNORE_FIRST_RECORD_YES = new AllowableValue(
+ "YES", "YES",
+ "Ignore first record during ingestion");
+
+ public static final AllowableValue IGNORE_FIRST_RECORD_NO = new AllowableValue(
+ "NO", "NO",
+ "Do not ignore first record during ingestion");
+
+ public static final PropertyDescriptor DATABASE_NAME = new PropertyDescriptor.Builder()
+ .name("Database Name")
+ .displayName("Database Name")
+ .description("Azure Data Explorer Database Name for querying")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+ .build();
+
+ public static final PropertyDescriptor TABLE_NAME = new PropertyDescriptor.Builder()
+ .name("Table Name")
+ .displayName("Table Name")
+ .description("Azure Data Explorer Table Name for ingesting data")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+ .build();
+
+ public static final PropertyDescriptor MAPPING_NAME = new PropertyDescriptor
+ .Builder().name("Mapping name")
+ .displayName("Mapping name")
+ .description("The name of the mapping responsible for storing the data in the appropriate columns.")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+
+ public static final PropertyDescriptor IS_STREAMING_ENABLED = new PropertyDescriptor
+ .Builder().name("Is Streaming enabled")
+ .displayName("Is Streaming enabled")
+ .description("This property determines whether we want to stream data to ADX.")
+ .required(false)
+ .allowableValues("true", "false")
+ .addValidator(StandardValidators.BOOLEAN_VALIDATOR)
+ .defaultValue("false")
+ .build();
+
+ public static final PropertyDescriptor POLL_ON_INGESTION_STATUS = new PropertyDescriptor
+ .Builder().name("Poll on ingestion status")
+ .displayName("Whether to Poll on ingestion status")
+ .description("This property determines whether we want to poll on ingestion status after an ingestion to ADX is completed")
+ .required(false)
+ .allowableValues("true", "false")
+ .addValidator(StandardValidators.BOOLEAN_VALIDATOR)
+ .defaultValue("false")
+ .build();
+ public static final PropertyDescriptor ADX_SERVICE = new PropertyDescriptor
+ .Builder().name("Kusto Ingest Service")
+ .displayName("Kusto Ingest Service")
+ .description("Azure Data Explorer Kusto Ingest Service")
+ .required(true)
+ .identifiesControllerService(KustoIngestService.class)
+ .build();
+
+ public static final Relationship SUCCESS = new Relationship.Builder()
+ .name("success")
+ .description("Relationship for success")
+ .build();
+ public static final Relationship FAILURE = new Relationship.Builder()
+ .name("failure")
+ .description("Relationship for failure")
+ .build();
Review Comment:
Recommend moving the relationship definitions after all of the property definitions, instead of between properties.
##########
nifi-nar-bundles/nifi-azure-bundle/nifi-azure-services-api/src/main/java/org/apache/nifi/services/azure/data/explorer/KustoIngestionRequest.java:
##########
@@ -0,0 +1,57 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.services.azure.data.explorer;
+
+import com.microsoft.azure.kusto.ingest.IngestionProperties;
+
+import java.io.InputStream;
+
+public class KustoIngestionRequest {
+
+ private boolean isStreamingEnabled;
+
+ private boolean pollOnIngestionStatus;
+ private InputStream inputStream;
+ private IngestionProperties ingestionProperties;
+
+ public KustoIngestionRequest(boolean isStreamingEnabled, boolean pollOnIngestionStatus , InputStream inputStream, IngestionProperties ingestionProperties) {
Review Comment:
```suggestion
public KustoIngestionRequest(boolean streamingEnabled, boolean pollOnIngestionStatus , InputStream inputStream, IngestionProperties ingestionProperties) {
```
##########
nifi-nar-bundles/nifi-azure-bundle/nifi-azure-processors/src/main/java/org/apache/nifi/processors/azure/data/explorer/PutAzureDataExplorer.java:
##########
@@ -0,0 +1,399 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.azure.data.explorer;
+
+import com.microsoft.azure.kusto.data.KustoResultSetTable;
+import com.microsoft.azure.kusto.ingest.IngestionMapping;
+import com.microsoft.azure.kusto.ingest.IngestionProperties;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.nifi.annotation.behavior.ReadsAttribute;
+import org.apache.nifi.annotation.behavior.ReadsAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.expression.ExpressionLanguageScope;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.services.azure.data.explorer.KustoIngestService;
+import org.apache.nifi.services.azure.data.explorer.KustoIngestionRequest;
+import org.apache.nifi.services.azure.data.explorer.KustoIngestionResult;
+import org.apache.nifi.services.azure.data.explorer.KustoQueryResponse;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.net.URISyntaxException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+
+@Tags({"Azure", "Kusto", "ADX", "Explorer", "Data"})
+@CapabilityDescription("The PutAzureDataExplorer acts as a ADX sink connector which sends flowFiles using the ADX-Service to the provided Azure Data" +
+ "Explorer Ingest Endpoint. The data can be sent through queued ingestion or streaming ingestion to the Azure Data Explorer cluster.")
+@ReadsAttributes({
+ @ReadsAttribute(attribute = "DB_NAME", description = "Specifies the name of the ADX database where the data needs to be stored."),
+ @ReadsAttribute(attribute = "TABLE_NAME", description = "Specifies the name of the ADX table where the data needs to be stored."),
+ @ReadsAttribute(attribute = "MAPPING_NAME", description = "Specifies the name of the mapping responsible for storing the data in appropriate columns."),
+ @ReadsAttribute(attribute = "FLUSH_IMMEDIATE", description = "In case of queued ingestion, this property determines whether the data should be flushed immediately to the ingest endpoint."),
+ @ReadsAttribute(attribute = "DATA_FORMAT", description = "Specifies the format of data that is send to Azure Data Explorer."),
+ @ReadsAttribute(attribute = "IGNORE_FIRST_RECORD", description = "Specifies whether we want to ignore ingestion of first record. " +
+ "This is primarily applicable for csv files. Default is set to NO"),
+ @ReadsAttribute(attribute = "POLL_ON_INGESTION_STATUS", description = "Specifies whether we want to poll on ingestion result during ingestion into ADX." +
+ "In case of applications that need high throughput it is recommended to keep the default value as false. Default is set to false." +
+ "This property should be set to true during Queued Ingestion for near realtime micro-batches of data that require acknowledgement of ingestion status.")
+})
Review Comment:
The Processor does not appear to read these attributes, so these annotations should be removed.
```suggestion
```
##########
nifi-nar-bundles/nifi-azure-bundle/nifi-azure-processors/src/main/java/org/apache/nifi/services/azure/data/explorer/StandardKustoIngestService.java:
##########
@@ -0,0 +1,293 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.services.azure.data.explorer;
+
+import com.microsoft.azure.kusto.data.Client;
+import com.microsoft.azure.kusto.data.ClientFactory;
+import com.microsoft.azure.kusto.data.KustoResultSetTable;
+import com.microsoft.azure.kusto.data.auth.ConnectionStringBuilder;
+import com.microsoft.azure.kusto.data.exceptions.DataClientException;
+import com.microsoft.azure.kusto.data.exceptions.DataServiceException;
+import com.microsoft.azure.kusto.ingest.IngestClientFactory;
+import com.microsoft.azure.kusto.ingest.ManagedStreamingIngestClient;
+import com.microsoft.azure.kusto.ingest.QueuedIngestClient;
+import com.microsoft.azure.kusto.ingest.exceptions.IngestionClientException;
+import com.microsoft.azure.kusto.ingest.exceptions.IngestionServiceException;
+import com.microsoft.azure.kusto.ingest.result.IngestionResult;
+import com.microsoft.azure.kusto.ingest.result.IngestionStatus;
+import com.microsoft.azure.kusto.ingest.result.OperationStatus;
+import com.microsoft.azure.kusto.ingest.source.StreamSourceInfo;
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.nifi.annotation.behavior.ReadsAttribute;
+import org.apache.nifi.annotation.behavior.ReadsAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnEnabled;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.controller.AbstractControllerService;
+import org.apache.nifi.controller.ConfigurationContext;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.reporting.InitializationException;
+
+import java.io.IOException;
+import java.net.URISyntaxException;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.List;
+import java.util.Objects;
+
+@Tags({"Azure", "ADX", "Kusto", "ingest", "azure"})
+@CapabilityDescription("Sends batches of flowfile content or stream flowfile content to an Azure ADX cluster.")
+@ReadsAttributes({
+ @ReadsAttribute(attribute = "AUTH_STRATEGY", description = "The strategy/method to authenticate against Azure Active Directory, either 'application' or 'managed_identity'."),
+ @ReadsAttribute(attribute = "APP_ID", description = "Specifies Azure application id for accessing the ADX-Cluster."),
+ @ReadsAttribute(attribute = "APP_KEY", description = "Specifies Azure application key for accessing the ADX-Cluster."),
+ @ReadsAttribute(attribute = "APP_TENANT", description = "Azure application tenant for accessing the ADX-Cluster."),
+ @ReadsAttribute(attribute = "CLUSTER_URL", description = "Endpoint of ADX cluster. This is required only when streaming data to ADX cluster is enabled."),
+})
Review Comment:
Controller Services do not read attributes, so these annotations should be removed.
```suggestion
```
##########
nifi-nar-bundles/nifi-azure-bundle/nifi-azure-processors/src/main/java/org/apache/nifi/services/azure/data/explorer/StandardKustoIngestService.java:
##########
@@ -0,0 +1,293 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.services.azure.data.explorer;
+
+import com.microsoft.azure.kusto.data.Client;
+import com.microsoft.azure.kusto.data.ClientFactory;
+import com.microsoft.azure.kusto.data.KustoResultSetTable;
+import com.microsoft.azure.kusto.data.auth.ConnectionStringBuilder;
+import com.microsoft.azure.kusto.data.exceptions.DataClientException;
+import com.microsoft.azure.kusto.data.exceptions.DataServiceException;
+import com.microsoft.azure.kusto.ingest.IngestClientFactory;
+import com.microsoft.azure.kusto.ingest.ManagedStreamingIngestClient;
+import com.microsoft.azure.kusto.ingest.QueuedIngestClient;
+import com.microsoft.azure.kusto.ingest.exceptions.IngestionClientException;
+import com.microsoft.azure.kusto.ingest.exceptions.IngestionServiceException;
+import com.microsoft.azure.kusto.ingest.result.IngestionResult;
+import com.microsoft.azure.kusto.ingest.result.IngestionStatus;
+import com.microsoft.azure.kusto.ingest.result.OperationStatus;
+import com.microsoft.azure.kusto.ingest.source.StreamSourceInfo;
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.nifi.annotation.behavior.ReadsAttribute;
+import org.apache.nifi.annotation.behavior.ReadsAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnEnabled;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.controller.AbstractControllerService;
+import org.apache.nifi.controller.ConfigurationContext;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.reporting.InitializationException;
+
+import java.io.IOException;
+import java.net.URISyntaxException;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.List;
+import java.util.Objects;
+
+@Tags({"Azure", "ADX", "Kusto", "ingest", "azure"})
+@CapabilityDescription("Sends batches of flowfile content or stream flowfile content to an Azure ADX cluster.")
+@ReadsAttributes({
+ @ReadsAttribute(attribute = "AUTH_STRATEGY", description = "The strategy/method to authenticate against Azure Active Directory, either 'application' or 'managed_identity'."),
+ @ReadsAttribute(attribute = "APP_ID", description = "Specifies Azure application id for accessing the ADX-Cluster."),
+ @ReadsAttribute(attribute = "APP_KEY", description = "Specifies Azure application key for accessing the ADX-Cluster."),
+ @ReadsAttribute(attribute = "APP_TENANT", description = "Azure application tenant for accessing the ADX-Cluster."),
+ @ReadsAttribute(attribute = "CLUSTER_URL", description = "Endpoint of ADX cluster. This is required only when streaming data to ADX cluster is enabled."),
+})
+public class StandardKustoIngestService extends AbstractControllerService implements KustoIngestService {
+
+ public static final PropertyDescriptor AUTHENTICATION_STRATEGY = new PropertyDescriptor
+ .Builder().name("Authentication Strategy")
+ .displayName("Authentication Strategy")
+ .description("Authentication method for access to Azure Data Explorer")
+ .required(true)
+ .defaultValue(KustoAuthenticationStrategy.MANAGED_IDENTITY.getValue())
+ .allowableValues(KustoAuthenticationStrategy.class)
+ .build();
+
+ public static final PropertyDescriptor APPLICATION_CLIENT_ID = new PropertyDescriptor
+ .Builder().name("Application Client ID")
+ .displayName("Application Client ID")
+ .description("Azure Data Explorer Application Client Identifier for Authentication")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+
+ public static final PropertyDescriptor APPLICATION_KEY = new PropertyDescriptor
+ .Builder().name("Application Key")
+ .displayName("Application Key")
+ .description("Azure Data Explorer Application Key for Authentication")
+ .required(true)
+ .sensitive(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .dependsOn(AUTHENTICATION_STRATEGY, KustoAuthenticationStrategy.APPLICATION_CREDENTIALS.getValue())
+ .build();
+
+ public static final PropertyDescriptor APPLICATION_TENANT_ID = new PropertyDescriptor.Builder()
+ .name("Application Tenant ID")
+ .displayName("Application Tenant ID")
+ .description("Azure Data Explorer Application Tenant Identifier for Authentication")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .dependsOn(AUTHENTICATION_STRATEGY, KustoAuthenticationStrategy.APPLICATION_CREDENTIALS.getValue())
+ .build();
+
+ public static final PropertyDescriptor CLUSTER_URI = new PropertyDescriptor
+ .Builder().name("Cluster URI")
+ .displayName("Cluster URI")
+ .description("Azure Data Explorer Cluster URI")
+ .required(true)
+ .addValidator(StandardValidators.URL_VALIDATOR)
+ .build();
+
+ private static final List<PropertyDescriptor> PROPERTY_DESCRIPTORS = Collections.unmodifiableList(Arrays.asList(
+ AUTHENTICATION_STRATEGY,
+ APPLICATION_CLIENT_ID,
+ APPLICATION_KEY,
+ APPLICATION_TENANT_ID,
+ CLUSTER_URI));
+
+ public static final Pair<String, String> NIFI_SINK = Pair.of("processor", "nifi-sink");
+
+ private volatile QueuedIngestClient queuedIngestClient;
+
+ private volatile ManagedStreamingIngestClient managedStreamingIngestClient;
+
+ private volatile Client executionClient;
+
+ @Override
+ public List<PropertyDescriptor> getSupportedPropertyDescriptors() {
+ return PROPERTY_DESCRIPTORS;
+ }
+
+ /**
+ * @param context the configuration context
+ * @throws InitializationException if unable to create a database connection
+ */
+ @OnEnabled
+ public void onEnabled(final ConfigurationContext context) throws ProcessException, URISyntaxException {
+
+ getLogger().info("Starting Azure ADX Connection Service...");
+ final String applicationClientId = context.getProperty(APPLICATION_CLIENT_ID).getValue();
+ final String applicationKey = context.getProperty(APPLICATION_KEY).getValue();
+ final String applicationTenantId = context.getProperty(APPLICATION_TENANT_ID).getValue();
+ final String clusterUri = context.getProperty(CLUSTER_URI).getValue();
+ final KustoAuthenticationStrategy kustoAuthenticationStrategy = KustoAuthenticationStrategy.valueOf(context.getProperty(AUTHENTICATION_STRATEGY).getValue());
+
+ if (this.queuedIngestClient == null) {
+ this.queuedIngestClient = createKustoQueuedIngestClient(clusterUri, applicationClientId, applicationKey, applicationTenantId, kustoAuthenticationStrategy);
+ }
+
+ if (this.managedStreamingIngestClient == null) {
+ this.managedStreamingIngestClient = createKustoStreamingIngestClient(clusterUri, applicationClientId, applicationKey, applicationTenantId, kustoAuthenticationStrategy);
+ }
+
+ if (this.executionClient == null) {
+ this.executionClient = createKustoExecutionClient(clusterUri, applicationClientId, applicationKey, applicationTenantId, kustoAuthenticationStrategy);
+ }
+
+ }
+
+ @OnStopped
+ public final void onStopped() {
+ if (this.queuedIngestClient != null) {
+ try {
+ this.queuedIngestClient.close();
+ } catch (IOException e) {
+ getLogger().error("Closing Azure ADX Queued Ingest Client failed ", e);
Review Comment:
The extra space should be removed.
```suggestion
getLogger().error("Closing Azure ADX Queued Ingest Client failed", e);
```
##########
nifi-nar-bundles/nifi-azure-bundle/nifi-azure-processors/src/main/java/org/apache/nifi/services/azure/data/explorer/StandardKustoIngestService.java:
##########
@@ -0,0 +1,293 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.services.azure.data.explorer;
+
+import com.microsoft.azure.kusto.data.Client;
+import com.microsoft.azure.kusto.data.ClientFactory;
+import com.microsoft.azure.kusto.data.KustoResultSetTable;
+import com.microsoft.azure.kusto.data.auth.ConnectionStringBuilder;
+import com.microsoft.azure.kusto.data.exceptions.DataClientException;
+import com.microsoft.azure.kusto.data.exceptions.DataServiceException;
+import com.microsoft.azure.kusto.ingest.IngestClientFactory;
+import com.microsoft.azure.kusto.ingest.ManagedStreamingIngestClient;
+import com.microsoft.azure.kusto.ingest.QueuedIngestClient;
+import com.microsoft.azure.kusto.ingest.exceptions.IngestionClientException;
+import com.microsoft.azure.kusto.ingest.exceptions.IngestionServiceException;
+import com.microsoft.azure.kusto.ingest.result.IngestionResult;
+import com.microsoft.azure.kusto.ingest.result.IngestionStatus;
+import com.microsoft.azure.kusto.ingest.result.OperationStatus;
+import com.microsoft.azure.kusto.ingest.source.StreamSourceInfo;
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.nifi.annotation.behavior.ReadsAttribute;
+import org.apache.nifi.annotation.behavior.ReadsAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnEnabled;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.controller.AbstractControllerService;
+import org.apache.nifi.controller.ConfigurationContext;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.reporting.InitializationException;
+
+import java.io.IOException;
+import java.net.URISyntaxException;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.List;
+import java.util.Objects;
+
+@Tags({"Azure", "ADX", "Kusto", "ingest", "azure"})
+@CapabilityDescription("Sends batches of flowfile content or stream flowfile content to an Azure ADX cluster.")
+@ReadsAttributes({
+ @ReadsAttribute(attribute = "AUTH_STRATEGY", description = "The strategy/method to authenticate against Azure Active Directory, either 'application' or 'managed_identity'."),
+ @ReadsAttribute(attribute = "APP_ID", description = "Specifies Azure application id for accessing the ADX-Cluster."),
+ @ReadsAttribute(attribute = "APP_KEY", description = "Specifies Azure application key for accessing the ADX-Cluster."),
+ @ReadsAttribute(attribute = "APP_TENANT", description = "Azure application tenant for accessing the ADX-Cluster."),
+ @ReadsAttribute(attribute = "CLUSTER_URL", description = "Endpoint of ADX cluster. This is required only when streaming data to ADX cluster is enabled."),
+})
+public class StandardKustoIngestService extends AbstractControllerService implements KustoIngestService {
+
+ public static final PropertyDescriptor AUTHENTICATION_STRATEGY = new PropertyDescriptor
+ .Builder().name("Authentication Strategy")
+ .displayName("Authentication Strategy")
+ .description("Authentication method for access to Azure Data Explorer")
+ .required(true)
+ .defaultValue(KustoAuthenticationStrategy.MANAGED_IDENTITY.getValue())
+ .allowableValues(KustoAuthenticationStrategy.class)
+ .build();
+
+ public static final PropertyDescriptor APPLICATION_CLIENT_ID = new PropertyDescriptor
+ .Builder().name("Application Client ID")
+ .displayName("Application Client ID")
+ .description("Azure Data Explorer Application Client Identifier for Authentication")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+
+ public static final PropertyDescriptor APPLICATION_KEY = new PropertyDescriptor
+ .Builder().name("Application Key")
+ .displayName("Application Key")
+ .description("Azure Data Explorer Application Key for Authentication")
+ .required(true)
+ .sensitive(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .dependsOn(AUTHENTICATION_STRATEGY, KustoAuthenticationStrategy.APPLICATION_CREDENTIALS.getValue())
+ .build();
+
+ public static final PropertyDescriptor APPLICATION_TENANT_ID = new PropertyDescriptor.Builder()
+ .name("Application Tenant ID")
+ .displayName("Application Tenant ID")
+ .description("Azure Data Explorer Application Tenant Identifier for Authentication")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .dependsOn(AUTHENTICATION_STRATEGY, KustoAuthenticationStrategy.APPLICATION_CREDENTIALS.getValue())
+ .build();
+
+ public static final PropertyDescriptor CLUSTER_URI = new PropertyDescriptor
+ .Builder().name("Cluster URI")
+ .displayName("Cluster URI")
+ .description("Azure Data Explorer Cluster URI")
+ .required(true)
+ .addValidator(StandardValidators.URL_VALIDATOR)
+ .build();
+
+ private static final List<PropertyDescriptor> PROPERTY_DESCRIPTORS = Collections.unmodifiableList(Arrays.asList(
+ AUTHENTICATION_STRATEGY,
+ APPLICATION_CLIENT_ID,
+ APPLICATION_KEY,
+ APPLICATION_TENANT_ID,
+ CLUSTER_URI));
+
+ public static final Pair<String, String> NIFI_SINK = Pair.of("processor", "nifi-sink");
+
+ private volatile QueuedIngestClient queuedIngestClient;
+
+ private volatile ManagedStreamingIngestClient managedStreamingIngestClient;
+
+ private volatile Client executionClient;
+
+ @Override
+ public List<PropertyDescriptor> getSupportedPropertyDescriptors() {
+ return PROPERTY_DESCRIPTORS;
+ }
+
+ /**
+ * @param context the configuration context
+ * @throws InitializationException if unable to create a database connection
+ */
+ @OnEnabled
+ public void onEnabled(final ConfigurationContext context) throws ProcessException, URISyntaxException {
+
+ getLogger().info("Starting Azure ADX Connection Service...");
+ final String applicationClientId = context.getProperty(APPLICATION_CLIENT_ID).getValue();
+ final String applicationKey = context.getProperty(APPLICATION_KEY).getValue();
+ final String applicationTenantId = context.getProperty(APPLICATION_TENANT_ID).getValue();
+ final String clusterUri = context.getProperty(CLUSTER_URI).getValue();
+ final KustoAuthenticationStrategy kustoAuthenticationStrategy = KustoAuthenticationStrategy.valueOf(context.getProperty(AUTHENTICATION_STRATEGY).getValue());
+
+ if (this.queuedIngestClient == null) {
+ this.queuedIngestClient = createKustoQueuedIngestClient(clusterUri, applicationClientId, applicationKey, applicationTenantId, kustoAuthenticationStrategy);
+ }
+
+ if (this.managedStreamingIngestClient == null) {
+ this.managedStreamingIngestClient = createKustoStreamingIngestClient(clusterUri, applicationClientId, applicationKey, applicationTenantId, kustoAuthenticationStrategy);
+ }
+
+ if (this.executionClient == null) {
+ this.executionClient = createKustoExecutionClient(clusterUri, applicationClientId, applicationKey, applicationTenantId, kustoAuthenticationStrategy);
+ }
+
+ }
+
+ @OnStopped
+ public final void onStopped() {
+ if (this.queuedIngestClient != null) {
+ try {
+ this.queuedIngestClient.close();
+ } catch (IOException e) {
+ getLogger().error("Closing Azure ADX Queued Ingest Client failed ", e);
+ } finally {
+ this.queuedIngestClient = null;
+ }
+ }
+ if (this.managedStreamingIngestClient != null) {
+ try {
+ this.managedStreamingIngestClient.close();
+ } catch (IOException e) {
+ getLogger().error("Closing Azure ADX Managed Streaming Ingest Client failed", e);
+ } finally {
+ this.managedStreamingIngestClient = null;
+ }
+ }
+ if (this.executionClient != null) {
+ try {
+ this.executionClient.close();
+ } catch (IOException e) {
+ getLogger().error("Closing Azure ADX Execution Client failed", e);
+ } finally {
+ this.executionClient = null;
+ }
+ }
+ }
+
+
+ protected QueuedIngestClient createKustoQueuedIngestClient(final String clusterUrl,
+ final String appId,
+ final String appKey,
+ final String appTenant,
+ final KustoAuthenticationStrategy kustoAuthStrategy) throws URISyntaxException {
+ ConnectionStringBuilder ingestConnectionStringBuilder = createKustoEngineConnectionString(clusterUrl, appId, appKey, appTenant, kustoAuthStrategy);
+ return IngestClientFactory.createClient(ingestConnectionStringBuilder);
+ }
+
+ protected ManagedStreamingIngestClient createKustoStreamingIngestClient(final String clusterUrl,
+ final String appId,
+ final String appKey,
+ final String appTenant,
+ final KustoAuthenticationStrategy kustoAuthStrategy) throws URISyntaxException {
+ ConnectionStringBuilder ingestConnectionStringBuilder = createKustoEngineConnectionString(clusterUrl, appId, appKey, appTenant, kustoAuthStrategy);
+ ConnectionStringBuilder streamingConnectionStringBuilder = createKustoEngineConnectionString(clusterUrl, appId, appKey, appTenant, kustoAuthStrategy);
+ return IngestClientFactory.createManagedStreamingIngestClient(ingestConnectionStringBuilder, streamingConnectionStringBuilder);
+ }
+
+ public KustoIngestionResult ingestData(KustoIngestionRequest kustoIngestionRequest) throws URISyntaxException {
+ StreamSourceInfo info = new StreamSourceInfo(kustoIngestionRequest.getInputStream());
+ //ingest data
+ IngestionResult ingestionResult;
+ try {
+ if (kustoIngestionRequest.isStreamingEnabled()) {
+ ingestionResult = managedStreamingIngestClient.ingestFromStream(info, kustoIngestionRequest.getIngestionProperties());
+ } else {
+ ingestionResult = queuedIngestClient.ingestFromStream(info, kustoIngestionRequest.getIngestionProperties());
+ }
+ if (kustoIngestionRequest.pollOnIngestionStatus()) {
+ List<IngestionStatus> statuses;
+ long startTime = System.currentTimeMillis();
+ long timeoutMillis = 600 * 1800; // 10 Minutes timeout
+ while (true) {
+ // Get the status of the ingestion operation
+ List<IngestionStatus> statuses1 = ingestionResult.getIngestionStatusCollection();
+ if (statuses1.get(0).status == OperationStatus.Succeeded
+ || statuses1.get(0).status == OperationStatus.Failed
+ || statuses1.get(0).status == OperationStatus.PartiallySucceeded) {
+ statuses = statuses1;
+ break;
+ }
+ // Check if the timeout has been exceeded
+ if (System.currentTimeMillis() - startTime >= timeoutMillis) {
+ throw new ProcessException("Timeout exceeded while waiting for ingestion status");
+ }
+
+ // Sleep for 5 seconds before checking again
+ Thread.sleep(5000);
Review Comment:
This is a long time to wait and will block other framework operations. At minimum, this value should be a configurable property, along the lines of `Status Polling Interval`.
##########
nifi-nar-bundles/nifi-azure-bundle/nifi-azure-processors/src/main/java/org/apache/nifi/services/azure/data/explorer/StandardKustoIngestService.java:
##########
@@ -0,0 +1,293 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.services.azure.data.explorer;
+
+import com.microsoft.azure.kusto.data.Client;
+import com.microsoft.azure.kusto.data.ClientFactory;
+import com.microsoft.azure.kusto.data.KustoResultSetTable;
+import com.microsoft.azure.kusto.data.auth.ConnectionStringBuilder;
+import com.microsoft.azure.kusto.data.exceptions.DataClientException;
+import com.microsoft.azure.kusto.data.exceptions.DataServiceException;
+import com.microsoft.azure.kusto.ingest.IngestClientFactory;
+import com.microsoft.azure.kusto.ingest.ManagedStreamingIngestClient;
+import com.microsoft.azure.kusto.ingest.QueuedIngestClient;
+import com.microsoft.azure.kusto.ingest.exceptions.IngestionClientException;
+import com.microsoft.azure.kusto.ingest.exceptions.IngestionServiceException;
+import com.microsoft.azure.kusto.ingest.result.IngestionResult;
+import com.microsoft.azure.kusto.ingest.result.IngestionStatus;
+import com.microsoft.azure.kusto.ingest.result.OperationStatus;
+import com.microsoft.azure.kusto.ingest.source.StreamSourceInfo;
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.nifi.annotation.behavior.ReadsAttribute;
+import org.apache.nifi.annotation.behavior.ReadsAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnEnabled;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.controller.AbstractControllerService;
+import org.apache.nifi.controller.ConfigurationContext;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.reporting.InitializationException;
+
+import java.io.IOException;
+import java.net.URISyntaxException;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.List;
+import java.util.Objects;
+
+@Tags({"Azure", "ADX", "Kusto", "ingest", "azure"})
+@CapabilityDescription("Sends batches of flowfile content or stream flowfile content to an Azure ADX cluster.")
+@ReadsAttributes({
+ @ReadsAttribute(attribute = "AUTH_STRATEGY", description = "The strategy/method to authenticate against Azure Active Directory, either 'application' or 'managed_identity'."),
+ @ReadsAttribute(attribute = "APP_ID", description = "Specifies Azure application id for accessing the ADX-Cluster."),
+ @ReadsAttribute(attribute = "APP_KEY", description = "Specifies Azure application key for accessing the ADX-Cluster."),
+ @ReadsAttribute(attribute = "APP_TENANT", description = "Azure application tenant for accessing the ADX-Cluster."),
+ @ReadsAttribute(attribute = "CLUSTER_URL", description = "Endpoint of ADX cluster. This is required only when streaming data to ADX cluster is enabled."),
+})
+public class StandardKustoIngestService extends AbstractControllerService implements KustoIngestService {
+
+ public static final PropertyDescriptor AUTHENTICATION_STRATEGY = new PropertyDescriptor
+ .Builder().name("Authentication Strategy")
+ .displayName("Authentication Strategy")
+ .description("Authentication method for access to Azure Data Explorer")
+ .required(true)
+ .defaultValue(KustoAuthenticationStrategy.MANAGED_IDENTITY.getValue())
+ .allowableValues(KustoAuthenticationStrategy.class)
+ .build();
+
+ public static final PropertyDescriptor APPLICATION_CLIENT_ID = new PropertyDescriptor
+ .Builder().name("Application Client ID")
+ .displayName("Application Client ID")
+ .description("Azure Data Explorer Application Client Identifier for Authentication")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+
+ public static final PropertyDescriptor APPLICATION_KEY = new PropertyDescriptor
+ .Builder().name("Application Key")
+ .displayName("Application Key")
+ .description("Azure Data Explorer Application Key for Authentication")
+ .required(true)
+ .sensitive(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .dependsOn(AUTHENTICATION_STRATEGY, KustoAuthenticationStrategy.APPLICATION_CREDENTIALS.getValue())
+ .build();
+
+ public static final PropertyDescriptor APPLICATION_TENANT_ID = new PropertyDescriptor.Builder()
+ .name("Application Tenant ID")
+ .displayName("Application Tenant ID")
+ .description("Azure Data Explorer Application Tenant Identifier for Authentication")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .dependsOn(AUTHENTICATION_STRATEGY, KustoAuthenticationStrategy.APPLICATION_CREDENTIALS.getValue())
+ .build();
+
+ public static final PropertyDescriptor CLUSTER_URI = new PropertyDescriptor
+ .Builder().name("Cluster URI")
+ .displayName("Cluster URI")
+ .description("Azure Data Explorer Cluster URI")
+ .required(true)
+ .addValidator(StandardValidators.URL_VALIDATOR)
+ .build();
+
+ private static final List<PropertyDescriptor> PROPERTY_DESCRIPTORS = Collections.unmodifiableList(Arrays.asList(
+ AUTHENTICATION_STRATEGY,
+ APPLICATION_CLIENT_ID,
+ APPLICATION_KEY,
+ APPLICATION_TENANT_ID,
+ CLUSTER_URI));
+
+ public static final Pair<String, String> NIFI_SINK = Pair.of("processor", "nifi-sink");
+
+ private volatile QueuedIngestClient queuedIngestClient;
+
+ private volatile ManagedStreamingIngestClient managedStreamingIngestClient;
+
+ private volatile Client executionClient;
+
+ @Override
+ public List<PropertyDescriptor> getSupportedPropertyDescriptors() {
+ return PROPERTY_DESCRIPTORS;
+ }
+
+ /**
+ * @param context the configuration context
+ * @throws InitializationException if unable to create a database connection
+ */
+ @OnEnabled
+ public void onEnabled(final ConfigurationContext context) throws ProcessException, URISyntaxException {
+
+ getLogger().info("Starting Azure ADX Connection Service...");
+ final String applicationClientId = context.getProperty(APPLICATION_CLIENT_ID).getValue();
+ final String applicationKey = context.getProperty(APPLICATION_KEY).getValue();
+ final String applicationTenantId = context.getProperty(APPLICATION_TENANT_ID).getValue();
+ final String clusterUri = context.getProperty(CLUSTER_URI).getValue();
+ final KustoAuthenticationStrategy kustoAuthenticationStrategy = KustoAuthenticationStrategy.valueOf(context.getProperty(AUTHENTICATION_STRATEGY).getValue());
+
+ if (this.queuedIngestClient == null) {
+ this.queuedIngestClient = createKustoQueuedIngestClient(clusterUri, applicationClientId, applicationKey, applicationTenantId, kustoAuthenticationStrategy);
+ }
+
+ if (this.managedStreamingIngestClient == null) {
+ this.managedStreamingIngestClient = createKustoStreamingIngestClient(clusterUri, applicationClientId, applicationKey, applicationTenantId, kustoAuthenticationStrategy);
+ }
+
+ if (this.executionClient == null) {
+ this.executionClient = createKustoExecutionClient(clusterUri, applicationClientId, applicationKey, applicationTenantId, kustoAuthenticationStrategy);
+ }
+
+ }
+
+ @OnStopped
+ public final void onStopped() {
+ if (this.queuedIngestClient != null) {
+ try {
+ this.queuedIngestClient.close();
+ } catch (IOException e) {
+ getLogger().error("Closing Azure ADX Queued Ingest Client failed ", e);
+ } finally {
+ this.queuedIngestClient = null;
+ }
+ }
+ if (this.managedStreamingIngestClient != null) {
+ try {
+ this.managedStreamingIngestClient.close();
+ } catch (IOException e) {
+ getLogger().error("Closing Azure ADX Managed Streaming Ingest Client failed", e);
+ } finally {
+ this.managedStreamingIngestClient = null;
+ }
+ }
+ if (this.executionClient != null) {
+ try {
+ this.executionClient.close();
+ } catch (IOException e) {
+ getLogger().error("Closing Azure ADX Execution Client failed", e);
+ } finally {
+ this.executionClient = null;
+ }
+ }
+ }
+
+
+ protected QueuedIngestClient createKustoQueuedIngestClient(final String clusterUrl,
+ final String appId,
+ final String appKey,
+ final String appTenant,
+ final KustoAuthenticationStrategy kustoAuthStrategy) throws URISyntaxException {
+ ConnectionStringBuilder ingestConnectionStringBuilder = createKustoEngineConnectionString(clusterUrl, appId, appKey, appTenant, kustoAuthStrategy);
+ return IngestClientFactory.createClient(ingestConnectionStringBuilder);
+ }
+
+ protected ManagedStreamingIngestClient createKustoStreamingIngestClient(final String clusterUrl,
+ final String appId,
+ final String appKey,
+ final String appTenant,
+ final KustoAuthenticationStrategy kustoAuthStrategy) throws URISyntaxException {
+ ConnectionStringBuilder ingestConnectionStringBuilder = createKustoEngineConnectionString(clusterUrl, appId, appKey, appTenant, kustoAuthStrategy);
+ ConnectionStringBuilder streamingConnectionStringBuilder = createKustoEngineConnectionString(clusterUrl, appId, appKey, appTenant, kustoAuthStrategy);
+ return IngestClientFactory.createManagedStreamingIngestClient(ingestConnectionStringBuilder, streamingConnectionStringBuilder);
+ }
+
+ public KustoIngestionResult ingestData(KustoIngestionRequest kustoIngestionRequest) throws URISyntaxException {
+ StreamSourceInfo info = new StreamSourceInfo(kustoIngestionRequest.getInputStream());
+ //ingest data
+ IngestionResult ingestionResult;
+ try {
+ if (kustoIngestionRequest.isStreamingEnabled()) {
+ ingestionResult = managedStreamingIngestClient.ingestFromStream(info, kustoIngestionRequest.getIngestionProperties());
+ } else {
+ ingestionResult = queuedIngestClient.ingestFromStream(info, kustoIngestionRequest.getIngestionProperties());
+ }
+ if (kustoIngestionRequest.pollOnIngestionStatus()) {
+ List<IngestionStatus> statuses;
+ long startTime = System.currentTimeMillis();
+ long timeoutMillis = 600 * 1800; // 10 Minutes timeout
+ while (true) {
+ // Get the status of the ingestion operation
+ List<IngestionStatus> statuses1 = ingestionResult.getIngestionStatusCollection();
+ if (statuses1.get(0).status == OperationStatus.Succeeded
+ || statuses1.get(0).status == OperationStatus.Failed
+ || statuses1.get(0).status == OperationStatus.PartiallySucceeded) {
+ statuses = statuses1;
+ break;
+ }
+ // Check if the timeout has been exceeded
+ if (System.currentTimeMillis() - startTime >= timeoutMillis) {
+ throw new ProcessException("Timeout exceeded while waiting for ingestion status");
+ }
+
+ // Sleep for 5 seconds before checking again
+ Thread.sleep(5000);
+ }
+ return KustoIngestionResult.fromString(statuses.get(0).status.toString());
+ } else {
+ return KustoIngestionResult.SUCCEEDED;
+ }
+ } catch (IngestionClientException | IngestionServiceException e) {
+ throw new ProcessException("Error occurred while ingesting data into ADX", e);
Review Comment:
Recommend rewording the message:
```suggestion
throw new ProcessException("Azure Data Explorer Ingest failed", e);
```
##########
nifi-nar-bundles/nifi-azure-bundle/nifi-azure-processors/src/main/java/org/apache/nifi/services/azure/data/explorer/StandardKustoIngestService.java:
##########
@@ -0,0 +1,293 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.services.azure.data.explorer;
+
+import com.microsoft.azure.kusto.data.Client;
+import com.microsoft.azure.kusto.data.ClientFactory;
+import com.microsoft.azure.kusto.data.KustoResultSetTable;
+import com.microsoft.azure.kusto.data.auth.ConnectionStringBuilder;
+import com.microsoft.azure.kusto.data.exceptions.DataClientException;
+import com.microsoft.azure.kusto.data.exceptions.DataServiceException;
+import com.microsoft.azure.kusto.ingest.IngestClientFactory;
+import com.microsoft.azure.kusto.ingest.ManagedStreamingIngestClient;
+import com.microsoft.azure.kusto.ingest.QueuedIngestClient;
+import com.microsoft.azure.kusto.ingest.exceptions.IngestionClientException;
+import com.microsoft.azure.kusto.ingest.exceptions.IngestionServiceException;
+import com.microsoft.azure.kusto.ingest.result.IngestionResult;
+import com.microsoft.azure.kusto.ingest.result.IngestionStatus;
+import com.microsoft.azure.kusto.ingest.result.OperationStatus;
+import com.microsoft.azure.kusto.ingest.source.StreamSourceInfo;
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.nifi.annotation.behavior.ReadsAttribute;
+import org.apache.nifi.annotation.behavior.ReadsAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnEnabled;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.controller.AbstractControllerService;
+import org.apache.nifi.controller.ConfigurationContext;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.reporting.InitializationException;
+
+import java.io.IOException;
+import java.net.URISyntaxException;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.List;
+import java.util.Objects;
+
+@Tags({"Azure", "ADX", "Kusto", "ingest", "azure"})
+@CapabilityDescription("Sends batches of flowfile content or stream flowfile content to an Azure ADX cluster.")
+@ReadsAttributes({
+ @ReadsAttribute(attribute = "AUTH_STRATEGY", description = "The strategy/method to authenticate against Azure Active Directory, either 'application' or 'managed_identity'."),
+ @ReadsAttribute(attribute = "APP_ID", description = "Specifies Azure application id for accessing the ADX-Cluster."),
+ @ReadsAttribute(attribute = "APP_KEY", description = "Specifies Azure application key for accessing the ADX-Cluster."),
+ @ReadsAttribute(attribute = "APP_TENANT", description = "Azure application tenant for accessing the ADX-Cluster."),
+ @ReadsAttribute(attribute = "CLUSTER_URL", description = "Endpoint of ADX cluster. This is required only when streaming data to ADX cluster is enabled."),
+})
+public class StandardKustoIngestService extends AbstractControllerService implements KustoIngestService {
+
+ public static final PropertyDescriptor AUTHENTICATION_STRATEGY = new PropertyDescriptor
+ .Builder().name("Authentication Strategy")
+ .displayName("Authentication Strategy")
+ .description("Authentication method for access to Azure Data Explorer")
+ .required(true)
+ .defaultValue(KustoAuthenticationStrategy.MANAGED_IDENTITY.getValue())
+ .allowableValues(KustoAuthenticationStrategy.class)
+ .build();
+
+ public static final PropertyDescriptor APPLICATION_CLIENT_ID = new PropertyDescriptor
+ .Builder().name("Application Client ID")
+ .displayName("Application Client ID")
+ .description("Azure Data Explorer Application Client Identifier for Authentication")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+
+ public static final PropertyDescriptor APPLICATION_KEY = new PropertyDescriptor
+ .Builder().name("Application Key")
+ .displayName("Application Key")
+ .description("Azure Data Explorer Application Key for Authentication")
+ .required(true)
+ .sensitive(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .dependsOn(AUTHENTICATION_STRATEGY, KustoAuthenticationStrategy.APPLICATION_CREDENTIALS.getValue())
+ .build();
+
+ public static final PropertyDescriptor APPLICATION_TENANT_ID = new PropertyDescriptor.Builder()
+ .name("Application Tenant ID")
+ .displayName("Application Tenant ID")
+ .description("Azure Data Explorer Application Tenant Identifier for Authentication")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .dependsOn(AUTHENTICATION_STRATEGY, KustoAuthenticationStrategy.APPLICATION_CREDENTIALS.getValue())
+ .build();
+
+ public static final PropertyDescriptor CLUSTER_URI = new PropertyDescriptor
+ .Builder().name("Cluster URI")
+ .displayName("Cluster URI")
+ .description("Azure Data Explorer Cluster URI")
+ .required(true)
+ .addValidator(StandardValidators.URL_VALIDATOR)
+ .build();
+
+ private static final List<PropertyDescriptor> PROPERTY_DESCRIPTORS = Collections.unmodifiableList(Arrays.asList(
+ AUTHENTICATION_STRATEGY,
+ APPLICATION_CLIENT_ID,
+ APPLICATION_KEY,
+ APPLICATION_TENANT_ID,
+ CLUSTER_URI));
+
+ public static final Pair<String, String> NIFI_SINK = Pair.of("processor", "nifi-sink");
+
+ private volatile QueuedIngestClient queuedIngestClient;
+
+ private volatile ManagedStreamingIngestClient managedStreamingIngestClient;
+
+ private volatile Client executionClient;
+
+ @Override
+ public List<PropertyDescriptor> getSupportedPropertyDescriptors() {
+ return PROPERTY_DESCRIPTORS;
+ }
+
+ /**
+ * @param context the configuration context
+ * @throws InitializationException if unable to create a database connection
+ */
+ @OnEnabled
+ public void onEnabled(final ConfigurationContext context) throws ProcessException, URISyntaxException {
+
+ getLogger().info("Starting Azure ADX Connection Service...");
+ final String applicationClientId = context.getProperty(APPLICATION_CLIENT_ID).getValue();
+ final String applicationKey = context.getProperty(APPLICATION_KEY).getValue();
+ final String applicationTenantId = context.getProperty(APPLICATION_TENANT_ID).getValue();
+ final String clusterUri = context.getProperty(CLUSTER_URI).getValue();
+ final KustoAuthenticationStrategy kustoAuthenticationStrategy = KustoAuthenticationStrategy.valueOf(context.getProperty(AUTHENTICATION_STRATEGY).getValue());
+
+ if (this.queuedIngestClient == null) {
+ this.queuedIngestClient = createKustoQueuedIngestClient(clusterUri, applicationClientId, applicationKey, applicationTenantId, kustoAuthenticationStrategy);
+ }
+
+ if (this.managedStreamingIngestClient == null) {
+ this.managedStreamingIngestClient = createKustoStreamingIngestClient(clusterUri, applicationClientId, applicationKey, applicationTenantId, kustoAuthenticationStrategy);
+ }
+
+ if (this.executionClient == null) {
+ this.executionClient = createKustoExecutionClient(clusterUri, applicationClientId, applicationKey, applicationTenantId, kustoAuthenticationStrategy);
+ }
+
+ }
+
+ @OnStopped
+ public final void onStopped() {
+ if (this.queuedIngestClient != null) {
+ try {
+ this.queuedIngestClient.close();
+ } catch (IOException e) {
+ getLogger().error("Closing Azure ADX Queued Ingest Client failed ", e);
+ } finally {
+ this.queuedIngestClient = null;
+ }
+ }
+ if (this.managedStreamingIngestClient != null) {
+ try {
+ this.managedStreamingIngestClient.close();
+ } catch (IOException e) {
+ getLogger().error("Closing Azure ADX Managed Streaming Ingest Client failed", e);
+ } finally {
+ this.managedStreamingIngestClient = null;
+ }
+ }
+ if (this.executionClient != null) {
+ try {
+ this.executionClient.close();
+ } catch (IOException e) {
+ getLogger().error("Closing Azure ADX Execution Client failed", e);
+ } finally {
+ this.executionClient = null;
+ }
+ }
+ }
+
+
+ protected QueuedIngestClient createKustoQueuedIngestClient(final String clusterUrl,
+ final String appId,
+ final String appKey,
+ final String appTenant,
+ final KustoAuthenticationStrategy kustoAuthStrategy) throws URISyntaxException {
+ ConnectionStringBuilder ingestConnectionStringBuilder = createKustoEngineConnectionString(clusterUrl, appId, appKey, appTenant, kustoAuthStrategy);
+ return IngestClientFactory.createClient(ingestConnectionStringBuilder);
+ }
+
+ protected ManagedStreamingIngestClient createKustoStreamingIngestClient(final String clusterUrl,
+ final String appId,
+ final String appKey,
+ final String appTenant,
+ final KustoAuthenticationStrategy kustoAuthStrategy) throws URISyntaxException {
+ ConnectionStringBuilder ingestConnectionStringBuilder = createKustoEngineConnectionString(clusterUrl, appId, appKey, appTenant, kustoAuthStrategy);
+ ConnectionStringBuilder streamingConnectionStringBuilder = createKustoEngineConnectionString(clusterUrl, appId, appKey, appTenant, kustoAuthStrategy);
+ return IngestClientFactory.createManagedStreamingIngestClient(ingestConnectionStringBuilder, streamingConnectionStringBuilder);
+ }
+
+ public KustoIngestionResult ingestData(KustoIngestionRequest kustoIngestionRequest) throws URISyntaxException {
+ StreamSourceInfo info = new StreamSourceInfo(kustoIngestionRequest.getInputStream());
+ //ingest data
+ IngestionResult ingestionResult;
+ try {
+ if (kustoIngestionRequest.isStreamingEnabled()) {
+ ingestionResult = managedStreamingIngestClient.ingestFromStream(info, kustoIngestionRequest.getIngestionProperties());
+ } else {
+ ingestionResult = queuedIngestClient.ingestFromStream(info, kustoIngestionRequest.getIngestionProperties());
+ }
+ if (kustoIngestionRequest.pollOnIngestionStatus()) {
+ List<IngestionStatus> statuses;
+ long startTime = System.currentTimeMillis();
+ long timeoutMillis = 600 * 1800; // 10 Minutes timeout
+ while (true) {
+ // Get the status of the ingestion operation
+ List<IngestionStatus> statuses1 = ingestionResult.getIngestionStatusCollection();
+ if (statuses1.get(0).status == OperationStatus.Succeeded
+ || statuses1.get(0).status == OperationStatus.Failed
+ || statuses1.get(0).status == OperationStatus.PartiallySucceeded) {
+ statuses = statuses1;
+ break;
+ }
+ // Check if the timeout has been exceeded
+ if (System.currentTimeMillis() - startTime >= timeoutMillis) {
+ throw new ProcessException("Timeout exceeded while waiting for ingestion status");
+ }
+
+ // Sleep for 5 seconds before checking again
+ Thread.sleep(5000);
+ }
+ return KustoIngestionResult.fromString(statuses.get(0).status.toString());
+ } else {
+ return KustoIngestionResult.SUCCEEDED;
+ }
+ } catch (IngestionClientException | IngestionServiceException e) {
+ throw new ProcessException("Error occurred while ingesting data into ADX", e);
+ } catch (InterruptedException e) {
+ Thread.currentThread().interrupt(); //Restore interrupted status
+ throw new ProcessException("Error occurred while waiting for ingestion status", e);
Review Comment:
```suggestion
throw new ProcessException("Azure Data Explorer Ingest interrupted", e);
```
##########
nifi-nar-bundles/nifi-azure-bundle/nifi-azure-processors/src/main/java/org/apache/nifi/services/azure/data/explorer/StandardKustoIngestService.java:
##########
@@ -0,0 +1,293 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.services.azure.data.explorer;
+
+import com.microsoft.azure.kusto.data.Client;
+import com.microsoft.azure.kusto.data.ClientFactory;
+import com.microsoft.azure.kusto.data.KustoResultSetTable;
+import com.microsoft.azure.kusto.data.auth.ConnectionStringBuilder;
+import com.microsoft.azure.kusto.data.exceptions.DataClientException;
+import com.microsoft.azure.kusto.data.exceptions.DataServiceException;
+import com.microsoft.azure.kusto.ingest.IngestClientFactory;
+import com.microsoft.azure.kusto.ingest.ManagedStreamingIngestClient;
+import com.microsoft.azure.kusto.ingest.QueuedIngestClient;
+import com.microsoft.azure.kusto.ingest.exceptions.IngestionClientException;
+import com.microsoft.azure.kusto.ingest.exceptions.IngestionServiceException;
+import com.microsoft.azure.kusto.ingest.result.IngestionResult;
+import com.microsoft.azure.kusto.ingest.result.IngestionStatus;
+import com.microsoft.azure.kusto.ingest.result.OperationStatus;
+import com.microsoft.azure.kusto.ingest.source.StreamSourceInfo;
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.nifi.annotation.behavior.ReadsAttribute;
+import org.apache.nifi.annotation.behavior.ReadsAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnEnabled;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.controller.AbstractControllerService;
+import org.apache.nifi.controller.ConfigurationContext;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.reporting.InitializationException;
+
+import java.io.IOException;
+import java.net.URISyntaxException;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.List;
+import java.util.Objects;
+
+@Tags({"Azure", "ADX", "Kusto", "ingest", "azure"})
+@CapabilityDescription("Sends batches of flowfile content or stream flowfile content to an Azure ADX cluster.")
+@ReadsAttributes({
+ @ReadsAttribute(attribute = "AUTH_STRATEGY", description = "The strategy/method to authenticate against Azure Active Directory, either 'application' or 'managed_identity'."),
+ @ReadsAttribute(attribute = "APP_ID", description = "Specifies Azure application id for accessing the ADX-Cluster."),
+ @ReadsAttribute(attribute = "APP_KEY", description = "Specifies Azure application key for accessing the ADX-Cluster."),
+ @ReadsAttribute(attribute = "APP_TENANT", description = "Azure application tenant for accessing the ADX-Cluster."),
+ @ReadsAttribute(attribute = "CLUSTER_URL", description = "Endpoint of ADX cluster. This is required only when streaming data to ADX cluster is enabled."),
+})
+public class StandardKustoIngestService extends AbstractControllerService implements KustoIngestService {
+
+ public static final PropertyDescriptor AUTHENTICATION_STRATEGY = new PropertyDescriptor
+ .Builder().name("Authentication Strategy")
+ .displayName("Authentication Strategy")
+ .description("Authentication method for access to Azure Data Explorer")
+ .required(true)
+ .defaultValue(KustoAuthenticationStrategy.MANAGED_IDENTITY.getValue())
+ .allowableValues(KustoAuthenticationStrategy.class)
+ .build();
+
+ public static final PropertyDescriptor APPLICATION_CLIENT_ID = new PropertyDescriptor
+ .Builder().name("Application Client ID")
+ .displayName("Application Client ID")
+ .description("Azure Data Explorer Application Client Identifier for Authentication")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+
+ public static final PropertyDescriptor APPLICATION_KEY = new PropertyDescriptor
+ .Builder().name("Application Key")
+ .displayName("Application Key")
+ .description("Azure Data Explorer Application Key for Authentication")
+ .required(true)
+ .sensitive(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .dependsOn(AUTHENTICATION_STRATEGY, KustoAuthenticationStrategy.APPLICATION_CREDENTIALS.getValue())
+ .build();
+
+ public static final PropertyDescriptor APPLICATION_TENANT_ID = new PropertyDescriptor.Builder()
+ .name("Application Tenant ID")
+ .displayName("Application Tenant ID")
+ .description("Azure Data Explorer Application Tenant Identifier for Authentication")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .dependsOn(AUTHENTICATION_STRATEGY, KustoAuthenticationStrategy.APPLICATION_CREDENTIALS.getValue())
+ .build();
+
+ public static final PropertyDescriptor CLUSTER_URI = new PropertyDescriptor
+ .Builder().name("Cluster URI")
+ .displayName("Cluster URI")
+ .description("Azure Data Explorer Cluster URI")
+ .required(true)
+ .addValidator(StandardValidators.URL_VALIDATOR)
+ .build();
+
+ private static final List<PropertyDescriptor> PROPERTY_DESCRIPTORS = Collections.unmodifiableList(Arrays.asList(
+ AUTHENTICATION_STRATEGY,
+ APPLICATION_CLIENT_ID,
+ APPLICATION_KEY,
+ APPLICATION_TENANT_ID,
+ CLUSTER_URI));
+
+ public static final Pair<String, String> NIFI_SINK = Pair.of("processor", "nifi-sink");
+
+ private volatile QueuedIngestClient queuedIngestClient;
+
+ private volatile ManagedStreamingIngestClient managedStreamingIngestClient;
+
+ private volatile Client executionClient;
+
+ @Override
+ public List<PropertyDescriptor> getSupportedPropertyDescriptors() {
+ return PROPERTY_DESCRIPTORS;
+ }
+
+ /**
+ * @param context the configuration context
+ * @throws InitializationException if unable to create a database connection
+ */
+ @OnEnabled
+ public void onEnabled(final ConfigurationContext context) throws ProcessException, URISyntaxException {
+
+ getLogger().info("Starting Azure ADX Connection Service...");
+ final String applicationClientId = context.getProperty(APPLICATION_CLIENT_ID).getValue();
+ final String applicationKey = context.getProperty(APPLICATION_KEY).getValue();
+ final String applicationTenantId = context.getProperty(APPLICATION_TENANT_ID).getValue();
+ final String clusterUri = context.getProperty(CLUSTER_URI).getValue();
+ final KustoAuthenticationStrategy kustoAuthenticationStrategy = KustoAuthenticationStrategy.valueOf(context.getProperty(AUTHENTICATION_STRATEGY).getValue());
+
+ if (this.queuedIngestClient == null) {
+ this.queuedIngestClient = createKustoQueuedIngestClient(clusterUri, applicationClientId, applicationKey, applicationTenantId, kustoAuthenticationStrategy);
+ }
+
+ if (this.managedStreamingIngestClient == null) {
+ this.managedStreamingIngestClient = createKustoStreamingIngestClient(clusterUri, applicationClientId, applicationKey, applicationTenantId, kustoAuthenticationStrategy);
+ }
+
+ if (this.executionClient == null) {
+ this.executionClient = createKustoExecutionClient(clusterUri, applicationClientId, applicationKey, applicationTenantId, kustoAuthenticationStrategy);
+ }
+
+ }
+
+ @OnStopped
+ public final void onStopped() {
+ if (this.queuedIngestClient != null) {
+ try {
+ this.queuedIngestClient.close();
+ } catch (IOException e) {
+ getLogger().error("Closing Azure ADX Queued Ingest Client failed ", e);
+ } finally {
+ this.queuedIngestClient = null;
+ }
+ }
+ if (this.managedStreamingIngestClient != null) {
+ try {
+ this.managedStreamingIngestClient.close();
+ } catch (IOException e) {
+ getLogger().error("Closing Azure ADX Managed Streaming Ingest Client failed", e);
+ } finally {
+ this.managedStreamingIngestClient = null;
+ }
+ }
+ if (this.executionClient != null) {
+ try {
+ this.executionClient.close();
+ } catch (IOException e) {
+ getLogger().error("Closing Azure ADX Execution Client failed", e);
+ } finally {
+ this.executionClient = null;
+ }
+ }
+ }
+
+
+ protected QueuedIngestClient createKustoQueuedIngestClient(final String clusterUrl,
+ final String appId,
+ final String appKey,
+ final String appTenant,
+ final KustoAuthenticationStrategy kustoAuthStrategy) throws URISyntaxException {
+ ConnectionStringBuilder ingestConnectionStringBuilder = createKustoEngineConnectionString(clusterUrl, appId, appKey, appTenant, kustoAuthStrategy);
+ return IngestClientFactory.createClient(ingestConnectionStringBuilder);
+ }
+
+ protected ManagedStreamingIngestClient createKustoStreamingIngestClient(final String clusterUrl,
+ final String appId,
+ final String appKey,
+ final String appTenant,
+ final KustoAuthenticationStrategy kustoAuthStrategy) throws URISyntaxException {
+ ConnectionStringBuilder ingestConnectionStringBuilder = createKustoEngineConnectionString(clusterUrl, appId, appKey, appTenant, kustoAuthStrategy);
+ ConnectionStringBuilder streamingConnectionStringBuilder = createKustoEngineConnectionString(clusterUrl, appId, appKey, appTenant, kustoAuthStrategy);
+ return IngestClientFactory.createManagedStreamingIngestClient(ingestConnectionStringBuilder, streamingConnectionStringBuilder);
+ }
+
+ public KustoIngestionResult ingestData(KustoIngestionRequest kustoIngestionRequest) throws URISyntaxException {
+ StreamSourceInfo info = new StreamSourceInfo(kustoIngestionRequest.getInputStream());
+ //ingest data
+ IngestionResult ingestionResult;
+ try {
+ if (kustoIngestionRequest.isStreamingEnabled()) {
+ ingestionResult = managedStreamingIngestClient.ingestFromStream(info, kustoIngestionRequest.getIngestionProperties());
+ } else {
+ ingestionResult = queuedIngestClient.ingestFromStream(info, kustoIngestionRequest.getIngestionProperties());
+ }
+ if (kustoIngestionRequest.pollOnIngestionStatus()) {
+ List<IngestionStatus> statuses;
+ long startTime = System.currentTimeMillis();
+ long timeoutMillis = 600 * 1800; // 10 Minutes timeout
+ while (true) {
+ // Get the status of the ingestion operation
+ List<IngestionStatus> statuses1 = ingestionResult.getIngestionStatusCollection();
+ if (statuses1.get(0).status == OperationStatus.Succeeded
+ || statuses1.get(0).status == OperationStatus.Failed
+ || statuses1.get(0).status == OperationStatus.PartiallySucceeded) {
+ statuses = statuses1;
+ break;
+ }
+ // Check if the timeout has been exceeded
+ if (System.currentTimeMillis() - startTime >= timeoutMillis) {
+ throw new ProcessException("Timeout exceeded while waiting for ingestion status");
+ }
+
+ // Sleep for 5 seconds before checking again
+ Thread.sleep(5000);
+ }
+ return KustoIngestionResult.fromString(statuses.get(0).status.toString());
+ } else {
+ return KustoIngestionResult.SUCCEEDED;
+ }
+ } catch (IngestionClientException | IngestionServiceException e) {
+ throw new ProcessException("Error occurred while ingesting data into ADX", e);
+ } catch (InterruptedException e) {
+ Thread.currentThread().interrupt(); //Restore interrupted status
+ throw new ProcessException("Error occurred while waiting for ingestion status", e);
+ }
+ }
+
+ private ConnectionStringBuilder createKustoEngineConnectionString(final String clusterUrl,
+ final String appId,
+ final String appKey,
+ final String appTenant,
+ final KustoAuthenticationStrategy kustoAuthStrategy) {
+ final ConnectionStringBuilder builder;
+ if (KustoAuthenticationStrategy.APPLICATION_CREDENTIALS == kustoAuthStrategy) {
+ builder = ConnectionStringBuilder.createWithAadApplicationCredentials(clusterUrl, appId, appKey, appTenant);
+ } else {
+ builder = ConnectionStringBuilder.createWithAadManagedIdentity(clusterUrl, appId);
+ }
+
+
+ builder.setConnectorDetails("Kusto.Nifi.Sink", StandardKustoIngestService.class.getPackage().getImplementationVersion(), null, null, false, null, NIFI_SINK);
+ return builder;
+ }
+
+ protected Client createKustoExecutionClient(final String clusterUrl,
+ final String appId,
+ final String appKey,
+ final String appTenant,
+ final KustoAuthenticationStrategy kustoAuthStrategy) throws URISyntaxException {
+ return ClientFactory.createClient(createKustoEngineConnectionString(clusterUrl, appId, appKey, appTenant, kustoAuthStrategy));
+ }
+
+ @Override
+ public KustoQueryResponse executeQuery(String databaseName, String query) {
+ Objects.requireNonNull(databaseName, "Database Name required");
+ Objects.requireNonNull(query, "Query required");
+ KustoQueryResponse kustoQueryResponse;
+ try {
+ KustoResultSetTable kustoResultSetTable = this.executionClient.execute(databaseName, query).getPrimaryResults();
+ kustoQueryResponse = new KustoQueryResponse(kustoResultSetTable);
+ } catch (DataServiceException | DataClientException e) {
+ getLogger().error("ADX Ingestion : Kusto Query execution failed", e);
+ kustoQueryResponse = new KustoQueryResponse(true, e.getMessage());
+ }
+ return kustoQueryResponse;
+ }
+
+
Review Comment:
```suggestion
```
##########
nifi-nar-bundles/nifi-azure-bundle/nifi-azure-processors/src/main/java/org/apache/nifi/services/azure/data/explorer/StandardKustoIngestService.java:
##########
@@ -0,0 +1,293 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.services.azure.data.explorer;
+
+import com.microsoft.azure.kusto.data.Client;
+import com.microsoft.azure.kusto.data.ClientFactory;
+import com.microsoft.azure.kusto.data.KustoResultSetTable;
+import com.microsoft.azure.kusto.data.auth.ConnectionStringBuilder;
+import com.microsoft.azure.kusto.data.exceptions.DataClientException;
+import com.microsoft.azure.kusto.data.exceptions.DataServiceException;
+import com.microsoft.azure.kusto.ingest.IngestClientFactory;
+import com.microsoft.azure.kusto.ingest.ManagedStreamingIngestClient;
+import com.microsoft.azure.kusto.ingest.QueuedIngestClient;
+import com.microsoft.azure.kusto.ingest.exceptions.IngestionClientException;
+import com.microsoft.azure.kusto.ingest.exceptions.IngestionServiceException;
+import com.microsoft.azure.kusto.ingest.result.IngestionResult;
+import com.microsoft.azure.kusto.ingest.result.IngestionStatus;
+import com.microsoft.azure.kusto.ingest.result.OperationStatus;
+import com.microsoft.azure.kusto.ingest.source.StreamSourceInfo;
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.nifi.annotation.behavior.ReadsAttribute;
+import org.apache.nifi.annotation.behavior.ReadsAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnEnabled;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.controller.AbstractControllerService;
+import org.apache.nifi.controller.ConfigurationContext;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.reporting.InitializationException;
+
+import java.io.IOException;
+import java.net.URISyntaxException;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.List;
+import java.util.Objects;
+
+@Tags({"Azure", "ADX", "Kusto", "ingest", "azure"})
+@CapabilityDescription("Sends batches of flowfile content or stream flowfile content to an Azure ADX cluster.")
+@ReadsAttributes({
+ @ReadsAttribute(attribute = "AUTH_STRATEGY", description = "The strategy/method to authenticate against Azure Active Directory, either 'application' or 'managed_identity'."),
+ @ReadsAttribute(attribute = "APP_ID", description = "Specifies Azure application id for accessing the ADX-Cluster."),
+ @ReadsAttribute(attribute = "APP_KEY", description = "Specifies Azure application key for accessing the ADX-Cluster."),
+ @ReadsAttribute(attribute = "APP_TENANT", description = "Azure application tenant for accessing the ADX-Cluster."),
+ @ReadsAttribute(attribute = "CLUSTER_URL", description = "Endpoint of ADX cluster. This is required only when streaming data to ADX cluster is enabled."),
+})
+public class StandardKustoIngestService extends AbstractControllerService implements KustoIngestService {
+
+ public static final PropertyDescriptor AUTHENTICATION_STRATEGY = new PropertyDescriptor
+ .Builder().name("Authentication Strategy")
+ .displayName("Authentication Strategy")
+ .description("Authentication method for access to Azure Data Explorer")
+ .required(true)
+ .defaultValue(KustoAuthenticationStrategy.MANAGED_IDENTITY.getValue())
+ .allowableValues(KustoAuthenticationStrategy.class)
+ .build();
+
+ public static final PropertyDescriptor APPLICATION_CLIENT_ID = new PropertyDescriptor
+ .Builder().name("Application Client ID")
+ .displayName("Application Client ID")
+ .description("Azure Data Explorer Application Client Identifier for Authentication")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+
+ public static final PropertyDescriptor APPLICATION_KEY = new PropertyDescriptor
+ .Builder().name("Application Key")
+ .displayName("Application Key")
+ .description("Azure Data Explorer Application Key for Authentication")
+ .required(true)
+ .sensitive(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .dependsOn(AUTHENTICATION_STRATEGY, KustoAuthenticationStrategy.APPLICATION_CREDENTIALS.getValue())
+ .build();
+
+ public static final PropertyDescriptor APPLICATION_TENANT_ID = new PropertyDescriptor.Builder()
+ .name("Application Tenant ID")
+ .displayName("Application Tenant ID")
+ .description("Azure Data Explorer Application Tenant Identifier for Authentication")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .dependsOn(AUTHENTICATION_STRATEGY, KustoAuthenticationStrategy.APPLICATION_CREDENTIALS.getValue())
+ .build();
+
+ public static final PropertyDescriptor CLUSTER_URI = new PropertyDescriptor
+ .Builder().name("Cluster URI")
+ .displayName("Cluster URI")
+ .description("Azure Data Explorer Cluster URI")
+ .required(true)
+ .addValidator(StandardValidators.URL_VALIDATOR)
+ .build();
+
+ private static final List<PropertyDescriptor> PROPERTY_DESCRIPTORS = Collections.unmodifiableList(Arrays.asList(
+ AUTHENTICATION_STRATEGY,
+ APPLICATION_CLIENT_ID,
+ APPLICATION_KEY,
+ APPLICATION_TENANT_ID,
+ CLUSTER_URI));
+
+ public static final Pair<String, String> NIFI_SINK = Pair.of("processor", "nifi-sink");
+
+ private volatile QueuedIngestClient queuedIngestClient;
+
+ private volatile ManagedStreamingIngestClient managedStreamingIngestClient;
+
+ private volatile Client executionClient;
+
+ @Override
+ public List<PropertyDescriptor> getSupportedPropertyDescriptors() {
+ return PROPERTY_DESCRIPTORS;
+ }
+
+ /**
+ * @param context the configuration context
+ * @throws InitializationException if unable to create a database connection
+ */
+ @OnEnabled
+ public void onEnabled(final ConfigurationContext context) throws ProcessException, URISyntaxException {
+
+ getLogger().info("Starting Azure ADX Connection Service...");
+ final String applicationClientId = context.getProperty(APPLICATION_CLIENT_ID).getValue();
+ final String applicationKey = context.getProperty(APPLICATION_KEY).getValue();
+ final String applicationTenantId = context.getProperty(APPLICATION_TENANT_ID).getValue();
+ final String clusterUri = context.getProperty(CLUSTER_URI).getValue();
+ final KustoAuthenticationStrategy kustoAuthenticationStrategy = KustoAuthenticationStrategy.valueOf(context.getProperty(AUTHENTICATION_STRATEGY).getValue());
+
+ if (this.queuedIngestClient == null) {
+ this.queuedIngestClient = createKustoQueuedIngestClient(clusterUri, applicationClientId, applicationKey, applicationTenantId, kustoAuthenticationStrategy);
+ }
+
+ if (this.managedStreamingIngestClient == null) {
+ this.managedStreamingIngestClient = createKustoStreamingIngestClient(clusterUri, applicationClientId, applicationKey, applicationTenantId, kustoAuthenticationStrategy);
+ }
+
+ if (this.executionClient == null) {
+ this.executionClient = createKustoExecutionClient(clusterUri, applicationClientId, applicationKey, applicationTenantId, kustoAuthenticationStrategy);
+ }
+
+ }
+
+ @OnStopped
+ public final void onStopped() {
+ if (this.queuedIngestClient != null) {
+ try {
+ this.queuedIngestClient.close();
+ } catch (IOException e) {
+ getLogger().error("Closing Azure ADX Queued Ingest Client failed ", e);
+ } finally {
+ this.queuedIngestClient = null;
+ }
+ }
+ if (this.managedStreamingIngestClient != null) {
+ try {
+ this.managedStreamingIngestClient.close();
+ } catch (IOException e) {
+ getLogger().error("Closing Azure ADX Managed Streaming Ingest Client failed", e);
+ } finally {
+ this.managedStreamingIngestClient = null;
+ }
+ }
+ if (this.executionClient != null) {
+ try {
+ this.executionClient.close();
+ } catch (IOException e) {
+ getLogger().error("Closing Azure ADX Execution Client failed", e);
+ } finally {
+ this.executionClient = null;
+ }
+ }
+ }
+
+
+ protected QueuedIngestClient createKustoQueuedIngestClient(final String clusterUrl,
+ final String appId,
+ final String appKey,
+ final String appTenant,
+ final KustoAuthenticationStrategy kustoAuthStrategy) throws URISyntaxException {
+ ConnectionStringBuilder ingestConnectionStringBuilder = createKustoEngineConnectionString(clusterUrl, appId, appKey, appTenant, kustoAuthStrategy);
+ return IngestClientFactory.createClient(ingestConnectionStringBuilder);
+ }
+
+ protected ManagedStreamingIngestClient createKustoStreamingIngestClient(final String clusterUrl,
+ final String appId,
+ final String appKey,
+ final String appTenant,
+ final KustoAuthenticationStrategy kustoAuthStrategy) throws URISyntaxException {
+ ConnectionStringBuilder ingestConnectionStringBuilder = createKustoEngineConnectionString(clusterUrl, appId, appKey, appTenant, kustoAuthStrategy);
+ ConnectionStringBuilder streamingConnectionStringBuilder = createKustoEngineConnectionString(clusterUrl, appId, appKey, appTenant, kustoAuthStrategy);
+ return IngestClientFactory.createManagedStreamingIngestClient(ingestConnectionStringBuilder, streamingConnectionStringBuilder);
+ }
+
+ public KustoIngestionResult ingestData(KustoIngestionRequest kustoIngestionRequest) throws URISyntaxException {
+ StreamSourceInfo info = new StreamSourceInfo(kustoIngestionRequest.getInputStream());
+ //ingest data
+ IngestionResult ingestionResult;
+ try {
+ if (kustoIngestionRequest.isStreamingEnabled()) {
+ ingestionResult = managedStreamingIngestClient.ingestFromStream(info, kustoIngestionRequest.getIngestionProperties());
+ } else {
+ ingestionResult = queuedIngestClient.ingestFromStream(info, kustoIngestionRequest.getIngestionProperties());
+ }
+ if (kustoIngestionRequest.pollOnIngestionStatus()) {
+ List<IngestionStatus> statuses;
+ long startTime = System.currentTimeMillis();
+ long timeoutMillis = 600 * 1800; // 10 Minutes timeout
+ while (true) {
+ // Get the status of the ingestion operation
+ List<IngestionStatus> statuses1 = ingestionResult.getIngestionStatusCollection();
+ if (statuses1.get(0).status == OperationStatus.Succeeded
+ || statuses1.get(0).status == OperationStatus.Failed
+ || statuses1.get(0).status == OperationStatus.PartiallySucceeded) {
+ statuses = statuses1;
+ break;
+ }
+ // Check if the timeout has been exceeded
+ if (System.currentTimeMillis() - startTime >= timeoutMillis) {
+ throw new ProcessException("Timeout exceeded while waiting for ingestion status");
+ }
+
+ // Sleep for 5 seconds before checking again
+ Thread.sleep(5000);
+ }
+ return KustoIngestionResult.fromString(statuses.get(0).status.toString());
+ } else {
+ return KustoIngestionResult.SUCCEEDED;
+ }
+ } catch (IngestionClientException | IngestionServiceException e) {
+ throw new ProcessException("Error occurred while ingesting data into ADX", e);
+ } catch (InterruptedException e) {
+ Thread.currentThread().interrupt(); //Restore interrupted status
+ throw new ProcessException("Error occurred while waiting for ingestion status", e);
+ }
+ }
+
+ private ConnectionStringBuilder createKustoEngineConnectionString(final String clusterUrl,
+ final String appId,
+ final String appKey,
+ final String appTenant,
+ final KustoAuthenticationStrategy kustoAuthStrategy) {
+ final ConnectionStringBuilder builder;
+ if (KustoAuthenticationStrategy.APPLICATION_CREDENTIALS == kustoAuthStrategy) {
+ builder = ConnectionStringBuilder.createWithAadApplicationCredentials(clusterUrl, appId, appKey, appTenant);
+ } else {
+ builder = ConnectionStringBuilder.createWithAadManagedIdentity(clusterUrl, appId);
+ }
+
+
+ builder.setConnectorDetails("Kusto.Nifi.Sink", StandardKustoIngestService.class.getPackage().getImplementationVersion(), null, null, false, null, NIFI_SINK);
+ return builder;
+ }
+
+ protected Client createKustoExecutionClient(final String clusterUrl,
+ final String appId,
+ final String appKey,
+ final String appTenant,
+ final KustoAuthenticationStrategy kustoAuthStrategy) throws URISyntaxException {
+ return ClientFactory.createClient(createKustoEngineConnectionString(clusterUrl, appId, appKey, appTenant, kustoAuthStrategy));
+ }
+
+ @Override
+ public KustoQueryResponse executeQuery(String databaseName, String query) {
+ Objects.requireNonNull(databaseName, "Database Name required");
+ Objects.requireNonNull(query, "Query required");
+ KustoQueryResponse kustoQueryResponse;
+ try {
+ KustoResultSetTable kustoResultSetTable = this.executionClient.execute(databaseName, query).getPrimaryResults();
+ kustoQueryResponse = new KustoQueryResponse(kustoResultSetTable);
+ } catch (DataServiceException | DataClientException e) {
+ getLogger().error("ADX Ingestion : Kusto Query execution failed", e);
Review Comment:
```suggestion
getLogger().error("Azure Data Explorer Ingest execution failed", e);
```
##########
nifi-nar-bundles/nifi-azure-bundle/nifi-azure-processors/src/main/java/org/apache/nifi/processors/azure/data/explorer/PutAzureDataExplorer.java:
##########
@@ -0,0 +1,399 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.azure.data.explorer;
+
+import com.microsoft.azure.kusto.data.KustoResultSetTable;
+import com.microsoft.azure.kusto.ingest.IngestionMapping;
+import com.microsoft.azure.kusto.ingest.IngestionProperties;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.nifi.annotation.behavior.ReadsAttribute;
+import org.apache.nifi.annotation.behavior.ReadsAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.expression.ExpressionLanguageScope;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.services.azure.data.explorer.KustoIngestService;
+import org.apache.nifi.services.azure.data.explorer.KustoIngestionRequest;
+import org.apache.nifi.services.azure.data.explorer.KustoIngestionResult;
+import org.apache.nifi.services.azure.data.explorer.KustoQueryResponse;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.net.URISyntaxException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+
+@Tags({"Azure", "Kusto", "ADX", "Explorer", "Data"})
+@CapabilityDescription("The PutAzureDataExplorer acts as a ADX sink connector which sends flowFiles using the ADX-Service to the provided Azure Data" +
+ "Explorer Ingest Endpoint. The data can be sent through queued ingestion or streaming ingestion to the Azure Data Explorer cluster.")
+@ReadsAttributes({
+ @ReadsAttribute(attribute = "DB_NAME", description = "Specifies the name of the ADX database where the data needs to be stored."),
+ @ReadsAttribute(attribute = "TABLE_NAME", description = "Specifies the name of the ADX table where the data needs to be stored."),
+ @ReadsAttribute(attribute = "MAPPING_NAME", description = "Specifies the name of the mapping responsible for storing the data in appropriate columns."),
+ @ReadsAttribute(attribute = "FLUSH_IMMEDIATE", description = "In case of queued ingestion, this property determines whether the data should be flushed immediately to the ingest endpoint."),
+ @ReadsAttribute(attribute = "DATA_FORMAT", description = "Specifies the format of data that is send to Azure Data Explorer."),
+ @ReadsAttribute(attribute = "IGNORE_FIRST_RECORD", description = "Specifies whether we want to ignore ingestion of first record. " +
+ "This is primarily applicable for csv files. Default is set to NO"),
+ @ReadsAttribute(attribute = "POLL_ON_INGESTION_STATUS", description = "Specifies whether we want to poll on ingestion result during ingestion into ADX." +
+ "In case of applications that need high throughput it is recommended to keep the default value as false. Default is set to false." +
+ "This property should be set to true during Queued Ingestion for near realtime micro-batches of data that require acknowledgement of ingestion status.")
+})
+public class PutAzureDataExplorer extends AbstractProcessor {
+
+ public static final String FETCH_TABLE_COMMAND = "%s | count";
+ public static final String STREAMING_POLICY_SHOW_COMMAND = ".show %s %s policy streamingingestion";
+ public static final String DATABASE = "database";
+
+ private List<PropertyDescriptor> descriptors;
+ private Set<Relationship> relationships;
+ private transient KustoIngestService service;
+ private boolean isStreamingEnabled;
+ private boolean pollOnIngestionStatus;
+
+ public static final AllowableValue AVRO = new AllowableValue(
+ IngestionProperties.DataFormat.AVRO.getKustoValue(), IngestionProperties.DataFormat.AVRO.getKustoValue(),
+ "An Avro format with support for logical types and for the snappy compression codec");
+
+ public static final AllowableValue APACHEAVRO = new AllowableValue(
+ IngestionProperties.DataFormat.APACHEAVRO.getKustoValue(), IngestionProperties.DataFormat.APACHEAVRO.getKustoValue(),
+ "An Avro format with support for logical types and for the snappy compression codec.");
+
+ public static final AllowableValue CSV = new AllowableValue(
+ IngestionProperties.DataFormat.CSV.name(), IngestionProperties.DataFormat.CSV.getKustoValue(),
+ "A text file with comma-separated values (,). For more information, see RFC 4180: Common Format " +
+ "and MIME Type for Comma-Separated Values (CSV) Files.");
+
+ public static final AllowableValue JSON = new AllowableValue(
+ IngestionProperties.DataFormat.JSON.name(), IngestionProperties.DataFormat.JSON.getKustoValue(),
+ "A text file containing JSON objects separated by \\n or \\r\\n. For more information, " +
+ "see JSON Lines (JSONL).");
+
+ public static final AllowableValue MULTIJSON = new AllowableValue(
+ IngestionProperties.DataFormat.MULTIJSON.name(), IngestionProperties.DataFormat.MULTIJSON.getKustoValue(),
+ "A text file containing a JSON array of property containers (each representing a record) or any " +
+ "number of property containers separated by spaces, \\n or \\r\\n. Each property container may be " +
+ "spread across multiple lines. This format is preferable to JSON unless the data is not property " +
+ "containers.");
+
+ public static final AllowableValue ORC = new AllowableValue(
+ IngestionProperties.DataFormat.ORC.name(), IngestionProperties.DataFormat.ORC.getKustoValue(), "An ORC file.");
+
+ public static final AllowableValue PARQUET = new AllowableValue(
+ IngestionProperties.DataFormat.PARQUET.name(), IngestionProperties.DataFormat.PARQUET.getKustoValue(), "A parquet file.");
+
+ public static final AllowableValue PSV = new AllowableValue(
+ IngestionProperties.DataFormat.PSV.name(), IngestionProperties.DataFormat.PSV.getKustoValue(), "A text file with values separated by vertical bars (|).");
+
+ public static final AllowableValue SCSV = new AllowableValue(
+ IngestionProperties.DataFormat.SCSV.name(), IngestionProperties.DataFormat.SCSV.getKustoValue(), "A text file with values separated by semicolons (;).");
+
+ public static final AllowableValue SOHSV = new AllowableValue(
+ IngestionProperties.DataFormat.SOHSV.name(), IngestionProperties.DataFormat.SOHSV.getKustoValue(),
+ "A text file with SOH-separated values. (SOH is the ASCII code point 1. " +
+ "This format is used by Hive in HDInsight).");
+
+ public static final AllowableValue TSV = new AllowableValue(
+ IngestionProperties.DataFormat.TSV.name(), IngestionProperties.DataFormat.TSV.getKustoValue(), "A text file with tab delimited values (\\t).");
+
+ public static final AllowableValue TSVE = new AllowableValue(
+ IngestionProperties.DataFormat.TSVE.name(), IngestionProperties.DataFormat.TSVE.getKustoValue(),
+ "A text file with tab-delimited values (\\t). A backslash (\\) is used as escape character.");
+
+ public static final AllowableValue TXT = new AllowableValue(
+ IngestionProperties.DataFormat.TXT.name(), IngestionProperties.DataFormat.TXT.getKustoValue(),
+ "A text file with lines separated by \\n. Empty lines are skipped.");
+
+ public static final AllowableValue IGNORE_FIRST_RECORD_YES = new AllowableValue(
+ "YES", "YES",
+ "Ignore first record during ingestion");
+
+ public static final AllowableValue IGNORE_FIRST_RECORD_NO = new AllowableValue(
+ "NO", "NO",
+ "Do not ignore first record during ingestion");
+
+ public static final PropertyDescriptor DATABASE_NAME = new PropertyDescriptor.Builder()
+ .name("Database Name")
+ .displayName("Database Name")
+ .description("Azure Data Explorer Database Name for querying")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+ .build();
+
+ public static final PropertyDescriptor TABLE_NAME = new PropertyDescriptor.Builder()
+ .name("Table Name")
+ .displayName("Table Name")
+ .description("Azure Data Explorer Table Name for ingesting data")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+ .build();
+
+ public static final PropertyDescriptor MAPPING_NAME = new PropertyDescriptor
+ .Builder().name("Mapping name")
+ .displayName("Mapping name")
+ .description("The name of the mapping responsible for storing the data in the appropriate columns.")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+
+ public static final PropertyDescriptor IS_STREAMING_ENABLED = new PropertyDescriptor
+ .Builder().name("Is Streaming enabled")
+ .displayName("Is Streaming enabled")
+ .description("This property determines whether we want to stream data to ADX.")
+ .required(false)
+ .allowableValues("true", "false")
+ .addValidator(StandardValidators.BOOLEAN_VALIDATOR)
+ .defaultValue("false")
+ .build();
+
+ public static final PropertyDescriptor POLL_ON_INGESTION_STATUS = new PropertyDescriptor
+ .Builder().name("Poll on ingestion status")
+ .displayName("Whether to Poll on ingestion status")
+ .description("This property determines whether we want to poll on ingestion status after an ingestion to ADX is completed")
+ .required(false)
+ .allowableValues("true", "false")
+ .addValidator(StandardValidators.BOOLEAN_VALIDATOR)
+ .defaultValue("false")
+ .build();
+ public static final PropertyDescriptor ADX_SERVICE = new PropertyDescriptor
+ .Builder().name("Kusto Ingest Service")
+ .displayName("Kusto Ingest Service")
+ .description("Azure Data Explorer Kusto Ingest Service")
+ .required(true)
+ .identifiesControllerService(KustoIngestService.class)
+ .build();
+
+ public static final Relationship SUCCESS = new Relationship.Builder()
+ .name("success")
+ .description("Relationship for success")
+ .build();
+ public static final Relationship FAILURE = new Relationship.Builder()
+ .name("failure")
+ .description("Relationship for failure")
+ .build();
+
+ static final PropertyDescriptor DATA_FORMAT = new PropertyDescriptor.Builder()
+ .name("Data Format")
+ .displayName("Data Format")
+ .description("The format of the data that is sent to ADX.")
+ .required(true)
+ .allowableValues(AVRO, APACHEAVRO, CSV, JSON, MULTIJSON, ORC, PARQUET, PSV, SCSV, SOHSV, TSV, TSVE, TXT)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+
+ static final PropertyDescriptor IGNORE_FIRST_RECORD = new PropertyDescriptor.Builder()
+ .name("Ingestion Ignore First Record")
+ .displayName("Ingestion Ignore First Record")
+ .description("Defines whether ignore first record while ingestion.")
+ .required(false)
+ .allowableValues(IGNORE_FIRST_RECORD_YES, IGNORE_FIRST_RECORD_NO)
+ .defaultValue(IGNORE_FIRST_RECORD_NO.getValue())
+ .build();
+
+ @Override
+ protected void init(final ProcessorInitializationContext context) {
+ final List<PropertyDescriptor> descriptorList = new ArrayList<>();
+ descriptorList.add(ADX_SERVICE);
+ descriptorList.add(DATABASE_NAME);
+ descriptorList.add(TABLE_NAME);
+ descriptorList.add(MAPPING_NAME);
+ descriptorList.add(DATA_FORMAT);
+ descriptorList.add(IGNORE_FIRST_RECORD);
+ descriptorList.add(IS_STREAMING_ENABLED);
+ descriptorList.add(POLL_ON_INGESTION_STATUS);
+ this.descriptors = Collections.unmodifiableList(descriptorList);
+
+ final Set<Relationship> relationshipSet = new HashSet<>();
+ relationshipSet.add(SUCCESS);
+ relationshipSet.add(FAILURE);
+ this.relationships = Collections.unmodifiableSet(relationshipSet);
+ }
+
+ @Override
+ public Set<Relationship> getRelationships() {
+ return this.relationships;
+ }
+
+ @Override
+ public final List<PropertyDescriptor> getSupportedPropertyDescriptors() {
+ return descriptors;
+ }
+
+ @OnScheduled
+ public void onScheduled(final ProcessContext context) {
+ service = context.getProperty(ADX_SERVICE).asControllerService(KustoIngestService.class);
+ if (checkIfIngestorRoleDoesntExist(context.getProperty(DATABASE_NAME).getValue(), context.getProperty(TABLE_NAME).getValue())) {
+ getLogger().error("User might not have ingestor privileges, table validation will be skipped for all table mappings.");
+ throw new ProcessException("User might not have ingestor privileges, table validation will be skipped for all table mappings. ");
+ }
+ isStreamingEnabled = context.getProperty(IS_STREAMING_ENABLED).evaluateAttributeExpressions().asBoolean();
+ if (isStreamingEnabled) {
+ checkIfStreamingPolicyIsEnabledInADX(context.getProperty(DATABASE_NAME).getValue(), context.getProperty(DATABASE_NAME).getValue());
+ }
+ }
+
+ @Override
+ public void onTrigger(final ProcessContext context, final ProcessSession session) throws ProcessException {
+ FlowFile flowFile = session.get();
+ if (flowFile == null) {
+ context.yield();
+ return;
+ }
+
+ IngestionProperties ingestionProperties = new IngestionProperties(context.getProperty(DATABASE_NAME).getValue(),
+ context.getProperty(TABLE_NAME).getValue());
+
+ IngestionMapping.IngestionMappingKind ingestionMappingKind = null;
+
+ isStreamingEnabled = context.getProperty(IS_STREAMING_ENABLED).evaluateAttributeExpressions().asBoolean();
+ pollOnIngestionStatus = context.getProperty(POLL_ON_INGESTION_STATUS).evaluateAttributeExpressions().asBoolean();
+
+ switch (IngestionProperties.DataFormat.valueOf(context.getProperty(DATA_FORMAT).getValue())) {
+ case AVRO:
+ ingestionProperties.setDataFormat(IngestionProperties.DataFormat.AVRO);
+ ingestionMappingKind = IngestionProperties.DataFormat.AVRO.getIngestionMappingKind();
+ break;
+ case APACHEAVRO:
+ ingestionProperties.setDataFormat(IngestionProperties.DataFormat.APACHEAVRO);
+ ingestionMappingKind = IngestionProperties.DataFormat.APACHEAVRO.getIngestionMappingKind();
+ break;
+ case CSV:
+ ingestionProperties.setDataFormat(IngestionProperties.DataFormat.CSV);
+ ingestionMappingKind = IngestionProperties.DataFormat.CSV.getIngestionMappingKind();
+ break;
+ case JSON:
+ ingestionProperties.setDataFormat(IngestionProperties.DataFormat.JSON);
+ ingestionMappingKind = IngestionProperties.DataFormat.JSON.getIngestionMappingKind();
+ break;
+ case MULTIJSON:
+ ingestionProperties.setDataFormat(IngestionProperties.DataFormat.MULTIJSON);
+ ingestionMappingKind = IngestionProperties.DataFormat.MULTIJSON.getIngestionMappingKind();
+ break;
+ case ORC:
+ ingestionProperties.setDataFormat(IngestionProperties.DataFormat.ORC);
+ ingestionMappingKind = IngestionProperties.DataFormat.ORC.getIngestionMappingKind();
+ break;
+ case PARQUET:
+ ingestionProperties.setDataFormat(IngestionProperties.DataFormat.PARQUET);
+ ingestionMappingKind = IngestionProperties.DataFormat.PARQUET.getIngestionMappingKind();
+ break;
+ case PSV:
+ ingestionProperties.setDataFormat(IngestionProperties.DataFormat.PSV);
+ ingestionMappingKind = IngestionProperties.DataFormat.PSV.getIngestionMappingKind();
+ break;
+ case SCSV:
+ ingestionProperties.setDataFormat(IngestionProperties.DataFormat.SCSV);
+ ingestionMappingKind = IngestionProperties.DataFormat.SCSV.getIngestionMappingKind();
+ break;
+ case SOHSV:
+ ingestionProperties.setDataFormat(IngestionProperties.DataFormat.SOHSV);
+ ingestionMappingKind = IngestionProperties.DataFormat.SOHSV.getIngestionMappingKind();
+ break;
+ case TSV:
+ ingestionProperties.setDataFormat(IngestionProperties.DataFormat.TSV);
+ ingestionMappingKind = IngestionProperties.DataFormat.TSV.getIngestionMappingKind();
+ break;
+ case TSVE:
+ ingestionProperties.setDataFormat(IngestionProperties.DataFormat.TSVE);
+ ingestionMappingKind = IngestionProperties.DataFormat.TSVE.getIngestionMappingKind();
+ break;
+ case TXT:
+ ingestionProperties.setDataFormat(IngestionProperties.DataFormat.TXT);
+ ingestionMappingKind = IngestionProperties.DataFormat.TXT.getIngestionMappingKind();
+ break;
+ }
+
+ if (StringUtils.isNotEmpty(context.getProperty(MAPPING_NAME).getValue()) && ingestionMappingKind != null) {
+ ingestionProperties.setIngestionMapping(context.getProperty(MAPPING_NAME).getValue(), ingestionMappingKind);
+ }
+
+ ingestionProperties.setReportLevel(IngestionProperties.IngestionReportLevel.FAILURES_AND_SUCCESSES);
+ ingestionProperties.setReportMethod(IngestionProperties.IngestionReportMethod.TABLE);
+ ingestionProperties.setFlushImmediately(false);
+ ingestionProperties.setIgnoreFirstRecord(StringUtils.equalsIgnoreCase(context.getProperty(IGNORE_FIRST_RECORD).getValue(), IGNORE_FIRST_RECORD_YES.getValue()));
+
+ boolean isError = false;
+
+ // ingestion into ADX
+ try (final InputStream inputStream = session.read(flowFile)) {
+ StringBuilder ingestLogString = new StringBuilder().append("Ingesting with: ")
+ .append("dataFormat - ").append(ingestionProperties.getDataFormat()).append("|")
+ .append("ingestionMapping - ").append(ingestionProperties.getIngestionMapping().getIngestionMappingReference()).append("|")
+ .append("reportLevel - ").append(ingestionProperties.getReportLevel()).append("|")
+ .append("reportMethod - ").append(ingestionProperties.getReportMethod()).append("|")
+ .append("databaseName - ").append(ingestionProperties.getDatabaseName()).append("|")
+ .append("tableName - ").append(ingestionProperties.getTableName()).append("|")
+ .append("flushImmediately - ").append(ingestionProperties.getFlushImmediately());
+ getLogger().info(ingestLogString.toString());
+
+ KustoIngestionResult result = service.ingestData(new KustoIngestionRequest(isStreamingEnabled, pollOnIngestionStatus, inputStream, ingestionProperties));
+ getLogger().info("Poll on Ingestion Status {} and Operation status: {} ", pollOnIngestionStatus, result.toString());
+
+ if (result == KustoIngestionResult.SUCCEEDED) {
+ getLogger().info("Operation status Succeeded - {}", result.toString());
+ } else if (result == KustoIngestionResult.FAILED) {
+ getLogger().error("Operation status Error - {}", result.toString());
+ isError = true;
+ } else if (result == KustoIngestionResult.PARTIALLY_SUCCEEDED) {
+ getLogger().error("Operation status Partially succeeded - {}", result.toString());
+ isError = true;
+ }
+
+ } catch (IOException | URISyntaxException e) {
+ getLogger().error("Exception occurred while ingesting data into ADX ", e);
+ isError = true;
+ }
+
+ if (isError) {
+ getLogger().error("Process failed - {}");
Review Comment:
```suggestion
getLogger().error("Ingest processing failed {}", flowFile);
```
##########
nifi-nar-bundles/nifi-azure-bundle/nifi-azure-processors/src/main/java/org/apache/nifi/processors/azure/data/explorer/PutAzureDataExplorer.java:
##########
@@ -0,0 +1,399 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.azure.data.explorer;
+
+import com.microsoft.azure.kusto.data.KustoResultSetTable;
+import com.microsoft.azure.kusto.ingest.IngestionMapping;
+import com.microsoft.azure.kusto.ingest.IngestionProperties;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.nifi.annotation.behavior.ReadsAttribute;
+import org.apache.nifi.annotation.behavior.ReadsAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.expression.ExpressionLanguageScope;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.services.azure.data.explorer.KustoIngestService;
+import org.apache.nifi.services.azure.data.explorer.KustoIngestionRequest;
+import org.apache.nifi.services.azure.data.explorer.KustoIngestionResult;
+import org.apache.nifi.services.azure.data.explorer.KustoQueryResponse;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.net.URISyntaxException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+
+@Tags({"Azure", "Kusto", "ADX", "Explorer", "Data"})
+@CapabilityDescription("The PutAzureDataExplorer acts as a ADX sink connector which sends flowFiles using the ADX-Service to the provided Azure Data" +
+ "Explorer Ingest Endpoint. The data can be sent through queued ingestion or streaming ingestion to the Azure Data Explorer cluster.")
+@ReadsAttributes({
+ @ReadsAttribute(attribute = "DB_NAME", description = "Specifies the name of the ADX database where the data needs to be stored."),
+ @ReadsAttribute(attribute = "TABLE_NAME", description = "Specifies the name of the ADX table where the data needs to be stored."),
+ @ReadsAttribute(attribute = "MAPPING_NAME", description = "Specifies the name of the mapping responsible for storing the data in appropriate columns."),
+ @ReadsAttribute(attribute = "FLUSH_IMMEDIATE", description = "In case of queued ingestion, this property determines whether the data should be flushed immediately to the ingest endpoint."),
+ @ReadsAttribute(attribute = "DATA_FORMAT", description = "Specifies the format of data that is send to Azure Data Explorer."),
+ @ReadsAttribute(attribute = "IGNORE_FIRST_RECORD", description = "Specifies whether we want to ignore ingestion of first record. " +
+ "This is primarily applicable for csv files. Default is set to NO"),
+ @ReadsAttribute(attribute = "POLL_ON_INGESTION_STATUS", description = "Specifies whether we want to poll on ingestion result during ingestion into ADX." +
+ "In case of applications that need high throughput it is recommended to keep the default value as false. Default is set to false." +
+ "This property should be set to true during Queued Ingestion for near realtime micro-batches of data that require acknowledgement of ingestion status.")
+})
+public class PutAzureDataExplorer extends AbstractProcessor {
+
+ public static final String FETCH_TABLE_COMMAND = "%s | count";
+ public static final String STREAMING_POLICY_SHOW_COMMAND = ".show %s %s policy streamingingestion";
+ public static final String DATABASE = "database";
+
+ private List<PropertyDescriptor> descriptors;
+ private Set<Relationship> relationships;
+ private transient KustoIngestService service;
+ private boolean isStreamingEnabled;
+ private boolean pollOnIngestionStatus;
+
+ public static final AllowableValue AVRO = new AllowableValue(
+ IngestionProperties.DataFormat.AVRO.getKustoValue(), IngestionProperties.DataFormat.AVRO.getKustoValue(),
+ "An Avro format with support for logical types and for the snappy compression codec");
+
+ public static final AllowableValue APACHEAVRO = new AllowableValue(
+ IngestionProperties.DataFormat.APACHEAVRO.getKustoValue(), IngestionProperties.DataFormat.APACHEAVRO.getKustoValue(),
+ "An Avro format with support for logical types and for the snappy compression codec.");
+
+ public static final AllowableValue CSV = new AllowableValue(
+ IngestionProperties.DataFormat.CSV.name(), IngestionProperties.DataFormat.CSV.getKustoValue(),
+ "A text file with comma-separated values (,). For more information, see RFC 4180: Common Format " +
+ "and MIME Type for Comma-Separated Values (CSV) Files.");
+
+ public static final AllowableValue JSON = new AllowableValue(
+ IngestionProperties.DataFormat.JSON.name(), IngestionProperties.DataFormat.JSON.getKustoValue(),
+ "A text file containing JSON objects separated by \\n or \\r\\n. For more information, " +
+ "see JSON Lines (JSONL).");
+
+ public static final AllowableValue MULTIJSON = new AllowableValue(
+ IngestionProperties.DataFormat.MULTIJSON.name(), IngestionProperties.DataFormat.MULTIJSON.getKustoValue(),
+ "A text file containing a JSON array of property containers (each representing a record) or any " +
+ "number of property containers separated by spaces, \\n or \\r\\n. Each property container may be " +
+ "spread across multiple lines. This format is preferable to JSON unless the data is not property " +
+ "containers.");
+
+ public static final AllowableValue ORC = new AllowableValue(
+ IngestionProperties.DataFormat.ORC.name(), IngestionProperties.DataFormat.ORC.getKustoValue(), "An ORC file.");
+
+ public static final AllowableValue PARQUET = new AllowableValue(
+ IngestionProperties.DataFormat.PARQUET.name(), IngestionProperties.DataFormat.PARQUET.getKustoValue(), "A parquet file.");
+
+ public static final AllowableValue PSV = new AllowableValue(
+ IngestionProperties.DataFormat.PSV.name(), IngestionProperties.DataFormat.PSV.getKustoValue(), "A text file with values separated by vertical bars (|).");
+
+ public static final AllowableValue SCSV = new AllowableValue(
+ IngestionProperties.DataFormat.SCSV.name(), IngestionProperties.DataFormat.SCSV.getKustoValue(), "A text file with values separated by semicolons (;).");
+
+ public static final AllowableValue SOHSV = new AllowableValue(
+ IngestionProperties.DataFormat.SOHSV.name(), IngestionProperties.DataFormat.SOHSV.getKustoValue(),
+ "A text file with SOH-separated values. (SOH is the ASCII code point 1. " +
+ "This format is used by Hive in HDInsight).");
+
+ public static final AllowableValue TSV = new AllowableValue(
+ IngestionProperties.DataFormat.TSV.name(), IngestionProperties.DataFormat.TSV.getKustoValue(), "A text file with tab delimited values (\\t).");
+
+ public static final AllowableValue TSVE = new AllowableValue(
+ IngestionProperties.DataFormat.TSVE.name(), IngestionProperties.DataFormat.TSVE.getKustoValue(),
+ "A text file with tab-delimited values (\\t). A backslash (\\) is used as escape character.");
+
+ public static final AllowableValue TXT = new AllowableValue(
+ IngestionProperties.DataFormat.TXT.name(), IngestionProperties.DataFormat.TXT.getKustoValue(),
+ "A text file with lines separated by \\n. Empty lines are skipped.");
+
+ public static final AllowableValue IGNORE_FIRST_RECORD_YES = new AllowableValue(
+ "YES", "YES",
+ "Ignore first record during ingestion");
+
+ public static final AllowableValue IGNORE_FIRST_RECORD_NO = new AllowableValue(
+ "NO", "NO",
+ "Do not ignore first record during ingestion");
+
+ public static final PropertyDescriptor DATABASE_NAME = new PropertyDescriptor.Builder()
+ .name("Database Name")
+ .displayName("Database Name")
+ .description("Azure Data Explorer Database Name for querying")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+ .build();
+
+ public static final PropertyDescriptor TABLE_NAME = new PropertyDescriptor.Builder()
+ .name("Table Name")
+ .displayName("Table Name")
+ .description("Azure Data Explorer Table Name for ingesting data")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+ .build();
+
+ public static final PropertyDescriptor MAPPING_NAME = new PropertyDescriptor
+ .Builder().name("Mapping name")
+ .displayName("Mapping name")
+ .description("The name of the mapping responsible for storing the data in the appropriate columns.")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+
+ public static final PropertyDescriptor IS_STREAMING_ENABLED = new PropertyDescriptor
+ .Builder().name("Is Streaming enabled")
+ .displayName("Is Streaming enabled")
+ .description("This property determines whether we want to stream data to ADX.")
+ .required(false)
+ .allowableValues("true", "false")
+ .addValidator(StandardValidators.BOOLEAN_VALIDATOR)
+ .defaultValue("false")
+ .build();
+
+ public static final PropertyDescriptor POLL_ON_INGESTION_STATUS = new PropertyDescriptor
+ .Builder().name("Poll on ingestion status")
+ .displayName("Whether to Poll on ingestion status")
+ .description("This property determines whether we want to poll on ingestion status after an ingestion to ADX is completed")
+ .required(false)
+ .allowableValues("true", "false")
+ .addValidator(StandardValidators.BOOLEAN_VALIDATOR)
+ .defaultValue("false")
+ .build();
+ public static final PropertyDescriptor ADX_SERVICE = new PropertyDescriptor
+ .Builder().name("Kusto Ingest Service")
+ .displayName("Kusto Ingest Service")
+ .description("Azure Data Explorer Kusto Ingest Service")
+ .required(true)
+ .identifiesControllerService(KustoIngestService.class)
+ .build();
+
+ public static final Relationship SUCCESS = new Relationship.Builder()
+ .name("success")
+ .description("Relationship for success")
+ .build();
+ public static final Relationship FAILURE = new Relationship.Builder()
+ .name("failure")
+ .description("Relationship for failure")
+ .build();
+
+ static final PropertyDescriptor DATA_FORMAT = new PropertyDescriptor.Builder()
+ .name("Data Format")
+ .displayName("Data Format")
+ .description("The format of the data that is sent to ADX.")
+ .required(true)
+ .allowableValues(AVRO, APACHEAVRO, CSV, JSON, MULTIJSON, ORC, PARQUET, PSV, SCSV, SOHSV, TSV, TSVE, TXT)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+
+ static final PropertyDescriptor IGNORE_FIRST_RECORD = new PropertyDescriptor.Builder()
+ .name("Ingestion Ignore First Record")
+ .displayName("Ingestion Ignore First Record")
+ .description("Defines whether ignore first record while ingestion.")
+ .required(false)
+ .allowableValues(IGNORE_FIRST_RECORD_YES, IGNORE_FIRST_RECORD_NO)
+ .defaultValue(IGNORE_FIRST_RECORD_NO.getValue())
+ .build();
+
+ @Override
+ protected void init(final ProcessorInitializationContext context) {
+ final List<PropertyDescriptor> descriptorList = new ArrayList<>();
+ descriptorList.add(ADX_SERVICE);
+ descriptorList.add(DATABASE_NAME);
+ descriptorList.add(TABLE_NAME);
+ descriptorList.add(MAPPING_NAME);
+ descriptorList.add(DATA_FORMAT);
+ descriptorList.add(IGNORE_FIRST_RECORD);
+ descriptorList.add(IS_STREAMING_ENABLED);
+ descriptorList.add(POLL_ON_INGESTION_STATUS);
+ this.descriptors = Collections.unmodifiableList(descriptorList);
+
+ final Set<Relationship> relationshipSet = new HashSet<>();
+ relationshipSet.add(SUCCESS);
+ relationshipSet.add(FAILURE);
+ this.relationships = Collections.unmodifiableSet(relationshipSet);
+ }
+
+ @Override
+ public Set<Relationship> getRelationships() {
+ return this.relationships;
+ }
+
+ @Override
+ public final List<PropertyDescriptor> getSupportedPropertyDescriptors() {
+ return descriptors;
+ }
+
+ @OnScheduled
+ public void onScheduled(final ProcessContext context) {
+ service = context.getProperty(ADX_SERVICE).asControllerService(KustoIngestService.class);
+ if (checkIfIngestorRoleDoesntExist(context.getProperty(DATABASE_NAME).getValue(), context.getProperty(TABLE_NAME).getValue())) {
+ getLogger().error("User might not have ingestor privileges, table validation will be skipped for all table mappings.");
+ throw new ProcessException("User might not have ingestor privileges, table validation will be skipped for all table mappings. ");
+ }
+ isStreamingEnabled = context.getProperty(IS_STREAMING_ENABLED).evaluateAttributeExpressions().asBoolean();
+ if (isStreamingEnabled) {
+ checkIfStreamingPolicyIsEnabledInADX(context.getProperty(DATABASE_NAME).getValue(), context.getProperty(DATABASE_NAME).getValue());
+ }
+ }
+
+ @Override
+ public void onTrigger(final ProcessContext context, final ProcessSession session) throws ProcessException {
+ FlowFile flowFile = session.get();
+ if (flowFile == null) {
+ context.yield();
+ return;
+ }
+
+ IngestionProperties ingestionProperties = new IngestionProperties(context.getProperty(DATABASE_NAME).getValue(),
+ context.getProperty(TABLE_NAME).getValue());
+
+ IngestionMapping.IngestionMappingKind ingestionMappingKind = null;
+
+ isStreamingEnabled = context.getProperty(IS_STREAMING_ENABLED).evaluateAttributeExpressions().asBoolean();
+ pollOnIngestionStatus = context.getProperty(POLL_ON_INGESTION_STATUS).evaluateAttributeExpressions().asBoolean();
+
+ switch (IngestionProperties.DataFormat.valueOf(context.getProperty(DATA_FORMAT).getValue())) {
+ case AVRO:
+ ingestionProperties.setDataFormat(IngestionProperties.DataFormat.AVRO);
+ ingestionMappingKind = IngestionProperties.DataFormat.AVRO.getIngestionMappingKind();
+ break;
+ case APACHEAVRO:
+ ingestionProperties.setDataFormat(IngestionProperties.DataFormat.APACHEAVRO);
+ ingestionMappingKind = IngestionProperties.DataFormat.APACHEAVRO.getIngestionMappingKind();
+ break;
+ case CSV:
+ ingestionProperties.setDataFormat(IngestionProperties.DataFormat.CSV);
+ ingestionMappingKind = IngestionProperties.DataFormat.CSV.getIngestionMappingKind();
+ break;
+ case JSON:
+ ingestionProperties.setDataFormat(IngestionProperties.DataFormat.JSON);
+ ingestionMappingKind = IngestionProperties.DataFormat.JSON.getIngestionMappingKind();
+ break;
+ case MULTIJSON:
+ ingestionProperties.setDataFormat(IngestionProperties.DataFormat.MULTIJSON);
+ ingestionMappingKind = IngestionProperties.DataFormat.MULTIJSON.getIngestionMappingKind();
+ break;
+ case ORC:
+ ingestionProperties.setDataFormat(IngestionProperties.DataFormat.ORC);
+ ingestionMappingKind = IngestionProperties.DataFormat.ORC.getIngestionMappingKind();
+ break;
+ case PARQUET:
+ ingestionProperties.setDataFormat(IngestionProperties.DataFormat.PARQUET);
+ ingestionMappingKind = IngestionProperties.DataFormat.PARQUET.getIngestionMappingKind();
+ break;
+ case PSV:
+ ingestionProperties.setDataFormat(IngestionProperties.DataFormat.PSV);
+ ingestionMappingKind = IngestionProperties.DataFormat.PSV.getIngestionMappingKind();
+ break;
+ case SCSV:
+ ingestionProperties.setDataFormat(IngestionProperties.DataFormat.SCSV);
+ ingestionMappingKind = IngestionProperties.DataFormat.SCSV.getIngestionMappingKind();
+ break;
+ case SOHSV:
+ ingestionProperties.setDataFormat(IngestionProperties.DataFormat.SOHSV);
+ ingestionMappingKind = IngestionProperties.DataFormat.SOHSV.getIngestionMappingKind();
+ break;
+ case TSV:
+ ingestionProperties.setDataFormat(IngestionProperties.DataFormat.TSV);
+ ingestionMappingKind = IngestionProperties.DataFormat.TSV.getIngestionMappingKind();
+ break;
+ case TSVE:
+ ingestionProperties.setDataFormat(IngestionProperties.DataFormat.TSVE);
+ ingestionMappingKind = IngestionProperties.DataFormat.TSVE.getIngestionMappingKind();
+ break;
+ case TXT:
+ ingestionProperties.setDataFormat(IngestionProperties.DataFormat.TXT);
+ ingestionMappingKind = IngestionProperties.DataFormat.TXT.getIngestionMappingKind();
+ break;
+ }
+
+ if (StringUtils.isNotEmpty(context.getProperty(MAPPING_NAME).getValue()) && ingestionMappingKind != null) {
+ ingestionProperties.setIngestionMapping(context.getProperty(MAPPING_NAME).getValue(), ingestionMappingKind);
+ }
+
+ ingestionProperties.setReportLevel(IngestionProperties.IngestionReportLevel.FAILURES_AND_SUCCESSES);
+ ingestionProperties.setReportMethod(IngestionProperties.IngestionReportMethod.TABLE);
+ ingestionProperties.setFlushImmediately(false);
+ ingestionProperties.setIgnoreFirstRecord(StringUtils.equalsIgnoreCase(context.getProperty(IGNORE_FIRST_RECORD).getValue(), IGNORE_FIRST_RECORD_YES.getValue()));
+
+ boolean isError = false;
+
+ // ingestion into ADX
+ try (final InputStream inputStream = session.read(flowFile)) {
+ StringBuilder ingestLogString = new StringBuilder().append("Ingesting with: ")
+ .append("dataFormat - ").append(ingestionProperties.getDataFormat()).append("|")
+ .append("ingestionMapping - ").append(ingestionProperties.getIngestionMapping().getIngestionMappingReference()).append("|")
+ .append("reportLevel - ").append(ingestionProperties.getReportLevel()).append("|")
+ .append("reportMethod - ").append(ingestionProperties.getReportMethod()).append("|")
+ .append("databaseName - ").append(ingestionProperties.getDatabaseName()).append("|")
+ .append("tableName - ").append(ingestionProperties.getTableName()).append("|")
+ .append("flushImmediately - ").append(ingestionProperties.getFlushImmediately());
+ getLogger().info(ingestLogString.toString());
+
+ KustoIngestionResult result = service.ingestData(new KustoIngestionRequest(isStreamingEnabled, pollOnIngestionStatus, inputStream, ingestionProperties));
+ getLogger().info("Poll on Ingestion Status {} and Operation status: {} ", pollOnIngestionStatus, result.toString());
+
+ if (result == KustoIngestionResult.SUCCEEDED) {
+ getLogger().info("Operation status Succeeded - {}", result.toString());
+ } else if (result == KustoIngestionResult.FAILED) {
+ getLogger().error("Operation status Error - {}", result.toString());
+ isError = true;
+ } else if (result == KustoIngestionResult.PARTIALLY_SUCCEEDED) {
+ getLogger().error("Operation status Partially succeeded - {}", result.toString());
Review Comment:
The `toString()` is not necessary, and this should be a warning, not and error.
```suggestion
getLogger().warn("Ingest Partially Succeeded: {}", result);
```
##########
nifi-nar-bundles/nifi-azure-bundle/nifi-azure-processors/src/main/java/org/apache/nifi/processors/azure/data/explorer/PutAzureDataExplorer.java:
##########
@@ -0,0 +1,399 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.azure.data.explorer;
+
+import com.microsoft.azure.kusto.data.KustoResultSetTable;
+import com.microsoft.azure.kusto.ingest.IngestionMapping;
+import com.microsoft.azure.kusto.ingest.IngestionProperties;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.nifi.annotation.behavior.ReadsAttribute;
+import org.apache.nifi.annotation.behavior.ReadsAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.expression.ExpressionLanguageScope;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.services.azure.data.explorer.KustoIngestService;
+import org.apache.nifi.services.azure.data.explorer.KustoIngestionRequest;
+import org.apache.nifi.services.azure.data.explorer.KustoIngestionResult;
+import org.apache.nifi.services.azure.data.explorer.KustoQueryResponse;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.net.URISyntaxException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+
+@Tags({"Azure", "Kusto", "ADX", "Explorer", "Data"})
+@CapabilityDescription("The PutAzureDataExplorer acts as a ADX sink connector which sends flowFiles using the ADX-Service to the provided Azure Data" +
+ "Explorer Ingest Endpoint. The data can be sent through queued ingestion or streaming ingestion to the Azure Data Explorer cluster.")
+@ReadsAttributes({
+ @ReadsAttribute(attribute = "DB_NAME", description = "Specifies the name of the ADX database where the data needs to be stored."),
+ @ReadsAttribute(attribute = "TABLE_NAME", description = "Specifies the name of the ADX table where the data needs to be stored."),
+ @ReadsAttribute(attribute = "MAPPING_NAME", description = "Specifies the name of the mapping responsible for storing the data in appropriate columns."),
+ @ReadsAttribute(attribute = "FLUSH_IMMEDIATE", description = "In case of queued ingestion, this property determines whether the data should be flushed immediately to the ingest endpoint."),
+ @ReadsAttribute(attribute = "DATA_FORMAT", description = "Specifies the format of data that is send to Azure Data Explorer."),
+ @ReadsAttribute(attribute = "IGNORE_FIRST_RECORD", description = "Specifies whether we want to ignore ingestion of first record. " +
+ "This is primarily applicable for csv files. Default is set to NO"),
+ @ReadsAttribute(attribute = "POLL_ON_INGESTION_STATUS", description = "Specifies whether we want to poll on ingestion result during ingestion into ADX." +
+ "In case of applications that need high throughput it is recommended to keep the default value as false. Default is set to false." +
+ "This property should be set to true during Queued Ingestion for near realtime micro-batches of data that require acknowledgement of ingestion status.")
+})
+public class PutAzureDataExplorer extends AbstractProcessor {
+
+ public static final String FETCH_TABLE_COMMAND = "%s | count";
+ public static final String STREAMING_POLICY_SHOW_COMMAND = ".show %s %s policy streamingingestion";
+ public static final String DATABASE = "database";
+
+ private List<PropertyDescriptor> descriptors;
+ private Set<Relationship> relationships;
+ private transient KustoIngestService service;
+ private boolean isStreamingEnabled;
+ private boolean pollOnIngestionStatus;
+
+ public static final AllowableValue AVRO = new AllowableValue(
+ IngestionProperties.DataFormat.AVRO.getKustoValue(), IngestionProperties.DataFormat.AVRO.getKustoValue(),
+ "An Avro format with support for logical types and for the snappy compression codec");
+
+ public static final AllowableValue APACHEAVRO = new AllowableValue(
+ IngestionProperties.DataFormat.APACHEAVRO.getKustoValue(), IngestionProperties.DataFormat.APACHEAVRO.getKustoValue(),
+ "An Avro format with support for logical types and for the snappy compression codec.");
+
+ public static final AllowableValue CSV = new AllowableValue(
+ IngestionProperties.DataFormat.CSV.name(), IngestionProperties.DataFormat.CSV.getKustoValue(),
+ "A text file with comma-separated values (,). For more information, see RFC 4180: Common Format " +
+ "and MIME Type for Comma-Separated Values (CSV) Files.");
+
+ public static final AllowableValue JSON = new AllowableValue(
+ IngestionProperties.DataFormat.JSON.name(), IngestionProperties.DataFormat.JSON.getKustoValue(),
+ "A text file containing JSON objects separated by \\n or \\r\\n. For more information, " +
+ "see JSON Lines (JSONL).");
+
+ public static final AllowableValue MULTIJSON = new AllowableValue(
+ IngestionProperties.DataFormat.MULTIJSON.name(), IngestionProperties.DataFormat.MULTIJSON.getKustoValue(),
+ "A text file containing a JSON array of property containers (each representing a record) or any " +
+ "number of property containers separated by spaces, \\n or \\r\\n. Each property container may be " +
+ "spread across multiple lines. This format is preferable to JSON unless the data is not property " +
+ "containers.");
+
+ public static final AllowableValue ORC = new AllowableValue(
+ IngestionProperties.DataFormat.ORC.name(), IngestionProperties.DataFormat.ORC.getKustoValue(), "An ORC file.");
+
+ public static final AllowableValue PARQUET = new AllowableValue(
+ IngestionProperties.DataFormat.PARQUET.name(), IngestionProperties.DataFormat.PARQUET.getKustoValue(), "A parquet file.");
+
+ public static final AllowableValue PSV = new AllowableValue(
+ IngestionProperties.DataFormat.PSV.name(), IngestionProperties.DataFormat.PSV.getKustoValue(), "A text file with values separated by vertical bars (|).");
+
+ public static final AllowableValue SCSV = new AllowableValue(
+ IngestionProperties.DataFormat.SCSV.name(), IngestionProperties.DataFormat.SCSV.getKustoValue(), "A text file with values separated by semicolons (;).");
+
+ public static final AllowableValue SOHSV = new AllowableValue(
+ IngestionProperties.DataFormat.SOHSV.name(), IngestionProperties.DataFormat.SOHSV.getKustoValue(),
+ "A text file with SOH-separated values. (SOH is the ASCII code point 1. " +
+ "This format is used by Hive in HDInsight).");
+
+ public static final AllowableValue TSV = new AllowableValue(
+ IngestionProperties.DataFormat.TSV.name(), IngestionProperties.DataFormat.TSV.getKustoValue(), "A text file with tab delimited values (\\t).");
+
+ public static final AllowableValue TSVE = new AllowableValue(
+ IngestionProperties.DataFormat.TSVE.name(), IngestionProperties.DataFormat.TSVE.getKustoValue(),
+ "A text file with tab-delimited values (\\t). A backslash (\\) is used as escape character.");
+
+ public static final AllowableValue TXT = new AllowableValue(
+ IngestionProperties.DataFormat.TXT.name(), IngestionProperties.DataFormat.TXT.getKustoValue(),
+ "A text file with lines separated by \\n. Empty lines are skipped.");
+
+ public static final AllowableValue IGNORE_FIRST_RECORD_YES = new AllowableValue(
+ "YES", "YES",
+ "Ignore first record during ingestion");
+
+ public static final AllowableValue IGNORE_FIRST_RECORD_NO = new AllowableValue(
+ "NO", "NO",
+ "Do not ignore first record during ingestion");
+
+ public static final PropertyDescriptor DATABASE_NAME = new PropertyDescriptor.Builder()
+ .name("Database Name")
+ .displayName("Database Name")
+ .description("Azure Data Explorer Database Name for querying")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+ .build();
+
+ public static final PropertyDescriptor TABLE_NAME = new PropertyDescriptor.Builder()
+ .name("Table Name")
+ .displayName("Table Name")
+ .description("Azure Data Explorer Table Name for ingesting data")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+ .build();
+
+ public static final PropertyDescriptor MAPPING_NAME = new PropertyDescriptor
+ .Builder().name("Mapping name")
+ .displayName("Mapping name")
+ .description("The name of the mapping responsible for storing the data in the appropriate columns.")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+
+ public static final PropertyDescriptor IS_STREAMING_ENABLED = new PropertyDescriptor
+ .Builder().name("Is Streaming enabled")
+ .displayName("Is Streaming enabled")
+ .description("This property determines whether we want to stream data to ADX.")
+ .required(false)
+ .allowableValues("true", "false")
+ .addValidator(StandardValidators.BOOLEAN_VALIDATOR)
+ .defaultValue("false")
+ .build();
+
+ public static final PropertyDescriptor POLL_ON_INGESTION_STATUS = new PropertyDescriptor
+ .Builder().name("Poll on ingestion status")
+ .displayName("Whether to Poll on ingestion status")
+ .description("This property determines whether we want to poll on ingestion status after an ingestion to ADX is completed")
+ .required(false)
+ .allowableValues("true", "false")
+ .addValidator(StandardValidators.BOOLEAN_VALIDATOR)
+ .defaultValue("false")
+ .build();
+ public static final PropertyDescriptor ADX_SERVICE = new PropertyDescriptor
+ .Builder().name("Kusto Ingest Service")
+ .displayName("Kusto Ingest Service")
+ .description("Azure Data Explorer Kusto Ingest Service")
+ .required(true)
+ .identifiesControllerService(KustoIngestService.class)
+ .build();
+
+ public static final Relationship SUCCESS = new Relationship.Builder()
+ .name("success")
+ .description("Relationship for success")
+ .build();
+ public static final Relationship FAILURE = new Relationship.Builder()
+ .name("failure")
+ .description("Relationship for failure")
+ .build();
+
+ static final PropertyDescriptor DATA_FORMAT = new PropertyDescriptor.Builder()
+ .name("Data Format")
+ .displayName("Data Format")
+ .description("The format of the data that is sent to ADX.")
+ .required(true)
+ .allowableValues(AVRO, APACHEAVRO, CSV, JSON, MULTIJSON, ORC, PARQUET, PSV, SCSV, SOHSV, TSV, TSVE, TXT)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+
+ static final PropertyDescriptor IGNORE_FIRST_RECORD = new PropertyDescriptor.Builder()
+ .name("Ingestion Ignore First Record")
+ .displayName("Ingestion Ignore First Record")
+ .description("Defines whether ignore first record while ingestion.")
+ .required(false)
+ .allowableValues(IGNORE_FIRST_RECORD_YES, IGNORE_FIRST_RECORD_NO)
+ .defaultValue(IGNORE_FIRST_RECORD_NO.getValue())
+ .build();
+
+ @Override
+ protected void init(final ProcessorInitializationContext context) {
+ final List<PropertyDescriptor> descriptorList = new ArrayList<>();
+ descriptorList.add(ADX_SERVICE);
+ descriptorList.add(DATABASE_NAME);
+ descriptorList.add(TABLE_NAME);
+ descriptorList.add(MAPPING_NAME);
+ descriptorList.add(DATA_FORMAT);
+ descriptorList.add(IGNORE_FIRST_RECORD);
+ descriptorList.add(IS_STREAMING_ENABLED);
+ descriptorList.add(POLL_ON_INGESTION_STATUS);
+ this.descriptors = Collections.unmodifiableList(descriptorList);
+
+ final Set<Relationship> relationshipSet = new HashSet<>();
+ relationshipSet.add(SUCCESS);
+ relationshipSet.add(FAILURE);
+ this.relationships = Collections.unmodifiableSet(relationshipSet);
+ }
+
+ @Override
+ public Set<Relationship> getRelationships() {
+ return this.relationships;
+ }
+
+ @Override
+ public final List<PropertyDescriptor> getSupportedPropertyDescriptors() {
+ return descriptors;
+ }
+
+ @OnScheduled
+ public void onScheduled(final ProcessContext context) {
+ service = context.getProperty(ADX_SERVICE).asControllerService(KustoIngestService.class);
+ if (checkIfIngestorRoleDoesntExist(context.getProperty(DATABASE_NAME).getValue(), context.getProperty(TABLE_NAME).getValue())) {
+ getLogger().error("User might not have ingestor privileges, table validation will be skipped for all table mappings.");
+ throw new ProcessException("User might not have ingestor privileges, table validation will be skipped for all table mappings. ");
+ }
+ isStreamingEnabled = context.getProperty(IS_STREAMING_ENABLED).evaluateAttributeExpressions().asBoolean();
+ if (isStreamingEnabled) {
+ checkIfStreamingPolicyIsEnabledInADX(context.getProperty(DATABASE_NAME).getValue(), context.getProperty(DATABASE_NAME).getValue());
+ }
+ }
+
+ @Override
+ public void onTrigger(final ProcessContext context, final ProcessSession session) throws ProcessException {
+ FlowFile flowFile = session.get();
+ if (flowFile == null) {
+ context.yield();
+ return;
+ }
+
+ IngestionProperties ingestionProperties = new IngestionProperties(context.getProperty(DATABASE_NAME).getValue(),
+ context.getProperty(TABLE_NAME).getValue());
+
+ IngestionMapping.IngestionMappingKind ingestionMappingKind = null;
+
+ isStreamingEnabled = context.getProperty(IS_STREAMING_ENABLED).evaluateAttributeExpressions().asBoolean();
+ pollOnIngestionStatus = context.getProperty(POLL_ON_INGESTION_STATUS).evaluateAttributeExpressions().asBoolean();
+
+ switch (IngestionProperties.DataFormat.valueOf(context.getProperty(DATA_FORMAT).getValue())) {
+ case AVRO:
+ ingestionProperties.setDataFormat(IngestionProperties.DataFormat.AVRO);
+ ingestionMappingKind = IngestionProperties.DataFormat.AVRO.getIngestionMappingKind();
+ break;
+ case APACHEAVRO:
+ ingestionProperties.setDataFormat(IngestionProperties.DataFormat.APACHEAVRO);
+ ingestionMappingKind = IngestionProperties.DataFormat.APACHEAVRO.getIngestionMappingKind();
+ break;
+ case CSV:
+ ingestionProperties.setDataFormat(IngestionProperties.DataFormat.CSV);
+ ingestionMappingKind = IngestionProperties.DataFormat.CSV.getIngestionMappingKind();
+ break;
+ case JSON:
+ ingestionProperties.setDataFormat(IngestionProperties.DataFormat.JSON);
+ ingestionMappingKind = IngestionProperties.DataFormat.JSON.getIngestionMappingKind();
+ break;
+ case MULTIJSON:
+ ingestionProperties.setDataFormat(IngestionProperties.DataFormat.MULTIJSON);
+ ingestionMappingKind = IngestionProperties.DataFormat.MULTIJSON.getIngestionMappingKind();
+ break;
+ case ORC:
+ ingestionProperties.setDataFormat(IngestionProperties.DataFormat.ORC);
+ ingestionMappingKind = IngestionProperties.DataFormat.ORC.getIngestionMappingKind();
+ break;
+ case PARQUET:
+ ingestionProperties.setDataFormat(IngestionProperties.DataFormat.PARQUET);
+ ingestionMappingKind = IngestionProperties.DataFormat.PARQUET.getIngestionMappingKind();
+ break;
+ case PSV:
+ ingestionProperties.setDataFormat(IngestionProperties.DataFormat.PSV);
+ ingestionMappingKind = IngestionProperties.DataFormat.PSV.getIngestionMappingKind();
+ break;
+ case SCSV:
+ ingestionProperties.setDataFormat(IngestionProperties.DataFormat.SCSV);
+ ingestionMappingKind = IngestionProperties.DataFormat.SCSV.getIngestionMappingKind();
+ break;
+ case SOHSV:
+ ingestionProperties.setDataFormat(IngestionProperties.DataFormat.SOHSV);
+ ingestionMappingKind = IngestionProperties.DataFormat.SOHSV.getIngestionMappingKind();
+ break;
+ case TSV:
+ ingestionProperties.setDataFormat(IngestionProperties.DataFormat.TSV);
+ ingestionMappingKind = IngestionProperties.DataFormat.TSV.getIngestionMappingKind();
+ break;
+ case TSVE:
+ ingestionProperties.setDataFormat(IngestionProperties.DataFormat.TSVE);
+ ingestionMappingKind = IngestionProperties.DataFormat.TSVE.getIngestionMappingKind();
+ break;
+ case TXT:
+ ingestionProperties.setDataFormat(IngestionProperties.DataFormat.TXT);
+ ingestionMappingKind = IngestionProperties.DataFormat.TXT.getIngestionMappingKind();
+ break;
+ }
+
+ if (StringUtils.isNotEmpty(context.getProperty(MAPPING_NAME).getValue()) && ingestionMappingKind != null) {
+ ingestionProperties.setIngestionMapping(context.getProperty(MAPPING_NAME).getValue(), ingestionMappingKind);
+ }
+
+ ingestionProperties.setReportLevel(IngestionProperties.IngestionReportLevel.FAILURES_AND_SUCCESSES);
+ ingestionProperties.setReportMethod(IngestionProperties.IngestionReportMethod.TABLE);
+ ingestionProperties.setFlushImmediately(false);
+ ingestionProperties.setIgnoreFirstRecord(StringUtils.equalsIgnoreCase(context.getProperty(IGNORE_FIRST_RECORD).getValue(), IGNORE_FIRST_RECORD_YES.getValue()));
+
+ boolean isError = false;
+
+ // ingestion into ADX
+ try (final InputStream inputStream = session.read(flowFile)) {
+ StringBuilder ingestLogString = new StringBuilder().append("Ingesting with: ")
+ .append("dataFormat - ").append(ingestionProperties.getDataFormat()).append("|")
+ .append("ingestionMapping - ").append(ingestionProperties.getIngestionMapping().getIngestionMappingReference()).append("|")
+ .append("reportLevel - ").append(ingestionProperties.getReportLevel()).append("|")
+ .append("reportMethod - ").append(ingestionProperties.getReportMethod()).append("|")
+ .append("databaseName - ").append(ingestionProperties.getDatabaseName()).append("|")
+ .append("tableName - ").append(ingestionProperties.getTableName()).append("|")
+ .append("flushImmediately - ").append(ingestionProperties.getFlushImmediately());
+ getLogger().info(ingestLogString.toString());
+
+ KustoIngestionResult result = service.ingestData(new KustoIngestionRequest(isStreamingEnabled, pollOnIngestionStatus, inputStream, ingestionProperties));
+ getLogger().info("Poll on Ingestion Status {} and Operation status: {} ", pollOnIngestionStatus, result.toString());
+
+ if (result == KustoIngestionResult.SUCCEEDED) {
+ getLogger().info("Operation status Succeeded - {}", result.toString());
+ } else if (result == KustoIngestionResult.FAILED) {
+ getLogger().error("Operation status Error - {}", result.toString());
+ isError = true;
+ } else if (result == KustoIngestionResult.PARTIALLY_SUCCEEDED) {
+ getLogger().error("Operation status Partially succeeded - {}", result.toString());
+ isError = true;
+ }
+
+ } catch (IOException | URISyntaxException e) {
+ getLogger().error("Exception occurred while ingesting data into ADX ", e);
Review Comment:
```suggestion
getLogger().error("Azure Data Explorer Ingest processing failed", e);
```
##########
nifi-nar-bundles/nifi-azure-bundle/nifi-azure-processors/src/main/java/org/apache/nifi/processors/azure/data/explorer/PutAzureDataExplorer.java:
##########
@@ -0,0 +1,399 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.azure.data.explorer;
+
+import com.microsoft.azure.kusto.data.KustoResultSetTable;
+import com.microsoft.azure.kusto.ingest.IngestionMapping;
+import com.microsoft.azure.kusto.ingest.IngestionProperties;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.nifi.annotation.behavior.ReadsAttribute;
+import org.apache.nifi.annotation.behavior.ReadsAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.expression.ExpressionLanguageScope;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.services.azure.data.explorer.KustoIngestService;
+import org.apache.nifi.services.azure.data.explorer.KustoIngestionRequest;
+import org.apache.nifi.services.azure.data.explorer.KustoIngestionResult;
+import org.apache.nifi.services.azure.data.explorer.KustoQueryResponse;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.net.URISyntaxException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+
+@Tags({"Azure", "Kusto", "ADX", "Explorer", "Data"})
+@CapabilityDescription("The PutAzureDataExplorer acts as a ADX sink connector which sends flowFiles using the ADX-Service to the provided Azure Data" +
+ "Explorer Ingest Endpoint. The data can be sent through queued ingestion or streaming ingestion to the Azure Data Explorer cluster.")
+@ReadsAttributes({
+ @ReadsAttribute(attribute = "DB_NAME", description = "Specifies the name of the ADX database where the data needs to be stored."),
+ @ReadsAttribute(attribute = "TABLE_NAME", description = "Specifies the name of the ADX table where the data needs to be stored."),
+ @ReadsAttribute(attribute = "MAPPING_NAME", description = "Specifies the name of the mapping responsible for storing the data in appropriate columns."),
+ @ReadsAttribute(attribute = "FLUSH_IMMEDIATE", description = "In case of queued ingestion, this property determines whether the data should be flushed immediately to the ingest endpoint."),
+ @ReadsAttribute(attribute = "DATA_FORMAT", description = "Specifies the format of data that is send to Azure Data Explorer."),
+ @ReadsAttribute(attribute = "IGNORE_FIRST_RECORD", description = "Specifies whether we want to ignore ingestion of first record. " +
+ "This is primarily applicable for csv files. Default is set to NO"),
+ @ReadsAttribute(attribute = "POLL_ON_INGESTION_STATUS", description = "Specifies whether we want to poll on ingestion result during ingestion into ADX." +
+ "In case of applications that need high throughput it is recommended to keep the default value as false. Default is set to false." +
+ "This property should be set to true during Queued Ingestion for near realtime micro-batches of data that require acknowledgement of ingestion status.")
+})
+public class PutAzureDataExplorer extends AbstractProcessor {
+
+ public static final String FETCH_TABLE_COMMAND = "%s | count";
+ public static final String STREAMING_POLICY_SHOW_COMMAND = ".show %s %s policy streamingingestion";
+ public static final String DATABASE = "database";
+
+ private List<PropertyDescriptor> descriptors;
+ private Set<Relationship> relationships;
+ private transient KustoIngestService service;
+ private boolean isStreamingEnabled;
+ private boolean pollOnIngestionStatus;
+
+ public static final AllowableValue AVRO = new AllowableValue(
+ IngestionProperties.DataFormat.AVRO.getKustoValue(), IngestionProperties.DataFormat.AVRO.getKustoValue(),
+ "An Avro format with support for logical types and for the snappy compression codec");
+
+ public static final AllowableValue APACHEAVRO = new AllowableValue(
+ IngestionProperties.DataFormat.APACHEAVRO.getKustoValue(), IngestionProperties.DataFormat.APACHEAVRO.getKustoValue(),
+ "An Avro format with support for logical types and for the snappy compression codec.");
+
+ public static final AllowableValue CSV = new AllowableValue(
+ IngestionProperties.DataFormat.CSV.name(), IngestionProperties.DataFormat.CSV.getKustoValue(),
+ "A text file with comma-separated values (,). For more information, see RFC 4180: Common Format " +
+ "and MIME Type for Comma-Separated Values (CSV) Files.");
+
+ public static final AllowableValue JSON = new AllowableValue(
+ IngestionProperties.DataFormat.JSON.name(), IngestionProperties.DataFormat.JSON.getKustoValue(),
+ "A text file containing JSON objects separated by \\n or \\r\\n. For more information, " +
+ "see JSON Lines (JSONL).");
+
+ public static final AllowableValue MULTIJSON = new AllowableValue(
+ IngestionProperties.DataFormat.MULTIJSON.name(), IngestionProperties.DataFormat.MULTIJSON.getKustoValue(),
+ "A text file containing a JSON array of property containers (each representing a record) or any " +
+ "number of property containers separated by spaces, \\n or \\r\\n. Each property container may be " +
+ "spread across multiple lines. This format is preferable to JSON unless the data is not property " +
+ "containers.");
+
+ public static final AllowableValue ORC = new AllowableValue(
+ IngestionProperties.DataFormat.ORC.name(), IngestionProperties.DataFormat.ORC.getKustoValue(), "An ORC file.");
+
+ public static final AllowableValue PARQUET = new AllowableValue(
+ IngestionProperties.DataFormat.PARQUET.name(), IngestionProperties.DataFormat.PARQUET.getKustoValue(), "A parquet file.");
+
+ public static final AllowableValue PSV = new AllowableValue(
+ IngestionProperties.DataFormat.PSV.name(), IngestionProperties.DataFormat.PSV.getKustoValue(), "A text file with values separated by vertical bars (|).");
+
+ public static final AllowableValue SCSV = new AllowableValue(
+ IngestionProperties.DataFormat.SCSV.name(), IngestionProperties.DataFormat.SCSV.getKustoValue(), "A text file with values separated by semicolons (;).");
+
+ public static final AllowableValue SOHSV = new AllowableValue(
+ IngestionProperties.DataFormat.SOHSV.name(), IngestionProperties.DataFormat.SOHSV.getKustoValue(),
+ "A text file with SOH-separated values. (SOH is the ASCII code point 1. " +
+ "This format is used by Hive in HDInsight).");
+
+ public static final AllowableValue TSV = new AllowableValue(
+ IngestionProperties.DataFormat.TSV.name(), IngestionProperties.DataFormat.TSV.getKustoValue(), "A text file with tab delimited values (\\t).");
+
+ public static final AllowableValue TSVE = new AllowableValue(
+ IngestionProperties.DataFormat.TSVE.name(), IngestionProperties.DataFormat.TSVE.getKustoValue(),
+ "A text file with tab-delimited values (\\t). A backslash (\\) is used as escape character.");
+
+ public static final AllowableValue TXT = new AllowableValue(
+ IngestionProperties.DataFormat.TXT.name(), IngestionProperties.DataFormat.TXT.getKustoValue(),
+ "A text file with lines separated by \\n. Empty lines are skipped.");
+
+ public static final AllowableValue IGNORE_FIRST_RECORD_YES = new AllowableValue(
+ "YES", "YES",
+ "Ignore first record during ingestion");
+
+ public static final AllowableValue IGNORE_FIRST_RECORD_NO = new AllowableValue(
+ "NO", "NO",
+ "Do not ignore first record during ingestion");
+
+ public static final PropertyDescriptor DATABASE_NAME = new PropertyDescriptor.Builder()
+ .name("Database Name")
+ .displayName("Database Name")
+ .description("Azure Data Explorer Database Name for querying")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+ .build();
+
+ public static final PropertyDescriptor TABLE_NAME = new PropertyDescriptor.Builder()
+ .name("Table Name")
+ .displayName("Table Name")
+ .description("Azure Data Explorer Table Name for ingesting data")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+ .build();
+
+ public static final PropertyDescriptor MAPPING_NAME = new PropertyDescriptor
+ .Builder().name("Mapping name")
+ .displayName("Mapping name")
+ .description("The name of the mapping responsible for storing the data in the appropriate columns.")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+
+ public static final PropertyDescriptor IS_STREAMING_ENABLED = new PropertyDescriptor
+ .Builder().name("Is Streaming enabled")
+ .displayName("Is Streaming enabled")
+ .description("This property determines whether we want to stream data to ADX.")
+ .required(false)
+ .allowableValues("true", "false")
+ .addValidator(StandardValidators.BOOLEAN_VALIDATOR)
+ .defaultValue("false")
+ .build();
+
+ public static final PropertyDescriptor POLL_ON_INGESTION_STATUS = new PropertyDescriptor
+ .Builder().name("Poll on ingestion status")
+ .displayName("Whether to Poll on ingestion status")
+ .description("This property determines whether we want to poll on ingestion status after an ingestion to ADX is completed")
+ .required(false)
+ .allowableValues("true", "false")
+ .addValidator(StandardValidators.BOOLEAN_VALIDATOR)
+ .defaultValue("false")
+ .build();
+ public static final PropertyDescriptor ADX_SERVICE = new PropertyDescriptor
+ .Builder().name("Kusto Ingest Service")
+ .displayName("Kusto Ingest Service")
+ .description("Azure Data Explorer Kusto Ingest Service")
+ .required(true)
+ .identifiesControllerService(KustoIngestService.class)
+ .build();
+
+ public static final Relationship SUCCESS = new Relationship.Builder()
+ .name("success")
+ .description("Relationship for success")
+ .build();
+ public static final Relationship FAILURE = new Relationship.Builder()
+ .name("failure")
+ .description("Relationship for failure")
+ .build();
+
+ static final PropertyDescriptor DATA_FORMAT = new PropertyDescriptor.Builder()
+ .name("Data Format")
+ .displayName("Data Format")
+ .description("The format of the data that is sent to ADX.")
+ .required(true)
+ .allowableValues(AVRO, APACHEAVRO, CSV, JSON, MULTIJSON, ORC, PARQUET, PSV, SCSV, SOHSV, TSV, TSVE, TXT)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+
+ static final PropertyDescriptor IGNORE_FIRST_RECORD = new PropertyDescriptor.Builder()
+ .name("Ingestion Ignore First Record")
+ .displayName("Ingestion Ignore First Record")
+ .description("Defines whether ignore first record while ingestion.")
+ .required(false)
+ .allowableValues(IGNORE_FIRST_RECORD_YES, IGNORE_FIRST_RECORD_NO)
+ .defaultValue(IGNORE_FIRST_RECORD_NO.getValue())
+ .build();
+
+ @Override
+ protected void init(final ProcessorInitializationContext context) {
+ final List<PropertyDescriptor> descriptorList = new ArrayList<>();
+ descriptorList.add(ADX_SERVICE);
+ descriptorList.add(DATABASE_NAME);
+ descriptorList.add(TABLE_NAME);
+ descriptorList.add(MAPPING_NAME);
+ descriptorList.add(DATA_FORMAT);
+ descriptorList.add(IGNORE_FIRST_RECORD);
+ descriptorList.add(IS_STREAMING_ENABLED);
+ descriptorList.add(POLL_ON_INGESTION_STATUS);
+ this.descriptors = Collections.unmodifiableList(descriptorList);
+
+ final Set<Relationship> relationshipSet = new HashSet<>();
+ relationshipSet.add(SUCCESS);
+ relationshipSet.add(FAILURE);
+ this.relationships = Collections.unmodifiableSet(relationshipSet);
+ }
+
+ @Override
+ public Set<Relationship> getRelationships() {
+ return this.relationships;
+ }
+
+ @Override
+ public final List<PropertyDescriptor> getSupportedPropertyDescriptors() {
+ return descriptors;
+ }
+
+ @OnScheduled
+ public void onScheduled(final ProcessContext context) {
+ service = context.getProperty(ADX_SERVICE).asControllerService(KustoIngestService.class);
+ if (checkIfIngestorRoleDoesntExist(context.getProperty(DATABASE_NAME).getValue(), context.getProperty(TABLE_NAME).getValue())) {
+ getLogger().error("User might not have ingestor privileges, table validation will be skipped for all table mappings.");
+ throw new ProcessException("User might not have ingestor privileges, table validation will be skipped for all table mappings. ");
+ }
+ isStreamingEnabled = context.getProperty(IS_STREAMING_ENABLED).evaluateAttributeExpressions().asBoolean();
+ if (isStreamingEnabled) {
+ checkIfStreamingPolicyIsEnabledInADX(context.getProperty(DATABASE_NAME).getValue(), context.getProperty(DATABASE_NAME).getValue());
+ }
+ }
+
+ @Override
+ public void onTrigger(final ProcessContext context, final ProcessSession session) throws ProcessException {
+ FlowFile flowFile = session.get();
+ if (flowFile == null) {
+ context.yield();
+ return;
+ }
+
+ IngestionProperties ingestionProperties = new IngestionProperties(context.getProperty(DATABASE_NAME).getValue(),
+ context.getProperty(TABLE_NAME).getValue());
+
+ IngestionMapping.IngestionMappingKind ingestionMappingKind = null;
+
+ isStreamingEnabled = context.getProperty(IS_STREAMING_ENABLED).evaluateAttributeExpressions().asBoolean();
+ pollOnIngestionStatus = context.getProperty(POLL_ON_INGESTION_STATUS).evaluateAttributeExpressions().asBoolean();
+
+ switch (IngestionProperties.DataFormat.valueOf(context.getProperty(DATA_FORMAT).getValue())) {
+ case AVRO:
+ ingestionProperties.setDataFormat(IngestionProperties.DataFormat.AVRO);
+ ingestionMappingKind = IngestionProperties.DataFormat.AVRO.getIngestionMappingKind();
+ break;
+ case APACHEAVRO:
+ ingestionProperties.setDataFormat(IngestionProperties.DataFormat.APACHEAVRO);
+ ingestionMappingKind = IngestionProperties.DataFormat.APACHEAVRO.getIngestionMappingKind();
+ break;
+ case CSV:
+ ingestionProperties.setDataFormat(IngestionProperties.DataFormat.CSV);
+ ingestionMappingKind = IngestionProperties.DataFormat.CSV.getIngestionMappingKind();
+ break;
+ case JSON:
+ ingestionProperties.setDataFormat(IngestionProperties.DataFormat.JSON);
+ ingestionMappingKind = IngestionProperties.DataFormat.JSON.getIngestionMappingKind();
+ break;
+ case MULTIJSON:
+ ingestionProperties.setDataFormat(IngestionProperties.DataFormat.MULTIJSON);
+ ingestionMappingKind = IngestionProperties.DataFormat.MULTIJSON.getIngestionMappingKind();
+ break;
+ case ORC:
+ ingestionProperties.setDataFormat(IngestionProperties.DataFormat.ORC);
+ ingestionMappingKind = IngestionProperties.DataFormat.ORC.getIngestionMappingKind();
+ break;
+ case PARQUET:
+ ingestionProperties.setDataFormat(IngestionProperties.DataFormat.PARQUET);
+ ingestionMappingKind = IngestionProperties.DataFormat.PARQUET.getIngestionMappingKind();
+ break;
+ case PSV:
+ ingestionProperties.setDataFormat(IngestionProperties.DataFormat.PSV);
+ ingestionMappingKind = IngestionProperties.DataFormat.PSV.getIngestionMappingKind();
+ break;
+ case SCSV:
+ ingestionProperties.setDataFormat(IngestionProperties.DataFormat.SCSV);
+ ingestionMappingKind = IngestionProperties.DataFormat.SCSV.getIngestionMappingKind();
+ break;
+ case SOHSV:
+ ingestionProperties.setDataFormat(IngestionProperties.DataFormat.SOHSV);
+ ingestionMappingKind = IngestionProperties.DataFormat.SOHSV.getIngestionMappingKind();
+ break;
+ case TSV:
+ ingestionProperties.setDataFormat(IngestionProperties.DataFormat.TSV);
+ ingestionMappingKind = IngestionProperties.DataFormat.TSV.getIngestionMappingKind();
+ break;
+ case TSVE:
+ ingestionProperties.setDataFormat(IngestionProperties.DataFormat.TSVE);
+ ingestionMappingKind = IngestionProperties.DataFormat.TSVE.getIngestionMappingKind();
+ break;
+ case TXT:
+ ingestionProperties.setDataFormat(IngestionProperties.DataFormat.TXT);
+ ingestionMappingKind = IngestionProperties.DataFormat.TXT.getIngestionMappingKind();
+ break;
+ }
+
+ if (StringUtils.isNotEmpty(context.getProperty(MAPPING_NAME).getValue()) && ingestionMappingKind != null) {
+ ingestionProperties.setIngestionMapping(context.getProperty(MAPPING_NAME).getValue(), ingestionMappingKind);
+ }
+
+ ingestionProperties.setReportLevel(IngestionProperties.IngestionReportLevel.FAILURES_AND_SUCCESSES);
+ ingestionProperties.setReportMethod(IngestionProperties.IngestionReportMethod.TABLE);
+ ingestionProperties.setFlushImmediately(false);
+ ingestionProperties.setIgnoreFirstRecord(StringUtils.equalsIgnoreCase(context.getProperty(IGNORE_FIRST_RECORD).getValue(), IGNORE_FIRST_RECORD_YES.getValue()));
+
+ boolean isError = false;
+
+ // ingestion into ADX
+ try (final InputStream inputStream = session.read(flowFile)) {
+ StringBuilder ingestLogString = new StringBuilder().append("Ingesting with: ")
+ .append("dataFormat - ").append(ingestionProperties.getDataFormat()).append("|")
+ .append("ingestionMapping - ").append(ingestionProperties.getIngestionMapping().getIngestionMappingReference()).append("|")
+ .append("reportLevel - ").append(ingestionProperties.getReportLevel()).append("|")
+ .append("reportMethod - ").append(ingestionProperties.getReportMethod()).append("|")
+ .append("databaseName - ").append(ingestionProperties.getDatabaseName()).append("|")
+ .append("tableName - ").append(ingestionProperties.getTableName()).append("|")
+ .append("flushImmediately - ").append(ingestionProperties.getFlushImmediately());
+ getLogger().info(ingestLogString.toString());
+
+ KustoIngestionResult result = service.ingestData(new KustoIngestionRequest(isStreamingEnabled, pollOnIngestionStatus, inputStream, ingestionProperties));
+ getLogger().info("Poll on Ingestion Status {} and Operation status: {} ", pollOnIngestionStatus, result.toString());
+
+ if (result == KustoIngestionResult.SUCCEEDED) {
+ getLogger().info("Operation status Succeeded - {}", result.toString());
+ } else if (result == KustoIngestionResult.FAILED) {
+ getLogger().error("Operation status Error - {}", result.toString());
Review Comment:
```suggestion
getLogger().error("Ingest Failed: {}", result);
```
##########
nifi-nar-bundles/nifi-azure-bundle/nifi-azure-processors/src/main/java/org/apache/nifi/processors/azure/data/explorer/PutAzureDataExplorer.java:
##########
@@ -0,0 +1,399 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.azure.data.explorer;
+
+import com.microsoft.azure.kusto.data.KustoResultSetTable;
+import com.microsoft.azure.kusto.ingest.IngestionMapping;
+import com.microsoft.azure.kusto.ingest.IngestionProperties;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.nifi.annotation.behavior.ReadsAttribute;
+import org.apache.nifi.annotation.behavior.ReadsAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.expression.ExpressionLanguageScope;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.services.azure.data.explorer.KustoIngestService;
+import org.apache.nifi.services.azure.data.explorer.KustoIngestionRequest;
+import org.apache.nifi.services.azure.data.explorer.KustoIngestionResult;
+import org.apache.nifi.services.azure.data.explorer.KustoQueryResponse;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.net.URISyntaxException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+
+@Tags({"Azure", "Kusto", "ADX", "Explorer", "Data"})
+@CapabilityDescription("The PutAzureDataExplorer acts as a ADX sink connector which sends flowFiles using the ADX-Service to the provided Azure Data" +
+ "Explorer Ingest Endpoint. The data can be sent through queued ingestion or streaming ingestion to the Azure Data Explorer cluster.")
+@ReadsAttributes({
+ @ReadsAttribute(attribute = "DB_NAME", description = "Specifies the name of the ADX database where the data needs to be stored."),
+ @ReadsAttribute(attribute = "TABLE_NAME", description = "Specifies the name of the ADX table where the data needs to be stored."),
+ @ReadsAttribute(attribute = "MAPPING_NAME", description = "Specifies the name of the mapping responsible for storing the data in appropriate columns."),
+ @ReadsAttribute(attribute = "FLUSH_IMMEDIATE", description = "In case of queued ingestion, this property determines whether the data should be flushed immediately to the ingest endpoint."),
+ @ReadsAttribute(attribute = "DATA_FORMAT", description = "Specifies the format of data that is send to Azure Data Explorer."),
+ @ReadsAttribute(attribute = "IGNORE_FIRST_RECORD", description = "Specifies whether we want to ignore ingestion of first record. " +
+ "This is primarily applicable for csv files. Default is set to NO"),
+ @ReadsAttribute(attribute = "POLL_ON_INGESTION_STATUS", description = "Specifies whether we want to poll on ingestion result during ingestion into ADX." +
+ "In case of applications that need high throughput it is recommended to keep the default value as false. Default is set to false." +
+ "This property should be set to true during Queued Ingestion for near realtime micro-batches of data that require acknowledgement of ingestion status.")
+})
+public class PutAzureDataExplorer extends AbstractProcessor {
+
+ public static final String FETCH_TABLE_COMMAND = "%s | count";
+ public static final String STREAMING_POLICY_SHOW_COMMAND = ".show %s %s policy streamingingestion";
+ public static final String DATABASE = "database";
+
+ private List<PropertyDescriptor> descriptors;
+ private Set<Relationship> relationships;
+ private transient KustoIngestService service;
+ private boolean isStreamingEnabled;
+ private boolean pollOnIngestionStatus;
+
+ public static final AllowableValue AVRO = new AllowableValue(
+ IngestionProperties.DataFormat.AVRO.getKustoValue(), IngestionProperties.DataFormat.AVRO.getKustoValue(),
+ "An Avro format with support for logical types and for the snappy compression codec");
+
+ public static final AllowableValue APACHEAVRO = new AllowableValue(
+ IngestionProperties.DataFormat.APACHEAVRO.getKustoValue(), IngestionProperties.DataFormat.APACHEAVRO.getKustoValue(),
+ "An Avro format with support for logical types and for the snappy compression codec.");
+
+ public static final AllowableValue CSV = new AllowableValue(
+ IngestionProperties.DataFormat.CSV.name(), IngestionProperties.DataFormat.CSV.getKustoValue(),
+ "A text file with comma-separated values (,). For more information, see RFC 4180: Common Format " +
+ "and MIME Type for Comma-Separated Values (CSV) Files.");
+
+ public static final AllowableValue JSON = new AllowableValue(
+ IngestionProperties.DataFormat.JSON.name(), IngestionProperties.DataFormat.JSON.getKustoValue(),
+ "A text file containing JSON objects separated by \\n or \\r\\n. For more information, " +
+ "see JSON Lines (JSONL).");
+
+ public static final AllowableValue MULTIJSON = new AllowableValue(
+ IngestionProperties.DataFormat.MULTIJSON.name(), IngestionProperties.DataFormat.MULTIJSON.getKustoValue(),
+ "A text file containing a JSON array of property containers (each representing a record) or any " +
+ "number of property containers separated by spaces, \\n or \\r\\n. Each property container may be " +
+ "spread across multiple lines. This format is preferable to JSON unless the data is not property " +
+ "containers.");
+
+ public static final AllowableValue ORC = new AllowableValue(
+ IngestionProperties.DataFormat.ORC.name(), IngestionProperties.DataFormat.ORC.getKustoValue(), "An ORC file.");
+
+ public static final AllowableValue PARQUET = new AllowableValue(
+ IngestionProperties.DataFormat.PARQUET.name(), IngestionProperties.DataFormat.PARQUET.getKustoValue(), "A parquet file.");
+
+ public static final AllowableValue PSV = new AllowableValue(
+ IngestionProperties.DataFormat.PSV.name(), IngestionProperties.DataFormat.PSV.getKustoValue(), "A text file with values separated by vertical bars (|).");
+
+ public static final AllowableValue SCSV = new AllowableValue(
+ IngestionProperties.DataFormat.SCSV.name(), IngestionProperties.DataFormat.SCSV.getKustoValue(), "A text file with values separated by semicolons (;).");
+
+ public static final AllowableValue SOHSV = new AllowableValue(
+ IngestionProperties.DataFormat.SOHSV.name(), IngestionProperties.DataFormat.SOHSV.getKustoValue(),
+ "A text file with SOH-separated values. (SOH is the ASCII code point 1. " +
+ "This format is used by Hive in HDInsight).");
+
+ public static final AllowableValue TSV = new AllowableValue(
+ IngestionProperties.DataFormat.TSV.name(), IngestionProperties.DataFormat.TSV.getKustoValue(), "A text file with tab delimited values (\\t).");
+
+ public static final AllowableValue TSVE = new AllowableValue(
+ IngestionProperties.DataFormat.TSVE.name(), IngestionProperties.DataFormat.TSVE.getKustoValue(),
+ "A text file with tab-delimited values (\\t). A backslash (\\) is used as escape character.");
+
+ public static final AllowableValue TXT = new AllowableValue(
+ IngestionProperties.DataFormat.TXT.name(), IngestionProperties.DataFormat.TXT.getKustoValue(),
+ "A text file with lines separated by \\n. Empty lines are skipped.");
+
+ public static final AllowableValue IGNORE_FIRST_RECORD_YES = new AllowableValue(
+ "YES", "YES",
+ "Ignore first record during ingestion");
+
+ public static final AllowableValue IGNORE_FIRST_RECORD_NO = new AllowableValue(
+ "NO", "NO",
+ "Do not ignore first record during ingestion");
+
+ public static final PropertyDescriptor DATABASE_NAME = new PropertyDescriptor.Builder()
+ .name("Database Name")
+ .displayName("Database Name")
+ .description("Azure Data Explorer Database Name for querying")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+ .build();
+
+ public static final PropertyDescriptor TABLE_NAME = new PropertyDescriptor.Builder()
+ .name("Table Name")
+ .displayName("Table Name")
+ .description("Azure Data Explorer Table Name for ingesting data")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+ .build();
+
+ public static final PropertyDescriptor MAPPING_NAME = new PropertyDescriptor
+ .Builder().name("Mapping name")
+ .displayName("Mapping name")
+ .description("The name of the mapping responsible for storing the data in the appropriate columns.")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+
+ public static final PropertyDescriptor IS_STREAMING_ENABLED = new PropertyDescriptor
+ .Builder().name("Is Streaming enabled")
+ .displayName("Is Streaming enabled")
+ .description("This property determines whether we want to stream data to ADX.")
+ .required(false)
+ .allowableValues("true", "false")
+ .addValidator(StandardValidators.BOOLEAN_VALIDATOR)
+ .defaultValue("false")
+ .build();
+
+ public static final PropertyDescriptor POLL_ON_INGESTION_STATUS = new PropertyDescriptor
+ .Builder().name("Poll on ingestion status")
+ .displayName("Whether to Poll on ingestion status")
+ .description("This property determines whether we want to poll on ingestion status after an ingestion to ADX is completed")
+ .required(false)
+ .allowableValues("true", "false")
+ .addValidator(StandardValidators.BOOLEAN_VALIDATOR)
+ .defaultValue("false")
+ .build();
+ public static final PropertyDescriptor ADX_SERVICE = new PropertyDescriptor
+ .Builder().name("Kusto Ingest Service")
+ .displayName("Kusto Ingest Service")
+ .description("Azure Data Explorer Kusto Ingest Service")
+ .required(true)
+ .identifiesControllerService(KustoIngestService.class)
+ .build();
+
+ public static final Relationship SUCCESS = new Relationship.Builder()
+ .name("success")
+ .description("Relationship for success")
+ .build();
+ public static final Relationship FAILURE = new Relationship.Builder()
+ .name("failure")
+ .description("Relationship for failure")
+ .build();
+
+ static final PropertyDescriptor DATA_FORMAT = new PropertyDescriptor.Builder()
+ .name("Data Format")
+ .displayName("Data Format")
+ .description("The format of the data that is sent to ADX.")
+ .required(true)
+ .allowableValues(AVRO, APACHEAVRO, CSV, JSON, MULTIJSON, ORC, PARQUET, PSV, SCSV, SOHSV, TSV, TSVE, TXT)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+
+ static final PropertyDescriptor IGNORE_FIRST_RECORD = new PropertyDescriptor.Builder()
+ .name("Ingestion Ignore First Record")
+ .displayName("Ingestion Ignore First Record")
+ .description("Defines whether ignore first record while ingestion.")
+ .required(false)
+ .allowableValues(IGNORE_FIRST_RECORD_YES, IGNORE_FIRST_RECORD_NO)
+ .defaultValue(IGNORE_FIRST_RECORD_NO.getValue())
+ .build();
+
+ @Override
+ protected void init(final ProcessorInitializationContext context) {
+ final List<PropertyDescriptor> descriptorList = new ArrayList<>();
+ descriptorList.add(ADX_SERVICE);
+ descriptorList.add(DATABASE_NAME);
+ descriptorList.add(TABLE_NAME);
+ descriptorList.add(MAPPING_NAME);
+ descriptorList.add(DATA_FORMAT);
+ descriptorList.add(IGNORE_FIRST_RECORD);
+ descriptorList.add(IS_STREAMING_ENABLED);
+ descriptorList.add(POLL_ON_INGESTION_STATUS);
+ this.descriptors = Collections.unmodifiableList(descriptorList);
+
+ final Set<Relationship> relationshipSet = new HashSet<>();
+ relationshipSet.add(SUCCESS);
+ relationshipSet.add(FAILURE);
+ this.relationships = Collections.unmodifiableSet(relationshipSet);
+ }
+
+ @Override
+ public Set<Relationship> getRelationships() {
+ return this.relationships;
+ }
+
+ @Override
+ public final List<PropertyDescriptor> getSupportedPropertyDescriptors() {
+ return descriptors;
+ }
+
+ @OnScheduled
+ public void onScheduled(final ProcessContext context) {
+ service = context.getProperty(ADX_SERVICE).asControllerService(KustoIngestService.class);
+ if (checkIfIngestorRoleDoesntExist(context.getProperty(DATABASE_NAME).getValue(), context.getProperty(TABLE_NAME).getValue())) {
+ getLogger().error("User might not have ingestor privileges, table validation will be skipped for all table mappings.");
+ throw new ProcessException("User might not have ingestor privileges, table validation will be skipped for all table mappings. ");
+ }
+ isStreamingEnabled = context.getProperty(IS_STREAMING_ENABLED).evaluateAttributeExpressions().asBoolean();
+ if (isStreamingEnabled) {
+ checkIfStreamingPolicyIsEnabledInADX(context.getProperty(DATABASE_NAME).getValue(), context.getProperty(DATABASE_NAME).getValue());
+ }
+ }
+
+ @Override
+ public void onTrigger(final ProcessContext context, final ProcessSession session) throws ProcessException {
+ FlowFile flowFile = session.get();
+ if (flowFile == null) {
+ context.yield();
+ return;
+ }
+
+ IngestionProperties ingestionProperties = new IngestionProperties(context.getProperty(DATABASE_NAME).getValue(),
+ context.getProperty(TABLE_NAME).getValue());
+
+ IngestionMapping.IngestionMappingKind ingestionMappingKind = null;
+
+ isStreamingEnabled = context.getProperty(IS_STREAMING_ENABLED).evaluateAttributeExpressions().asBoolean();
+ pollOnIngestionStatus = context.getProperty(POLL_ON_INGESTION_STATUS).evaluateAttributeExpressions().asBoolean();
+
+ switch (IngestionProperties.DataFormat.valueOf(context.getProperty(DATA_FORMAT).getValue())) {
+ case AVRO:
+ ingestionProperties.setDataFormat(IngestionProperties.DataFormat.AVRO);
+ ingestionMappingKind = IngestionProperties.DataFormat.AVRO.getIngestionMappingKind();
+ break;
+ case APACHEAVRO:
+ ingestionProperties.setDataFormat(IngestionProperties.DataFormat.APACHEAVRO);
+ ingestionMappingKind = IngestionProperties.DataFormat.APACHEAVRO.getIngestionMappingKind();
+ break;
+ case CSV:
+ ingestionProperties.setDataFormat(IngestionProperties.DataFormat.CSV);
+ ingestionMappingKind = IngestionProperties.DataFormat.CSV.getIngestionMappingKind();
+ break;
+ case JSON:
+ ingestionProperties.setDataFormat(IngestionProperties.DataFormat.JSON);
+ ingestionMappingKind = IngestionProperties.DataFormat.JSON.getIngestionMappingKind();
+ break;
+ case MULTIJSON:
+ ingestionProperties.setDataFormat(IngestionProperties.DataFormat.MULTIJSON);
+ ingestionMappingKind = IngestionProperties.DataFormat.MULTIJSON.getIngestionMappingKind();
+ break;
+ case ORC:
+ ingestionProperties.setDataFormat(IngestionProperties.DataFormat.ORC);
+ ingestionMappingKind = IngestionProperties.DataFormat.ORC.getIngestionMappingKind();
+ break;
+ case PARQUET:
+ ingestionProperties.setDataFormat(IngestionProperties.DataFormat.PARQUET);
+ ingestionMappingKind = IngestionProperties.DataFormat.PARQUET.getIngestionMappingKind();
+ break;
+ case PSV:
+ ingestionProperties.setDataFormat(IngestionProperties.DataFormat.PSV);
+ ingestionMappingKind = IngestionProperties.DataFormat.PSV.getIngestionMappingKind();
+ break;
+ case SCSV:
+ ingestionProperties.setDataFormat(IngestionProperties.DataFormat.SCSV);
+ ingestionMappingKind = IngestionProperties.DataFormat.SCSV.getIngestionMappingKind();
+ break;
+ case SOHSV:
+ ingestionProperties.setDataFormat(IngestionProperties.DataFormat.SOHSV);
+ ingestionMappingKind = IngestionProperties.DataFormat.SOHSV.getIngestionMappingKind();
+ break;
+ case TSV:
+ ingestionProperties.setDataFormat(IngestionProperties.DataFormat.TSV);
+ ingestionMappingKind = IngestionProperties.DataFormat.TSV.getIngestionMappingKind();
+ break;
+ case TSVE:
+ ingestionProperties.setDataFormat(IngestionProperties.DataFormat.TSVE);
+ ingestionMappingKind = IngestionProperties.DataFormat.TSVE.getIngestionMappingKind();
+ break;
+ case TXT:
+ ingestionProperties.setDataFormat(IngestionProperties.DataFormat.TXT);
+ ingestionMappingKind = IngestionProperties.DataFormat.TXT.getIngestionMappingKind();
+ break;
+ }
+
+ if (StringUtils.isNotEmpty(context.getProperty(MAPPING_NAME).getValue()) && ingestionMappingKind != null) {
+ ingestionProperties.setIngestionMapping(context.getProperty(MAPPING_NAME).getValue(), ingestionMappingKind);
+ }
+
+ ingestionProperties.setReportLevel(IngestionProperties.IngestionReportLevel.FAILURES_AND_SUCCESSES);
+ ingestionProperties.setReportMethod(IngestionProperties.IngestionReportMethod.TABLE);
+ ingestionProperties.setFlushImmediately(false);
+ ingestionProperties.setIgnoreFirstRecord(StringUtils.equalsIgnoreCase(context.getProperty(IGNORE_FIRST_RECORD).getValue(), IGNORE_FIRST_RECORD_YES.getValue()));
+
+ boolean isError = false;
+
+ // ingestion into ADX
+ try (final InputStream inputStream = session.read(flowFile)) {
+ StringBuilder ingestLogString = new StringBuilder().append("Ingesting with: ")
+ .append("dataFormat - ").append(ingestionProperties.getDataFormat()).append("|")
+ .append("ingestionMapping - ").append(ingestionProperties.getIngestionMapping().getIngestionMappingReference()).append("|")
+ .append("reportLevel - ").append(ingestionProperties.getReportLevel()).append("|")
+ .append("reportMethod - ").append(ingestionProperties.getReportMethod()).append("|")
+ .append("databaseName - ").append(ingestionProperties.getDatabaseName()).append("|")
+ .append("tableName - ").append(ingestionProperties.getTableName()).append("|")
+ .append("flushImmediately - ").append(ingestionProperties.getFlushImmediately());
+ getLogger().info(ingestLogString.toString());
Review Comment:
This seems better as a `debug`, and placeholder elements should be used instead of `StringBuilder`.
##########
nifi-nar-bundles/nifi-azure-bundle/nifi-azure-processors/src/main/java/org/apache/nifi/processors/azure/data/explorer/PutAzureDataExplorer.java:
##########
@@ -0,0 +1,399 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.azure.data.explorer;
+
+import com.microsoft.azure.kusto.data.KustoResultSetTable;
+import com.microsoft.azure.kusto.ingest.IngestionMapping;
+import com.microsoft.azure.kusto.ingest.IngestionProperties;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.nifi.annotation.behavior.ReadsAttribute;
+import org.apache.nifi.annotation.behavior.ReadsAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.expression.ExpressionLanguageScope;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.services.azure.data.explorer.KustoIngestService;
+import org.apache.nifi.services.azure.data.explorer.KustoIngestionRequest;
+import org.apache.nifi.services.azure.data.explorer.KustoIngestionResult;
+import org.apache.nifi.services.azure.data.explorer.KustoQueryResponse;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.net.URISyntaxException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+
+@Tags({"Azure", "Kusto", "ADX", "Explorer", "Data"})
+@CapabilityDescription("The PutAzureDataExplorer acts as a ADX sink connector which sends flowFiles using the ADX-Service to the provided Azure Data" +
+ "Explorer Ingest Endpoint. The data can be sent through queued ingestion or streaming ingestion to the Azure Data Explorer cluster.")
+@ReadsAttributes({
+ @ReadsAttribute(attribute = "DB_NAME", description = "Specifies the name of the ADX database where the data needs to be stored."),
+ @ReadsAttribute(attribute = "TABLE_NAME", description = "Specifies the name of the ADX table where the data needs to be stored."),
+ @ReadsAttribute(attribute = "MAPPING_NAME", description = "Specifies the name of the mapping responsible for storing the data in appropriate columns."),
+ @ReadsAttribute(attribute = "FLUSH_IMMEDIATE", description = "In case of queued ingestion, this property determines whether the data should be flushed immediately to the ingest endpoint."),
+ @ReadsAttribute(attribute = "DATA_FORMAT", description = "Specifies the format of data that is send to Azure Data Explorer."),
+ @ReadsAttribute(attribute = "IGNORE_FIRST_RECORD", description = "Specifies whether we want to ignore ingestion of first record. " +
+ "This is primarily applicable for csv files. Default is set to NO"),
+ @ReadsAttribute(attribute = "POLL_ON_INGESTION_STATUS", description = "Specifies whether we want to poll on ingestion result during ingestion into ADX." +
+ "In case of applications that need high throughput it is recommended to keep the default value as false. Default is set to false." +
+ "This property should be set to true during Queued Ingestion for near realtime micro-batches of data that require acknowledgement of ingestion status.")
+})
+public class PutAzureDataExplorer extends AbstractProcessor {
+
+ public static final String FETCH_TABLE_COMMAND = "%s | count";
+ public static final String STREAMING_POLICY_SHOW_COMMAND = ".show %s %s policy streamingingestion";
+ public static final String DATABASE = "database";
+
+ private List<PropertyDescriptor> descriptors;
+ private Set<Relationship> relationships;
+ private transient KustoIngestService service;
+ private boolean isStreamingEnabled;
+ private boolean pollOnIngestionStatus;
+
+ public static final AllowableValue AVRO = new AllowableValue(
+ IngestionProperties.DataFormat.AVRO.getKustoValue(), IngestionProperties.DataFormat.AVRO.getKustoValue(),
+ "An Avro format with support for logical types and for the snappy compression codec");
+
+ public static final AllowableValue APACHEAVRO = new AllowableValue(
+ IngestionProperties.DataFormat.APACHEAVRO.getKustoValue(), IngestionProperties.DataFormat.APACHEAVRO.getKustoValue(),
+ "An Avro format with support for logical types and for the snappy compression codec.");
+
+ public static final AllowableValue CSV = new AllowableValue(
+ IngestionProperties.DataFormat.CSV.name(), IngestionProperties.DataFormat.CSV.getKustoValue(),
+ "A text file with comma-separated values (,). For more information, see RFC 4180: Common Format " +
+ "and MIME Type for Comma-Separated Values (CSV) Files.");
+
+ public static final AllowableValue JSON = new AllowableValue(
+ IngestionProperties.DataFormat.JSON.name(), IngestionProperties.DataFormat.JSON.getKustoValue(),
+ "A text file containing JSON objects separated by \\n or \\r\\n. For more information, " +
+ "see JSON Lines (JSONL).");
+
+ public static final AllowableValue MULTIJSON = new AllowableValue(
+ IngestionProperties.DataFormat.MULTIJSON.name(), IngestionProperties.DataFormat.MULTIJSON.getKustoValue(),
+ "A text file containing a JSON array of property containers (each representing a record) or any " +
+ "number of property containers separated by spaces, \\n or \\r\\n. Each property container may be " +
+ "spread across multiple lines. This format is preferable to JSON unless the data is not property " +
+ "containers.");
+
+ public static final AllowableValue ORC = new AllowableValue(
+ IngestionProperties.DataFormat.ORC.name(), IngestionProperties.DataFormat.ORC.getKustoValue(), "An ORC file.");
+
+ public static final AllowableValue PARQUET = new AllowableValue(
+ IngestionProperties.DataFormat.PARQUET.name(), IngestionProperties.DataFormat.PARQUET.getKustoValue(), "A parquet file.");
+
+ public static final AllowableValue PSV = new AllowableValue(
+ IngestionProperties.DataFormat.PSV.name(), IngestionProperties.DataFormat.PSV.getKustoValue(), "A text file with values separated by vertical bars (|).");
+
+ public static final AllowableValue SCSV = new AllowableValue(
+ IngestionProperties.DataFormat.SCSV.name(), IngestionProperties.DataFormat.SCSV.getKustoValue(), "A text file with values separated by semicolons (;).");
+
+ public static final AllowableValue SOHSV = new AllowableValue(
+ IngestionProperties.DataFormat.SOHSV.name(), IngestionProperties.DataFormat.SOHSV.getKustoValue(),
+ "A text file with SOH-separated values. (SOH is the ASCII code point 1. " +
+ "This format is used by Hive in HDInsight).");
+
+ public static final AllowableValue TSV = new AllowableValue(
+ IngestionProperties.DataFormat.TSV.name(), IngestionProperties.DataFormat.TSV.getKustoValue(), "A text file with tab delimited values (\\t).");
+
+ public static final AllowableValue TSVE = new AllowableValue(
+ IngestionProperties.DataFormat.TSVE.name(), IngestionProperties.DataFormat.TSVE.getKustoValue(),
+ "A text file with tab-delimited values (\\t). A backslash (\\) is used as escape character.");
+
+ public static final AllowableValue TXT = new AllowableValue(
+ IngestionProperties.DataFormat.TXT.name(), IngestionProperties.DataFormat.TXT.getKustoValue(),
+ "A text file with lines separated by \\n. Empty lines are skipped.");
+
+ public static final AllowableValue IGNORE_FIRST_RECORD_YES = new AllowableValue(
+ "YES", "YES",
+ "Ignore first record during ingestion");
+
+ public static final AllowableValue IGNORE_FIRST_RECORD_NO = new AllowableValue(
+ "NO", "NO",
+ "Do not ignore first record during ingestion");
+
+ public static final PropertyDescriptor DATABASE_NAME = new PropertyDescriptor.Builder()
+ .name("Database Name")
+ .displayName("Database Name")
+ .description("Azure Data Explorer Database Name for querying")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+ .build();
+
+ public static final PropertyDescriptor TABLE_NAME = new PropertyDescriptor.Builder()
+ .name("Table Name")
+ .displayName("Table Name")
+ .description("Azure Data Explorer Table Name for ingesting data")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+ .build();
+
+ public static final PropertyDescriptor MAPPING_NAME = new PropertyDescriptor
+ .Builder().name("Mapping name")
+ .displayName("Mapping name")
+ .description("The name of the mapping responsible for storing the data in the appropriate columns.")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+
+ public static final PropertyDescriptor IS_STREAMING_ENABLED = new PropertyDescriptor
+ .Builder().name("Is Streaming enabled")
+ .displayName("Is Streaming enabled")
+ .description("This property determines whether we want to stream data to ADX.")
+ .required(false)
+ .allowableValues("true", "false")
+ .addValidator(StandardValidators.BOOLEAN_VALIDATOR)
+ .defaultValue("false")
+ .build();
+
+ public static final PropertyDescriptor POLL_ON_INGESTION_STATUS = new PropertyDescriptor
+ .Builder().name("Poll on ingestion status")
+ .displayName("Whether to Poll on ingestion status")
+ .description("This property determines whether we want to poll on ingestion status after an ingestion to ADX is completed")
+ .required(false)
+ .allowableValues("true", "false")
+ .addValidator(StandardValidators.BOOLEAN_VALIDATOR)
+ .defaultValue("false")
+ .build();
+ public static final PropertyDescriptor ADX_SERVICE = new PropertyDescriptor
+ .Builder().name("Kusto Ingest Service")
+ .displayName("Kusto Ingest Service")
+ .description("Azure Data Explorer Kusto Ingest Service")
+ .required(true)
+ .identifiesControllerService(KustoIngestService.class)
+ .build();
+
+ public static final Relationship SUCCESS = new Relationship.Builder()
+ .name("success")
+ .description("Relationship for success")
+ .build();
+ public static final Relationship FAILURE = new Relationship.Builder()
+ .name("failure")
+ .description("Relationship for failure")
+ .build();
+
+ static final PropertyDescriptor DATA_FORMAT = new PropertyDescriptor.Builder()
+ .name("Data Format")
+ .displayName("Data Format")
+ .description("The format of the data that is sent to ADX.")
+ .required(true)
+ .allowableValues(AVRO, APACHEAVRO, CSV, JSON, MULTIJSON, ORC, PARQUET, PSV, SCSV, SOHSV, TSV, TSVE, TXT)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+
+ static final PropertyDescriptor IGNORE_FIRST_RECORD = new PropertyDescriptor.Builder()
+ .name("Ingestion Ignore First Record")
+ .displayName("Ingestion Ignore First Record")
+ .description("Defines whether ignore first record while ingestion.")
+ .required(false)
+ .allowableValues(IGNORE_FIRST_RECORD_YES, IGNORE_FIRST_RECORD_NO)
+ .defaultValue(IGNORE_FIRST_RECORD_NO.getValue())
+ .build();
+
+ @Override
+ protected void init(final ProcessorInitializationContext context) {
+ final List<PropertyDescriptor> descriptorList = new ArrayList<>();
+ descriptorList.add(ADX_SERVICE);
+ descriptorList.add(DATABASE_NAME);
+ descriptorList.add(TABLE_NAME);
+ descriptorList.add(MAPPING_NAME);
+ descriptorList.add(DATA_FORMAT);
+ descriptorList.add(IGNORE_FIRST_RECORD);
+ descriptorList.add(IS_STREAMING_ENABLED);
+ descriptorList.add(POLL_ON_INGESTION_STATUS);
+ this.descriptors = Collections.unmodifiableList(descriptorList);
+
+ final Set<Relationship> relationshipSet = new HashSet<>();
+ relationshipSet.add(SUCCESS);
+ relationshipSet.add(FAILURE);
+ this.relationships = Collections.unmodifiableSet(relationshipSet);
+ }
+
+ @Override
+ public Set<Relationship> getRelationships() {
+ return this.relationships;
+ }
+
+ @Override
+ public final List<PropertyDescriptor> getSupportedPropertyDescriptors() {
+ return descriptors;
+ }
+
+ @OnScheduled
+ public void onScheduled(final ProcessContext context) {
+ service = context.getProperty(ADX_SERVICE).asControllerService(KustoIngestService.class);
+ if (checkIfIngestorRoleDoesntExist(context.getProperty(DATABASE_NAME).getValue(), context.getProperty(TABLE_NAME).getValue())) {
+ getLogger().error("User might not have ingestor privileges, table validation will be skipped for all table mappings.");
+ throw new ProcessException("User might not have ingestor privileges, table validation will be skipped for all table mappings. ");
+ }
+ isStreamingEnabled = context.getProperty(IS_STREAMING_ENABLED).evaluateAttributeExpressions().asBoolean();
+ if (isStreamingEnabled) {
+ checkIfStreamingPolicyIsEnabledInADX(context.getProperty(DATABASE_NAME).getValue(), context.getProperty(DATABASE_NAME).getValue());
+ }
+ }
+
+ @Override
+ public void onTrigger(final ProcessContext context, final ProcessSession session) throws ProcessException {
+ FlowFile flowFile = session.get();
+ if (flowFile == null) {
+ context.yield();
Review Comment:
This `yield` is not necessary.
```suggestion
```
##########
nifi-nar-bundles/nifi-azure-bundle/nifi-azure-processors/src/main/java/org/apache/nifi/processors/azure/data/explorer/PutAzureDataExplorer.java:
##########
@@ -0,0 +1,399 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.azure.data.explorer;
+
+import com.microsoft.azure.kusto.data.KustoResultSetTable;
+import com.microsoft.azure.kusto.ingest.IngestionMapping;
+import com.microsoft.azure.kusto.ingest.IngestionProperties;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.nifi.annotation.behavior.ReadsAttribute;
+import org.apache.nifi.annotation.behavior.ReadsAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.expression.ExpressionLanguageScope;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.services.azure.data.explorer.KustoIngestService;
+import org.apache.nifi.services.azure.data.explorer.KustoIngestionRequest;
+import org.apache.nifi.services.azure.data.explorer.KustoIngestionResult;
+import org.apache.nifi.services.azure.data.explorer.KustoQueryResponse;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.net.URISyntaxException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+
+@Tags({"Azure", "Kusto", "ADX", "Explorer", "Data"})
+@CapabilityDescription("The PutAzureDataExplorer acts as a ADX sink connector which sends flowFiles using the ADX-Service to the provided Azure Data" +
+ "Explorer Ingest Endpoint. The data can be sent through queued ingestion or streaming ingestion to the Azure Data Explorer cluster.")
+@ReadsAttributes({
+ @ReadsAttribute(attribute = "DB_NAME", description = "Specifies the name of the ADX database where the data needs to be stored."),
+ @ReadsAttribute(attribute = "TABLE_NAME", description = "Specifies the name of the ADX table where the data needs to be stored."),
+ @ReadsAttribute(attribute = "MAPPING_NAME", description = "Specifies the name of the mapping responsible for storing the data in appropriate columns."),
+ @ReadsAttribute(attribute = "FLUSH_IMMEDIATE", description = "In case of queued ingestion, this property determines whether the data should be flushed immediately to the ingest endpoint."),
+ @ReadsAttribute(attribute = "DATA_FORMAT", description = "Specifies the format of data that is send to Azure Data Explorer."),
+ @ReadsAttribute(attribute = "IGNORE_FIRST_RECORD", description = "Specifies whether we want to ignore ingestion of first record. " +
+ "This is primarily applicable for csv files. Default is set to NO"),
+ @ReadsAttribute(attribute = "POLL_ON_INGESTION_STATUS", description = "Specifies whether we want to poll on ingestion result during ingestion into ADX." +
+ "In case of applications that need high throughput it is recommended to keep the default value as false. Default is set to false." +
+ "This property should be set to true during Queued Ingestion for near realtime micro-batches of data that require acknowledgement of ingestion status.")
+})
+public class PutAzureDataExplorer extends AbstractProcessor {
+
+ public static final String FETCH_TABLE_COMMAND = "%s | count";
+ public static final String STREAMING_POLICY_SHOW_COMMAND = ".show %s %s policy streamingingestion";
+ public static final String DATABASE = "database";
+
+ private List<PropertyDescriptor> descriptors;
+ private Set<Relationship> relationships;
+ private transient KustoIngestService service;
+ private boolean isStreamingEnabled;
+ private boolean pollOnIngestionStatus;
+
+ public static final AllowableValue AVRO = new AllowableValue(
+ IngestionProperties.DataFormat.AVRO.getKustoValue(), IngestionProperties.DataFormat.AVRO.getKustoValue(),
+ "An Avro format with support for logical types and for the snappy compression codec");
+
+ public static final AllowableValue APACHEAVRO = new AllowableValue(
+ IngestionProperties.DataFormat.APACHEAVRO.getKustoValue(), IngestionProperties.DataFormat.APACHEAVRO.getKustoValue(),
+ "An Avro format with support for logical types and for the snappy compression codec.");
+
+ public static final AllowableValue CSV = new AllowableValue(
+ IngestionProperties.DataFormat.CSV.name(), IngestionProperties.DataFormat.CSV.getKustoValue(),
+ "A text file with comma-separated values (,). For more information, see RFC 4180: Common Format " +
+ "and MIME Type for Comma-Separated Values (CSV) Files.");
+
+ public static final AllowableValue JSON = new AllowableValue(
+ IngestionProperties.DataFormat.JSON.name(), IngestionProperties.DataFormat.JSON.getKustoValue(),
+ "A text file containing JSON objects separated by \\n or \\r\\n. For more information, " +
+ "see JSON Lines (JSONL).");
+
+ public static final AllowableValue MULTIJSON = new AllowableValue(
+ IngestionProperties.DataFormat.MULTIJSON.name(), IngestionProperties.DataFormat.MULTIJSON.getKustoValue(),
+ "A text file containing a JSON array of property containers (each representing a record) or any " +
+ "number of property containers separated by spaces, \\n or \\r\\n. Each property container may be " +
+ "spread across multiple lines. This format is preferable to JSON unless the data is not property " +
+ "containers.");
+
+ public static final AllowableValue ORC = new AllowableValue(
+ IngestionProperties.DataFormat.ORC.name(), IngestionProperties.DataFormat.ORC.getKustoValue(), "An ORC file.");
+
+ public static final AllowableValue PARQUET = new AllowableValue(
+ IngestionProperties.DataFormat.PARQUET.name(), IngestionProperties.DataFormat.PARQUET.getKustoValue(), "A parquet file.");
+
+ public static final AllowableValue PSV = new AllowableValue(
+ IngestionProperties.DataFormat.PSV.name(), IngestionProperties.DataFormat.PSV.getKustoValue(), "A text file with values separated by vertical bars (|).");
+
+ public static final AllowableValue SCSV = new AllowableValue(
+ IngestionProperties.DataFormat.SCSV.name(), IngestionProperties.DataFormat.SCSV.getKustoValue(), "A text file with values separated by semicolons (;).");
+
+ public static final AllowableValue SOHSV = new AllowableValue(
+ IngestionProperties.DataFormat.SOHSV.name(), IngestionProperties.DataFormat.SOHSV.getKustoValue(),
+ "A text file with SOH-separated values. (SOH is the ASCII code point 1. " +
+ "This format is used by Hive in HDInsight).");
+
+ public static final AllowableValue TSV = new AllowableValue(
+ IngestionProperties.DataFormat.TSV.name(), IngestionProperties.DataFormat.TSV.getKustoValue(), "A text file with tab delimited values (\\t).");
+
+ public static final AllowableValue TSVE = new AllowableValue(
+ IngestionProperties.DataFormat.TSVE.name(), IngestionProperties.DataFormat.TSVE.getKustoValue(),
+ "A text file with tab-delimited values (\\t). A backslash (\\) is used as escape character.");
+
+ public static final AllowableValue TXT = new AllowableValue(
+ IngestionProperties.DataFormat.TXT.name(), IngestionProperties.DataFormat.TXT.getKustoValue(),
+ "A text file with lines separated by \\n. Empty lines are skipped.");
+
+ public static final AllowableValue IGNORE_FIRST_RECORD_YES = new AllowableValue(
+ "YES", "YES",
+ "Ignore first record during ingestion");
+
+ public static final AllowableValue IGNORE_FIRST_RECORD_NO = new AllowableValue(
+ "NO", "NO",
+ "Do not ignore first record during ingestion");
+
+ public static final PropertyDescriptor DATABASE_NAME = new PropertyDescriptor.Builder()
+ .name("Database Name")
+ .displayName("Database Name")
+ .description("Azure Data Explorer Database Name for querying")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+ .build();
+
+ public static final PropertyDescriptor TABLE_NAME = new PropertyDescriptor.Builder()
+ .name("Table Name")
+ .displayName("Table Name")
+ .description("Azure Data Explorer Table Name for ingesting data")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+ .build();
+
+ public static final PropertyDescriptor MAPPING_NAME = new PropertyDescriptor
+ .Builder().name("Mapping name")
+ .displayName("Mapping name")
+ .description("The name of the mapping responsible for storing the data in the appropriate columns.")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+
+ public static final PropertyDescriptor IS_STREAMING_ENABLED = new PropertyDescriptor
+ .Builder().name("Is Streaming enabled")
+ .displayName("Is Streaming enabled")
+ .description("This property determines whether we want to stream data to ADX.")
+ .required(false)
+ .allowableValues("true", "false")
+ .addValidator(StandardValidators.BOOLEAN_VALIDATOR)
+ .defaultValue("false")
+ .build();
+
+ public static final PropertyDescriptor POLL_ON_INGESTION_STATUS = new PropertyDescriptor
+ .Builder().name("Poll on ingestion status")
+ .displayName("Whether to Poll on ingestion status")
+ .description("This property determines whether we want to poll on ingestion status after an ingestion to ADX is completed")
+ .required(false)
+ .allowableValues("true", "false")
+ .addValidator(StandardValidators.BOOLEAN_VALIDATOR)
+ .defaultValue("false")
+ .build();
+ public static final PropertyDescriptor ADX_SERVICE = new PropertyDescriptor
+ .Builder().name("Kusto Ingest Service")
+ .displayName("Kusto Ingest Service")
+ .description("Azure Data Explorer Kusto Ingest Service")
+ .required(true)
+ .identifiesControllerService(KustoIngestService.class)
+ .build();
+
+ public static final Relationship SUCCESS = new Relationship.Builder()
+ .name("success")
+ .description("Relationship for success")
+ .build();
+ public static final Relationship FAILURE = new Relationship.Builder()
+ .name("failure")
+ .description("Relationship for failure")
+ .build();
+
+ static final PropertyDescriptor DATA_FORMAT = new PropertyDescriptor.Builder()
+ .name("Data Format")
+ .displayName("Data Format")
+ .description("The format of the data that is sent to ADX.")
+ .required(true)
+ .allowableValues(AVRO, APACHEAVRO, CSV, JSON, MULTIJSON, ORC, PARQUET, PSV, SCSV, SOHSV, TSV, TSVE, TXT)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+
+ static final PropertyDescriptor IGNORE_FIRST_RECORD = new PropertyDescriptor.Builder()
+ .name("Ingestion Ignore First Record")
+ .displayName("Ingestion Ignore First Record")
+ .description("Defines whether ignore first record while ingestion.")
+ .required(false)
+ .allowableValues(IGNORE_FIRST_RECORD_YES, IGNORE_FIRST_RECORD_NO)
+ .defaultValue(IGNORE_FIRST_RECORD_NO.getValue())
+ .build();
+
+ @Override
+ protected void init(final ProcessorInitializationContext context) {
+ final List<PropertyDescriptor> descriptorList = new ArrayList<>();
+ descriptorList.add(ADX_SERVICE);
+ descriptorList.add(DATABASE_NAME);
+ descriptorList.add(TABLE_NAME);
+ descriptorList.add(MAPPING_NAME);
+ descriptorList.add(DATA_FORMAT);
+ descriptorList.add(IGNORE_FIRST_RECORD);
+ descriptorList.add(IS_STREAMING_ENABLED);
+ descriptorList.add(POLL_ON_INGESTION_STATUS);
+ this.descriptors = Collections.unmodifiableList(descriptorList);
+
+ final Set<Relationship> relationshipSet = new HashSet<>();
+ relationshipSet.add(SUCCESS);
+ relationshipSet.add(FAILURE);
+ this.relationships = Collections.unmodifiableSet(relationshipSet);
+ }
+
+ @Override
+ public Set<Relationship> getRelationships() {
+ return this.relationships;
+ }
+
+ @Override
+ public final List<PropertyDescriptor> getSupportedPropertyDescriptors() {
+ return descriptors;
+ }
+
+ @OnScheduled
+ public void onScheduled(final ProcessContext context) {
+ service = context.getProperty(ADX_SERVICE).asControllerService(KustoIngestService.class);
+ if (checkIfIngestorRoleDoesntExist(context.getProperty(DATABASE_NAME).getValue(), context.getProperty(TABLE_NAME).getValue())) {
+ getLogger().error("User might not have ingestor privileges, table validation will be skipped for all table mappings.");
+ throw new ProcessException("User might not have ingestor privileges, table validation will be skipped for all table mappings. ");
+ }
+ isStreamingEnabled = context.getProperty(IS_STREAMING_ENABLED).evaluateAttributeExpressions().asBoolean();
+ if (isStreamingEnabled) {
+ checkIfStreamingPolicyIsEnabledInADX(context.getProperty(DATABASE_NAME).getValue(), context.getProperty(DATABASE_NAME).getValue());
+ }
+ }
+
+ @Override
+ public void onTrigger(final ProcessContext context, final ProcessSession session) throws ProcessException {
+ FlowFile flowFile = session.get();
+ if (flowFile == null) {
+ context.yield();
+ return;
+ }
+
+ IngestionProperties ingestionProperties = new IngestionProperties(context.getProperty(DATABASE_NAME).getValue(),
+ context.getProperty(TABLE_NAME).getValue());
+
+ IngestionMapping.IngestionMappingKind ingestionMappingKind = null;
+
+ isStreamingEnabled = context.getProperty(IS_STREAMING_ENABLED).evaluateAttributeExpressions().asBoolean();
+ pollOnIngestionStatus = context.getProperty(POLL_ON_INGESTION_STATUS).evaluateAttributeExpressions().asBoolean();
+
+ switch (IngestionProperties.DataFormat.valueOf(context.getProperty(DATA_FORMAT).getValue())) {
+ case AVRO:
+ ingestionProperties.setDataFormat(IngestionProperties.DataFormat.AVRO);
+ ingestionMappingKind = IngestionProperties.DataFormat.AVRO.getIngestionMappingKind();
+ break;
+ case APACHEAVRO:
+ ingestionProperties.setDataFormat(IngestionProperties.DataFormat.APACHEAVRO);
+ ingestionMappingKind = IngestionProperties.DataFormat.APACHEAVRO.getIngestionMappingKind();
+ break;
+ case CSV:
+ ingestionProperties.setDataFormat(IngestionProperties.DataFormat.CSV);
+ ingestionMappingKind = IngestionProperties.DataFormat.CSV.getIngestionMappingKind();
+ break;
+ case JSON:
+ ingestionProperties.setDataFormat(IngestionProperties.DataFormat.JSON);
+ ingestionMappingKind = IngestionProperties.DataFormat.JSON.getIngestionMappingKind();
+ break;
+ case MULTIJSON:
+ ingestionProperties.setDataFormat(IngestionProperties.DataFormat.MULTIJSON);
+ ingestionMappingKind = IngestionProperties.DataFormat.MULTIJSON.getIngestionMappingKind();
+ break;
+ case ORC:
+ ingestionProperties.setDataFormat(IngestionProperties.DataFormat.ORC);
+ ingestionMappingKind = IngestionProperties.DataFormat.ORC.getIngestionMappingKind();
+ break;
+ case PARQUET:
+ ingestionProperties.setDataFormat(IngestionProperties.DataFormat.PARQUET);
+ ingestionMappingKind = IngestionProperties.DataFormat.PARQUET.getIngestionMappingKind();
+ break;
+ case PSV:
+ ingestionProperties.setDataFormat(IngestionProperties.DataFormat.PSV);
+ ingestionMappingKind = IngestionProperties.DataFormat.PSV.getIngestionMappingKind();
+ break;
+ case SCSV:
+ ingestionProperties.setDataFormat(IngestionProperties.DataFormat.SCSV);
+ ingestionMappingKind = IngestionProperties.DataFormat.SCSV.getIngestionMappingKind();
+ break;
+ case SOHSV:
+ ingestionProperties.setDataFormat(IngestionProperties.DataFormat.SOHSV);
+ ingestionMappingKind = IngestionProperties.DataFormat.SOHSV.getIngestionMappingKind();
+ break;
+ case TSV:
+ ingestionProperties.setDataFormat(IngestionProperties.DataFormat.TSV);
+ ingestionMappingKind = IngestionProperties.DataFormat.TSV.getIngestionMappingKind();
+ break;
+ case TSVE:
+ ingestionProperties.setDataFormat(IngestionProperties.DataFormat.TSVE);
+ ingestionMappingKind = IngestionProperties.DataFormat.TSVE.getIngestionMappingKind();
+ break;
+ case TXT:
+ ingestionProperties.setDataFormat(IngestionProperties.DataFormat.TXT);
+ ingestionMappingKind = IngestionProperties.DataFormat.TXT.getIngestionMappingKind();
+ break;
+ }
+
+ if (StringUtils.isNotEmpty(context.getProperty(MAPPING_NAME).getValue()) && ingestionMappingKind != null) {
+ ingestionProperties.setIngestionMapping(context.getProperty(MAPPING_NAME).getValue(), ingestionMappingKind);
+ }
+
+ ingestionProperties.setReportLevel(IngestionProperties.IngestionReportLevel.FAILURES_AND_SUCCESSES);
+ ingestionProperties.setReportMethod(IngestionProperties.IngestionReportMethod.TABLE);
+ ingestionProperties.setFlushImmediately(false);
+ ingestionProperties.setIgnoreFirstRecord(StringUtils.equalsIgnoreCase(context.getProperty(IGNORE_FIRST_RECORD).getValue(), IGNORE_FIRST_RECORD_YES.getValue()));
+
+ boolean isError = false;
+
+ // ingestion into ADX
+ try (final InputStream inputStream = session.read(flowFile)) {
+ StringBuilder ingestLogString = new StringBuilder().append("Ingesting with: ")
+ .append("dataFormat - ").append(ingestionProperties.getDataFormat()).append("|")
+ .append("ingestionMapping - ").append(ingestionProperties.getIngestionMapping().getIngestionMappingReference()).append("|")
+ .append("reportLevel - ").append(ingestionProperties.getReportLevel()).append("|")
+ .append("reportMethod - ").append(ingestionProperties.getReportMethod()).append("|")
+ .append("databaseName - ").append(ingestionProperties.getDatabaseName()).append("|")
+ .append("tableName - ").append(ingestionProperties.getTableName()).append("|")
+ .append("flushImmediately - ").append(ingestionProperties.getFlushImmediately());
+ getLogger().info(ingestLogString.toString());
+
+ KustoIngestionResult result = service.ingestData(new KustoIngestionRequest(isStreamingEnabled, pollOnIngestionStatus, inputStream, ingestionProperties));
+ getLogger().info("Poll on Ingestion Status {} and Operation status: {} ", pollOnIngestionStatus, result.toString());
+
+ if (result == KustoIngestionResult.SUCCEEDED) {
+ getLogger().info("Operation status Succeeded - {}", result.toString());
Review Comment:
```suggestion
getLogger().info("Ingest Completed: {}", result);
```
##########
nifi-nar-bundles/nifi-azure-bundle/nifi-azure-processors/src/main/java/org/apache/nifi/processors/azure/data/explorer/PutAzureDataExplorer.java:
##########
@@ -0,0 +1,399 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.azure.data.explorer;
+
+import com.microsoft.azure.kusto.data.KustoResultSetTable;
+import com.microsoft.azure.kusto.ingest.IngestionMapping;
+import com.microsoft.azure.kusto.ingest.IngestionProperties;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.nifi.annotation.behavior.ReadsAttribute;
+import org.apache.nifi.annotation.behavior.ReadsAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.expression.ExpressionLanguageScope;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.services.azure.data.explorer.KustoIngestService;
+import org.apache.nifi.services.azure.data.explorer.KustoIngestionRequest;
+import org.apache.nifi.services.azure.data.explorer.KustoIngestionResult;
+import org.apache.nifi.services.azure.data.explorer.KustoQueryResponse;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.net.URISyntaxException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+
+@Tags({"Azure", "Kusto", "ADX", "Explorer", "Data"})
+@CapabilityDescription("The PutAzureDataExplorer acts as a ADX sink connector which sends flowFiles using the ADX-Service to the provided Azure Data" +
+ "Explorer Ingest Endpoint. The data can be sent through queued ingestion or streaming ingestion to the Azure Data Explorer cluster.")
+@ReadsAttributes({
+ @ReadsAttribute(attribute = "DB_NAME", description = "Specifies the name of the ADX database where the data needs to be stored."),
+ @ReadsAttribute(attribute = "TABLE_NAME", description = "Specifies the name of the ADX table where the data needs to be stored."),
+ @ReadsAttribute(attribute = "MAPPING_NAME", description = "Specifies the name of the mapping responsible for storing the data in appropriate columns."),
+ @ReadsAttribute(attribute = "FLUSH_IMMEDIATE", description = "In case of queued ingestion, this property determines whether the data should be flushed immediately to the ingest endpoint."),
+ @ReadsAttribute(attribute = "DATA_FORMAT", description = "Specifies the format of data that is send to Azure Data Explorer."),
+ @ReadsAttribute(attribute = "IGNORE_FIRST_RECORD", description = "Specifies whether we want to ignore ingestion of first record. " +
+ "This is primarily applicable for csv files. Default is set to NO"),
+ @ReadsAttribute(attribute = "POLL_ON_INGESTION_STATUS", description = "Specifies whether we want to poll on ingestion result during ingestion into ADX." +
+ "In case of applications that need high throughput it is recommended to keep the default value as false. Default is set to false." +
+ "This property should be set to true during Queued Ingestion for near realtime micro-batches of data that require acknowledgement of ingestion status.")
+})
+public class PutAzureDataExplorer extends AbstractProcessor {
+
+ public static final String FETCH_TABLE_COMMAND = "%s | count";
+ public static final String STREAMING_POLICY_SHOW_COMMAND = ".show %s %s policy streamingingestion";
+ public static final String DATABASE = "database";
+
+ private List<PropertyDescriptor> descriptors;
+ private Set<Relationship> relationships;
+ private transient KustoIngestService service;
+ private boolean isStreamingEnabled;
+ private boolean pollOnIngestionStatus;
+
+ public static final AllowableValue AVRO = new AllowableValue(
+ IngestionProperties.DataFormat.AVRO.getKustoValue(), IngestionProperties.DataFormat.AVRO.getKustoValue(),
+ "An Avro format with support for logical types and for the snappy compression codec");
+
+ public static final AllowableValue APACHEAVRO = new AllowableValue(
+ IngestionProperties.DataFormat.APACHEAVRO.getKustoValue(), IngestionProperties.DataFormat.APACHEAVRO.getKustoValue(),
+ "An Avro format with support for logical types and for the snappy compression codec.");
+
+ public static final AllowableValue CSV = new AllowableValue(
+ IngestionProperties.DataFormat.CSV.name(), IngestionProperties.DataFormat.CSV.getKustoValue(),
+ "A text file with comma-separated values (,). For more information, see RFC 4180: Common Format " +
+ "and MIME Type for Comma-Separated Values (CSV) Files.");
+
+ public static final AllowableValue JSON = new AllowableValue(
+ IngestionProperties.DataFormat.JSON.name(), IngestionProperties.DataFormat.JSON.getKustoValue(),
+ "A text file containing JSON objects separated by \\n or \\r\\n. For more information, " +
+ "see JSON Lines (JSONL).");
+
+ public static final AllowableValue MULTIJSON = new AllowableValue(
+ IngestionProperties.DataFormat.MULTIJSON.name(), IngestionProperties.DataFormat.MULTIJSON.getKustoValue(),
+ "A text file containing a JSON array of property containers (each representing a record) or any " +
+ "number of property containers separated by spaces, \\n or \\r\\n. Each property container may be " +
+ "spread across multiple lines. This format is preferable to JSON unless the data is not property " +
+ "containers.");
+
+ public static final AllowableValue ORC = new AllowableValue(
+ IngestionProperties.DataFormat.ORC.name(), IngestionProperties.DataFormat.ORC.getKustoValue(), "An ORC file.");
+
+ public static final AllowableValue PARQUET = new AllowableValue(
+ IngestionProperties.DataFormat.PARQUET.name(), IngestionProperties.DataFormat.PARQUET.getKustoValue(), "A parquet file.");
+
+ public static final AllowableValue PSV = new AllowableValue(
+ IngestionProperties.DataFormat.PSV.name(), IngestionProperties.DataFormat.PSV.getKustoValue(), "A text file with values separated by vertical bars (|).");
+
+ public static final AllowableValue SCSV = new AllowableValue(
+ IngestionProperties.DataFormat.SCSV.name(), IngestionProperties.DataFormat.SCSV.getKustoValue(), "A text file with values separated by semicolons (;).");
+
+ public static final AllowableValue SOHSV = new AllowableValue(
+ IngestionProperties.DataFormat.SOHSV.name(), IngestionProperties.DataFormat.SOHSV.getKustoValue(),
+ "A text file with SOH-separated values. (SOH is the ASCII code point 1. " +
+ "This format is used by Hive in HDInsight).");
+
+ public static final AllowableValue TSV = new AllowableValue(
+ IngestionProperties.DataFormat.TSV.name(), IngestionProperties.DataFormat.TSV.getKustoValue(), "A text file with tab delimited values (\\t).");
+
+ public static final AllowableValue TSVE = new AllowableValue(
+ IngestionProperties.DataFormat.TSVE.name(), IngestionProperties.DataFormat.TSVE.getKustoValue(),
+ "A text file with tab-delimited values (\\t). A backslash (\\) is used as escape character.");
+
+ public static final AllowableValue TXT = new AllowableValue(
+ IngestionProperties.DataFormat.TXT.name(), IngestionProperties.DataFormat.TXT.getKustoValue(),
+ "A text file with lines separated by \\n. Empty lines are skipped.");
+
+ public static final AllowableValue IGNORE_FIRST_RECORD_YES = new AllowableValue(
+ "YES", "YES",
+ "Ignore first record during ingestion");
+
+ public static final AllowableValue IGNORE_FIRST_RECORD_NO = new AllowableValue(
+ "NO", "NO",
+ "Do not ignore first record during ingestion");
+
+ public static final PropertyDescriptor DATABASE_NAME = new PropertyDescriptor.Builder()
+ .name("Database Name")
+ .displayName("Database Name")
+ .description("Azure Data Explorer Database Name for querying")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+ .build();
+
+ public static final PropertyDescriptor TABLE_NAME = new PropertyDescriptor.Builder()
+ .name("Table Name")
+ .displayName("Table Name")
+ .description("Azure Data Explorer Table Name for ingesting data")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+ .build();
+
+ public static final PropertyDescriptor MAPPING_NAME = new PropertyDescriptor
+ .Builder().name("Mapping name")
+ .displayName("Mapping name")
+ .description("The name of the mapping responsible for storing the data in the appropriate columns.")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+
+ public static final PropertyDescriptor IS_STREAMING_ENABLED = new PropertyDescriptor
+ .Builder().name("Is Streaming enabled")
+ .displayName("Is Streaming enabled")
+ .description("This property determines whether we want to stream data to ADX.")
+ .required(false)
+ .allowableValues("true", "false")
+ .addValidator(StandardValidators.BOOLEAN_VALIDATOR)
+ .defaultValue("false")
+ .build();
+
+ public static final PropertyDescriptor POLL_ON_INGESTION_STATUS = new PropertyDescriptor
+ .Builder().name("Poll on ingestion status")
+ .displayName("Whether to Poll on ingestion status")
+ .description("This property determines whether we want to poll on ingestion status after an ingestion to ADX is completed")
+ .required(false)
+ .allowableValues("true", "false")
+ .addValidator(StandardValidators.BOOLEAN_VALIDATOR)
+ .defaultValue("false")
+ .build();
+ public static final PropertyDescriptor ADX_SERVICE = new PropertyDescriptor
+ .Builder().name("Kusto Ingest Service")
+ .displayName("Kusto Ingest Service")
+ .description("Azure Data Explorer Kusto Ingest Service")
+ .required(true)
+ .identifiesControllerService(KustoIngestService.class)
+ .build();
+
+ public static final Relationship SUCCESS = new Relationship.Builder()
+ .name("success")
+ .description("Relationship for success")
+ .build();
+ public static final Relationship FAILURE = new Relationship.Builder()
+ .name("failure")
+ .description("Relationship for failure")
+ .build();
+
+ static final PropertyDescriptor DATA_FORMAT = new PropertyDescriptor.Builder()
+ .name("Data Format")
+ .displayName("Data Format")
+ .description("The format of the data that is sent to ADX.")
+ .required(true)
+ .allowableValues(AVRO, APACHEAVRO, CSV, JSON, MULTIJSON, ORC, PARQUET, PSV, SCSV, SOHSV, TSV, TSVE, TXT)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+
+ static final PropertyDescriptor IGNORE_FIRST_RECORD = new PropertyDescriptor.Builder()
+ .name("Ingestion Ignore First Record")
+ .displayName("Ingestion Ignore First Record")
+ .description("Defines whether ignore first record while ingestion.")
+ .required(false)
+ .allowableValues(IGNORE_FIRST_RECORD_YES, IGNORE_FIRST_RECORD_NO)
+ .defaultValue(IGNORE_FIRST_RECORD_NO.getValue())
+ .build();
+
+ @Override
+ protected void init(final ProcessorInitializationContext context) {
+ final List<PropertyDescriptor> descriptorList = new ArrayList<>();
+ descriptorList.add(ADX_SERVICE);
+ descriptorList.add(DATABASE_NAME);
+ descriptorList.add(TABLE_NAME);
+ descriptorList.add(MAPPING_NAME);
+ descriptorList.add(DATA_FORMAT);
+ descriptorList.add(IGNORE_FIRST_RECORD);
+ descriptorList.add(IS_STREAMING_ENABLED);
+ descriptorList.add(POLL_ON_INGESTION_STATUS);
+ this.descriptors = Collections.unmodifiableList(descriptorList);
+
+ final Set<Relationship> relationshipSet = new HashSet<>();
+ relationshipSet.add(SUCCESS);
+ relationshipSet.add(FAILURE);
+ this.relationships = Collections.unmodifiableSet(relationshipSet);
+ }
+
+ @Override
+ public Set<Relationship> getRelationships() {
+ return this.relationships;
+ }
+
+ @Override
+ public final List<PropertyDescriptor> getSupportedPropertyDescriptors() {
+ return descriptors;
+ }
+
+ @OnScheduled
+ public void onScheduled(final ProcessContext context) {
+ service = context.getProperty(ADX_SERVICE).asControllerService(KustoIngestService.class);
+ if (checkIfIngestorRoleDoesntExist(context.getProperty(DATABASE_NAME).getValue(), context.getProperty(TABLE_NAME).getValue())) {
+ getLogger().error("User might not have ingestor privileges, table validation will be skipped for all table mappings.");
+ throw new ProcessException("User might not have ingestor privileges, table validation will be skipped for all table mappings. ");
+ }
+ isStreamingEnabled = context.getProperty(IS_STREAMING_ENABLED).evaluateAttributeExpressions().asBoolean();
Review Comment:
```suggestion
streamingEnabled = context.getProperty(IS_STREAMING_ENABLED).evaluateAttributeExpressions().asBoolean();
```
##########
nifi-nar-bundles/nifi-azure-bundle/nifi-azure-processors/src/main/java/org/apache/nifi/processors/azure/data/explorer/PutAzureDataExplorer.java:
##########
@@ -0,0 +1,399 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.azure.data.explorer;
+
+import com.microsoft.azure.kusto.data.KustoResultSetTable;
+import com.microsoft.azure.kusto.ingest.IngestionMapping;
+import com.microsoft.azure.kusto.ingest.IngestionProperties;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.nifi.annotation.behavior.ReadsAttribute;
+import org.apache.nifi.annotation.behavior.ReadsAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.expression.ExpressionLanguageScope;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.services.azure.data.explorer.KustoIngestService;
+import org.apache.nifi.services.azure.data.explorer.KustoIngestionRequest;
+import org.apache.nifi.services.azure.data.explorer.KustoIngestionResult;
+import org.apache.nifi.services.azure.data.explorer.KustoQueryResponse;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.net.URISyntaxException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+
+@Tags({"Azure", "Kusto", "ADX", "Explorer", "Data"})
+@CapabilityDescription("The PutAzureDataExplorer acts as a ADX sink connector which sends flowFiles using the ADX-Service to the provided Azure Data" +
+ "Explorer Ingest Endpoint. The data can be sent through queued ingestion or streaming ingestion to the Azure Data Explorer cluster.")
+@ReadsAttributes({
+ @ReadsAttribute(attribute = "DB_NAME", description = "Specifies the name of the ADX database where the data needs to be stored."),
+ @ReadsAttribute(attribute = "TABLE_NAME", description = "Specifies the name of the ADX table where the data needs to be stored."),
+ @ReadsAttribute(attribute = "MAPPING_NAME", description = "Specifies the name of the mapping responsible for storing the data in appropriate columns."),
+ @ReadsAttribute(attribute = "FLUSH_IMMEDIATE", description = "In case of queued ingestion, this property determines whether the data should be flushed immediately to the ingest endpoint."),
+ @ReadsAttribute(attribute = "DATA_FORMAT", description = "Specifies the format of data that is send to Azure Data Explorer."),
+ @ReadsAttribute(attribute = "IGNORE_FIRST_RECORD", description = "Specifies whether we want to ignore ingestion of first record. " +
+ "This is primarily applicable for csv files. Default is set to NO"),
+ @ReadsAttribute(attribute = "POLL_ON_INGESTION_STATUS", description = "Specifies whether we want to poll on ingestion result during ingestion into ADX." +
+ "In case of applications that need high throughput it is recommended to keep the default value as false. Default is set to false." +
+ "This property should be set to true during Queued Ingestion for near realtime micro-batches of data that require acknowledgement of ingestion status.")
+})
+public class PutAzureDataExplorer extends AbstractProcessor {
+
+ public static final String FETCH_TABLE_COMMAND = "%s | count";
+ public static final String STREAMING_POLICY_SHOW_COMMAND = ".show %s %s policy streamingingestion";
+ public static final String DATABASE = "database";
+
+ private List<PropertyDescriptor> descriptors;
+ private Set<Relationship> relationships;
+ private transient KustoIngestService service;
+ private boolean isStreamingEnabled;
+ private boolean pollOnIngestionStatus;
+
+ public static final AllowableValue AVRO = new AllowableValue(
+ IngestionProperties.DataFormat.AVRO.getKustoValue(), IngestionProperties.DataFormat.AVRO.getKustoValue(),
+ "An Avro format with support for logical types and for the snappy compression codec");
+
+ public static final AllowableValue APACHEAVRO = new AllowableValue(
+ IngestionProperties.DataFormat.APACHEAVRO.getKustoValue(), IngestionProperties.DataFormat.APACHEAVRO.getKustoValue(),
+ "An Avro format with support for logical types and for the snappy compression codec.");
+
+ public static final AllowableValue CSV = new AllowableValue(
+ IngestionProperties.DataFormat.CSV.name(), IngestionProperties.DataFormat.CSV.getKustoValue(),
+ "A text file with comma-separated values (,). For more information, see RFC 4180: Common Format " +
+ "and MIME Type for Comma-Separated Values (CSV) Files.");
+
+ public static final AllowableValue JSON = new AllowableValue(
+ IngestionProperties.DataFormat.JSON.name(), IngestionProperties.DataFormat.JSON.getKustoValue(),
+ "A text file containing JSON objects separated by \\n or \\r\\n. For more information, " +
+ "see JSON Lines (JSONL).");
+
+ public static final AllowableValue MULTIJSON = new AllowableValue(
+ IngestionProperties.DataFormat.MULTIJSON.name(), IngestionProperties.DataFormat.MULTIJSON.getKustoValue(),
+ "A text file containing a JSON array of property containers (each representing a record) or any " +
+ "number of property containers separated by spaces, \\n or \\r\\n. Each property container may be " +
+ "spread across multiple lines. This format is preferable to JSON unless the data is not property " +
+ "containers.");
+
+ public static final AllowableValue ORC = new AllowableValue(
+ IngestionProperties.DataFormat.ORC.name(), IngestionProperties.DataFormat.ORC.getKustoValue(), "An ORC file.");
+
+ public static final AllowableValue PARQUET = new AllowableValue(
+ IngestionProperties.DataFormat.PARQUET.name(), IngestionProperties.DataFormat.PARQUET.getKustoValue(), "A parquet file.");
+
+ public static final AllowableValue PSV = new AllowableValue(
+ IngestionProperties.DataFormat.PSV.name(), IngestionProperties.DataFormat.PSV.getKustoValue(), "A text file with values separated by vertical bars (|).");
+
+ public static final AllowableValue SCSV = new AllowableValue(
+ IngestionProperties.DataFormat.SCSV.name(), IngestionProperties.DataFormat.SCSV.getKustoValue(), "A text file with values separated by semicolons (;).");
+
+ public static final AllowableValue SOHSV = new AllowableValue(
+ IngestionProperties.DataFormat.SOHSV.name(), IngestionProperties.DataFormat.SOHSV.getKustoValue(),
+ "A text file with SOH-separated values. (SOH is the ASCII code point 1. " +
+ "This format is used by Hive in HDInsight).");
+
+ public static final AllowableValue TSV = new AllowableValue(
+ IngestionProperties.DataFormat.TSV.name(), IngestionProperties.DataFormat.TSV.getKustoValue(), "A text file with tab delimited values (\\t).");
+
+ public static final AllowableValue TSVE = new AllowableValue(
+ IngestionProperties.DataFormat.TSVE.name(), IngestionProperties.DataFormat.TSVE.getKustoValue(),
+ "A text file with tab-delimited values (\\t). A backslash (\\) is used as escape character.");
+
+ public static final AllowableValue TXT = new AllowableValue(
+ IngestionProperties.DataFormat.TXT.name(), IngestionProperties.DataFormat.TXT.getKustoValue(),
+ "A text file with lines separated by \\n. Empty lines are skipped.");
Review Comment:
Recommend creating a new `enum` named `IngestDataFormat` that implements the `DescribedValue` interface. That provides a clear enumeration of supported values that can be used in Property Descriptor definitions, and also avoids the need to define AllowableValues in the Processor itself.
##########
nifi-nar-bundles/nifi-azure-bundle/nifi-azure-processors/src/main/java/org/apache/nifi/processors/azure/data/explorer/PutAzureDataExplorer.java:
##########
@@ -0,0 +1,399 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.azure.data.explorer;
+
+import com.microsoft.azure.kusto.data.KustoResultSetTable;
+import com.microsoft.azure.kusto.ingest.IngestionMapping;
+import com.microsoft.azure.kusto.ingest.IngestionProperties;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.nifi.annotation.behavior.ReadsAttribute;
+import org.apache.nifi.annotation.behavior.ReadsAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.expression.ExpressionLanguageScope;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.services.azure.data.explorer.KustoIngestService;
+import org.apache.nifi.services.azure.data.explorer.KustoIngestionRequest;
+import org.apache.nifi.services.azure.data.explorer.KustoIngestionResult;
+import org.apache.nifi.services.azure.data.explorer.KustoQueryResponse;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.net.URISyntaxException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+
+@Tags({"Azure", "Kusto", "ADX", "Explorer", "Data"})
+@CapabilityDescription("The PutAzureDataExplorer acts as a ADX sink connector which sends flowFiles using the ADX-Service to the provided Azure Data" +
+ "Explorer Ingest Endpoint. The data can be sent through queued ingestion or streaming ingestion to the Azure Data Explorer cluster.")
+@ReadsAttributes({
+ @ReadsAttribute(attribute = "DB_NAME", description = "Specifies the name of the ADX database where the data needs to be stored."),
+ @ReadsAttribute(attribute = "TABLE_NAME", description = "Specifies the name of the ADX table where the data needs to be stored."),
+ @ReadsAttribute(attribute = "MAPPING_NAME", description = "Specifies the name of the mapping responsible for storing the data in appropriate columns."),
+ @ReadsAttribute(attribute = "FLUSH_IMMEDIATE", description = "In case of queued ingestion, this property determines whether the data should be flushed immediately to the ingest endpoint."),
+ @ReadsAttribute(attribute = "DATA_FORMAT", description = "Specifies the format of data that is send to Azure Data Explorer."),
+ @ReadsAttribute(attribute = "IGNORE_FIRST_RECORD", description = "Specifies whether we want to ignore ingestion of first record. " +
+ "This is primarily applicable for csv files. Default is set to NO"),
+ @ReadsAttribute(attribute = "POLL_ON_INGESTION_STATUS", description = "Specifies whether we want to poll on ingestion result during ingestion into ADX." +
+ "In case of applications that need high throughput it is recommended to keep the default value as false. Default is set to false." +
+ "This property should be set to true during Queued Ingestion for near realtime micro-batches of data that require acknowledgement of ingestion status.")
+})
+public class PutAzureDataExplorer extends AbstractProcessor {
+
+ public static final String FETCH_TABLE_COMMAND = "%s | count";
+ public static final String STREAMING_POLICY_SHOW_COMMAND = ".show %s %s policy streamingingestion";
+ public static final String DATABASE = "database";
+
+ private List<PropertyDescriptor> descriptors;
+ private Set<Relationship> relationships;
+ private transient KustoIngestService service;
+ private boolean isStreamingEnabled;
+ private boolean pollOnIngestionStatus;
+
+ public static final AllowableValue AVRO = new AllowableValue(
+ IngestionProperties.DataFormat.AVRO.getKustoValue(), IngestionProperties.DataFormat.AVRO.getKustoValue(),
+ "An Avro format with support for logical types and for the snappy compression codec");
+
+ public static final AllowableValue APACHEAVRO = new AllowableValue(
+ IngestionProperties.DataFormat.APACHEAVRO.getKustoValue(), IngestionProperties.DataFormat.APACHEAVRO.getKustoValue(),
+ "An Avro format with support for logical types and for the snappy compression codec.");
+
+ public static final AllowableValue CSV = new AllowableValue(
+ IngestionProperties.DataFormat.CSV.name(), IngestionProperties.DataFormat.CSV.getKustoValue(),
+ "A text file with comma-separated values (,). For more information, see RFC 4180: Common Format " +
+ "and MIME Type for Comma-Separated Values (CSV) Files.");
+
+ public static final AllowableValue JSON = new AllowableValue(
+ IngestionProperties.DataFormat.JSON.name(), IngestionProperties.DataFormat.JSON.getKustoValue(),
+ "A text file containing JSON objects separated by \\n or \\r\\n. For more information, " +
+ "see JSON Lines (JSONL).");
+
+ public static final AllowableValue MULTIJSON = new AllowableValue(
+ IngestionProperties.DataFormat.MULTIJSON.name(), IngestionProperties.DataFormat.MULTIJSON.getKustoValue(),
+ "A text file containing a JSON array of property containers (each representing a record) or any " +
+ "number of property containers separated by spaces, \\n or \\r\\n. Each property container may be " +
+ "spread across multiple lines. This format is preferable to JSON unless the data is not property " +
+ "containers.");
+
+ public static final AllowableValue ORC = new AllowableValue(
+ IngestionProperties.DataFormat.ORC.name(), IngestionProperties.DataFormat.ORC.getKustoValue(), "An ORC file.");
+
+ public static final AllowableValue PARQUET = new AllowableValue(
+ IngestionProperties.DataFormat.PARQUET.name(), IngestionProperties.DataFormat.PARQUET.getKustoValue(), "A parquet file.");
+
+ public static final AllowableValue PSV = new AllowableValue(
+ IngestionProperties.DataFormat.PSV.name(), IngestionProperties.DataFormat.PSV.getKustoValue(), "A text file with values separated by vertical bars (|).");
+
+ public static final AllowableValue SCSV = new AllowableValue(
+ IngestionProperties.DataFormat.SCSV.name(), IngestionProperties.DataFormat.SCSV.getKustoValue(), "A text file with values separated by semicolons (;).");
+
+ public static final AllowableValue SOHSV = new AllowableValue(
+ IngestionProperties.DataFormat.SOHSV.name(), IngestionProperties.DataFormat.SOHSV.getKustoValue(),
+ "A text file with SOH-separated values. (SOH is the ASCII code point 1. " +
+ "This format is used by Hive in HDInsight).");
+
+ public static final AllowableValue TSV = new AllowableValue(
+ IngestionProperties.DataFormat.TSV.name(), IngestionProperties.DataFormat.TSV.getKustoValue(), "A text file with tab delimited values (\\t).");
+
+ public static final AllowableValue TSVE = new AllowableValue(
+ IngestionProperties.DataFormat.TSVE.name(), IngestionProperties.DataFormat.TSVE.getKustoValue(),
+ "A text file with tab-delimited values (\\t). A backslash (\\) is used as escape character.");
+
+ public static final AllowableValue TXT = new AllowableValue(
+ IngestionProperties.DataFormat.TXT.name(), IngestionProperties.DataFormat.TXT.getKustoValue(),
+ "A text file with lines separated by \\n. Empty lines are skipped.");
+
+ public static final AllowableValue IGNORE_FIRST_RECORD_YES = new AllowableValue(
+ "YES", "YES",
+ "Ignore first record during ingestion");
+
+ public static final AllowableValue IGNORE_FIRST_RECORD_NO = new AllowableValue(
+ "NO", "NO",
+ "Do not ignore first record during ingestion");
+
+ public static final PropertyDescriptor DATABASE_NAME = new PropertyDescriptor.Builder()
+ .name("Database Name")
+ .displayName("Database Name")
+ .description("Azure Data Explorer Database Name for querying")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+ .build();
+
+ public static final PropertyDescriptor TABLE_NAME = new PropertyDescriptor.Builder()
+ .name("Table Name")
+ .displayName("Table Name")
+ .description("Azure Data Explorer Table Name for ingesting data")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+ .build();
+
+ public static final PropertyDescriptor MAPPING_NAME = new PropertyDescriptor
+ .Builder().name("Mapping name")
+ .displayName("Mapping name")
Review Comment:
```suggestion
.Builder().name("Ingest Mapping Name")
.displayName("Ingest Mapping Name")
```
##########
nifi-nar-bundles/nifi-azure-bundle/nifi-azure-processors/src/main/java/org/apache/nifi/processors/azure/data/explorer/PutAzureDataExplorer.java:
##########
@@ -0,0 +1,399 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.azure.data.explorer;
+
+import com.microsoft.azure.kusto.data.KustoResultSetTable;
+import com.microsoft.azure.kusto.ingest.IngestionMapping;
+import com.microsoft.azure.kusto.ingest.IngestionProperties;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.nifi.annotation.behavior.ReadsAttribute;
+import org.apache.nifi.annotation.behavior.ReadsAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.expression.ExpressionLanguageScope;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.services.azure.data.explorer.KustoIngestService;
+import org.apache.nifi.services.azure.data.explorer.KustoIngestionRequest;
+import org.apache.nifi.services.azure.data.explorer.KustoIngestionResult;
+import org.apache.nifi.services.azure.data.explorer.KustoQueryResponse;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.net.URISyntaxException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+
+@Tags({"Azure", "Kusto", "ADX", "Explorer", "Data"})
+@CapabilityDescription("The PutAzureDataExplorer acts as a ADX sink connector which sends flowFiles using the ADX-Service to the provided Azure Data" +
+ "Explorer Ingest Endpoint. The data can be sent through queued ingestion or streaming ingestion to the Azure Data Explorer cluster.")
+@ReadsAttributes({
+ @ReadsAttribute(attribute = "DB_NAME", description = "Specifies the name of the ADX database where the data needs to be stored."),
+ @ReadsAttribute(attribute = "TABLE_NAME", description = "Specifies the name of the ADX table where the data needs to be stored."),
+ @ReadsAttribute(attribute = "MAPPING_NAME", description = "Specifies the name of the mapping responsible for storing the data in appropriate columns."),
+ @ReadsAttribute(attribute = "FLUSH_IMMEDIATE", description = "In case of queued ingestion, this property determines whether the data should be flushed immediately to the ingest endpoint."),
+ @ReadsAttribute(attribute = "DATA_FORMAT", description = "Specifies the format of data that is send to Azure Data Explorer."),
+ @ReadsAttribute(attribute = "IGNORE_FIRST_RECORD", description = "Specifies whether we want to ignore ingestion of first record. " +
+ "This is primarily applicable for csv files. Default is set to NO"),
+ @ReadsAttribute(attribute = "POLL_ON_INGESTION_STATUS", description = "Specifies whether we want to poll on ingestion result during ingestion into ADX." +
+ "In case of applications that need high throughput it is recommended to keep the default value as false. Default is set to false." +
+ "This property should be set to true during Queued Ingestion for near realtime micro-batches of data that require acknowledgement of ingestion status.")
+})
+public class PutAzureDataExplorer extends AbstractProcessor {
+
+ public static final String FETCH_TABLE_COMMAND = "%s | count";
+ public static final String STREAMING_POLICY_SHOW_COMMAND = ".show %s %s policy streamingingestion";
+ public static final String DATABASE = "database";
+
+ private List<PropertyDescriptor> descriptors;
+ private Set<Relationship> relationships;
+ private transient KustoIngestService service;
+ private boolean isStreamingEnabled;
+ private boolean pollOnIngestionStatus;
+
+ public static final AllowableValue AVRO = new AllowableValue(
+ IngestionProperties.DataFormat.AVRO.getKustoValue(), IngestionProperties.DataFormat.AVRO.getKustoValue(),
+ "An Avro format with support for logical types and for the snappy compression codec");
+
+ public static final AllowableValue APACHEAVRO = new AllowableValue(
+ IngestionProperties.DataFormat.APACHEAVRO.getKustoValue(), IngestionProperties.DataFormat.APACHEAVRO.getKustoValue(),
+ "An Avro format with support for logical types and for the snappy compression codec.");
+
+ public static final AllowableValue CSV = new AllowableValue(
+ IngestionProperties.DataFormat.CSV.name(), IngestionProperties.DataFormat.CSV.getKustoValue(),
+ "A text file with comma-separated values (,). For more information, see RFC 4180: Common Format " +
+ "and MIME Type for Comma-Separated Values (CSV) Files.");
+
+ public static final AllowableValue JSON = new AllowableValue(
+ IngestionProperties.DataFormat.JSON.name(), IngestionProperties.DataFormat.JSON.getKustoValue(),
+ "A text file containing JSON objects separated by \\n or \\r\\n. For more information, " +
+ "see JSON Lines (JSONL).");
+
+ public static final AllowableValue MULTIJSON = new AllowableValue(
+ IngestionProperties.DataFormat.MULTIJSON.name(), IngestionProperties.DataFormat.MULTIJSON.getKustoValue(),
+ "A text file containing a JSON array of property containers (each representing a record) or any " +
+ "number of property containers separated by spaces, \\n or \\r\\n. Each property container may be " +
+ "spread across multiple lines. This format is preferable to JSON unless the data is not property " +
+ "containers.");
+
+ public static final AllowableValue ORC = new AllowableValue(
+ IngestionProperties.DataFormat.ORC.name(), IngestionProperties.DataFormat.ORC.getKustoValue(), "An ORC file.");
+
+ public static final AllowableValue PARQUET = new AllowableValue(
+ IngestionProperties.DataFormat.PARQUET.name(), IngestionProperties.DataFormat.PARQUET.getKustoValue(), "A parquet file.");
+
+ public static final AllowableValue PSV = new AllowableValue(
+ IngestionProperties.DataFormat.PSV.name(), IngestionProperties.DataFormat.PSV.getKustoValue(), "A text file with values separated by vertical bars (|).");
+
+ public static final AllowableValue SCSV = new AllowableValue(
+ IngestionProperties.DataFormat.SCSV.name(), IngestionProperties.DataFormat.SCSV.getKustoValue(), "A text file with values separated by semicolons (;).");
+
+ public static final AllowableValue SOHSV = new AllowableValue(
+ IngestionProperties.DataFormat.SOHSV.name(), IngestionProperties.DataFormat.SOHSV.getKustoValue(),
+ "A text file with SOH-separated values. (SOH is the ASCII code point 1. " +
+ "This format is used by Hive in HDInsight).");
+
+ public static final AllowableValue TSV = new AllowableValue(
+ IngestionProperties.DataFormat.TSV.name(), IngestionProperties.DataFormat.TSV.getKustoValue(), "A text file with tab delimited values (\\t).");
+
+ public static final AllowableValue TSVE = new AllowableValue(
+ IngestionProperties.DataFormat.TSVE.name(), IngestionProperties.DataFormat.TSVE.getKustoValue(),
+ "A text file with tab-delimited values (\\t). A backslash (\\) is used as escape character.");
+
+ public static final AllowableValue TXT = new AllowableValue(
+ IngestionProperties.DataFormat.TXT.name(), IngestionProperties.DataFormat.TXT.getKustoValue(),
+ "A text file with lines separated by \\n. Empty lines are skipped.");
+
+ public static final AllowableValue IGNORE_FIRST_RECORD_YES = new AllowableValue(
+ "YES", "YES",
+ "Ignore first record during ingestion");
+
+ public static final AllowableValue IGNORE_FIRST_RECORD_NO = new AllowableValue(
+ "NO", "NO",
+ "Do not ignore first record during ingestion");
+
+ public static final PropertyDescriptor DATABASE_NAME = new PropertyDescriptor.Builder()
+ .name("Database Name")
+ .displayName("Database Name")
+ .description("Azure Data Explorer Database Name for querying")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+ .build();
+
+ public static final PropertyDescriptor TABLE_NAME = new PropertyDescriptor.Builder()
+ .name("Table Name")
+ .displayName("Table Name")
+ .description("Azure Data Explorer Table Name for ingesting data")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+ .build();
+
+ public static final PropertyDescriptor MAPPING_NAME = new PropertyDescriptor
+ .Builder().name("Mapping name")
+ .displayName("Mapping name")
+ .description("The name of the mapping responsible for storing the data in the appropriate columns.")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+
+ public static final PropertyDescriptor IS_STREAMING_ENABLED = new PropertyDescriptor
+ .Builder().name("Is Streaming enabled")
+ .displayName("Is Streaming enabled")
+ .description("This property determines whether we want to stream data to ADX.")
+ .required(false)
+ .allowableValues("true", "false")
+ .addValidator(StandardValidators.BOOLEAN_VALIDATOR)
+ .defaultValue("false")
+ .build();
+
+ public static final PropertyDescriptor POLL_ON_INGESTION_STATUS = new PropertyDescriptor
+ .Builder().name("Poll on ingestion status")
+ .displayName("Whether to Poll on ingestion status")
Review Comment:
Property names should use Title Case:
```suggestion
.Builder().name("Ingest Status Polling Enabled")
.displayName("Ingest Status Polling Enabled")
```
##########
nifi-nar-bundles/nifi-azure-bundle/nifi-azure-services-api/pom.xml:
##########
@@ -36,6 +36,16 @@
<groupId>org.apache.nifi</groupId>
<artifactId>nifi-api</artifactId>
</dependency>
+ <dependency>
+ <groupId>com.microsoft.azure.kusto</groupId>
+ <artifactId>kusto-data</artifactId>
+ <version>5.0.1</version>
+ </dependency>
+ <dependency>
+ <groupId>com.microsoft.azure.kusto</groupId>
+ <artifactId>kusto-ingest</artifactId>
+ <version>5.0.1</version>
+ </dependency>
Review Comment:
@tanmaya-panda1 The `KustResultSetTable` does not appear to be used in the `PutAzureDataExplorer` Processor, so it seems like the class reference could be removed, and perhaps replaced with a few of the relevant properties.
Although the `IngestProperties` provide a way to pass the information, this is an opportunity to extract the properties to a NiFi API class, which could be called `KustoIngestProperties`. Although this is somewhat duplicative, it would basically involve copying the properties from `KustoIngestProperties` to `IngestProperties` in the Controller Service implementation method. The reason for this approach is to avoid hard links between the NiFi Controller Service API and the particular Kusto API. Given that these are just configuration properties, maintaining the abstraction seems worthwhile as it can avoid including these new dependencies in the NiFi API.
##########
nifi-nar-bundles/nifi-azure-bundle/nifi-azure-processors/src/main/java/org/apache/nifi/processors/azure/data/explorer/PutAzureDataExplorer.java:
##########
@@ -0,0 +1,399 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.azure.data.explorer;
+
+import com.microsoft.azure.kusto.data.KustoResultSetTable;
+import com.microsoft.azure.kusto.ingest.IngestionMapping;
+import com.microsoft.azure.kusto.ingest.IngestionProperties;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.nifi.annotation.behavior.ReadsAttribute;
+import org.apache.nifi.annotation.behavior.ReadsAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.expression.ExpressionLanguageScope;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.services.azure.data.explorer.KustoIngestService;
+import org.apache.nifi.services.azure.data.explorer.KustoIngestionRequest;
+import org.apache.nifi.services.azure.data.explorer.KustoIngestionResult;
+import org.apache.nifi.services.azure.data.explorer.KustoQueryResponse;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.net.URISyntaxException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+
+@Tags({"Azure", "Kusto", "ADX", "Explorer", "Data"})
+@CapabilityDescription("The PutAzureDataExplorer acts as a ADX sink connector which sends flowFiles using the ADX-Service to the provided Azure Data" +
+ "Explorer Ingest Endpoint. The data can be sent through queued ingestion or streaming ingestion to the Azure Data Explorer cluster.")
+@ReadsAttributes({
+ @ReadsAttribute(attribute = "DB_NAME", description = "Specifies the name of the ADX database where the data needs to be stored."),
+ @ReadsAttribute(attribute = "TABLE_NAME", description = "Specifies the name of the ADX table where the data needs to be stored."),
+ @ReadsAttribute(attribute = "MAPPING_NAME", description = "Specifies the name of the mapping responsible for storing the data in appropriate columns."),
+ @ReadsAttribute(attribute = "FLUSH_IMMEDIATE", description = "In case of queued ingestion, this property determines whether the data should be flushed immediately to the ingest endpoint."),
+ @ReadsAttribute(attribute = "DATA_FORMAT", description = "Specifies the format of data that is send to Azure Data Explorer."),
+ @ReadsAttribute(attribute = "IGNORE_FIRST_RECORD", description = "Specifies whether we want to ignore ingestion of first record. " +
+ "This is primarily applicable for csv files. Default is set to NO"),
+ @ReadsAttribute(attribute = "POLL_ON_INGESTION_STATUS", description = "Specifies whether we want to poll on ingestion result during ingestion into ADX." +
+ "In case of applications that need high throughput it is recommended to keep the default value as false. Default is set to false." +
+ "This property should be set to true during Queued Ingestion for near realtime micro-batches of data that require acknowledgement of ingestion status.")
+})
+public class PutAzureDataExplorer extends AbstractProcessor {
+
+ public static final String FETCH_TABLE_COMMAND = "%s | count";
+ public static final String STREAMING_POLICY_SHOW_COMMAND = ".show %s %s policy streamingingestion";
+ public static final String DATABASE = "database";
+
+ private List<PropertyDescriptor> descriptors;
+ private Set<Relationship> relationships;
+ private transient KustoIngestService service;
+ private boolean isStreamingEnabled;
+ private boolean pollOnIngestionStatus;
+
+ public static final AllowableValue AVRO = new AllowableValue(
+ IngestionProperties.DataFormat.AVRO.getKustoValue(), IngestionProperties.DataFormat.AVRO.getKustoValue(),
+ "An Avro format with support for logical types and for the snappy compression codec");
+
+ public static final AllowableValue APACHEAVRO = new AllowableValue(
+ IngestionProperties.DataFormat.APACHEAVRO.getKustoValue(), IngestionProperties.DataFormat.APACHEAVRO.getKustoValue(),
+ "An Avro format with support for logical types and for the snappy compression codec.");
+
+ public static final AllowableValue CSV = new AllowableValue(
+ IngestionProperties.DataFormat.CSV.name(), IngestionProperties.DataFormat.CSV.getKustoValue(),
+ "A text file with comma-separated values (,). For more information, see RFC 4180: Common Format " +
+ "and MIME Type for Comma-Separated Values (CSV) Files.");
+
+ public static final AllowableValue JSON = new AllowableValue(
+ IngestionProperties.DataFormat.JSON.name(), IngestionProperties.DataFormat.JSON.getKustoValue(),
+ "A text file containing JSON objects separated by \\n or \\r\\n. For more information, " +
+ "see JSON Lines (JSONL).");
+
+ public static final AllowableValue MULTIJSON = new AllowableValue(
+ IngestionProperties.DataFormat.MULTIJSON.name(), IngestionProperties.DataFormat.MULTIJSON.getKustoValue(),
+ "A text file containing a JSON array of property containers (each representing a record) or any " +
+ "number of property containers separated by spaces, \\n or \\r\\n. Each property container may be " +
+ "spread across multiple lines. This format is preferable to JSON unless the data is not property " +
+ "containers.");
+
+ public static final AllowableValue ORC = new AllowableValue(
+ IngestionProperties.DataFormat.ORC.name(), IngestionProperties.DataFormat.ORC.getKustoValue(), "An ORC file.");
+
+ public static final AllowableValue PARQUET = new AllowableValue(
+ IngestionProperties.DataFormat.PARQUET.name(), IngestionProperties.DataFormat.PARQUET.getKustoValue(), "A parquet file.");
+
+ public static final AllowableValue PSV = new AllowableValue(
+ IngestionProperties.DataFormat.PSV.name(), IngestionProperties.DataFormat.PSV.getKustoValue(), "A text file with values separated by vertical bars (|).");
+
+ public static final AllowableValue SCSV = new AllowableValue(
+ IngestionProperties.DataFormat.SCSV.name(), IngestionProperties.DataFormat.SCSV.getKustoValue(), "A text file with values separated by semicolons (;).");
+
+ public static final AllowableValue SOHSV = new AllowableValue(
+ IngestionProperties.DataFormat.SOHSV.name(), IngestionProperties.DataFormat.SOHSV.getKustoValue(),
+ "A text file with SOH-separated values. (SOH is the ASCII code point 1. " +
+ "This format is used by Hive in HDInsight).");
+
+ public static final AllowableValue TSV = new AllowableValue(
+ IngestionProperties.DataFormat.TSV.name(), IngestionProperties.DataFormat.TSV.getKustoValue(), "A text file with tab delimited values (\\t).");
+
+ public static final AllowableValue TSVE = new AllowableValue(
+ IngestionProperties.DataFormat.TSVE.name(), IngestionProperties.DataFormat.TSVE.getKustoValue(),
+ "A text file with tab-delimited values (\\t). A backslash (\\) is used as escape character.");
+
+ public static final AllowableValue TXT = new AllowableValue(
+ IngestionProperties.DataFormat.TXT.name(), IngestionProperties.DataFormat.TXT.getKustoValue(),
+ "A text file with lines separated by \\n. Empty lines are skipped.");
+
+ public static final AllowableValue IGNORE_FIRST_RECORD_YES = new AllowableValue(
+ "YES", "YES",
+ "Ignore first record during ingestion");
+
+ public static final AllowableValue IGNORE_FIRST_RECORD_NO = new AllowableValue(
+ "NO", "NO",
+ "Do not ignore first record during ingestion");
+
+ public static final PropertyDescriptor DATABASE_NAME = new PropertyDescriptor.Builder()
+ .name("Database Name")
+ .displayName("Database Name")
+ .description("Azure Data Explorer Database Name for querying")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+ .build();
+
+ public static final PropertyDescriptor TABLE_NAME = new PropertyDescriptor.Builder()
+ .name("Table Name")
+ .displayName("Table Name")
+ .description("Azure Data Explorer Table Name for ingesting data")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+ .build();
+
+ public static final PropertyDescriptor MAPPING_NAME = new PropertyDescriptor
+ .Builder().name("Mapping name")
+ .displayName("Mapping name")
+ .description("The name of the mapping responsible for storing the data in the appropriate columns.")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+
+ public static final PropertyDescriptor IS_STREAMING_ENABLED = new PropertyDescriptor
+ .Builder().name("Is Streaming enabled")
+ .displayName("Is Streaming enabled")
+ .description("This property determines whether we want to stream data to ADX.")
+ .required(false)
+ .allowableValues("true", "false")
+ .addValidator(StandardValidators.BOOLEAN_VALIDATOR)
+ .defaultValue("false")
+ .build();
+
+ public static final PropertyDescriptor POLL_ON_INGESTION_STATUS = new PropertyDescriptor
+ .Builder().name("Poll on ingestion status")
+ .displayName("Whether to Poll on ingestion status")
+ .description("This property determines whether we want to poll on ingestion status after an ingestion to ADX is completed")
+ .required(false)
+ .allowableValues("true", "false")
+ .addValidator(StandardValidators.BOOLEAN_VALIDATOR)
+ .defaultValue("false")
+ .build();
+ public static final PropertyDescriptor ADX_SERVICE = new PropertyDescriptor
+ .Builder().name("Kusto Ingest Service")
+ .displayName("Kusto Ingest Service")
+ .description("Azure Data Explorer Kusto Ingest Service")
+ .required(true)
+ .identifiesControllerService(KustoIngestService.class)
+ .build();
+
+ public static final Relationship SUCCESS = new Relationship.Builder()
+ .name("success")
+ .description("Relationship for success")
+ .build();
+ public static final Relationship FAILURE = new Relationship.Builder()
+ .name("failure")
+ .description("Relationship for failure")
+ .build();
+
+ static final PropertyDescriptor DATA_FORMAT = new PropertyDescriptor.Builder()
+ .name("Data Format")
+ .displayName("Data Format")
+ .description("The format of the data that is sent to ADX.")
Review Comment:
```suggestion
.description("The format of the data that is sent to Azure Data Explorer.")
```
##########
nifi-nar-bundles/nifi-azure-bundle/nifi-azure-processors/src/main/java/org/apache/nifi/processors/azure/data/explorer/PutAzureDataExplorer.java:
##########
@@ -0,0 +1,399 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.azure.data.explorer;
+
+import com.microsoft.azure.kusto.data.KustoResultSetTable;
+import com.microsoft.azure.kusto.ingest.IngestionMapping;
+import com.microsoft.azure.kusto.ingest.IngestionProperties;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.nifi.annotation.behavior.ReadsAttribute;
+import org.apache.nifi.annotation.behavior.ReadsAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.expression.ExpressionLanguageScope;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.services.azure.data.explorer.KustoIngestService;
+import org.apache.nifi.services.azure.data.explorer.KustoIngestionRequest;
+import org.apache.nifi.services.azure.data.explorer.KustoIngestionResult;
+import org.apache.nifi.services.azure.data.explorer.KustoQueryResponse;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.net.URISyntaxException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+
+@Tags({"Azure", "Kusto", "ADX", "Explorer", "Data"})
+@CapabilityDescription("The PutAzureDataExplorer acts as a ADX sink connector which sends flowFiles using the ADX-Service to the provided Azure Data" +
+ "Explorer Ingest Endpoint. The data can be sent through queued ingestion or streaming ingestion to the Azure Data Explorer cluster.")
+@ReadsAttributes({
+ @ReadsAttribute(attribute = "DB_NAME", description = "Specifies the name of the ADX database where the data needs to be stored."),
+ @ReadsAttribute(attribute = "TABLE_NAME", description = "Specifies the name of the ADX table where the data needs to be stored."),
+ @ReadsAttribute(attribute = "MAPPING_NAME", description = "Specifies the name of the mapping responsible for storing the data in appropriate columns."),
+ @ReadsAttribute(attribute = "FLUSH_IMMEDIATE", description = "In case of queued ingestion, this property determines whether the data should be flushed immediately to the ingest endpoint."),
+ @ReadsAttribute(attribute = "DATA_FORMAT", description = "Specifies the format of data that is send to Azure Data Explorer."),
+ @ReadsAttribute(attribute = "IGNORE_FIRST_RECORD", description = "Specifies whether we want to ignore ingestion of first record. " +
+ "This is primarily applicable for csv files. Default is set to NO"),
+ @ReadsAttribute(attribute = "POLL_ON_INGESTION_STATUS", description = "Specifies whether we want to poll on ingestion result during ingestion into ADX." +
+ "In case of applications that need high throughput it is recommended to keep the default value as false. Default is set to false." +
+ "This property should be set to true during Queued Ingestion for near realtime micro-batches of data that require acknowledgement of ingestion status.")
+})
+public class PutAzureDataExplorer extends AbstractProcessor {
+
+ public static final String FETCH_TABLE_COMMAND = "%s | count";
+ public static final String STREAMING_POLICY_SHOW_COMMAND = ".show %s %s policy streamingingestion";
+ public static final String DATABASE = "database";
+
+ private List<PropertyDescriptor> descriptors;
+ private Set<Relationship> relationships;
+ private transient KustoIngestService service;
+ private boolean isStreamingEnabled;
+ private boolean pollOnIngestionStatus;
+
+ public static final AllowableValue AVRO = new AllowableValue(
+ IngestionProperties.DataFormat.AVRO.getKustoValue(), IngestionProperties.DataFormat.AVRO.getKustoValue(),
+ "An Avro format with support for logical types and for the snappy compression codec");
+
+ public static final AllowableValue APACHEAVRO = new AllowableValue(
+ IngestionProperties.DataFormat.APACHEAVRO.getKustoValue(), IngestionProperties.DataFormat.APACHEAVRO.getKustoValue(),
+ "An Avro format with support for logical types and for the snappy compression codec.");
+
+ public static final AllowableValue CSV = new AllowableValue(
+ IngestionProperties.DataFormat.CSV.name(), IngestionProperties.DataFormat.CSV.getKustoValue(),
+ "A text file with comma-separated values (,). For more information, see RFC 4180: Common Format " +
+ "and MIME Type for Comma-Separated Values (CSV) Files.");
+
+ public static final AllowableValue JSON = new AllowableValue(
+ IngestionProperties.DataFormat.JSON.name(), IngestionProperties.DataFormat.JSON.getKustoValue(),
+ "A text file containing JSON objects separated by \\n or \\r\\n. For more information, " +
+ "see JSON Lines (JSONL).");
+
+ public static final AllowableValue MULTIJSON = new AllowableValue(
+ IngestionProperties.DataFormat.MULTIJSON.name(), IngestionProperties.DataFormat.MULTIJSON.getKustoValue(),
+ "A text file containing a JSON array of property containers (each representing a record) or any " +
+ "number of property containers separated by spaces, \\n or \\r\\n. Each property container may be " +
+ "spread across multiple lines. This format is preferable to JSON unless the data is not property " +
+ "containers.");
+
+ public static final AllowableValue ORC = new AllowableValue(
+ IngestionProperties.DataFormat.ORC.name(), IngestionProperties.DataFormat.ORC.getKustoValue(), "An ORC file.");
+
+ public static final AllowableValue PARQUET = new AllowableValue(
+ IngestionProperties.DataFormat.PARQUET.name(), IngestionProperties.DataFormat.PARQUET.getKustoValue(), "A parquet file.");
+
+ public static final AllowableValue PSV = new AllowableValue(
+ IngestionProperties.DataFormat.PSV.name(), IngestionProperties.DataFormat.PSV.getKustoValue(), "A text file with values separated by vertical bars (|).");
+
+ public static final AllowableValue SCSV = new AllowableValue(
+ IngestionProperties.DataFormat.SCSV.name(), IngestionProperties.DataFormat.SCSV.getKustoValue(), "A text file with values separated by semicolons (;).");
+
+ public static final AllowableValue SOHSV = new AllowableValue(
+ IngestionProperties.DataFormat.SOHSV.name(), IngestionProperties.DataFormat.SOHSV.getKustoValue(),
+ "A text file with SOH-separated values. (SOH is the ASCII code point 1. " +
+ "This format is used by Hive in HDInsight).");
+
+ public static final AllowableValue TSV = new AllowableValue(
+ IngestionProperties.DataFormat.TSV.name(), IngestionProperties.DataFormat.TSV.getKustoValue(), "A text file with tab delimited values (\\t).");
+
+ public static final AllowableValue TSVE = new AllowableValue(
+ IngestionProperties.DataFormat.TSVE.name(), IngestionProperties.DataFormat.TSVE.getKustoValue(),
+ "A text file with tab-delimited values (\\t). A backslash (\\) is used as escape character.");
+
+ public static final AllowableValue TXT = new AllowableValue(
+ IngestionProperties.DataFormat.TXT.name(), IngestionProperties.DataFormat.TXT.getKustoValue(),
+ "A text file with lines separated by \\n. Empty lines are skipped.");
+
+ public static final AllowableValue IGNORE_FIRST_RECORD_YES = new AllowableValue(
+ "YES", "YES",
+ "Ignore first record during ingestion");
+
+ public static final AllowableValue IGNORE_FIRST_RECORD_NO = new AllowableValue(
+ "NO", "NO",
+ "Do not ignore first record during ingestion");
+
+ public static final PropertyDescriptor DATABASE_NAME = new PropertyDescriptor.Builder()
+ .name("Database Name")
+ .displayName("Database Name")
+ .description("Azure Data Explorer Database Name for querying")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+ .build();
+
+ public static final PropertyDescriptor TABLE_NAME = new PropertyDescriptor.Builder()
+ .name("Table Name")
+ .displayName("Table Name")
+ .description("Azure Data Explorer Table Name for ingesting data")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+ .build();
+
+ public static final PropertyDescriptor MAPPING_NAME = new PropertyDescriptor
+ .Builder().name("Mapping name")
+ .displayName("Mapping name")
+ .description("The name of the mapping responsible for storing the data in the appropriate columns.")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+
+ public static final PropertyDescriptor IS_STREAMING_ENABLED = new PropertyDescriptor
+ .Builder().name("Is Streaming enabled")
+ .displayName("Is Streaming enabled")
Review Comment:
```suggestion
.Builder().name("Streaming Enabled")
.displayName("Streaming Enabled")
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] tanmaya-panda1 commented on a diff in pull request #7624: NIFI-11958 Azure Data Explorer as sink
Posted by "tanmaya-panda1 (via GitHub)" <gi...@apache.org>.
tanmaya-panda1 commented on code in PR #7624:
URL: https://github.com/apache/nifi/pull/7624#discussion_r1336374183
##########
nifi-nar-bundles/nifi-azure-bundle/nifi-azure-processors/src/main/java/org/apache/nifi/services/azure/data/explorer/StandardKustoQueryService.java:
##########
@@ -159,10 +162,7 @@ private ConnectionStringBuilder getConnectionStringBuilder(final ConfigurationCo
builder = ConnectionStringBuilder.createWithAadApplicationCredentials(clusterUrl, clientId, applicationKey, tenantId);
}
- final String vendor = System.getProperty("java.vendor");
- final String version = System.getProperty("java.version");
-
- builder.setConnectorDetails(vendor, version, null, null, false, null);
+ builder.setConnectorDetails("Kusto.Nifi.Source", StandardKustoQueryService.class.getPackage().getImplementationVersion(), null, null, false, null, NIFI_SOURCE);
Review Comment:
Yes this change is necessary since connector details/identifier along with version information is required for identification of request in ADX(Azure Data Explorer) cluster for debugging and tracking purpose. This is a standard across all the connectors we have implemented for Azure Data Explorer.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] NIFI-11958 Azure Data Explorer as sink [nifi]
Posted by "exceptionfactory (via GitHub)" <gi...@apache.org>.
exceptionfactory commented on code in PR #7624:
URL: https://github.com/apache/nifi/pull/7624#discussion_r1377749253
##########
nifi-nar-bundles/nifi-azure-bundle/nifi-azure-processors/src/main/java/org/apache/nifi/processors/azure/data/explorer/PutAzureDataExplorer.java:
##########
@@ -0,0 +1,309 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.azure.data.explorer;
+
+import org.apache.commons.lang3.StringUtils;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.expression.ExpressionLanguageScope;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.services.azure.data.explorer.KustoIngestDataFormat;
+import org.apache.nifi.services.azure.data.explorer.KustoIngestService;
+import org.apache.nifi.services.azure.data.explorer.KustoIngestionRequest;
+import org.apache.nifi.services.azure.data.explorer.KustoIngestionResult;
+import org.apache.nifi.services.azure.data.explorer.KustoQueryResponse;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.net.URISyntaxException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+
Review Comment:
The `InputRequirement` annotation should be added with `Requirement.INPUT_REQUIRED` specified.
##########
nifi-nar-bundles/nifi-azure-bundle/nifi-azure-processors/src/main/java/org/apache/nifi/processors/azure/data/explorer/PutAzureDataExplorer.java:
##########
@@ -0,0 +1,309 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.azure.data.explorer;
+
+import org.apache.commons.lang3.StringUtils;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.expression.ExpressionLanguageScope;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.services.azure.data.explorer.KustoIngestDataFormat;
+import org.apache.nifi.services.azure.data.explorer.KustoIngestService;
+import org.apache.nifi.services.azure.data.explorer.KustoIngestionRequest;
+import org.apache.nifi.services.azure.data.explorer.KustoIngestionResult;
+import org.apache.nifi.services.azure.data.explorer.KustoQueryResponse;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.net.URISyntaxException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+
+@Tags({"Azure", "Kusto", "ADX", "Explorer", "Data"})
+@CapabilityDescription("The PutAzureDataExplorer acts as a ADX sink connector which sends flowFiles using the ADX-Service to the provided Azure Data" +
+ "Explorer Ingest Endpoint. The data can be sent through queued ingestion or streaming ingestion to the Azure Data Explorer cluster.")
+public class PutAzureDataExplorer extends AbstractProcessor {
+
+ public static final String FETCH_TABLE_COMMAND = "%s | count";
+ public static final String STREAMING_POLICY_SHOW_COMMAND = ".show %s %s policy streamingingestion";
+ public static final String DATABASE = "database";
+
+ private List<PropertyDescriptor> descriptors;
+ private Set<Relationship> relationships;
+ private transient KustoIngestService service;
+ private boolean streamingEnabled;
+ private boolean pollOnIngestionStatus;
Review Comment:
These member variables should be moved down, following after `static` values, to maintain common conventions.
##########
nifi-nar-bundles/nifi-azure-bundle/nifi-azure-processors/src/main/java/org/apache/nifi/processors/azure/data/explorer/PutAzureDataExplorer.java:
##########
@@ -0,0 +1,309 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.azure.data.explorer;
+
+import org.apache.commons.lang3.StringUtils;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.expression.ExpressionLanguageScope;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.services.azure.data.explorer.KustoIngestDataFormat;
+import org.apache.nifi.services.azure.data.explorer.KustoIngestService;
+import org.apache.nifi.services.azure.data.explorer.KustoIngestionRequest;
+import org.apache.nifi.services.azure.data.explorer.KustoIngestionResult;
+import org.apache.nifi.services.azure.data.explorer.KustoQueryResponse;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.net.URISyntaxException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+
+@Tags({"Azure", "Kusto", "ADX", "Explorer", "Data"})
+@CapabilityDescription("The PutAzureDataExplorer acts as a ADX sink connector which sends flowFiles using the ADX-Service to the provided Azure Data" +
+ "Explorer Ingest Endpoint. The data can be sent through queued ingestion or streaming ingestion to the Azure Data Explorer cluster.")
+public class PutAzureDataExplorer extends AbstractProcessor {
+
+ public static final String FETCH_TABLE_COMMAND = "%s | count";
+ public static final String STREAMING_POLICY_SHOW_COMMAND = ".show %s %s policy streamingingestion";
+ public static final String DATABASE = "database";
+
+ private List<PropertyDescriptor> descriptors;
+ private Set<Relationship> relationships;
+ private transient KustoIngestService service;
+ private boolean streamingEnabled;
+ private boolean pollOnIngestionStatus;
+
+ public static final AllowableValue IGNORE_FIRST_RECORD_YES = new AllowableValue(
+ "YES", "YES",
+ "Ignore first record during ingestion");
+
+ public static final AllowableValue IGNORE_FIRST_RECORD_NO = new AllowableValue(
+ "NO", "NO",
+ "Do not ignore first record during ingestion");
+
+ public static final PropertyDescriptor DATABASE_NAME = new PropertyDescriptor.Builder()
+ .name("Database Name")
+ .displayName("Database Name")
+ .description("Azure Data Explorer Database Name for querying")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+ .build();
+
+ public static final PropertyDescriptor TABLE_NAME = new PropertyDescriptor.Builder()
+ .name("Table Name")
+ .displayName("Table Name")
+ .description("Azure Data Explorer Table Name for ingesting data")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+ .build();
+
+ public static final PropertyDescriptor MAPPING_NAME = new PropertyDescriptor
+ .Builder().name("Ingest Mapping name")
+ .displayName("Ingest Mapping name")
+ .description("The name of the mapping responsible for storing the data in the appropriate columns.")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+
+ public static final PropertyDescriptor IS_STREAMING_ENABLED = new PropertyDescriptor
+ .Builder().name("Streaming enabled")
+ .displayName("Streaming enabled")
Review Comment:
```suggestion
.Builder().name("Streaming Enabled")
.displayName("Streaming Enabled")
```
##########
nifi-nar-bundles/nifi-azure-bundle/nifi-azure-processors/src/main/java/org/apache/nifi/processors/azure/data/explorer/PutAzureDataExplorer.java:
##########
@@ -0,0 +1,309 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.azure.data.explorer;
+
+import org.apache.commons.lang3.StringUtils;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.expression.ExpressionLanguageScope;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.services.azure.data.explorer.KustoIngestDataFormat;
+import org.apache.nifi.services.azure.data.explorer.KustoIngestService;
+import org.apache.nifi.services.azure.data.explorer.KustoIngestionRequest;
+import org.apache.nifi.services.azure.data.explorer.KustoIngestionResult;
+import org.apache.nifi.services.azure.data.explorer.KustoQueryResponse;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.net.URISyntaxException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+
+@Tags({"Azure", "Kusto", "ADX", "Explorer", "Data"})
+@CapabilityDescription("The PutAzureDataExplorer acts as a ADX sink connector which sends flowFiles using the ADX-Service to the provided Azure Data" +
+ "Explorer Ingest Endpoint. The data can be sent through queued ingestion or streaming ingestion to the Azure Data Explorer cluster.")
+public class PutAzureDataExplorer extends AbstractProcessor {
+
+ public static final String FETCH_TABLE_COMMAND = "%s | count";
+ public static final String STREAMING_POLICY_SHOW_COMMAND = ".show %s %s policy streamingingestion";
+ public static final String DATABASE = "database";
+
+ private List<PropertyDescriptor> descriptors;
+ private Set<Relationship> relationships;
+ private transient KustoIngestService service;
+ private boolean streamingEnabled;
+ private boolean pollOnIngestionStatus;
+
+ public static final AllowableValue IGNORE_FIRST_RECORD_YES = new AllowableValue(
+ "YES", "YES",
+ "Ignore first record during ingestion");
+
+ public static final AllowableValue IGNORE_FIRST_RECORD_NO = new AllowableValue(
+ "NO", "NO",
+ "Do not ignore first record during ingestion");
+
+ public static final PropertyDescriptor DATABASE_NAME = new PropertyDescriptor.Builder()
+ .name("Database Name")
+ .displayName("Database Name")
+ .description("Azure Data Explorer Database Name for querying")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+ .build();
+
+ public static final PropertyDescriptor TABLE_NAME = new PropertyDescriptor.Builder()
+ .name("Table Name")
+ .displayName("Table Name")
+ .description("Azure Data Explorer Table Name for ingesting data")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+ .build();
+
+ public static final PropertyDescriptor MAPPING_NAME = new PropertyDescriptor
+ .Builder().name("Ingest Mapping name")
+ .displayName("Ingest Mapping name")
+ .description("The name of the mapping responsible for storing the data in the appropriate columns.")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+
+ public static final PropertyDescriptor IS_STREAMING_ENABLED = new PropertyDescriptor
+ .Builder().name("Streaming enabled")
+ .displayName("Streaming enabled")
+ .description("This property determines whether we want to stream data to ADX.")
+ .required(false)
+ .allowableValues("true", "false")
+ .addValidator(StandardValidators.BOOLEAN_VALIDATOR)
+ .defaultValue("false")
+ .build();
+
+ public static final PropertyDescriptor POLL_ON_INGESTION_STATUS = new PropertyDescriptor
+ .Builder().name("Poll on ingestion status")
+ .displayName("Whether to Poll on ingestion status")
+ .description("This property determines whether we want to poll on ingestion status after an ingestion to ADX is completed")
+ .required(false)
+ .allowableValues("true", "false")
+ .addValidator(StandardValidators.BOOLEAN_VALIDATOR)
+ .defaultValue("false")
+ .build();
+ public static final PropertyDescriptor ADX_SERVICE = new PropertyDescriptor
+ .Builder().name("Kusto Ingest Service")
+ .displayName("Kusto Ingest Service")
+ .description("Azure Data Explorer Kusto Ingest Service")
+ .required(true)
+ .identifiesControllerService(KustoIngestService.class)
+ .build();
+
+ static final PropertyDescriptor DATA_FORMAT = new PropertyDescriptor.Builder()
+ .name("Data Format")
+ .displayName("Data Format")
+ .description("The format of the data that is sent to Azure Data Explorer.")
+ .required(true)
+ .allowableValues(KustoIngestDataFormat.values())
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+
+ static final PropertyDescriptor IGNORE_FIRST_RECORD = new PropertyDescriptor.Builder()
+ .name("Ingestion Ignore First Record")
+ .displayName("Ingestion Ignore First Record")
+ .description("Defines whether ignore first record while ingestion.")
+ .required(false)
+ .allowableValues(IGNORE_FIRST_RECORD_YES, IGNORE_FIRST_RECORD_NO)
+ .defaultValue(IGNORE_FIRST_RECORD_NO.getValue())
+ .build();
+
+ static final PropertyDescriptor ROUTE_PARTIALLY_SUCCESSFUL_INGESTION = new PropertyDescriptor.Builder()
+ .name("Route partially successful ingestion records")
+ .displayName("Route partially successful ingestion records")
+ .description("Defines where to route partially successful ingestion records.")
+ .required(false)
+ .allowableValues("Success", "Failure")
+ .defaultValue("Failure")
+ .build();
+
+ static final PropertyDescriptor INGESTION_STATUS_POLLING_TIMEOUT = new PropertyDescriptor.Builder()
+ .name("Timeout for polling on ingestion status")
+ .displayName("Timeout for polling on ingestion status in seconds")
+ .description("Defines the value of timeout for polling on ingestion status in seconds")
+ .required(false)
+ .dependsOn(POLL_ON_INGESTION_STATUS, "true")
+ .addValidator(StandardValidators.POSITIVE_INTEGER_VALIDATOR)
+ .defaultValue("600")
+ .build();
+
+ static final PropertyDescriptor INGESTION_STATUS_POLLING_INTERVAL = new PropertyDescriptor.Builder()
+ .name("Ingestion status polling interval")
+ .displayName("Ingestion status polling interval in seconds")
Review Comment:
```suggestion
.name("Ingest Status Polling Interval")
.displayName("Ingest Status Polling Interval")
```
##########
nifi-nar-bundles/nifi-azure-bundle/nifi-azure-processors/src/main/java/org/apache/nifi/processors/azure/data/explorer/PutAzureDataExplorer.java:
##########
@@ -0,0 +1,309 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.azure.data.explorer;
+
+import org.apache.commons.lang3.StringUtils;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.expression.ExpressionLanguageScope;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.services.azure.data.explorer.KustoIngestDataFormat;
+import org.apache.nifi.services.azure.data.explorer.KustoIngestService;
+import org.apache.nifi.services.azure.data.explorer.KustoIngestionRequest;
+import org.apache.nifi.services.azure.data.explorer.KustoIngestionResult;
+import org.apache.nifi.services.azure.data.explorer.KustoQueryResponse;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.net.URISyntaxException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+
+@Tags({"Azure", "Kusto", "ADX", "Explorer", "Data"})
+@CapabilityDescription("The PutAzureDataExplorer acts as a ADX sink connector which sends flowFiles using the ADX-Service to the provided Azure Data" +
+ "Explorer Ingest Endpoint. The data can be sent through queued ingestion or streaming ingestion to the Azure Data Explorer cluster.")
+public class PutAzureDataExplorer extends AbstractProcessor {
+
+ public static final String FETCH_TABLE_COMMAND = "%s | count";
+ public static final String STREAMING_POLICY_SHOW_COMMAND = ".show %s %s policy streamingingestion";
+ public static final String DATABASE = "database";
+
+ private List<PropertyDescriptor> descriptors;
+ private Set<Relationship> relationships;
+ private transient KustoIngestService service;
+ private boolean streamingEnabled;
+ private boolean pollOnIngestionStatus;
+
+ public static final AllowableValue IGNORE_FIRST_RECORD_YES = new AllowableValue(
+ "YES", "YES",
+ "Ignore first record during ingestion");
+
+ public static final AllowableValue IGNORE_FIRST_RECORD_NO = new AllowableValue(
+ "NO", "NO",
+ "Do not ignore first record during ingestion");
+
+ public static final PropertyDescriptor DATABASE_NAME = new PropertyDescriptor.Builder()
+ .name("Database Name")
+ .displayName("Database Name")
+ .description("Azure Data Explorer Database Name for querying")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+ .build();
+
+ public static final PropertyDescriptor TABLE_NAME = new PropertyDescriptor.Builder()
+ .name("Table Name")
+ .displayName("Table Name")
+ .description("Azure Data Explorer Table Name for ingesting data")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+ .build();
+
+ public static final PropertyDescriptor MAPPING_NAME = new PropertyDescriptor
+ .Builder().name("Ingest Mapping name")
+ .displayName("Ingest Mapping name")
+ .description("The name of the mapping responsible for storing the data in the appropriate columns.")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+
+ public static final PropertyDescriptor IS_STREAMING_ENABLED = new PropertyDescriptor
+ .Builder().name("Streaming enabled")
+ .displayName("Streaming enabled")
+ .description("This property determines whether we want to stream data to ADX.")
+ .required(false)
+ .allowableValues("true", "false")
+ .addValidator(StandardValidators.BOOLEAN_VALIDATOR)
+ .defaultValue("false")
+ .build();
+
+ public static final PropertyDescriptor POLL_ON_INGESTION_STATUS = new PropertyDescriptor
+ .Builder().name("Poll on ingestion status")
+ .displayName("Whether to Poll on ingestion status")
+ .description("This property determines whether we want to poll on ingestion status after an ingestion to ADX is completed")
+ .required(false)
+ .allowableValues("true", "false")
+ .addValidator(StandardValidators.BOOLEAN_VALIDATOR)
+ .defaultValue("false")
+ .build();
+ public static final PropertyDescriptor ADX_SERVICE = new PropertyDescriptor
+ .Builder().name("Kusto Ingest Service")
+ .displayName("Kusto Ingest Service")
+ .description("Azure Data Explorer Kusto Ingest Service")
+ .required(true)
+ .identifiesControllerService(KustoIngestService.class)
+ .build();
+
+ static final PropertyDescriptor DATA_FORMAT = new PropertyDescriptor.Builder()
+ .name("Data Format")
+ .displayName("Data Format")
+ .description("The format of the data that is sent to Azure Data Explorer.")
+ .required(true)
+ .allowableValues(KustoIngestDataFormat.values())
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+
+ static final PropertyDescriptor IGNORE_FIRST_RECORD = new PropertyDescriptor.Builder()
+ .name("Ingestion Ignore First Record")
+ .displayName("Ingestion Ignore First Record")
+ .description("Defines whether ignore first record while ingestion.")
+ .required(false)
+ .allowableValues(IGNORE_FIRST_RECORD_YES, IGNORE_FIRST_RECORD_NO)
+ .defaultValue(IGNORE_FIRST_RECORD_NO.getValue())
+ .build();
+
+ static final PropertyDescriptor ROUTE_PARTIALLY_SUCCESSFUL_INGESTION = new PropertyDescriptor.Builder()
+ .name("Route partially successful ingestion records")
+ .displayName("Route partially successful ingestion records")
+ .description("Defines where to route partially successful ingestion records.")
+ .required(false)
+ .allowableValues("Success", "Failure")
+ .defaultValue("Failure")
+ .build();
+
+ static final PropertyDescriptor INGESTION_STATUS_POLLING_TIMEOUT = new PropertyDescriptor.Builder()
+ .name("Timeout for polling on ingestion status")
+ .displayName("Timeout for polling on ingestion status in seconds")
+ .description("Defines the value of timeout for polling on ingestion status in seconds")
+ .required(false)
+ .dependsOn(POLL_ON_INGESTION_STATUS, "true")
+ .addValidator(StandardValidators.POSITIVE_INTEGER_VALIDATOR)
+ .defaultValue("600")
+ .build();
+
+ static final PropertyDescriptor INGESTION_STATUS_POLLING_INTERVAL = new PropertyDescriptor.Builder()
+ .name("Ingestion status polling interval")
+ .displayName("Ingestion status polling interval in seconds")
+ .description("Defines the value of timeout for polling on ingestion status in seconds.")
+ .required(false)
+ .dependsOn(POLL_ON_INGESTION_STATUS, "true")
+ .addValidator(StandardValidators.POSITIVE_INTEGER_VALIDATOR)
+ .defaultValue("5")
Review Comment:
See above for recommendation on using the Time Period validator.
```suggestion
.addValidator(StandardValidators.TIME_PERIOD_VALIDATOR)
.defaultValue("5 s")
```
##########
nifi-nar-bundles/nifi-azure-bundle/nifi-azure-processors/src/main/java/org/apache/nifi/processors/azure/data/explorer/PutAzureDataExplorer.java:
##########
@@ -0,0 +1,309 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.azure.data.explorer;
+
+import org.apache.commons.lang3.StringUtils;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.expression.ExpressionLanguageScope;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.services.azure.data.explorer.KustoIngestDataFormat;
+import org.apache.nifi.services.azure.data.explorer.KustoIngestService;
+import org.apache.nifi.services.azure.data.explorer.KustoIngestionRequest;
+import org.apache.nifi.services.azure.data.explorer.KustoIngestionResult;
+import org.apache.nifi.services.azure.data.explorer.KustoQueryResponse;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.net.URISyntaxException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+
+@Tags({"Azure", "Kusto", "ADX", "Explorer", "Data"})
+@CapabilityDescription("The PutAzureDataExplorer acts as a ADX sink connector which sends flowFiles using the ADX-Service to the provided Azure Data" +
+ "Explorer Ingest Endpoint. The data can be sent through queued ingestion or streaming ingestion to the Azure Data Explorer cluster.")
+public class PutAzureDataExplorer extends AbstractProcessor {
+
+ public static final String FETCH_TABLE_COMMAND = "%s | count";
+ public static final String STREAMING_POLICY_SHOW_COMMAND = ".show %s %s policy streamingingestion";
+ public static final String DATABASE = "database";
+
+ private List<PropertyDescriptor> descriptors;
+ private Set<Relationship> relationships;
+ private transient KustoIngestService service;
+ private boolean streamingEnabled;
+ private boolean pollOnIngestionStatus;
+
+ public static final AllowableValue IGNORE_FIRST_RECORD_YES = new AllowableValue(
+ "YES", "YES",
+ "Ignore first record during ingestion");
+
+ public static final AllowableValue IGNORE_FIRST_RECORD_NO = new AllowableValue(
+ "NO", "NO",
+ "Do not ignore first record during ingestion");
+
+ public static final PropertyDescriptor DATABASE_NAME = new PropertyDescriptor.Builder()
+ .name("Database Name")
+ .displayName("Database Name")
+ .description("Azure Data Explorer Database Name for querying")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+ .build();
+
+ public static final PropertyDescriptor TABLE_NAME = new PropertyDescriptor.Builder()
+ .name("Table Name")
+ .displayName("Table Name")
+ .description("Azure Data Explorer Table Name for ingesting data")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+ .build();
+
+ public static final PropertyDescriptor MAPPING_NAME = new PropertyDescriptor
+ .Builder().name("Ingest Mapping name")
+ .displayName("Ingest Mapping name")
+ .description("The name of the mapping responsible for storing the data in the appropriate columns.")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+
+ public static final PropertyDescriptor IS_STREAMING_ENABLED = new PropertyDescriptor
+ .Builder().name("Streaming enabled")
+ .displayName("Streaming enabled")
+ .description("This property determines whether we want to stream data to ADX.")
+ .required(false)
+ .allowableValues("true", "false")
+ .addValidator(StandardValidators.BOOLEAN_VALIDATOR)
+ .defaultValue("false")
+ .build();
+
+ public static final PropertyDescriptor POLL_ON_INGESTION_STATUS = new PropertyDescriptor
+ .Builder().name("Poll on ingestion status")
+ .displayName("Whether to Poll on ingestion status")
+ .description("This property determines whether we want to poll on ingestion status after an ingestion to ADX is completed")
+ .required(false)
+ .allowableValues("true", "false")
+ .addValidator(StandardValidators.BOOLEAN_VALIDATOR)
+ .defaultValue("false")
+ .build();
+ public static final PropertyDescriptor ADX_SERVICE = new PropertyDescriptor
+ .Builder().name("Kusto Ingest Service")
+ .displayName("Kusto Ingest Service")
+ .description("Azure Data Explorer Kusto Ingest Service")
+ .required(true)
+ .identifiesControllerService(KustoIngestService.class)
+ .build();
+
+ static final PropertyDescriptor DATA_FORMAT = new PropertyDescriptor.Builder()
+ .name("Data Format")
+ .displayName("Data Format")
+ .description("The format of the data that is sent to Azure Data Explorer.")
+ .required(true)
+ .allowableValues(KustoIngestDataFormat.values())
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+
+ static final PropertyDescriptor IGNORE_FIRST_RECORD = new PropertyDescriptor.Builder()
+ .name("Ingestion Ignore First Record")
+ .displayName("Ingestion Ignore First Record")
+ .description("Defines whether ignore first record while ingestion.")
+ .required(false)
+ .allowableValues(IGNORE_FIRST_RECORD_YES, IGNORE_FIRST_RECORD_NO)
+ .defaultValue(IGNORE_FIRST_RECORD_NO.getValue())
+ .build();
+
+ static final PropertyDescriptor ROUTE_PARTIALLY_SUCCESSFUL_INGESTION = new PropertyDescriptor.Builder()
+ .name("Route partially successful ingestion records")
+ .displayName("Route partially successful ingestion records")
+ .description("Defines where to route partially successful ingestion records.")
+ .required(false)
+ .allowableValues("Success", "Failure")
+ .defaultValue("Failure")
+ .build();
+
+ static final PropertyDescriptor INGESTION_STATUS_POLLING_TIMEOUT = new PropertyDescriptor.Builder()
+ .name("Timeout for polling on ingestion status")
+ .displayName("Timeout for polling on ingestion status in seconds")
+ .description("Defines the value of timeout for polling on ingestion status in seconds")
+ .required(false)
+ .dependsOn(POLL_ON_INGESTION_STATUS, "true")
+ .addValidator(StandardValidators.POSITIVE_INTEGER_VALIDATOR)
+ .defaultValue("600")
+ .build();
+
+ static final PropertyDescriptor INGESTION_STATUS_POLLING_INTERVAL = new PropertyDescriptor.Builder()
+ .name("Ingestion status polling interval")
+ .displayName("Ingestion status polling interval in seconds")
+ .description("Defines the value of timeout for polling on ingestion status in seconds.")
+ .required(false)
+ .dependsOn(POLL_ON_INGESTION_STATUS, "true")
+ .addValidator(StandardValidators.POSITIVE_INTEGER_VALIDATOR)
+ .defaultValue("5")
+ .build();
+
+ public static final Relationship SUCCESS = new Relationship.Builder()
+ .name("success")
+ .description("Relationship for success")
+ .build();
+ public static final Relationship FAILURE = new Relationship.Builder()
+ .name("failure")
+ .description("Relationship for failure")
+ .build();
+
+ @Override
+ protected void init(final ProcessorInitializationContext context) {
+ final List<PropertyDescriptor> descriptorList = new ArrayList<>();
+ descriptorList.add(ADX_SERVICE);
+ descriptorList.add(DATABASE_NAME);
+ descriptorList.add(TABLE_NAME);
+ descriptorList.add(MAPPING_NAME);
+ descriptorList.add(DATA_FORMAT);
+ descriptorList.add(IGNORE_FIRST_RECORD);
+ descriptorList.add(IS_STREAMING_ENABLED);
+ descriptorList.add(POLL_ON_INGESTION_STATUS);
+ descriptorList.add(ROUTE_PARTIALLY_SUCCESSFUL_INGESTION);
+ descriptorList.add(INGESTION_STATUS_POLLING_TIMEOUT);
+ descriptorList.add(INGESTION_STATUS_POLLING_INTERVAL);
+ this.descriptors = Collections.unmodifiableList(descriptorList);
+
+ final Set<Relationship> relationshipSet = new HashSet<>();
+ relationshipSet.add(SUCCESS);
+ relationshipSet.add(FAILURE);
+ this.relationships = Collections.unmodifiableSet(relationshipSet);
+ }
+
+ @Override
+ public Set<Relationship> getRelationships() {
+ return this.relationships;
+ }
+
+ @Override
+ public final List<PropertyDescriptor> getSupportedPropertyDescriptors() {
+ return descriptors;
+ }
+
+ @OnScheduled
+ public void onScheduled(final ProcessContext context) {
+ service = context.getProperty(ADX_SERVICE).asControllerService(KustoIngestService.class);
+ if (checkIfIngestorRoleDoesntExist(context.getProperty(DATABASE_NAME).getValue(), context.getProperty(TABLE_NAME).getValue())) {
+ getLogger().error("User might not have ingestor privileges, table validation will be skipped for all table mappings.");
+ throw new ProcessException("User might not have ingestor privileges, table validation will be skipped for all table mappings. ");
+ }
+ streamingEnabled = context.getProperty(IS_STREAMING_ENABLED).evaluateAttributeExpressions().asBoolean();
+ if (streamingEnabled && !checkIfStreamingPolicyIsEnabledInADX(context.getProperty(DATABASE_NAME).getValue(), context.getProperty(DATABASE_NAME).getValue())) {
+ getLogger().error("Streaming policy is not enabled in database {}", context.getProperty(DATABASE_NAME).getValue());
+ throw new ProcessException("Streaming policy is not enabled in database {}" + context.getProperty(DATABASE_NAME).getValue());
+ }
+ }
+
+ @Override
+ public void onTrigger(final ProcessContext context, final ProcessSession session) throws ProcessException {
+ FlowFile flowFile = session.get();
+ if (flowFile == null) {
+ return;
+ }
+
+ String databaseName = context.getProperty(DATABASE_NAME).getValue();
+ String tableName = context.getProperty(TABLE_NAME).getValue();
+ String dataFormat = context.getProperty(DATA_FORMAT).getValue();
+ String mappingName = context.getProperty(MAPPING_NAME).getValue();
+ String routePartiallySuccessfulIngestion = context.getProperty(ROUTE_PARTIALLY_SUCCESSFUL_INGESTION).getValue();
+ String ingestionStatusPollingTimeout = context.getProperty(INGESTION_STATUS_POLLING_TIMEOUT).getValue();
+ String ingestionStatusPollingInterval = context.getProperty(INGESTION_STATUS_POLLING_INTERVAL).getValue();
+ String ignoreFirstRecord = context.getProperty(IGNORE_FIRST_RECORD).getValue();
+ streamingEnabled = context.getProperty(IS_STREAMING_ENABLED).evaluateAttributeExpressions().asBoolean();
+ pollOnIngestionStatus = context.getProperty(POLL_ON_INGESTION_STATUS).evaluateAttributeExpressions().asBoolean();
+
+ boolean isError = false;
+
+ // ingestion into ADX
+ try (final InputStream inputStream = session.read(flowFile)) {
+ StringBuilder ingestLogString = new StringBuilder().append("Ingesting with: ")
+ .append("dataFormat - ").append(dataFormat).append("|")
+ .append("ingestionMappingName - ").append(mappingName).append("|")
+ .append("databaseName - ").append(databaseName).append("|")
+ .append("tableName - ").append(tableName).append("|")
+ .append("pollOnIngestionStatus - ").append(pollOnIngestionStatus).append("|")
+ .append("ingestionStatusPollingTimeout - ").append(ingestionStatusPollingTimeout).append("|")
+ .append("ingestionStatusPollingInterval - ").append(ingestionStatusPollingInterval).append("|")
+ .append("routePartiallySuccessfulIngestion - ").append(routePartiallySuccessfulIngestion);
+ getLogger().debug(ingestLogString.toString());
+
+ KustoIngestionResult result = service.ingestData(
+ new KustoIngestionRequest(streamingEnabled, pollOnIngestionStatus, inputStream, databaseName,
+ tableName, dataFormat, mappingName, ignoreFirstRecord, ingestionStatusPollingTimeout, ingestionStatusPollingInterval)
+ );
+ getLogger().info("Ingest Status Polling Enabled {} and Ingest Status Polling Enabled {} ", pollOnIngestionStatus, result.toString());
+
+ if (result == KustoIngestionResult.SUCCEEDED) {
+ getLogger().info("Ingest Completed - {}", result.getStatus());
+ } else if (result == KustoIngestionResult.FAILED) {
+ getLogger().error("Ingest Failed - {}", result.getStatus());
+ isError = true;
+ } else if (result == KustoIngestionResult.PARTIALLY_SUCCEEDED) {
+ getLogger().warn("Ingest Partially succeeded - {}", result.getStatus());
+ flowFile = session.putAttribute(flowFile, "ingestion_status", "partial_success");
+ if (StringUtils.equalsIgnoreCase(routePartiallySuccessfulIngestion, "Failure")) {
+ isError = true;
+ }
+ }
+
+ } catch (IOException | URISyntaxException e) {
+ getLogger().error("Azure Data Explorer Ingest processing failed", e);
+ isError = true;
Review Comment:
The `isError` variable is not very clear. Instead, recommend naming it `transferRelationship` and setting the appropriate relationship value.
##########
nifi-nar-bundles/nifi-azure-bundle/nifi-azure-processors/src/main/java/org/apache/nifi/processors/azure/data/explorer/PutAzureDataExplorer.java:
##########
@@ -0,0 +1,309 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.azure.data.explorer;
+
+import org.apache.commons.lang3.StringUtils;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.expression.ExpressionLanguageScope;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.services.azure.data.explorer.KustoIngestDataFormat;
+import org.apache.nifi.services.azure.data.explorer.KustoIngestService;
+import org.apache.nifi.services.azure.data.explorer.KustoIngestionRequest;
+import org.apache.nifi.services.azure.data.explorer.KustoIngestionResult;
+import org.apache.nifi.services.azure.data.explorer.KustoQueryResponse;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.net.URISyntaxException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+
+@Tags({"Azure", "Kusto", "ADX", "Explorer", "Data"})
+@CapabilityDescription("The PutAzureDataExplorer acts as a ADX sink connector which sends flowFiles using the ADX-Service to the provided Azure Data" +
+ "Explorer Ingest Endpoint. The data can be sent through queued ingestion or streaming ingestion to the Azure Data Explorer cluster.")
+public class PutAzureDataExplorer extends AbstractProcessor {
+
+ public static final String FETCH_TABLE_COMMAND = "%s | count";
+ public static final String STREAMING_POLICY_SHOW_COMMAND = ".show %s %s policy streamingingestion";
+ public static final String DATABASE = "database";
+
+ private List<PropertyDescriptor> descriptors;
+ private Set<Relationship> relationships;
+ private transient KustoIngestService service;
+ private boolean streamingEnabled;
+ private boolean pollOnIngestionStatus;
+
+ public static final AllowableValue IGNORE_FIRST_RECORD_YES = new AllowableValue(
+ "YES", "YES",
+ "Ignore first record during ingestion");
+
+ public static final AllowableValue IGNORE_FIRST_RECORD_NO = new AllowableValue(
+ "NO", "NO",
+ "Do not ignore first record during ingestion");
+
+ public static final PropertyDescriptor DATABASE_NAME = new PropertyDescriptor.Builder()
+ .name("Database Name")
+ .displayName("Database Name")
+ .description("Azure Data Explorer Database Name for querying")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+ .build();
+
+ public static final PropertyDescriptor TABLE_NAME = new PropertyDescriptor.Builder()
+ .name("Table Name")
+ .displayName("Table Name")
+ .description("Azure Data Explorer Table Name for ingesting data")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+ .build();
+
+ public static final PropertyDescriptor MAPPING_NAME = new PropertyDescriptor
+ .Builder().name("Ingest Mapping name")
+ .displayName("Ingest Mapping name")
+ .description("The name of the mapping responsible for storing the data in the appropriate columns.")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+
+ public static final PropertyDescriptor IS_STREAMING_ENABLED = new PropertyDescriptor
+ .Builder().name("Streaming enabled")
+ .displayName("Streaming enabled")
+ .description("This property determines whether we want to stream data to ADX.")
+ .required(false)
+ .allowableValues("true", "false")
+ .addValidator(StandardValidators.BOOLEAN_VALIDATOR)
+ .defaultValue("false")
+ .build();
+
+ public static final PropertyDescriptor POLL_ON_INGESTION_STATUS = new PropertyDescriptor
+ .Builder().name("Poll on ingestion status")
+ .displayName("Whether to Poll on ingestion status")
+ .description("This property determines whether we want to poll on ingestion status after an ingestion to ADX is completed")
+ .required(false)
+ .allowableValues("true", "false")
+ .addValidator(StandardValidators.BOOLEAN_VALIDATOR)
+ .defaultValue("false")
+ .build();
+ public static final PropertyDescriptor ADX_SERVICE = new PropertyDescriptor
+ .Builder().name("Kusto Ingest Service")
+ .displayName("Kusto Ingest Service")
+ .description("Azure Data Explorer Kusto Ingest Service")
+ .required(true)
+ .identifiesControllerService(KustoIngestService.class)
+ .build();
+
+ static final PropertyDescriptor DATA_FORMAT = new PropertyDescriptor.Builder()
+ .name("Data Format")
+ .displayName("Data Format")
+ .description("The format of the data that is sent to Azure Data Explorer.")
+ .required(true)
+ .allowableValues(KustoIngestDataFormat.values())
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+
+ static final PropertyDescriptor IGNORE_FIRST_RECORD = new PropertyDescriptor.Builder()
+ .name("Ingestion Ignore First Record")
+ .displayName("Ingestion Ignore First Record")
+ .description("Defines whether ignore first record while ingestion.")
+ .required(false)
+ .allowableValues(IGNORE_FIRST_RECORD_YES, IGNORE_FIRST_RECORD_NO)
+ .defaultValue(IGNORE_FIRST_RECORD_NO.getValue())
+ .build();
+
+ static final PropertyDescriptor ROUTE_PARTIALLY_SUCCESSFUL_INGESTION = new PropertyDescriptor.Builder()
+ .name("Route partially successful ingestion records")
+ .displayName("Route partially successful ingestion records")
+ .description("Defines where to route partially successful ingestion records.")
+ .required(false)
+ .allowableValues("Success", "Failure")
+ .defaultValue("Failure")
+ .build();
+
+ static final PropertyDescriptor INGESTION_STATUS_POLLING_TIMEOUT = new PropertyDescriptor.Builder()
+ .name("Timeout for polling on ingestion status")
+ .displayName("Timeout for polling on ingestion status in seconds")
+ .description("Defines the value of timeout for polling on ingestion status in seconds")
+ .required(false)
+ .dependsOn(POLL_ON_INGESTION_STATUS, "true")
+ .addValidator(StandardValidators.POSITIVE_INTEGER_VALIDATOR)
+ .defaultValue("600")
+ .build();
+
+ static final PropertyDescriptor INGESTION_STATUS_POLLING_INTERVAL = new PropertyDescriptor.Builder()
+ .name("Ingestion status polling interval")
+ .displayName("Ingestion status polling interval in seconds")
+ .description("Defines the value of timeout for polling on ingestion status in seconds.")
+ .required(false)
+ .dependsOn(POLL_ON_INGESTION_STATUS, "true")
+ .addValidator(StandardValidators.POSITIVE_INTEGER_VALIDATOR)
+ .defaultValue("5")
+ .build();
+
+ public static final Relationship SUCCESS = new Relationship.Builder()
+ .name("success")
+ .description("Relationship for success")
+ .build();
+ public static final Relationship FAILURE = new Relationship.Builder()
+ .name("failure")
+ .description("Relationship for failure")
+ .build();
+
+ @Override
+ protected void init(final ProcessorInitializationContext context) {
+ final List<PropertyDescriptor> descriptorList = new ArrayList<>();
+ descriptorList.add(ADX_SERVICE);
+ descriptorList.add(DATABASE_NAME);
+ descriptorList.add(TABLE_NAME);
+ descriptorList.add(MAPPING_NAME);
+ descriptorList.add(DATA_FORMAT);
+ descriptorList.add(IGNORE_FIRST_RECORD);
+ descriptorList.add(IS_STREAMING_ENABLED);
+ descriptorList.add(POLL_ON_INGESTION_STATUS);
+ descriptorList.add(ROUTE_PARTIALLY_SUCCESSFUL_INGESTION);
+ descriptorList.add(INGESTION_STATUS_POLLING_TIMEOUT);
+ descriptorList.add(INGESTION_STATUS_POLLING_INTERVAL);
+ this.descriptors = Collections.unmodifiableList(descriptorList);
+
+ final Set<Relationship> relationshipSet = new HashSet<>();
+ relationshipSet.add(SUCCESS);
+ relationshipSet.add(FAILURE);
+ this.relationships = Collections.unmodifiableSet(relationshipSet);
+ }
+
+ @Override
+ public Set<Relationship> getRelationships() {
+ return this.relationships;
+ }
+
+ @Override
+ public final List<PropertyDescriptor> getSupportedPropertyDescriptors() {
+ return descriptors;
+ }
+
+ @OnScheduled
+ public void onScheduled(final ProcessContext context) {
+ service = context.getProperty(ADX_SERVICE).asControllerService(KustoIngestService.class);
+ if (checkIfIngestorRoleDoesntExist(context.getProperty(DATABASE_NAME).getValue(), context.getProperty(TABLE_NAME).getValue())) {
+ getLogger().error("User might not have ingestor privileges, table validation will be skipped for all table mappings.");
+ throw new ProcessException("User might not have ingestor privileges, table validation will be skipped for all table mappings. ");
+ }
+ streamingEnabled = context.getProperty(IS_STREAMING_ENABLED).evaluateAttributeExpressions().asBoolean();
+ if (streamingEnabled && !checkIfStreamingPolicyIsEnabledInADX(context.getProperty(DATABASE_NAME).getValue(), context.getProperty(DATABASE_NAME).getValue())) {
+ getLogger().error("Streaming policy is not enabled in database {}", context.getProperty(DATABASE_NAME).getValue());
+ throw new ProcessException("Streaming policy is not enabled in database {}" + context.getProperty(DATABASE_NAME).getValue());
+ }
+ }
+
+ @Override
+ public void onTrigger(final ProcessContext context, final ProcessSession session) throws ProcessException {
+ FlowFile flowFile = session.get();
+ if (flowFile == null) {
+ return;
+ }
+
+ String databaseName = context.getProperty(DATABASE_NAME).getValue();
+ String tableName = context.getProperty(TABLE_NAME).getValue();
+ String dataFormat = context.getProperty(DATA_FORMAT).getValue();
+ String mappingName = context.getProperty(MAPPING_NAME).getValue();
+ String routePartiallySuccessfulIngestion = context.getProperty(ROUTE_PARTIALLY_SUCCESSFUL_INGESTION).getValue();
+ String ingestionStatusPollingTimeout = context.getProperty(INGESTION_STATUS_POLLING_TIMEOUT).getValue();
+ String ingestionStatusPollingInterval = context.getProperty(INGESTION_STATUS_POLLING_INTERVAL).getValue();
+ String ignoreFirstRecord = context.getProperty(IGNORE_FIRST_RECORD).getValue();
+ streamingEnabled = context.getProperty(IS_STREAMING_ENABLED).evaluateAttributeExpressions().asBoolean();
+ pollOnIngestionStatus = context.getProperty(POLL_ON_INGESTION_STATUS).evaluateAttributeExpressions().asBoolean();
+
+ boolean isError = false;
+
+ // ingestion into ADX
+ try (final InputStream inputStream = session.read(flowFile)) {
+ StringBuilder ingestLogString = new StringBuilder().append("Ingesting with: ")
+ .append("dataFormat - ").append(dataFormat).append("|")
+ .append("ingestionMappingName - ").append(mappingName).append("|")
+ .append("databaseName - ").append(databaseName).append("|")
+ .append("tableName - ").append(tableName).append("|")
+ .append("pollOnIngestionStatus - ").append(pollOnIngestionStatus).append("|")
+ .append("ingestionStatusPollingTimeout - ").append(ingestionStatusPollingTimeout).append("|")
+ .append("ingestionStatusPollingInterval - ").append(ingestionStatusPollingInterval).append("|")
+ .append("routePartiallySuccessfulIngestion - ").append(routePartiallySuccessfulIngestion);
+ getLogger().debug(ingestLogString.toString());
+
+ KustoIngestionResult result = service.ingestData(
+ new KustoIngestionRequest(streamingEnabled, pollOnIngestionStatus, inputStream, databaseName,
+ tableName, dataFormat, mappingName, ignoreFirstRecord, ingestionStatusPollingTimeout, ingestionStatusPollingInterval)
+ );
+ getLogger().info("Ingest Status Polling Enabled {} and Ingest Status Polling Enabled {} ", pollOnIngestionStatus, result.toString());
+
+ if (result == KustoIngestionResult.SUCCEEDED) {
+ getLogger().info("Ingest Completed - {}", result.getStatus());
+ } else if (result == KustoIngestionResult.FAILED) {
+ getLogger().error("Ingest Failed - {}", result.getStatus());
+ isError = true;
+ } else if (result == KustoIngestionResult.PARTIALLY_SUCCEEDED) {
+ getLogger().warn("Ingest Partially succeeded - {}", result.getStatus());
+ flowFile = session.putAttribute(flowFile, "ingestion_status", "partial_success");
+ if (StringUtils.equalsIgnoreCase(routePartiallySuccessfulIngestion, "Failure")) {
Review Comment:
This should be compared using the failure relationship name instead of repeating the `Failure` string for the comparison.
##########
nifi-nar-bundles/nifi-azure-bundle/nifi-azure-processors/src/main/java/org/apache/nifi/services/azure/data/explorer/StandardKustoIngestService.java:
##########
@@ -0,0 +1,362 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.services.azure.data.explorer;
+
+import com.microsoft.azure.kusto.data.Client;
+import com.microsoft.azure.kusto.data.ClientFactory;
+import com.microsoft.azure.kusto.data.KustoResultSetTable;
+import com.microsoft.azure.kusto.data.auth.ConnectionStringBuilder;
+import com.microsoft.azure.kusto.data.exceptions.DataClientException;
+import com.microsoft.azure.kusto.data.exceptions.DataServiceException;
+import com.microsoft.azure.kusto.ingest.IngestClientFactory;
+import com.microsoft.azure.kusto.ingest.IngestionMapping;
+import com.microsoft.azure.kusto.ingest.IngestionProperties;
+import com.microsoft.azure.kusto.ingest.ManagedStreamingIngestClient;
+import com.microsoft.azure.kusto.ingest.QueuedIngestClient;
+import com.microsoft.azure.kusto.ingest.exceptions.IngestionClientException;
+import com.microsoft.azure.kusto.ingest.exceptions.IngestionServiceException;
+import com.microsoft.azure.kusto.ingest.result.IngestionResult;
+import com.microsoft.azure.kusto.ingest.result.IngestionStatus;
+import com.microsoft.azure.kusto.ingest.result.OperationStatus;
+import com.microsoft.azure.kusto.ingest.source.StreamSourceInfo;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnEnabled;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.controller.AbstractControllerService;
+import org.apache.nifi.controller.ConfigurationContext;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.reporting.InitializationException;
+
+import java.io.IOException;
+import java.net.URISyntaxException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.List;
+import java.util.Objects;
+
+@Tags({"Azure", "ADX", "Kusto", "ingest", "azure"})
+@CapabilityDescription("Sends batches of flowfile content or stream flowfile content to an Azure ADX cluster.")
+public class StandardKustoIngestService extends AbstractControllerService implements KustoIngestService {
+
+ public static final PropertyDescriptor AUTHENTICATION_STRATEGY = new PropertyDescriptor
+ .Builder().name("Authentication Strategy")
+ .displayName("Authentication Strategy")
+ .description("Authentication method for access to Azure Data Explorer")
+ .required(true)
+ .defaultValue(KustoAuthenticationStrategy.MANAGED_IDENTITY.getValue())
+ .allowableValues(KustoAuthenticationStrategy.class)
+ .build();
+
+ public static final PropertyDescriptor APPLICATION_CLIENT_ID = new PropertyDescriptor
+ .Builder().name("Application Client ID")
+ .displayName("Application Client ID")
+ .description("Azure Data Explorer Application Client Identifier for Authentication")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+
+ public static final PropertyDescriptor APPLICATION_KEY = new PropertyDescriptor
+ .Builder().name("Application Key")
+ .displayName("Application Key")
+ .description("Azure Data Explorer Application Key for Authentication")
+ .required(true)
+ .sensitive(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .dependsOn(AUTHENTICATION_STRATEGY, KustoAuthenticationStrategy.APPLICATION_CREDENTIALS.getValue())
+ .build();
+
+ public static final PropertyDescriptor APPLICATION_TENANT_ID = new PropertyDescriptor.Builder()
+ .name("Application Tenant ID")
+ .displayName("Application Tenant ID")
+ .description("Azure Data Explorer Application Tenant Identifier for Authentication")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .dependsOn(AUTHENTICATION_STRATEGY, KustoAuthenticationStrategy.APPLICATION_CREDENTIALS.getValue())
+ .build();
+
+ public static final PropertyDescriptor CLUSTER_URI = new PropertyDescriptor
+ .Builder().name("Cluster URI")
+ .displayName("Cluster URI")
+ .description("Azure Data Explorer Cluster URI")
+ .required(true)
+ .addValidator(StandardValidators.URL_VALIDATOR)
+ .build();
+
+ private static final List<PropertyDescriptor> PROPERTY_DESCRIPTORS = Collections.unmodifiableList(Arrays.asList(
+ AUTHENTICATION_STRATEGY,
+ APPLICATION_CLIENT_ID,
+ APPLICATION_KEY,
+ APPLICATION_TENANT_ID,
+ CLUSTER_URI));
+
+ public static final Pair<String, String> NIFI_SINK = Pair.of("processor", "nifi-sink");
+
+ private volatile QueuedIngestClient queuedIngestClient;
+
+ private volatile ManagedStreamingIngestClient managedStreamingIngestClient;
+
+ private volatile Client executionClient;
+
+ @Override
+ public List<PropertyDescriptor> getSupportedPropertyDescriptors() {
+ return PROPERTY_DESCRIPTORS;
+ }
+
+ /**
+ * @param context the configuration context
+ * @throws InitializationException if unable to create a database connection
+ */
+ @OnEnabled
+ public void onEnabled(final ConfigurationContext context) throws ProcessException, URISyntaxException {
+
+ final String applicationClientId = context.getProperty(APPLICATION_CLIENT_ID).getValue();
+ final String applicationKey = context.getProperty(APPLICATION_KEY).getValue();
+ final String applicationTenantId = context.getProperty(APPLICATION_TENANT_ID).getValue();
+ final String clusterUri = context.getProperty(CLUSTER_URI).getValue();
+ final KustoAuthenticationStrategy kustoAuthenticationStrategy = KustoAuthenticationStrategy.valueOf(context.getProperty(AUTHENTICATION_STRATEGY).getValue());
+
+ if (this.queuedIngestClient == null) {
+ this.queuedIngestClient = createKustoQueuedIngestClient(clusterUri, applicationClientId, applicationKey, applicationTenantId, kustoAuthenticationStrategy);
+ }
+
+ if (this.managedStreamingIngestClient == null) {
+ this.managedStreamingIngestClient = createKustoStreamingIngestClient(clusterUri, applicationClientId, applicationKey, applicationTenantId, kustoAuthenticationStrategy);
+ }
+
+ if (this.executionClient == null) {
+ this.executionClient = createKustoExecutionClient(clusterUri, applicationClientId, applicationKey, applicationTenantId, kustoAuthenticationStrategy);
+ }
+
+ }
+
+ @OnStopped
+ public final void onStopped() {
+ if (this.queuedIngestClient != null) {
+ try {
+ this.queuedIngestClient.close();
+ } catch (IOException e) {
+ getLogger().error("Closing Azure ADX Queued Ingest Client failed", e);
+ } finally {
+ this.queuedIngestClient = null;
+ }
+ }
+ if (this.managedStreamingIngestClient != null) {
+ try {
+ this.managedStreamingIngestClient.close();
+ } catch (IOException e) {
+ getLogger().error("Closing Azure ADX Managed Streaming Ingest Client failed", e);
+ } finally {
+ this.managedStreamingIngestClient = null;
+ }
+ }
+ if (this.executionClient != null) {
+ try {
+ this.executionClient.close();
+ } catch (IOException e) {
+ getLogger().error("Closing Azure ADX Execution Client failed", e);
+ } finally {
+ this.executionClient = null;
+ }
+ }
+ }
+
+
+ protected QueuedIngestClient createKustoQueuedIngestClient(final String clusterUrl,
+ final String appId,
+ final String appKey,
+ final String appTenant,
+ final KustoAuthenticationStrategy kustoAuthStrategy) throws URISyntaxException {
+ ConnectionStringBuilder ingestConnectionStringBuilder = createKustoEngineConnectionString(clusterUrl, appId, appKey, appTenant, kustoAuthStrategy);
+ return IngestClientFactory.createClient(ingestConnectionStringBuilder);
+ }
+
+ protected ManagedStreamingIngestClient createKustoStreamingIngestClient(final String clusterUrl,
+ final String appId,
+ final String appKey,
+ final String appTenant,
+ final KustoAuthenticationStrategy kustoAuthStrategy) throws URISyntaxException {
+ ConnectionStringBuilder ingestConnectionStringBuilder = createKustoEngineConnectionString(clusterUrl, appId, appKey, appTenant, kustoAuthStrategy);
+ ConnectionStringBuilder streamingConnectionStringBuilder = createKustoEngineConnectionString(clusterUrl, appId, appKey, appTenant, kustoAuthStrategy);
+ return IngestClientFactory.createManagedStreamingIngestClient(ingestConnectionStringBuilder, streamingConnectionStringBuilder);
+ }
+
+ public KustoIngestionResult ingestData(KustoIngestionRequest kustoIngestionRequest) throws URISyntaxException {
+ StreamSourceInfo info = new StreamSourceInfo(kustoIngestionRequest.getInputStream());
+ //ingest data
+ IngestionResult ingestionResult;
+ IngestionProperties ingestionProperties = new IngestionProperties(kustoIngestionRequest.getDatabaseName(),
+ kustoIngestionRequest.getTableName());
+
+ IngestionMapping.IngestionMappingKind ingestionMappingKind = setDataFormatAndMapping(kustoIngestionRequest.getDataFormat(), ingestionProperties);
+ if (StringUtils.isNotEmpty(kustoIngestionRequest.getMappingName()) && ingestionMappingKind != null) {
+ ingestionProperties.setIngestionMapping(kustoIngestionRequest.getMappingName(), ingestionMappingKind);
+ }
+
+ ingestionProperties.setReportLevel(IngestionProperties.IngestionReportLevel.FAILURES_AND_SUCCESSES);
+ ingestionProperties.setReportMethod(IngestionProperties.IngestionReportMethod.TABLE);
+ ingestionProperties.setFlushImmediately(false);
+ ingestionProperties.setIgnoreFirstRecord(StringUtils.equalsIgnoreCase(kustoIngestionRequest.getIgnoreFirstRecord(), "YES"));
+
+ try {
+ if (kustoIngestionRequest.isStreamingEnabled()) {
+ ingestionResult = managedStreamingIngestClient.ingestFromStream(info, ingestionProperties);
+ } else {
+ ingestionResult = queuedIngestClient.ingestFromStream(info, ingestionProperties);
+ }
+ if (kustoIngestionRequest.pollOnIngestionStatus()) {
+ List<IngestionStatus> statuses;
+ long startTime = System.currentTimeMillis();
+ long timeoutMillis = Long.parseLong(kustoIngestionRequest.getIngestionStatusPollingTimeout()) * 1000L;
Review Comment:
Instead of parsing the value in this method, the KustoIngestRequest should use a `java.time.Duration` and the value should be parsed in the Processor.
##########
nifi-nar-bundles/nifi-azure-bundle/nifi-azure-processors/src/main/java/org/apache/nifi/processors/azure/data/explorer/PutAzureDataExplorer.java:
##########
@@ -0,0 +1,399 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.azure.data.explorer;
+
+import com.microsoft.azure.kusto.data.KustoResultSetTable;
+import com.microsoft.azure.kusto.ingest.IngestionMapping;
+import com.microsoft.azure.kusto.ingest.IngestionProperties;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.nifi.annotation.behavior.ReadsAttribute;
+import org.apache.nifi.annotation.behavior.ReadsAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.expression.ExpressionLanguageScope;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.services.azure.data.explorer.KustoIngestService;
+import org.apache.nifi.services.azure.data.explorer.KustoIngestionRequest;
+import org.apache.nifi.services.azure.data.explorer.KustoIngestionResult;
+import org.apache.nifi.services.azure.data.explorer.KustoQueryResponse;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.net.URISyntaxException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+
+@Tags({"Azure", "Kusto", "ADX", "Explorer", "Data"})
+@CapabilityDescription("The PutAzureDataExplorer acts as a ADX sink connector which sends flowFiles using the ADX-Service to the provided Azure Data" +
+ "Explorer Ingest Endpoint. The data can be sent through queued ingestion or streaming ingestion to the Azure Data Explorer cluster.")
+@ReadsAttributes({
+ @ReadsAttribute(attribute = "DB_NAME", description = "Specifies the name of the ADX database where the data needs to be stored."),
+ @ReadsAttribute(attribute = "TABLE_NAME", description = "Specifies the name of the ADX table where the data needs to be stored."),
+ @ReadsAttribute(attribute = "MAPPING_NAME", description = "Specifies the name of the mapping responsible for storing the data in appropriate columns."),
+ @ReadsAttribute(attribute = "FLUSH_IMMEDIATE", description = "In case of queued ingestion, this property determines whether the data should be flushed immediately to the ingest endpoint."),
+ @ReadsAttribute(attribute = "DATA_FORMAT", description = "Specifies the format of data that is send to Azure Data Explorer."),
+ @ReadsAttribute(attribute = "IGNORE_FIRST_RECORD", description = "Specifies whether we want to ignore ingestion of first record. " +
+ "This is primarily applicable for csv files. Default is set to NO"),
+ @ReadsAttribute(attribute = "POLL_ON_INGESTION_STATUS", description = "Specifies whether we want to poll on ingestion result during ingestion into ADX." +
+ "In case of applications that need high throughput it is recommended to keep the default value as false. Default is set to false." +
+ "This property should be set to true during Queued Ingestion for near realtime micro-batches of data that require acknowledgement of ingestion status.")
+})
+public class PutAzureDataExplorer extends AbstractProcessor {
+
+ public static final String FETCH_TABLE_COMMAND = "%s | count";
+ public static final String STREAMING_POLICY_SHOW_COMMAND = ".show %s %s policy streamingingestion";
+ public static final String DATABASE = "database";
+
+ private List<PropertyDescriptor> descriptors;
+ private Set<Relationship> relationships;
+ private transient KustoIngestService service;
+ private boolean isStreamingEnabled;
+ private boolean pollOnIngestionStatus;
+
+ public static final AllowableValue AVRO = new AllowableValue(
+ IngestionProperties.DataFormat.AVRO.getKustoValue(), IngestionProperties.DataFormat.AVRO.getKustoValue(),
+ "An Avro format with support for logical types and for the snappy compression codec");
+
+ public static final AllowableValue APACHEAVRO = new AllowableValue(
+ IngestionProperties.DataFormat.APACHEAVRO.getKustoValue(), IngestionProperties.DataFormat.APACHEAVRO.getKustoValue(),
+ "An Avro format with support for logical types and for the snappy compression codec.");
+
+ public static final AllowableValue CSV = new AllowableValue(
+ IngestionProperties.DataFormat.CSV.name(), IngestionProperties.DataFormat.CSV.getKustoValue(),
+ "A text file with comma-separated values (,). For more information, see RFC 4180: Common Format " +
+ "and MIME Type for Comma-Separated Values (CSV) Files.");
+
+ public static final AllowableValue JSON = new AllowableValue(
+ IngestionProperties.DataFormat.JSON.name(), IngestionProperties.DataFormat.JSON.getKustoValue(),
+ "A text file containing JSON objects separated by \\n or \\r\\n. For more information, " +
+ "see JSON Lines (JSONL).");
+
+ public static final AllowableValue MULTIJSON = new AllowableValue(
+ IngestionProperties.DataFormat.MULTIJSON.name(), IngestionProperties.DataFormat.MULTIJSON.getKustoValue(),
+ "A text file containing a JSON array of property containers (each representing a record) or any " +
+ "number of property containers separated by spaces, \\n or \\r\\n. Each property container may be " +
+ "spread across multiple lines. This format is preferable to JSON unless the data is not property " +
+ "containers.");
+
+ public static final AllowableValue ORC = new AllowableValue(
+ IngestionProperties.DataFormat.ORC.name(), IngestionProperties.DataFormat.ORC.getKustoValue(), "An ORC file.");
+
+ public static final AllowableValue PARQUET = new AllowableValue(
+ IngestionProperties.DataFormat.PARQUET.name(), IngestionProperties.DataFormat.PARQUET.getKustoValue(), "A parquet file.");
+
+ public static final AllowableValue PSV = new AllowableValue(
+ IngestionProperties.DataFormat.PSV.name(), IngestionProperties.DataFormat.PSV.getKustoValue(), "A text file with values separated by vertical bars (|).");
+
+ public static final AllowableValue SCSV = new AllowableValue(
+ IngestionProperties.DataFormat.SCSV.name(), IngestionProperties.DataFormat.SCSV.getKustoValue(), "A text file with values separated by semicolons (;).");
+
+ public static final AllowableValue SOHSV = new AllowableValue(
+ IngestionProperties.DataFormat.SOHSV.name(), IngestionProperties.DataFormat.SOHSV.getKustoValue(),
+ "A text file with SOH-separated values. (SOH is the ASCII code point 1. " +
+ "This format is used by Hive in HDInsight).");
+
+ public static final AllowableValue TSV = new AllowableValue(
+ IngestionProperties.DataFormat.TSV.name(), IngestionProperties.DataFormat.TSV.getKustoValue(), "A text file with tab delimited values (\\t).");
+
+ public static final AllowableValue TSVE = new AllowableValue(
+ IngestionProperties.DataFormat.TSVE.name(), IngestionProperties.DataFormat.TSVE.getKustoValue(),
+ "A text file with tab-delimited values (\\t). A backslash (\\) is used as escape character.");
+
+ public static final AllowableValue TXT = new AllowableValue(
+ IngestionProperties.DataFormat.TXT.name(), IngestionProperties.DataFormat.TXT.getKustoValue(),
+ "A text file with lines separated by \\n. Empty lines are skipped.");
+
+ public static final AllowableValue IGNORE_FIRST_RECORD_YES = new AllowableValue(
+ "YES", "YES",
+ "Ignore first record during ingestion");
+
+ public static final AllowableValue IGNORE_FIRST_RECORD_NO = new AllowableValue(
+ "NO", "NO",
+ "Do not ignore first record during ingestion");
+
+ public static final PropertyDescriptor DATABASE_NAME = new PropertyDescriptor.Builder()
+ .name("Database Name")
+ .displayName("Database Name")
+ .description("Azure Data Explorer Database Name for querying")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+ .build();
+
+ public static final PropertyDescriptor TABLE_NAME = new PropertyDescriptor.Builder()
+ .name("Table Name")
+ .displayName("Table Name")
+ .description("Azure Data Explorer Table Name for ingesting data")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+ .build();
+
+ public static final PropertyDescriptor MAPPING_NAME = new PropertyDescriptor
+ .Builder().name("Mapping name")
+ .displayName("Mapping name")
+ .description("The name of the mapping responsible for storing the data in the appropriate columns.")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+
+ public static final PropertyDescriptor IS_STREAMING_ENABLED = new PropertyDescriptor
+ .Builder().name("Is Streaming enabled")
+ .displayName("Is Streaming enabled")
+ .description("This property determines whether we want to stream data to ADX.")
+ .required(false)
+ .allowableValues("true", "false")
+ .addValidator(StandardValidators.BOOLEAN_VALIDATOR)
+ .defaultValue("false")
+ .build();
+
+ public static final PropertyDescriptor POLL_ON_INGESTION_STATUS = new PropertyDescriptor
+ .Builder().name("Poll on ingestion status")
+ .displayName("Whether to Poll on ingestion status")
Review Comment:
The name should be changed to use Title Case.
##########
nifi-nar-bundles/nifi-azure-bundle/nifi-azure-processors/src/main/java/org/apache/nifi/services/azure/data/explorer/StandardKustoIngestService.java:
##########
@@ -0,0 +1,293 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.services.azure.data.explorer;
+
+import com.microsoft.azure.kusto.data.Client;
+import com.microsoft.azure.kusto.data.ClientFactory;
+import com.microsoft.azure.kusto.data.KustoResultSetTable;
+import com.microsoft.azure.kusto.data.auth.ConnectionStringBuilder;
+import com.microsoft.azure.kusto.data.exceptions.DataClientException;
+import com.microsoft.azure.kusto.data.exceptions.DataServiceException;
+import com.microsoft.azure.kusto.ingest.IngestClientFactory;
+import com.microsoft.azure.kusto.ingest.ManagedStreamingIngestClient;
+import com.microsoft.azure.kusto.ingest.QueuedIngestClient;
+import com.microsoft.azure.kusto.ingest.exceptions.IngestionClientException;
+import com.microsoft.azure.kusto.ingest.exceptions.IngestionServiceException;
+import com.microsoft.azure.kusto.ingest.result.IngestionResult;
+import com.microsoft.azure.kusto.ingest.result.IngestionStatus;
+import com.microsoft.azure.kusto.ingest.result.OperationStatus;
+import com.microsoft.azure.kusto.ingest.source.StreamSourceInfo;
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.nifi.annotation.behavior.ReadsAttribute;
+import org.apache.nifi.annotation.behavior.ReadsAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnEnabled;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.controller.AbstractControllerService;
+import org.apache.nifi.controller.ConfigurationContext;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.reporting.InitializationException;
+
+import java.io.IOException;
+import java.net.URISyntaxException;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.List;
+import java.util.Objects;
+
+@Tags({"Azure", "ADX", "Kusto", "ingest", "azure"})
+@CapabilityDescription("Sends batches of flowfile content or stream flowfile content to an Azure ADX cluster.")
+@ReadsAttributes({
+ @ReadsAttribute(attribute = "AUTH_STRATEGY", description = "The strategy/method to authenticate against Azure Active Directory, either 'application' or 'managed_identity'."),
+ @ReadsAttribute(attribute = "APP_ID", description = "Specifies Azure application id for accessing the ADX-Cluster."),
+ @ReadsAttribute(attribute = "APP_KEY", description = "Specifies Azure application key for accessing the ADX-Cluster."),
+ @ReadsAttribute(attribute = "APP_TENANT", description = "Azure application tenant for accessing the ADX-Cluster."),
+ @ReadsAttribute(attribute = "CLUSTER_URL", description = "Endpoint of ADX cluster. This is required only when streaming data to ADX cluster is enabled."),
+})
+public class StandardKustoIngestService extends AbstractControllerService implements KustoIngestService {
+
+ public static final PropertyDescriptor AUTHENTICATION_STRATEGY = new PropertyDescriptor
+ .Builder().name("Authentication Strategy")
+ .displayName("Authentication Strategy")
+ .description("Authentication method for access to Azure Data Explorer")
+ .required(true)
+ .defaultValue(KustoAuthenticationStrategy.MANAGED_IDENTITY.getValue())
+ .allowableValues(KustoAuthenticationStrategy.class)
+ .build();
+
+ public static final PropertyDescriptor APPLICATION_CLIENT_ID = new PropertyDescriptor
+ .Builder().name("Application Client ID")
+ .displayName("Application Client ID")
+ .description("Azure Data Explorer Application Client Identifier for Authentication")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+
+ public static final PropertyDescriptor APPLICATION_KEY = new PropertyDescriptor
+ .Builder().name("Application Key")
+ .displayName("Application Key")
+ .description("Azure Data Explorer Application Key for Authentication")
+ .required(true)
+ .sensitive(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .dependsOn(AUTHENTICATION_STRATEGY, KustoAuthenticationStrategy.APPLICATION_CREDENTIALS.getValue())
+ .build();
+
+ public static final PropertyDescriptor APPLICATION_TENANT_ID = new PropertyDescriptor.Builder()
+ .name("Application Tenant ID")
+ .displayName("Application Tenant ID")
+ .description("Azure Data Explorer Application Tenant Identifier for Authentication")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .dependsOn(AUTHENTICATION_STRATEGY, KustoAuthenticationStrategy.APPLICATION_CREDENTIALS.getValue())
+ .build();
+
+ public static final PropertyDescriptor CLUSTER_URI = new PropertyDescriptor
+ .Builder().name("Cluster URI")
+ .displayName("Cluster URI")
+ .description("Azure Data Explorer Cluster URI")
+ .required(true)
+ .addValidator(StandardValidators.URL_VALIDATOR)
+ .build();
+
+ private static final List<PropertyDescriptor> PROPERTY_DESCRIPTORS = Collections.unmodifiableList(Arrays.asList(
+ AUTHENTICATION_STRATEGY,
+ APPLICATION_CLIENT_ID,
+ APPLICATION_KEY,
+ APPLICATION_TENANT_ID,
+ CLUSTER_URI));
+
+ public static final Pair<String, String> NIFI_SINK = Pair.of("processor", "nifi-sink");
+
+ private volatile QueuedIngestClient queuedIngestClient;
+
+ private volatile ManagedStreamingIngestClient managedStreamingIngestClient;
+
+ private volatile Client executionClient;
+
+ @Override
+ public List<PropertyDescriptor> getSupportedPropertyDescriptors() {
+ return PROPERTY_DESCRIPTORS;
+ }
+
+ /**
+ * @param context the configuration context
+ * @throws InitializationException if unable to create a database connection
+ */
+ @OnEnabled
+ public void onEnabled(final ConfigurationContext context) throws ProcessException, URISyntaxException {
+
+ getLogger().info("Starting Azure ADX Connection Service...");
+ final String applicationClientId = context.getProperty(APPLICATION_CLIENT_ID).getValue();
+ final String applicationKey = context.getProperty(APPLICATION_KEY).getValue();
+ final String applicationTenantId = context.getProperty(APPLICATION_TENANT_ID).getValue();
+ final String clusterUri = context.getProperty(CLUSTER_URI).getValue();
+ final KustoAuthenticationStrategy kustoAuthenticationStrategy = KustoAuthenticationStrategy.valueOf(context.getProperty(AUTHENTICATION_STRATEGY).getValue());
+
+ if (this.queuedIngestClient == null) {
+ this.queuedIngestClient = createKustoQueuedIngestClient(clusterUri, applicationClientId, applicationKey, applicationTenantId, kustoAuthenticationStrategy);
+ }
+
+ if (this.managedStreamingIngestClient == null) {
+ this.managedStreamingIngestClient = createKustoStreamingIngestClient(clusterUri, applicationClientId, applicationKey, applicationTenantId, kustoAuthenticationStrategy);
+ }
+
+ if (this.executionClient == null) {
+ this.executionClient = createKustoExecutionClient(clusterUri, applicationClientId, applicationKey, applicationTenantId, kustoAuthenticationStrategy);
+ }
+
+ }
+
+ @OnStopped
+ public final void onStopped() {
+ if (this.queuedIngestClient != null) {
+ try {
+ this.queuedIngestClient.close();
+ } catch (IOException e) {
+ getLogger().error("Closing Azure ADX Queued Ingest Client failed ", e);
+ } finally {
+ this.queuedIngestClient = null;
+ }
+ }
+ if (this.managedStreamingIngestClient != null) {
+ try {
+ this.managedStreamingIngestClient.close();
+ } catch (IOException e) {
+ getLogger().error("Closing Azure ADX Managed Streaming Ingest Client failed", e);
+ } finally {
+ this.managedStreamingIngestClient = null;
+ }
+ }
+ if (this.executionClient != null) {
+ try {
+ this.executionClient.close();
+ } catch (IOException e) {
+ getLogger().error("Closing Azure ADX Execution Client failed", e);
+ } finally {
+ this.executionClient = null;
+ }
+ }
+ }
+
+
+ protected QueuedIngestClient createKustoQueuedIngestClient(final String clusterUrl,
+ final String appId,
+ final String appKey,
+ final String appTenant,
+ final KustoAuthenticationStrategy kustoAuthStrategy) throws URISyntaxException {
+ ConnectionStringBuilder ingestConnectionStringBuilder = createKustoEngineConnectionString(clusterUrl, appId, appKey, appTenant, kustoAuthStrategy);
+ return IngestClientFactory.createClient(ingestConnectionStringBuilder);
+ }
+
+ protected ManagedStreamingIngestClient createKustoStreamingIngestClient(final String clusterUrl,
+ final String appId,
+ final String appKey,
+ final String appTenant,
+ final KustoAuthenticationStrategy kustoAuthStrategy) throws URISyntaxException {
+ ConnectionStringBuilder ingestConnectionStringBuilder = createKustoEngineConnectionString(clusterUrl, appId, appKey, appTenant, kustoAuthStrategy);
+ ConnectionStringBuilder streamingConnectionStringBuilder = createKustoEngineConnectionString(clusterUrl, appId, appKey, appTenant, kustoAuthStrategy);
+ return IngestClientFactory.createManagedStreamingIngestClient(ingestConnectionStringBuilder, streamingConnectionStringBuilder);
+ }
+
+ public KustoIngestionResult ingestData(KustoIngestionRequest kustoIngestionRequest) throws URISyntaxException {
+ StreamSourceInfo info = new StreamSourceInfo(kustoIngestionRequest.getInputStream());
+ //ingest data
+ IngestionResult ingestionResult;
+ try {
+ if (kustoIngestionRequest.isStreamingEnabled()) {
+ ingestionResult = managedStreamingIngestClient.ingestFromStream(info, kustoIngestionRequest.getIngestionProperties());
+ } else {
+ ingestionResult = queuedIngestClient.ingestFromStream(info, kustoIngestionRequest.getIngestionProperties());
+ }
+ if (kustoIngestionRequest.pollOnIngestionStatus()) {
+ List<IngestionStatus> statuses;
+ long startTime = System.currentTimeMillis();
+ long timeoutMillis = 600 * 1800; // 10 Minutes timeout
+ while (true) {
Review Comment:
Please review this recommendation to adjust the loop approach.
##########
nifi-nar-bundles/nifi-azure-bundle/nifi-azure-processors/src/main/java/org/apache/nifi/processors/azure/data/explorer/PutAzureDataExplorer.java:
##########
@@ -0,0 +1,309 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.azure.data.explorer;
+
+import org.apache.commons.lang3.StringUtils;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.expression.ExpressionLanguageScope;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.services.azure.data.explorer.KustoIngestDataFormat;
+import org.apache.nifi.services.azure.data.explorer.KustoIngestService;
+import org.apache.nifi.services.azure.data.explorer.KustoIngestionRequest;
+import org.apache.nifi.services.azure.data.explorer.KustoIngestionResult;
+import org.apache.nifi.services.azure.data.explorer.KustoQueryResponse;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.net.URISyntaxException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+
+@Tags({"Azure", "Kusto", "ADX", "Explorer", "Data"})
+@CapabilityDescription("The PutAzureDataExplorer acts as a ADX sink connector which sends flowFiles using the ADX-Service to the provided Azure Data" +
+ "Explorer Ingest Endpoint. The data can be sent through queued ingestion or streaming ingestion to the Azure Data Explorer cluster.")
+public class PutAzureDataExplorer extends AbstractProcessor {
+
+ public static final String FETCH_TABLE_COMMAND = "%s | count";
+ public static final String STREAMING_POLICY_SHOW_COMMAND = ".show %s %s policy streamingingestion";
+ public static final String DATABASE = "database";
+
+ private List<PropertyDescriptor> descriptors;
+ private Set<Relationship> relationships;
+ private transient KustoIngestService service;
+ private boolean streamingEnabled;
+ private boolean pollOnIngestionStatus;
+
+ public static final AllowableValue IGNORE_FIRST_RECORD_YES = new AllowableValue(
+ "YES", "YES",
+ "Ignore first record during ingestion");
+
+ public static final AllowableValue IGNORE_FIRST_RECORD_NO = new AllowableValue(
+ "NO", "NO",
+ "Do not ignore first record during ingestion");
+
+ public static final PropertyDescriptor DATABASE_NAME = new PropertyDescriptor.Builder()
+ .name("Database Name")
+ .displayName("Database Name")
+ .description("Azure Data Explorer Database Name for querying")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+ .build();
+
+ public static final PropertyDescriptor TABLE_NAME = new PropertyDescriptor.Builder()
+ .name("Table Name")
+ .displayName("Table Name")
+ .description("Azure Data Explorer Table Name for ingesting data")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+ .build();
+
+ public static final PropertyDescriptor MAPPING_NAME = new PropertyDescriptor
+ .Builder().name("Ingest Mapping name")
+ .displayName("Ingest Mapping name")
+ .description("The name of the mapping responsible for storing the data in the appropriate columns.")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+
+ public static final PropertyDescriptor IS_STREAMING_ENABLED = new PropertyDescriptor
+ .Builder().name("Streaming enabled")
+ .displayName("Streaming enabled")
+ .description("This property determines whether we want to stream data to ADX.")
+ .required(false)
+ .allowableValues("true", "false")
+ .addValidator(StandardValidators.BOOLEAN_VALIDATOR)
+ .defaultValue("false")
+ .build();
+
+ public static final PropertyDescriptor POLL_ON_INGESTION_STATUS = new PropertyDescriptor
+ .Builder().name("Poll on ingestion status")
+ .displayName("Whether to Poll on ingestion status")
+ .description("This property determines whether we want to poll on ingestion status after an ingestion to ADX is completed")
+ .required(false)
+ .allowableValues("true", "false")
+ .addValidator(StandardValidators.BOOLEAN_VALIDATOR)
+ .defaultValue("false")
+ .build();
+ public static final PropertyDescriptor ADX_SERVICE = new PropertyDescriptor
+ .Builder().name("Kusto Ingest Service")
+ .displayName("Kusto Ingest Service")
+ .description("Azure Data Explorer Kusto Ingest Service")
+ .required(true)
+ .identifiesControllerService(KustoIngestService.class)
+ .build();
+
+ static final PropertyDescriptor DATA_FORMAT = new PropertyDescriptor.Builder()
+ .name("Data Format")
+ .displayName("Data Format")
+ .description("The format of the data that is sent to Azure Data Explorer.")
+ .required(true)
+ .allowableValues(KustoIngestDataFormat.values())
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+
+ static final PropertyDescriptor IGNORE_FIRST_RECORD = new PropertyDescriptor.Builder()
+ .name("Ingestion Ignore First Record")
+ .displayName("Ingestion Ignore First Record")
+ .description("Defines whether ignore first record while ingestion.")
+ .required(false)
+ .allowableValues(IGNORE_FIRST_RECORD_YES, IGNORE_FIRST_RECORD_NO)
+ .defaultValue(IGNORE_FIRST_RECORD_NO.getValue())
+ .build();
+
+ static final PropertyDescriptor ROUTE_PARTIALLY_SUCCESSFUL_INGESTION = new PropertyDescriptor.Builder()
+ .name("Route partially successful ingestion records")
+ .displayName("Route partially successful ingestion records")
+ .description("Defines where to route partially successful ingestion records.")
+ .required(false)
+ .allowableValues("Success", "Failure")
+ .defaultValue("Failure")
Review Comment:
These values should reuse the available relationship names of `success` and `failure`.
##########
nifi-nar-bundles/nifi-azure-bundle/nifi-azure-processors/src/main/java/org/apache/nifi/processors/azure/data/explorer/PutAzureDataExplorer.java:
##########
@@ -0,0 +1,309 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.azure.data.explorer;
+
+import org.apache.commons.lang3.StringUtils;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.expression.ExpressionLanguageScope;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.services.azure.data.explorer.KustoIngestDataFormat;
+import org.apache.nifi.services.azure.data.explorer.KustoIngestService;
+import org.apache.nifi.services.azure.data.explorer.KustoIngestionRequest;
+import org.apache.nifi.services.azure.data.explorer.KustoIngestionResult;
+import org.apache.nifi.services.azure.data.explorer.KustoQueryResponse;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.net.URISyntaxException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+
+@Tags({"Azure", "Kusto", "ADX", "Explorer", "Data"})
+@CapabilityDescription("The PutAzureDataExplorer acts as a ADX sink connector which sends flowFiles using the ADX-Service to the provided Azure Data" +
+ "Explorer Ingest Endpoint. The data can be sent through queued ingestion or streaming ingestion to the Azure Data Explorer cluster.")
+public class PutAzureDataExplorer extends AbstractProcessor {
+
+ public static final String FETCH_TABLE_COMMAND = "%s | count";
+ public static final String STREAMING_POLICY_SHOW_COMMAND = ".show %s %s policy streamingingestion";
+ public static final String DATABASE = "database";
+
+ private List<PropertyDescriptor> descriptors;
+ private Set<Relationship> relationships;
+ private transient KustoIngestService service;
+ private boolean streamingEnabled;
+ private boolean pollOnIngestionStatus;
+
+ public static final AllowableValue IGNORE_FIRST_RECORD_YES = new AllowableValue(
+ "YES", "YES",
+ "Ignore first record during ingestion");
+
+ public static final AllowableValue IGNORE_FIRST_RECORD_NO = new AllowableValue(
+ "NO", "NO",
+ "Do not ignore first record during ingestion");
+
+ public static final PropertyDescriptor DATABASE_NAME = new PropertyDescriptor.Builder()
+ .name("Database Name")
+ .displayName("Database Name")
+ .description("Azure Data Explorer Database Name for querying")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+ .build();
+
+ public static final PropertyDescriptor TABLE_NAME = new PropertyDescriptor.Builder()
+ .name("Table Name")
+ .displayName("Table Name")
+ .description("Azure Data Explorer Table Name for ingesting data")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+ .build();
+
+ public static final PropertyDescriptor MAPPING_NAME = new PropertyDescriptor
+ .Builder().name("Ingest Mapping name")
+ .displayName("Ingest Mapping name")
+ .description("The name of the mapping responsible for storing the data in the appropriate columns.")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+
+ public static final PropertyDescriptor IS_STREAMING_ENABLED = new PropertyDescriptor
+ .Builder().name("Streaming enabled")
+ .displayName("Streaming enabled")
+ .description("This property determines whether we want to stream data to ADX.")
+ .required(false)
+ .allowableValues("true", "false")
+ .addValidator(StandardValidators.BOOLEAN_VALIDATOR)
+ .defaultValue("false")
+ .build();
+
+ public static final PropertyDescriptor POLL_ON_INGESTION_STATUS = new PropertyDescriptor
+ .Builder().name("Poll on ingestion status")
+ .displayName("Whether to Poll on ingestion status")
+ .description("This property determines whether we want to poll on ingestion status after an ingestion to ADX is completed")
+ .required(false)
+ .allowableValues("true", "false")
+ .addValidator(StandardValidators.BOOLEAN_VALIDATOR)
+ .defaultValue("false")
+ .build();
+ public static final PropertyDescriptor ADX_SERVICE = new PropertyDescriptor
+ .Builder().name("Kusto Ingest Service")
+ .displayName("Kusto Ingest Service")
+ .description("Azure Data Explorer Kusto Ingest Service")
+ .required(true)
+ .identifiesControllerService(KustoIngestService.class)
+ .build();
+
+ static final PropertyDescriptor DATA_FORMAT = new PropertyDescriptor.Builder()
+ .name("Data Format")
+ .displayName("Data Format")
+ .description("The format of the data that is sent to Azure Data Explorer.")
+ .required(true)
+ .allowableValues(KustoIngestDataFormat.values())
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+
+ static final PropertyDescriptor IGNORE_FIRST_RECORD = new PropertyDescriptor.Builder()
+ .name("Ingestion Ignore First Record")
+ .displayName("Ingestion Ignore First Record")
+ .description("Defines whether ignore first record while ingestion.")
+ .required(false)
+ .allowableValues(IGNORE_FIRST_RECORD_YES, IGNORE_FIRST_RECORD_NO)
+ .defaultValue(IGNORE_FIRST_RECORD_NO.getValue())
+ .build();
+
+ static final PropertyDescriptor ROUTE_PARTIALLY_SUCCESSFUL_INGESTION = new PropertyDescriptor.Builder()
+ .name("Route partially successful ingestion records")
+ .displayName("Route partially successful ingestion records")
+ .description("Defines where to route partially successful ingestion records.")
+ .required(false)
+ .allowableValues("Success", "Failure")
+ .defaultValue("Failure")
+ .build();
+
+ static final PropertyDescriptor INGESTION_STATUS_POLLING_TIMEOUT = new PropertyDescriptor.Builder()
+ .name("Timeout for polling on ingestion status")
+ .displayName("Timeout for polling on ingestion status in seconds")
+ .description("Defines the value of timeout for polling on ingestion status in seconds")
+ .required(false)
+ .dependsOn(POLL_ON_INGESTION_STATUS, "true")
+ .addValidator(StandardValidators.POSITIVE_INTEGER_VALIDATOR)
+ .defaultValue("600")
+ .build();
+
+ static final PropertyDescriptor INGESTION_STATUS_POLLING_INTERVAL = new PropertyDescriptor.Builder()
+ .name("Ingestion status polling interval")
+ .displayName("Ingestion status polling interval in seconds")
+ .description("Defines the value of timeout for polling on ingestion status in seconds.")
+ .required(false)
+ .dependsOn(POLL_ON_INGESTION_STATUS, "true")
Review Comment:
```suggestion
.dependsOn(POLL_ON_INGESTION_STATUS, Boolean.TRUE.toString())
```
##########
nifi-nar-bundles/nifi-azure-bundle/nifi-azure-processors/src/main/java/org/apache/nifi/processors/azure/data/explorer/PutAzureDataExplorer.java:
##########
@@ -0,0 +1,309 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.azure.data.explorer;
+
+import org.apache.commons.lang3.StringUtils;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.expression.ExpressionLanguageScope;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.services.azure.data.explorer.KustoIngestDataFormat;
+import org.apache.nifi.services.azure.data.explorer.KustoIngestService;
+import org.apache.nifi.services.azure.data.explorer.KustoIngestionRequest;
+import org.apache.nifi.services.azure.data.explorer.KustoIngestionResult;
+import org.apache.nifi.services.azure.data.explorer.KustoQueryResponse;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.net.URISyntaxException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+
+@Tags({"Azure", "Kusto", "ADX", "Explorer", "Data"})
+@CapabilityDescription("The PutAzureDataExplorer acts as a ADX sink connector which sends flowFiles using the ADX-Service to the provided Azure Data" +
+ "Explorer Ingest Endpoint. The data can be sent through queued ingestion or streaming ingestion to the Azure Data Explorer cluster.")
+public class PutAzureDataExplorer extends AbstractProcessor {
+
+ public static final String FETCH_TABLE_COMMAND = "%s | count";
+ public static final String STREAMING_POLICY_SHOW_COMMAND = ".show %s %s policy streamingingestion";
+ public static final String DATABASE = "database";
+
+ private List<PropertyDescriptor> descriptors;
+ private Set<Relationship> relationships;
+ private transient KustoIngestService service;
+ private boolean streamingEnabled;
+ private boolean pollOnIngestionStatus;
+
+ public static final AllowableValue IGNORE_FIRST_RECORD_YES = new AllowableValue(
+ "YES", "YES",
+ "Ignore first record during ingestion");
+
+ public static final AllowableValue IGNORE_FIRST_RECORD_NO = new AllowableValue(
+ "NO", "NO",
+ "Do not ignore first record during ingestion");
+
+ public static final PropertyDescriptor DATABASE_NAME = new PropertyDescriptor.Builder()
+ .name("Database Name")
+ .displayName("Database Name")
+ .description("Azure Data Explorer Database Name for querying")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+ .build();
+
+ public static final PropertyDescriptor TABLE_NAME = new PropertyDescriptor.Builder()
+ .name("Table Name")
+ .displayName("Table Name")
+ .description("Azure Data Explorer Table Name for ingesting data")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+ .build();
+
+ public static final PropertyDescriptor MAPPING_NAME = new PropertyDescriptor
+ .Builder().name("Ingest Mapping name")
+ .displayName("Ingest Mapping name")
+ .description("The name of the mapping responsible for storing the data in the appropriate columns.")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+
+ public static final PropertyDescriptor IS_STREAMING_ENABLED = new PropertyDescriptor
+ .Builder().name("Streaming enabled")
+ .displayName("Streaming enabled")
+ .description("This property determines whether we want to stream data to ADX.")
+ .required(false)
+ .allowableValues("true", "false")
+ .addValidator(StandardValidators.BOOLEAN_VALIDATOR)
+ .defaultValue("false")
+ .build();
+
+ public static final PropertyDescriptor POLL_ON_INGESTION_STATUS = new PropertyDescriptor
+ .Builder().name("Poll on ingestion status")
+ .displayName("Whether to Poll on ingestion status")
+ .description("This property determines whether we want to poll on ingestion status after an ingestion to ADX is completed")
+ .required(false)
+ .allowableValues("true", "false")
+ .addValidator(StandardValidators.BOOLEAN_VALIDATOR)
+ .defaultValue("false")
+ .build();
+ public static final PropertyDescriptor ADX_SERVICE = new PropertyDescriptor
+ .Builder().name("Kusto Ingest Service")
+ .displayName("Kusto Ingest Service")
+ .description("Azure Data Explorer Kusto Ingest Service")
+ .required(true)
+ .identifiesControllerService(KustoIngestService.class)
+ .build();
+
+ static final PropertyDescriptor DATA_FORMAT = new PropertyDescriptor.Builder()
+ .name("Data Format")
+ .displayName("Data Format")
+ .description("The format of the data that is sent to Azure Data Explorer.")
+ .required(true)
+ .allowableValues(KustoIngestDataFormat.values())
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+
+ static final PropertyDescriptor IGNORE_FIRST_RECORD = new PropertyDescriptor.Builder()
+ .name("Ingestion Ignore First Record")
+ .displayName("Ingestion Ignore First Record")
+ .description("Defines whether ignore first record while ingestion.")
+ .required(false)
+ .allowableValues(IGNORE_FIRST_RECORD_YES, IGNORE_FIRST_RECORD_NO)
+ .defaultValue(IGNORE_FIRST_RECORD_NO.getValue())
+ .build();
+
+ static final PropertyDescriptor ROUTE_PARTIALLY_SUCCESSFUL_INGESTION = new PropertyDescriptor.Builder()
+ .name("Route partially successful ingestion records")
+ .displayName("Route partially successful ingestion records")
+ .description("Defines where to route partially successful ingestion records.")
+ .required(false)
+ .allowableValues("Success", "Failure")
+ .defaultValue("Failure")
+ .build();
+
+ static final PropertyDescriptor INGESTION_STATUS_POLLING_TIMEOUT = new PropertyDescriptor.Builder()
+ .name("Timeout for polling on ingestion status")
+ .displayName("Timeout for polling on ingestion status in seconds")
Review Comment:
The name should be shorter and use Title Case.
```suggestion
.name("Ingest Status Polling Timeout")
.displayName("Ingest Status Polling Timeout")
```
##########
nifi-nar-bundles/nifi-azure-bundle/nifi-azure-processors/src/main/java/org/apache/nifi/processors/azure/data/explorer/PutAzureDataExplorer.java:
##########
@@ -0,0 +1,309 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.azure.data.explorer;
+
+import org.apache.commons.lang3.StringUtils;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.expression.ExpressionLanguageScope;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.services.azure.data.explorer.KustoIngestDataFormat;
+import org.apache.nifi.services.azure.data.explorer.KustoIngestService;
+import org.apache.nifi.services.azure.data.explorer.KustoIngestionRequest;
+import org.apache.nifi.services.azure.data.explorer.KustoIngestionResult;
+import org.apache.nifi.services.azure.data.explorer.KustoQueryResponse;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.net.URISyntaxException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+
+@Tags({"Azure", "Kusto", "ADX", "Explorer", "Data"})
+@CapabilityDescription("The PutAzureDataExplorer acts as a ADX sink connector which sends flowFiles using the ADX-Service to the provided Azure Data" +
+ "Explorer Ingest Endpoint. The data can be sent through queued ingestion or streaming ingestion to the Azure Data Explorer cluster.")
+public class PutAzureDataExplorer extends AbstractProcessor {
+
+ public static final String FETCH_TABLE_COMMAND = "%s | count";
+ public static final String STREAMING_POLICY_SHOW_COMMAND = ".show %s %s policy streamingingestion";
+ public static final String DATABASE = "database";
+
+ private List<PropertyDescriptor> descriptors;
+ private Set<Relationship> relationships;
+ private transient KustoIngestService service;
+ private boolean streamingEnabled;
+ private boolean pollOnIngestionStatus;
+
+ public static final AllowableValue IGNORE_FIRST_RECORD_YES = new AllowableValue(
+ "YES", "YES",
+ "Ignore first record during ingestion");
+
+ public static final AllowableValue IGNORE_FIRST_RECORD_NO = new AllowableValue(
+ "NO", "NO",
+ "Do not ignore first record during ingestion");
+
+ public static final PropertyDescriptor DATABASE_NAME = new PropertyDescriptor.Builder()
+ .name("Database Name")
+ .displayName("Database Name")
+ .description("Azure Data Explorer Database Name for querying")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+ .build();
+
+ public static final PropertyDescriptor TABLE_NAME = new PropertyDescriptor.Builder()
+ .name("Table Name")
+ .displayName("Table Name")
+ .description("Azure Data Explorer Table Name for ingesting data")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+ .build();
+
+ public static final PropertyDescriptor MAPPING_NAME = new PropertyDescriptor
+ .Builder().name("Ingest Mapping name")
+ .displayName("Ingest Mapping name")
+ .description("The name of the mapping responsible for storing the data in the appropriate columns.")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+
+ public static final PropertyDescriptor IS_STREAMING_ENABLED = new PropertyDescriptor
+ .Builder().name("Streaming enabled")
+ .displayName("Streaming enabled")
+ .description("This property determines whether we want to stream data to ADX.")
+ .required(false)
+ .allowableValues("true", "false")
+ .addValidator(StandardValidators.BOOLEAN_VALIDATOR)
+ .defaultValue("false")
+ .build();
+
+ public static final PropertyDescriptor POLL_ON_INGESTION_STATUS = new PropertyDescriptor
+ .Builder().name("Poll on ingestion status")
+ .displayName("Whether to Poll on ingestion status")
+ .description("This property determines whether we want to poll on ingestion status after an ingestion to ADX is completed")
+ .required(false)
+ .allowableValues("true", "false")
+ .addValidator(StandardValidators.BOOLEAN_VALIDATOR)
+ .defaultValue("false")
+ .build();
+ public static final PropertyDescriptor ADX_SERVICE = new PropertyDescriptor
+ .Builder().name("Kusto Ingest Service")
+ .displayName("Kusto Ingest Service")
+ .description("Azure Data Explorer Kusto Ingest Service")
+ .required(true)
+ .identifiesControllerService(KustoIngestService.class)
+ .build();
+
+ static final PropertyDescriptor DATA_FORMAT = new PropertyDescriptor.Builder()
+ .name("Data Format")
+ .displayName("Data Format")
+ .description("The format of the data that is sent to Azure Data Explorer.")
+ .required(true)
+ .allowableValues(KustoIngestDataFormat.values())
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+
+ static final PropertyDescriptor IGNORE_FIRST_RECORD = new PropertyDescriptor.Builder()
+ .name("Ingestion Ignore First Record")
+ .displayName("Ingestion Ignore First Record")
+ .description("Defines whether ignore first record while ingestion.")
+ .required(false)
+ .allowableValues(IGNORE_FIRST_RECORD_YES, IGNORE_FIRST_RECORD_NO)
+ .defaultValue(IGNORE_FIRST_RECORD_NO.getValue())
+ .build();
+
+ static final PropertyDescriptor ROUTE_PARTIALLY_SUCCESSFUL_INGESTION = new PropertyDescriptor.Builder()
+ .name("Route partially successful ingestion records")
+ .displayName("Route partially successful ingestion records")
+ .description("Defines where to route partially successful ingestion records.")
+ .required(false)
+ .allowableValues("Success", "Failure")
+ .defaultValue("Failure")
+ .build();
+
+ static final PropertyDescriptor INGESTION_STATUS_POLLING_TIMEOUT = new PropertyDescriptor.Builder()
+ .name("Timeout for polling on ingestion status")
+ .displayName("Timeout for polling on ingestion status in seconds")
+ .description("Defines the value of timeout for polling on ingestion status in seconds")
+ .required(false)
+ .dependsOn(POLL_ON_INGESTION_STATUS, "true")
+ .addValidator(StandardValidators.POSITIVE_INTEGER_VALIDATOR)
+ .defaultValue("600")
Review Comment:
The value should not be in seconds, but should use the available `TIME_PERIOD_VALIDATOR`, which supports expressions such as `5 minutes` or `30 s`.
##########
nifi-nar-bundles/nifi-azure-bundle/nifi-azure-processors/src/main/java/org/apache/nifi/processors/azure/data/explorer/PutAzureDataExplorer.java:
##########
@@ -0,0 +1,309 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.azure.data.explorer;
+
+import org.apache.commons.lang3.StringUtils;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.expression.ExpressionLanguageScope;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.services.azure.data.explorer.KustoIngestDataFormat;
+import org.apache.nifi.services.azure.data.explorer.KustoIngestService;
+import org.apache.nifi.services.azure.data.explorer.KustoIngestionRequest;
+import org.apache.nifi.services.azure.data.explorer.KustoIngestionResult;
+import org.apache.nifi.services.azure.data.explorer.KustoQueryResponse;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.net.URISyntaxException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+
+@Tags({"Azure", "Kusto", "ADX", "Explorer", "Data"})
+@CapabilityDescription("The PutAzureDataExplorer acts as a ADX sink connector which sends flowFiles using the ADX-Service to the provided Azure Data" +
+ "Explorer Ingest Endpoint. The data can be sent through queued ingestion or streaming ingestion to the Azure Data Explorer cluster.")
+public class PutAzureDataExplorer extends AbstractProcessor {
+
+ public static final String FETCH_TABLE_COMMAND = "%s | count";
+ public static final String STREAMING_POLICY_SHOW_COMMAND = ".show %s %s policy streamingingestion";
+ public static final String DATABASE = "database";
+
+ private List<PropertyDescriptor> descriptors;
+ private Set<Relationship> relationships;
+ private transient KustoIngestService service;
+ private boolean streamingEnabled;
+ private boolean pollOnIngestionStatus;
+
+ public static final AllowableValue IGNORE_FIRST_RECORD_YES = new AllowableValue(
+ "YES", "YES",
+ "Ignore first record during ingestion");
+
+ public static final AllowableValue IGNORE_FIRST_RECORD_NO = new AllowableValue(
+ "NO", "NO",
+ "Do not ignore first record during ingestion");
+
+ public static final PropertyDescriptor DATABASE_NAME = new PropertyDescriptor.Builder()
+ .name("Database Name")
+ .displayName("Database Name")
+ .description("Azure Data Explorer Database Name for querying")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+ .build();
+
+ public static final PropertyDescriptor TABLE_NAME = new PropertyDescriptor.Builder()
+ .name("Table Name")
+ .displayName("Table Name")
+ .description("Azure Data Explorer Table Name for ingesting data")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+ .build();
+
+ public static final PropertyDescriptor MAPPING_NAME = new PropertyDescriptor
+ .Builder().name("Ingest Mapping name")
+ .displayName("Ingest Mapping name")
+ .description("The name of the mapping responsible for storing the data in the appropriate columns.")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+
+ public static final PropertyDescriptor IS_STREAMING_ENABLED = new PropertyDescriptor
+ .Builder().name("Streaming enabled")
+ .displayName("Streaming enabled")
+ .description("This property determines whether we want to stream data to ADX.")
+ .required(false)
+ .allowableValues("true", "false")
+ .addValidator(StandardValidators.BOOLEAN_VALIDATOR)
+ .defaultValue("false")
+ .build();
+
+ public static final PropertyDescriptor POLL_ON_INGESTION_STATUS = new PropertyDescriptor
+ .Builder().name("Poll on ingestion status")
+ .displayName("Whether to Poll on ingestion status")
+ .description("This property determines whether we want to poll on ingestion status after an ingestion to ADX is completed")
+ .required(false)
+ .allowableValues("true", "false")
+ .addValidator(StandardValidators.BOOLEAN_VALIDATOR)
+ .defaultValue("false")
+ .build();
+ public static final PropertyDescriptor ADX_SERVICE = new PropertyDescriptor
+ .Builder().name("Kusto Ingest Service")
+ .displayName("Kusto Ingest Service")
+ .description("Azure Data Explorer Kusto Ingest Service")
+ .required(true)
+ .identifiesControllerService(KustoIngestService.class)
+ .build();
+
+ static final PropertyDescriptor DATA_FORMAT = new PropertyDescriptor.Builder()
+ .name("Data Format")
+ .displayName("Data Format")
+ .description("The format of the data that is sent to Azure Data Explorer.")
+ .required(true)
+ .allowableValues(KustoIngestDataFormat.values())
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+
+ static final PropertyDescriptor IGNORE_FIRST_RECORD = new PropertyDescriptor.Builder()
+ .name("Ingestion Ignore First Record")
+ .displayName("Ingestion Ignore First Record")
+ .description("Defines whether ignore first record while ingestion.")
+ .required(false)
+ .allowableValues(IGNORE_FIRST_RECORD_YES, IGNORE_FIRST_RECORD_NO)
+ .defaultValue(IGNORE_FIRST_RECORD_NO.getValue())
+ .build();
+
+ static final PropertyDescriptor ROUTE_PARTIALLY_SUCCESSFUL_INGESTION = new PropertyDescriptor.Builder()
+ .name("Route partially successful ingestion records")
+ .displayName("Route partially successful ingestion records")
+ .description("Defines where to route partially successful ingestion records.")
+ .required(false)
+ .allowableValues("Success", "Failure")
+ .defaultValue("Failure")
+ .build();
+
+ static final PropertyDescriptor INGESTION_STATUS_POLLING_TIMEOUT = new PropertyDescriptor.Builder()
+ .name("Timeout for polling on ingestion status")
+ .displayName("Timeout for polling on ingestion status in seconds")
+ .description("Defines the value of timeout for polling on ingestion status in seconds")
+ .required(false)
+ .dependsOn(POLL_ON_INGESTION_STATUS, "true")
+ .addValidator(StandardValidators.POSITIVE_INTEGER_VALIDATOR)
+ .defaultValue("600")
+ .build();
+
+ static final PropertyDescriptor INGESTION_STATUS_POLLING_INTERVAL = new PropertyDescriptor.Builder()
+ .name("Ingestion status polling interval")
+ .displayName("Ingestion status polling interval in seconds")
+ .description("Defines the value of timeout for polling on ingestion status in seconds.")
+ .required(false)
+ .dependsOn(POLL_ON_INGESTION_STATUS, "true")
+ .addValidator(StandardValidators.POSITIVE_INTEGER_VALIDATOR)
+ .defaultValue("5")
+ .build();
+
+ public static final Relationship SUCCESS = new Relationship.Builder()
+ .name("success")
+ .description("Relationship for success")
+ .build();
+ public static final Relationship FAILURE = new Relationship.Builder()
+ .name("failure")
+ .description("Relationship for failure")
+ .build();
+
+ @Override
+ protected void init(final ProcessorInitializationContext context) {
+ final List<PropertyDescriptor> descriptorList = new ArrayList<>();
+ descriptorList.add(ADX_SERVICE);
+ descriptorList.add(DATABASE_NAME);
+ descriptorList.add(TABLE_NAME);
+ descriptorList.add(MAPPING_NAME);
+ descriptorList.add(DATA_FORMAT);
+ descriptorList.add(IGNORE_FIRST_RECORD);
+ descriptorList.add(IS_STREAMING_ENABLED);
+ descriptorList.add(POLL_ON_INGESTION_STATUS);
+ descriptorList.add(ROUTE_PARTIALLY_SUCCESSFUL_INGESTION);
+ descriptorList.add(INGESTION_STATUS_POLLING_TIMEOUT);
+ descriptorList.add(INGESTION_STATUS_POLLING_INTERVAL);
+ this.descriptors = Collections.unmodifiableList(descriptorList);
+
+ final Set<Relationship> relationshipSet = new HashSet<>();
+ relationshipSet.add(SUCCESS);
+ relationshipSet.add(FAILURE);
+ this.relationships = Collections.unmodifiableSet(relationshipSet);
Review Comment:
These collections should be created in a static initializer instead, so this `init()` method can be removed.
##########
nifi-nar-bundles/nifi-azure-bundle/nifi-azure-processors/src/main/java/org/apache/nifi/processors/azure/data/explorer/PutAzureDataExplorer.java:
##########
@@ -0,0 +1,309 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.azure.data.explorer;
+
+import org.apache.commons.lang3.StringUtils;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.expression.ExpressionLanguageScope;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.services.azure.data.explorer.KustoIngestDataFormat;
+import org.apache.nifi.services.azure.data.explorer.KustoIngestService;
+import org.apache.nifi.services.azure.data.explorer.KustoIngestionRequest;
+import org.apache.nifi.services.azure.data.explorer.KustoIngestionResult;
+import org.apache.nifi.services.azure.data.explorer.KustoQueryResponse;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.net.URISyntaxException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+
+@Tags({"Azure", "Kusto", "ADX", "Explorer", "Data"})
+@CapabilityDescription("The PutAzureDataExplorer acts as a ADX sink connector which sends flowFiles using the ADX-Service to the provided Azure Data" +
+ "Explorer Ingest Endpoint. The data can be sent through queued ingestion or streaming ingestion to the Azure Data Explorer cluster.")
+public class PutAzureDataExplorer extends AbstractProcessor {
+
+ public static final String FETCH_TABLE_COMMAND = "%s | count";
+ public static final String STREAMING_POLICY_SHOW_COMMAND = ".show %s %s policy streamingingestion";
+ public static final String DATABASE = "database";
+
+ private List<PropertyDescriptor> descriptors;
+ private Set<Relationship> relationships;
+ private transient KustoIngestService service;
+ private boolean streamingEnabled;
+ private boolean pollOnIngestionStatus;
+
+ public static final AllowableValue IGNORE_FIRST_RECORD_YES = new AllowableValue(
+ "YES", "YES",
+ "Ignore first record during ingestion");
+
+ public static final AllowableValue IGNORE_FIRST_RECORD_NO = new AllowableValue(
+ "NO", "NO",
+ "Do not ignore first record during ingestion");
+
+ public static final PropertyDescriptor DATABASE_NAME = new PropertyDescriptor.Builder()
+ .name("Database Name")
+ .displayName("Database Name")
+ .description("Azure Data Explorer Database Name for querying")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+ .build();
+
+ public static final PropertyDescriptor TABLE_NAME = new PropertyDescriptor.Builder()
+ .name("Table Name")
+ .displayName("Table Name")
+ .description("Azure Data Explorer Table Name for ingesting data")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+ .build();
+
+ public static final PropertyDescriptor MAPPING_NAME = new PropertyDescriptor
+ .Builder().name("Ingest Mapping name")
+ .displayName("Ingest Mapping name")
+ .description("The name of the mapping responsible for storing the data in the appropriate columns.")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+
+ public static final PropertyDescriptor IS_STREAMING_ENABLED = new PropertyDescriptor
+ .Builder().name("Streaming enabled")
+ .displayName("Streaming enabled")
+ .description("This property determines whether we want to stream data to ADX.")
+ .required(false)
+ .allowableValues("true", "false")
+ .addValidator(StandardValidators.BOOLEAN_VALIDATOR)
+ .defaultValue("false")
+ .build();
+
+ public static final PropertyDescriptor POLL_ON_INGESTION_STATUS = new PropertyDescriptor
+ .Builder().name("Poll on ingestion status")
+ .displayName("Whether to Poll on ingestion status")
+ .description("This property determines whether we want to poll on ingestion status after an ingestion to ADX is completed")
+ .required(false)
+ .allowableValues("true", "false")
+ .addValidator(StandardValidators.BOOLEAN_VALIDATOR)
+ .defaultValue("false")
+ .build();
+ public static final PropertyDescriptor ADX_SERVICE = new PropertyDescriptor
+ .Builder().name("Kusto Ingest Service")
+ .displayName("Kusto Ingest Service")
+ .description("Azure Data Explorer Kusto Ingest Service")
+ .required(true)
+ .identifiesControllerService(KustoIngestService.class)
+ .build();
+
+ static final PropertyDescriptor DATA_FORMAT = new PropertyDescriptor.Builder()
+ .name("Data Format")
+ .displayName("Data Format")
+ .description("The format of the data that is sent to Azure Data Explorer.")
+ .required(true)
+ .allowableValues(KustoIngestDataFormat.values())
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+
+ static final PropertyDescriptor IGNORE_FIRST_RECORD = new PropertyDescriptor.Builder()
+ .name("Ingestion Ignore First Record")
+ .displayName("Ingestion Ignore First Record")
+ .description("Defines whether ignore first record while ingestion.")
+ .required(false)
+ .allowableValues(IGNORE_FIRST_RECORD_YES, IGNORE_FIRST_RECORD_NO)
+ .defaultValue(IGNORE_FIRST_RECORD_NO.getValue())
+ .build();
+
+ static final PropertyDescriptor ROUTE_PARTIALLY_SUCCESSFUL_INGESTION = new PropertyDescriptor.Builder()
+ .name("Route partially successful ingestion records")
+ .displayName("Route partially successful ingestion records")
+ .description("Defines where to route partially successful ingestion records.")
+ .required(false)
+ .allowableValues("Success", "Failure")
+ .defaultValue("Failure")
+ .build();
+
+ static final PropertyDescriptor INGESTION_STATUS_POLLING_TIMEOUT = new PropertyDescriptor.Builder()
+ .name("Timeout for polling on ingestion status")
+ .displayName("Timeout for polling on ingestion status in seconds")
+ .description("Defines the value of timeout for polling on ingestion status in seconds")
+ .required(false)
+ .dependsOn(POLL_ON_INGESTION_STATUS, "true")
+ .addValidator(StandardValidators.POSITIVE_INTEGER_VALIDATOR)
+ .defaultValue("600")
+ .build();
+
+ static final PropertyDescriptor INGESTION_STATUS_POLLING_INTERVAL = new PropertyDescriptor.Builder()
+ .name("Ingestion status polling interval")
+ .displayName("Ingestion status polling interval in seconds")
+ .description("Defines the value of timeout for polling on ingestion status in seconds.")
+ .required(false)
+ .dependsOn(POLL_ON_INGESTION_STATUS, "true")
+ .addValidator(StandardValidators.POSITIVE_INTEGER_VALIDATOR)
+ .defaultValue("5")
+ .build();
+
+ public static final Relationship SUCCESS = new Relationship.Builder()
+ .name("success")
+ .description("Relationship for success")
+ .build();
+ public static final Relationship FAILURE = new Relationship.Builder()
+ .name("failure")
+ .description("Relationship for failure")
+ .build();
+
+ @Override
+ protected void init(final ProcessorInitializationContext context) {
+ final List<PropertyDescriptor> descriptorList = new ArrayList<>();
+ descriptorList.add(ADX_SERVICE);
+ descriptorList.add(DATABASE_NAME);
+ descriptorList.add(TABLE_NAME);
+ descriptorList.add(MAPPING_NAME);
+ descriptorList.add(DATA_FORMAT);
+ descriptorList.add(IGNORE_FIRST_RECORD);
+ descriptorList.add(IS_STREAMING_ENABLED);
+ descriptorList.add(POLL_ON_INGESTION_STATUS);
+ descriptorList.add(ROUTE_PARTIALLY_SUCCESSFUL_INGESTION);
+ descriptorList.add(INGESTION_STATUS_POLLING_TIMEOUT);
+ descriptorList.add(INGESTION_STATUS_POLLING_INTERVAL);
+ this.descriptors = Collections.unmodifiableList(descriptorList);
+
+ final Set<Relationship> relationshipSet = new HashSet<>();
+ relationshipSet.add(SUCCESS);
+ relationshipSet.add(FAILURE);
+ this.relationships = Collections.unmodifiableSet(relationshipSet);
+ }
+
+ @Override
+ public Set<Relationship> getRelationships() {
+ return this.relationships;
+ }
+
+ @Override
+ public final List<PropertyDescriptor> getSupportedPropertyDescriptors() {
+ return descriptors;
+ }
+
+ @OnScheduled
+ public void onScheduled(final ProcessContext context) {
+ service = context.getProperty(ADX_SERVICE).asControllerService(KustoIngestService.class);
+ if (checkIfIngestorRoleDoesntExist(context.getProperty(DATABASE_NAME).getValue(), context.getProperty(TABLE_NAME).getValue())) {
+ getLogger().error("User might not have ingestor privileges, table validation will be skipped for all table mappings.");
+ throw new ProcessException("User might not have ingestor privileges, table validation will be skipped for all table mappings. ");
Review Comment:
It is not necessary to both log an error and throw an exception. Recommend removing the error log and include the database and table name in the exception message.
The message says that validation will be skipped, but throwing an exception prevents the Processor from starting, so the behavior should be clarified.
##########
nifi-nar-bundles/nifi-azure-bundle/nifi-azure-processors/src/main/java/org/apache/nifi/processors/azure/data/explorer/PutAzureDataExplorer.java:
##########
@@ -0,0 +1,309 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.azure.data.explorer;
+
+import org.apache.commons.lang3.StringUtils;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.expression.ExpressionLanguageScope;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.services.azure.data.explorer.KustoIngestDataFormat;
+import org.apache.nifi.services.azure.data.explorer.KustoIngestService;
+import org.apache.nifi.services.azure.data.explorer.KustoIngestionRequest;
+import org.apache.nifi.services.azure.data.explorer.KustoIngestionResult;
+import org.apache.nifi.services.azure.data.explorer.KustoQueryResponse;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.net.URISyntaxException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+
+@Tags({"Azure", "Kusto", "ADX", "Explorer", "Data"})
+@CapabilityDescription("The PutAzureDataExplorer acts as a ADX sink connector which sends flowFiles using the ADX-Service to the provided Azure Data" +
+ "Explorer Ingest Endpoint. The data can be sent through queued ingestion or streaming ingestion to the Azure Data Explorer cluster.")
+public class PutAzureDataExplorer extends AbstractProcessor {
+
+ public static final String FETCH_TABLE_COMMAND = "%s | count";
+ public static final String STREAMING_POLICY_SHOW_COMMAND = ".show %s %s policy streamingingestion";
+ public static final String DATABASE = "database";
+
+ private List<PropertyDescriptor> descriptors;
+ private Set<Relationship> relationships;
+ private transient KustoIngestService service;
+ private boolean streamingEnabled;
+ private boolean pollOnIngestionStatus;
+
+ public static final AllowableValue IGNORE_FIRST_RECORD_YES = new AllowableValue(
+ "YES", "YES",
+ "Ignore first record during ingestion");
+
+ public static final AllowableValue IGNORE_FIRST_RECORD_NO = new AllowableValue(
+ "NO", "NO",
+ "Do not ignore first record during ingestion");
+
+ public static final PropertyDescriptor DATABASE_NAME = new PropertyDescriptor.Builder()
+ .name("Database Name")
+ .displayName("Database Name")
+ .description("Azure Data Explorer Database Name for querying")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+ .build();
+
+ public static final PropertyDescriptor TABLE_NAME = new PropertyDescriptor.Builder()
+ .name("Table Name")
+ .displayName("Table Name")
+ .description("Azure Data Explorer Table Name for ingesting data")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+ .build();
+
+ public static final PropertyDescriptor MAPPING_NAME = new PropertyDescriptor
+ .Builder().name("Ingest Mapping name")
+ .displayName("Ingest Mapping name")
+ .description("The name of the mapping responsible for storing the data in the appropriate columns.")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+
+ public static final PropertyDescriptor IS_STREAMING_ENABLED = new PropertyDescriptor
+ .Builder().name("Streaming enabled")
+ .displayName("Streaming enabled")
+ .description("This property determines whether we want to stream data to ADX.")
+ .required(false)
+ .allowableValues("true", "false")
+ .addValidator(StandardValidators.BOOLEAN_VALIDATOR)
+ .defaultValue("false")
+ .build();
+
+ public static final PropertyDescriptor POLL_ON_INGESTION_STATUS = new PropertyDescriptor
+ .Builder().name("Poll on ingestion status")
+ .displayName("Whether to Poll on ingestion status")
+ .description("This property determines whether we want to poll on ingestion status after an ingestion to ADX is completed")
+ .required(false)
+ .allowableValues("true", "false")
+ .addValidator(StandardValidators.BOOLEAN_VALIDATOR)
+ .defaultValue("false")
+ .build();
+ public static final PropertyDescriptor ADX_SERVICE = new PropertyDescriptor
+ .Builder().name("Kusto Ingest Service")
+ .displayName("Kusto Ingest Service")
+ .description("Azure Data Explorer Kusto Ingest Service")
+ .required(true)
+ .identifiesControllerService(KustoIngestService.class)
+ .build();
+
+ static final PropertyDescriptor DATA_FORMAT = new PropertyDescriptor.Builder()
+ .name("Data Format")
+ .displayName("Data Format")
+ .description("The format of the data that is sent to Azure Data Explorer.")
+ .required(true)
+ .allowableValues(KustoIngestDataFormat.values())
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+
+ static final PropertyDescriptor IGNORE_FIRST_RECORD = new PropertyDescriptor.Builder()
+ .name("Ingestion Ignore First Record")
+ .displayName("Ingestion Ignore First Record")
+ .description("Defines whether ignore first record while ingestion.")
+ .required(false)
+ .allowableValues(IGNORE_FIRST_RECORD_YES, IGNORE_FIRST_RECORD_NO)
+ .defaultValue(IGNORE_FIRST_RECORD_NO.getValue())
+ .build();
+
+ static final PropertyDescriptor ROUTE_PARTIALLY_SUCCESSFUL_INGESTION = new PropertyDescriptor.Builder()
+ .name("Route partially successful ingestion records")
+ .displayName("Route partially successful ingestion records")
+ .description("Defines where to route partially successful ingestion records.")
+ .required(false)
+ .allowableValues("Success", "Failure")
+ .defaultValue("Failure")
+ .build();
+
+ static final PropertyDescriptor INGESTION_STATUS_POLLING_TIMEOUT = new PropertyDescriptor.Builder()
+ .name("Timeout for polling on ingestion status")
+ .displayName("Timeout for polling on ingestion status in seconds")
+ .description("Defines the value of timeout for polling on ingestion status in seconds")
+ .required(false)
+ .dependsOn(POLL_ON_INGESTION_STATUS, "true")
+ .addValidator(StandardValidators.POSITIVE_INTEGER_VALIDATOR)
+ .defaultValue("600")
+ .build();
+
+ static final PropertyDescriptor INGESTION_STATUS_POLLING_INTERVAL = new PropertyDescriptor.Builder()
+ .name("Ingestion status polling interval")
+ .displayName("Ingestion status polling interval in seconds")
+ .description("Defines the value of timeout for polling on ingestion status in seconds.")
+ .required(false)
+ .dependsOn(POLL_ON_INGESTION_STATUS, "true")
+ .addValidator(StandardValidators.POSITIVE_INTEGER_VALIDATOR)
+ .defaultValue("5")
+ .build();
+
+ public static final Relationship SUCCESS = new Relationship.Builder()
+ .name("success")
+ .description("Relationship for success")
+ .build();
+ public static final Relationship FAILURE = new Relationship.Builder()
+ .name("failure")
+ .description("Relationship for failure")
+ .build();
+
+ @Override
+ protected void init(final ProcessorInitializationContext context) {
+ final List<PropertyDescriptor> descriptorList = new ArrayList<>();
+ descriptorList.add(ADX_SERVICE);
+ descriptorList.add(DATABASE_NAME);
+ descriptorList.add(TABLE_NAME);
+ descriptorList.add(MAPPING_NAME);
+ descriptorList.add(DATA_FORMAT);
+ descriptorList.add(IGNORE_FIRST_RECORD);
+ descriptorList.add(IS_STREAMING_ENABLED);
+ descriptorList.add(POLL_ON_INGESTION_STATUS);
+ descriptorList.add(ROUTE_PARTIALLY_SUCCESSFUL_INGESTION);
+ descriptorList.add(INGESTION_STATUS_POLLING_TIMEOUT);
+ descriptorList.add(INGESTION_STATUS_POLLING_INTERVAL);
+ this.descriptors = Collections.unmodifiableList(descriptorList);
+
+ final Set<Relationship> relationshipSet = new HashSet<>();
+ relationshipSet.add(SUCCESS);
+ relationshipSet.add(FAILURE);
+ this.relationships = Collections.unmodifiableSet(relationshipSet);
+ }
+
+ @Override
+ public Set<Relationship> getRelationships() {
+ return this.relationships;
+ }
+
+ @Override
+ public final List<PropertyDescriptor> getSupportedPropertyDescriptors() {
+ return descriptors;
+ }
+
+ @OnScheduled
+ public void onScheduled(final ProcessContext context) {
+ service = context.getProperty(ADX_SERVICE).asControllerService(KustoIngestService.class);
+ if (checkIfIngestorRoleDoesntExist(context.getProperty(DATABASE_NAME).getValue(), context.getProperty(TABLE_NAME).getValue())) {
+ getLogger().error("User might not have ingestor privileges, table validation will be skipped for all table mappings.");
+ throw new ProcessException("User might not have ingestor privileges, table validation will be skipped for all table mappings. ");
+ }
+ streamingEnabled = context.getProperty(IS_STREAMING_ENABLED).evaluateAttributeExpressions().asBoolean();
+ if (streamingEnabled && !checkIfStreamingPolicyIsEnabledInADX(context.getProperty(DATABASE_NAME).getValue(), context.getProperty(DATABASE_NAME).getValue())) {
+ getLogger().error("Streaming policy is not enabled in database {}", context.getProperty(DATABASE_NAME).getValue());
+ throw new ProcessException("Streaming policy is not enabled in database {}" + context.getProperty(DATABASE_NAME).getValue());
+ }
+ }
+
+ @Override
+ public void onTrigger(final ProcessContext context, final ProcessSession session) throws ProcessException {
+ FlowFile flowFile = session.get();
+ if (flowFile == null) {
+ return;
+ }
+
+ String databaseName = context.getProperty(DATABASE_NAME).getValue();
+ String tableName = context.getProperty(TABLE_NAME).getValue();
+ String dataFormat = context.getProperty(DATA_FORMAT).getValue();
+ String mappingName = context.getProperty(MAPPING_NAME).getValue();
+ String routePartiallySuccessfulIngestion = context.getProperty(ROUTE_PARTIALLY_SUCCESSFUL_INGESTION).getValue();
+ String ingestionStatusPollingTimeout = context.getProperty(INGESTION_STATUS_POLLING_TIMEOUT).getValue();
+ String ingestionStatusPollingInterval = context.getProperty(INGESTION_STATUS_POLLING_INTERVAL).getValue();
+ String ignoreFirstRecord = context.getProperty(IGNORE_FIRST_RECORD).getValue();
+ streamingEnabled = context.getProperty(IS_STREAMING_ENABLED).evaluateAttributeExpressions().asBoolean();
+ pollOnIngestionStatus = context.getProperty(POLL_ON_INGESTION_STATUS).evaluateAttributeExpressions().asBoolean();
+
+ boolean isError = false;
+
+ // ingestion into ADX
+ try (final InputStream inputStream = session.read(flowFile)) {
+ StringBuilder ingestLogString = new StringBuilder().append("Ingesting with: ")
+ .append("dataFormat - ").append(dataFormat).append("|")
+ .append("ingestionMappingName - ").append(mappingName).append("|")
+ .append("databaseName - ").append(databaseName).append("|")
+ .append("tableName - ").append(tableName).append("|")
+ .append("pollOnIngestionStatus - ").append(pollOnIngestionStatus).append("|")
+ .append("ingestionStatusPollingTimeout - ").append(ingestionStatusPollingTimeout).append("|")
+ .append("ingestionStatusPollingInterval - ").append(ingestionStatusPollingInterval).append("|")
+ .append("routePartiallySuccessfulIngestion - ").append(routePartiallySuccessfulIngestion);
+ getLogger().debug(ingestLogString.toString());
Review Comment:
This StringBuilder approach is not efficient because it requires a processing for concatenation just to create a debug log. The StringBuilder should be removed and replaced with a parameterized log statement.
##########
nifi-nar-bundles/nifi-azure-bundle/nifi-azure-processors/src/main/java/org/apache/nifi/processors/azure/data/explorer/PutAzureDataExplorer.java:
##########
@@ -0,0 +1,309 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.azure.data.explorer;
+
+import org.apache.commons.lang3.StringUtils;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.expression.ExpressionLanguageScope;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.services.azure.data.explorer.KustoIngestDataFormat;
+import org.apache.nifi.services.azure.data.explorer.KustoIngestService;
+import org.apache.nifi.services.azure.data.explorer.KustoIngestionRequest;
+import org.apache.nifi.services.azure.data.explorer.KustoIngestionResult;
+import org.apache.nifi.services.azure.data.explorer.KustoQueryResponse;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.net.URISyntaxException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+
+@Tags({"Azure", "Kusto", "ADX", "Explorer", "Data"})
+@CapabilityDescription("The PutAzureDataExplorer acts as a ADX sink connector which sends flowFiles using the ADX-Service to the provided Azure Data" +
+ "Explorer Ingest Endpoint. The data can be sent through queued ingestion or streaming ingestion to the Azure Data Explorer cluster.")
+public class PutAzureDataExplorer extends AbstractProcessor {
+
+ public static final String FETCH_TABLE_COMMAND = "%s | count";
+ public static final String STREAMING_POLICY_SHOW_COMMAND = ".show %s %s policy streamingingestion";
+ public static final String DATABASE = "database";
+
+ private List<PropertyDescriptor> descriptors;
+ private Set<Relationship> relationships;
+ private transient KustoIngestService service;
+ private boolean streamingEnabled;
+ private boolean pollOnIngestionStatus;
+
+ public static final AllowableValue IGNORE_FIRST_RECORD_YES = new AllowableValue(
+ "YES", "YES",
+ "Ignore first record during ingestion");
+
+ public static final AllowableValue IGNORE_FIRST_RECORD_NO = new AllowableValue(
+ "NO", "NO",
+ "Do not ignore first record during ingestion");
+
+ public static final PropertyDescriptor DATABASE_NAME = new PropertyDescriptor.Builder()
+ .name("Database Name")
+ .displayName("Database Name")
+ .description("Azure Data Explorer Database Name for querying")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+ .build();
+
+ public static final PropertyDescriptor TABLE_NAME = new PropertyDescriptor.Builder()
+ .name("Table Name")
+ .displayName("Table Name")
+ .description("Azure Data Explorer Table Name for ingesting data")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+ .build();
+
+ public static final PropertyDescriptor MAPPING_NAME = new PropertyDescriptor
+ .Builder().name("Ingest Mapping name")
+ .displayName("Ingest Mapping name")
+ .description("The name of the mapping responsible for storing the data in the appropriate columns.")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+
+ public static final PropertyDescriptor IS_STREAMING_ENABLED = new PropertyDescriptor
+ .Builder().name("Streaming enabled")
+ .displayName("Streaming enabled")
+ .description("This property determines whether we want to stream data to ADX.")
+ .required(false)
+ .allowableValues("true", "false")
+ .addValidator(StandardValidators.BOOLEAN_VALIDATOR)
+ .defaultValue("false")
+ .build();
+
+ public static final PropertyDescriptor POLL_ON_INGESTION_STATUS = new PropertyDescriptor
+ .Builder().name("Poll on ingestion status")
+ .displayName("Whether to Poll on ingestion status")
+ .description("This property determines whether we want to poll on ingestion status after an ingestion to ADX is completed")
+ .required(false)
+ .allowableValues("true", "false")
+ .addValidator(StandardValidators.BOOLEAN_VALIDATOR)
+ .defaultValue("false")
+ .build();
+ public static final PropertyDescriptor ADX_SERVICE = new PropertyDescriptor
+ .Builder().name("Kusto Ingest Service")
+ .displayName("Kusto Ingest Service")
+ .description("Azure Data Explorer Kusto Ingest Service")
+ .required(true)
+ .identifiesControllerService(KustoIngestService.class)
+ .build();
+
+ static final PropertyDescriptor DATA_FORMAT = new PropertyDescriptor.Builder()
+ .name("Data Format")
+ .displayName("Data Format")
+ .description("The format of the data that is sent to Azure Data Explorer.")
+ .required(true)
+ .allowableValues(KustoIngestDataFormat.values())
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+
+ static final PropertyDescriptor IGNORE_FIRST_RECORD = new PropertyDescriptor.Builder()
+ .name("Ingestion Ignore First Record")
+ .displayName("Ingestion Ignore First Record")
+ .description("Defines whether ignore first record while ingestion.")
+ .required(false)
+ .allowableValues(IGNORE_FIRST_RECORD_YES, IGNORE_FIRST_RECORD_NO)
+ .defaultValue(IGNORE_FIRST_RECORD_NO.getValue())
+ .build();
+
+ static final PropertyDescriptor ROUTE_PARTIALLY_SUCCESSFUL_INGESTION = new PropertyDescriptor.Builder()
+ .name("Route partially successful ingestion records")
+ .displayName("Route partially successful ingestion records")
+ .description("Defines where to route partially successful ingestion records.")
+ .required(false)
+ .allowableValues("Success", "Failure")
+ .defaultValue("Failure")
+ .build();
+
+ static final PropertyDescriptor INGESTION_STATUS_POLLING_TIMEOUT = new PropertyDescriptor.Builder()
+ .name("Timeout for polling on ingestion status")
+ .displayName("Timeout for polling on ingestion status in seconds")
+ .description("Defines the value of timeout for polling on ingestion status in seconds")
+ .required(false)
+ .dependsOn(POLL_ON_INGESTION_STATUS, "true")
+ .addValidator(StandardValidators.POSITIVE_INTEGER_VALIDATOR)
+ .defaultValue("600")
+ .build();
+
+ static final PropertyDescriptor INGESTION_STATUS_POLLING_INTERVAL = new PropertyDescriptor.Builder()
+ .name("Ingestion status polling interval")
+ .displayName("Ingestion status polling interval in seconds")
+ .description("Defines the value of timeout for polling on ingestion status in seconds.")
+ .required(false)
+ .dependsOn(POLL_ON_INGESTION_STATUS, "true")
+ .addValidator(StandardValidators.POSITIVE_INTEGER_VALIDATOR)
+ .defaultValue("5")
+ .build();
+
+ public static final Relationship SUCCESS = new Relationship.Builder()
+ .name("success")
+ .description("Relationship for success")
+ .build();
+ public static final Relationship FAILURE = new Relationship.Builder()
+ .name("failure")
+ .description("Relationship for failure")
+ .build();
+
+ @Override
+ protected void init(final ProcessorInitializationContext context) {
+ final List<PropertyDescriptor> descriptorList = new ArrayList<>();
+ descriptorList.add(ADX_SERVICE);
+ descriptorList.add(DATABASE_NAME);
+ descriptorList.add(TABLE_NAME);
+ descriptorList.add(MAPPING_NAME);
+ descriptorList.add(DATA_FORMAT);
+ descriptorList.add(IGNORE_FIRST_RECORD);
+ descriptorList.add(IS_STREAMING_ENABLED);
+ descriptorList.add(POLL_ON_INGESTION_STATUS);
+ descriptorList.add(ROUTE_PARTIALLY_SUCCESSFUL_INGESTION);
+ descriptorList.add(INGESTION_STATUS_POLLING_TIMEOUT);
+ descriptorList.add(INGESTION_STATUS_POLLING_INTERVAL);
+ this.descriptors = Collections.unmodifiableList(descriptorList);
+
+ final Set<Relationship> relationshipSet = new HashSet<>();
+ relationshipSet.add(SUCCESS);
+ relationshipSet.add(FAILURE);
+ this.relationships = Collections.unmodifiableSet(relationshipSet);
+ }
+
+ @Override
+ public Set<Relationship> getRelationships() {
+ return this.relationships;
+ }
+
+ @Override
+ public final List<PropertyDescriptor> getSupportedPropertyDescriptors() {
+ return descriptors;
+ }
+
+ @OnScheduled
+ public void onScheduled(final ProcessContext context) {
+ service = context.getProperty(ADX_SERVICE).asControllerService(KustoIngestService.class);
+ if (checkIfIngestorRoleDoesntExist(context.getProperty(DATABASE_NAME).getValue(), context.getProperty(TABLE_NAME).getValue())) {
+ getLogger().error("User might not have ingestor privileges, table validation will be skipped for all table mappings.");
+ throw new ProcessException("User might not have ingestor privileges, table validation will be skipped for all table mappings. ");
+ }
+ streamingEnabled = context.getProperty(IS_STREAMING_ENABLED).evaluateAttributeExpressions().asBoolean();
+ if (streamingEnabled && !checkIfStreamingPolicyIsEnabledInADX(context.getProperty(DATABASE_NAME).getValue(), context.getProperty(DATABASE_NAME).getValue())) {
+ getLogger().error("Streaming policy is not enabled in database {}", context.getProperty(DATABASE_NAME).getValue());
+ throw new ProcessException("Streaming policy is not enabled in database {}" + context.getProperty(DATABASE_NAME).getValue());
Review Comment:
The `{}` placeholder is for log messages, but that doesn't work for exception messages.
##########
nifi-nar-bundles/nifi-azure-bundle/nifi-azure-processors/src/main/java/org/apache/nifi/services/azure/data/explorer/StandardKustoIngestService.java:
##########
@@ -0,0 +1,362 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.services.azure.data.explorer;
+
+import com.microsoft.azure.kusto.data.Client;
+import com.microsoft.azure.kusto.data.ClientFactory;
+import com.microsoft.azure.kusto.data.KustoResultSetTable;
+import com.microsoft.azure.kusto.data.auth.ConnectionStringBuilder;
+import com.microsoft.azure.kusto.data.exceptions.DataClientException;
+import com.microsoft.azure.kusto.data.exceptions.DataServiceException;
+import com.microsoft.azure.kusto.ingest.IngestClientFactory;
+import com.microsoft.azure.kusto.ingest.IngestionMapping;
+import com.microsoft.azure.kusto.ingest.IngestionProperties;
+import com.microsoft.azure.kusto.ingest.ManagedStreamingIngestClient;
+import com.microsoft.azure.kusto.ingest.QueuedIngestClient;
+import com.microsoft.azure.kusto.ingest.exceptions.IngestionClientException;
+import com.microsoft.azure.kusto.ingest.exceptions.IngestionServiceException;
+import com.microsoft.azure.kusto.ingest.result.IngestionResult;
+import com.microsoft.azure.kusto.ingest.result.IngestionStatus;
+import com.microsoft.azure.kusto.ingest.result.OperationStatus;
+import com.microsoft.azure.kusto.ingest.source.StreamSourceInfo;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnEnabled;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.controller.AbstractControllerService;
+import org.apache.nifi.controller.ConfigurationContext;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.reporting.InitializationException;
+
+import java.io.IOException;
+import java.net.URISyntaxException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.List;
+import java.util.Objects;
+
+@Tags({"Azure", "ADX", "Kusto", "ingest", "azure"})
+@CapabilityDescription("Sends batches of flowfile content or stream flowfile content to an Azure ADX cluster.")
+public class StandardKustoIngestService extends AbstractControllerService implements KustoIngestService {
+
+ public static final PropertyDescriptor AUTHENTICATION_STRATEGY = new PropertyDescriptor
+ .Builder().name("Authentication Strategy")
+ .displayName("Authentication Strategy")
+ .description("Authentication method for access to Azure Data Explorer")
+ .required(true)
+ .defaultValue(KustoAuthenticationStrategy.MANAGED_IDENTITY.getValue())
+ .allowableValues(KustoAuthenticationStrategy.class)
+ .build();
+
+ public static final PropertyDescriptor APPLICATION_CLIENT_ID = new PropertyDescriptor
+ .Builder().name("Application Client ID")
+ .displayName("Application Client ID")
+ .description("Azure Data Explorer Application Client Identifier for Authentication")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+
+ public static final PropertyDescriptor APPLICATION_KEY = new PropertyDescriptor
+ .Builder().name("Application Key")
+ .displayName("Application Key")
+ .description("Azure Data Explorer Application Key for Authentication")
+ .required(true)
+ .sensitive(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .dependsOn(AUTHENTICATION_STRATEGY, KustoAuthenticationStrategy.APPLICATION_CREDENTIALS.getValue())
+ .build();
+
+ public static final PropertyDescriptor APPLICATION_TENANT_ID = new PropertyDescriptor.Builder()
+ .name("Application Tenant ID")
+ .displayName("Application Tenant ID")
+ .description("Azure Data Explorer Application Tenant Identifier for Authentication")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .dependsOn(AUTHENTICATION_STRATEGY, KustoAuthenticationStrategy.APPLICATION_CREDENTIALS.getValue())
+ .build();
+
+ public static final PropertyDescriptor CLUSTER_URI = new PropertyDescriptor
+ .Builder().name("Cluster URI")
+ .displayName("Cluster URI")
+ .description("Azure Data Explorer Cluster URI")
+ .required(true)
+ .addValidator(StandardValidators.URL_VALIDATOR)
+ .build();
+
+ private static final List<PropertyDescriptor> PROPERTY_DESCRIPTORS = Collections.unmodifiableList(Arrays.asList(
+ AUTHENTICATION_STRATEGY,
+ APPLICATION_CLIENT_ID,
+ APPLICATION_KEY,
+ APPLICATION_TENANT_ID,
+ CLUSTER_URI));
+
+ public static final Pair<String, String> NIFI_SINK = Pair.of("processor", "nifi-sink");
Review Comment:
Instead of using the word `nifi-sink`, it seems better to use the component class simple name.
```suggestion
public static final Pair<String, String> NIFI_SINK = Pair.of("processor", StandardKustoIngestService.class.getSimpleName());
```
##########
nifi-nar-bundles/nifi-azure-bundle/nifi-azure-processors/src/main/java/org/apache/nifi/processors/azure/data/explorer/PutAzureDataExplorer.java:
##########
@@ -0,0 +1,309 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.azure.data.explorer;
+
+import org.apache.commons.lang3.StringUtils;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.expression.ExpressionLanguageScope;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.services.azure.data.explorer.KustoIngestDataFormat;
+import org.apache.nifi.services.azure.data.explorer.KustoIngestService;
+import org.apache.nifi.services.azure.data.explorer.KustoIngestionRequest;
+import org.apache.nifi.services.azure.data.explorer.KustoIngestionResult;
+import org.apache.nifi.services.azure.data.explorer.KustoQueryResponse;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.net.URISyntaxException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+
+@Tags({"Azure", "Kusto", "ADX", "Explorer", "Data"})
+@CapabilityDescription("The PutAzureDataExplorer acts as a ADX sink connector which sends flowFiles using the ADX-Service to the provided Azure Data" +
+ "Explorer Ingest Endpoint. The data can be sent through queued ingestion or streaming ingestion to the Azure Data Explorer cluster.")
+public class PutAzureDataExplorer extends AbstractProcessor {
+
+ public static final String FETCH_TABLE_COMMAND = "%s | count";
+ public static final String STREAMING_POLICY_SHOW_COMMAND = ".show %s %s policy streamingingestion";
+ public static final String DATABASE = "database";
+
+ private List<PropertyDescriptor> descriptors;
+ private Set<Relationship> relationships;
+ private transient KustoIngestService service;
+ private boolean streamingEnabled;
+ private boolean pollOnIngestionStatus;
+
+ public static final AllowableValue IGNORE_FIRST_RECORD_YES = new AllowableValue(
+ "YES", "YES",
+ "Ignore first record during ingestion");
+
+ public static final AllowableValue IGNORE_FIRST_RECORD_NO = new AllowableValue(
+ "NO", "NO",
+ "Do not ignore first record during ingestion");
+
+ public static final PropertyDescriptor DATABASE_NAME = new PropertyDescriptor.Builder()
+ .name("Database Name")
+ .displayName("Database Name")
+ .description("Azure Data Explorer Database Name for querying")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+ .build();
+
+ public static final PropertyDescriptor TABLE_NAME = new PropertyDescriptor.Builder()
+ .name("Table Name")
+ .displayName("Table Name")
+ .description("Azure Data Explorer Table Name for ingesting data")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+ .build();
+
+ public static final PropertyDescriptor MAPPING_NAME = new PropertyDescriptor
+ .Builder().name("Ingest Mapping name")
+ .displayName("Ingest Mapping name")
+ .description("The name of the mapping responsible for storing the data in the appropriate columns.")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+
+ public static final PropertyDescriptor IS_STREAMING_ENABLED = new PropertyDescriptor
+ .Builder().name("Streaming enabled")
+ .displayName("Streaming enabled")
+ .description("This property determines whether we want to stream data to ADX.")
+ .required(false)
+ .allowableValues("true", "false")
+ .addValidator(StandardValidators.BOOLEAN_VALIDATOR)
+ .defaultValue("false")
+ .build();
+
+ public static final PropertyDescriptor POLL_ON_INGESTION_STATUS = new PropertyDescriptor
+ .Builder().name("Poll on ingestion status")
+ .displayName("Whether to Poll on ingestion status")
+ .description("This property determines whether we want to poll on ingestion status after an ingestion to ADX is completed")
+ .required(false)
+ .allowableValues("true", "false")
+ .addValidator(StandardValidators.BOOLEAN_VALIDATOR)
+ .defaultValue("false")
+ .build();
+ public static final PropertyDescriptor ADX_SERVICE = new PropertyDescriptor
+ .Builder().name("Kusto Ingest Service")
+ .displayName("Kusto Ingest Service")
+ .description("Azure Data Explorer Kusto Ingest Service")
+ .required(true)
+ .identifiesControllerService(KustoIngestService.class)
+ .build();
+
+ static final PropertyDescriptor DATA_FORMAT = new PropertyDescriptor.Builder()
+ .name("Data Format")
+ .displayName("Data Format")
+ .description("The format of the data that is sent to Azure Data Explorer.")
+ .required(true)
+ .allowableValues(KustoIngestDataFormat.values())
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+
+ static final PropertyDescriptor IGNORE_FIRST_RECORD = new PropertyDescriptor.Builder()
+ .name("Ingestion Ignore First Record")
+ .displayName("Ingestion Ignore First Record")
+ .description("Defines whether ignore first record while ingestion.")
+ .required(false)
+ .allowableValues(IGNORE_FIRST_RECORD_YES, IGNORE_FIRST_RECORD_NO)
+ .defaultValue(IGNORE_FIRST_RECORD_NO.getValue())
+ .build();
+
+ static final PropertyDescriptor ROUTE_PARTIALLY_SUCCESSFUL_INGESTION = new PropertyDescriptor.Builder()
+ .name("Route partially successful ingestion records")
+ .displayName("Route partially successful ingestion records")
+ .description("Defines where to route partially successful ingestion records.")
+ .required(false)
+ .allowableValues("Success", "Failure")
+ .defaultValue("Failure")
+ .build();
+
+ static final PropertyDescriptor INGESTION_STATUS_POLLING_TIMEOUT = new PropertyDescriptor.Builder()
+ .name("Timeout for polling on ingestion status")
+ .displayName("Timeout for polling on ingestion status in seconds")
+ .description("Defines the value of timeout for polling on ingestion status in seconds")
+ .required(false)
+ .dependsOn(POLL_ON_INGESTION_STATUS, "true")
+ .addValidator(StandardValidators.POSITIVE_INTEGER_VALIDATOR)
+ .defaultValue("600")
+ .build();
+
+ static final PropertyDescriptor INGESTION_STATUS_POLLING_INTERVAL = new PropertyDescriptor.Builder()
+ .name("Ingestion status polling interval")
+ .displayName("Ingestion status polling interval in seconds")
+ .description("Defines the value of timeout for polling on ingestion status in seconds.")
+ .required(false)
+ .dependsOn(POLL_ON_INGESTION_STATUS, "true")
+ .addValidator(StandardValidators.POSITIVE_INTEGER_VALIDATOR)
+ .defaultValue("5")
+ .build();
+
+ public static final Relationship SUCCESS = new Relationship.Builder()
+ .name("success")
+ .description("Relationship for success")
+ .build();
+ public static final Relationship FAILURE = new Relationship.Builder()
+ .name("failure")
+ .description("Relationship for failure")
+ .build();
+
+ @Override
+ protected void init(final ProcessorInitializationContext context) {
+ final List<PropertyDescriptor> descriptorList = new ArrayList<>();
+ descriptorList.add(ADX_SERVICE);
+ descriptorList.add(DATABASE_NAME);
+ descriptorList.add(TABLE_NAME);
+ descriptorList.add(MAPPING_NAME);
+ descriptorList.add(DATA_FORMAT);
+ descriptorList.add(IGNORE_FIRST_RECORD);
+ descriptorList.add(IS_STREAMING_ENABLED);
+ descriptorList.add(POLL_ON_INGESTION_STATUS);
+ descriptorList.add(ROUTE_PARTIALLY_SUCCESSFUL_INGESTION);
+ descriptorList.add(INGESTION_STATUS_POLLING_TIMEOUT);
+ descriptorList.add(INGESTION_STATUS_POLLING_INTERVAL);
+ this.descriptors = Collections.unmodifiableList(descriptorList);
+
+ final Set<Relationship> relationshipSet = new HashSet<>();
+ relationshipSet.add(SUCCESS);
+ relationshipSet.add(FAILURE);
+ this.relationships = Collections.unmodifiableSet(relationshipSet);
+ }
+
+ @Override
+ public Set<Relationship> getRelationships() {
+ return this.relationships;
+ }
+
+ @Override
+ public final List<PropertyDescriptor> getSupportedPropertyDescriptors() {
+ return descriptors;
+ }
+
+ @OnScheduled
+ public void onScheduled(final ProcessContext context) {
+ service = context.getProperty(ADX_SERVICE).asControllerService(KustoIngestService.class);
+ if (checkIfIngestorRoleDoesntExist(context.getProperty(DATABASE_NAME).getValue(), context.getProperty(TABLE_NAME).getValue())) {
+ getLogger().error("User might not have ingestor privileges, table validation will be skipped for all table mappings.");
+ throw new ProcessException("User might not have ingestor privileges, table validation will be skipped for all table mappings. ");
+ }
+ streamingEnabled = context.getProperty(IS_STREAMING_ENABLED).evaluateAttributeExpressions().asBoolean();
+ if (streamingEnabled && !checkIfStreamingPolicyIsEnabledInADX(context.getProperty(DATABASE_NAME).getValue(), context.getProperty(DATABASE_NAME).getValue())) {
+ getLogger().error("Streaming policy is not enabled in database {}", context.getProperty(DATABASE_NAME).getValue());
+ throw new ProcessException("Streaming policy is not enabled in database {}" + context.getProperty(DATABASE_NAME).getValue());
+ }
+ }
+
+ @Override
+ public void onTrigger(final ProcessContext context, final ProcessSession session) throws ProcessException {
+ FlowFile flowFile = session.get();
+ if (flowFile == null) {
+ return;
+ }
+
+ String databaseName = context.getProperty(DATABASE_NAME).getValue();
+ String tableName = context.getProperty(TABLE_NAME).getValue();
+ String dataFormat = context.getProperty(DATA_FORMAT).getValue();
+ String mappingName = context.getProperty(MAPPING_NAME).getValue();
+ String routePartiallySuccessfulIngestion = context.getProperty(ROUTE_PARTIALLY_SUCCESSFUL_INGESTION).getValue();
+ String ingestionStatusPollingTimeout = context.getProperty(INGESTION_STATUS_POLLING_TIMEOUT).getValue();
+ String ingestionStatusPollingInterval = context.getProperty(INGESTION_STATUS_POLLING_INTERVAL).getValue();
+ String ignoreFirstRecord = context.getProperty(IGNORE_FIRST_RECORD).getValue();
+ streamingEnabled = context.getProperty(IS_STREAMING_ENABLED).evaluateAttributeExpressions().asBoolean();
+ pollOnIngestionStatus = context.getProperty(POLL_ON_INGESTION_STATUS).evaluateAttributeExpressions().asBoolean();
+
+ boolean isError = false;
+
+ // ingestion into ADX
+ try (final InputStream inputStream = session.read(flowFile)) {
+ StringBuilder ingestLogString = new StringBuilder().append("Ingesting with: ")
+ .append("dataFormat - ").append(dataFormat).append("|")
+ .append("ingestionMappingName - ").append(mappingName).append("|")
+ .append("databaseName - ").append(databaseName).append("|")
+ .append("tableName - ").append(tableName).append("|")
+ .append("pollOnIngestionStatus - ").append(pollOnIngestionStatus).append("|")
+ .append("ingestionStatusPollingTimeout - ").append(ingestionStatusPollingTimeout).append("|")
+ .append("ingestionStatusPollingInterval - ").append(ingestionStatusPollingInterval).append("|")
+ .append("routePartiallySuccessfulIngestion - ").append(routePartiallySuccessfulIngestion);
+ getLogger().debug(ingestLogString.toString());
+
+ KustoIngestionResult result = service.ingestData(
+ new KustoIngestionRequest(streamingEnabled, pollOnIngestionStatus, inputStream, databaseName,
+ tableName, dataFormat, mappingName, ignoreFirstRecord, ingestionStatusPollingTimeout, ingestionStatusPollingInterval)
+ );
+ getLogger().info("Ingest Status Polling Enabled {} and Ingest Status Polling Enabled {} ", pollOnIngestionStatus, result.toString());
+
+ if (result == KustoIngestionResult.SUCCEEDED) {
+ getLogger().info("Ingest Completed - {}", result.getStatus());
+ } else if (result == KustoIngestionResult.FAILED) {
+ getLogger().error("Ingest Failed - {}", result.getStatus());
+ isError = true;
+ } else if (result == KustoIngestionResult.PARTIALLY_SUCCEEDED) {
+ getLogger().warn("Ingest Partially succeeded - {}", result.getStatus());
+ flowFile = session.putAttribute(flowFile, "ingestion_status", "partial_success");
+ if (StringUtils.equalsIgnoreCase(routePartiallySuccessfulIngestion, "Failure")) {
+ isError = true;
+ }
+ }
+
+ } catch (IOException | URISyntaxException e) {
+ getLogger().error("Azure Data Explorer Ingest processing failed", e);
+ isError = true;
+ }
+
+ if (isError) {
+ getLogger().error("Ingest processing failed {}", flowFile);
+ session.transfer(flowFile, FAILURE);
+ }else {
+ getLogger().info("Ingest processing completed {}", flowFile);
+ session.transfer(flowFile, SUCCESS);
+ }
+ }
+
+ protected boolean checkIfStreamingPolicyIsEnabledInADX(String entityName, String database) {
+ KustoQueryResponse kustoQueryResponse = service.executeQuery(database, String.format(STREAMING_POLICY_SHOW_COMMAND, PutAzureDataExplorer.DATABASE, entityName));
+ if (kustoQueryResponse.isError()) {
+ throw new ProcessException(String.format("Error occurred while checking if streaming policy is enabled for the table for entity %s in database %s",entityName,database));
+ }
+ List<List<Object>> queryResult = kustoQueryResponse.getQueryResult();;
+ if ( queryResult.get(0) !=null && queryResult.get(0).get(2)!=null && StringUtils.isNotEmpty(queryResult.get(0).get(2).toString())) {
Review Comment:
This conditional is difficult to follow based on the construction and formatting. The `List<List<Object>>` structure is also very generic, which makes this difficult to understand. If this is a value that should always be checked, recommend evaluating the value in the service and providing a separate named property.
##########
nifi-nar-bundles/nifi-azure-bundle/nifi-azure-processors/src/main/java/org/apache/nifi/processors/azure/data/explorer/PutAzureDataExplorer.java:
##########
@@ -0,0 +1,309 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.azure.data.explorer;
+
+import org.apache.commons.lang3.StringUtils;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.expression.ExpressionLanguageScope;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.services.azure.data.explorer.KustoIngestDataFormat;
+import org.apache.nifi.services.azure.data.explorer.KustoIngestService;
+import org.apache.nifi.services.azure.data.explorer.KustoIngestionRequest;
+import org.apache.nifi.services.azure.data.explorer.KustoIngestionResult;
+import org.apache.nifi.services.azure.data.explorer.KustoQueryResponse;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.net.URISyntaxException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+
+@Tags({"Azure", "Kusto", "ADX", "Explorer", "Data"})
+@CapabilityDescription("The PutAzureDataExplorer acts as a ADX sink connector which sends flowFiles using the ADX-Service to the provided Azure Data" +
+ "Explorer Ingest Endpoint. The data can be sent through queued ingestion or streaming ingestion to the Azure Data Explorer cluster.")
+public class PutAzureDataExplorer extends AbstractProcessor {
+
+ public static final String FETCH_TABLE_COMMAND = "%s | count";
+ public static final String STREAMING_POLICY_SHOW_COMMAND = ".show %s %s policy streamingingestion";
+ public static final String DATABASE = "database";
+
+ private List<PropertyDescriptor> descriptors;
+ private Set<Relationship> relationships;
+ private transient KustoIngestService service;
+ private boolean streamingEnabled;
+ private boolean pollOnIngestionStatus;
+
+ public static final AllowableValue IGNORE_FIRST_RECORD_YES = new AllowableValue(
+ "YES", "YES",
+ "Ignore first record during ingestion");
+
+ public static final AllowableValue IGNORE_FIRST_RECORD_NO = new AllowableValue(
+ "NO", "NO",
+ "Do not ignore first record during ingestion");
+
+ public static final PropertyDescriptor DATABASE_NAME = new PropertyDescriptor.Builder()
+ .name("Database Name")
+ .displayName("Database Name")
+ .description("Azure Data Explorer Database Name for querying")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+ .build();
+
+ public static final PropertyDescriptor TABLE_NAME = new PropertyDescriptor.Builder()
+ .name("Table Name")
+ .displayName("Table Name")
+ .description("Azure Data Explorer Table Name for ingesting data")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+ .build();
+
+ public static final PropertyDescriptor MAPPING_NAME = new PropertyDescriptor
+ .Builder().name("Ingest Mapping name")
+ .displayName("Ingest Mapping name")
+ .description("The name of the mapping responsible for storing the data in the appropriate columns.")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+
+ public static final PropertyDescriptor IS_STREAMING_ENABLED = new PropertyDescriptor
+ .Builder().name("Streaming enabled")
+ .displayName("Streaming enabled")
+ .description("This property determines whether we want to stream data to ADX.")
+ .required(false)
+ .allowableValues("true", "false")
+ .addValidator(StandardValidators.BOOLEAN_VALIDATOR)
+ .defaultValue("false")
+ .build();
+
+ public static final PropertyDescriptor POLL_ON_INGESTION_STATUS = new PropertyDescriptor
+ .Builder().name("Poll on ingestion status")
+ .displayName("Whether to Poll on ingestion status")
+ .description("This property determines whether we want to poll on ingestion status after an ingestion to ADX is completed")
+ .required(false)
+ .allowableValues("true", "false")
+ .addValidator(StandardValidators.BOOLEAN_VALIDATOR)
+ .defaultValue("false")
+ .build();
+ public static final PropertyDescriptor ADX_SERVICE = new PropertyDescriptor
+ .Builder().name("Kusto Ingest Service")
+ .displayName("Kusto Ingest Service")
+ .description("Azure Data Explorer Kusto Ingest Service")
+ .required(true)
+ .identifiesControllerService(KustoIngestService.class)
+ .build();
+
+ static final PropertyDescriptor DATA_FORMAT = new PropertyDescriptor.Builder()
+ .name("Data Format")
+ .displayName("Data Format")
+ .description("The format of the data that is sent to Azure Data Explorer.")
+ .required(true)
+ .allowableValues(KustoIngestDataFormat.values())
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+
+ static final PropertyDescriptor IGNORE_FIRST_RECORD = new PropertyDescriptor.Builder()
+ .name("Ingestion Ignore First Record")
+ .displayName("Ingestion Ignore First Record")
+ .description("Defines whether ignore first record while ingestion.")
+ .required(false)
+ .allowableValues(IGNORE_FIRST_RECORD_YES, IGNORE_FIRST_RECORD_NO)
+ .defaultValue(IGNORE_FIRST_RECORD_NO.getValue())
+ .build();
+
+ static final PropertyDescriptor ROUTE_PARTIALLY_SUCCESSFUL_INGESTION = new PropertyDescriptor.Builder()
+ .name("Route partially successful ingestion records")
+ .displayName("Route partially successful ingestion records")
+ .description("Defines where to route partially successful ingestion records.")
+ .required(false)
+ .allowableValues("Success", "Failure")
+ .defaultValue("Failure")
+ .build();
+
+ static final PropertyDescriptor INGESTION_STATUS_POLLING_TIMEOUT = new PropertyDescriptor.Builder()
+ .name("Timeout for polling on ingestion status")
+ .displayName("Timeout for polling on ingestion status in seconds")
+ .description("Defines the value of timeout for polling on ingestion status in seconds")
+ .required(false)
+ .dependsOn(POLL_ON_INGESTION_STATUS, "true")
+ .addValidator(StandardValidators.POSITIVE_INTEGER_VALIDATOR)
+ .defaultValue("600")
+ .build();
+
+ static final PropertyDescriptor INGESTION_STATUS_POLLING_INTERVAL = new PropertyDescriptor.Builder()
+ .name("Ingestion status polling interval")
+ .displayName("Ingestion status polling interval in seconds")
+ .description("Defines the value of timeout for polling on ingestion status in seconds.")
+ .required(false)
+ .dependsOn(POLL_ON_INGESTION_STATUS, "true")
+ .addValidator(StandardValidators.POSITIVE_INTEGER_VALIDATOR)
+ .defaultValue("5")
+ .build();
+
+ public static final Relationship SUCCESS = new Relationship.Builder()
+ .name("success")
+ .description("Relationship for success")
+ .build();
+ public static final Relationship FAILURE = new Relationship.Builder()
+ .name("failure")
+ .description("Relationship for failure")
+ .build();
+
+ @Override
+ protected void init(final ProcessorInitializationContext context) {
+ final List<PropertyDescriptor> descriptorList = new ArrayList<>();
+ descriptorList.add(ADX_SERVICE);
+ descriptorList.add(DATABASE_NAME);
+ descriptorList.add(TABLE_NAME);
+ descriptorList.add(MAPPING_NAME);
+ descriptorList.add(DATA_FORMAT);
+ descriptorList.add(IGNORE_FIRST_RECORD);
+ descriptorList.add(IS_STREAMING_ENABLED);
+ descriptorList.add(POLL_ON_INGESTION_STATUS);
+ descriptorList.add(ROUTE_PARTIALLY_SUCCESSFUL_INGESTION);
+ descriptorList.add(INGESTION_STATUS_POLLING_TIMEOUT);
+ descriptorList.add(INGESTION_STATUS_POLLING_INTERVAL);
+ this.descriptors = Collections.unmodifiableList(descriptorList);
+
+ final Set<Relationship> relationshipSet = new HashSet<>();
+ relationshipSet.add(SUCCESS);
+ relationshipSet.add(FAILURE);
+ this.relationships = Collections.unmodifiableSet(relationshipSet);
+ }
+
+ @Override
+ public Set<Relationship> getRelationships() {
+ return this.relationships;
+ }
+
+ @Override
+ public final List<PropertyDescriptor> getSupportedPropertyDescriptors() {
+ return descriptors;
+ }
+
+ @OnScheduled
+ public void onScheduled(final ProcessContext context) {
+ service = context.getProperty(ADX_SERVICE).asControllerService(KustoIngestService.class);
+ if (checkIfIngestorRoleDoesntExist(context.getProperty(DATABASE_NAME).getValue(), context.getProperty(TABLE_NAME).getValue())) {
+ getLogger().error("User might not have ingestor privileges, table validation will be skipped for all table mappings.");
+ throw new ProcessException("User might not have ingestor privileges, table validation will be skipped for all table mappings. ");
+ }
+ streamingEnabled = context.getProperty(IS_STREAMING_ENABLED).evaluateAttributeExpressions().asBoolean();
+ if (streamingEnabled && !checkIfStreamingPolicyIsEnabledInADX(context.getProperty(DATABASE_NAME).getValue(), context.getProperty(DATABASE_NAME).getValue())) {
+ getLogger().error("Streaming policy is not enabled in database {}", context.getProperty(DATABASE_NAME).getValue());
+ throw new ProcessException("Streaming policy is not enabled in database {}" + context.getProperty(DATABASE_NAME).getValue());
+ }
+ }
+
+ @Override
+ public void onTrigger(final ProcessContext context, final ProcessSession session) throws ProcessException {
+ FlowFile flowFile = session.get();
+ if (flowFile == null) {
+ return;
+ }
+
+ String databaseName = context.getProperty(DATABASE_NAME).getValue();
+ String tableName = context.getProperty(TABLE_NAME).getValue();
+ String dataFormat = context.getProperty(DATA_FORMAT).getValue();
+ String mappingName = context.getProperty(MAPPING_NAME).getValue();
+ String routePartiallySuccessfulIngestion = context.getProperty(ROUTE_PARTIALLY_SUCCESSFUL_INGESTION).getValue();
+ String ingestionStatusPollingTimeout = context.getProperty(INGESTION_STATUS_POLLING_TIMEOUT).getValue();
+ String ingestionStatusPollingInterval = context.getProperty(INGESTION_STATUS_POLLING_INTERVAL).getValue();
+ String ignoreFirstRecord = context.getProperty(IGNORE_FIRST_RECORD).getValue();
+ streamingEnabled = context.getProperty(IS_STREAMING_ENABLED).evaluateAttributeExpressions().asBoolean();
+ pollOnIngestionStatus = context.getProperty(POLL_ON_INGESTION_STATUS).evaluateAttributeExpressions().asBoolean();
+
+ boolean isError = false;
+
+ // ingestion into ADX
+ try (final InputStream inputStream = session.read(flowFile)) {
+ StringBuilder ingestLogString = new StringBuilder().append("Ingesting with: ")
+ .append("dataFormat - ").append(dataFormat).append("|")
+ .append("ingestionMappingName - ").append(mappingName).append("|")
+ .append("databaseName - ").append(databaseName).append("|")
+ .append("tableName - ").append(tableName).append("|")
+ .append("pollOnIngestionStatus - ").append(pollOnIngestionStatus).append("|")
+ .append("ingestionStatusPollingTimeout - ").append(ingestionStatusPollingTimeout).append("|")
+ .append("ingestionStatusPollingInterval - ").append(ingestionStatusPollingInterval).append("|")
+ .append("routePartiallySuccessfulIngestion - ").append(routePartiallySuccessfulIngestion);
+ getLogger().debug(ingestLogString.toString());
+
+ KustoIngestionResult result = service.ingestData(
+ new KustoIngestionRequest(streamingEnabled, pollOnIngestionStatus, inputStream, databaseName,
+ tableName, dataFormat, mappingName, ignoreFirstRecord, ingestionStatusPollingTimeout, ingestionStatusPollingInterval)
+ );
+ getLogger().info("Ingest Status Polling Enabled {} and Ingest Status Polling Enabled {} ", pollOnIngestionStatus, result.toString());
Review Comment:
The `result.toString()` is not necessary. This seems more appropriate as a debug log.
```suggestion
getLogger().debug("Ingest Status Polling Enabled [{}] and Ingest Status Polling Enabled {}", pollOnIngestionStatus, result);
```
##########
nifi-nar-bundles/nifi-azure-bundle/nifi-azure-services-api/src/main/java/org/apache/nifi/services/azure/data/explorer/KustoIngestionRequest.java:
##########
@@ -0,0 +1,100 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.services.azure.data.explorer;
+
+import java.io.InputStream;
+
+public class KustoIngestionRequest {
+
+ private boolean streamingEnabled;
+ private boolean pollOnIngestionStatus;
+ private InputStream inputStream;
+ private String dataFormat;
+ private String mappingName;
+
+ private String databaseName;
+
+ private String tableName;
+
+ private String ignoreFirstRecord;
+
+ private String ingestionStatusPollingTimeout;
+
+ private String ingestionStatusPollingInterval;
Review Comment:
As noted elsewhere, these properties should be changed to `java.time.Duration`.
##########
nifi-nar-bundles/nifi-azure-bundle/nifi-azure-processors/src/main/java/org/apache/nifi/services/azure/data/explorer/StandardKustoIngestService.java:
##########
@@ -0,0 +1,362 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.services.azure.data.explorer;
+
+import com.microsoft.azure.kusto.data.Client;
+import com.microsoft.azure.kusto.data.ClientFactory;
+import com.microsoft.azure.kusto.data.KustoResultSetTable;
+import com.microsoft.azure.kusto.data.auth.ConnectionStringBuilder;
+import com.microsoft.azure.kusto.data.exceptions.DataClientException;
+import com.microsoft.azure.kusto.data.exceptions.DataServiceException;
+import com.microsoft.azure.kusto.ingest.IngestClientFactory;
+import com.microsoft.azure.kusto.ingest.IngestionMapping;
+import com.microsoft.azure.kusto.ingest.IngestionProperties;
+import com.microsoft.azure.kusto.ingest.ManagedStreamingIngestClient;
+import com.microsoft.azure.kusto.ingest.QueuedIngestClient;
+import com.microsoft.azure.kusto.ingest.exceptions.IngestionClientException;
+import com.microsoft.azure.kusto.ingest.exceptions.IngestionServiceException;
+import com.microsoft.azure.kusto.ingest.result.IngestionResult;
+import com.microsoft.azure.kusto.ingest.result.IngestionStatus;
+import com.microsoft.azure.kusto.ingest.result.OperationStatus;
+import com.microsoft.azure.kusto.ingest.source.StreamSourceInfo;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnEnabled;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.controller.AbstractControllerService;
+import org.apache.nifi.controller.ConfigurationContext;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.reporting.InitializationException;
+
+import java.io.IOException;
+import java.net.URISyntaxException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.List;
+import java.util.Objects;
+
+@Tags({"Azure", "ADX", "Kusto", "ingest", "azure"})
+@CapabilityDescription("Sends batches of flowfile content or stream flowfile content to an Azure ADX cluster.")
+public class StandardKustoIngestService extends AbstractControllerService implements KustoIngestService {
+
+ public static final PropertyDescriptor AUTHENTICATION_STRATEGY = new PropertyDescriptor
+ .Builder().name("Authentication Strategy")
+ .displayName("Authentication Strategy")
+ .description("Authentication method for access to Azure Data Explorer")
+ .required(true)
+ .defaultValue(KustoAuthenticationStrategy.MANAGED_IDENTITY.getValue())
+ .allowableValues(KustoAuthenticationStrategy.class)
+ .build();
+
+ public static final PropertyDescriptor APPLICATION_CLIENT_ID = new PropertyDescriptor
+ .Builder().name("Application Client ID")
+ .displayName("Application Client ID")
+ .description("Azure Data Explorer Application Client Identifier for Authentication")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+
+ public static final PropertyDescriptor APPLICATION_KEY = new PropertyDescriptor
+ .Builder().name("Application Key")
+ .displayName("Application Key")
+ .description("Azure Data Explorer Application Key for Authentication")
+ .required(true)
+ .sensitive(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .dependsOn(AUTHENTICATION_STRATEGY, KustoAuthenticationStrategy.APPLICATION_CREDENTIALS.getValue())
+ .build();
+
+ public static final PropertyDescriptor APPLICATION_TENANT_ID = new PropertyDescriptor.Builder()
+ .name("Application Tenant ID")
+ .displayName("Application Tenant ID")
+ .description("Azure Data Explorer Application Tenant Identifier for Authentication")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .dependsOn(AUTHENTICATION_STRATEGY, KustoAuthenticationStrategy.APPLICATION_CREDENTIALS.getValue())
+ .build();
+
+ public static final PropertyDescriptor CLUSTER_URI = new PropertyDescriptor
+ .Builder().name("Cluster URI")
+ .displayName("Cluster URI")
+ .description("Azure Data Explorer Cluster URI")
+ .required(true)
+ .addValidator(StandardValidators.URL_VALIDATOR)
+ .build();
+
+ private static final List<PropertyDescriptor> PROPERTY_DESCRIPTORS = Collections.unmodifiableList(Arrays.asList(
+ AUTHENTICATION_STRATEGY,
+ APPLICATION_CLIENT_ID,
+ APPLICATION_KEY,
+ APPLICATION_TENANT_ID,
+ CLUSTER_URI));
+
+ public static final Pair<String, String> NIFI_SINK = Pair.of("processor", "nifi-sink");
+
+ private volatile QueuedIngestClient queuedIngestClient;
+
+ private volatile ManagedStreamingIngestClient managedStreamingIngestClient;
+
+ private volatile Client executionClient;
+
+ @Override
+ public List<PropertyDescriptor> getSupportedPropertyDescriptors() {
+ return PROPERTY_DESCRIPTORS;
+ }
+
+ /**
+ * @param context the configuration context
+ * @throws InitializationException if unable to create a database connection
Review Comment:
This `throws` line is not correct. It is probably better to remove the Javadoc comment if it is not completely written.
##########
nifi-nar-bundles/nifi-azure-bundle/nifi-azure-processors/src/main/java/org/apache/nifi/processors/azure/data/explorer/PutAzureDataExplorer.java:
##########
@@ -0,0 +1,309 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.azure.data.explorer;
+
+import org.apache.commons.lang3.StringUtils;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.expression.ExpressionLanguageScope;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.services.azure.data.explorer.KustoIngestDataFormat;
+import org.apache.nifi.services.azure.data.explorer.KustoIngestService;
+import org.apache.nifi.services.azure.data.explorer.KustoIngestionRequest;
+import org.apache.nifi.services.azure.data.explorer.KustoIngestionResult;
+import org.apache.nifi.services.azure.data.explorer.KustoQueryResponse;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.net.URISyntaxException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+
+@Tags({"Azure", "Kusto", "ADX", "Explorer", "Data"})
+@CapabilityDescription("The PutAzureDataExplorer acts as a ADX sink connector which sends flowFiles using the ADX-Service to the provided Azure Data" +
+ "Explorer Ingest Endpoint. The data can be sent through queued ingestion or streaming ingestion to the Azure Data Explorer cluster.")
+public class PutAzureDataExplorer extends AbstractProcessor {
+
+ public static final String FETCH_TABLE_COMMAND = "%s | count";
+ public static final String STREAMING_POLICY_SHOW_COMMAND = ".show %s %s policy streamingingestion";
+ public static final String DATABASE = "database";
+
+ private List<PropertyDescriptor> descriptors;
+ private Set<Relationship> relationships;
+ private transient KustoIngestService service;
+ private boolean streamingEnabled;
+ private boolean pollOnIngestionStatus;
+
+ public static final AllowableValue IGNORE_FIRST_RECORD_YES = new AllowableValue(
+ "YES", "YES",
+ "Ignore first record during ingestion");
+
+ public static final AllowableValue IGNORE_FIRST_RECORD_NO = new AllowableValue(
+ "NO", "NO",
+ "Do not ignore first record during ingestion");
+
+ public static final PropertyDescriptor DATABASE_NAME = new PropertyDescriptor.Builder()
+ .name("Database Name")
+ .displayName("Database Name")
+ .description("Azure Data Explorer Database Name for querying")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+ .build();
+
+ public static final PropertyDescriptor TABLE_NAME = new PropertyDescriptor.Builder()
+ .name("Table Name")
+ .displayName("Table Name")
+ .description("Azure Data Explorer Table Name for ingesting data")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+ .build();
+
+ public static final PropertyDescriptor MAPPING_NAME = new PropertyDescriptor
+ .Builder().name("Ingest Mapping name")
+ .displayName("Ingest Mapping name")
+ .description("The name of the mapping responsible for storing the data in the appropriate columns.")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+
+ public static final PropertyDescriptor IS_STREAMING_ENABLED = new PropertyDescriptor
+ .Builder().name("Streaming enabled")
+ .displayName("Streaming enabled")
+ .description("This property determines whether we want to stream data to ADX.")
+ .required(false)
+ .allowableValues("true", "false")
+ .addValidator(StandardValidators.BOOLEAN_VALIDATOR)
+ .defaultValue("false")
+ .build();
+
+ public static final PropertyDescriptor POLL_ON_INGESTION_STATUS = new PropertyDescriptor
+ .Builder().name("Poll on ingestion status")
+ .displayName("Whether to Poll on ingestion status")
+ .description("This property determines whether we want to poll on ingestion status after an ingestion to ADX is completed")
+ .required(false)
+ .allowableValues("true", "false")
+ .addValidator(StandardValidators.BOOLEAN_VALIDATOR)
+ .defaultValue("false")
+ .build();
+ public static final PropertyDescriptor ADX_SERVICE = new PropertyDescriptor
+ .Builder().name("Kusto Ingest Service")
+ .displayName("Kusto Ingest Service")
+ .description("Azure Data Explorer Kusto Ingest Service")
+ .required(true)
+ .identifiesControllerService(KustoIngestService.class)
+ .build();
+
+ static final PropertyDescriptor DATA_FORMAT = new PropertyDescriptor.Builder()
+ .name("Data Format")
+ .displayName("Data Format")
+ .description("The format of the data that is sent to Azure Data Explorer.")
+ .required(true)
+ .allowableValues(KustoIngestDataFormat.values())
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+
+ static final PropertyDescriptor IGNORE_FIRST_RECORD = new PropertyDescriptor.Builder()
+ .name("Ingestion Ignore First Record")
+ .displayName("Ingestion Ignore First Record")
+ .description("Defines whether ignore first record while ingestion.")
+ .required(false)
+ .allowableValues(IGNORE_FIRST_RECORD_YES, IGNORE_FIRST_RECORD_NO)
+ .defaultValue(IGNORE_FIRST_RECORD_NO.getValue())
+ .build();
+
+ static final PropertyDescriptor ROUTE_PARTIALLY_SUCCESSFUL_INGESTION = new PropertyDescriptor.Builder()
+ .name("Route partially successful ingestion records")
+ .displayName("Route partially successful ingestion records")
+ .description("Defines where to route partially successful ingestion records.")
+ .required(false)
+ .allowableValues("Success", "Failure")
+ .defaultValue("Failure")
+ .build();
+
+ static final PropertyDescriptor INGESTION_STATUS_POLLING_TIMEOUT = new PropertyDescriptor.Builder()
+ .name("Timeout for polling on ingestion status")
+ .displayName("Timeout for polling on ingestion status in seconds")
+ .description("Defines the value of timeout for polling on ingestion status in seconds")
+ .required(false)
+ .dependsOn(POLL_ON_INGESTION_STATUS, "true")
+ .addValidator(StandardValidators.POSITIVE_INTEGER_VALIDATOR)
+ .defaultValue("600")
+ .build();
+
+ static final PropertyDescriptor INGESTION_STATUS_POLLING_INTERVAL = new PropertyDescriptor.Builder()
+ .name("Ingestion status polling interval")
+ .displayName("Ingestion status polling interval in seconds")
+ .description("Defines the value of timeout for polling on ingestion status in seconds.")
+ .required(false)
+ .dependsOn(POLL_ON_INGESTION_STATUS, "true")
+ .addValidator(StandardValidators.POSITIVE_INTEGER_VALIDATOR)
+ .defaultValue("5")
+ .build();
+
+ public static final Relationship SUCCESS = new Relationship.Builder()
+ .name("success")
+ .description("Relationship for success")
+ .build();
+ public static final Relationship FAILURE = new Relationship.Builder()
+ .name("failure")
+ .description("Relationship for failure")
+ .build();
+
+ @Override
+ protected void init(final ProcessorInitializationContext context) {
+ final List<PropertyDescriptor> descriptorList = new ArrayList<>();
+ descriptorList.add(ADX_SERVICE);
+ descriptorList.add(DATABASE_NAME);
+ descriptorList.add(TABLE_NAME);
+ descriptorList.add(MAPPING_NAME);
+ descriptorList.add(DATA_FORMAT);
+ descriptorList.add(IGNORE_FIRST_RECORD);
+ descriptorList.add(IS_STREAMING_ENABLED);
+ descriptorList.add(POLL_ON_INGESTION_STATUS);
+ descriptorList.add(ROUTE_PARTIALLY_SUCCESSFUL_INGESTION);
+ descriptorList.add(INGESTION_STATUS_POLLING_TIMEOUT);
+ descriptorList.add(INGESTION_STATUS_POLLING_INTERVAL);
+ this.descriptors = Collections.unmodifiableList(descriptorList);
+
+ final Set<Relationship> relationshipSet = new HashSet<>();
+ relationshipSet.add(SUCCESS);
+ relationshipSet.add(FAILURE);
+ this.relationships = Collections.unmodifiableSet(relationshipSet);
+ }
+
+ @Override
+ public Set<Relationship> getRelationships() {
+ return this.relationships;
+ }
+
+ @Override
+ public final List<PropertyDescriptor> getSupportedPropertyDescriptors() {
+ return descriptors;
+ }
+
+ @OnScheduled
+ public void onScheduled(final ProcessContext context) {
+ service = context.getProperty(ADX_SERVICE).asControllerService(KustoIngestService.class);
+ if (checkIfIngestorRoleDoesntExist(context.getProperty(DATABASE_NAME).getValue(), context.getProperty(TABLE_NAME).getValue())) {
+ getLogger().error("User might not have ingestor privileges, table validation will be skipped for all table mappings.");
+ throw new ProcessException("User might not have ingestor privileges, table validation will be skipped for all table mappings. ");
+ }
+ streamingEnabled = context.getProperty(IS_STREAMING_ENABLED).evaluateAttributeExpressions().asBoolean();
+ if (streamingEnabled && !checkIfStreamingPolicyIsEnabledInADX(context.getProperty(DATABASE_NAME).getValue(), context.getProperty(DATABASE_NAME).getValue())) {
+ getLogger().error("Streaming policy is not enabled in database {}", context.getProperty(DATABASE_NAME).getValue());
+ throw new ProcessException("Streaming policy is not enabled in database {}" + context.getProperty(DATABASE_NAME).getValue());
+ }
+ }
+
+ @Override
+ public void onTrigger(final ProcessContext context, final ProcessSession session) throws ProcessException {
+ FlowFile flowFile = session.get();
+ if (flowFile == null) {
+ return;
+ }
+
+ String databaseName = context.getProperty(DATABASE_NAME).getValue();
+ String tableName = context.getProperty(TABLE_NAME).getValue();
+ String dataFormat = context.getProperty(DATA_FORMAT).getValue();
+ String mappingName = context.getProperty(MAPPING_NAME).getValue();
+ String routePartiallySuccessfulIngestion = context.getProperty(ROUTE_PARTIALLY_SUCCESSFUL_INGESTION).getValue();
+ String ingestionStatusPollingTimeout = context.getProperty(INGESTION_STATUS_POLLING_TIMEOUT).getValue();
+ String ingestionStatusPollingInterval = context.getProperty(INGESTION_STATUS_POLLING_INTERVAL).getValue();
+ String ignoreFirstRecord = context.getProperty(IGNORE_FIRST_RECORD).getValue();
+ streamingEnabled = context.getProperty(IS_STREAMING_ENABLED).evaluateAttributeExpressions().asBoolean();
+ pollOnIngestionStatus = context.getProperty(POLL_ON_INGESTION_STATUS).evaluateAttributeExpressions().asBoolean();
+
+ boolean isError = false;
+
+ // ingestion into ADX
+ try (final InputStream inputStream = session.read(flowFile)) {
+ StringBuilder ingestLogString = new StringBuilder().append("Ingesting with: ")
+ .append("dataFormat - ").append(dataFormat).append("|")
+ .append("ingestionMappingName - ").append(mappingName).append("|")
+ .append("databaseName - ").append(databaseName).append("|")
+ .append("tableName - ").append(tableName).append("|")
+ .append("pollOnIngestionStatus - ").append(pollOnIngestionStatus).append("|")
+ .append("ingestionStatusPollingTimeout - ").append(ingestionStatusPollingTimeout).append("|")
+ .append("ingestionStatusPollingInterval - ").append(ingestionStatusPollingInterval).append("|")
+ .append("routePartiallySuccessfulIngestion - ").append(routePartiallySuccessfulIngestion);
+ getLogger().debug(ingestLogString.toString());
+
+ KustoIngestionResult result = service.ingestData(
+ new KustoIngestionRequest(streamingEnabled, pollOnIngestionStatus, inputStream, databaseName,
+ tableName, dataFormat, mappingName, ignoreFirstRecord, ingestionStatusPollingTimeout, ingestionStatusPollingInterval)
+ );
+ getLogger().info("Ingest Status Polling Enabled {} and Ingest Status Polling Enabled {} ", pollOnIngestionStatus, result.toString());
+
+ if (result == KustoIngestionResult.SUCCEEDED) {
+ getLogger().info("Ingest Completed - {}", result.getStatus());
+ } else if (result == KustoIngestionResult.FAILED) {
+ getLogger().error("Ingest Failed - {}", result.getStatus());
+ isError = true;
+ } else if (result == KustoIngestionResult.PARTIALLY_SUCCEEDED) {
+ getLogger().warn("Ingest Partially succeeded - {}", result.getStatus());
+ flowFile = session.putAttribute(flowFile, "ingestion_status", "partial_success");
+ if (StringUtils.equalsIgnoreCase(routePartiallySuccessfulIngestion, "Failure")) {
+ isError = true;
+ }
+ }
+
+ } catch (IOException | URISyntaxException e) {
+ getLogger().error("Azure Data Explorer Ingest processing failed", e);
+ isError = true;
+ }
+
+ if (isError) {
+ getLogger().error("Ingest processing failed {}", flowFile);
+ session.transfer(flowFile, FAILURE);
+ }else {
+ getLogger().info("Ingest processing completed {}", flowFile);
+ session.transfer(flowFile, SUCCESS);
+ }
+ }
+
+ protected boolean checkIfStreamingPolicyIsEnabledInADX(String entityName, String database) {
+ KustoQueryResponse kustoQueryResponse = service.executeQuery(database, String.format(STREAMING_POLICY_SHOW_COMMAND, PutAzureDataExplorer.DATABASE, entityName));
+ if (kustoQueryResponse.isError()) {
+ throw new ProcessException(String.format("Error occurred while checking if streaming policy is enabled for the table for entity %s in database %s",entityName,database));
Review Comment:
```suggestion
throw new ProcessException(String.format("Error occurred while checking if streaming policy is enabled for the table for entity %s in database %s", entityName, database));
```
##########
nifi-nar-bundles/nifi-azure-bundle/nifi-azure-processors/src/main/java/org/apache/nifi/services/azure/data/explorer/StandardKustoIngestService.java:
##########
@@ -0,0 +1,362 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.services.azure.data.explorer;
+
+import com.microsoft.azure.kusto.data.Client;
+import com.microsoft.azure.kusto.data.ClientFactory;
+import com.microsoft.azure.kusto.data.KustoResultSetTable;
+import com.microsoft.azure.kusto.data.auth.ConnectionStringBuilder;
+import com.microsoft.azure.kusto.data.exceptions.DataClientException;
+import com.microsoft.azure.kusto.data.exceptions.DataServiceException;
+import com.microsoft.azure.kusto.ingest.IngestClientFactory;
+import com.microsoft.azure.kusto.ingest.IngestionMapping;
+import com.microsoft.azure.kusto.ingest.IngestionProperties;
+import com.microsoft.azure.kusto.ingest.ManagedStreamingIngestClient;
+import com.microsoft.azure.kusto.ingest.QueuedIngestClient;
+import com.microsoft.azure.kusto.ingest.exceptions.IngestionClientException;
+import com.microsoft.azure.kusto.ingest.exceptions.IngestionServiceException;
+import com.microsoft.azure.kusto.ingest.result.IngestionResult;
+import com.microsoft.azure.kusto.ingest.result.IngestionStatus;
+import com.microsoft.azure.kusto.ingest.result.OperationStatus;
+import com.microsoft.azure.kusto.ingest.source.StreamSourceInfo;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnEnabled;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.controller.AbstractControllerService;
+import org.apache.nifi.controller.ConfigurationContext;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.reporting.InitializationException;
+
+import java.io.IOException;
+import java.net.URISyntaxException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.List;
+import java.util.Objects;
+
+@Tags({"Azure", "ADX", "Kusto", "ingest", "azure"})
+@CapabilityDescription("Sends batches of flowfile content or stream flowfile content to an Azure ADX cluster.")
+public class StandardKustoIngestService extends AbstractControllerService implements KustoIngestService {
+
+ public static final PropertyDescriptor AUTHENTICATION_STRATEGY = new PropertyDescriptor
+ .Builder().name("Authentication Strategy")
+ .displayName("Authentication Strategy")
+ .description("Authentication method for access to Azure Data Explorer")
+ .required(true)
+ .defaultValue(KustoAuthenticationStrategy.MANAGED_IDENTITY.getValue())
+ .allowableValues(KustoAuthenticationStrategy.class)
+ .build();
+
+ public static final PropertyDescriptor APPLICATION_CLIENT_ID = new PropertyDescriptor
+ .Builder().name("Application Client ID")
+ .displayName("Application Client ID")
+ .description("Azure Data Explorer Application Client Identifier for Authentication")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+
+ public static final PropertyDescriptor APPLICATION_KEY = new PropertyDescriptor
+ .Builder().name("Application Key")
+ .displayName("Application Key")
+ .description("Azure Data Explorer Application Key for Authentication")
+ .required(true)
+ .sensitive(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .dependsOn(AUTHENTICATION_STRATEGY, KustoAuthenticationStrategy.APPLICATION_CREDENTIALS.getValue())
+ .build();
+
+ public static final PropertyDescriptor APPLICATION_TENANT_ID = new PropertyDescriptor.Builder()
+ .name("Application Tenant ID")
+ .displayName("Application Tenant ID")
+ .description("Azure Data Explorer Application Tenant Identifier for Authentication")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .dependsOn(AUTHENTICATION_STRATEGY, KustoAuthenticationStrategy.APPLICATION_CREDENTIALS.getValue())
+ .build();
+
+ public static final PropertyDescriptor CLUSTER_URI = new PropertyDescriptor
+ .Builder().name("Cluster URI")
+ .displayName("Cluster URI")
+ .description("Azure Data Explorer Cluster URI")
+ .required(true)
+ .addValidator(StandardValidators.URL_VALIDATOR)
+ .build();
+
+ private static final List<PropertyDescriptor> PROPERTY_DESCRIPTORS = Collections.unmodifiableList(Arrays.asList(
+ AUTHENTICATION_STRATEGY,
+ APPLICATION_CLIENT_ID,
+ APPLICATION_KEY,
+ APPLICATION_TENANT_ID,
+ CLUSTER_URI));
+
+ public static final Pair<String, String> NIFI_SINK = Pair.of("processor", "nifi-sink");
+
+ private volatile QueuedIngestClient queuedIngestClient;
+
+ private volatile ManagedStreamingIngestClient managedStreamingIngestClient;
+
+ private volatile Client executionClient;
+
+ @Override
+ public List<PropertyDescriptor> getSupportedPropertyDescriptors() {
+ return PROPERTY_DESCRIPTORS;
+ }
+
+ /**
+ * @param context the configuration context
+ * @throws InitializationException if unable to create a database connection
+ */
+ @OnEnabled
+ public void onEnabled(final ConfigurationContext context) throws ProcessException, URISyntaxException {
+
+ final String applicationClientId = context.getProperty(APPLICATION_CLIENT_ID).getValue();
+ final String applicationKey = context.getProperty(APPLICATION_KEY).getValue();
+ final String applicationTenantId = context.getProperty(APPLICATION_TENANT_ID).getValue();
+ final String clusterUri = context.getProperty(CLUSTER_URI).getValue();
+ final KustoAuthenticationStrategy kustoAuthenticationStrategy = KustoAuthenticationStrategy.valueOf(context.getProperty(AUTHENTICATION_STRATEGY).getValue());
+
+ if (this.queuedIngestClient == null) {
+ this.queuedIngestClient = createKustoQueuedIngestClient(clusterUri, applicationClientId, applicationKey, applicationTenantId, kustoAuthenticationStrategy);
+ }
+
+ if (this.managedStreamingIngestClient == null) {
+ this.managedStreamingIngestClient = createKustoStreamingIngestClient(clusterUri, applicationClientId, applicationKey, applicationTenantId, kustoAuthenticationStrategy);
+ }
+
+ if (this.executionClient == null) {
+ this.executionClient = createKustoExecutionClient(clusterUri, applicationClientId, applicationKey, applicationTenantId, kustoAuthenticationStrategy);
+ }
+
+ }
+
+ @OnStopped
+ public final void onStopped() {
+ if (this.queuedIngestClient != null) {
+ try {
+ this.queuedIngestClient.close();
+ } catch (IOException e) {
+ getLogger().error("Closing Azure ADX Queued Ingest Client failed", e);
+ } finally {
+ this.queuedIngestClient = null;
+ }
+ }
+ if (this.managedStreamingIngestClient != null) {
+ try {
+ this.managedStreamingIngestClient.close();
+ } catch (IOException e) {
+ getLogger().error("Closing Azure ADX Managed Streaming Ingest Client failed", e);
+ } finally {
+ this.managedStreamingIngestClient = null;
+ }
+ }
+ if (this.executionClient != null) {
+ try {
+ this.executionClient.close();
+ } catch (IOException e) {
+ getLogger().error("Closing Azure ADX Execution Client failed", e);
+ } finally {
+ this.executionClient = null;
+ }
+ }
+ }
+
+
+ protected QueuedIngestClient createKustoQueuedIngestClient(final String clusterUrl,
+ final String appId,
+ final String appKey,
+ final String appTenant,
+ final KustoAuthenticationStrategy kustoAuthStrategy) throws URISyntaxException {
+ ConnectionStringBuilder ingestConnectionStringBuilder = createKustoEngineConnectionString(clusterUrl, appId, appKey, appTenant, kustoAuthStrategy);
+ return IngestClientFactory.createClient(ingestConnectionStringBuilder);
+ }
+
+ protected ManagedStreamingIngestClient createKustoStreamingIngestClient(final String clusterUrl,
+ final String appId,
+ final String appKey,
+ final String appTenant,
+ final KustoAuthenticationStrategy kustoAuthStrategy) throws URISyntaxException {
+ ConnectionStringBuilder ingestConnectionStringBuilder = createKustoEngineConnectionString(clusterUrl, appId, appKey, appTenant, kustoAuthStrategy);
+ ConnectionStringBuilder streamingConnectionStringBuilder = createKustoEngineConnectionString(clusterUrl, appId, appKey, appTenant, kustoAuthStrategy);
+ return IngestClientFactory.createManagedStreamingIngestClient(ingestConnectionStringBuilder, streamingConnectionStringBuilder);
+ }
+
+ public KustoIngestionResult ingestData(KustoIngestionRequest kustoIngestionRequest) throws URISyntaxException {
+ StreamSourceInfo info = new StreamSourceInfo(kustoIngestionRequest.getInputStream());
+ //ingest data
+ IngestionResult ingestionResult;
+ IngestionProperties ingestionProperties = new IngestionProperties(kustoIngestionRequest.getDatabaseName(),
+ kustoIngestionRequest.getTableName());
+
+ IngestionMapping.IngestionMappingKind ingestionMappingKind = setDataFormatAndMapping(kustoIngestionRequest.getDataFormat(), ingestionProperties);
+ if (StringUtils.isNotEmpty(kustoIngestionRequest.getMappingName()) && ingestionMappingKind != null) {
+ ingestionProperties.setIngestionMapping(kustoIngestionRequest.getMappingName(), ingestionMappingKind);
+ }
+
+ ingestionProperties.setReportLevel(IngestionProperties.IngestionReportLevel.FAILURES_AND_SUCCESSES);
+ ingestionProperties.setReportMethod(IngestionProperties.IngestionReportMethod.TABLE);
+ ingestionProperties.setFlushImmediately(false);
+ ingestionProperties.setIgnoreFirstRecord(StringUtils.equalsIgnoreCase(kustoIngestionRequest.getIgnoreFirstRecord(), "YES"));
+
+ try {
+ if (kustoIngestionRequest.isStreamingEnabled()) {
+ ingestionResult = managedStreamingIngestClient.ingestFromStream(info, ingestionProperties);
+ } else {
+ ingestionResult = queuedIngestClient.ingestFromStream(info, ingestionProperties);
+ }
+ if (kustoIngestionRequest.pollOnIngestionStatus()) {
+ List<IngestionStatus> statuses;
+ long startTime = System.currentTimeMillis();
+ long timeoutMillis = Long.parseLong(kustoIngestionRequest.getIngestionStatusPollingTimeout()) * 1000L;
+ while (true) {
+ // Get the status of the ingestion operation
+ List<IngestionStatus> statuses1 = ingestionResult.getIngestionStatusCollection();
+ if (statuses1.get(0).status == OperationStatus.Succeeded
+ || statuses1.get(0).status == OperationStatus.Failed
+ || statuses1.get(0).status == OperationStatus.PartiallySucceeded) {
+ statuses = statuses1;
+ break;
+ }
+ // Check if the timeout has been exceeded
+ if (System.currentTimeMillis() - startTime >= timeoutMillis) {
+ throw new ProcessException("Timeout exceeded while waiting for ingestion status");
+ }
+
+ // Sleep for 5 seconds before checking again
+ Thread.sleep(Integer.parseInt(kustoIngestionRequest.getIngestionStatusPollingInterval()) * 1000L);
Review Comment:
See the related note on parsing this value in the calling method and using `java.time.Duration`.
##########
nifi-nar-bundles/nifi-azure-bundle/nifi-azure-processors/src/main/java/org/apache/nifi/services/azure/data/explorer/StandardKustoIngestService.java:
##########
@@ -0,0 +1,362 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.services.azure.data.explorer;
+
+import com.microsoft.azure.kusto.data.Client;
+import com.microsoft.azure.kusto.data.ClientFactory;
+import com.microsoft.azure.kusto.data.KustoResultSetTable;
+import com.microsoft.azure.kusto.data.auth.ConnectionStringBuilder;
+import com.microsoft.azure.kusto.data.exceptions.DataClientException;
+import com.microsoft.azure.kusto.data.exceptions.DataServiceException;
+import com.microsoft.azure.kusto.ingest.IngestClientFactory;
+import com.microsoft.azure.kusto.ingest.IngestionMapping;
+import com.microsoft.azure.kusto.ingest.IngestionProperties;
+import com.microsoft.azure.kusto.ingest.ManagedStreamingIngestClient;
+import com.microsoft.azure.kusto.ingest.QueuedIngestClient;
+import com.microsoft.azure.kusto.ingest.exceptions.IngestionClientException;
+import com.microsoft.azure.kusto.ingest.exceptions.IngestionServiceException;
+import com.microsoft.azure.kusto.ingest.result.IngestionResult;
+import com.microsoft.azure.kusto.ingest.result.IngestionStatus;
+import com.microsoft.azure.kusto.ingest.result.OperationStatus;
+import com.microsoft.azure.kusto.ingest.source.StreamSourceInfo;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnEnabled;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.controller.AbstractControllerService;
+import org.apache.nifi.controller.ConfigurationContext;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.reporting.InitializationException;
+
+import java.io.IOException;
+import java.net.URISyntaxException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.List;
+import java.util.Objects;
+
+@Tags({"Azure", "ADX", "Kusto", "ingest", "azure"})
+@CapabilityDescription("Sends batches of flowfile content or stream flowfile content to an Azure ADX cluster.")
+public class StandardKustoIngestService extends AbstractControllerService implements KustoIngestService {
+
+ public static final PropertyDescriptor AUTHENTICATION_STRATEGY = new PropertyDescriptor
+ .Builder().name("Authentication Strategy")
+ .displayName("Authentication Strategy")
+ .description("Authentication method for access to Azure Data Explorer")
+ .required(true)
+ .defaultValue(KustoAuthenticationStrategy.MANAGED_IDENTITY.getValue())
+ .allowableValues(KustoAuthenticationStrategy.class)
+ .build();
+
+ public static final PropertyDescriptor APPLICATION_CLIENT_ID = new PropertyDescriptor
+ .Builder().name("Application Client ID")
+ .displayName("Application Client ID")
+ .description("Azure Data Explorer Application Client Identifier for Authentication")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+
+ public static final PropertyDescriptor APPLICATION_KEY = new PropertyDescriptor
+ .Builder().name("Application Key")
+ .displayName("Application Key")
+ .description("Azure Data Explorer Application Key for Authentication")
+ .required(true)
+ .sensitive(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .dependsOn(AUTHENTICATION_STRATEGY, KustoAuthenticationStrategy.APPLICATION_CREDENTIALS.getValue())
+ .build();
+
+ public static final PropertyDescriptor APPLICATION_TENANT_ID = new PropertyDescriptor.Builder()
+ .name("Application Tenant ID")
+ .displayName("Application Tenant ID")
+ .description("Azure Data Explorer Application Tenant Identifier for Authentication")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .dependsOn(AUTHENTICATION_STRATEGY, KustoAuthenticationStrategy.APPLICATION_CREDENTIALS.getValue())
+ .build();
+
+ public static final PropertyDescriptor CLUSTER_URI = new PropertyDescriptor
+ .Builder().name("Cluster URI")
+ .displayName("Cluster URI")
+ .description("Azure Data Explorer Cluster URI")
+ .required(true)
+ .addValidator(StandardValidators.URL_VALIDATOR)
+ .build();
+
+ private static final List<PropertyDescriptor> PROPERTY_DESCRIPTORS = Collections.unmodifiableList(Arrays.asList(
+ AUTHENTICATION_STRATEGY,
+ APPLICATION_CLIENT_ID,
+ APPLICATION_KEY,
+ APPLICATION_TENANT_ID,
+ CLUSTER_URI));
+
+ public static final Pair<String, String> NIFI_SINK = Pair.of("processor", "nifi-sink");
+
+ private volatile QueuedIngestClient queuedIngestClient;
+
+ private volatile ManagedStreamingIngestClient managedStreamingIngestClient;
+
+ private volatile Client executionClient;
+
+ @Override
+ public List<PropertyDescriptor> getSupportedPropertyDescriptors() {
+ return PROPERTY_DESCRIPTORS;
+ }
+
+ /**
+ * @param context the configuration context
+ * @throws InitializationException if unable to create a database connection
+ */
+ @OnEnabled
+ public void onEnabled(final ConfigurationContext context) throws ProcessException, URISyntaxException {
+
+ final String applicationClientId = context.getProperty(APPLICATION_CLIENT_ID).getValue();
+ final String applicationKey = context.getProperty(APPLICATION_KEY).getValue();
+ final String applicationTenantId = context.getProperty(APPLICATION_TENANT_ID).getValue();
+ final String clusterUri = context.getProperty(CLUSTER_URI).getValue();
+ final KustoAuthenticationStrategy kustoAuthenticationStrategy = KustoAuthenticationStrategy.valueOf(context.getProperty(AUTHENTICATION_STRATEGY).getValue());
+
+ if (this.queuedIngestClient == null) {
+ this.queuedIngestClient = createKustoQueuedIngestClient(clusterUri, applicationClientId, applicationKey, applicationTenantId, kustoAuthenticationStrategy);
+ }
+
+ if (this.managedStreamingIngestClient == null) {
+ this.managedStreamingIngestClient = createKustoStreamingIngestClient(clusterUri, applicationClientId, applicationKey, applicationTenantId, kustoAuthenticationStrategy);
+ }
+
+ if (this.executionClient == null) {
+ this.executionClient = createKustoExecutionClient(clusterUri, applicationClientId, applicationKey, applicationTenantId, kustoAuthenticationStrategy);
+ }
+
+ }
+
+ @OnStopped
+ public final void onStopped() {
+ if (this.queuedIngestClient != null) {
+ try {
+ this.queuedIngestClient.close();
+ } catch (IOException e) {
+ getLogger().error("Closing Azure ADX Queued Ingest Client failed", e);
+ } finally {
+ this.queuedIngestClient = null;
+ }
+ }
+ if (this.managedStreamingIngestClient != null) {
+ try {
+ this.managedStreamingIngestClient.close();
+ } catch (IOException e) {
+ getLogger().error("Closing Azure ADX Managed Streaming Ingest Client failed", e);
+ } finally {
+ this.managedStreamingIngestClient = null;
+ }
+ }
+ if (this.executionClient != null) {
+ try {
+ this.executionClient.close();
+ } catch (IOException e) {
+ getLogger().error("Closing Azure ADX Execution Client failed", e);
+ } finally {
+ this.executionClient = null;
+ }
+ }
+ }
+
+
+ protected QueuedIngestClient createKustoQueuedIngestClient(final String clusterUrl,
+ final String appId,
+ final String appKey,
+ final String appTenant,
+ final KustoAuthenticationStrategy kustoAuthStrategy) throws URISyntaxException {
+ ConnectionStringBuilder ingestConnectionStringBuilder = createKustoEngineConnectionString(clusterUrl, appId, appKey, appTenant, kustoAuthStrategy);
+ return IngestClientFactory.createClient(ingestConnectionStringBuilder);
+ }
+
+ protected ManagedStreamingIngestClient createKustoStreamingIngestClient(final String clusterUrl,
+ final String appId,
+ final String appKey,
+ final String appTenant,
+ final KustoAuthenticationStrategy kustoAuthStrategy) throws URISyntaxException {
+ ConnectionStringBuilder ingestConnectionStringBuilder = createKustoEngineConnectionString(clusterUrl, appId, appKey, appTenant, kustoAuthStrategy);
+ ConnectionStringBuilder streamingConnectionStringBuilder = createKustoEngineConnectionString(clusterUrl, appId, appKey, appTenant, kustoAuthStrategy);
+ return IngestClientFactory.createManagedStreamingIngestClient(ingestConnectionStringBuilder, streamingConnectionStringBuilder);
+ }
+
+ public KustoIngestionResult ingestData(KustoIngestionRequest kustoIngestionRequest) throws URISyntaxException {
+ StreamSourceInfo info = new StreamSourceInfo(kustoIngestionRequest.getInputStream());
+ //ingest data
+ IngestionResult ingestionResult;
+ IngestionProperties ingestionProperties = new IngestionProperties(kustoIngestionRequest.getDatabaseName(),
+ kustoIngestionRequest.getTableName());
+
+ IngestionMapping.IngestionMappingKind ingestionMappingKind = setDataFormatAndMapping(kustoIngestionRequest.getDataFormat(), ingestionProperties);
+ if (StringUtils.isNotEmpty(kustoIngestionRequest.getMappingName()) && ingestionMappingKind != null) {
+ ingestionProperties.setIngestionMapping(kustoIngestionRequest.getMappingName(), ingestionMappingKind);
+ }
+
+ ingestionProperties.setReportLevel(IngestionProperties.IngestionReportLevel.FAILURES_AND_SUCCESSES);
+ ingestionProperties.setReportMethod(IngestionProperties.IngestionReportMethod.TABLE);
+ ingestionProperties.setFlushImmediately(false);
+ ingestionProperties.setIgnoreFirstRecord(StringUtils.equalsIgnoreCase(kustoIngestionRequest.getIgnoreFirstRecord(), "YES"));
+
+ try {
+ if (kustoIngestionRequest.isStreamingEnabled()) {
+ ingestionResult = managedStreamingIngestClient.ingestFromStream(info, ingestionProperties);
+ } else {
+ ingestionResult = queuedIngestClient.ingestFromStream(info, ingestionProperties);
+ }
+ if (kustoIngestionRequest.pollOnIngestionStatus()) {
+ List<IngestionStatus> statuses;
+ long startTime = System.currentTimeMillis();
+ long timeoutMillis = Long.parseLong(kustoIngestionRequest.getIngestionStatusPollingTimeout()) * 1000L;
+ while (true) {
+ // Get the status of the ingestion operation
+ List<IngestionStatus> statuses1 = ingestionResult.getIngestionStatusCollection();
+ if (statuses1.get(0).status == OperationStatus.Succeeded
+ || statuses1.get(0).status == OperationStatus.Failed
+ || statuses1.get(0).status == OperationStatus.PartiallySucceeded) {
+ statuses = statuses1;
+ break;
+ }
+ // Check if the timeout has been exceeded
+ if (System.currentTimeMillis() - startTime >= timeoutMillis) {
+ throw new ProcessException("Timeout exceeded while waiting for ingestion status");
+ }
+
+ // Sleep for 5 seconds before checking again
+ Thread.sleep(Integer.parseInt(kustoIngestionRequest.getIngestionStatusPollingInterval()) * 1000L);
+ }
+ return KustoIngestionResult.fromString(statuses.get(0).status.toString());
+ } else {
+ return KustoIngestionResult.fromString("Succeeded");
+ }
+ } catch (IngestionClientException | IngestionServiceException e) {
+ throw new ProcessException("Azure Data Explorer Ingest failed", e);
+ } catch (InterruptedException e) {
+ Thread.currentThread().interrupt(); //Restore interrupted status
+ throw new ProcessException("Azure Data Explorer Ingest interrupted", e);
+ }
+ }
+
+ private ConnectionStringBuilder createKustoEngineConnectionString(final String clusterUrl,
+ final String appId,
+ final String appKey,
+ final String appTenant,
+ final KustoAuthenticationStrategy kustoAuthStrategy) {
+ final ConnectionStringBuilder builder;
+ if (KustoAuthenticationStrategy.APPLICATION_CREDENTIALS == kustoAuthStrategy) {
+ builder = ConnectionStringBuilder.createWithAadApplicationCredentials(clusterUrl, appId, appKey, appTenant);
+ } else {
+ builder = ConnectionStringBuilder.createWithAadManagedIdentity(clusterUrl, appId);
+ }
+
+
+ builder.setConnectorDetails("Kusto.Nifi.Sink", StandardKustoIngestService.class.getPackage().getImplementationVersion(), null, null, false, null, NIFI_SINK);
+ return builder;
+ }
+
+ protected Client createKustoExecutionClient(final String clusterUrl,
+ final String appId,
+ final String appKey,
+ final String appTenant,
+ final KustoAuthenticationStrategy kustoAuthStrategy) throws URISyntaxException {
+ return ClientFactory.createClient(createKustoEngineConnectionString(clusterUrl, appId, appKey, appTenant, kustoAuthStrategy));
+ }
+
+ @Override
+ public KustoQueryResponse executeQuery(String databaseName, String query) {
+ Objects.requireNonNull(databaseName, "Database Name required");
+ Objects.requireNonNull(query, "Query required");
+ KustoQueryResponse kustoQueryResponse;
+ try {
+ KustoResultSetTable kustoResultSetTable = this.executionClient.execute(databaseName, query).getPrimaryResults();
+ if(kustoResultSetTable.hasNext()){
+ kustoResultSetTable.next();
+ kustoQueryResponse = new KustoQueryResponse(kustoResultSetTable.getData());
+ } else {
+ kustoQueryResponse = new KustoQueryResponse(new ArrayList<>());
Review Comment:
```suggestion
kustoQueryResponse = new KustoQueryResponse(Collections.emptyList());
```
##########
nifi-nar-bundles/nifi-azure-bundle/nifi-azure-services-api/src/main/java/org/apache/nifi/services/azure/data/explorer/KustoIngestService.java:
##########
@@ -0,0 +1,28 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.services.azure.data.explorer;
+
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+
+import java.net.URISyntaxException;
+
+@Tags({"azure", "adx", "explorer", "kusto", "ingest"})
+@CapabilityDescription("Connection-Service for Azure ADX (Kusto) ingestion cluster to ingest data.")
+public interface KustoIngestService extends KustoQueryService {
Review Comment:
On further review of the response, it seems like it would be better to introduce a new method for executing the ingest status query, instead of returning the generic `List<List<Object>>` result.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] NIFI-11958 Azure Data Explorer as sink [nifi]
Posted by "exceptionfactory (via GitHub)" <gi...@apache.org>.
exceptionfactory commented on PR #7624:
URL: https://github.com/apache/nifi/pull/7624#issuecomment-1840798813
> @exceptionfactory : Hope you are doing good. Can you please take a look at this PR. This is long pending item, we wanted to close this before the end of this year. We would appreciate your review on this item whenever you have some time.
Thanks for the follow-up @tanmaya-panda1. I am in the process of reviewing several pull requests, and I plan to return to this one soon.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] tanmaya-panda1 commented on a diff in pull request #7624: NIFI-11958 Azure Data Explorer as sink
Posted by "tanmaya-panda1 (via GitHub)" <gi...@apache.org>.
tanmaya-panda1 commented on code in PR #7624:
URL: https://github.com/apache/nifi/pull/7624#discussion_r1336369541
##########
nifi-nar-bundles/nifi-azure-bundle/nifi-azure-processors/src/main/java/org/apache/nifi/services/azure/data/explorer/StandardKustoIngestService.java:
##########
@@ -0,0 +1,314 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.services.azure.data.explorer;
+
+import com.microsoft.azure.kusto.data.Client;
+import com.microsoft.azure.kusto.data.ClientFactory;
+import com.microsoft.azure.kusto.data.KustoResultSetTable;
+import com.microsoft.azure.kusto.data.auth.ConnectionStringBuilder;
+import com.microsoft.azure.kusto.data.exceptions.DataClientException;
+import com.microsoft.azure.kusto.data.exceptions.DataServiceException;
+import com.microsoft.azure.kusto.ingest.IngestClientFactory;
+import com.microsoft.azure.kusto.ingest.ManagedStreamingIngestClient;
+import com.microsoft.azure.kusto.ingest.QueuedIngestClient;
+import com.microsoft.azure.kusto.ingest.exceptions.IngestionClientException;
+import com.microsoft.azure.kusto.ingest.exceptions.IngestionServiceException;
+import com.microsoft.azure.kusto.ingest.result.IngestionResult;
+import com.microsoft.azure.kusto.ingest.result.IngestionStatus;
+import com.microsoft.azure.kusto.ingest.result.OperationStatus;
+import com.microsoft.azure.kusto.ingest.source.StreamSourceInfo;
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.nifi.annotation.behavior.ReadsAttribute;
+import org.apache.nifi.annotation.behavior.ReadsAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnEnabled;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.controller.AbstractControllerService;
+import org.apache.nifi.controller.ConfigurationContext;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.reporting.InitializationException;
+
+import java.io.IOException;
+import java.net.URISyntaxException;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.List;
+import java.util.Objects;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+
+@Tags({"Azure", "ADX", "Kusto", "ingest", "azure"})
+@CapabilityDescription("Sends batches of flowfile content or stream flowfile content to an Azure ADX cluster.")
+@ReadsAttributes({
+ @ReadsAttribute(attribute = "AUTH_STRATEGY", description = "The strategy/method to authenticate against Azure Active Directory, either 'application' or 'managed_identity'."),
+ @ReadsAttribute(attribute = "APP_ID", description = "Specifies Azure application id for accessing the ADX-Cluster."),
+ @ReadsAttribute(attribute = "APP_KEY", description = "Specifies Azure application key for accessing the ADX-Cluster."),
+ @ReadsAttribute(attribute = "APP_TENANT", description = "Azure application tenant for accessing the ADX-Cluster."),
+ @ReadsAttribute(attribute = "CLUSTER_URL", description = "Endpoint of ADX cluster. This is required only when streaming data to ADX cluster is enabled."),
+})
+public class StandardKustoIngestService extends AbstractControllerService implements KustoIngestService {
+
+ public static final PropertyDescriptor AUTHENTICATION_STRATEGY = new PropertyDescriptor
+ .Builder().name("Authentication Strategy")
+ .displayName("Authentication Strategy")
+ .description("Authentication method for access to Azure Data Explorer")
+ .required(true)
+ .defaultValue(KustoAuthenticationStrategy.MANAGED_IDENTITY.getValue())
+ .allowableValues(KustoAuthenticationStrategy.class)
+ .build();
+
+ public static final PropertyDescriptor APPLICATION_CLIENT_ID = new PropertyDescriptor
+ .Builder().name("Application Client ID")
+ .displayName("Application Client ID")
+ .description("Azure Data Explorer Application Client Identifier for Authentication")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+
+ public static final PropertyDescriptor APPLICATION_KEY = new PropertyDescriptor
+ .Builder().name("Application Key")
+ .displayName("Application Key")
+ .description("Azure Data Explorer Application Key for Authentication")
+ .required(true)
+ .sensitive(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .dependsOn(AUTHENTICATION_STRATEGY, KustoAuthenticationStrategy.APPLICATION_CREDENTIALS.getValue())
+ .build();
+
+ public static final PropertyDescriptor APPLICATION_TENANT_ID = new PropertyDescriptor.Builder()
+ .name("Application Tenant ID")
+ .displayName("Application Tenant ID")
+ .description("Azure Data Explorer Application Tenant Identifier for Authentication")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .dependsOn(AUTHENTICATION_STRATEGY, KustoAuthenticationStrategy.APPLICATION_CREDENTIALS.getValue())
+ .build();
+
+ public static final PropertyDescriptor CLUSTER_URI = new PropertyDescriptor
+ .Builder().name("Cluster URI")
+ .displayName("Cluster URI")
+ .description("Azure Data Explorer Cluster URI")
+ .required(true)
+ .addValidator(StandardValidators.URL_VALIDATOR)
+ .build();
+
+ private static final List<PropertyDescriptor> PROPERTY_DESCRIPTORS = Collections.unmodifiableList(Arrays.asList(
+ AUTHENTICATION_STRATEGY,
+ APPLICATION_CLIENT_ID,
+ APPLICATION_KEY,
+ APPLICATION_TENANT_ID,
+ CLUSTER_URI));
+
+ public static final Pair<String, String> NIFI_SINK = Pair.of("processor", "nifi-sink");
+
+ private volatile QueuedIngestClient queuedIngestClient;
+
+ private volatile ManagedStreamingIngestClient managedStreamingIngestClient;
+
+ private volatile Client executionClient;
+
+ @Override
+ public List<PropertyDescriptor> getSupportedPropertyDescriptors() {
+ return PROPERTY_DESCRIPTORS;
+ }
+
+ /**
+ * @param context the configuration context
+ * @throws InitializationException if unable to create a database connection
+ */
+ @OnEnabled
+ public void onEnabled(final ConfigurationContext context) throws ProcessException, URISyntaxException {
+
+ getLogger().info("Starting Azure ADX Connection Service...");
+ final String applicationClientId = context.getProperty(APPLICATION_CLIENT_ID).getValue();
+ final String applicationKey = context.getProperty(APPLICATION_KEY).getValue();
+ final String applicationTenantId = context.getProperty(APPLICATION_TENANT_ID).getValue();
+ final String clusterUri = context.getProperty(CLUSTER_URI).getValue();
+ final KustoAuthenticationStrategy kustoAuthenticationStrategy = KustoAuthenticationStrategy.valueOf(context.getProperty(AUTHENTICATION_STRATEGY).getValue());
+
+ if (this.queuedIngestClient == null) {
+ this.queuedIngestClient = createKustoQueuedIngestClient(clusterUri, applicationClientId, applicationKey, applicationTenantId, kustoAuthenticationStrategy);
+ }
+
+ if (this.managedStreamingIngestClient == null) {
+ this.managedStreamingIngestClient = createKustoStreamingIngestClient(clusterUri, applicationClientId, applicationKey, applicationTenantId, kustoAuthenticationStrategy);
+ }
+
+ if (this.executionClient == null) {
+ this.executionClient = createKustoExecutionClient(clusterUri, applicationClientId, applicationKey, applicationTenantId, kustoAuthenticationStrategy);
+ }
+
+ }
+
+ @OnStopped
+ public final void onStopped() {
+ if (this.queuedIngestClient != null) {
+ try {
+ this.queuedIngestClient.close();
+ } catch (IOException e) {
+ getLogger().error("Closing Azure ADX Queued Ingest Client failed with: " + e.getMessage(), e);
+ } finally {
+ this.queuedIngestClient = null;
+ }
+ }
+ if (this.managedStreamingIngestClient != null) {
+ try {
+ this.managedStreamingIngestClient.close();
+ } catch (IOException e) {
+ getLogger().error("Closing Azure ADX Managed Streaming Ingest Client failed with: " + e.getMessage(), e);
+ } finally {
+ this.managedStreamingIngestClient = null;
+ }
+ }
+ if (this.executionClient != null) {
+ try {
+ this.executionClient.close();
+ } catch (IOException e) {
+ getLogger().error("Closing Azure ADX Execution Client failed with: " + e.getMessage(), e);
+ } finally {
+ this.executionClient = null;
+ }
+ }
+ }
+
+
+ protected QueuedIngestClient createKustoQueuedIngestClient(final String clusterUrl,
+ final String appId,
+ final String appKey,
+ final String appTenant,
+ final KustoAuthenticationStrategy kustoAuthStrategy) throws URISyntaxException {
+ ConnectionStringBuilder ingestConnectionStringBuilder = createKustoEngineConnectionString(clusterUrl, appId, appKey, appTenant, kustoAuthStrategy);
+ return IngestClientFactory.createClient(ingestConnectionStringBuilder);
+ }
+
+ protected ManagedStreamingIngestClient createKustoStreamingIngestClient(final String clusterUrl,
+ final String appId,
+ final String appKey,
+ final String appTenant,
+ final KustoAuthenticationStrategy kustoAuthStrategy) throws URISyntaxException {
+ ConnectionStringBuilder ingestConnectionStringBuilder = createKustoEngineConnectionString(clusterUrl, appId, appKey, appTenant, kustoAuthStrategy);
+ ConnectionStringBuilder streamingConnectionStringBuilder = createKustoEngineConnectionString(clusterUrl, appId, appKey, appTenant, kustoAuthStrategy);
+ return IngestClientFactory.createManagedStreamingIngestClient(ingestConnectionStringBuilder, streamingConnectionStringBuilder);
+ }
+
+ public KustoIngestionResult ingestData(KustoIngestionRequest kustoIngestionRequest) {
+ StreamSourceInfo info = new StreamSourceInfo(kustoIngestionRequest.getInputStream());
+ //ingest data
+ IngestionResult ingestionResult;
+ ScheduledExecutorService scheduler = Executors.newScheduledThreadPool(1);
Review Comment:
Agreed, modified the ingestionStatusCheck to execute in the same thread
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] NIFI-11958 Azure Data Explorer as sink [nifi]
Posted by "exceptionfactory (via GitHub)" <gi...@apache.org>.
exceptionfactory commented on PR #7624:
URL: https://github.com/apache/nifi/pull/7624#issuecomment-1894538604
Rebased to resolve merge conflicts.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] NIFI-11958 Azure Data Explorer as sink [nifi]
Posted by "tanmaya-panda1 (via GitHub)" <gi...@apache.org>.
tanmaya-panda1 commented on code in PR #7624:
URL: https://github.com/apache/nifi/pull/7624#discussion_r1364001567
##########
nifi-nar-bundles/nifi-azure-bundle/nifi-azure-processors/src/main/java/org/apache/nifi/processors/azure/data/explorer/PutAzureDataExplorer.java:
##########
@@ -0,0 +1,399 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.azure.data.explorer;
+
+import com.microsoft.azure.kusto.data.KustoResultSetTable;
+import com.microsoft.azure.kusto.ingest.IngestionMapping;
+import com.microsoft.azure.kusto.ingest.IngestionProperties;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.nifi.annotation.behavior.ReadsAttribute;
+import org.apache.nifi.annotation.behavior.ReadsAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.expression.ExpressionLanguageScope;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.services.azure.data.explorer.KustoIngestService;
+import org.apache.nifi.services.azure.data.explorer.KustoIngestionRequest;
+import org.apache.nifi.services.azure.data.explorer.KustoIngestionResult;
+import org.apache.nifi.services.azure.data.explorer.KustoQueryResponse;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.net.URISyntaxException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+
+@Tags({"Azure", "Kusto", "ADX", "Explorer", "Data"})
+@CapabilityDescription("The PutAzureDataExplorer acts as a ADX sink connector which sends flowFiles using the ADX-Service to the provided Azure Data" +
+ "Explorer Ingest Endpoint. The data can be sent through queued ingestion or streaming ingestion to the Azure Data Explorer cluster.")
+@ReadsAttributes({
+ @ReadsAttribute(attribute = "DB_NAME", description = "Specifies the name of the ADX database where the data needs to be stored."),
+ @ReadsAttribute(attribute = "TABLE_NAME", description = "Specifies the name of the ADX table where the data needs to be stored."),
+ @ReadsAttribute(attribute = "MAPPING_NAME", description = "Specifies the name of the mapping responsible for storing the data in appropriate columns."),
+ @ReadsAttribute(attribute = "FLUSH_IMMEDIATE", description = "In case of queued ingestion, this property determines whether the data should be flushed immediately to the ingest endpoint."),
+ @ReadsAttribute(attribute = "DATA_FORMAT", description = "Specifies the format of data that is send to Azure Data Explorer."),
+ @ReadsAttribute(attribute = "IGNORE_FIRST_RECORD", description = "Specifies whether we want to ignore ingestion of first record. " +
+ "This is primarily applicable for csv files. Default is set to NO"),
+ @ReadsAttribute(attribute = "POLL_ON_INGESTION_STATUS", description = "Specifies whether we want to poll on ingestion result during ingestion into ADX." +
+ "In case of applications that need high throughput it is recommended to keep the default value as false. Default is set to false." +
+ "This property should be set to true during Queued Ingestion for near realtime micro-batches of data that require acknowledgement of ingestion status.")
+})
+public class PutAzureDataExplorer extends AbstractProcessor {
+
+ public static final String FETCH_TABLE_COMMAND = "%s | count";
+ public static final String STREAMING_POLICY_SHOW_COMMAND = ".show %s %s policy streamingingestion";
+ public static final String DATABASE = "database";
+
+ private List<PropertyDescriptor> descriptors;
+ private Set<Relationship> relationships;
+ private transient KustoIngestService service;
+ private boolean isStreamingEnabled;
+ private boolean pollOnIngestionStatus;
+
+ public static final AllowableValue AVRO = new AllowableValue(
+ IngestionProperties.DataFormat.AVRO.getKustoValue(), IngestionProperties.DataFormat.AVRO.getKustoValue(),
+ "An Avro format with support for logical types and for the snappy compression codec");
+
+ public static final AllowableValue APACHEAVRO = new AllowableValue(
+ IngestionProperties.DataFormat.APACHEAVRO.getKustoValue(), IngestionProperties.DataFormat.APACHEAVRO.getKustoValue(),
+ "An Avro format with support for logical types and for the snappy compression codec.");
+
+ public static final AllowableValue CSV = new AllowableValue(
+ IngestionProperties.DataFormat.CSV.name(), IngestionProperties.DataFormat.CSV.getKustoValue(),
+ "A text file with comma-separated values (,). For more information, see RFC 4180: Common Format " +
+ "and MIME Type for Comma-Separated Values (CSV) Files.");
+
+ public static final AllowableValue JSON = new AllowableValue(
+ IngestionProperties.DataFormat.JSON.name(), IngestionProperties.DataFormat.JSON.getKustoValue(),
+ "A text file containing JSON objects separated by \\n or \\r\\n. For more information, " +
+ "see JSON Lines (JSONL).");
+
+ public static final AllowableValue MULTIJSON = new AllowableValue(
+ IngestionProperties.DataFormat.MULTIJSON.name(), IngestionProperties.DataFormat.MULTIJSON.getKustoValue(),
+ "A text file containing a JSON array of property containers (each representing a record) or any " +
+ "number of property containers separated by spaces, \\n or \\r\\n. Each property container may be " +
+ "spread across multiple lines. This format is preferable to JSON unless the data is not property " +
+ "containers.");
+
+ public static final AllowableValue ORC = new AllowableValue(
+ IngestionProperties.DataFormat.ORC.name(), IngestionProperties.DataFormat.ORC.getKustoValue(), "An ORC file.");
+
+ public static final AllowableValue PARQUET = new AllowableValue(
+ IngestionProperties.DataFormat.PARQUET.name(), IngestionProperties.DataFormat.PARQUET.getKustoValue(), "A parquet file.");
+
+ public static final AllowableValue PSV = new AllowableValue(
+ IngestionProperties.DataFormat.PSV.name(), IngestionProperties.DataFormat.PSV.getKustoValue(), "A text file with values separated by vertical bars (|).");
+
+ public static final AllowableValue SCSV = new AllowableValue(
+ IngestionProperties.DataFormat.SCSV.name(), IngestionProperties.DataFormat.SCSV.getKustoValue(), "A text file with values separated by semicolons (;).");
+
+ public static final AllowableValue SOHSV = new AllowableValue(
+ IngestionProperties.DataFormat.SOHSV.name(), IngestionProperties.DataFormat.SOHSV.getKustoValue(),
+ "A text file with SOH-separated values. (SOH is the ASCII code point 1. " +
+ "This format is used by Hive in HDInsight).");
+
+ public static final AllowableValue TSV = new AllowableValue(
+ IngestionProperties.DataFormat.TSV.name(), IngestionProperties.DataFormat.TSV.getKustoValue(), "A text file with tab delimited values (\\t).");
+
+ public static final AllowableValue TSVE = new AllowableValue(
+ IngestionProperties.DataFormat.TSVE.name(), IngestionProperties.DataFormat.TSVE.getKustoValue(),
+ "A text file with tab-delimited values (\\t). A backslash (\\) is used as escape character.");
+
+ public static final AllowableValue TXT = new AllowableValue(
+ IngestionProperties.DataFormat.TXT.name(), IngestionProperties.DataFormat.TXT.getKustoValue(),
+ "A text file with lines separated by \\n. Empty lines are skipped.");
+
+ public static final AllowableValue IGNORE_FIRST_RECORD_YES = new AllowableValue(
+ "YES", "YES",
+ "Ignore first record during ingestion");
+
+ public static final AllowableValue IGNORE_FIRST_RECORD_NO = new AllowableValue(
+ "NO", "NO",
+ "Do not ignore first record during ingestion");
+
+ public static final PropertyDescriptor DATABASE_NAME = new PropertyDescriptor.Builder()
+ .name("Database Name")
+ .displayName("Database Name")
+ .description("Azure Data Explorer Database Name for querying")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+ .build();
+
+ public static final PropertyDescriptor TABLE_NAME = new PropertyDescriptor.Builder()
+ .name("Table Name")
+ .displayName("Table Name")
+ .description("Azure Data Explorer Table Name for ingesting data")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+ .build();
+
+ public static final PropertyDescriptor MAPPING_NAME = new PropertyDescriptor
+ .Builder().name("Mapping name")
+ .displayName("Mapping name")
+ .description("The name of the mapping responsible for storing the data in the appropriate columns.")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+
+ public static final PropertyDescriptor IS_STREAMING_ENABLED = new PropertyDescriptor
+ .Builder().name("Is Streaming enabled")
+ .displayName("Is Streaming enabled")
+ .description("This property determines whether we want to stream data to ADX.")
+ .required(false)
+ .allowableValues("true", "false")
+ .addValidator(StandardValidators.BOOLEAN_VALIDATOR)
+ .defaultValue("false")
+ .build();
+
+ public static final PropertyDescriptor POLL_ON_INGESTION_STATUS = new PropertyDescriptor
+ .Builder().name("Poll on ingestion status")
+ .displayName("Whether to Poll on ingestion status")
+ .description("This property determines whether we want to poll on ingestion status after an ingestion to ADX is completed")
+ .required(false)
+ .allowableValues("true", "false")
+ .addValidator(StandardValidators.BOOLEAN_VALIDATOR)
+ .defaultValue("false")
+ .build();
+ public static final PropertyDescriptor ADX_SERVICE = new PropertyDescriptor
+ .Builder().name("Kusto Ingest Service")
+ .displayName("Kusto Ingest Service")
+ .description("Azure Data Explorer Kusto Ingest Service")
+ .required(true)
+ .identifiesControllerService(KustoIngestService.class)
+ .build();
+
+ public static final Relationship SUCCESS = new Relationship.Builder()
+ .name("success")
+ .description("Relationship for success")
+ .build();
+ public static final Relationship FAILURE = new Relationship.Builder()
+ .name("failure")
+ .description("Relationship for failure")
+ .build();
+
+ static final PropertyDescriptor DATA_FORMAT = new PropertyDescriptor.Builder()
+ .name("Data Format")
+ .displayName("Data Format")
+ .description("The format of the data that is sent to ADX.")
+ .required(true)
+ .allowableValues(AVRO, APACHEAVRO, CSV, JSON, MULTIJSON, ORC, PARQUET, PSV, SCSV, SOHSV, TSV, TSVE, TXT)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+
+ static final PropertyDescriptor IGNORE_FIRST_RECORD = new PropertyDescriptor.Builder()
+ .name("Ingestion Ignore First Record")
+ .displayName("Ingestion Ignore First Record")
+ .description("Defines whether ignore first record while ingestion.")
+ .required(false)
+ .allowableValues(IGNORE_FIRST_RECORD_YES, IGNORE_FIRST_RECORD_NO)
+ .defaultValue(IGNORE_FIRST_RECORD_NO.getValue())
+ .build();
+
+ @Override
+ protected void init(final ProcessorInitializationContext context) {
+ final List<PropertyDescriptor> descriptorList = new ArrayList<>();
+ descriptorList.add(ADX_SERVICE);
+ descriptorList.add(DATABASE_NAME);
+ descriptorList.add(TABLE_NAME);
+ descriptorList.add(MAPPING_NAME);
+ descriptorList.add(DATA_FORMAT);
+ descriptorList.add(IGNORE_FIRST_RECORD);
+ descriptorList.add(IS_STREAMING_ENABLED);
+ descriptorList.add(POLL_ON_INGESTION_STATUS);
+ this.descriptors = Collections.unmodifiableList(descriptorList);
+
+ final Set<Relationship> relationshipSet = new HashSet<>();
+ relationshipSet.add(SUCCESS);
+ relationshipSet.add(FAILURE);
+ this.relationships = Collections.unmodifiableSet(relationshipSet);
+ }
+
+ @Override
+ public Set<Relationship> getRelationships() {
+ return this.relationships;
+ }
+
+ @Override
+ public final List<PropertyDescriptor> getSupportedPropertyDescriptors() {
+ return descriptors;
+ }
+
+ @OnScheduled
+ public void onScheduled(final ProcessContext context) {
+ service = context.getProperty(ADX_SERVICE).asControllerService(KustoIngestService.class);
+ if (checkIfIngestorRoleDoesntExist(context.getProperty(DATABASE_NAME).getValue(), context.getProperty(TABLE_NAME).getValue())) {
+ getLogger().error("User might not have ingestor privileges, table validation will be skipped for all table mappings.");
+ throw new ProcessException("User might not have ingestor privileges, table validation will be skipped for all table mappings. ");
+ }
+ isStreamingEnabled = context.getProperty(IS_STREAMING_ENABLED).evaluateAttributeExpressions().asBoolean();
+ if (isStreamingEnabled) {
+ checkIfStreamingPolicyIsEnabledInADX(context.getProperty(DATABASE_NAME).getValue(), context.getProperty(DATABASE_NAME).getValue());
+ }
+ }
+
+ @Override
+ public void onTrigger(final ProcessContext context, final ProcessSession session) throws ProcessException {
+ FlowFile flowFile = session.get();
+ if (flowFile == null) {
+ context.yield();
+ return;
+ }
+
+ IngestionProperties ingestionProperties = new IngestionProperties(context.getProperty(DATABASE_NAME).getValue(),
+ context.getProperty(TABLE_NAME).getValue());
+
+ IngestionMapping.IngestionMappingKind ingestionMappingKind = null;
+
+ isStreamingEnabled = context.getProperty(IS_STREAMING_ENABLED).evaluateAttributeExpressions().asBoolean();
+ pollOnIngestionStatus = context.getProperty(POLL_ON_INGESTION_STATUS).evaluateAttributeExpressions().asBoolean();
+
+ switch (IngestionProperties.DataFormat.valueOf(context.getProperty(DATA_FORMAT).getValue())) {
+ case AVRO:
+ ingestionProperties.setDataFormat(IngestionProperties.DataFormat.AVRO);
+ ingestionMappingKind = IngestionProperties.DataFormat.AVRO.getIngestionMappingKind();
+ break;
+ case APACHEAVRO:
+ ingestionProperties.setDataFormat(IngestionProperties.DataFormat.APACHEAVRO);
+ ingestionMappingKind = IngestionProperties.DataFormat.APACHEAVRO.getIngestionMappingKind();
+ break;
+ case CSV:
+ ingestionProperties.setDataFormat(IngestionProperties.DataFormat.CSV);
+ ingestionMappingKind = IngestionProperties.DataFormat.CSV.getIngestionMappingKind();
+ break;
+ case JSON:
+ ingestionProperties.setDataFormat(IngestionProperties.DataFormat.JSON);
+ ingestionMappingKind = IngestionProperties.DataFormat.JSON.getIngestionMappingKind();
+ break;
+ case MULTIJSON:
+ ingestionProperties.setDataFormat(IngestionProperties.DataFormat.MULTIJSON);
+ ingestionMappingKind = IngestionProperties.DataFormat.MULTIJSON.getIngestionMappingKind();
+ break;
+ case ORC:
+ ingestionProperties.setDataFormat(IngestionProperties.DataFormat.ORC);
+ ingestionMappingKind = IngestionProperties.DataFormat.ORC.getIngestionMappingKind();
+ break;
+ case PARQUET:
+ ingestionProperties.setDataFormat(IngestionProperties.DataFormat.PARQUET);
+ ingestionMappingKind = IngestionProperties.DataFormat.PARQUET.getIngestionMappingKind();
+ break;
+ case PSV:
+ ingestionProperties.setDataFormat(IngestionProperties.DataFormat.PSV);
+ ingestionMappingKind = IngestionProperties.DataFormat.PSV.getIngestionMappingKind();
+ break;
+ case SCSV:
+ ingestionProperties.setDataFormat(IngestionProperties.DataFormat.SCSV);
+ ingestionMappingKind = IngestionProperties.DataFormat.SCSV.getIngestionMappingKind();
+ break;
+ case SOHSV:
+ ingestionProperties.setDataFormat(IngestionProperties.DataFormat.SOHSV);
+ ingestionMappingKind = IngestionProperties.DataFormat.SOHSV.getIngestionMappingKind();
+ break;
+ case TSV:
+ ingestionProperties.setDataFormat(IngestionProperties.DataFormat.TSV);
+ ingestionMappingKind = IngestionProperties.DataFormat.TSV.getIngestionMappingKind();
+ break;
+ case TSVE:
+ ingestionProperties.setDataFormat(IngestionProperties.DataFormat.TSVE);
+ ingestionMappingKind = IngestionProperties.DataFormat.TSVE.getIngestionMappingKind();
+ break;
+ case TXT:
+ ingestionProperties.setDataFormat(IngestionProperties.DataFormat.TXT);
+ ingestionMappingKind = IngestionProperties.DataFormat.TXT.getIngestionMappingKind();
+ break;
+ }
+
+ if (StringUtils.isNotEmpty(context.getProperty(MAPPING_NAME).getValue()) && ingestionMappingKind != null) {
+ ingestionProperties.setIngestionMapping(context.getProperty(MAPPING_NAME).getValue(), ingestionMappingKind);
+ }
+
+ ingestionProperties.setReportLevel(IngestionProperties.IngestionReportLevel.FAILURES_AND_SUCCESSES);
+ ingestionProperties.setReportMethod(IngestionProperties.IngestionReportMethod.TABLE);
+ ingestionProperties.setFlushImmediately(false);
+ ingestionProperties.setIgnoreFirstRecord(StringUtils.equalsIgnoreCase(context.getProperty(IGNORE_FIRST_RECORD).getValue(), IGNORE_FIRST_RECORD_YES.getValue()));
+
+ boolean isError = false;
+
+ // ingestion into ADX
+ try (final InputStream inputStream = session.read(flowFile)) {
+ StringBuilder ingestLogString = new StringBuilder().append("Ingesting with: ")
+ .append("dataFormat - ").append(ingestionProperties.getDataFormat()).append("|")
+ .append("ingestionMapping - ").append(ingestionProperties.getIngestionMapping().getIngestionMappingReference()).append("|")
+ .append("reportLevel - ").append(ingestionProperties.getReportLevel()).append("|")
+ .append("reportMethod - ").append(ingestionProperties.getReportMethod()).append("|")
+ .append("databaseName - ").append(ingestionProperties.getDatabaseName()).append("|")
+ .append("tableName - ").append(ingestionProperties.getTableName()).append("|")
+ .append("flushImmediately - ").append(ingestionProperties.getFlushImmediately());
+ getLogger().info(ingestLogString.toString());
+
+ KustoIngestionResult result = service.ingestData(new KustoIngestionRequest(isStreamingEnabled, pollOnIngestionStatus, inputStream, ingestionProperties));
+ getLogger().info("Poll on Ingestion Status {} and Operation status: {} ", pollOnIngestionStatus, result.toString());
+
+ if (result == KustoIngestionResult.SUCCEEDED) {
+ getLogger().info("Operation status Succeeded - {}", result.toString());
+ } else if (result == KustoIngestionResult.FAILED) {
+ getLogger().error("Operation status Error - {}", result.toString());
+ isError = true;
+ } else if (result == KustoIngestionResult.PARTIALLY_SUCCEEDED) {
+ getLogger().error("Operation status Partially succeeded - {}", result.toString());
Review Comment:
Considering partially succeeded status as success might result in silent data loss when the user ignores warning logs for a huge set of data, Hence added a new "partial success" relationship where the partially succeeded data can be routed and appropriate action can be taken by the user. What are your thoughts on this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] NIFI-11958 Azure Data Explorer as sink [nifi]
Posted by "tanmaya-panda1 (via GitHub)" <gi...@apache.org>.
tanmaya-panda1 commented on code in PR #7624:
URL: https://github.com/apache/nifi/pull/7624#discussion_r1366722167
##########
nifi-nar-bundles/nifi-azure-bundle/nifi-azure-services-api/src/main/java/org/apache/nifi/services/azure/data/explorer/KustoIngestService.java:
##########
@@ -0,0 +1,28 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.services.azure.data.explorer;
+
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+
+import java.net.URISyntaxException;
+
+@Tags({"azure", "adx", "explorer", "kusto", "ingest"})
+@CapabilityDescription("Connection-Service for Azure ADX (Kusto) ingestion cluster to ingest data.")
+public interface KustoIngestService extends KustoQueryService {
Review Comment:
While using KustoIngestService, we will need to query for ingestor privileges and then start ingestion hence we need executeQuery during the ingest service. We didn't want to duplicate executeQuery in KustoIngestService, so extended KustoQueryService.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] NIFI-11958 Azure Data Explorer as sink [nifi]
Posted by "tanmaya-panda1 (via GitHub)" <gi...@apache.org>.
tanmaya-panda1 commented on code in PR #7624:
URL: https://github.com/apache/nifi/pull/7624#discussion_r1430938458
##########
nifi-nar-bundles/nifi-azure-bundle/nifi-azure-processors/src/main/java/org/apache/nifi/processors/azure/data/explorer/PutAzureDataExplorer.java:
##########
@@ -0,0 +1,282 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.azure.data.explorer;
+
+import org.apache.commons.lang3.StringUtils;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.expression.ExpressionLanguageScope;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.services.azure.data.explorer.KustoIngestDataFormat;
+import org.apache.nifi.services.azure.data.explorer.KustoIngestQueryResponse;
+import org.apache.nifi.services.azure.data.explorer.KustoIngestService;
+import org.apache.nifi.services.azure.data.explorer.KustoIngestionRequest;
+import org.apache.nifi.services.azure.data.explorer.KustoIngestionResult;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.time.Duration;
+import java.util.List;
+import java.util.Set;
+import java.util.concurrent.TimeUnit;
+
+@Tags({"Azure", "Kusto", "ADX", "Explorer", "Data"})
+@CapabilityDescription("The PutAzureDataExplorer acts as a ADX sink connector which sends flowFiles using the ADX-Service to the provided Azure Data" +
+ "Explorer Ingest Endpoint. The data can be sent through queued ingestion or streaming ingestion to the Azure Data Explorer cluster.")
+@InputRequirement(InputRequirement.Requirement.INPUT_REQUIRED)
+public class PutAzureDataExplorer extends AbstractProcessor {
+
+ public static final String FETCH_TABLE_COMMAND = "%s | count";
+ public static final String STREAMING_POLICY_SHOW_COMMAND = ".show %s %s policy streamingingestion";
+ public static final String DATABASE = "database";
+ public static final AllowableValue IGNORE_FIRST_RECORD_YES = new AllowableValue(
+ "YES", "YES",
+ "Ignore first record during ingestion");
+
+ public static final AllowableValue IGNORE_FIRST_RECORD_NO = new AllowableValue(
+ "NO", "NO",
+ "Do not ignore first record during ingestion");
+
+ public static final PropertyDescriptor DATABASE_NAME = new PropertyDescriptor.Builder()
+ .name("Database Name")
+ .displayName("Database Name")
+ .description("Azure Data Explorer Database Name for querying")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+ .build();
+ public static final PropertyDescriptor TABLE_NAME = new PropertyDescriptor.Builder()
+ .name("Table Name")
+ .displayName("Table Name")
+ .description("Azure Data Explorer Table Name for ingesting data")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+ .build();
+ public static final PropertyDescriptor MAPPING_NAME = new PropertyDescriptor
+ .Builder().name("Ingest Mapping name")
+ .displayName("Ingest Mapping name")
+ .description("The name of the mapping responsible for storing the data in the appropriate columns.")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor IS_STREAMING_ENABLED = new PropertyDescriptor
+ .Builder().name("Streaming Enabled")
+ .displayName("Streaming Enabled")
+ .description("This property determines whether we want to stream data to ADX.")
+ .required(false)
+ .allowableValues(Boolean.TRUE.toString(), Boolean.FALSE.toString())
+ .addValidator(StandardValidators.BOOLEAN_VALIDATOR)
+ .defaultValue(Boolean.FALSE.toString())
+ .build();
+ public static final PropertyDescriptor POLL_ON_INGESTION_STATUS = new PropertyDescriptor
+ .Builder().name("Poll for Ingestion Status")
+ .displayName("Poll for Ingestion Status")
+ .description("Determines whether to poll on ingestion status after an ingestion to Azure Data Explorer is completed")
+ .required(false)
+ .allowableValues(Boolean.TRUE.toString(), Boolean.FALSE.toString())
+ .addValidator(StandardValidators.BOOLEAN_VALIDATOR)
+ .defaultValue(Boolean.FALSE.toString())
+ .build();
+ public static final PropertyDescriptor ADX_SERVICE = new PropertyDescriptor
+ .Builder().name("Kusto Ingest Service")
+ .displayName("Kusto Ingest Service")
+ .description("Azure Data Explorer Kusto Ingest Service")
+ .required(true)
+ .identifiesControllerService(KustoIngestService.class)
+ .build();
+ public static final PropertyDescriptor DATA_FORMAT = new PropertyDescriptor.Builder()
+ .name("Data Format")
Review Comment:
yeah makes sense.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] tanmaya-panda1 commented on a diff in pull request #7624: NIFI-11958 Azure Data Explorer as sink
Posted by "tanmaya-panda1 (via GitHub)" <gi...@apache.org>.
tanmaya-panda1 commented on code in PR #7624:
URL: https://github.com/apache/nifi/pull/7624#discussion_r1336392441
##########
nifi-nar-bundles/nifi-azure-bundle/nifi-azure-services-api/pom.xml:
##########
@@ -36,6 +36,16 @@
<groupId>org.apache.nifi</groupId>
<artifactId>nifi-api</artifactId>
</dependency>
+ <dependency>
+ <groupId>com.microsoft.azure.kusto</groupId>
+ <artifactId>kusto-data</artifactId>
+ <version>5.0.1</version>
+ </dependency>
+ <dependency>
+ <groupId>com.microsoft.azure.kusto</groupId>
+ <artifactId>kusto-ingest</artifactId>
+ <version>5.0.1</version>
+ </dependency>
Review Comment:
yeah, I struggled to decouple the dependencies of azure-kusto-java sdk from API package level. The interfaces KustoIngestService and KustoQueryService are decoupled from kusto java SDK dependency.
But had to include dependencies in the API package level since we had com.microsoft.azure.kusto.ingest.IngestionProperties while submitting an ingestion request to Kusto cluster and com.microsoft.azure.kusto.data.KustoResultSetTable as a part ingestionResult from kusto after ingestion.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] NIFI-11958 Azure Data Explorer as sink [nifi]
Posted by "exceptionfactory (via GitHub)" <gi...@apache.org>.
exceptionfactory commented on code in PR #7624:
URL: https://github.com/apache/nifi/pull/7624#discussion_r1394354709
##########
nifi-nar-bundles/nifi-azure-bundle/nifi-azure-processors/src/main/java/org/apache/nifi/processors/azure/data/explorer/PutAzureDataExplorer.java:
##########
@@ -0,0 +1,291 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.azure.data.explorer;
+
+import org.apache.commons.lang3.StringUtils;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.expression.ExpressionLanguageScope;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.services.azure.data.explorer.KustoIngestDataFormat;
+import org.apache.nifi.services.azure.data.explorer.KustoIngestQueryResponse;
+import org.apache.nifi.services.azure.data.explorer.KustoIngestService;
+import org.apache.nifi.services.azure.data.explorer.KustoIngestionRequest;
+import org.apache.nifi.services.azure.data.explorer.KustoIngestionResult;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.net.URISyntaxException;
+import java.time.Duration;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+import java.util.concurrent.TimeUnit;
+
+@Tags({"Azure", "Kusto", "ADX", "Explorer", "Data"})
+@CapabilityDescription("The PutAzureDataExplorer acts as a ADX sink connector which sends flowFiles using the ADX-Service to the provided Azure Data" +
+ "Explorer Ingest Endpoint. The data can be sent through queued ingestion or streaming ingestion to the Azure Data Explorer cluster.")
+@InputRequirement(InputRequirement.Requirement.INPUT_REQUIRED)
+public class PutAzureDataExplorer extends AbstractProcessor {
+
+ public static final String FETCH_TABLE_COMMAND = "%s | count";
+ public static final String STREAMING_POLICY_SHOW_COMMAND = ".show %s %s policy streamingingestion";
+ public static final String DATABASE = "database";
+ public static final AllowableValue IGNORE_FIRST_RECORD_YES = new AllowableValue(
+ "YES", "YES",
+ "Ignore first record during ingestion");
+
+ public static final AllowableValue IGNORE_FIRST_RECORD_NO = new AllowableValue(
+ "NO", "NO",
+ "Do not ignore first record during ingestion");
+
+ public static final PropertyDescriptor DATABASE_NAME = new PropertyDescriptor.Builder()
+ .name("Database Name")
+ .displayName("Database Name")
+ .description("Azure Data Explorer Database Name for querying")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+ .build();
+ public static final PropertyDescriptor TABLE_NAME = new PropertyDescriptor.Builder()
+ .name("Table Name")
+ .displayName("Table Name")
+ .description("Azure Data Explorer Table Name for ingesting data")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+ .build();
+ public static final PropertyDescriptor MAPPING_NAME = new PropertyDescriptor
+ .Builder().name("Ingest Mapping name")
+ .displayName("Ingest Mapping name")
+ .description("The name of the mapping responsible for storing the data in the appropriate columns.")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor IS_STREAMING_ENABLED = new PropertyDescriptor
+ .Builder().name("Streaming Enabled")
+ .displayName("Streaming Enabled")
+ .description("This property determines whether we want to stream data to ADX.")
+ .required(false)
+ .allowableValues(Boolean.TRUE.toString(), Boolean.FALSE.toString())
+ .addValidator(StandardValidators.BOOLEAN_VALIDATOR)
+ .defaultValue(Boolean.FALSE.toString())
+ .build();
+ public static final PropertyDescriptor POLL_ON_INGESTION_STATUS = new PropertyDescriptor
+ .Builder().name("Poll On Ingestion Status")
+ .displayName("Whether To Poll On Ingestion Status")
+ .description("This property determines whether we want to poll on ingestion status after an ingestion to ADX is completed")
+ .required(false)
+ .allowableValues(Boolean.TRUE.toString(), Boolean.FALSE.toString())
+ .addValidator(StandardValidators.BOOLEAN_VALIDATOR)
+ .defaultValue(Boolean.FALSE.toString())
+ .build();
+ public static final PropertyDescriptor ADX_SERVICE = new PropertyDescriptor
+ .Builder().name("Kusto Ingest Service")
+ .displayName("Kusto Ingest Service")
+ .description("Azure Data Explorer Kusto Ingest Service")
+ .required(true)
+ .identifiesControllerService(KustoIngestService.class)
+ .build();
+ public static final PropertyDescriptor DATA_FORMAT = new PropertyDescriptor.Builder()
+ .name("Data Format")
+ .displayName("Data Format")
+ .description("The format of the data that is sent to Azure Data Explorer.")
+ .required(true)
+ .allowableValues(KustoIngestDataFormat.values())
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor IGNORE_FIRST_RECORD = new PropertyDescriptor.Builder()
+ .name("Ingestion Ignore First Record")
+ .displayName("Ingestion Ignore First Record")
+ .description("Defines whether ignore first record while ingestion.")
+ .required(false)
+ .allowableValues(IGNORE_FIRST_RECORD_YES, IGNORE_FIRST_RECORD_NO)
+ .defaultValue(IGNORE_FIRST_RECORD_NO.getValue())
+ .build();
+ public static final PropertyDescriptor INGESTION_STATUS_POLLING_TIMEOUT = new PropertyDescriptor.Builder()
+ .name("Ingest Status Polling Timeout")
+ .displayName("Ingest Status Polling Timeout")
+ .description("Defines the value of timeout for polling on ingestion status")
+ .required(false)
+ .dependsOn(POLL_ON_INGESTION_STATUS, Boolean.TRUE.toString())
+ .addValidator(StandardValidators.TIME_PERIOD_VALIDATOR)
+ .defaultValue("10 m")
+ .build();
+ public static final PropertyDescriptor INGESTION_STATUS_POLLING_INTERVAL = new PropertyDescriptor.Builder()
+ .name("Ingest Status Polling Interval")
+ .displayName("Ingest Status Polling Interval")
+ .description("Defines the value of timeout for polling on ingestion status in seconds.")
+ .required(false)
+ .dependsOn(POLL_ON_INGESTION_STATUS, Boolean.TRUE.toString())
+ .addValidator(StandardValidators.TIME_PERIOD_VALIDATOR)
+ .defaultValue("5 s")
+ .build();
+ public static final Relationship SUCCESS = new Relationship.Builder()
+ .name("Success")
+ .description("Relationship For Success")
+ .build();
+ public static final Relationship FAILURE = new Relationship.Builder()
+ .name("Failure")
Review Comment:
```suggestion
.name("failure")
```
##########
nifi-nar-bundles/nifi-azure-bundle/nifi-azure-services-api/src/main/java/org/apache/nifi/services/azure/data/explorer/KustoIngestService.java:
##########
@@ -0,0 +1,35 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.services.azure.data.explorer;
+
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.controller.ControllerService;
+
+import java.net.URISyntaxException;
+
+@Tags({"azure", "adx", "explorer", "kusto", "ingest"})
+@CapabilityDescription("Connection-Service for Azure ADX (Kusto) ingestion cluster to ingest data.")
+public interface KustoIngestService extends ControllerService {
+ KustoIngestionResult ingestData(KustoIngestionRequest kustoIngestionRequest) throws URISyntaxException;
Review Comment:
The `URISyntaxException` is too specific to the implementation and should be removed, it could be wrapped in some other type of `IllegalArgumentException` in the implementation if needed.
```suggestion
KustoIngestionResult ingestData(KustoIngestionRequest kustoIngestionRequest);
```
##########
nifi-nar-bundles/nifi-azure-bundle/nifi-azure-services-api/src/main/java/org/apache/nifi/services/azure/data/explorer/KustoIngestService.java:
##########
@@ -0,0 +1,35 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.services.azure.data.explorer;
+
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.controller.ControllerService;
+
+import java.net.URISyntaxException;
+
+@Tags({"azure", "adx", "explorer", "kusto", "ingest"})
+@CapabilityDescription("Connection-Service for Azure ADX (Kusto) ingestion cluster to ingest data.")
+public interface KustoIngestService extends ControllerService {
+ KustoIngestionResult ingestData(KustoIngestionRequest kustoIngestionRequest) throws URISyntaxException;
+
+ KustoIngestQueryResponse executeQuery(String databaseName, String query);
+
+ KustoIngestQueryResponse checkIfStreamingIsEnabled(String databaseName, String query);
Review Comment:
The method name should be shortened. The `query` parameter should be changed to `tableName` and the query itself should be constructed in the service implementation, instead of the Processor.
```suggestion
KustoIngestQueryResponse isStreamingEnabled(String databaseName, String query);
```
##########
nifi-nar-bundles/nifi-azure-bundle/nifi-azure-processors/src/main/java/org/apache/nifi/processors/azure/data/explorer/PutAzureDataExplorer.java:
##########
@@ -0,0 +1,291 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.azure.data.explorer;
+
+import org.apache.commons.lang3.StringUtils;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.expression.ExpressionLanguageScope;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.services.azure.data.explorer.KustoIngestDataFormat;
+import org.apache.nifi.services.azure.data.explorer.KustoIngestQueryResponse;
+import org.apache.nifi.services.azure.data.explorer.KustoIngestService;
+import org.apache.nifi.services.azure.data.explorer.KustoIngestionRequest;
+import org.apache.nifi.services.azure.data.explorer.KustoIngestionResult;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.net.URISyntaxException;
+import java.time.Duration;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+import java.util.concurrent.TimeUnit;
+
+@Tags({"Azure", "Kusto", "ADX", "Explorer", "Data"})
+@CapabilityDescription("The PutAzureDataExplorer acts as a ADX sink connector which sends flowFiles using the ADX-Service to the provided Azure Data" +
+ "Explorer Ingest Endpoint. The data can be sent through queued ingestion or streaming ingestion to the Azure Data Explorer cluster.")
+@InputRequirement(InputRequirement.Requirement.INPUT_REQUIRED)
+public class PutAzureDataExplorer extends AbstractProcessor {
+
+ public static final String FETCH_TABLE_COMMAND = "%s | count";
+ public static final String STREAMING_POLICY_SHOW_COMMAND = ".show %s %s policy streamingingestion";
+ public static final String DATABASE = "database";
+ public static final AllowableValue IGNORE_FIRST_RECORD_YES = new AllowableValue(
+ "YES", "YES",
+ "Ignore first record during ingestion");
+
+ public static final AllowableValue IGNORE_FIRST_RECORD_NO = new AllowableValue(
+ "NO", "NO",
+ "Do not ignore first record during ingestion");
+
+ public static final PropertyDescriptor DATABASE_NAME = new PropertyDescriptor.Builder()
+ .name("Database Name")
+ .displayName("Database Name")
+ .description("Azure Data Explorer Database Name for querying")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+ .build();
+ public static final PropertyDescriptor TABLE_NAME = new PropertyDescriptor.Builder()
+ .name("Table Name")
+ .displayName("Table Name")
+ .description("Azure Data Explorer Table Name for ingesting data")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+ .build();
+ public static final PropertyDescriptor MAPPING_NAME = new PropertyDescriptor
+ .Builder().name("Ingest Mapping name")
+ .displayName("Ingest Mapping name")
+ .description("The name of the mapping responsible for storing the data in the appropriate columns.")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor IS_STREAMING_ENABLED = new PropertyDescriptor
+ .Builder().name("Streaming Enabled")
+ .displayName("Streaming Enabled")
+ .description("This property determines whether we want to stream data to ADX.")
+ .required(false)
+ .allowableValues(Boolean.TRUE.toString(), Boolean.FALSE.toString())
+ .addValidator(StandardValidators.BOOLEAN_VALIDATOR)
+ .defaultValue(Boolean.FALSE.toString())
+ .build();
+ public static final PropertyDescriptor POLL_ON_INGESTION_STATUS = new PropertyDescriptor
+ .Builder().name("Poll On Ingestion Status")
+ .displayName("Whether To Poll On Ingestion Status")
+ .description("This property determines whether we want to poll on ingestion status after an ingestion to ADX is completed")
+ .required(false)
+ .allowableValues(Boolean.TRUE.toString(), Boolean.FALSE.toString())
+ .addValidator(StandardValidators.BOOLEAN_VALIDATOR)
+ .defaultValue(Boolean.FALSE.toString())
+ .build();
+ public static final PropertyDescriptor ADX_SERVICE = new PropertyDescriptor
+ .Builder().name("Kusto Ingest Service")
+ .displayName("Kusto Ingest Service")
+ .description("Azure Data Explorer Kusto Ingest Service")
+ .required(true)
+ .identifiesControllerService(KustoIngestService.class)
+ .build();
+ public static final PropertyDescriptor DATA_FORMAT = new PropertyDescriptor.Builder()
+ .name("Data Format")
+ .displayName("Data Format")
+ .description("The format of the data that is sent to Azure Data Explorer.")
+ .required(true)
+ .allowableValues(KustoIngestDataFormat.values())
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor IGNORE_FIRST_RECORD = new PropertyDescriptor.Builder()
+ .name("Ingestion Ignore First Record")
+ .displayName("Ingestion Ignore First Record")
+ .description("Defines whether ignore first record while ingestion.")
+ .required(false)
+ .allowableValues(IGNORE_FIRST_RECORD_YES, IGNORE_FIRST_RECORD_NO)
+ .defaultValue(IGNORE_FIRST_RECORD_NO.getValue())
+ .build();
+ public static final PropertyDescriptor INGESTION_STATUS_POLLING_TIMEOUT = new PropertyDescriptor.Builder()
+ .name("Ingest Status Polling Timeout")
+ .displayName("Ingest Status Polling Timeout")
+ .description("Defines the value of timeout for polling on ingestion status")
+ .required(false)
+ .dependsOn(POLL_ON_INGESTION_STATUS, Boolean.TRUE.toString())
+ .addValidator(StandardValidators.TIME_PERIOD_VALIDATOR)
+ .defaultValue("10 m")
+ .build();
+ public static final PropertyDescriptor INGESTION_STATUS_POLLING_INTERVAL = new PropertyDescriptor.Builder()
+ .name("Ingest Status Polling Interval")
+ .displayName("Ingest Status Polling Interval")
+ .description("Defines the value of timeout for polling on ingestion status in seconds.")
+ .required(false)
+ .dependsOn(POLL_ON_INGESTION_STATUS, Boolean.TRUE.toString())
+ .addValidator(StandardValidators.TIME_PERIOD_VALIDATOR)
+ .defaultValue("5 s")
+ .build();
+ public static final Relationship SUCCESS = new Relationship.Builder()
+ .name("Success")
Review Comment:
```suggestion
.name("success")
```
##########
nifi-nar-bundles/nifi-azure-bundle/nifi-azure-services-api/src/main/java/org/apache/nifi/services/azure/data/explorer/KustoQueryResponse.java:
##########
@@ -27,16 +28,27 @@ public class KustoQueryResponse {
private final String errorMessage;
+ private final List<List<Object>> queryResult;
Review Comment:
Now that the interface has been redefined, can this list of results be removed? The values should be passed in the service implementation methods, which would avoid adding this member variable to the public response object.
##########
nifi-nar-bundles/nifi-azure-bundle/nifi-azure-processors/src/main/java/org/apache/nifi/services/azure/data/explorer/StandardKustoIngestService.java:
##########
@@ -0,0 +1,404 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.services.azure.data.explorer;
+
+import com.microsoft.azure.kusto.data.Client;
+import com.microsoft.azure.kusto.data.ClientFactory;
+import com.microsoft.azure.kusto.data.KustoResultColumn;
+import com.microsoft.azure.kusto.data.KustoResultSetTable;
+import com.microsoft.azure.kusto.data.auth.ConnectionStringBuilder;
+import com.microsoft.azure.kusto.data.exceptions.DataClientException;
+import com.microsoft.azure.kusto.data.exceptions.DataServiceException;
+import com.microsoft.azure.kusto.ingest.IngestClientFactory;
+import com.microsoft.azure.kusto.ingest.IngestionMapping;
+import com.microsoft.azure.kusto.ingest.IngestionProperties;
+import com.microsoft.azure.kusto.ingest.ManagedStreamingIngestClient;
+import com.microsoft.azure.kusto.ingest.QueuedIngestClient;
+import com.microsoft.azure.kusto.ingest.exceptions.IngestionClientException;
+import com.microsoft.azure.kusto.ingest.exceptions.IngestionServiceException;
+import com.microsoft.azure.kusto.ingest.result.IngestionResult;
+import com.microsoft.azure.kusto.ingest.result.IngestionStatus;
+import com.microsoft.azure.kusto.ingest.result.OperationStatus;
+import com.microsoft.azure.kusto.ingest.source.StreamSourceInfo;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnEnabled;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.controller.AbstractControllerService;
+import org.apache.nifi.controller.ConfigurationContext;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+
+import java.io.IOException;
+import java.net.URISyntaxException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+
+@Tags({"Azure", "ADX", "Kusto", "ingest", "azure"})
+@CapabilityDescription("Sends batches of flowfile content or stream flowfile content to an Azure ADX cluster.")
+public class StandardKustoIngestService extends AbstractControllerService implements KustoIngestService {
+
+ public static final PropertyDescriptor AUTHENTICATION_STRATEGY = new PropertyDescriptor
+ .Builder().name("Authentication Strategy")
+ .displayName("Authentication Strategy")
+ .description("Authentication method for access to Azure Data Explorer")
+ .required(true)
+ .defaultValue(KustoAuthenticationStrategy.MANAGED_IDENTITY.getValue())
+ .allowableValues(KustoAuthenticationStrategy.class)
+ .build();
+
+ public static final PropertyDescriptor APPLICATION_CLIENT_ID = new PropertyDescriptor
+ .Builder().name("Application Client ID")
+ .displayName("Application Client ID")
+ .description("Azure Data Explorer Application Client Identifier for Authentication")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+
+ public static final PropertyDescriptor APPLICATION_KEY = new PropertyDescriptor
+ .Builder().name("Application Key")
+ .displayName("Application Key")
+ .description("Azure Data Explorer Application Key for Authentication")
+ .required(true)
+ .sensitive(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .dependsOn(AUTHENTICATION_STRATEGY, KustoAuthenticationStrategy.APPLICATION_CREDENTIALS.getValue())
+ .build();
+
+ public static final PropertyDescriptor APPLICATION_TENANT_ID = new PropertyDescriptor.Builder()
+ .name("Application Tenant ID")
+ .displayName("Application Tenant ID")
+ .description("Azure Data Explorer Application Tenant Identifier for Authentication")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .dependsOn(AUTHENTICATION_STRATEGY, KustoAuthenticationStrategy.APPLICATION_CREDENTIALS.getValue())
+ .build();
+
+ public static final PropertyDescriptor CLUSTER_URI = new PropertyDescriptor
+ .Builder().name("Cluster URI")
+ .displayName("Cluster URI")
+ .description("Azure Data Explorer Cluster URI")
+ .required(true)
+ .addValidator(StandardValidators.URL_VALIDATOR)
+ .build();
+
+ private static final List<PropertyDescriptor> PROPERTY_DESCRIPTORS = Collections.unmodifiableList(Arrays.asList(
+ AUTHENTICATION_STRATEGY,
+ APPLICATION_CLIENT_ID,
+ APPLICATION_KEY,
+ APPLICATION_TENANT_ID,
+ CLUSTER_URI));
+
+ public static final Pair<String, String> NIFI_SINK = Pair.of("processor", StandardKustoIngestService.class.getSimpleName());
+
+ private volatile QueuedIngestClient queuedIngestClient;
+
+ private volatile ManagedStreamingIngestClient managedStreamingIngestClient;
+
+ private volatile Client executionClient;
+
+ @Override
+ public List<PropertyDescriptor> getSupportedPropertyDescriptors() {
+ return PROPERTY_DESCRIPTORS;
+ }
+
+ /**
+ * @param context the configuration context
+ */
+ @OnEnabled
+ public void onEnabled(final ConfigurationContext context) throws ProcessException, URISyntaxException {
+
+ final String applicationClientId = context.getProperty(APPLICATION_CLIENT_ID).getValue();
+ final String applicationKey = context.getProperty(APPLICATION_KEY).getValue();
+ final String applicationTenantId = context.getProperty(APPLICATION_TENANT_ID).getValue();
+ final String clusterUri = context.getProperty(CLUSTER_URI).getValue();
+ final KustoAuthenticationStrategy kustoAuthenticationStrategy = KustoAuthenticationStrategy.valueOf(context.getProperty(AUTHENTICATION_STRATEGY).getValue());
+
+ if (this.queuedIngestClient == null) {
+ this.queuedIngestClient = createKustoQueuedIngestClient(clusterUri, applicationClientId, applicationKey, applicationTenantId, kustoAuthenticationStrategy);
+ }
+
+ if (this.managedStreamingIngestClient == null) {
+ this.managedStreamingIngestClient = createKustoStreamingIngestClient(clusterUri, applicationClientId, applicationKey, applicationTenantId, kustoAuthenticationStrategy);
+ }
+
+ if (this.executionClient == null) {
+ this.executionClient = createKustoExecutionClient(clusterUri, applicationClientId, applicationKey, applicationTenantId, kustoAuthenticationStrategy);
+ }
+
+ }
+
+ @OnStopped
+ public final void onStopped() {
+ if (this.queuedIngestClient != null) {
+ try {
+ this.queuedIngestClient.close();
+ } catch (IOException e) {
+ getLogger().error("Closing Azure ADX Queued Ingest Client failed", e);
+ } finally {
+ this.queuedIngestClient = null;
+ }
+ }
+ if (this.managedStreamingIngestClient != null) {
+ try {
+ this.managedStreamingIngestClient.close();
+ } catch (IOException e) {
+ getLogger().error("Closing Azure ADX Managed Streaming Ingest Client failed", e);
+ } finally {
+ this.managedStreamingIngestClient = null;
+ }
+ }
+ if (this.executionClient != null) {
+ try {
+ this.executionClient.close();
+ } catch (IOException e) {
+ getLogger().error("Closing Azure ADX Execution Client failed", e);
+ } finally {
+ this.executionClient = null;
+ }
+ }
+ }
+
+
+ protected QueuedIngestClient createKustoQueuedIngestClient(final String clusterUrl,
+ final String appId,
+ final String appKey,
+ final String appTenant,
+ final KustoAuthenticationStrategy kustoAuthStrategy) throws URISyntaxException {
+ ConnectionStringBuilder ingestConnectionStringBuilder = createKustoEngineConnectionString(clusterUrl, appId, appKey, appTenant, kustoAuthStrategy);
+ return IngestClientFactory.createClient(ingestConnectionStringBuilder);
+ }
+
+ protected ManagedStreamingIngestClient createKustoStreamingIngestClient(final String clusterUrl,
+ final String appId,
+ final String appKey,
+ final String appTenant,
+ final KustoAuthenticationStrategy kustoAuthStrategy) throws URISyntaxException {
+ ConnectionStringBuilder ingestConnectionStringBuilder = createKustoEngineConnectionString(clusterUrl, appId, appKey, appTenant, kustoAuthStrategy);
+ ConnectionStringBuilder streamingConnectionStringBuilder = createKustoEngineConnectionString(clusterUrl, appId, appKey, appTenant, kustoAuthStrategy);
+ return IngestClientFactory.createManagedStreamingIngestClient(ingestConnectionStringBuilder, streamingConnectionStringBuilder);
+ }
+
+ public KustoIngestionResult ingestData(KustoIngestionRequest kustoIngestionRequest) throws URISyntaxException {
+ StreamSourceInfo info = new StreamSourceInfo(kustoIngestionRequest.getInputStream());
+ //ingest data
+ IngestionResult ingestionResult;
+ IngestionProperties ingestionProperties = new IngestionProperties(kustoIngestionRequest.getDatabaseName(),
+ kustoIngestionRequest.getTableName());
+
+ IngestionMapping.IngestionMappingKind ingestionMappingKind = setDataFormatAndMapping(kustoIngestionRequest.getDataFormat(), ingestionProperties);
+ if (StringUtils.isNotEmpty(kustoIngestionRequest.getMappingName()) && ingestionMappingKind != null) {
+ ingestionProperties.setIngestionMapping(kustoIngestionRequest.getMappingName(), ingestionMappingKind);
+ }
+
+ ingestionProperties.setReportLevel(IngestionProperties.IngestionReportLevel.FAILURES_AND_SUCCESSES);
+ ingestionProperties.setReportMethod(IngestionProperties.IngestionReportMethod.TABLE);
+ ingestionProperties.setFlushImmediately(false);
+ ingestionProperties.setIgnoreFirstRecord(StringUtils.equalsIgnoreCase(kustoIngestionRequest.getIgnoreFirstRecord(), "YES"));
+
+ try {
+ if (kustoIngestionRequest.isStreamingEnabled()) {
+ ingestionResult = managedStreamingIngestClient.ingestFromStream(info, ingestionProperties);
+ } else {
+ ingestionResult = queuedIngestClient.ingestFromStream(info, ingestionProperties);
+ }
+ if (kustoIngestionRequest.pollOnIngestionStatus()) {
+ List<IngestionStatus> statuses = initializeKustoIngestionStatusAsPending();
+ long startTime = System.currentTimeMillis();
+ long timeoutMillis = kustoIngestionRequest.getIngestionStatusPollingTimeout().toMillis();
+ // Calculate the end time based on the timeout duration
+ long endTime = startTime + timeoutMillis;
+ while (System.currentTimeMillis() < endTime) {
+ // Get the status of the ingestion operation
+ List<IngestionStatus> statuses1 = ingestionResult.getIngestionStatusCollection();
+ if (statuses1.get(0).status == OperationStatus.Succeeded
+ || statuses1.get(0).status == OperationStatus.Failed
+ || statuses1.get(0).status == OperationStatus.PartiallySucceeded) {
+ statuses = statuses1;
+ break;
+ }
+ // Sleep for before checking again
+ Thread.sleep(kustoIngestionRequest.getIngestionStatusPollingInterval().toMillis());
+ }
+ // Check if the timeout has been exceeded
+ if (System.currentTimeMillis() - startTime >= timeoutMillis) {
+ throw new ProcessException(String.format("Timeout of %s exceeded while waiting for ingestion status", kustoIngestionRequest.getIngestionStatusPollingTimeout()));
+ }
+ return KustoIngestionResult.fromString(statuses.get(0).status.toString());
Review Comment:
Recommend moving this block into a separate method named something like `pollIngestionStatus()`.
##########
nifi-nar-bundles/nifi-azure-bundle/nifi-azure-processors/src/main/java/org/apache/nifi/processors/azure/data/explorer/PutAzureDataExplorer.java:
##########
@@ -0,0 +1,291 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.azure.data.explorer;
+
+import org.apache.commons.lang3.StringUtils;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.expression.ExpressionLanguageScope;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.services.azure.data.explorer.KustoIngestDataFormat;
+import org.apache.nifi.services.azure.data.explorer.KustoIngestQueryResponse;
+import org.apache.nifi.services.azure.data.explorer.KustoIngestService;
+import org.apache.nifi.services.azure.data.explorer.KustoIngestionRequest;
+import org.apache.nifi.services.azure.data.explorer.KustoIngestionResult;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.net.URISyntaxException;
+import java.time.Duration;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+import java.util.concurrent.TimeUnit;
+
+@Tags({"Azure", "Kusto", "ADX", "Explorer", "Data"})
+@CapabilityDescription("The PutAzureDataExplorer acts as a ADX sink connector which sends flowFiles using the ADX-Service to the provided Azure Data" +
+ "Explorer Ingest Endpoint. The data can be sent through queued ingestion or streaming ingestion to the Azure Data Explorer cluster.")
+@InputRequirement(InputRequirement.Requirement.INPUT_REQUIRED)
+public class PutAzureDataExplorer extends AbstractProcessor {
+
+ public static final String FETCH_TABLE_COMMAND = "%s | count";
+ public static final String STREAMING_POLICY_SHOW_COMMAND = ".show %s %s policy streamingingestion";
+ public static final String DATABASE = "database";
+ public static final AllowableValue IGNORE_FIRST_RECORD_YES = new AllowableValue(
+ "YES", "YES",
+ "Ignore first record during ingestion");
+
+ public static final AllowableValue IGNORE_FIRST_RECORD_NO = new AllowableValue(
+ "NO", "NO",
+ "Do not ignore first record during ingestion");
+
+ public static final PropertyDescriptor DATABASE_NAME = new PropertyDescriptor.Builder()
+ .name("Database Name")
+ .displayName("Database Name")
+ .description("Azure Data Explorer Database Name for querying")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+ .build();
+ public static final PropertyDescriptor TABLE_NAME = new PropertyDescriptor.Builder()
+ .name("Table Name")
+ .displayName("Table Name")
+ .description("Azure Data Explorer Table Name for ingesting data")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+ .build();
+ public static final PropertyDescriptor MAPPING_NAME = new PropertyDescriptor
+ .Builder().name("Ingest Mapping name")
+ .displayName("Ingest Mapping name")
+ .description("The name of the mapping responsible for storing the data in the appropriate columns.")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor IS_STREAMING_ENABLED = new PropertyDescriptor
+ .Builder().name("Streaming Enabled")
+ .displayName("Streaming Enabled")
+ .description("This property determines whether we want to stream data to ADX.")
+ .required(false)
+ .allowableValues(Boolean.TRUE.toString(), Boolean.FALSE.toString())
+ .addValidator(StandardValidators.BOOLEAN_VALIDATOR)
+ .defaultValue(Boolean.FALSE.toString())
+ .build();
+ public static final PropertyDescriptor POLL_ON_INGESTION_STATUS = new PropertyDescriptor
+ .Builder().name("Poll On Ingestion Status")
+ .displayName("Whether To Poll On Ingestion Status")
+ .description("This property determines whether we want to poll on ingestion status after an ingestion to ADX is completed")
+ .required(false)
+ .allowableValues(Boolean.TRUE.toString(), Boolean.FALSE.toString())
+ .addValidator(StandardValidators.BOOLEAN_VALIDATOR)
+ .defaultValue(Boolean.FALSE.toString())
+ .build();
+ public static final PropertyDescriptor ADX_SERVICE = new PropertyDescriptor
+ .Builder().name("Kusto Ingest Service")
+ .displayName("Kusto Ingest Service")
+ .description("Azure Data Explorer Kusto Ingest Service")
+ .required(true)
+ .identifiesControllerService(KustoIngestService.class)
+ .build();
+ public static final PropertyDescriptor DATA_FORMAT = new PropertyDescriptor.Builder()
+ .name("Data Format")
+ .displayName("Data Format")
+ .description("The format of the data that is sent to Azure Data Explorer.")
+ .required(true)
+ .allowableValues(KustoIngestDataFormat.values())
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor IGNORE_FIRST_RECORD = new PropertyDescriptor.Builder()
+ .name("Ingestion Ignore First Record")
+ .displayName("Ingestion Ignore First Record")
+ .description("Defines whether ignore first record while ingestion.")
+ .required(false)
+ .allowableValues(IGNORE_FIRST_RECORD_YES, IGNORE_FIRST_RECORD_NO)
+ .defaultValue(IGNORE_FIRST_RECORD_NO.getValue())
+ .build();
+ public static final PropertyDescriptor INGESTION_STATUS_POLLING_TIMEOUT = new PropertyDescriptor.Builder()
+ .name("Ingest Status Polling Timeout")
+ .displayName("Ingest Status Polling Timeout")
+ .description("Defines the value of timeout for polling on ingestion status")
+ .required(false)
+ .dependsOn(POLL_ON_INGESTION_STATUS, Boolean.TRUE.toString())
+ .addValidator(StandardValidators.TIME_PERIOD_VALIDATOR)
+ .defaultValue("10 m")
+ .build();
+ public static final PropertyDescriptor INGESTION_STATUS_POLLING_INTERVAL = new PropertyDescriptor.Builder()
+ .name("Ingest Status Polling Interval")
+ .displayName("Ingest Status Polling Interval")
+ .description("Defines the value of timeout for polling on ingestion status in seconds.")
+ .required(false)
+ .dependsOn(POLL_ON_INGESTION_STATUS, Boolean.TRUE.toString())
+ .addValidator(StandardValidators.TIME_PERIOD_VALIDATOR)
+ .defaultValue("5 s")
+ .build();
+ public static final Relationship SUCCESS = new Relationship.Builder()
+ .name("Success")
+ .description("Relationship For Success")
+ .build();
+ public static final Relationship FAILURE = new Relationship.Builder()
+ .name("Failure")
+ .description("Relationship For Failure")
+ .build();
+
+ public static final PropertyDescriptor ROUTE_PARTIALLY_SUCCESSFUL_INGESTION = new PropertyDescriptor.Builder()
+ .name("Route Partially Successful Ingestion Records")
+ .displayName("Route Partially Successful Ingestion Records")
+ .description("Defines where to route partially successful ingestion records.")
+ .required(false)
+ .allowableValues(SUCCESS.getName(), FAILURE.getName())
+ .defaultValue(FAILURE.getName())
+ .build();
+ private static final List<PropertyDescriptor> descriptors;
+ private static final Set<Relationship> relationships;
+ private transient KustoIngestService service;
+ private boolean streamingEnabled;
+ private boolean pollOnIngestionStatus;
Review Comment:
These values should not be member variables.
```suggestion
```
##########
nifi-nar-bundles/nifi-azure-bundle/nifi-azure-services-api/src/main/java/org/apache/nifi/services/azure/data/explorer/KustoIngestService.java:
##########
@@ -0,0 +1,35 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.services.azure.data.explorer;
+
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.controller.ControllerService;
+
+import java.net.URISyntaxException;
+
+@Tags({"azure", "adx", "explorer", "kusto", "ingest"})
+@CapabilityDescription("Connection-Service for Azure ADX (Kusto) ingestion cluster to ingest data.")
+public interface KustoIngestService extends ControllerService {
+ KustoIngestionResult ingestData(KustoIngestionRequest kustoIngestionRequest) throws URISyntaxException;
+
+ KustoIngestQueryResponse executeQuery(String databaseName, String query);
+
+ KustoIngestQueryResponse checkIfStreamingIsEnabled(String databaseName, String query);
+
+ KustoIngestQueryResponse checkIfIngestorPrivilegeIsEnabled(String databaseName, String query);
Review Comment:
```suggestion
KustoIngestQueryResponse isIngestorPrivilegeEnabled(String databaseName, String tableName);
```
##########
nifi-nar-bundles/nifi-azure-bundle/nifi-azure-processors/src/main/java/org/apache/nifi/processors/azure/data/explorer/PutAzureDataExplorer.java:
##########
@@ -0,0 +1,291 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.azure.data.explorer;
+
+import org.apache.commons.lang3.StringUtils;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.expression.ExpressionLanguageScope;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.services.azure.data.explorer.KustoIngestDataFormat;
+import org.apache.nifi.services.azure.data.explorer.KustoIngestQueryResponse;
+import org.apache.nifi.services.azure.data.explorer.KustoIngestService;
+import org.apache.nifi.services.azure.data.explorer.KustoIngestionRequest;
+import org.apache.nifi.services.azure.data.explorer.KustoIngestionResult;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.net.URISyntaxException;
+import java.time.Duration;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+import java.util.concurrent.TimeUnit;
+
+@Tags({"Azure", "Kusto", "ADX", "Explorer", "Data"})
+@CapabilityDescription("The PutAzureDataExplorer acts as a ADX sink connector which sends flowFiles using the ADX-Service to the provided Azure Data" +
+ "Explorer Ingest Endpoint. The data can be sent through queued ingestion or streaming ingestion to the Azure Data Explorer cluster.")
+@InputRequirement(InputRequirement.Requirement.INPUT_REQUIRED)
+public class PutAzureDataExplorer extends AbstractProcessor {
+
+ public static final String FETCH_TABLE_COMMAND = "%s | count";
+ public static final String STREAMING_POLICY_SHOW_COMMAND = ".show %s %s policy streamingingestion";
+ public static final String DATABASE = "database";
+ public static final AllowableValue IGNORE_FIRST_RECORD_YES = new AllowableValue(
+ "YES", "YES",
+ "Ignore first record during ingestion");
+
+ public static final AllowableValue IGNORE_FIRST_RECORD_NO = new AllowableValue(
+ "NO", "NO",
+ "Do not ignore first record during ingestion");
+
+ public static final PropertyDescriptor DATABASE_NAME = new PropertyDescriptor.Builder()
+ .name("Database Name")
+ .displayName("Database Name")
+ .description("Azure Data Explorer Database Name for querying")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+ .build();
+ public static final PropertyDescriptor TABLE_NAME = new PropertyDescriptor.Builder()
+ .name("Table Name")
+ .displayName("Table Name")
+ .description("Azure Data Explorer Table Name for ingesting data")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+ .build();
+ public static final PropertyDescriptor MAPPING_NAME = new PropertyDescriptor
+ .Builder().name("Ingest Mapping name")
+ .displayName("Ingest Mapping name")
+ .description("The name of the mapping responsible for storing the data in the appropriate columns.")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor IS_STREAMING_ENABLED = new PropertyDescriptor
+ .Builder().name("Streaming Enabled")
+ .displayName("Streaming Enabled")
+ .description("This property determines whether we want to stream data to ADX.")
+ .required(false)
+ .allowableValues(Boolean.TRUE.toString(), Boolean.FALSE.toString())
+ .addValidator(StandardValidators.BOOLEAN_VALIDATOR)
+ .defaultValue(Boolean.FALSE.toString())
+ .build();
+ public static final PropertyDescriptor POLL_ON_INGESTION_STATUS = new PropertyDescriptor
+ .Builder().name("Poll On Ingestion Status")
+ .displayName("Whether To Poll On Ingestion Status")
Review Comment:
```suggestion
.Builder().name("Poll for Ingestion Status")
.displayName("Poll for Ingestion Status")
```
##########
nifi-nar-bundles/nifi-azure-bundle/nifi-azure-processors/src/main/java/org/apache/nifi/processors/azure/data/explorer/PutAzureDataExplorer.java:
##########
@@ -0,0 +1,291 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.azure.data.explorer;
+
+import org.apache.commons.lang3.StringUtils;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.expression.ExpressionLanguageScope;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.services.azure.data.explorer.KustoIngestDataFormat;
+import org.apache.nifi.services.azure.data.explorer.KustoIngestQueryResponse;
+import org.apache.nifi.services.azure.data.explorer.KustoIngestService;
+import org.apache.nifi.services.azure.data.explorer.KustoIngestionRequest;
+import org.apache.nifi.services.azure.data.explorer.KustoIngestionResult;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.net.URISyntaxException;
+import java.time.Duration;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+import java.util.concurrent.TimeUnit;
+
+@Tags({"Azure", "Kusto", "ADX", "Explorer", "Data"})
+@CapabilityDescription("The PutAzureDataExplorer acts as a ADX sink connector which sends flowFiles using the ADX-Service to the provided Azure Data" +
+ "Explorer Ingest Endpoint. The data can be sent through queued ingestion or streaming ingestion to the Azure Data Explorer cluster.")
+@InputRequirement(InputRequirement.Requirement.INPUT_REQUIRED)
+public class PutAzureDataExplorer extends AbstractProcessor {
+
+ public static final String FETCH_TABLE_COMMAND = "%s | count";
+ public static final String STREAMING_POLICY_SHOW_COMMAND = ".show %s %s policy streamingingestion";
+ public static final String DATABASE = "database";
+ public static final AllowableValue IGNORE_FIRST_RECORD_YES = new AllowableValue(
+ "YES", "YES",
+ "Ignore first record during ingestion");
+
+ public static final AllowableValue IGNORE_FIRST_RECORD_NO = new AllowableValue(
+ "NO", "NO",
+ "Do not ignore first record during ingestion");
+
+ public static final PropertyDescriptor DATABASE_NAME = new PropertyDescriptor.Builder()
+ .name("Database Name")
+ .displayName("Database Name")
+ .description("Azure Data Explorer Database Name for querying")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+ .build();
+ public static final PropertyDescriptor TABLE_NAME = new PropertyDescriptor.Builder()
+ .name("Table Name")
+ .displayName("Table Name")
+ .description("Azure Data Explorer Table Name for ingesting data")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+ .build();
+ public static final PropertyDescriptor MAPPING_NAME = new PropertyDescriptor
+ .Builder().name("Ingest Mapping name")
+ .displayName("Ingest Mapping name")
+ .description("The name of the mapping responsible for storing the data in the appropriate columns.")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor IS_STREAMING_ENABLED = new PropertyDescriptor
+ .Builder().name("Streaming Enabled")
+ .displayName("Streaming Enabled")
+ .description("This property determines whether we want to stream data to ADX.")
+ .required(false)
+ .allowableValues(Boolean.TRUE.toString(), Boolean.FALSE.toString())
+ .addValidator(StandardValidators.BOOLEAN_VALIDATOR)
+ .defaultValue(Boolean.FALSE.toString())
+ .build();
+ public static final PropertyDescriptor POLL_ON_INGESTION_STATUS = new PropertyDescriptor
+ .Builder().name("Poll On Ingestion Status")
+ .displayName("Whether To Poll On Ingestion Status")
+ .description("This property determines whether we want to poll on ingestion status after an ingestion to ADX is completed")
Review Comment:
```suggestion
.description("Determines whether to poll on ingestion status after an ingestion to Azure Data Explorer is completed")
```
##########
nifi-nar-bundles/nifi-azure-bundle/nifi-azure-processors/src/main/java/org/apache/nifi/processors/azure/data/explorer/PutAzureDataExplorer.java:
##########
@@ -0,0 +1,291 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.azure.data.explorer;
+
+import org.apache.commons.lang3.StringUtils;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.expression.ExpressionLanguageScope;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.services.azure.data.explorer.KustoIngestDataFormat;
+import org.apache.nifi.services.azure.data.explorer.KustoIngestQueryResponse;
+import org.apache.nifi.services.azure.data.explorer.KustoIngestService;
+import org.apache.nifi.services.azure.data.explorer.KustoIngestionRequest;
+import org.apache.nifi.services.azure.data.explorer.KustoIngestionResult;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.net.URISyntaxException;
+import java.time.Duration;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+import java.util.concurrent.TimeUnit;
+
+@Tags({"Azure", "Kusto", "ADX", "Explorer", "Data"})
+@CapabilityDescription("The PutAzureDataExplorer acts as a ADX sink connector which sends flowFiles using the ADX-Service to the provided Azure Data" +
+ "Explorer Ingest Endpoint. The data can be sent through queued ingestion or streaming ingestion to the Azure Data Explorer cluster.")
+@InputRequirement(InputRequirement.Requirement.INPUT_REQUIRED)
+public class PutAzureDataExplorer extends AbstractProcessor {
+
+ public static final String FETCH_TABLE_COMMAND = "%s | count";
+ public static final String STREAMING_POLICY_SHOW_COMMAND = ".show %s %s policy streamingingestion";
+ public static final String DATABASE = "database";
+ public static final AllowableValue IGNORE_FIRST_RECORD_YES = new AllowableValue(
+ "YES", "YES",
+ "Ignore first record during ingestion");
+
+ public static final AllowableValue IGNORE_FIRST_RECORD_NO = new AllowableValue(
+ "NO", "NO",
+ "Do not ignore first record during ingestion");
+
+ public static final PropertyDescriptor DATABASE_NAME = new PropertyDescriptor.Builder()
+ .name("Database Name")
+ .displayName("Database Name")
+ .description("Azure Data Explorer Database Name for querying")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+ .build();
+ public static final PropertyDescriptor TABLE_NAME = new PropertyDescriptor.Builder()
+ .name("Table Name")
+ .displayName("Table Name")
+ .description("Azure Data Explorer Table Name for ingesting data")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+ .build();
+ public static final PropertyDescriptor MAPPING_NAME = new PropertyDescriptor
+ .Builder().name("Ingest Mapping name")
+ .displayName("Ingest Mapping name")
+ .description("The name of the mapping responsible for storing the data in the appropriate columns.")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor IS_STREAMING_ENABLED = new PropertyDescriptor
+ .Builder().name("Streaming Enabled")
+ .displayName("Streaming Enabled")
+ .description("This property determines whether we want to stream data to ADX.")
+ .required(false)
+ .allowableValues(Boolean.TRUE.toString(), Boolean.FALSE.toString())
+ .addValidator(StandardValidators.BOOLEAN_VALIDATOR)
+ .defaultValue(Boolean.FALSE.toString())
+ .build();
+ public static final PropertyDescriptor POLL_ON_INGESTION_STATUS = new PropertyDescriptor
+ .Builder().name("Poll On Ingestion Status")
+ .displayName("Whether To Poll On Ingestion Status")
+ .description("This property determines whether we want to poll on ingestion status after an ingestion to ADX is completed")
+ .required(false)
+ .allowableValues(Boolean.TRUE.toString(), Boolean.FALSE.toString())
+ .addValidator(StandardValidators.BOOLEAN_VALIDATOR)
+ .defaultValue(Boolean.FALSE.toString())
+ .build();
+ public static final PropertyDescriptor ADX_SERVICE = new PropertyDescriptor
+ .Builder().name("Kusto Ingest Service")
+ .displayName("Kusto Ingest Service")
+ .description("Azure Data Explorer Kusto Ingest Service")
+ .required(true)
+ .identifiesControllerService(KustoIngestService.class)
+ .build();
+ public static final PropertyDescriptor DATA_FORMAT = new PropertyDescriptor.Builder()
+ .name("Data Format")
+ .displayName("Data Format")
+ .description("The format of the data that is sent to Azure Data Explorer.")
+ .required(true)
+ .allowableValues(KustoIngestDataFormat.values())
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor IGNORE_FIRST_RECORD = new PropertyDescriptor.Builder()
+ .name("Ingestion Ignore First Record")
+ .displayName("Ingestion Ignore First Record")
+ .description("Defines whether ignore first record while ingestion.")
+ .required(false)
+ .allowableValues(IGNORE_FIRST_RECORD_YES, IGNORE_FIRST_RECORD_NO)
+ .defaultValue(IGNORE_FIRST_RECORD_NO.getValue())
+ .build();
+ public static final PropertyDescriptor INGESTION_STATUS_POLLING_TIMEOUT = new PropertyDescriptor.Builder()
+ .name("Ingest Status Polling Timeout")
+ .displayName("Ingest Status Polling Timeout")
+ .description("Defines the value of timeout for polling on ingestion status")
+ .required(false)
+ .dependsOn(POLL_ON_INGESTION_STATUS, Boolean.TRUE.toString())
+ .addValidator(StandardValidators.TIME_PERIOD_VALIDATOR)
+ .defaultValue("10 m")
+ .build();
+ public static final PropertyDescriptor INGESTION_STATUS_POLLING_INTERVAL = new PropertyDescriptor.Builder()
+ .name("Ingest Status Polling Interval")
+ .displayName("Ingest Status Polling Interval")
+ .description("Defines the value of timeout for polling on ingestion status in seconds.")
+ .required(false)
+ .dependsOn(POLL_ON_INGESTION_STATUS, Boolean.TRUE.toString())
+ .addValidator(StandardValidators.TIME_PERIOD_VALIDATOR)
+ .defaultValue("5 s")
+ .build();
+ public static final Relationship SUCCESS = new Relationship.Builder()
+ .name("Success")
+ .description("Relationship For Success")
+ .build();
+ public static final Relationship FAILURE = new Relationship.Builder()
+ .name("Failure")
+ .description("Relationship For Failure")
+ .build();
+
+ public static final PropertyDescriptor ROUTE_PARTIALLY_SUCCESSFUL_INGESTION = new PropertyDescriptor.Builder()
+ .name("Route Partially Successful Ingestion Records")
+ .displayName("Route Partially Successful Ingestion Records")
+ .description("Defines where to route partially successful ingestion records.")
+ .required(false)
+ .allowableValues(SUCCESS.getName(), FAILURE.getName())
+ .defaultValue(FAILURE.getName())
+ .build();
+ private static final List<PropertyDescriptor> descriptors;
+ private static final Set<Relationship> relationships;
+ private transient KustoIngestService service;
+ private boolean streamingEnabled;
+ private boolean pollOnIngestionStatus;
+
+ static {
+ final List<PropertyDescriptor> descriptorList = new ArrayList<>();
+ descriptorList.add(ADX_SERVICE);
+ descriptorList.add(DATABASE_NAME);
+ descriptorList.add(TABLE_NAME);
+ descriptorList.add(MAPPING_NAME);
+ descriptorList.add(DATA_FORMAT);
+ descriptorList.add(IGNORE_FIRST_RECORD);
+ descriptorList.add(IS_STREAMING_ENABLED);
+ descriptorList.add(POLL_ON_INGESTION_STATUS);
+ descriptorList.add(ROUTE_PARTIALLY_SUCCESSFUL_INGESTION);
+ descriptorList.add(INGESTION_STATUS_POLLING_TIMEOUT);
+ descriptorList.add(INGESTION_STATUS_POLLING_INTERVAL);
+ descriptors = Collections.unmodifiableList(descriptorList);
Review Comment:
This should be changed to use List.of()
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] NIFI-11958 Azure Data Explorer as sink [nifi]
Posted by "tanmaya-panda1 (via GitHub)" <gi...@apache.org>.
tanmaya-panda1 commented on PR #7624:
URL: https://github.com/apache/nifi/pull/7624#issuecomment-1748661908
@exceptionfactory , Hope you are doing well. Can you please take a look the PR again.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] NIFI-11958 Azure Data Explorer as sink [nifi]
Posted by "exceptionfactory (via GitHub)" <gi...@apache.org>.
exceptionfactory commented on code in PR #7624:
URL: https://github.com/apache/nifi/pull/7624#discussion_r1428874499
##########
nifi-nar-bundles/nifi-azure-bundle/nifi-azure-processors/src/main/java/org/apache/nifi/services/azure/data/explorer/StandardKustoIngestService.java:
##########
@@ -0,0 +1,409 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.services.azure.data.explorer;
+
+import com.microsoft.azure.kusto.data.Client;
+import com.microsoft.azure.kusto.data.ClientFactory;
+import com.microsoft.azure.kusto.data.KustoResultColumn;
+import com.microsoft.azure.kusto.data.KustoResultSetTable;
+import com.microsoft.azure.kusto.data.auth.ConnectionStringBuilder;
+import com.microsoft.azure.kusto.data.exceptions.DataClientException;
+import com.microsoft.azure.kusto.data.exceptions.DataServiceException;
+import com.microsoft.azure.kusto.ingest.IngestClientFactory;
+import com.microsoft.azure.kusto.ingest.IngestionMapping;
+import com.microsoft.azure.kusto.ingest.IngestionProperties;
+import com.microsoft.azure.kusto.ingest.ManagedStreamingIngestClient;
+import com.microsoft.azure.kusto.ingest.QueuedIngestClient;
+import com.microsoft.azure.kusto.ingest.exceptions.IngestionClientException;
+import com.microsoft.azure.kusto.ingest.exceptions.IngestionServiceException;
+import com.microsoft.azure.kusto.ingest.result.IngestionResult;
+import com.microsoft.azure.kusto.ingest.result.IngestionStatus;
+import com.microsoft.azure.kusto.ingest.result.OperationStatus;
+import com.microsoft.azure.kusto.ingest.source.StreamSourceInfo;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnEnabled;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.controller.AbstractControllerService;
+import org.apache.nifi.controller.ConfigurationContext;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+
+import java.io.IOException;
+import java.net.URISyntaxException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+
+@Tags({"Azure", "ADX", "Kusto", "ingest", "azure"})
+@CapabilityDescription("Sends batches of flowfile content or stream flowfile content to an Azure ADX cluster.")
+public class StandardKustoIngestService extends AbstractControllerService implements KustoIngestService {
+
+ public static final PropertyDescriptor AUTHENTICATION_STRATEGY = new PropertyDescriptor
+ .Builder().name("Authentication Strategy")
+ .displayName("Authentication Strategy")
+ .description("Authentication method for access to Azure Data Explorer")
+ .required(true)
+ .defaultValue(KustoAuthenticationStrategy.MANAGED_IDENTITY.getValue())
+ .allowableValues(KustoAuthenticationStrategy.class)
+ .build();
+
+ public static final PropertyDescriptor APPLICATION_CLIENT_ID = new PropertyDescriptor
+ .Builder().name("Application Client ID")
+ .displayName("Application Client ID")
+ .description("Azure Data Explorer Application Client Identifier for Authentication")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+
+ public static final PropertyDescriptor APPLICATION_KEY = new PropertyDescriptor
+ .Builder().name("Application Key")
+ .displayName("Application Key")
+ .description("Azure Data Explorer Application Key for Authentication")
+ .required(true)
+ .sensitive(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .dependsOn(AUTHENTICATION_STRATEGY, KustoAuthenticationStrategy.APPLICATION_CREDENTIALS.getValue())
+ .build();
+
+ public static final PropertyDescriptor APPLICATION_TENANT_ID = new PropertyDescriptor.Builder()
+ .name("Application Tenant ID")
+ .displayName("Application Tenant ID")
+ .description("Azure Data Explorer Application Tenant Identifier for Authentication")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .dependsOn(AUTHENTICATION_STRATEGY, KustoAuthenticationStrategy.APPLICATION_CREDENTIALS.getValue())
+ .build();
+
+ public static final PropertyDescriptor CLUSTER_URI = new PropertyDescriptor
+ .Builder().name("Cluster URI")
+ .displayName("Cluster URI")
+ .description("Azure Data Explorer Cluster URI")
+ .required(true)
+ .addValidator(StandardValidators.URL_VALIDATOR)
+ .build();
+
+ private static final List<PropertyDescriptor> PROPERTY_DESCRIPTORS = Collections.unmodifiableList(Arrays.asList(
+ AUTHENTICATION_STRATEGY,
+ APPLICATION_CLIENT_ID,
+ APPLICATION_KEY,
+ APPLICATION_TENANT_ID,
+ CLUSTER_URI));
+
+ public static final Pair<String, String> NIFI_SINK = Pair.of("processor", StandardKustoIngestService.class.getSimpleName());
+
+ private volatile QueuedIngestClient queuedIngestClient;
+
+ private volatile ManagedStreamingIngestClient managedStreamingIngestClient;
+
+ private volatile Client executionClient;
+
+ @Override
+ public List<PropertyDescriptor> getSupportedPropertyDescriptors() {
+ return PROPERTY_DESCRIPTORS;
+ }
+
+ /**
+ * @param context the configuration context
+ */
+ @OnEnabled
+ public void onEnabled(final ConfigurationContext context) throws ProcessException, URISyntaxException {
+
+ final String applicationClientId = context.getProperty(APPLICATION_CLIENT_ID).getValue();
+ final String applicationKey = context.getProperty(APPLICATION_KEY).getValue();
+ final String applicationTenantId = context.getProperty(APPLICATION_TENANT_ID).getValue();
+ final String clusterUri = context.getProperty(CLUSTER_URI).getValue();
+ final KustoAuthenticationStrategy kustoAuthenticationStrategy = KustoAuthenticationStrategy.valueOf(context.getProperty(AUTHENTICATION_STRATEGY).getValue());
+
+ if (this.queuedIngestClient == null) {
+ this.queuedIngestClient = createKustoQueuedIngestClient(clusterUri, applicationClientId, applicationKey, applicationTenantId, kustoAuthenticationStrategy);
+ }
+
+ if (this.managedStreamingIngestClient == null) {
+ this.managedStreamingIngestClient = createKustoStreamingIngestClient(clusterUri, applicationClientId, applicationKey, applicationTenantId, kustoAuthenticationStrategy);
+ }
+
+ if (this.executionClient == null) {
+ this.executionClient = createKustoExecutionClient(clusterUri, applicationClientId, applicationKey, applicationTenantId, kustoAuthenticationStrategy);
+ }
Review Comment:
The conditionals can be removed, since the clients should always be reset when enabling the Controller Service.
```suggestion
this.queuedIngestClient = createKustoQueuedIngestClient(clusterUri, applicationClientId, applicationKey, applicationTenantId, kustoAuthenticationStrategy);
this.managedStreamingIngestClient = createKustoStreamingIngestClient(clusterUri, applicationClientId, applicationKey, applicationTenantId, kustoAuthenticationStrategy);
this.executionClient = createKustoExecutionClient(clusterUri, applicationClientId, applicationKey, applicationTenantId, kustoAuthenticationStrategy);
```
##########
nifi-nar-bundles/nifi-azure-bundle/nifi-azure-processors/src/main/java/org/apache/nifi/processors/azure/data/explorer/PutAzureDataExplorer.java:
##########
@@ -0,0 +1,282 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.azure.data.explorer;
+
+import org.apache.commons.lang3.StringUtils;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.expression.ExpressionLanguageScope;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.services.azure.data.explorer.KustoIngestDataFormat;
+import org.apache.nifi.services.azure.data.explorer.KustoIngestQueryResponse;
+import org.apache.nifi.services.azure.data.explorer.KustoIngestService;
+import org.apache.nifi.services.azure.data.explorer.KustoIngestionRequest;
+import org.apache.nifi.services.azure.data.explorer.KustoIngestionResult;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.time.Duration;
+import java.util.List;
+import java.util.Set;
+import java.util.concurrent.TimeUnit;
+
+@Tags({"Azure", "Kusto", "ADX", "Explorer", "Data"})
+@CapabilityDescription("The PutAzureDataExplorer acts as a ADX sink connector which sends flowFiles using the ADX-Service to the provided Azure Data" +
+ "Explorer Ingest Endpoint. The data can be sent through queued ingestion or streaming ingestion to the Azure Data Explorer cluster.")
+@InputRequirement(InputRequirement.Requirement.INPUT_REQUIRED)
+public class PutAzureDataExplorer extends AbstractProcessor {
+
+ public static final String FETCH_TABLE_COMMAND = "%s | count";
+ public static final String STREAMING_POLICY_SHOW_COMMAND = ".show %s %s policy streamingingestion";
+ public static final String DATABASE = "database";
+ public static final AllowableValue IGNORE_FIRST_RECORD_YES = new AllowableValue(
+ "YES", "YES",
+ "Ignore first record during ingestion");
+
+ public static final AllowableValue IGNORE_FIRST_RECORD_NO = new AllowableValue(
+ "NO", "NO",
+ "Do not ignore first record during ingestion");
+
+ public static final PropertyDescriptor DATABASE_NAME = new PropertyDescriptor.Builder()
+ .name("Database Name")
+ .displayName("Database Name")
+ .description("Azure Data Explorer Database Name for querying")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+ .build();
+ public static final PropertyDescriptor TABLE_NAME = new PropertyDescriptor.Builder()
+ .name("Table Name")
+ .displayName("Table Name")
+ .description("Azure Data Explorer Table Name for ingesting data")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+ .build();
+ public static final PropertyDescriptor MAPPING_NAME = new PropertyDescriptor
+ .Builder().name("Ingest Mapping name")
+ .displayName("Ingest Mapping name")
+ .description("The name of the mapping responsible for storing the data in the appropriate columns.")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor IS_STREAMING_ENABLED = new PropertyDescriptor
+ .Builder().name("Streaming Enabled")
+ .displayName("Streaming Enabled")
+ .description("This property determines whether we want to stream data to ADX.")
+ .required(false)
+ .allowableValues(Boolean.TRUE.toString(), Boolean.FALSE.toString())
+ .addValidator(StandardValidators.BOOLEAN_VALIDATOR)
+ .defaultValue(Boolean.FALSE.toString())
+ .build();
+ public static final PropertyDescriptor POLL_ON_INGESTION_STATUS = new PropertyDescriptor
+ .Builder().name("Poll for Ingestion Status")
+ .displayName("Poll for Ingestion Status")
+ .description("Determines whether to poll on ingestion status after an ingestion to Azure Data Explorer is completed")
+ .required(false)
+ .allowableValues(Boolean.TRUE.toString(), Boolean.FALSE.toString())
+ .addValidator(StandardValidators.BOOLEAN_VALIDATOR)
+ .defaultValue(Boolean.FALSE.toString())
+ .build();
+ public static final PropertyDescriptor ADX_SERVICE = new PropertyDescriptor
+ .Builder().name("Kusto Ingest Service")
+ .displayName("Kusto Ingest Service")
+ .description("Azure Data Explorer Kusto Ingest Service")
+ .required(true)
+ .identifiesControllerService(KustoIngestService.class)
+ .build();
+ public static final PropertyDescriptor DATA_FORMAT = new PropertyDescriptor.Builder()
+ .name("Data Format")
Review Comment:
What do you think about supporting Expression Language for the Data Format property? This would allow the property value to be resolved based on a FlowFile attribute, so that one Processor could handle different types of data. That would require call `evaluateAttributeExpressions(flowFile)` in the `onTrigger()` method.
##########
nifi-nar-bundles/nifi-azure-bundle/nifi-azure-processors/src/main/java/org/apache/nifi/processors/azure/data/explorer/PutAzureDataExplorer.java:
##########
@@ -0,0 +1,282 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.azure.data.explorer;
+
+import org.apache.commons.lang3.StringUtils;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.expression.ExpressionLanguageScope;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.services.azure.data.explorer.KustoIngestDataFormat;
+import org.apache.nifi.services.azure.data.explorer.KustoIngestQueryResponse;
+import org.apache.nifi.services.azure.data.explorer.KustoIngestService;
+import org.apache.nifi.services.azure.data.explorer.KustoIngestionRequest;
+import org.apache.nifi.services.azure.data.explorer.KustoIngestionResult;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.time.Duration;
+import java.util.List;
+import java.util.Set;
+import java.util.concurrent.TimeUnit;
+
+@Tags({"Azure", "Kusto", "ADX", "Explorer", "Data"})
+@CapabilityDescription("The PutAzureDataExplorer acts as a ADX sink connector which sends flowFiles using the ADX-Service to the provided Azure Data" +
+ "Explorer Ingest Endpoint. The data can be sent through queued ingestion or streaming ingestion to the Azure Data Explorer cluster.")
+@InputRequirement(InputRequirement.Requirement.INPUT_REQUIRED)
+public class PutAzureDataExplorer extends AbstractProcessor {
+
+ public static final String FETCH_TABLE_COMMAND = "%s | count";
+ public static final String STREAMING_POLICY_SHOW_COMMAND = ".show %s %s policy streamingingestion";
+ public static final String DATABASE = "database";
+ public static final AllowableValue IGNORE_FIRST_RECORD_YES = new AllowableValue(
+ "YES", "YES",
+ "Ignore first record during ingestion");
+
+ public static final AllowableValue IGNORE_FIRST_RECORD_NO = new AllowableValue(
+ "NO", "NO",
+ "Do not ignore first record during ingestion");
+
+ public static final PropertyDescriptor DATABASE_NAME = new PropertyDescriptor.Builder()
+ .name("Database Name")
+ .displayName("Database Name")
+ .description("Azure Data Explorer Database Name for querying")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+ .build();
+ public static final PropertyDescriptor TABLE_NAME = new PropertyDescriptor.Builder()
+ .name("Table Name")
+ .displayName("Table Name")
+ .description("Azure Data Explorer Table Name for ingesting data")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+ .build();
+ public static final PropertyDescriptor MAPPING_NAME = new PropertyDescriptor
+ .Builder().name("Ingest Mapping name")
+ .displayName("Ingest Mapping name")
+ .description("The name of the mapping responsible for storing the data in the appropriate columns.")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor IS_STREAMING_ENABLED = new PropertyDescriptor
+ .Builder().name("Streaming Enabled")
+ .displayName("Streaming Enabled")
+ .description("This property determines whether we want to stream data to ADX.")
+ .required(false)
+ .allowableValues(Boolean.TRUE.toString(), Boolean.FALSE.toString())
+ .addValidator(StandardValidators.BOOLEAN_VALIDATOR)
+ .defaultValue(Boolean.FALSE.toString())
+ .build();
+ public static final PropertyDescriptor POLL_ON_INGESTION_STATUS = new PropertyDescriptor
+ .Builder().name("Poll for Ingestion Status")
+ .displayName("Poll for Ingestion Status")
+ .description("Determines whether to poll on ingestion status after an ingestion to Azure Data Explorer is completed")
+ .required(false)
+ .allowableValues(Boolean.TRUE.toString(), Boolean.FALSE.toString())
+ .addValidator(StandardValidators.BOOLEAN_VALIDATOR)
+ .defaultValue(Boolean.FALSE.toString())
+ .build();
+ public static final PropertyDescriptor ADX_SERVICE = new PropertyDescriptor
+ .Builder().name("Kusto Ingest Service")
+ .displayName("Kusto Ingest Service")
+ .description("Azure Data Explorer Kusto Ingest Service")
+ .required(true)
+ .identifiesControllerService(KustoIngestService.class)
+ .build();
+ public static final PropertyDescriptor DATA_FORMAT = new PropertyDescriptor.Builder()
+ .name("Data Format")
+ .displayName("Data Format")
+ .description("The format of the data that is sent to Azure Data Explorer.")
+ .required(true)
+ .allowableValues(KustoIngestDataFormat.values())
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor IGNORE_FIRST_RECORD = new PropertyDescriptor.Builder()
+ .name("Ingestion Ignore First Record")
+ .displayName("Ingestion Ignore First Record")
+ .description("Defines whether ignore first record while ingestion.")
+ .required(false)
+ .allowableValues(IGNORE_FIRST_RECORD_YES, IGNORE_FIRST_RECORD_NO)
+ .defaultValue(IGNORE_FIRST_RECORD_NO.getValue())
+ .build();
+ public static final PropertyDescriptor INGESTION_STATUS_POLLING_TIMEOUT = new PropertyDescriptor.Builder()
+ .name("Ingest Status Polling Timeout")
+ .displayName("Ingest Status Polling Timeout")
+ .description("Defines the value of timeout for polling on ingestion status")
+ .required(false)
+ .dependsOn(POLL_ON_INGESTION_STATUS, Boolean.TRUE.toString())
+ .addValidator(StandardValidators.TIME_PERIOD_VALIDATOR)
+ .defaultValue("10 m")
+ .build();
+ public static final PropertyDescriptor INGESTION_STATUS_POLLING_INTERVAL = new PropertyDescriptor.Builder()
+ .name("Ingest Status Polling Interval")
+ .displayName("Ingest Status Polling Interval")
+ .description("Defines the value of timeout for polling on ingestion status in seconds.")
+ .required(false)
+ .dependsOn(POLL_ON_INGESTION_STATUS, Boolean.TRUE.toString())
+ .addValidator(StandardValidators.TIME_PERIOD_VALIDATOR)
+ .defaultValue("5 s")
+ .build();
+ public static final Relationship SUCCESS = new Relationship.Builder()
+ .name("success")
+ .description("Relationship For Success")
+ .build();
+ public static final Relationship FAILURE = new Relationship.Builder()
+ .name("failure")
+ .description("Relationship For Failure")
+ .build();
+
+ public static final PropertyDescriptor ROUTE_PARTIALLY_SUCCESSFUL_INGESTION = new PropertyDescriptor.Builder()
+ .name("Route Partially Successful Ingestion Records")
+ .displayName("Route Partially Successful Ingestion Records")
+ .description("Defines where to route partially successful ingestion records.")
+ .required(false)
+ .allowableValues(SUCCESS.getName(), FAILURE.getName())
+ .defaultValue(FAILURE.getName())
+ .build();
+ private static final List<PropertyDescriptor> descriptors;
+ private static final Set<Relationship> relationships;
+ private transient KustoIngestService service;
+
+ static {
+ descriptors = List.of(
+ ADX_SERVICE,
+ DATABASE_NAME,
+ TABLE_NAME,
+ MAPPING_NAME,
+ DATA_FORMAT,
+ IGNORE_FIRST_RECORD,
+ IS_STREAMING_ENABLED,
+ POLL_ON_INGESTION_STATUS,
+ ROUTE_PARTIALLY_SUCCESSFUL_INGESTION,
+ INGESTION_STATUS_POLLING_TIMEOUT,
+ INGESTION_STATUS_POLLING_INTERVAL
+ );
+
+ relationships = Set.of(SUCCESS, FAILURE);
+ }
+
+ @Override
+ public Set<Relationship> getRelationships() {
+ return this.relationships;
+ }
+
+ @Override
+ public final List<PropertyDescriptor> getSupportedPropertyDescriptors() {
+ return descriptors;
+ }
+
+ @OnScheduled
+ public void onScheduled(final ProcessContext context) {
+ service = context.getProperty(ADX_SERVICE).asControllerService(KustoIngestService.class);
+ if (!checkIfIngestorRoleExistInADX(context.getProperty(DATABASE_NAME).getValue(), context.getProperty(TABLE_NAME).getValue())) {
+ throw new ProcessException(
+ String.format("User does not have ingestor privileges for database %s and table %s", context.getProperty(DATABASE_NAME).getValue(), context.getProperty(TABLE_NAME).getValue())
+ );
+ }
+ Boolean streamingEnabled = context.getProperty(IS_STREAMING_ENABLED).evaluateAttributeExpressions().asBoolean();
+ if (streamingEnabled && !checkIfStreamingPolicyIsEnabledInADX(context.getProperty(DATABASE_NAME).getValue(), context.getProperty(DATABASE_NAME).getValue())) {
+ throw new ProcessException(String.format("Streaming policy is not enabled in database %s",context.getProperty(DATABASE_NAME).getValue()));
+ }
+ }
+
+ @Override
+ public void onTrigger(final ProcessContext context, final ProcessSession session) throws ProcessException {
+ FlowFile flowFile = session.get();
+ if (flowFile == null) {
+ return;
+ }
+
+ Relationship transferRelationship = Relationship.SELF;
Review Comment:
This should be defaulted to either `SUCCESS` or `FAILURE`, then the value can be used at the end of the method, instead of the conditional.
##########
nifi-nar-bundles/nifi-azure-bundle/nifi-azure-processors/src/main/java/org/apache/nifi/processors/azure/data/explorer/PutAzureDataExplorer.java:
##########
@@ -0,0 +1,282 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.azure.data.explorer;
+
+import org.apache.commons.lang3.StringUtils;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.expression.ExpressionLanguageScope;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.services.azure.data.explorer.KustoIngestDataFormat;
+import org.apache.nifi.services.azure.data.explorer.KustoIngestQueryResponse;
+import org.apache.nifi.services.azure.data.explorer.KustoIngestService;
+import org.apache.nifi.services.azure.data.explorer.KustoIngestionRequest;
+import org.apache.nifi.services.azure.data.explorer.KustoIngestionResult;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.time.Duration;
+import java.util.List;
+import java.util.Set;
+import java.util.concurrent.TimeUnit;
+
+@Tags({"Azure", "Kusto", "ADX", "Explorer", "Data"})
+@CapabilityDescription("The PutAzureDataExplorer acts as a ADX sink connector which sends flowFiles using the ADX-Service to the provided Azure Data" +
+ "Explorer Ingest Endpoint. The data can be sent through queued ingestion or streaming ingestion to the Azure Data Explorer cluster.")
+@InputRequirement(InputRequirement.Requirement.INPUT_REQUIRED)
+public class PutAzureDataExplorer extends AbstractProcessor {
+
+ public static final String FETCH_TABLE_COMMAND = "%s | count";
+ public static final String STREAMING_POLICY_SHOW_COMMAND = ".show %s %s policy streamingingestion";
+ public static final String DATABASE = "database";
+ public static final AllowableValue IGNORE_FIRST_RECORD_YES = new AllowableValue(
+ "YES", "YES",
+ "Ignore first record during ingestion");
+
+ public static final AllowableValue IGNORE_FIRST_RECORD_NO = new AllowableValue(
+ "NO", "NO",
+ "Do not ignore first record during ingestion");
+
+ public static final PropertyDescriptor DATABASE_NAME = new PropertyDescriptor.Builder()
+ .name("Database Name")
+ .displayName("Database Name")
+ .description("Azure Data Explorer Database Name for querying")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+ .build();
+ public static final PropertyDescriptor TABLE_NAME = new PropertyDescriptor.Builder()
+ .name("Table Name")
+ .displayName("Table Name")
+ .description("Azure Data Explorer Table Name for ingesting data")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+ .build();
+ public static final PropertyDescriptor MAPPING_NAME = new PropertyDescriptor
+ .Builder().name("Ingest Mapping name")
+ .displayName("Ingest Mapping name")
+ .description("The name of the mapping responsible for storing the data in the appropriate columns.")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor IS_STREAMING_ENABLED = new PropertyDescriptor
+ .Builder().name("Streaming Enabled")
+ .displayName("Streaming Enabled")
+ .description("This property determines whether we want to stream data to ADX.")
+ .required(false)
+ .allowableValues(Boolean.TRUE.toString(), Boolean.FALSE.toString())
+ .addValidator(StandardValidators.BOOLEAN_VALIDATOR)
+ .defaultValue(Boolean.FALSE.toString())
+ .build();
+ public static final PropertyDescriptor POLL_ON_INGESTION_STATUS = new PropertyDescriptor
+ .Builder().name("Poll for Ingestion Status")
+ .displayName("Poll for Ingestion Status")
+ .description("Determines whether to poll on ingestion status after an ingestion to Azure Data Explorer is completed")
+ .required(false)
+ .allowableValues(Boolean.TRUE.toString(), Boolean.FALSE.toString())
+ .addValidator(StandardValidators.BOOLEAN_VALIDATOR)
+ .defaultValue(Boolean.FALSE.toString())
+ .build();
+ public static final PropertyDescriptor ADX_SERVICE = new PropertyDescriptor
+ .Builder().name("Kusto Ingest Service")
+ .displayName("Kusto Ingest Service")
+ .description("Azure Data Explorer Kusto Ingest Service")
+ .required(true)
+ .identifiesControllerService(KustoIngestService.class)
+ .build();
+ public static final PropertyDescriptor DATA_FORMAT = new PropertyDescriptor.Builder()
+ .name("Data Format")
+ .displayName("Data Format")
+ .description("The format of the data that is sent to Azure Data Explorer.")
+ .required(true)
+ .allowableValues(KustoIngestDataFormat.values())
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+ public static final PropertyDescriptor IGNORE_FIRST_RECORD = new PropertyDescriptor.Builder()
+ .name("Ingestion Ignore First Record")
+ .displayName("Ingestion Ignore First Record")
+ .description("Defines whether ignore first record while ingestion.")
+ .required(false)
+ .allowableValues(IGNORE_FIRST_RECORD_YES, IGNORE_FIRST_RECORD_NO)
+ .defaultValue(IGNORE_FIRST_RECORD_NO.getValue())
+ .build();
+ public static final PropertyDescriptor INGESTION_STATUS_POLLING_TIMEOUT = new PropertyDescriptor.Builder()
+ .name("Ingest Status Polling Timeout")
+ .displayName("Ingest Status Polling Timeout")
+ .description("Defines the value of timeout for polling on ingestion status")
+ .required(false)
+ .dependsOn(POLL_ON_INGESTION_STATUS, Boolean.TRUE.toString())
+ .addValidator(StandardValidators.TIME_PERIOD_VALIDATOR)
+ .defaultValue("10 m")
+ .build();
+ public static final PropertyDescriptor INGESTION_STATUS_POLLING_INTERVAL = new PropertyDescriptor.Builder()
+ .name("Ingest Status Polling Interval")
+ .displayName("Ingest Status Polling Interval")
+ .description("Defines the value of timeout for polling on ingestion status in seconds.")
+ .required(false)
+ .dependsOn(POLL_ON_INGESTION_STATUS, Boolean.TRUE.toString())
+ .addValidator(StandardValidators.TIME_PERIOD_VALIDATOR)
+ .defaultValue("5 s")
+ .build();
+ public static final Relationship SUCCESS = new Relationship.Builder()
+ .name("success")
+ .description("Relationship For Success")
+ .build();
+ public static final Relationship FAILURE = new Relationship.Builder()
+ .name("failure")
+ .description("Relationship For Failure")
+ .build();
+
+ public static final PropertyDescriptor ROUTE_PARTIALLY_SUCCESSFUL_INGESTION = new PropertyDescriptor.Builder()
+ .name("Route Partially Successful Ingestion Records")
+ .displayName("Route Partially Successful Ingestion Records")
+ .description("Defines where to route partially successful ingestion records.")
+ .required(false)
+ .allowableValues(SUCCESS.getName(), FAILURE.getName())
+ .defaultValue(FAILURE.getName())
+ .build();
+ private static final List<PropertyDescriptor> descriptors;
+ private static final Set<Relationship> relationships;
+ private transient KustoIngestService service;
+
+ static {
+ descriptors = List.of(
+ ADX_SERVICE,
+ DATABASE_NAME,
+ TABLE_NAME,
+ MAPPING_NAME,
+ DATA_FORMAT,
+ IGNORE_FIRST_RECORD,
+ IS_STREAMING_ENABLED,
+ POLL_ON_INGESTION_STATUS,
+ ROUTE_PARTIALLY_SUCCESSFUL_INGESTION,
+ INGESTION_STATUS_POLLING_TIMEOUT,
+ INGESTION_STATUS_POLLING_INTERVAL
+ );
+
+ relationships = Set.of(SUCCESS, FAILURE);
+ }
+
+ @Override
+ public Set<Relationship> getRelationships() {
+ return this.relationships;
+ }
+
+ @Override
+ public final List<PropertyDescriptor> getSupportedPropertyDescriptors() {
+ return descriptors;
+ }
+
+ @OnScheduled
+ public void onScheduled(final ProcessContext context) {
+ service = context.getProperty(ADX_SERVICE).asControllerService(KustoIngestService.class);
+ if (!checkIfIngestorRoleExistInADX(context.getProperty(DATABASE_NAME).getValue(), context.getProperty(TABLE_NAME).getValue())) {
+ throw new ProcessException(
+ String.format("User does not have ingestor privileges for database %s and table %s", context.getProperty(DATABASE_NAME).getValue(), context.getProperty(TABLE_NAME).getValue())
+ );
+ }
+ Boolean streamingEnabled = context.getProperty(IS_STREAMING_ENABLED).evaluateAttributeExpressions().asBoolean();
+ if (streamingEnabled && !checkIfStreamingPolicyIsEnabledInADX(context.getProperty(DATABASE_NAME).getValue(), context.getProperty(DATABASE_NAME).getValue())) {
+ throw new ProcessException(String.format("Streaming policy is not enabled in database %s",context.getProperty(DATABASE_NAME).getValue()));
+ }
+ }
+
+ @Override
+ public void onTrigger(final ProcessContext context, final ProcessSession session) throws ProcessException {
+ FlowFile flowFile = session.get();
+ if (flowFile == null) {
+ return;
+ }
+
+ Relationship transferRelationship = Relationship.SELF;
+
+ String databaseName = context.getProperty(DATABASE_NAME).getValue();
+ String tableName = context.getProperty(TABLE_NAME).getValue();
+ String dataFormat = context.getProperty(DATA_FORMAT).getValue();
+ String mappingName = context.getProperty(MAPPING_NAME).getValue();
+ String routePartiallySuccessfulIngestion = context.getProperty(ROUTE_PARTIALLY_SUCCESSFUL_INGESTION).getValue();
+ Duration ingestionStatusPollingTimeout = Duration.ofSeconds(context.getProperty(INGESTION_STATUS_POLLING_TIMEOUT).asTimePeriod(TimeUnit.SECONDS));
+ Duration ingestionStatusPollingInterval = Duration.ofSeconds(context.getProperty(INGESTION_STATUS_POLLING_INTERVAL).asTimePeriod(TimeUnit.SECONDS));
+ String ignoreFirstRecord = context.getProperty(IGNORE_FIRST_RECORD).getValue();
+ Boolean streamingEnabled = context.getProperty(IS_STREAMING_ENABLED).evaluateAttributeExpressions().asBoolean();
+ Boolean pollOnIngestionStatus = context.getProperty(POLL_ON_INGESTION_STATUS).evaluateAttributeExpressions().asBoolean();
+
+ // ingestion into ADX
+ try (final InputStream inputStream = session.read(flowFile)) {
+ getLogger().debug("Ingesting with: dataFormat - {} | ingestionMappingName - {} | databaseName - {} | tableName - {} | " +
+ "pollOnIngestionStatus - {} | ingestionStatusPollingTimeout - {} | ingestionStatusPollingInterval - {} | " +
+ "routePartiallySuccessfulIngestion - {}",
+ dataFormat, mappingName, databaseName, tableName, pollOnIngestionStatus,
+ ingestionStatusPollingTimeout, ingestionStatusPollingInterval, routePartiallySuccessfulIngestion);
+
+ KustoIngestionResult result = service.ingestData(
+ new KustoIngestionRequest(streamingEnabled, pollOnIngestionStatus, inputStream, databaseName,
+ tableName, dataFormat, mappingName, ignoreFirstRecord, ingestionStatusPollingTimeout, ingestionStatusPollingInterval)
+ );
+ getLogger().debug("Ingest Status Polling Enabled {} and Ingest Status Polling Enabled {} ", pollOnIngestionStatus, result);
+
+ if (result == KustoIngestionResult.SUCCEEDED) {
+ getLogger().info("Ingest Completed - {}", result.getStatus());
+ transferRelationship = SUCCESS;
+ } else if (result == KustoIngestionResult.FAILED) {
+ getLogger().error("Ingest Failed - {}", result.getStatus());
+ transferRelationship = FAILURE;
+ } else if (result == KustoIngestionResult.PARTIALLY_SUCCEEDED) {
+ getLogger().warn("Ingest Partially succeeded - {}", result.getStatus());
+ flowFile = session.putAttribute(flowFile, "ingestion_status", "partial_success");
+ if (StringUtils.equalsIgnoreCase(routePartiallySuccessfulIngestion, FAILURE.getName())) {
+ transferRelationship = FAILURE;
+ }else {
+ transferRelationship = SUCCESS;
+ }
+ }
+ } catch (IOException e) {
+ getLogger().error("Azure Data Explorer Ingest processing failed", e);
+ transferRelationship = FAILURE;
+ }
+
+ if (transferRelationship == SUCCESS) {
+ session.transfer(flowFile, SUCCESS);
+ } else {
+ session.transfer(flowFile, FAILURE);
+ }
Review Comment:
With changes above, this could be simplified:
```suggestion
session.transfer(flowFile, transferRelationship);
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] NIFI-11958 Azure Data Explorer as sink [nifi]
Posted by "tanmaya-panda1 (via GitHub)" <gi...@apache.org>.
tanmaya-panda1 commented on PR #7624:
URL: https://github.com/apache/nifi/pull/7624#issuecomment-1772420811
@exceptionfactory : Thanks for the detailed review. I have addressed most of your comments, please feel free to review and comment if anything is missing. Thanks once again.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] exceptionfactory commented on a diff in pull request #7624: NIFI-11958 Azure Data Explorer as sink
Posted by "exceptionfactory (via GitHub)" <gi...@apache.org>.
exceptionfactory commented on code in PR #7624:
URL: https://github.com/apache/nifi/pull/7624#discussion_r1330232784
##########
nifi-nar-bundles/nifi-azure-bundle/nifi-azure-processors/src/main/java/org/apache/nifi/processors/azure/data/explorer/IngestAzureDataExplorer.java:
##########
@@ -0,0 +1,401 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.azure.data.explorer;
+
+import com.microsoft.azure.kusto.data.KustoResultSetTable;
+import com.microsoft.azure.kusto.ingest.IngestionMapping;
+import com.microsoft.azure.kusto.ingest.IngestionProperties;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.nifi.annotation.behavior.ReadsAttribute;
+import org.apache.nifi.annotation.behavior.ReadsAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.expression.ExpressionLanguageScope;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.services.azure.data.explorer.KustoIngestService;
+import org.apache.nifi.services.azure.data.explorer.KustoIngestionRequest;
+import org.apache.nifi.services.azure.data.explorer.KustoIngestionResult;
+import org.apache.nifi.services.azure.data.explorer.KustoQueryResponse;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.net.URISyntaxException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+
+@Tags({"Azure", "Kusto", "ADX", "Explorer", "Data"})
+@CapabilityDescription("The Azure ADX Ingest Processor acts as a ADX sink connector which sends flowFiles using the ADX-Service to the provided Azure Data" +
+ "Explorer Ingest Endpoint. The data can be sent through queued ingestion or streaming ingestion to the Azure Data Explorer cluster.")
+@ReadsAttributes({
+ @ReadsAttribute(attribute = "DB_NAME", description = "Specifies the name of the ADX database where the data needs to be stored."),
+ @ReadsAttribute(attribute = "TABLE_NAME", description = "Specifies the name of the ADX table where the data needs to be stored."),
+ @ReadsAttribute(attribute = "MAPPING_NAME", description = "Specifies the name of the mapping responsible for storing the data in appropriate columns."),
+ @ReadsAttribute(attribute = "FLUSH_IMMEDIATE", description = "In case of queued ingestion, this property determines whether the data should be flushed immediately to the ingest endpoint."),
+ @ReadsAttribute(attribute = "DATA_FORMAT", description = "Specifies the format of data that is send to Azure Data Explorer."),
+ @ReadsAttribute(attribute = "IGNORE_FIRST_RECORD", description = "Specifies whether we want to ignore ingestion of first record. " +
+ "This is primarily applicable for csv files. Default is set to NO"),
+ @ReadsAttribute(attribute = "POLL_ON_INGESTION_STATUS", description = "Specifies whether we want to poll on ingestion result during ingestion into ADX." +
+ "In case of applications that need high throughput it is recommended to keep the default value as false. Default is set to false." +
+ "This property should be set to true during Queued Ingestion for near realtime micro-batches of data that require acknowledgement of ingestion status.")
+})
+public class IngestAzureDataExplorer extends AbstractProcessor {
Review Comment:
The name should be changed to `PutAzureDataExplorer` following the convention of other Processors.
##########
nifi-nar-bundles/nifi-azure-bundle/nifi-azure-processors/src/main/java/org/apache/nifi/services/azure/data/explorer/StandardKustoIngestService.java:
##########
@@ -0,0 +1,314 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.services.azure.data.explorer;
+
+import com.microsoft.azure.kusto.data.Client;
+import com.microsoft.azure.kusto.data.ClientFactory;
+import com.microsoft.azure.kusto.data.KustoResultSetTable;
+import com.microsoft.azure.kusto.data.auth.ConnectionStringBuilder;
+import com.microsoft.azure.kusto.data.exceptions.DataClientException;
+import com.microsoft.azure.kusto.data.exceptions.DataServiceException;
+import com.microsoft.azure.kusto.ingest.IngestClientFactory;
+import com.microsoft.azure.kusto.ingest.ManagedStreamingIngestClient;
+import com.microsoft.azure.kusto.ingest.QueuedIngestClient;
+import com.microsoft.azure.kusto.ingest.exceptions.IngestionClientException;
+import com.microsoft.azure.kusto.ingest.exceptions.IngestionServiceException;
+import com.microsoft.azure.kusto.ingest.result.IngestionResult;
+import com.microsoft.azure.kusto.ingest.result.IngestionStatus;
+import com.microsoft.azure.kusto.ingest.result.OperationStatus;
+import com.microsoft.azure.kusto.ingest.source.StreamSourceInfo;
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.nifi.annotation.behavior.ReadsAttribute;
+import org.apache.nifi.annotation.behavior.ReadsAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnEnabled;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.controller.AbstractControllerService;
+import org.apache.nifi.controller.ConfigurationContext;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.reporting.InitializationException;
+
+import java.io.IOException;
+import java.net.URISyntaxException;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.List;
+import java.util.Objects;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+
+@Tags({"Azure", "ADX", "Kusto", "ingest", "azure"})
+@CapabilityDescription("Sends batches of flowfile content or stream flowfile content to an Azure ADX cluster.")
+@ReadsAttributes({
+ @ReadsAttribute(attribute = "AUTH_STRATEGY", description = "The strategy/method to authenticate against Azure Active Directory, either 'application' or 'managed_identity'."),
+ @ReadsAttribute(attribute = "APP_ID", description = "Specifies Azure application id for accessing the ADX-Cluster."),
+ @ReadsAttribute(attribute = "APP_KEY", description = "Specifies Azure application key for accessing the ADX-Cluster."),
+ @ReadsAttribute(attribute = "APP_TENANT", description = "Azure application tenant for accessing the ADX-Cluster."),
+ @ReadsAttribute(attribute = "CLUSTER_URL", description = "Endpoint of ADX cluster. This is required only when streaming data to ADX cluster is enabled."),
+})
+public class StandardKustoIngestService extends AbstractControllerService implements KustoIngestService {
+
+ public static final PropertyDescriptor AUTHENTICATION_STRATEGY = new PropertyDescriptor
+ .Builder().name("Authentication Strategy")
+ .displayName("Authentication Strategy")
+ .description("Authentication method for access to Azure Data Explorer")
+ .required(true)
+ .defaultValue(KustoAuthenticationStrategy.MANAGED_IDENTITY.getValue())
+ .allowableValues(KustoAuthenticationStrategy.class)
+ .build();
+
+ public static final PropertyDescriptor APPLICATION_CLIENT_ID = new PropertyDescriptor
+ .Builder().name("Application Client ID")
+ .displayName("Application Client ID")
+ .description("Azure Data Explorer Application Client Identifier for Authentication")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+
+ public static final PropertyDescriptor APPLICATION_KEY = new PropertyDescriptor
+ .Builder().name("Application Key")
+ .displayName("Application Key")
+ .description("Azure Data Explorer Application Key for Authentication")
+ .required(true)
+ .sensitive(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .dependsOn(AUTHENTICATION_STRATEGY, KustoAuthenticationStrategy.APPLICATION_CREDENTIALS.getValue())
+ .build();
+
+ public static final PropertyDescriptor APPLICATION_TENANT_ID = new PropertyDescriptor.Builder()
+ .name("Application Tenant ID")
+ .displayName("Application Tenant ID")
+ .description("Azure Data Explorer Application Tenant Identifier for Authentication")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .dependsOn(AUTHENTICATION_STRATEGY, KustoAuthenticationStrategy.APPLICATION_CREDENTIALS.getValue())
+ .build();
+
+ public static final PropertyDescriptor CLUSTER_URI = new PropertyDescriptor
+ .Builder().name("Cluster URI")
+ .displayName("Cluster URI")
+ .description("Azure Data Explorer Cluster URI")
+ .required(true)
+ .addValidator(StandardValidators.URL_VALIDATOR)
+ .build();
+
+ private static final List<PropertyDescriptor> PROPERTY_DESCRIPTORS = Collections.unmodifiableList(Arrays.asList(
+ AUTHENTICATION_STRATEGY,
+ APPLICATION_CLIENT_ID,
+ APPLICATION_KEY,
+ APPLICATION_TENANT_ID,
+ CLUSTER_URI));
+
+ public static final Pair<String, String> NIFI_SINK = Pair.of("processor", "nifi-sink");
+
+ private volatile QueuedIngestClient queuedIngestClient;
+
+ private volatile ManagedStreamingIngestClient managedStreamingIngestClient;
+
+ private volatile Client executionClient;
+
+ @Override
+ public List<PropertyDescriptor> getSupportedPropertyDescriptors() {
+ return PROPERTY_DESCRIPTORS;
+ }
+
+ /**
+ * @param context the configuration context
+ * @throws InitializationException if unable to create a database connection
+ */
+ @OnEnabled
+ public void onEnabled(final ConfigurationContext context) throws ProcessException, URISyntaxException {
+
+ getLogger().info("Starting Azure ADX Connection Service...");
+ final String applicationClientId = context.getProperty(APPLICATION_CLIENT_ID).getValue();
+ final String applicationKey = context.getProperty(APPLICATION_KEY).getValue();
+ final String applicationTenantId = context.getProperty(APPLICATION_TENANT_ID).getValue();
+ final String clusterUri = context.getProperty(CLUSTER_URI).getValue();
+ final KustoAuthenticationStrategy kustoAuthenticationStrategy = KustoAuthenticationStrategy.valueOf(context.getProperty(AUTHENTICATION_STRATEGY).getValue());
+
+ if (this.queuedIngestClient == null) {
+ this.queuedIngestClient = createKustoQueuedIngestClient(clusterUri, applicationClientId, applicationKey, applicationTenantId, kustoAuthenticationStrategy);
+ }
+
+ if (this.managedStreamingIngestClient == null) {
+ this.managedStreamingIngestClient = createKustoStreamingIngestClient(clusterUri, applicationClientId, applicationKey, applicationTenantId, kustoAuthenticationStrategy);
+ }
+
+ if (this.executionClient == null) {
+ this.executionClient = createKustoExecutionClient(clusterUri, applicationClientId, applicationKey, applicationTenantId, kustoAuthenticationStrategy);
+ }
+
+ }
+
+ @OnStopped
+ public final void onStopped() {
+ if (this.queuedIngestClient != null) {
+ try {
+ this.queuedIngestClient.close();
+ } catch (IOException e) {
+ getLogger().error("Closing Azure ADX Queued Ingest Client failed with: " + e.getMessage(), e);
+ } finally {
+ this.queuedIngestClient = null;
+ }
+ }
+ if (this.managedStreamingIngestClient != null) {
+ try {
+ this.managedStreamingIngestClient.close();
+ } catch (IOException e) {
+ getLogger().error("Closing Azure ADX Managed Streaming Ingest Client failed with: " + e.getMessage(), e);
+ } finally {
+ this.managedStreamingIngestClient = null;
+ }
+ }
+ if (this.executionClient != null) {
+ try {
+ this.executionClient.close();
+ } catch (IOException e) {
+ getLogger().error("Closing Azure ADX Execution Client failed with: " + e.getMessage(), e);
Review Comment:
```suggestion
getLogger().error("Closing Azure ADX Execution Client failed", e);
```
##########
nifi-nar-bundles/nifi-azure-bundle/nifi-azure-processors/src/main/java/org/apache/nifi/services/azure/data/explorer/StandardKustoIngestService.java:
##########
@@ -0,0 +1,314 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.services.azure.data.explorer;
+
+import com.microsoft.azure.kusto.data.Client;
+import com.microsoft.azure.kusto.data.ClientFactory;
+import com.microsoft.azure.kusto.data.KustoResultSetTable;
+import com.microsoft.azure.kusto.data.auth.ConnectionStringBuilder;
+import com.microsoft.azure.kusto.data.exceptions.DataClientException;
+import com.microsoft.azure.kusto.data.exceptions.DataServiceException;
+import com.microsoft.azure.kusto.ingest.IngestClientFactory;
+import com.microsoft.azure.kusto.ingest.ManagedStreamingIngestClient;
+import com.microsoft.azure.kusto.ingest.QueuedIngestClient;
+import com.microsoft.azure.kusto.ingest.exceptions.IngestionClientException;
+import com.microsoft.azure.kusto.ingest.exceptions.IngestionServiceException;
+import com.microsoft.azure.kusto.ingest.result.IngestionResult;
+import com.microsoft.azure.kusto.ingest.result.IngestionStatus;
+import com.microsoft.azure.kusto.ingest.result.OperationStatus;
+import com.microsoft.azure.kusto.ingest.source.StreamSourceInfo;
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.nifi.annotation.behavior.ReadsAttribute;
+import org.apache.nifi.annotation.behavior.ReadsAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnEnabled;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.controller.AbstractControllerService;
+import org.apache.nifi.controller.ConfigurationContext;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.reporting.InitializationException;
+
+import java.io.IOException;
+import java.net.URISyntaxException;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.List;
+import java.util.Objects;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+
+@Tags({"Azure", "ADX", "Kusto", "ingest", "azure"})
+@CapabilityDescription("Sends batches of flowfile content or stream flowfile content to an Azure ADX cluster.")
+@ReadsAttributes({
+ @ReadsAttribute(attribute = "AUTH_STRATEGY", description = "The strategy/method to authenticate against Azure Active Directory, either 'application' or 'managed_identity'."),
+ @ReadsAttribute(attribute = "APP_ID", description = "Specifies Azure application id for accessing the ADX-Cluster."),
+ @ReadsAttribute(attribute = "APP_KEY", description = "Specifies Azure application key for accessing the ADX-Cluster."),
+ @ReadsAttribute(attribute = "APP_TENANT", description = "Azure application tenant for accessing the ADX-Cluster."),
+ @ReadsAttribute(attribute = "CLUSTER_URL", description = "Endpoint of ADX cluster. This is required only when streaming data to ADX cluster is enabled."),
+})
+public class StandardKustoIngestService extends AbstractControllerService implements KustoIngestService {
+
+ public static final PropertyDescriptor AUTHENTICATION_STRATEGY = new PropertyDescriptor
+ .Builder().name("Authentication Strategy")
+ .displayName("Authentication Strategy")
+ .description("Authentication method for access to Azure Data Explorer")
+ .required(true)
+ .defaultValue(KustoAuthenticationStrategy.MANAGED_IDENTITY.getValue())
+ .allowableValues(KustoAuthenticationStrategy.class)
+ .build();
+
+ public static final PropertyDescriptor APPLICATION_CLIENT_ID = new PropertyDescriptor
+ .Builder().name("Application Client ID")
+ .displayName("Application Client ID")
+ .description("Azure Data Explorer Application Client Identifier for Authentication")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+
+ public static final PropertyDescriptor APPLICATION_KEY = new PropertyDescriptor
+ .Builder().name("Application Key")
+ .displayName("Application Key")
+ .description("Azure Data Explorer Application Key for Authentication")
+ .required(true)
+ .sensitive(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .dependsOn(AUTHENTICATION_STRATEGY, KustoAuthenticationStrategy.APPLICATION_CREDENTIALS.getValue())
+ .build();
+
+ public static final PropertyDescriptor APPLICATION_TENANT_ID = new PropertyDescriptor.Builder()
+ .name("Application Tenant ID")
+ .displayName("Application Tenant ID")
+ .description("Azure Data Explorer Application Tenant Identifier for Authentication")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .dependsOn(AUTHENTICATION_STRATEGY, KustoAuthenticationStrategy.APPLICATION_CREDENTIALS.getValue())
+ .build();
+
+ public static final PropertyDescriptor CLUSTER_URI = new PropertyDescriptor
+ .Builder().name("Cluster URI")
+ .displayName("Cluster URI")
+ .description("Azure Data Explorer Cluster URI")
+ .required(true)
+ .addValidator(StandardValidators.URL_VALIDATOR)
+ .build();
+
+ private static final List<PropertyDescriptor> PROPERTY_DESCRIPTORS = Collections.unmodifiableList(Arrays.asList(
+ AUTHENTICATION_STRATEGY,
+ APPLICATION_CLIENT_ID,
+ APPLICATION_KEY,
+ APPLICATION_TENANT_ID,
+ CLUSTER_URI));
+
+ public static final Pair<String, String> NIFI_SINK = Pair.of("processor", "nifi-sink");
+
+ private volatile QueuedIngestClient queuedIngestClient;
+
+ private volatile ManagedStreamingIngestClient managedStreamingIngestClient;
+
+ private volatile Client executionClient;
+
+ @Override
+ public List<PropertyDescriptor> getSupportedPropertyDescriptors() {
+ return PROPERTY_DESCRIPTORS;
+ }
+
+ /**
+ * @param context the configuration context
+ * @throws InitializationException if unable to create a database connection
+ */
+ @OnEnabled
+ public void onEnabled(final ConfigurationContext context) throws ProcessException, URISyntaxException {
+
+ getLogger().info("Starting Azure ADX Connection Service...");
+ final String applicationClientId = context.getProperty(APPLICATION_CLIENT_ID).getValue();
+ final String applicationKey = context.getProperty(APPLICATION_KEY).getValue();
+ final String applicationTenantId = context.getProperty(APPLICATION_TENANT_ID).getValue();
+ final String clusterUri = context.getProperty(CLUSTER_URI).getValue();
+ final KustoAuthenticationStrategy kustoAuthenticationStrategy = KustoAuthenticationStrategy.valueOf(context.getProperty(AUTHENTICATION_STRATEGY).getValue());
+
+ if (this.queuedIngestClient == null) {
+ this.queuedIngestClient = createKustoQueuedIngestClient(clusterUri, applicationClientId, applicationKey, applicationTenantId, kustoAuthenticationStrategy);
+ }
+
+ if (this.managedStreamingIngestClient == null) {
+ this.managedStreamingIngestClient = createKustoStreamingIngestClient(clusterUri, applicationClientId, applicationKey, applicationTenantId, kustoAuthenticationStrategy);
+ }
+
+ if (this.executionClient == null) {
+ this.executionClient = createKustoExecutionClient(clusterUri, applicationClientId, applicationKey, applicationTenantId, kustoAuthenticationStrategy);
+ }
+
+ }
+
+ @OnStopped
+ public final void onStopped() {
+ if (this.queuedIngestClient != null) {
+ try {
+ this.queuedIngestClient.close();
+ } catch (IOException e) {
+ getLogger().error("Closing Azure ADX Queued Ingest Client failed with: " + e.getMessage(), e);
Review Comment:
The exception message should not be repeated because it will be included as part of the stack trace.
```suggestion
getLogger().error("Closing Azure ADX Queued Ingest Client failed", e);
```
##########
nifi-nar-bundles/nifi-azure-bundle/nifi-azure-processors/src/main/java/org/apache/nifi/processors/azure/data/explorer/IngestAzureDataExplorer.java:
##########
@@ -0,0 +1,401 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.azure.data.explorer;
+
+import com.microsoft.azure.kusto.data.KustoResultSetTable;
+import com.microsoft.azure.kusto.ingest.IngestionMapping;
+import com.microsoft.azure.kusto.ingest.IngestionProperties;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.nifi.annotation.behavior.ReadsAttribute;
+import org.apache.nifi.annotation.behavior.ReadsAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.expression.ExpressionLanguageScope;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.services.azure.data.explorer.KustoIngestService;
+import org.apache.nifi.services.azure.data.explorer.KustoIngestionRequest;
+import org.apache.nifi.services.azure.data.explorer.KustoIngestionResult;
+import org.apache.nifi.services.azure.data.explorer.KustoQueryResponse;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.net.URISyntaxException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+
+@Tags({"Azure", "Kusto", "ADX", "Explorer", "Data"})
+@CapabilityDescription("The Azure ADX Ingest Processor acts as a ADX sink connector which sends flowFiles using the ADX-Service to the provided Azure Data" +
+ "Explorer Ingest Endpoint. The data can be sent through queued ingestion or streaming ingestion to the Azure Data Explorer cluster.")
+@ReadsAttributes({
+ @ReadsAttribute(attribute = "DB_NAME", description = "Specifies the name of the ADX database where the data needs to be stored."),
+ @ReadsAttribute(attribute = "TABLE_NAME", description = "Specifies the name of the ADX table where the data needs to be stored."),
+ @ReadsAttribute(attribute = "MAPPING_NAME", description = "Specifies the name of the mapping responsible for storing the data in appropriate columns."),
+ @ReadsAttribute(attribute = "FLUSH_IMMEDIATE", description = "In case of queued ingestion, this property determines whether the data should be flushed immediately to the ingest endpoint."),
+ @ReadsAttribute(attribute = "DATA_FORMAT", description = "Specifies the format of data that is send to Azure Data Explorer."),
+ @ReadsAttribute(attribute = "IGNORE_FIRST_RECORD", description = "Specifies whether we want to ignore ingestion of first record. " +
+ "This is primarily applicable for csv files. Default is set to NO"),
+ @ReadsAttribute(attribute = "POLL_ON_INGESTION_STATUS", description = "Specifies whether we want to poll on ingestion result during ingestion into ADX." +
+ "In case of applications that need high throughput it is recommended to keep the default value as false. Default is set to false." +
+ "This property should be set to true during Queued Ingestion for near realtime micro-batches of data that require acknowledgement of ingestion status.")
+})
+public class IngestAzureDataExplorer extends AbstractProcessor {
+
+ public static final String FETCH_TABLE_COMMAND = "%s | count";
+ public static final String STREAMING_POLICY_SHOW_COMMAND = ".show %s %s policy streamingingestion";
+ public static final String DATABASE = "database";
+
+ private List<PropertyDescriptor> descriptors;
+ private Set<Relationship> relationships;
+ private transient KustoIngestService service;
+ private boolean isStreamingEnabled;
+ private boolean pollOnIngestionStatus;
+
+ public static final AllowableValue AVRO = new AllowableValue(
+ IngestionProperties.DataFormat.AVRO.getKustoValue(), IngestionProperties.DataFormat.AVRO.getKustoValue(),
+ "An Avro format with support for logical types and for the snappy compression codec");
+
+ public static final AllowableValue APACHEAVRO = new AllowableValue(
+ IngestionProperties.DataFormat.APACHEAVRO.getKustoValue(), IngestionProperties.DataFormat.APACHEAVRO.getKustoValue(),
+ "An Avro format with support for logical types and for the snappy compression codec.");
+
+ public static final AllowableValue CSV = new AllowableValue(
+ IngestionProperties.DataFormat.CSV.name(), IngestionProperties.DataFormat.CSV.getKustoValue(),
+ "A text file with comma-separated values (,). For more information, see RFC 4180: Common Format " +
+ "and MIME Type for Comma-Separated Values (CSV) Files.");
+
+ public static final AllowableValue JSON = new AllowableValue(
+ IngestionProperties.DataFormat.JSON.name(), IngestionProperties.DataFormat.JSON.getKustoValue(),
+ "A text file containing JSON objects separated by \\n or \\r\\n. For more information, " +
+ "see JSON Lines (JSONL).");
+
+ public static final AllowableValue MULTIJSON = new AllowableValue(
+ IngestionProperties.DataFormat.MULTIJSON.name(), IngestionProperties.DataFormat.MULTIJSON.getKustoValue(),
+ "A text file containing a JSON array of property containers (each representing a record) or any " +
+ "number of property containers separated by spaces, \\n or \\r\\n. Each property container may be " +
+ "spread across multiple lines. This format is preferable to JSON unless the data is not property " +
+ "containers.");
+
+ public static final AllowableValue ORC = new AllowableValue(
+ IngestionProperties.DataFormat.ORC.name(), IngestionProperties.DataFormat.ORC.getKustoValue(), "An ORC file.");
+
+ public static final AllowableValue PARQUET = new AllowableValue(
+ IngestionProperties.DataFormat.PARQUET.name(), IngestionProperties.DataFormat.PARQUET.getKustoValue(), "A parquet file.");
+
+ public static final AllowableValue PSV = new AllowableValue(
+ IngestionProperties.DataFormat.PSV.name(), IngestionProperties.DataFormat.PSV.getKustoValue(), "A text file with values separated by vertical bars (|).");
+
+ public static final AllowableValue SCSV = new AllowableValue(
+ IngestionProperties.DataFormat.SCSV.name(), IngestionProperties.DataFormat.SCSV.getKustoValue(), "A text file with values separated by semicolons (;).");
+
+ public static final AllowableValue SOHSV = new AllowableValue(
+ IngestionProperties.DataFormat.SOHSV.name(), IngestionProperties.DataFormat.SOHSV.getKustoValue(),
+ "A text file with SOH-separated values. (SOH is the ASCII code point 1. " +
+ "This format is used by Hive in HDInsight).");
+
+ public static final AllowableValue TSV = new AllowableValue(
+ IngestionProperties.DataFormat.TSV.name(), IngestionProperties.DataFormat.TSV.getKustoValue(), "A text file with tab delimited values (\\t).");
+
+ public static final AllowableValue TSVE = new AllowableValue(
+ IngestionProperties.DataFormat.TSVE.name(), IngestionProperties.DataFormat.TSVE.getKustoValue(),
+ "A text file with tab-delimited values (\\t). A backslash (\\) is used as escape character.");
+
+ public static final AllowableValue TXT = new AllowableValue(
+ IngestionProperties.DataFormat.TXT.name(), IngestionProperties.DataFormat.TXT.getKustoValue(),
+ "A text file with lines separated by \\n. Empty lines are skipped.");
+
+ public static final AllowableValue IGNORE_FIRST_RECORD_YES = new AllowableValue(
+ "YES", "YES",
+ "Ignore first record during ingestion");
+
+ public static final AllowableValue IGNORE_FIRST_RECORD_NO = new AllowableValue(
+ "NO", "NO",
+ "Do not ignore first record during ingestion");
+
+ public static final PropertyDescriptor DATABASE_NAME = new PropertyDescriptor.Builder()
+ .name("Database Name")
+ .displayName("Database Name")
+ .description("Azure Data Explorer Database Name for querying")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+ .build();
+
+ public static final PropertyDescriptor TABLE_NAME = new PropertyDescriptor.Builder()
+ .name("Table Name")
+ .displayName("Table Name")
+ .description("Azure Data Explorer Table Name for ingesting data")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+ .build();
+
+ public static final PropertyDescriptor MAPPING_NAME = new PropertyDescriptor
+ .Builder().name("Mapping name")
+ .displayName("Mapping name")
+ .description("The name of the mapping responsible for storing the data in the appropriate columns.")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+
+ public static final PropertyDescriptor IS_STREAMING_ENABLED = new PropertyDescriptor
+ .Builder().name("Is Streaming enabled")
+ .displayName("Is Streaming enabled")
+ .description("This property determines whether we want to stream data to ADX.")
+ .required(false)
+ .allowableValues("true", "false")
+ .addValidator(StandardValidators.BOOLEAN_VALIDATOR)
+ .defaultValue("false")
+ .build();
+
+ public static final PropertyDescriptor POLL_ON_INGESTION_STATUS = new PropertyDescriptor
+ .Builder().name("Poll on ingestion status")
+ .displayName("Whether to Poll on ingestion status")
+ .description("This property determines whether we want to poll on ingestion status after an ingestion to ADX is completed")
+ .required(false)
+ .allowableValues("true", "false")
+ .addValidator(StandardValidators.BOOLEAN_VALIDATOR)
+ .defaultValue("false")
+ .build();
+ public static final PropertyDescriptor ADX_SERVICE = new PropertyDescriptor
+ .Builder().name("Kusto Ingest Service")
+ .displayName("Kusto Ingest Service")
+ .description("Azure Data Explorer Kusto Ingest Service")
+ .required(true)
+ .identifiesControllerService(KustoIngestService.class)
+ .build();
+
+ public static final Relationship SUCCESS = new Relationship.Builder()
+ .name("success")
+ .description("Relationship for success")
+ .build();
+ public static final Relationship FAILURE = new Relationship.Builder()
+ .name("failure")
+ .description("Relationship for failure")
+ .build();
+
+ static final PropertyDescriptor DATA_FORMAT = new PropertyDescriptor.Builder()
+ .name("Data Format")
+ .displayName("Data Format")
+ .description("The format of the data that is sent to ADX.")
+ .required(true)
+ .allowableValues(AVRO, APACHEAVRO, CSV, JSON, MULTIJSON, ORC, PARQUET, PSV, SCSV, SOHSV, TSV, TSVE, TXT)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+
+ static final PropertyDescriptor IGNORE_FIRST_RECORD = new PropertyDescriptor.Builder()
+ .name("Ingestion Ignore First Record")
+ .displayName("Ingestion Ignore First Record")
+ .description("Defines whether ignore first record while ingestion.")
+ .required(false)
+ .allowableValues(IGNORE_FIRST_RECORD_YES, IGNORE_FIRST_RECORD_NO)
+ .defaultValue(IGNORE_FIRST_RECORD_NO.getValue())
+ .build();
+
+ @Override
+ protected void init(final ProcessorInitializationContext context) {
+ final List<PropertyDescriptor> descriptorList = new ArrayList<>();
+ descriptorList.add(ADX_SERVICE);
+ descriptorList.add(DATABASE_NAME);
+ descriptorList.add(TABLE_NAME);
+ descriptorList.add(MAPPING_NAME);
+ descriptorList.add(DATA_FORMAT);
+ descriptorList.add(IGNORE_FIRST_RECORD);
+ descriptorList.add(IS_STREAMING_ENABLED);
+ descriptorList.add(POLL_ON_INGESTION_STATUS);
+ this.descriptors = Collections.unmodifiableList(descriptorList);
+
+ final Set<Relationship> relationshipSet = new HashSet<>();
+ relationshipSet.add(SUCCESS);
+ relationshipSet.add(FAILURE);
+ this.relationships = Collections.unmodifiableSet(relationshipSet);
+ }
+
+ @Override
+ public Set<Relationship> getRelationships() {
+ return this.relationships;
+ }
+
+ @Override
+ public final List<PropertyDescriptor> getSupportedPropertyDescriptors() {
+ return descriptors;
+ }
+
+ @OnScheduled
+ public void onScheduled(final ProcessContext context) {
+ service = context.getProperty(ADX_SERVICE).asControllerService(KustoIngestService.class);
+ if (checkIfIngestorRoleDoesntExist(context.getProperty(DATABASE_NAME).getValue(), context.getProperty(TABLE_NAME).getValue())) {
+ getLogger().error("User might not have ingestor privileges, table validation will be skipped for all table mappings.");
+ throw new ProcessException("User might not have ingestor privileges, table validation will be skipped for all table mappings. ");
+ }
+ isStreamingEnabled = context.getProperty(IS_STREAMING_ENABLED).evaluateAttributeExpressions().asBoolean();
+ if (isStreamingEnabled) {
+ checkIfStreamingPolicyIsEnabledInADX(context.getProperty(DATABASE_NAME).getValue(), context.getProperty(DATABASE_NAME).getValue());
+ }
+ }
+
+ @Override
+ public void onTrigger(final ProcessContext context, final ProcessSession session) throws ProcessException {
+ FlowFile flowFile = session.get();
+ if (flowFile == null) {
+ context.yield();
+ return;
+ }
+
+ IngestionProperties ingestionProperties = new IngestionProperties(context.getProperty(DATABASE_NAME).getValue(),
+ context.getProperty(TABLE_NAME).getValue());
+
+ IngestionMapping.IngestionMappingKind ingestionMappingKind = null;
+
+ isStreamingEnabled = context.getProperty(IS_STREAMING_ENABLED).evaluateAttributeExpressions().asBoolean();
+ pollOnIngestionStatus = context.getProperty(POLL_ON_INGESTION_STATUS).evaluateAttributeExpressions().asBoolean();
+
+ switch (IngestionProperties.DataFormat.valueOf(context.getProperty(DATA_FORMAT).getValue())) {
+ case AVRO:
+ ingestionProperties.setDataFormat(IngestionProperties.DataFormat.AVRO);
+ ingestionMappingKind = IngestionProperties.DataFormat.AVRO.getIngestionMappingKind();
+ break;
+ case APACHEAVRO:
+ ingestionProperties.setDataFormat(IngestionProperties.DataFormat.APACHEAVRO);
+ ingestionMappingKind = IngestionProperties.DataFormat.APACHEAVRO.getIngestionMappingKind();
+ break;
+ case CSV:
+ ingestionProperties.setDataFormat(IngestionProperties.DataFormat.CSV);
+ ingestionMappingKind = IngestionProperties.DataFormat.CSV.getIngestionMappingKind();
+ break;
+ case JSON:
+ ingestionProperties.setDataFormat(IngestionProperties.DataFormat.JSON);
+ ingestionMappingKind = IngestionProperties.DataFormat.JSON.getIngestionMappingKind();
+ break;
+ case MULTIJSON:
+ ingestionProperties.setDataFormat(IngestionProperties.DataFormat.MULTIJSON);
+ ingestionMappingKind = IngestionProperties.DataFormat.MULTIJSON.getIngestionMappingKind();
+ break;
+ case ORC:
+ ingestionProperties.setDataFormat(IngestionProperties.DataFormat.ORC);
+ ingestionMappingKind = IngestionProperties.DataFormat.ORC.getIngestionMappingKind();
+ break;
+ case PARQUET:
+ ingestionProperties.setDataFormat(IngestionProperties.DataFormat.PARQUET);
+ ingestionMappingKind = IngestionProperties.DataFormat.PARQUET.getIngestionMappingKind();
+ break;
+ case PSV:
+ ingestionProperties.setDataFormat(IngestionProperties.DataFormat.PSV);
+ ingestionMappingKind = IngestionProperties.DataFormat.PSV.getIngestionMappingKind();
+ break;
+ case SCSV:
+ ingestionProperties.setDataFormat(IngestionProperties.DataFormat.SCSV);
+ ingestionMappingKind = IngestionProperties.DataFormat.SCSV.getIngestionMappingKind();
+ break;
+ case SOHSV:
+ ingestionProperties.setDataFormat(IngestionProperties.DataFormat.SOHSV);
+ ingestionMappingKind = IngestionProperties.DataFormat.SOHSV.getIngestionMappingKind();
+ break;
+ case TSV:
+ ingestionProperties.setDataFormat(IngestionProperties.DataFormat.TSV);
+ ingestionMappingKind = IngestionProperties.DataFormat.TSV.getIngestionMappingKind();
+ break;
+ case TSVE:
+ ingestionProperties.setDataFormat(IngestionProperties.DataFormat.TSVE);
+ ingestionMappingKind = IngestionProperties.DataFormat.TSVE.getIngestionMappingKind();
+ break;
+ case TXT:
+ ingestionProperties.setDataFormat(IngestionProperties.DataFormat.TXT);
+ ingestionMappingKind = IngestionProperties.DataFormat.TXT.getIngestionMappingKind();
+ break;
+ }
+
+ if (StringUtils.isNotEmpty(context.getProperty(MAPPING_NAME).getValue()) && ingestionMappingKind != null) {
+ ingestionProperties.setIngestionMapping(context.getProperty(MAPPING_NAME).getValue(), ingestionMappingKind);
+ }
+
+ ingestionProperties.setReportLevel(IngestionProperties.IngestionReportLevel.FAILURES_AND_SUCCESSES);
+ ingestionProperties.setReportMethod(IngestionProperties.IngestionReportMethod.TABLE);
+ ingestionProperties.setFlushImmediately(false);
+ ingestionProperties.setIgnoreFirstRecord(StringUtils.equalsIgnoreCase(context.getProperty(IGNORE_FIRST_RECORD).getValue(), IGNORE_FIRST_RECORD_YES.getValue()));
+
+ boolean isError = false;
+
+ // ingestion into ADX
+ try (final InputStream inputStream = session.read(flowFile)) {
+ StringBuilder ingestLogString = new StringBuilder().append("Ingesting with: ")
+ .append("dataFormat - ").append(ingestionProperties.getDataFormat()).append("|")
+ .append("ingestionMapping - ").append(ingestionProperties.getIngestionMapping().getIngestionMappingReference()).append("|")
+ .append("reportLevel - ").append(ingestionProperties.getReportLevel()).append("|")
+ .append("reportMethod - ").append(ingestionProperties.getReportMethod()).append("|")
+ .append("databaseName - ").append(ingestionProperties.getDatabaseName()).append("|")
+ .append("tableName - ").append(ingestionProperties.getTableName()).append("|")
+ .append("flushImmediately - ").append(ingestionProperties.getFlushImmediately());
+ getLogger().info(ingestLogString.toString());
+
+ KustoIngestionResult result = service.ingestData(new KustoIngestionRequest(isStreamingEnabled, pollOnIngestionStatus, inputStream, ingestionProperties));
+ getLogger().info("Poll on Ingestion Status {} and Operation status: {} ", pollOnIngestionStatus, result.toString());
+ if (result == KustoIngestionResult.SUCCEEDED) {
+ getLogger().info("Operation status Succeeded - {}", result.toString());
+ }
+
+ if (result == KustoIngestionResult.FAILED) {
+ getLogger().error("Operation status Error - {}", result.toString());
+ isError = true;
+ }
+
+ if (result == KustoIngestionResult.PARTIALLY_SUCCEEDED) {
+ getLogger().error("Operation status Partially succeeded - {}", result.toString());
+ isError = true;
+ }
+ } catch (IOException | URISyntaxException e) {
+ getLogger().error("Non Transactional/Streaming Ingestion mode : Exception occurred while ingesting data into ADX with exception {} ", e);
Review Comment:
The placeholder is not necessary for exceptions.
```suggestion
getLogger().error("Non Transactional/Streaming Ingestion mode : Exception occurred while ingesting data into ADX ", e);
```
##########
nifi-nar-bundles/nifi-azure-bundle/nifi-azure-processors/src/main/java/org/apache/nifi/services/azure/data/explorer/StandardKustoIngestService.java:
##########
@@ -0,0 +1,314 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.services.azure.data.explorer;
+
+import com.microsoft.azure.kusto.data.Client;
+import com.microsoft.azure.kusto.data.ClientFactory;
+import com.microsoft.azure.kusto.data.KustoResultSetTable;
+import com.microsoft.azure.kusto.data.auth.ConnectionStringBuilder;
+import com.microsoft.azure.kusto.data.exceptions.DataClientException;
+import com.microsoft.azure.kusto.data.exceptions.DataServiceException;
+import com.microsoft.azure.kusto.ingest.IngestClientFactory;
+import com.microsoft.azure.kusto.ingest.ManagedStreamingIngestClient;
+import com.microsoft.azure.kusto.ingest.QueuedIngestClient;
+import com.microsoft.azure.kusto.ingest.exceptions.IngestionClientException;
+import com.microsoft.azure.kusto.ingest.exceptions.IngestionServiceException;
+import com.microsoft.azure.kusto.ingest.result.IngestionResult;
+import com.microsoft.azure.kusto.ingest.result.IngestionStatus;
+import com.microsoft.azure.kusto.ingest.result.OperationStatus;
+import com.microsoft.azure.kusto.ingest.source.StreamSourceInfo;
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.nifi.annotation.behavior.ReadsAttribute;
+import org.apache.nifi.annotation.behavior.ReadsAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnEnabled;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.controller.AbstractControllerService;
+import org.apache.nifi.controller.ConfigurationContext;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.reporting.InitializationException;
+
+import java.io.IOException;
+import java.net.URISyntaxException;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.List;
+import java.util.Objects;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+
+@Tags({"Azure", "ADX", "Kusto", "ingest", "azure"})
+@CapabilityDescription("Sends batches of flowfile content or stream flowfile content to an Azure ADX cluster.")
+@ReadsAttributes({
+ @ReadsAttribute(attribute = "AUTH_STRATEGY", description = "The strategy/method to authenticate against Azure Active Directory, either 'application' or 'managed_identity'."),
+ @ReadsAttribute(attribute = "APP_ID", description = "Specifies Azure application id for accessing the ADX-Cluster."),
+ @ReadsAttribute(attribute = "APP_KEY", description = "Specifies Azure application key for accessing the ADX-Cluster."),
+ @ReadsAttribute(attribute = "APP_TENANT", description = "Azure application tenant for accessing the ADX-Cluster."),
+ @ReadsAttribute(attribute = "CLUSTER_URL", description = "Endpoint of ADX cluster. This is required only when streaming data to ADX cluster is enabled."),
+})
+public class StandardKustoIngestService extends AbstractControllerService implements KustoIngestService {
+
+ public static final PropertyDescriptor AUTHENTICATION_STRATEGY = new PropertyDescriptor
+ .Builder().name("Authentication Strategy")
+ .displayName("Authentication Strategy")
+ .description("Authentication method for access to Azure Data Explorer")
+ .required(true)
+ .defaultValue(KustoAuthenticationStrategy.MANAGED_IDENTITY.getValue())
+ .allowableValues(KustoAuthenticationStrategy.class)
+ .build();
+
+ public static final PropertyDescriptor APPLICATION_CLIENT_ID = new PropertyDescriptor
+ .Builder().name("Application Client ID")
+ .displayName("Application Client ID")
+ .description("Azure Data Explorer Application Client Identifier for Authentication")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+
+ public static final PropertyDescriptor APPLICATION_KEY = new PropertyDescriptor
+ .Builder().name("Application Key")
+ .displayName("Application Key")
+ .description("Azure Data Explorer Application Key for Authentication")
+ .required(true)
+ .sensitive(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .dependsOn(AUTHENTICATION_STRATEGY, KustoAuthenticationStrategy.APPLICATION_CREDENTIALS.getValue())
+ .build();
+
+ public static final PropertyDescriptor APPLICATION_TENANT_ID = new PropertyDescriptor.Builder()
+ .name("Application Tenant ID")
+ .displayName("Application Tenant ID")
+ .description("Azure Data Explorer Application Tenant Identifier for Authentication")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .dependsOn(AUTHENTICATION_STRATEGY, KustoAuthenticationStrategy.APPLICATION_CREDENTIALS.getValue())
+ .build();
+
+ public static final PropertyDescriptor CLUSTER_URI = new PropertyDescriptor
+ .Builder().name("Cluster URI")
+ .displayName("Cluster URI")
+ .description("Azure Data Explorer Cluster URI")
+ .required(true)
+ .addValidator(StandardValidators.URL_VALIDATOR)
+ .build();
+
+ private static final List<PropertyDescriptor> PROPERTY_DESCRIPTORS = Collections.unmodifiableList(Arrays.asList(
+ AUTHENTICATION_STRATEGY,
+ APPLICATION_CLIENT_ID,
+ APPLICATION_KEY,
+ APPLICATION_TENANT_ID,
+ CLUSTER_URI));
+
+ public static final Pair<String, String> NIFI_SINK = Pair.of("processor", "nifi-sink");
+
+ private volatile QueuedIngestClient queuedIngestClient;
+
+ private volatile ManagedStreamingIngestClient managedStreamingIngestClient;
+
+ private volatile Client executionClient;
+
+ @Override
+ public List<PropertyDescriptor> getSupportedPropertyDescriptors() {
+ return PROPERTY_DESCRIPTORS;
+ }
+
+ /**
+ * @param context the configuration context
+ * @throws InitializationException if unable to create a database connection
+ */
+ @OnEnabled
+ public void onEnabled(final ConfigurationContext context) throws ProcessException, URISyntaxException {
+
+ getLogger().info("Starting Azure ADX Connection Service...");
+ final String applicationClientId = context.getProperty(APPLICATION_CLIENT_ID).getValue();
+ final String applicationKey = context.getProperty(APPLICATION_KEY).getValue();
+ final String applicationTenantId = context.getProperty(APPLICATION_TENANT_ID).getValue();
+ final String clusterUri = context.getProperty(CLUSTER_URI).getValue();
+ final KustoAuthenticationStrategy kustoAuthenticationStrategy = KustoAuthenticationStrategy.valueOf(context.getProperty(AUTHENTICATION_STRATEGY).getValue());
+
+ if (this.queuedIngestClient == null) {
+ this.queuedIngestClient = createKustoQueuedIngestClient(clusterUri, applicationClientId, applicationKey, applicationTenantId, kustoAuthenticationStrategy);
+ }
+
+ if (this.managedStreamingIngestClient == null) {
+ this.managedStreamingIngestClient = createKustoStreamingIngestClient(clusterUri, applicationClientId, applicationKey, applicationTenantId, kustoAuthenticationStrategy);
+ }
+
+ if (this.executionClient == null) {
+ this.executionClient = createKustoExecutionClient(clusterUri, applicationClientId, applicationKey, applicationTenantId, kustoAuthenticationStrategy);
+ }
+
+ }
+
+ @OnStopped
+ public final void onStopped() {
+ if (this.queuedIngestClient != null) {
+ try {
+ this.queuedIngestClient.close();
+ } catch (IOException e) {
+ getLogger().error("Closing Azure ADX Queued Ingest Client failed with: " + e.getMessage(), e);
+ } finally {
+ this.queuedIngestClient = null;
+ }
+ }
+ if (this.managedStreamingIngestClient != null) {
+ try {
+ this.managedStreamingIngestClient.close();
+ } catch (IOException e) {
+ getLogger().error("Closing Azure ADX Managed Streaming Ingest Client failed with: " + e.getMessage(), e);
+ } finally {
+ this.managedStreamingIngestClient = null;
+ }
+ }
+ if (this.executionClient != null) {
+ try {
+ this.executionClient.close();
+ } catch (IOException e) {
+ getLogger().error("Closing Azure ADX Execution Client failed with: " + e.getMessage(), e);
+ } finally {
+ this.executionClient = null;
+ }
+ }
+ }
+
+
+ protected QueuedIngestClient createKustoQueuedIngestClient(final String clusterUrl,
+ final String appId,
+ final String appKey,
+ final String appTenant,
+ final KustoAuthenticationStrategy kustoAuthStrategy) throws URISyntaxException {
+ ConnectionStringBuilder ingestConnectionStringBuilder = createKustoEngineConnectionString(clusterUrl, appId, appKey, appTenant, kustoAuthStrategy);
+ return IngestClientFactory.createClient(ingestConnectionStringBuilder);
+ }
+
+ protected ManagedStreamingIngestClient createKustoStreamingIngestClient(final String clusterUrl,
+ final String appId,
+ final String appKey,
+ final String appTenant,
+ final KustoAuthenticationStrategy kustoAuthStrategy) throws URISyntaxException {
+ ConnectionStringBuilder ingestConnectionStringBuilder = createKustoEngineConnectionString(clusterUrl, appId, appKey, appTenant, kustoAuthStrategy);
+ ConnectionStringBuilder streamingConnectionStringBuilder = createKustoEngineConnectionString(clusterUrl, appId, appKey, appTenant, kustoAuthStrategy);
+ return IngestClientFactory.createManagedStreamingIngestClient(ingestConnectionStringBuilder, streamingConnectionStringBuilder);
+ }
+
+ public KustoIngestionResult ingestData(KustoIngestionRequest kustoIngestionRequest) {
+ StreamSourceInfo info = new StreamSourceInfo(kustoIngestionRequest.getInputStream());
+ //ingest data
+ IngestionResult ingestionResult;
+ ScheduledExecutorService scheduler = Executors.newScheduledThreadPool(1);
Review Comment:
This is problematic because it creates a new scheduler for every invocation, which is very inefficient. Is there a reason to run this in a separate thread? If so, a shared Thread Pool should be created once and reused. However, creating a separate thread for running may not be necessary.
##########
nifi-nar-bundles/nifi-azure-bundle/nifi-azure-processors/src/main/java/org/apache/nifi/services/azure/data/explorer/StandardKustoQueryService.java:
##########
@@ -159,10 +162,7 @@ private ConnectionStringBuilder getConnectionStringBuilder(final ConfigurationCo
builder = ConnectionStringBuilder.createWithAadApplicationCredentials(clusterUrl, clientId, applicationKey, tenantId);
}
- final String vendor = System.getProperty("java.vendor");
- final String version = System.getProperty("java.version");
-
- builder.setConnectorDetails(vendor, version, null, null, false, null);
+ builder.setConnectorDetails("Kusto.Nifi.Source", StandardKustoQueryService.class.getPackage().getImplementationVersion(), null, null, false, null, NIFI_SOURCE);
Review Comment:
Is this change necessary? Keeping the Java version information makes this more generic than the NiFi-specific details.
##########
nifi-nar-bundles/nifi-azure-bundle/nifi-azure-processors/src/main/java/org/apache/nifi/services/azure/data/explorer/StandardKustoIngestService.java:
##########
@@ -0,0 +1,314 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.services.azure.data.explorer;
+
+import com.microsoft.azure.kusto.data.Client;
+import com.microsoft.azure.kusto.data.ClientFactory;
+import com.microsoft.azure.kusto.data.KustoResultSetTable;
+import com.microsoft.azure.kusto.data.auth.ConnectionStringBuilder;
+import com.microsoft.azure.kusto.data.exceptions.DataClientException;
+import com.microsoft.azure.kusto.data.exceptions.DataServiceException;
+import com.microsoft.azure.kusto.ingest.IngestClientFactory;
+import com.microsoft.azure.kusto.ingest.ManagedStreamingIngestClient;
+import com.microsoft.azure.kusto.ingest.QueuedIngestClient;
+import com.microsoft.azure.kusto.ingest.exceptions.IngestionClientException;
+import com.microsoft.azure.kusto.ingest.exceptions.IngestionServiceException;
+import com.microsoft.azure.kusto.ingest.result.IngestionResult;
+import com.microsoft.azure.kusto.ingest.result.IngestionStatus;
+import com.microsoft.azure.kusto.ingest.result.OperationStatus;
+import com.microsoft.azure.kusto.ingest.source.StreamSourceInfo;
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.nifi.annotation.behavior.ReadsAttribute;
+import org.apache.nifi.annotation.behavior.ReadsAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnEnabled;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.controller.AbstractControllerService;
+import org.apache.nifi.controller.ConfigurationContext;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.reporting.InitializationException;
+
+import java.io.IOException;
+import java.net.URISyntaxException;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.List;
+import java.util.Objects;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+
+@Tags({"Azure", "ADX", "Kusto", "ingest", "azure"})
+@CapabilityDescription("Sends batches of flowfile content or stream flowfile content to an Azure ADX cluster.")
+@ReadsAttributes({
+ @ReadsAttribute(attribute = "AUTH_STRATEGY", description = "The strategy/method to authenticate against Azure Active Directory, either 'application' or 'managed_identity'."),
+ @ReadsAttribute(attribute = "APP_ID", description = "Specifies Azure application id for accessing the ADX-Cluster."),
+ @ReadsAttribute(attribute = "APP_KEY", description = "Specifies Azure application key for accessing the ADX-Cluster."),
+ @ReadsAttribute(attribute = "APP_TENANT", description = "Azure application tenant for accessing the ADX-Cluster."),
+ @ReadsAttribute(attribute = "CLUSTER_URL", description = "Endpoint of ADX cluster. This is required only when streaming data to ADX cluster is enabled."),
+})
+public class StandardKustoIngestService extends AbstractControllerService implements KustoIngestService {
+
+ public static final PropertyDescriptor AUTHENTICATION_STRATEGY = new PropertyDescriptor
+ .Builder().name("Authentication Strategy")
+ .displayName("Authentication Strategy")
+ .description("Authentication method for access to Azure Data Explorer")
+ .required(true)
+ .defaultValue(KustoAuthenticationStrategy.MANAGED_IDENTITY.getValue())
+ .allowableValues(KustoAuthenticationStrategy.class)
+ .build();
+
+ public static final PropertyDescriptor APPLICATION_CLIENT_ID = new PropertyDescriptor
+ .Builder().name("Application Client ID")
+ .displayName("Application Client ID")
+ .description("Azure Data Explorer Application Client Identifier for Authentication")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+
+ public static final PropertyDescriptor APPLICATION_KEY = new PropertyDescriptor
+ .Builder().name("Application Key")
+ .displayName("Application Key")
+ .description("Azure Data Explorer Application Key for Authentication")
+ .required(true)
+ .sensitive(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .dependsOn(AUTHENTICATION_STRATEGY, KustoAuthenticationStrategy.APPLICATION_CREDENTIALS.getValue())
+ .build();
+
+ public static final PropertyDescriptor APPLICATION_TENANT_ID = new PropertyDescriptor.Builder()
+ .name("Application Tenant ID")
+ .displayName("Application Tenant ID")
+ .description("Azure Data Explorer Application Tenant Identifier for Authentication")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .dependsOn(AUTHENTICATION_STRATEGY, KustoAuthenticationStrategy.APPLICATION_CREDENTIALS.getValue())
+ .build();
+
+ public static final PropertyDescriptor CLUSTER_URI = new PropertyDescriptor
+ .Builder().name("Cluster URI")
+ .displayName("Cluster URI")
+ .description("Azure Data Explorer Cluster URI")
+ .required(true)
+ .addValidator(StandardValidators.URL_VALIDATOR)
+ .build();
+
+ private static final List<PropertyDescriptor> PROPERTY_DESCRIPTORS = Collections.unmodifiableList(Arrays.asList(
+ AUTHENTICATION_STRATEGY,
+ APPLICATION_CLIENT_ID,
+ APPLICATION_KEY,
+ APPLICATION_TENANT_ID,
+ CLUSTER_URI));
+
+ public static final Pair<String, String> NIFI_SINK = Pair.of("processor", "nifi-sink");
+
+ private volatile QueuedIngestClient queuedIngestClient;
+
+ private volatile ManagedStreamingIngestClient managedStreamingIngestClient;
+
+ private volatile Client executionClient;
+
+ @Override
+ public List<PropertyDescriptor> getSupportedPropertyDescriptors() {
+ return PROPERTY_DESCRIPTORS;
+ }
+
+ /**
+ * @param context the configuration context
+ * @throws InitializationException if unable to create a database connection
+ */
+ @OnEnabled
+ public void onEnabled(final ConfigurationContext context) throws ProcessException, URISyntaxException {
+
+ getLogger().info("Starting Azure ADX Connection Service...");
+ final String applicationClientId = context.getProperty(APPLICATION_CLIENT_ID).getValue();
+ final String applicationKey = context.getProperty(APPLICATION_KEY).getValue();
+ final String applicationTenantId = context.getProperty(APPLICATION_TENANT_ID).getValue();
+ final String clusterUri = context.getProperty(CLUSTER_URI).getValue();
+ final KustoAuthenticationStrategy kustoAuthenticationStrategy = KustoAuthenticationStrategy.valueOf(context.getProperty(AUTHENTICATION_STRATEGY).getValue());
+
+ if (this.queuedIngestClient == null) {
+ this.queuedIngestClient = createKustoQueuedIngestClient(clusterUri, applicationClientId, applicationKey, applicationTenantId, kustoAuthenticationStrategy);
+ }
+
+ if (this.managedStreamingIngestClient == null) {
+ this.managedStreamingIngestClient = createKustoStreamingIngestClient(clusterUri, applicationClientId, applicationKey, applicationTenantId, kustoAuthenticationStrategy);
+ }
+
+ if (this.executionClient == null) {
+ this.executionClient = createKustoExecutionClient(clusterUri, applicationClientId, applicationKey, applicationTenantId, kustoAuthenticationStrategy);
+ }
+
+ }
+
+ @OnStopped
+ public final void onStopped() {
+ if (this.queuedIngestClient != null) {
+ try {
+ this.queuedIngestClient.close();
+ } catch (IOException e) {
+ getLogger().error("Closing Azure ADX Queued Ingest Client failed with: " + e.getMessage(), e);
+ } finally {
+ this.queuedIngestClient = null;
+ }
+ }
+ if (this.managedStreamingIngestClient != null) {
+ try {
+ this.managedStreamingIngestClient.close();
+ } catch (IOException e) {
+ getLogger().error("Closing Azure ADX Managed Streaming Ingest Client failed with: " + e.getMessage(), e);
Review Comment:
```suggestion
getLogger().error("Closing Azure ADX Managed Streaming Ingest Client failed", e);
```
##########
nifi-nar-bundles/nifi-azure-bundle/nifi-azure-services-api/pom.xml:
##########
@@ -36,6 +36,16 @@
<groupId>org.apache.nifi</groupId>
<artifactId>nifi-api</artifactId>
</dependency>
+ <dependency>
+ <groupId>com.microsoft.azure.kusto</groupId>
+ <artifactId>kusto-data</artifactId>
+ <version>5.0.1</version>
+ </dependency>
+ <dependency>
+ <groupId>com.microsoft.azure.kusto</groupId>
+ <artifactId>kusto-ingest</artifactId>
+ <version>5.0.1</version>
+ </dependency>
Review Comment:
Are these additions necessary at the API package level? It would be better to structure the interface to avoid the need for these dependencies if possible.
##########
nifi-nar-bundles/nifi-azure-bundle/nifi-azure-processors/src/main/java/org/apache/nifi/processors/azure/data/explorer/IngestAzureDataExplorer.java:
##########
@@ -0,0 +1,401 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.azure.data.explorer;
+
+import com.microsoft.azure.kusto.data.KustoResultSetTable;
+import com.microsoft.azure.kusto.ingest.IngestionMapping;
+import com.microsoft.azure.kusto.ingest.IngestionProperties;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.nifi.annotation.behavior.ReadsAttribute;
+import org.apache.nifi.annotation.behavior.ReadsAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.expression.ExpressionLanguageScope;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.services.azure.data.explorer.KustoIngestService;
+import org.apache.nifi.services.azure.data.explorer.KustoIngestionRequest;
+import org.apache.nifi.services.azure.data.explorer.KustoIngestionResult;
+import org.apache.nifi.services.azure.data.explorer.KustoQueryResponse;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.net.URISyntaxException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+
+@Tags({"Azure", "Kusto", "ADX", "Explorer", "Data"})
+@CapabilityDescription("The Azure ADX Ingest Processor acts as a ADX sink connector which sends flowFiles using the ADX-Service to the provided Azure Data" +
+ "Explorer Ingest Endpoint. The data can be sent through queued ingestion or streaming ingestion to the Azure Data Explorer cluster.")
+@ReadsAttributes({
+ @ReadsAttribute(attribute = "DB_NAME", description = "Specifies the name of the ADX database where the data needs to be stored."),
+ @ReadsAttribute(attribute = "TABLE_NAME", description = "Specifies the name of the ADX table where the data needs to be stored."),
+ @ReadsAttribute(attribute = "MAPPING_NAME", description = "Specifies the name of the mapping responsible for storing the data in appropriate columns."),
+ @ReadsAttribute(attribute = "FLUSH_IMMEDIATE", description = "In case of queued ingestion, this property determines whether the data should be flushed immediately to the ingest endpoint."),
+ @ReadsAttribute(attribute = "DATA_FORMAT", description = "Specifies the format of data that is send to Azure Data Explorer."),
+ @ReadsAttribute(attribute = "IGNORE_FIRST_RECORD", description = "Specifies whether we want to ignore ingestion of first record. " +
+ "This is primarily applicable for csv files. Default is set to NO"),
+ @ReadsAttribute(attribute = "POLL_ON_INGESTION_STATUS", description = "Specifies whether we want to poll on ingestion result during ingestion into ADX." +
+ "In case of applications that need high throughput it is recommended to keep the default value as false. Default is set to false." +
+ "This property should be set to true during Queued Ingestion for near realtime micro-batches of data that require acknowledgement of ingestion status.")
+})
+public class IngestAzureDataExplorer extends AbstractProcessor {
+
+ public static final String FETCH_TABLE_COMMAND = "%s | count";
+ public static final String STREAMING_POLICY_SHOW_COMMAND = ".show %s %s policy streamingingestion";
+ public static final String DATABASE = "database";
+
+ private List<PropertyDescriptor> descriptors;
+ private Set<Relationship> relationships;
+ private transient KustoIngestService service;
+ private boolean isStreamingEnabled;
+ private boolean pollOnIngestionStatus;
+
+ public static final AllowableValue AVRO = new AllowableValue(
+ IngestionProperties.DataFormat.AVRO.getKustoValue(), IngestionProperties.DataFormat.AVRO.getKustoValue(),
+ "An Avro format with support for logical types and for the snappy compression codec");
+
+ public static final AllowableValue APACHEAVRO = new AllowableValue(
+ IngestionProperties.DataFormat.APACHEAVRO.getKustoValue(), IngestionProperties.DataFormat.APACHEAVRO.getKustoValue(),
+ "An Avro format with support for logical types and for the snappy compression codec.");
+
+ public static final AllowableValue CSV = new AllowableValue(
+ IngestionProperties.DataFormat.CSV.name(), IngestionProperties.DataFormat.CSV.getKustoValue(),
+ "A text file with comma-separated values (,). For more information, see RFC 4180: Common Format " +
+ "and MIME Type for Comma-Separated Values (CSV) Files.");
+
+ public static final AllowableValue JSON = new AllowableValue(
+ IngestionProperties.DataFormat.JSON.name(), IngestionProperties.DataFormat.JSON.getKustoValue(),
+ "A text file containing JSON objects separated by \\n or \\r\\n. For more information, " +
+ "see JSON Lines (JSONL).");
+
+ public static final AllowableValue MULTIJSON = new AllowableValue(
+ IngestionProperties.DataFormat.MULTIJSON.name(), IngestionProperties.DataFormat.MULTIJSON.getKustoValue(),
+ "A text file containing a JSON array of property containers (each representing a record) or any " +
+ "number of property containers separated by spaces, \\n or \\r\\n. Each property container may be " +
+ "spread across multiple lines. This format is preferable to JSON unless the data is not property " +
+ "containers.");
+
+ public static final AllowableValue ORC = new AllowableValue(
+ IngestionProperties.DataFormat.ORC.name(), IngestionProperties.DataFormat.ORC.getKustoValue(), "An ORC file.");
+
+ public static final AllowableValue PARQUET = new AllowableValue(
+ IngestionProperties.DataFormat.PARQUET.name(), IngestionProperties.DataFormat.PARQUET.getKustoValue(), "A parquet file.");
+
+ public static final AllowableValue PSV = new AllowableValue(
+ IngestionProperties.DataFormat.PSV.name(), IngestionProperties.DataFormat.PSV.getKustoValue(), "A text file with values separated by vertical bars (|).");
+
+ public static final AllowableValue SCSV = new AllowableValue(
+ IngestionProperties.DataFormat.SCSV.name(), IngestionProperties.DataFormat.SCSV.getKustoValue(), "A text file with values separated by semicolons (;).");
+
+ public static final AllowableValue SOHSV = new AllowableValue(
+ IngestionProperties.DataFormat.SOHSV.name(), IngestionProperties.DataFormat.SOHSV.getKustoValue(),
+ "A text file with SOH-separated values. (SOH is the ASCII code point 1. " +
+ "This format is used by Hive in HDInsight).");
+
+ public static final AllowableValue TSV = new AllowableValue(
+ IngestionProperties.DataFormat.TSV.name(), IngestionProperties.DataFormat.TSV.getKustoValue(), "A text file with tab delimited values (\\t).");
+
+ public static final AllowableValue TSVE = new AllowableValue(
+ IngestionProperties.DataFormat.TSVE.name(), IngestionProperties.DataFormat.TSVE.getKustoValue(),
+ "A text file with tab-delimited values (\\t). A backslash (\\) is used as escape character.");
+
+ public static final AllowableValue TXT = new AllowableValue(
+ IngestionProperties.DataFormat.TXT.name(), IngestionProperties.DataFormat.TXT.getKustoValue(),
+ "A text file with lines separated by \\n. Empty lines are skipped.");
+
+ public static final AllowableValue IGNORE_FIRST_RECORD_YES = new AllowableValue(
+ "YES", "YES",
+ "Ignore first record during ingestion");
+
+ public static final AllowableValue IGNORE_FIRST_RECORD_NO = new AllowableValue(
+ "NO", "NO",
+ "Do not ignore first record during ingestion");
+
+ public static final PropertyDescriptor DATABASE_NAME = new PropertyDescriptor.Builder()
+ .name("Database Name")
+ .displayName("Database Name")
+ .description("Azure Data Explorer Database Name for querying")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+ .build();
+
+ public static final PropertyDescriptor TABLE_NAME = new PropertyDescriptor.Builder()
+ .name("Table Name")
+ .displayName("Table Name")
+ .description("Azure Data Explorer Table Name for ingesting data")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+ .build();
+
+ public static final PropertyDescriptor MAPPING_NAME = new PropertyDescriptor
+ .Builder().name("Mapping name")
+ .displayName("Mapping name")
+ .description("The name of the mapping responsible for storing the data in the appropriate columns.")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+
+ public static final PropertyDescriptor IS_STREAMING_ENABLED = new PropertyDescriptor
+ .Builder().name("Is Streaming enabled")
+ .displayName("Is Streaming enabled")
+ .description("This property determines whether we want to stream data to ADX.")
+ .required(false)
+ .allowableValues("true", "false")
+ .addValidator(StandardValidators.BOOLEAN_VALIDATOR)
+ .defaultValue("false")
+ .build();
+
+ public static final PropertyDescriptor POLL_ON_INGESTION_STATUS = new PropertyDescriptor
+ .Builder().name("Poll on ingestion status")
+ .displayName("Whether to Poll on ingestion status")
+ .description("This property determines whether we want to poll on ingestion status after an ingestion to ADX is completed")
+ .required(false)
+ .allowableValues("true", "false")
+ .addValidator(StandardValidators.BOOLEAN_VALIDATOR)
+ .defaultValue("false")
+ .build();
+ public static final PropertyDescriptor ADX_SERVICE = new PropertyDescriptor
+ .Builder().name("Kusto Ingest Service")
+ .displayName("Kusto Ingest Service")
+ .description("Azure Data Explorer Kusto Ingest Service")
+ .required(true)
+ .identifiesControllerService(KustoIngestService.class)
+ .build();
+
+ public static final Relationship SUCCESS = new Relationship.Builder()
+ .name("success")
+ .description("Relationship for success")
+ .build();
+ public static final Relationship FAILURE = new Relationship.Builder()
+ .name("failure")
+ .description("Relationship for failure")
+ .build();
+
+ static final PropertyDescriptor DATA_FORMAT = new PropertyDescriptor.Builder()
+ .name("Data Format")
+ .displayName("Data Format")
+ .description("The format of the data that is sent to ADX.")
+ .required(true)
+ .allowableValues(AVRO, APACHEAVRO, CSV, JSON, MULTIJSON, ORC, PARQUET, PSV, SCSV, SOHSV, TSV, TSVE, TXT)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+
+ static final PropertyDescriptor IGNORE_FIRST_RECORD = new PropertyDescriptor.Builder()
+ .name("Ingestion Ignore First Record")
+ .displayName("Ingestion Ignore First Record")
+ .description("Defines whether ignore first record while ingestion.")
+ .required(false)
+ .allowableValues(IGNORE_FIRST_RECORD_YES, IGNORE_FIRST_RECORD_NO)
+ .defaultValue(IGNORE_FIRST_RECORD_NO.getValue())
+ .build();
+
+ @Override
+ protected void init(final ProcessorInitializationContext context) {
+ final List<PropertyDescriptor> descriptorList = new ArrayList<>();
+ descriptorList.add(ADX_SERVICE);
+ descriptorList.add(DATABASE_NAME);
+ descriptorList.add(TABLE_NAME);
+ descriptorList.add(MAPPING_NAME);
+ descriptorList.add(DATA_FORMAT);
+ descriptorList.add(IGNORE_FIRST_RECORD);
+ descriptorList.add(IS_STREAMING_ENABLED);
+ descriptorList.add(POLL_ON_INGESTION_STATUS);
+ this.descriptors = Collections.unmodifiableList(descriptorList);
+
+ final Set<Relationship> relationshipSet = new HashSet<>();
+ relationshipSet.add(SUCCESS);
+ relationshipSet.add(FAILURE);
+ this.relationships = Collections.unmodifiableSet(relationshipSet);
+ }
+
+ @Override
+ public Set<Relationship> getRelationships() {
+ return this.relationships;
+ }
+
+ @Override
+ public final List<PropertyDescriptor> getSupportedPropertyDescriptors() {
+ return descriptors;
+ }
+
+ @OnScheduled
+ public void onScheduled(final ProcessContext context) {
+ service = context.getProperty(ADX_SERVICE).asControllerService(KustoIngestService.class);
+ if (checkIfIngestorRoleDoesntExist(context.getProperty(DATABASE_NAME).getValue(), context.getProperty(TABLE_NAME).getValue())) {
+ getLogger().error("User might not have ingestor privileges, table validation will be skipped for all table mappings.");
+ throw new ProcessException("User might not have ingestor privileges, table validation will be skipped for all table mappings. ");
+ }
+ isStreamingEnabled = context.getProperty(IS_STREAMING_ENABLED).evaluateAttributeExpressions().asBoolean();
+ if (isStreamingEnabled) {
+ checkIfStreamingPolicyIsEnabledInADX(context.getProperty(DATABASE_NAME).getValue(), context.getProperty(DATABASE_NAME).getValue());
+ }
+ }
+
+ @Override
+ public void onTrigger(final ProcessContext context, final ProcessSession session) throws ProcessException {
+ FlowFile flowFile = session.get();
+ if (flowFile == null) {
+ context.yield();
+ return;
+ }
+
+ IngestionProperties ingestionProperties = new IngestionProperties(context.getProperty(DATABASE_NAME).getValue(),
+ context.getProperty(TABLE_NAME).getValue());
+
+ IngestionMapping.IngestionMappingKind ingestionMappingKind = null;
+
+ isStreamingEnabled = context.getProperty(IS_STREAMING_ENABLED).evaluateAttributeExpressions().asBoolean();
+ pollOnIngestionStatus = context.getProperty(POLL_ON_INGESTION_STATUS).evaluateAttributeExpressions().asBoolean();
+
+ switch (IngestionProperties.DataFormat.valueOf(context.getProperty(DATA_FORMAT).getValue())) {
+ case AVRO:
+ ingestionProperties.setDataFormat(IngestionProperties.DataFormat.AVRO);
+ ingestionMappingKind = IngestionProperties.DataFormat.AVRO.getIngestionMappingKind();
+ break;
+ case APACHEAVRO:
+ ingestionProperties.setDataFormat(IngestionProperties.DataFormat.APACHEAVRO);
+ ingestionMappingKind = IngestionProperties.DataFormat.APACHEAVRO.getIngestionMappingKind();
+ break;
+ case CSV:
+ ingestionProperties.setDataFormat(IngestionProperties.DataFormat.CSV);
+ ingestionMappingKind = IngestionProperties.DataFormat.CSV.getIngestionMappingKind();
+ break;
+ case JSON:
+ ingestionProperties.setDataFormat(IngestionProperties.DataFormat.JSON);
+ ingestionMappingKind = IngestionProperties.DataFormat.JSON.getIngestionMappingKind();
+ break;
+ case MULTIJSON:
+ ingestionProperties.setDataFormat(IngestionProperties.DataFormat.MULTIJSON);
+ ingestionMappingKind = IngestionProperties.DataFormat.MULTIJSON.getIngestionMappingKind();
+ break;
+ case ORC:
+ ingestionProperties.setDataFormat(IngestionProperties.DataFormat.ORC);
+ ingestionMappingKind = IngestionProperties.DataFormat.ORC.getIngestionMappingKind();
+ break;
+ case PARQUET:
+ ingestionProperties.setDataFormat(IngestionProperties.DataFormat.PARQUET);
+ ingestionMappingKind = IngestionProperties.DataFormat.PARQUET.getIngestionMappingKind();
+ break;
+ case PSV:
+ ingestionProperties.setDataFormat(IngestionProperties.DataFormat.PSV);
+ ingestionMappingKind = IngestionProperties.DataFormat.PSV.getIngestionMappingKind();
+ break;
+ case SCSV:
+ ingestionProperties.setDataFormat(IngestionProperties.DataFormat.SCSV);
+ ingestionMappingKind = IngestionProperties.DataFormat.SCSV.getIngestionMappingKind();
+ break;
+ case SOHSV:
+ ingestionProperties.setDataFormat(IngestionProperties.DataFormat.SOHSV);
+ ingestionMappingKind = IngestionProperties.DataFormat.SOHSV.getIngestionMappingKind();
+ break;
+ case TSV:
+ ingestionProperties.setDataFormat(IngestionProperties.DataFormat.TSV);
+ ingestionMappingKind = IngestionProperties.DataFormat.TSV.getIngestionMappingKind();
+ break;
+ case TSVE:
+ ingestionProperties.setDataFormat(IngestionProperties.DataFormat.TSVE);
+ ingestionMappingKind = IngestionProperties.DataFormat.TSVE.getIngestionMappingKind();
+ break;
+ case TXT:
+ ingestionProperties.setDataFormat(IngestionProperties.DataFormat.TXT);
+ ingestionMappingKind = IngestionProperties.DataFormat.TXT.getIngestionMappingKind();
+ break;
+ }
+
+ if (StringUtils.isNotEmpty(context.getProperty(MAPPING_NAME).getValue()) && ingestionMappingKind != null) {
+ ingestionProperties.setIngestionMapping(context.getProperty(MAPPING_NAME).getValue(), ingestionMappingKind);
+ }
+
+ ingestionProperties.setReportLevel(IngestionProperties.IngestionReportLevel.FAILURES_AND_SUCCESSES);
+ ingestionProperties.setReportMethod(IngestionProperties.IngestionReportMethod.TABLE);
+ ingestionProperties.setFlushImmediately(false);
+ ingestionProperties.setIgnoreFirstRecord(StringUtils.equalsIgnoreCase(context.getProperty(IGNORE_FIRST_RECORD).getValue(), IGNORE_FIRST_RECORD_YES.getValue()));
+
+ boolean isError = false;
+
+ // ingestion into ADX
+ try (final InputStream inputStream = session.read(flowFile)) {
+ StringBuilder ingestLogString = new StringBuilder().append("Ingesting with: ")
+ .append("dataFormat - ").append(ingestionProperties.getDataFormat()).append("|")
+ .append("ingestionMapping - ").append(ingestionProperties.getIngestionMapping().getIngestionMappingReference()).append("|")
+ .append("reportLevel - ").append(ingestionProperties.getReportLevel()).append("|")
+ .append("reportMethod - ").append(ingestionProperties.getReportMethod()).append("|")
+ .append("databaseName - ").append(ingestionProperties.getDatabaseName()).append("|")
+ .append("tableName - ").append(ingestionProperties.getTableName()).append("|")
+ .append("flushImmediately - ").append(ingestionProperties.getFlushImmediately());
+ getLogger().info(ingestLogString.toString());
+
+ KustoIngestionResult result = service.ingestData(new KustoIngestionRequest(isStreamingEnabled, pollOnIngestionStatus, inputStream, ingestionProperties));
+ getLogger().info("Poll on Ingestion Status {} and Operation status: {} ", pollOnIngestionStatus, result.toString());
+ if (result == KustoIngestionResult.SUCCEEDED) {
+ getLogger().info("Operation status Succeeded - {}", result.toString());
+ }
+
+ if (result == KustoIngestionResult.FAILED) {
+ getLogger().error("Operation status Error - {}", result.toString());
+ isError = true;
+ }
+
+ if (result == KustoIngestionResult.PARTIALLY_SUCCEEDED) {
+ getLogger().error("Operation status Partially succeeded - {}", result.toString());
+ isError = true;
+ }
Review Comment:
These conditional should be linked together with `else if` instead of individual conditions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] joewitt commented on pull request #7624: NIFI-11958 Azure Data Explorer as sink
Posted by "joewitt (via GitHub)" <gi...@apache.org>.
joewitt commented on PR #7624:
URL: https://github.com/apache/nifi/pull/7624#issuecomment-1719853104
@tanmaya-panda1 Hello. Can you please rename the processor to be more specific/clear to what it is doing. If we're sending to ADX a name such as 'PublishAzureDataExplorer' is more clear in intent as "IngestAzureDataExplorer" could mean we're getting data from it OR we're sending data to it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] NIFI-11958 Azure Data Explorer as sink [nifi]
Posted by "exceptionfactory (via GitHub)" <gi...@apache.org>.
exceptionfactory commented on code in PR #7624:
URL: https://github.com/apache/nifi/pull/7624#discussion_r1366908724
##########
nifi-nar-bundles/nifi-azure-bundle/nifi-azure-services-api/src/main/java/org/apache/nifi/services/azure/data/explorer/KustoIngestService.java:
##########
@@ -0,0 +1,28 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.services.azure.data.explorer;
+
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+
+import java.net.URISyntaxException;
+
+@Tags({"azure", "adx", "explorer", "kusto", "ingest"})
+@CapabilityDescription("Connection-Service for Azure ADX (Kusto) ingestion cluster to ingest data.")
+public interface KustoIngestService extends KustoQueryService {
Review Comment:
Thanks for clarifying, that makes sense.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] NIFI-11958 Azure Data Explorer as sink [nifi]
Posted by "exceptionfactory (via GitHub)" <gi...@apache.org>.
exceptionfactory commented on code in PR #7624:
URL: https://github.com/apache/nifi/pull/7624#discussion_r1364558334
##########
nifi-nar-bundles/nifi-azure-bundle/nifi-azure-processors/src/main/java/org/apache/nifi/processors/azure/data/explorer/PutAzureDataExplorer.java:
##########
@@ -0,0 +1,399 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.azure.data.explorer;
+
+import com.microsoft.azure.kusto.data.KustoResultSetTable;
+import com.microsoft.azure.kusto.ingest.IngestionMapping;
+import com.microsoft.azure.kusto.ingest.IngestionProperties;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.nifi.annotation.behavior.ReadsAttribute;
+import org.apache.nifi.annotation.behavior.ReadsAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.expression.ExpressionLanguageScope;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.services.azure.data.explorer.KustoIngestService;
+import org.apache.nifi.services.azure.data.explorer.KustoIngestionRequest;
+import org.apache.nifi.services.azure.data.explorer.KustoIngestionResult;
+import org.apache.nifi.services.azure.data.explorer.KustoQueryResponse;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.net.URISyntaxException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+
+@Tags({"Azure", "Kusto", "ADX", "Explorer", "Data"})
+@CapabilityDescription("The PutAzureDataExplorer acts as a ADX sink connector which sends flowFiles using the ADX-Service to the provided Azure Data" +
+ "Explorer Ingest Endpoint. The data can be sent through queued ingestion or streaming ingestion to the Azure Data Explorer cluster.")
+@ReadsAttributes({
+ @ReadsAttribute(attribute = "DB_NAME", description = "Specifies the name of the ADX database where the data needs to be stored."),
+ @ReadsAttribute(attribute = "TABLE_NAME", description = "Specifies the name of the ADX table where the data needs to be stored."),
+ @ReadsAttribute(attribute = "MAPPING_NAME", description = "Specifies the name of the mapping responsible for storing the data in appropriate columns."),
+ @ReadsAttribute(attribute = "FLUSH_IMMEDIATE", description = "In case of queued ingestion, this property determines whether the data should be flushed immediately to the ingest endpoint."),
+ @ReadsAttribute(attribute = "DATA_FORMAT", description = "Specifies the format of data that is send to Azure Data Explorer."),
+ @ReadsAttribute(attribute = "IGNORE_FIRST_RECORD", description = "Specifies whether we want to ignore ingestion of first record. " +
+ "This is primarily applicable for csv files. Default is set to NO"),
+ @ReadsAttribute(attribute = "POLL_ON_INGESTION_STATUS", description = "Specifies whether we want to poll on ingestion result during ingestion into ADX." +
+ "In case of applications that need high throughput it is recommended to keep the default value as false. Default is set to false." +
+ "This property should be set to true during Queued Ingestion for near realtime micro-batches of data that require acknowledgement of ingestion status.")
+})
+public class PutAzureDataExplorer extends AbstractProcessor {
+
+ public static final String FETCH_TABLE_COMMAND = "%s | count";
+ public static final String STREAMING_POLICY_SHOW_COMMAND = ".show %s %s policy streamingingestion";
+ public static final String DATABASE = "database";
+
+ private List<PropertyDescriptor> descriptors;
+ private Set<Relationship> relationships;
+ private transient KustoIngestService service;
+ private boolean isStreamingEnabled;
+ private boolean pollOnIngestionStatus;
+
+ public static final AllowableValue AVRO = new AllowableValue(
+ IngestionProperties.DataFormat.AVRO.getKustoValue(), IngestionProperties.DataFormat.AVRO.getKustoValue(),
+ "An Avro format with support for logical types and for the snappy compression codec");
+
+ public static final AllowableValue APACHEAVRO = new AllowableValue(
+ IngestionProperties.DataFormat.APACHEAVRO.getKustoValue(), IngestionProperties.DataFormat.APACHEAVRO.getKustoValue(),
+ "An Avro format with support for logical types and for the snappy compression codec.");
+
+ public static final AllowableValue CSV = new AllowableValue(
+ IngestionProperties.DataFormat.CSV.name(), IngestionProperties.DataFormat.CSV.getKustoValue(),
+ "A text file with comma-separated values (,). For more information, see RFC 4180: Common Format " +
+ "and MIME Type for Comma-Separated Values (CSV) Files.");
+
+ public static final AllowableValue JSON = new AllowableValue(
+ IngestionProperties.DataFormat.JSON.name(), IngestionProperties.DataFormat.JSON.getKustoValue(),
+ "A text file containing JSON objects separated by \\n or \\r\\n. For more information, " +
+ "see JSON Lines (JSONL).");
+
+ public static final AllowableValue MULTIJSON = new AllowableValue(
+ IngestionProperties.DataFormat.MULTIJSON.name(), IngestionProperties.DataFormat.MULTIJSON.getKustoValue(),
+ "A text file containing a JSON array of property containers (each representing a record) or any " +
+ "number of property containers separated by spaces, \\n or \\r\\n. Each property container may be " +
+ "spread across multiple lines. This format is preferable to JSON unless the data is not property " +
+ "containers.");
+
+ public static final AllowableValue ORC = new AllowableValue(
+ IngestionProperties.DataFormat.ORC.name(), IngestionProperties.DataFormat.ORC.getKustoValue(), "An ORC file.");
+
+ public static final AllowableValue PARQUET = new AllowableValue(
+ IngestionProperties.DataFormat.PARQUET.name(), IngestionProperties.DataFormat.PARQUET.getKustoValue(), "A parquet file.");
+
+ public static final AllowableValue PSV = new AllowableValue(
+ IngestionProperties.DataFormat.PSV.name(), IngestionProperties.DataFormat.PSV.getKustoValue(), "A text file with values separated by vertical bars (|).");
+
+ public static final AllowableValue SCSV = new AllowableValue(
+ IngestionProperties.DataFormat.SCSV.name(), IngestionProperties.DataFormat.SCSV.getKustoValue(), "A text file with values separated by semicolons (;).");
+
+ public static final AllowableValue SOHSV = new AllowableValue(
+ IngestionProperties.DataFormat.SOHSV.name(), IngestionProperties.DataFormat.SOHSV.getKustoValue(),
+ "A text file with SOH-separated values. (SOH is the ASCII code point 1. " +
+ "This format is used by Hive in HDInsight).");
+
+ public static final AllowableValue TSV = new AllowableValue(
+ IngestionProperties.DataFormat.TSV.name(), IngestionProperties.DataFormat.TSV.getKustoValue(), "A text file with tab delimited values (\\t).");
+
+ public static final AllowableValue TSVE = new AllowableValue(
+ IngestionProperties.DataFormat.TSVE.name(), IngestionProperties.DataFormat.TSVE.getKustoValue(),
+ "A text file with tab-delimited values (\\t). A backslash (\\) is used as escape character.");
+
+ public static final AllowableValue TXT = new AllowableValue(
+ IngestionProperties.DataFormat.TXT.name(), IngestionProperties.DataFormat.TXT.getKustoValue(),
+ "A text file with lines separated by \\n. Empty lines are skipped.");
+
+ public static final AllowableValue IGNORE_FIRST_RECORD_YES = new AllowableValue(
+ "YES", "YES",
+ "Ignore first record during ingestion");
+
+ public static final AllowableValue IGNORE_FIRST_RECORD_NO = new AllowableValue(
+ "NO", "NO",
+ "Do not ignore first record during ingestion");
+
+ public static final PropertyDescriptor DATABASE_NAME = new PropertyDescriptor.Builder()
+ .name("Database Name")
+ .displayName("Database Name")
+ .description("Azure Data Explorer Database Name for querying")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+ .build();
+
+ public static final PropertyDescriptor TABLE_NAME = new PropertyDescriptor.Builder()
+ .name("Table Name")
+ .displayName("Table Name")
+ .description("Azure Data Explorer Table Name for ingesting data")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+ .build();
+
+ public static final PropertyDescriptor MAPPING_NAME = new PropertyDescriptor
+ .Builder().name("Mapping name")
+ .displayName("Mapping name")
+ .description("The name of the mapping responsible for storing the data in the appropriate columns.")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+
+ public static final PropertyDescriptor IS_STREAMING_ENABLED = new PropertyDescriptor
+ .Builder().name("Is Streaming enabled")
+ .displayName("Is Streaming enabled")
+ .description("This property determines whether we want to stream data to ADX.")
+ .required(false)
+ .allowableValues("true", "false")
+ .addValidator(StandardValidators.BOOLEAN_VALIDATOR)
+ .defaultValue("false")
+ .build();
+
+ public static final PropertyDescriptor POLL_ON_INGESTION_STATUS = new PropertyDescriptor
+ .Builder().name("Poll on ingestion status")
+ .displayName("Whether to Poll on ingestion status")
+ .description("This property determines whether we want to poll on ingestion status after an ingestion to ADX is completed")
+ .required(false)
+ .allowableValues("true", "false")
+ .addValidator(StandardValidators.BOOLEAN_VALIDATOR)
+ .defaultValue("false")
+ .build();
+ public static final PropertyDescriptor ADX_SERVICE = new PropertyDescriptor
+ .Builder().name("Kusto Ingest Service")
+ .displayName("Kusto Ingest Service")
+ .description("Azure Data Explorer Kusto Ingest Service")
+ .required(true)
+ .identifiesControllerService(KustoIngestService.class)
+ .build();
+
+ public static final Relationship SUCCESS = new Relationship.Builder()
+ .name("success")
+ .description("Relationship for success")
+ .build();
+ public static final Relationship FAILURE = new Relationship.Builder()
+ .name("failure")
+ .description("Relationship for failure")
+ .build();
+
+ static final PropertyDescriptor DATA_FORMAT = new PropertyDescriptor.Builder()
+ .name("Data Format")
+ .displayName("Data Format")
+ .description("The format of the data that is sent to ADX.")
+ .required(true)
+ .allowableValues(AVRO, APACHEAVRO, CSV, JSON, MULTIJSON, ORC, PARQUET, PSV, SCSV, SOHSV, TSV, TSVE, TXT)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+
+ static final PropertyDescriptor IGNORE_FIRST_RECORD = new PropertyDescriptor.Builder()
+ .name("Ingestion Ignore First Record")
+ .displayName("Ingestion Ignore First Record")
+ .description("Defines whether ignore first record while ingestion.")
+ .required(false)
+ .allowableValues(IGNORE_FIRST_RECORD_YES, IGNORE_FIRST_RECORD_NO)
+ .defaultValue(IGNORE_FIRST_RECORD_NO.getValue())
+ .build();
+
+ @Override
+ protected void init(final ProcessorInitializationContext context) {
+ final List<PropertyDescriptor> descriptorList = new ArrayList<>();
+ descriptorList.add(ADX_SERVICE);
+ descriptorList.add(DATABASE_NAME);
+ descriptorList.add(TABLE_NAME);
+ descriptorList.add(MAPPING_NAME);
+ descriptorList.add(DATA_FORMAT);
+ descriptorList.add(IGNORE_FIRST_RECORD);
+ descriptorList.add(IS_STREAMING_ENABLED);
+ descriptorList.add(POLL_ON_INGESTION_STATUS);
+ this.descriptors = Collections.unmodifiableList(descriptorList);
+
+ final Set<Relationship> relationshipSet = new HashSet<>();
+ relationshipSet.add(SUCCESS);
+ relationshipSet.add(FAILURE);
+ this.relationships = Collections.unmodifiableSet(relationshipSet);
+ }
+
+ @Override
+ public Set<Relationship> getRelationships() {
+ return this.relationships;
+ }
+
+ @Override
+ public final List<PropertyDescriptor> getSupportedPropertyDescriptors() {
+ return descriptors;
+ }
+
+ @OnScheduled
+ public void onScheduled(final ProcessContext context) {
+ service = context.getProperty(ADX_SERVICE).asControllerService(KustoIngestService.class);
+ if (checkIfIngestorRoleDoesntExist(context.getProperty(DATABASE_NAME).getValue(), context.getProperty(TABLE_NAME).getValue())) {
+ getLogger().error("User might not have ingestor privileges, table validation will be skipped for all table mappings.");
+ throw new ProcessException("User might not have ingestor privileges, table validation will be skipped for all table mappings. ");
+ }
+ isStreamingEnabled = context.getProperty(IS_STREAMING_ENABLED).evaluateAttributeExpressions().asBoolean();
+ if (isStreamingEnabled) {
+ checkIfStreamingPolicyIsEnabledInADX(context.getProperty(DATABASE_NAME).getValue(), context.getProperty(DATABASE_NAME).getValue());
+ }
+ }
+
+ @Override
+ public void onTrigger(final ProcessContext context, final ProcessSession session) throws ProcessException {
+ FlowFile flowFile = session.get();
+ if (flowFile == null) {
+ context.yield();
+ return;
+ }
+
+ IngestionProperties ingestionProperties = new IngestionProperties(context.getProperty(DATABASE_NAME).getValue(),
+ context.getProperty(TABLE_NAME).getValue());
+
+ IngestionMapping.IngestionMappingKind ingestionMappingKind = null;
+
+ isStreamingEnabled = context.getProperty(IS_STREAMING_ENABLED).evaluateAttributeExpressions().asBoolean();
+ pollOnIngestionStatus = context.getProperty(POLL_ON_INGESTION_STATUS).evaluateAttributeExpressions().asBoolean();
+
+ switch (IngestionProperties.DataFormat.valueOf(context.getProperty(DATA_FORMAT).getValue())) {
+ case AVRO:
+ ingestionProperties.setDataFormat(IngestionProperties.DataFormat.AVRO);
+ ingestionMappingKind = IngestionProperties.DataFormat.AVRO.getIngestionMappingKind();
+ break;
+ case APACHEAVRO:
+ ingestionProperties.setDataFormat(IngestionProperties.DataFormat.APACHEAVRO);
+ ingestionMappingKind = IngestionProperties.DataFormat.APACHEAVRO.getIngestionMappingKind();
+ break;
+ case CSV:
+ ingestionProperties.setDataFormat(IngestionProperties.DataFormat.CSV);
+ ingestionMappingKind = IngestionProperties.DataFormat.CSV.getIngestionMappingKind();
+ break;
+ case JSON:
+ ingestionProperties.setDataFormat(IngestionProperties.DataFormat.JSON);
+ ingestionMappingKind = IngestionProperties.DataFormat.JSON.getIngestionMappingKind();
+ break;
+ case MULTIJSON:
+ ingestionProperties.setDataFormat(IngestionProperties.DataFormat.MULTIJSON);
+ ingestionMappingKind = IngestionProperties.DataFormat.MULTIJSON.getIngestionMappingKind();
+ break;
+ case ORC:
+ ingestionProperties.setDataFormat(IngestionProperties.DataFormat.ORC);
+ ingestionMappingKind = IngestionProperties.DataFormat.ORC.getIngestionMappingKind();
+ break;
+ case PARQUET:
+ ingestionProperties.setDataFormat(IngestionProperties.DataFormat.PARQUET);
+ ingestionMappingKind = IngestionProperties.DataFormat.PARQUET.getIngestionMappingKind();
+ break;
+ case PSV:
+ ingestionProperties.setDataFormat(IngestionProperties.DataFormat.PSV);
+ ingestionMappingKind = IngestionProperties.DataFormat.PSV.getIngestionMappingKind();
+ break;
+ case SCSV:
+ ingestionProperties.setDataFormat(IngestionProperties.DataFormat.SCSV);
+ ingestionMappingKind = IngestionProperties.DataFormat.SCSV.getIngestionMappingKind();
+ break;
+ case SOHSV:
+ ingestionProperties.setDataFormat(IngestionProperties.DataFormat.SOHSV);
+ ingestionMappingKind = IngestionProperties.DataFormat.SOHSV.getIngestionMappingKind();
+ break;
+ case TSV:
+ ingestionProperties.setDataFormat(IngestionProperties.DataFormat.TSV);
+ ingestionMappingKind = IngestionProperties.DataFormat.TSV.getIngestionMappingKind();
+ break;
+ case TSVE:
+ ingestionProperties.setDataFormat(IngestionProperties.DataFormat.TSVE);
+ ingestionMappingKind = IngestionProperties.DataFormat.TSVE.getIngestionMappingKind();
+ break;
+ case TXT:
+ ingestionProperties.setDataFormat(IngestionProperties.DataFormat.TXT);
+ ingestionMappingKind = IngestionProperties.DataFormat.TXT.getIngestionMappingKind();
+ break;
+ }
+
+ if (StringUtils.isNotEmpty(context.getProperty(MAPPING_NAME).getValue()) && ingestionMappingKind != null) {
+ ingestionProperties.setIngestionMapping(context.getProperty(MAPPING_NAME).getValue(), ingestionMappingKind);
+ }
+
+ ingestionProperties.setReportLevel(IngestionProperties.IngestionReportLevel.FAILURES_AND_SUCCESSES);
+ ingestionProperties.setReportMethod(IngestionProperties.IngestionReportMethod.TABLE);
+ ingestionProperties.setFlushImmediately(false);
+ ingestionProperties.setIgnoreFirstRecord(StringUtils.equalsIgnoreCase(context.getProperty(IGNORE_FIRST_RECORD).getValue(), IGNORE_FIRST_RECORD_YES.getValue()));
+
+ boolean isError = false;
+
+ // ingestion into ADX
+ try (final InputStream inputStream = session.read(flowFile)) {
+ StringBuilder ingestLogString = new StringBuilder().append("Ingesting with: ")
+ .append("dataFormat - ").append(ingestionProperties.getDataFormat()).append("|")
+ .append("ingestionMapping - ").append(ingestionProperties.getIngestionMapping().getIngestionMappingReference()).append("|")
+ .append("reportLevel - ").append(ingestionProperties.getReportLevel()).append("|")
+ .append("reportMethod - ").append(ingestionProperties.getReportMethod()).append("|")
+ .append("databaseName - ").append(ingestionProperties.getDatabaseName()).append("|")
+ .append("tableName - ").append(ingestionProperties.getTableName()).append("|")
+ .append("flushImmediately - ").append(ingestionProperties.getFlushImmediately());
+ getLogger().info(ingestLogString.toString());
+
+ KustoIngestionResult result = service.ingestData(new KustoIngestionRequest(isStreamingEnabled, pollOnIngestionStatus, inputStream, ingestionProperties));
+ getLogger().info("Poll on Ingestion Status {} and Operation status: {} ", pollOnIngestionStatus, result.toString());
+
+ if (result == KustoIngestionResult.SUCCEEDED) {
+ getLogger().info("Operation status Succeeded - {}", result.toString());
+ } else if (result == KustoIngestionResult.FAILED) {
+ getLogger().error("Operation status Error - {}", result.toString());
+ isError = true;
+ } else if (result == KustoIngestionResult.PARTIALLY_SUCCEEDED) {
+ getLogger().error("Operation status Partially succeeded - {}", result.toString());
Review Comment:
Thanks for flagging this scenario @tanmaya-panda1. What would the user be expected to do in this scenario? The problem seems to be knowing what was ingested and what was not ingested. One option might be to introduce a configurable property for `Partial Success Routing`, with values of `success` or `failure`. This would let the user decide on a high-level strategy. In addition, the status should be noted in a FlowFile attribute, which would support routing decisions that could include retrying.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] NIFI-11958 Azure Data Explorer as sink [nifi]
Posted by "kustonaut (via GitHub)" <gi...@apache.org>.
kustonaut commented on PR #7624:
URL: https://github.com/apache/nifi/pull/7624#issuecomment-1766312186
@joewitt & @exceptionfactory - Thank you so much for your review and comments. @tanmaya-panda1 has fixed the issues. Could you please take a look when feasible?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] NIFI-11958 Azure Data Explorer as sink [nifi]
Posted by "tanmaya-panda1 (via GitHub)" <gi...@apache.org>.
tanmaya-panda1 commented on code in PR #7624:
URL: https://github.com/apache/nifi/pull/7624#discussion_r1381171499
##########
nifi-nar-bundles/nifi-azure-bundle/nifi-azure-processors/src/main/java/org/apache/nifi/processors/azure/data/explorer/PutAzureDataExplorer.java:
##########
@@ -0,0 +1,309 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.azure.data.explorer;
+
+import org.apache.commons.lang3.StringUtils;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.expression.ExpressionLanguageScope;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.services.azure.data.explorer.KustoIngestDataFormat;
+import org.apache.nifi.services.azure.data.explorer.KustoIngestService;
+import org.apache.nifi.services.azure.data.explorer.KustoIngestionRequest;
+import org.apache.nifi.services.azure.data.explorer.KustoIngestionResult;
+import org.apache.nifi.services.azure.data.explorer.KustoQueryResponse;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.net.URISyntaxException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+
+@Tags({"Azure", "Kusto", "ADX", "Explorer", "Data"})
+@CapabilityDescription("The PutAzureDataExplorer acts as a ADX sink connector which sends flowFiles using the ADX-Service to the provided Azure Data" +
+ "Explorer Ingest Endpoint. The data can be sent through queued ingestion or streaming ingestion to the Azure Data Explorer cluster.")
+public class PutAzureDataExplorer extends AbstractProcessor {
+
+ public static final String FETCH_TABLE_COMMAND = "%s | count";
+ public static final String STREAMING_POLICY_SHOW_COMMAND = ".show %s %s policy streamingingestion";
+ public static final String DATABASE = "database";
+
+ private List<PropertyDescriptor> descriptors;
+ private Set<Relationship> relationships;
+ private transient KustoIngestService service;
+ private boolean streamingEnabled;
+ private boolean pollOnIngestionStatus;
+
+ public static final AllowableValue IGNORE_FIRST_RECORD_YES = new AllowableValue(
+ "YES", "YES",
+ "Ignore first record during ingestion");
+
+ public static final AllowableValue IGNORE_FIRST_RECORD_NO = new AllowableValue(
+ "NO", "NO",
+ "Do not ignore first record during ingestion");
+
+ public static final PropertyDescriptor DATABASE_NAME = new PropertyDescriptor.Builder()
+ .name("Database Name")
+ .displayName("Database Name")
+ .description("Azure Data Explorer Database Name for querying")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+ .build();
+
+ public static final PropertyDescriptor TABLE_NAME = new PropertyDescriptor.Builder()
+ .name("Table Name")
+ .displayName("Table Name")
+ .description("Azure Data Explorer Table Name for ingesting data")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+ .build();
+
+ public static final PropertyDescriptor MAPPING_NAME = new PropertyDescriptor
+ .Builder().name("Ingest Mapping name")
+ .displayName("Ingest Mapping name")
+ .description("The name of the mapping responsible for storing the data in the appropriate columns.")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+
+ public static final PropertyDescriptor IS_STREAMING_ENABLED = new PropertyDescriptor
+ .Builder().name("Streaming enabled")
+ .displayName("Streaming enabled")
+ .description("This property determines whether we want to stream data to ADX.")
+ .required(false)
+ .allowableValues("true", "false")
+ .addValidator(StandardValidators.BOOLEAN_VALIDATOR)
+ .defaultValue("false")
+ .build();
+
+ public static final PropertyDescriptor POLL_ON_INGESTION_STATUS = new PropertyDescriptor
+ .Builder().name("Poll on ingestion status")
+ .displayName("Whether to Poll on ingestion status")
+ .description("This property determines whether we want to poll on ingestion status after an ingestion to ADX is completed")
+ .required(false)
+ .allowableValues("true", "false")
+ .addValidator(StandardValidators.BOOLEAN_VALIDATOR)
+ .defaultValue("false")
+ .build();
+ public static final PropertyDescriptor ADX_SERVICE = new PropertyDescriptor
+ .Builder().name("Kusto Ingest Service")
+ .displayName("Kusto Ingest Service")
+ .description("Azure Data Explorer Kusto Ingest Service")
+ .required(true)
+ .identifiesControllerService(KustoIngestService.class)
+ .build();
+
+ static final PropertyDescriptor DATA_FORMAT = new PropertyDescriptor.Builder()
+ .name("Data Format")
+ .displayName("Data Format")
+ .description("The format of the data that is sent to Azure Data Explorer.")
+ .required(true)
+ .allowableValues(KustoIngestDataFormat.values())
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+
+ static final PropertyDescriptor IGNORE_FIRST_RECORD = new PropertyDescriptor.Builder()
+ .name("Ingestion Ignore First Record")
+ .displayName("Ingestion Ignore First Record")
+ .description("Defines whether ignore first record while ingestion.")
+ .required(false)
+ .allowableValues(IGNORE_FIRST_RECORD_YES, IGNORE_FIRST_RECORD_NO)
+ .defaultValue(IGNORE_FIRST_RECORD_NO.getValue())
+ .build();
+
+ static final PropertyDescriptor ROUTE_PARTIALLY_SUCCESSFUL_INGESTION = new PropertyDescriptor.Builder()
+ .name("Route partially successful ingestion records")
+ .displayName("Route partially successful ingestion records")
+ .description("Defines where to route partially successful ingestion records.")
+ .required(false)
+ .allowableValues("Success", "Failure")
+ .defaultValue("Failure")
+ .build();
+
+ static final PropertyDescriptor INGESTION_STATUS_POLLING_TIMEOUT = new PropertyDescriptor.Builder()
+ .name("Timeout for polling on ingestion status")
+ .displayName("Timeout for polling on ingestion status in seconds")
+ .description("Defines the value of timeout for polling on ingestion status in seconds")
+ .required(false)
+ .dependsOn(POLL_ON_INGESTION_STATUS, "true")
+ .addValidator(StandardValidators.POSITIVE_INTEGER_VALIDATOR)
+ .defaultValue("600")
+ .build();
+
+ static final PropertyDescriptor INGESTION_STATUS_POLLING_INTERVAL = new PropertyDescriptor.Builder()
+ .name("Ingestion status polling interval")
+ .displayName("Ingestion status polling interval in seconds")
+ .description("Defines the value of timeout for polling on ingestion status in seconds.")
+ .required(false)
+ .dependsOn(POLL_ON_INGESTION_STATUS, "true")
+ .addValidator(StandardValidators.POSITIVE_INTEGER_VALIDATOR)
+ .defaultValue("5")
+ .build();
+
+ public static final Relationship SUCCESS = new Relationship.Builder()
+ .name("success")
+ .description("Relationship for success")
+ .build();
+ public static final Relationship FAILURE = new Relationship.Builder()
+ .name("failure")
+ .description("Relationship for failure")
+ .build();
+
+ @Override
+ protected void init(final ProcessorInitializationContext context) {
+ final List<PropertyDescriptor> descriptorList = new ArrayList<>();
+ descriptorList.add(ADX_SERVICE);
+ descriptorList.add(DATABASE_NAME);
+ descriptorList.add(TABLE_NAME);
+ descriptorList.add(MAPPING_NAME);
+ descriptorList.add(DATA_FORMAT);
+ descriptorList.add(IGNORE_FIRST_RECORD);
+ descriptorList.add(IS_STREAMING_ENABLED);
+ descriptorList.add(POLL_ON_INGESTION_STATUS);
+ descriptorList.add(ROUTE_PARTIALLY_SUCCESSFUL_INGESTION);
+ descriptorList.add(INGESTION_STATUS_POLLING_TIMEOUT);
+ descriptorList.add(INGESTION_STATUS_POLLING_INTERVAL);
+ this.descriptors = Collections.unmodifiableList(descriptorList);
+
+ final Set<Relationship> relationshipSet = new HashSet<>();
+ relationshipSet.add(SUCCESS);
+ relationshipSet.add(FAILURE);
+ this.relationships = Collections.unmodifiableSet(relationshipSet);
+ }
+
+ @Override
+ public Set<Relationship> getRelationships() {
+ return this.relationships;
+ }
+
+ @Override
+ public final List<PropertyDescriptor> getSupportedPropertyDescriptors() {
+ return descriptors;
+ }
+
+ @OnScheduled
+ public void onScheduled(final ProcessContext context) {
+ service = context.getProperty(ADX_SERVICE).asControllerService(KustoIngestService.class);
+ if (checkIfIngestorRoleDoesntExist(context.getProperty(DATABASE_NAME).getValue(), context.getProperty(TABLE_NAME).getValue())) {
+ getLogger().error("User might not have ingestor privileges, table validation will be skipped for all table mappings.");
+ throw new ProcessException("User might not have ingestor privileges, table validation will be skipped for all table mappings. ");
Review Comment:
Since without ingestor privilege, there is no point in starting the ingestion of data, hence we throw an exception. Also removed skipping of table validation message which might confuse the user.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] NIFI-11958 Azure Data Explorer as sink [nifi]
Posted by "tanmaya-panda1 (via GitHub)" <gi...@apache.org>.
tanmaya-panda1 commented on code in PR #7624:
URL: https://github.com/apache/nifi/pull/7624#discussion_r1381215021
##########
nifi-nar-bundles/nifi-azure-bundle/nifi-azure-processors/src/main/java/org/apache/nifi/processors/azure/data/explorer/PutAzureDataExplorer.java:
##########
@@ -0,0 +1,309 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.azure.data.explorer;
+
+import org.apache.commons.lang3.StringUtils;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.expression.ExpressionLanguageScope;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.services.azure.data.explorer.KustoIngestDataFormat;
+import org.apache.nifi.services.azure.data.explorer.KustoIngestService;
+import org.apache.nifi.services.azure.data.explorer.KustoIngestionRequest;
+import org.apache.nifi.services.azure.data.explorer.KustoIngestionResult;
+import org.apache.nifi.services.azure.data.explorer.KustoQueryResponse;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.net.URISyntaxException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+
+@Tags({"Azure", "Kusto", "ADX", "Explorer", "Data"})
+@CapabilityDescription("The PutAzureDataExplorer acts as a ADX sink connector which sends flowFiles using the ADX-Service to the provided Azure Data" +
+ "Explorer Ingest Endpoint. The data can be sent through queued ingestion or streaming ingestion to the Azure Data Explorer cluster.")
+public class PutAzureDataExplorer extends AbstractProcessor {
+
+ public static final String FETCH_TABLE_COMMAND = "%s | count";
+ public static final String STREAMING_POLICY_SHOW_COMMAND = ".show %s %s policy streamingingestion";
+ public static final String DATABASE = "database";
+
+ private List<PropertyDescriptor> descriptors;
+ private Set<Relationship> relationships;
+ private transient KustoIngestService service;
+ private boolean streamingEnabled;
+ private boolean pollOnIngestionStatus;
+
+ public static final AllowableValue IGNORE_FIRST_RECORD_YES = new AllowableValue(
+ "YES", "YES",
+ "Ignore first record during ingestion");
+
+ public static final AllowableValue IGNORE_FIRST_RECORD_NO = new AllowableValue(
+ "NO", "NO",
+ "Do not ignore first record during ingestion");
+
+ public static final PropertyDescriptor DATABASE_NAME = new PropertyDescriptor.Builder()
+ .name("Database Name")
+ .displayName("Database Name")
+ .description("Azure Data Explorer Database Name for querying")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+ .build();
+
+ public static final PropertyDescriptor TABLE_NAME = new PropertyDescriptor.Builder()
+ .name("Table Name")
+ .displayName("Table Name")
+ .description("Azure Data Explorer Table Name for ingesting data")
+ .required(true)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+ .build();
+
+ public static final PropertyDescriptor MAPPING_NAME = new PropertyDescriptor
+ .Builder().name("Ingest Mapping name")
+ .displayName("Ingest Mapping name")
+ .description("The name of the mapping responsible for storing the data in the appropriate columns.")
+ .required(false)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+
+ public static final PropertyDescriptor IS_STREAMING_ENABLED = new PropertyDescriptor
+ .Builder().name("Streaming enabled")
+ .displayName("Streaming enabled")
+ .description("This property determines whether we want to stream data to ADX.")
+ .required(false)
+ .allowableValues("true", "false")
+ .addValidator(StandardValidators.BOOLEAN_VALIDATOR)
+ .defaultValue("false")
+ .build();
+
+ public static final PropertyDescriptor POLL_ON_INGESTION_STATUS = new PropertyDescriptor
+ .Builder().name("Poll on ingestion status")
+ .displayName("Whether to Poll on ingestion status")
+ .description("This property determines whether we want to poll on ingestion status after an ingestion to ADX is completed")
+ .required(false)
+ .allowableValues("true", "false")
+ .addValidator(StandardValidators.BOOLEAN_VALIDATOR)
+ .defaultValue("false")
+ .build();
+ public static final PropertyDescriptor ADX_SERVICE = new PropertyDescriptor
+ .Builder().name("Kusto Ingest Service")
+ .displayName("Kusto Ingest Service")
+ .description("Azure Data Explorer Kusto Ingest Service")
+ .required(true)
+ .identifiesControllerService(KustoIngestService.class)
+ .build();
+
+ static final PropertyDescriptor DATA_FORMAT = new PropertyDescriptor.Builder()
+ .name("Data Format")
+ .displayName("Data Format")
+ .description("The format of the data that is sent to Azure Data Explorer.")
+ .required(true)
+ .allowableValues(KustoIngestDataFormat.values())
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+ .build();
+
+ static final PropertyDescriptor IGNORE_FIRST_RECORD = new PropertyDescriptor.Builder()
+ .name("Ingestion Ignore First Record")
+ .displayName("Ingestion Ignore First Record")
+ .description("Defines whether ignore first record while ingestion.")
+ .required(false)
+ .allowableValues(IGNORE_FIRST_RECORD_YES, IGNORE_FIRST_RECORD_NO)
+ .defaultValue(IGNORE_FIRST_RECORD_NO.getValue())
+ .build();
+
+ static final PropertyDescriptor ROUTE_PARTIALLY_SUCCESSFUL_INGESTION = new PropertyDescriptor.Builder()
+ .name("Route partially successful ingestion records")
+ .displayName("Route partially successful ingestion records")
+ .description("Defines where to route partially successful ingestion records.")
+ .required(false)
+ .allowableValues("Success", "Failure")
+ .defaultValue("Failure")
+ .build();
+
+ static final PropertyDescriptor INGESTION_STATUS_POLLING_TIMEOUT = new PropertyDescriptor.Builder()
+ .name("Timeout for polling on ingestion status")
+ .displayName("Timeout for polling on ingestion status in seconds")
+ .description("Defines the value of timeout for polling on ingestion status in seconds")
+ .required(false)
+ .dependsOn(POLL_ON_INGESTION_STATUS, "true")
+ .addValidator(StandardValidators.POSITIVE_INTEGER_VALIDATOR)
+ .defaultValue("600")
+ .build();
+
+ static final PropertyDescriptor INGESTION_STATUS_POLLING_INTERVAL = new PropertyDescriptor.Builder()
+ .name("Ingestion status polling interval")
+ .displayName("Ingestion status polling interval in seconds")
+ .description("Defines the value of timeout for polling on ingestion status in seconds.")
+ .required(false)
+ .dependsOn(POLL_ON_INGESTION_STATUS, "true")
+ .addValidator(StandardValidators.POSITIVE_INTEGER_VALIDATOR)
+ .defaultValue("5")
+ .build();
+
+ public static final Relationship SUCCESS = new Relationship.Builder()
+ .name("success")
+ .description("Relationship for success")
+ .build();
+ public static final Relationship FAILURE = new Relationship.Builder()
+ .name("failure")
+ .description("Relationship for failure")
+ .build();
+
+ @Override
+ protected void init(final ProcessorInitializationContext context) {
+ final List<PropertyDescriptor> descriptorList = new ArrayList<>();
+ descriptorList.add(ADX_SERVICE);
+ descriptorList.add(DATABASE_NAME);
+ descriptorList.add(TABLE_NAME);
+ descriptorList.add(MAPPING_NAME);
+ descriptorList.add(DATA_FORMAT);
+ descriptorList.add(IGNORE_FIRST_RECORD);
+ descriptorList.add(IS_STREAMING_ENABLED);
+ descriptorList.add(POLL_ON_INGESTION_STATUS);
+ descriptorList.add(ROUTE_PARTIALLY_SUCCESSFUL_INGESTION);
+ descriptorList.add(INGESTION_STATUS_POLLING_TIMEOUT);
+ descriptorList.add(INGESTION_STATUS_POLLING_INTERVAL);
+ this.descriptors = Collections.unmodifiableList(descriptorList);
+
+ final Set<Relationship> relationshipSet = new HashSet<>();
+ relationshipSet.add(SUCCESS);
+ relationshipSet.add(FAILURE);
+ this.relationships = Collections.unmodifiableSet(relationshipSet);
+ }
+
+ @Override
+ public Set<Relationship> getRelationships() {
+ return this.relationships;
+ }
+
+ @Override
+ public final List<PropertyDescriptor> getSupportedPropertyDescriptors() {
+ return descriptors;
+ }
+
+ @OnScheduled
+ public void onScheduled(final ProcessContext context) {
+ service = context.getProperty(ADX_SERVICE).asControllerService(KustoIngestService.class);
+ if (checkIfIngestorRoleDoesntExist(context.getProperty(DATABASE_NAME).getValue(), context.getProperty(TABLE_NAME).getValue())) {
+ getLogger().error("User might not have ingestor privileges, table validation will be skipped for all table mappings.");
+ throw new ProcessException("User might not have ingestor privileges, table validation will be skipped for all table mappings. ");
+ }
+ streamingEnabled = context.getProperty(IS_STREAMING_ENABLED).evaluateAttributeExpressions().asBoolean();
+ if (streamingEnabled && !checkIfStreamingPolicyIsEnabledInADX(context.getProperty(DATABASE_NAME).getValue(), context.getProperty(DATABASE_NAME).getValue())) {
+ getLogger().error("Streaming policy is not enabled in database {}", context.getProperty(DATABASE_NAME).getValue());
+ throw new ProcessException("Streaming policy is not enabled in database {}" + context.getProperty(DATABASE_NAME).getValue());
+ }
+ }
+
+ @Override
+ public void onTrigger(final ProcessContext context, final ProcessSession session) throws ProcessException {
+ FlowFile flowFile = session.get();
+ if (flowFile == null) {
+ return;
+ }
+
+ String databaseName = context.getProperty(DATABASE_NAME).getValue();
+ String tableName = context.getProperty(TABLE_NAME).getValue();
+ String dataFormat = context.getProperty(DATA_FORMAT).getValue();
+ String mappingName = context.getProperty(MAPPING_NAME).getValue();
+ String routePartiallySuccessfulIngestion = context.getProperty(ROUTE_PARTIALLY_SUCCESSFUL_INGESTION).getValue();
+ String ingestionStatusPollingTimeout = context.getProperty(INGESTION_STATUS_POLLING_TIMEOUT).getValue();
+ String ingestionStatusPollingInterval = context.getProperty(INGESTION_STATUS_POLLING_INTERVAL).getValue();
+ String ignoreFirstRecord = context.getProperty(IGNORE_FIRST_RECORD).getValue();
+ streamingEnabled = context.getProperty(IS_STREAMING_ENABLED).evaluateAttributeExpressions().asBoolean();
+ pollOnIngestionStatus = context.getProperty(POLL_ON_INGESTION_STATUS).evaluateAttributeExpressions().asBoolean();
+
+ boolean isError = false;
+
+ // ingestion into ADX
+ try (final InputStream inputStream = session.read(flowFile)) {
+ StringBuilder ingestLogString = new StringBuilder().append("Ingesting with: ")
+ .append("dataFormat - ").append(dataFormat).append("|")
+ .append("ingestionMappingName - ").append(mappingName).append("|")
+ .append("databaseName - ").append(databaseName).append("|")
+ .append("tableName - ").append(tableName).append("|")
+ .append("pollOnIngestionStatus - ").append(pollOnIngestionStatus).append("|")
+ .append("ingestionStatusPollingTimeout - ").append(ingestionStatusPollingTimeout).append("|")
+ .append("ingestionStatusPollingInterval - ").append(ingestionStatusPollingInterval).append("|")
+ .append("routePartiallySuccessfulIngestion - ").append(routePartiallySuccessfulIngestion);
+ getLogger().debug(ingestLogString.toString());
+
+ KustoIngestionResult result = service.ingestData(
+ new KustoIngestionRequest(streamingEnabled, pollOnIngestionStatus, inputStream, databaseName,
+ tableName, dataFormat, mappingName, ignoreFirstRecord, ingestionStatusPollingTimeout, ingestionStatusPollingInterval)
+ );
+ getLogger().info("Ingest Status Polling Enabled {} and Ingest Status Polling Enabled {} ", pollOnIngestionStatus, result.toString());
+
+ if (result == KustoIngestionResult.SUCCEEDED) {
+ getLogger().info("Ingest Completed - {}", result.getStatus());
+ } else if (result == KustoIngestionResult.FAILED) {
+ getLogger().error("Ingest Failed - {}", result.getStatus());
+ isError = true;
+ } else if (result == KustoIngestionResult.PARTIALLY_SUCCEEDED) {
+ getLogger().warn("Ingest Partially succeeded - {}", result.getStatus());
+ flowFile = session.putAttribute(flowFile, "ingestion_status", "partial_success");
+ if (StringUtils.equalsIgnoreCase(routePartiallySuccessfulIngestion, "Failure")) {
+ isError = true;
+ }
+ }
+
+ } catch (IOException | URISyntaxException e) {
+ getLogger().error("Azure Data Explorer Ingest processing failed", e);
+ isError = true;
Review Comment:
removed isError variable
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] NIFI-11958 Azure Data Explorer as sink [nifi]
Posted by "exceptionfactory (via GitHub)" <gi...@apache.org>.
exceptionfactory closed pull request #7624: NIFI-11958 Azure Data Explorer as sink
URL: https://github.com/apache/nifi/pull/7624
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] NIFI-11958 Azure Data Explorer as sink [nifi]
Posted by "tanmaya-panda1 (via GitHub)" <gi...@apache.org>.
tanmaya-panda1 commented on PR #7624:
URL: https://github.com/apache/nifi/pull/7624#issuecomment-1837866306
@exceptionfactory : Can you please take a look at this PR. This is long pending item, we wanted to close this before the end of this year. We would appreciate your review on this item whenever you have some time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] tanmaya-panda1 commented on pull request #7624: NIFI-11958 Azure Data Explorer as sink
Posted by "tanmaya-panda1 (via GitHub)" <gi...@apache.org>.
tanmaya-panda1 commented on PR #7624:
URL: https://github.com/apache/nifi/pull/7624#issuecomment-1700347242
@exceptionfactory : Can you please take a look at this PR for Azure Data Explorer acting as sink.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org