You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@nifi.apache.org by "Joe Witt (Jira)" <ji...@apache.org> on 2021/07/06 17:40:00 UTC
[jira] [Updated] (NIFI-8611) GCP BigQuery processors support using
designate project resource for ingestion
[ https://issues.apache.org/jira/browse/NIFI-8611?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Joe Witt updated NIFI-8611:
---------------------------
Fix Version/s: (was: 1.14.0)
> GCP BigQuery processors support using designate project resource for ingestion
> ------------------------------------------------------------------------------
>
> Key: NIFI-8611
> URL: https://issues.apache.org/jira/browse/NIFI-8611
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Extensions
> Affects Versions: 1.11.4
> Reporter: Chih Han Yu
> Assignee: Joe Witt
> Priority: Major
> Labels: GCP, bigquery
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> For now, *PutBigQueryBatch* processor and *PutBigQueryStreaming* processor can only assign a single project id for consuming resources and do ingestion. But in some business cases, the project providing resources and the project which be inserted are not always the same.
> src/main/java/org/apache/nifi/processors/gcp/AbstractGCPProcessor.java
>
> {code:java}
> ......
> public static final PropertyDescriptor PROJECT_ID = new PropertyDescriptor
> .Builder().name("gcp-project-id")
> .displayName("Project ID")
> .description("Google Cloud Project ID")
> .required(false)
> .expressionLanguageSupported(ExpressionLanguageScope.VARIABLE_REGISTRY)
> .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
> .build();
> ......{code}
>
> We've test a solution which is workable, which is, adding another property *DESIGNATE_PROJECT_ID* in *AbstractBigQueryProcessor*, it'll only impact *PutBigQueryBatch* processor and *PutBigQueryStreaming* processor.
> If user provides designate project id:
> * Use *PROJECT_ID* (defined in AbstractGCPProcessor) as resource consuming project.
> * Put data into *DESIGNATE_PROJECT_ID* (defined in AbstractBigQueryProcessor).
> If user does {color:#ff0000}not{color} provide designate project id:
> * Use *PROJECT_ID* (defined in AbstractGCPProcessor) as resource consuming project.
> * Put data into *PROJECT_ID* (defined in AbstractGCPProcessor).
> Since we already implemented this solution in production environment, I'll submit a PR later for this improvement.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)