You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@nifi.apache.org by GitBox <gi...@apache.org> on 2022/08/30 16:34:18 UTC
[GitHub] [nifi] krisztina-zsihovszki commented on a diff in pull request #6350: NIFI-10242 Create QueryAirtableTable processor
krisztina-zsihovszki commented on code in PR #6350:
URL: https://github.com/apache/nifi/pull/6350#discussion_r958528713
##########
nifi-nar-bundles/nifi-airtable-bundle/nifi-airtable-processors/src/main/java/org/apache/nifi/processors/airtable/service/AirtableRestService.java:
##########
@@ -0,0 +1,108 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.processors.airtable.service;
+
+import java.io.IOException;
+import java.net.URI;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.List;
+import javax.ws.rs.core.UriBuilder;
+import org.apache.commons.io.IOUtils;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.web.client.api.HttpResponseEntity;
+import org.apache.nifi.web.client.api.WebClientService;
+
+public class AirtableRestService {
+
+ public static final String API_V0_BASE_URL = "https://api.airtable.com/v0";
+
+ private final WebClientService webClientService;
+ private final String apiUrl;
+ private final String apiToken;
+ private final String baseId;
+ private final String tableId;
+
+ public AirtableRestService(final WebClientService webClientService,
+ final String apiUrl,
+ final String apiToken,
+ final String baseId,
+ final String tableId) {
+ this.webClientService = webClientService;
+ this.apiUrl = apiUrl;
+ this.apiToken = apiToken;
+ this.baseId = baseId;
+ this.tableId = tableId;
+ }
+
+ public byte[] getRecords(final AirtableGetRecordsParameters filter) throws RateLimitExceeded {
+ final URI uri = buildUri(filter);
+ try (final HttpResponseEntity response = webClientService.get()
+ .uri(uri)
+ .header("Authorization", "Bearer " + apiToken)
+ .retrieve()) {
+
+ if (response.statusCode() >= 200 && response.statusCode() < 300) {
+ return IOUtils.toByteArray(response.body());
+ }
+ if (response.statusCode() == 429) {
Review Comment:
Please consider introducing constant for the status code 429 ("Too many requests") to increase readability.
##########
nifi-nar-bundles/nifi-airtable-bundle/nifi-airtable-processors/src/main/resources/docs/org.apache.nifi.processors.airtable.QueryAirtableTable/additionalDetails.html:
##########
@@ -0,0 +1,83 @@
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one or more
+ ~ contributor license agreements. See the NOTICE file distributed with
+ ~ this work for additional information regarding copyright ownership.
+ ~ The ASF licenses this file to You under the Apache License, Version 2.0
+ ~ (the "License"); you may not use this file except in compliance with
+ ~ the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing, software
+ ~ distributed under the License is distributed on an "AS IS" BASIS,
+ ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ ~ See the License for the specific language governing permissions and
+ ~ limitations under the License.
+ -->
+
+<!DOCTYPE html>
+<html lang="en" xmlns="http://www.w3.org/1999/html">
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
Review Comment:
The "Licensed to the Apache Software Foundation" is duplicated by mistake.
##########
nifi-nar-bundles/nifi-airtable-bundle/nifi-airtable-processors/src/main/resources/docs/org.apache.nifi.processors.airtable.QueryAirtableTable/additionalDetails.html:
##########
@@ -0,0 +1,83 @@
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one or more
+ ~ contributor license agreements. See the NOTICE file distributed with
+ ~ this work for additional information regarding copyright ownership.
+ ~ The ASF licenses this file to You under the Apache License, Version 2.0
+ ~ (the "License"); you may not use this file except in compliance with
+ ~ the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing, software
+ ~ distributed under the License is distributed on an "AS IS" BASIS,
+ ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ ~ See the License for the specific language governing permissions and
+ ~ limitations under the License.
+ -->
+
+<!DOCTYPE html>
+<html lang="en" xmlns="http://www.w3.org/1999/html">
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+ -->
+
+<head>
+ <meta charset="utf-8"/>
+ <title>QueryAirtableTable</title>
+ <link rel="stylesheet" href="../../../../../css/component-usage.css" type="text/css"/>
+ <style>
+h2 {margin-top: 4em}
+h3 {margin-top: 3em}
+td {text-align: left}
+ </style>
+</head>
+
+<body>
+
+<h1>QueryAirtableTable</h1>
+
+<h3>Description</h3>
+<p>
+ Airtable is a spreadsheet-database hybrid. In Airtable an application is called base and each base can have multiple tables.
+ A table consists of records (rows) and each record can have multiple fields (columns).
+ The QueryAirtableTable processor can query records from a single base and table via Airtable's REST API.
+ The processor utilizes streams and NiFi record-based processing to be able to handle a large number of records and to allow arbitrary output format.
+ It can also split large record sets to multiple FlowFiles and transfer them in batches just like a database processor.
+</p>
+
+<h3>API Token</h3>
+<p>
+ Airtable REST API calls require a so-called "API Token" that can needs to be passed in a request. An Airtable account is required to generate such token.
Review Comment:
typo, it should be "needs to be passed" I guess
##########
nifi-nar-bundles/nifi-airtable-bundle/nifi-airtable-processors/src/main/resources/docs/org.apache.nifi.processors.airtable.QueryAirtableTable/additionalDetails.html:
##########
@@ -0,0 +1,83 @@
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one or more
+ ~ contributor license agreements. See the NOTICE file distributed with
+ ~ this work for additional information regarding copyright ownership.
+ ~ The ASF licenses this file to You under the Apache License, Version 2.0
+ ~ (the "License"); you may not use this file except in compliance with
+ ~ the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing, software
+ ~ distributed under the License is distributed on an "AS IS" BASIS,
+ ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ ~ See the License for the specific language governing permissions and
+ ~ limitations under the License.
+ -->
+
+<!DOCTYPE html>
+<html lang="en" xmlns="http://www.w3.org/1999/html">
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+ -->
+
+<head>
+ <meta charset="utf-8"/>
+ <title>QueryAirtableTable</title>
+ <link rel="stylesheet" href="../../../../../css/component-usage.css" type="text/css"/>
+ <style>
+h2 {margin-top: 4em}
+h3 {margin-top: 3em}
+td {text-align: left}
+ </style>
+</head>
+
+<body>
+
+<h1>QueryAirtableTable</h1>
+
+<h3>Description</h3>
+<p>
+ Airtable is a spreadsheet-database hybrid. In Airtable an application is called base and each base can have multiple tables.
+ A table consists of records (rows) and each record can have multiple fields (columns).
+ The QueryAirtableTable processor can query records from a single base and table via Airtable's REST API.
+ The processor utilizes streams and NiFi record-based processing to be able to handle a large number of records and to allow arbitrary output format.
+ It can also split large record sets to multiple FlowFiles and transfer them in batches just like a database processor.
+</p>
+
+<h3>API Token</h3>
+<p>
+ Airtable REST API calls require a so-called "API Token" that can needs to be passed in a request. An Airtable account is required to generate such token.
+</p>
+
+<h3>API rate limit</h3>
+<p>
+ The Airtable REST API limits the number of requests that can be sent on a per-base basis to avoid bottlenecks. Currently, this limit is 5 requests per second per base.
+ If this limit is exceeded the user can't make another request for 30 seconds. It's the user's responsibility to handle this rate limit.
+</p>
+
+<h3>Metadata API</h3>
+<p>
+ Currently the Metadata API of Airtable is unstable, and we don't provide a way to use it. Until it becomes stable the user should set up a JsonTreeReader service to provide a schema.
+ The JsonTreeReader is need to be set up in a specific way to properly work with the response of an Airtable REST API call.
Review Comment:
typo, "it needs to be"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org