You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@nifi.apache.org by "bbende (via GitHub)" <gi...@apache.org> on 2025/12/15 22:05:55 UTC
[PR] NIFI-15315 Add support for assets in connectors [nifi]
bbende opened a new pull request, #10647:
URL: https://github.com/apache/nifi/pull/10647
- Support multiple asset references
- Clean up of assets in StandardConnectorDAO after apply and discard
- Add endpoint for listing assets for a given connector
- Add endpoint for discarding the working config of a connector
- Add endpoint for retrieving asset content
- Add system test for connectors that uploads an asset
<!-- Licensed to the Apache Software Foundation (ASF) under one or more -->
<!-- contributor license agreements. See the NOTICE file distributed with -->
<!-- this work for additional information regarding copyright ownership. -->
<!-- The ASF licenses this file to You under the Apache License, Version 2.0 -->
<!-- (the "License"); you may not use this file except in compliance with -->
<!-- the License. You may obtain a copy of the License at -->
<!-- http://www.apache.org/licenses/LICENSE-2.0 -->
<!-- Unless required by applicable law or agreed to in writing, software -->
<!-- distributed under the License is distributed on an "AS IS" BASIS, -->
<!-- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -->
<!-- See the License for the specific language governing permissions and -->
<!-- limitations under the License. -->
# Summary
[NIFI-15315](https://issues.apache.org/jira/browse/NIFI-15315)
Adds support for creating/using assets in Connectors...
- Creates a new AssetManager implementation for Connectors
- Adds REST APIs in ConnectorsResource for interacting with assets
- Adds an asset synchronizer for Connectors, similar to the approach for parameter contexts
# Tracking
Please complete the following tracking steps prior to pull request creation.
### Issue Tracking
- [X] [Apache NiFi Jira](https://issues.apache.org/jira/browse/NIFI) issue created
### Pull Request Tracking
- [X] Pull Request title starts with Apache NiFi Jira issue number, such as `NIFI-00000`
- [X] Pull Request commit message starts with Apache NiFi Jira issue number, as such `NIFI-00000`
### Pull Request Formatting
- [X] Pull Request based on current revision of the `main` branch
- [X] Pull Request refers to a feature branch with one commit containing changes
# Verification
Please indicate the verification steps performed prior to pull request creation.
### Build
- [X] Build completed using `./mvnw clean install -P contrib-check`
- [X] JDK 21
- [ ] JDK 25
### Licensing
- [ ] New dependencies are compatible with the [Apache License 2.0](https://apache.org/licenses/LICENSE-2.0) according to the [License Policy](https://www.apache.org/legal/resolved.html)
- [ ] New dependencies are documented in applicable `LICENSE` and `NOTICE` files
### Documentation
- [ ] Documentation formatting appears as expected in rendered files
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] NIFI-15315 Add support for assets in connectors [nifi]
Posted by "kevdoran (via GitHub)" <gi...@apache.org>.
kevdoran commented on code in PR #10647:
URL: https://github.com/apache/nifi/pull/10647#discussion_r2628703187
##########
nifi-framework-bundle/nifi-framework/nifi-framework-components/src/main/java/org/apache/nifi/asset/StandardConnectorAssetManager.java:
##########
@@ -0,0 +1,225 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.asset;
+
+import org.apache.nifi.nar.FileDigestUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.File;
+import java.io.IOException;
+import java.io.InputStream;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.nio.file.Paths;
+import java.nio.file.StandardCopyOption;
+import java.util.ArrayList;
+import java.util.HexFormat;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.UUID;
+import java.util.concurrent.ConcurrentHashMap;
+
+/**
+ * AssetManager implementation for Connectors. This differs from the StandardAssetManager used for Parameter Contexts in that
+ * Connectors want every asset added to be treated as a new asset, even if the same filename is being saved to the same Connector.
+ * This is because all changes to the Connector needs to be stored separately in the working configuration until applied and can
+ * not interfere with the active configuration.
+ */
+public class StandardConnectorAssetManager implements AssetManager {
+
+ private static final Logger logger = LoggerFactory.getLogger(StandardConnectorAssetManager.class);
+
+ public static final String ASSET_STORAGE_LOCATION_PROPERTY = "directory";
+ public static final String DEFAULT_ASSET_STORAGE_LOCATION = "./connector-assets";
+
+ private volatile File assetStorageLocation;
+ private final Map<String, Asset> assets = new ConcurrentHashMap<>();
+
+ @Override
+ public void initialize(final AssetManagerInitializationContext context) {
+ final String storageLocation = getStorageLocation(context);
+
+ assetStorageLocation = new File(storageLocation);
+ if (!assetStorageLocation.exists()) {
+ try {
+ Files.createDirectories(assetStorageLocation.toPath());
+ } catch (IOException e) {
+ throw new RuntimeException("The Connector Asset Manager's [%s] property is set to [%s] but the directory does not exist and cannot be created"
+ .formatted(ASSET_STORAGE_LOCATION_PROPERTY, storageLocation), e);
+ }
+ }
+
+ try {
+ recoverLocalAssets();
+ } catch (final IOException e) {
+ throw new RuntimeException("Unable to access connector assets", e);
+ }
+ }
+
+ @Override
+ public Asset createAsset(final String connectorId, final String assetName, final InputStream contents) throws IOException {
+ final String assetId = generateAssetId();
+ final File assetFile = getFile(connectorId, assetId, assetName);
+ return saveAsset(connectorId, assetId, assetName, contents, assetFile);
+ }
+
+ @Override
+ public Asset saveAsset(final String connectorId, final String assetId, final String assetName, final InputStream contents) throws IOException {
+ final File assetFile = getFile(connectorId, assetId, assetName);
+ return saveAsset(connectorId, assetId, assetName, contents, assetFile);
+ }
+
+ @Override
+ public Optional<Asset> getAsset(final String id) {
+ return Optional.ofNullable(assets.get(id));
+ }
+
+ @Override
+ public List<Asset> getAssets(final String connectorId) {
+ final List<Asset> allAssets = new ArrayList<>(assets.values());
+ final List<Asset> ownerAssets = new ArrayList<>();
+ for (final Asset asset : allAssets) {
+ if (asset.getOwnerIdentifier().equals(connectorId)) {
+ ownerAssets.add(asset);
+ }
+ }
+ return ownerAssets;
+ }
+
+ @Override
+ public Asset createMissingAsset(final String connectorId, final String assetName) {
+ final String assetId = generateAssetId();
+ final File file = getFile(connectorId, assetId, assetName);
+ final Asset asset = new StandardAsset(assetId, connectorId, assetName, file, null);
+ assets.put(assetId, asset);
+ return asset;
+ }
+
+ @Override
+ public Optional<Asset> deleteAsset(final String id) {
+ final Asset removed = assets.remove(id);
+ if (removed == null) {
+ return Optional.empty();
+ }
+
+ final File assetFile = removed.getFile();
+ if (assetFile.exists()) {
+ deleteFile(assetFile);
+
+ final File assetIdDir = assetFile.getParentFile();
+ deleteFile(assetIdDir);
+
+ final File connectorDir = assetIdDir.getParentFile();
+ final File[] children = connectorDir.listFiles();
+ if (children != null && children.length == 0) {
+ deleteFile(connectorDir);
+ }
+ }
+
+ return Optional.of(removed);
+ }
+
+ private Asset saveAsset(final String connectorId, final String assetId, final String assetName, final InputStream contents, final File assetFile) throws IOException {
+ final File dir = assetFile.getParentFile();
+ if (!dir.exists()) {
+ try {
+ Files.createDirectories(dir.toPath());
+ } catch (final IOException ioe) {
+ throw new IOException("Could not create directory in order to store connector asset", ioe);
+ }
+ }
+
+ try {
+ Files.copy(contents, assetFile.toPath(), StandardCopyOption.REPLACE_EXISTING);
+ } catch (final Exception e) {
+ throw new IOException("Failed to write connector asset to file " + assetFile.getAbsolutePath(), e);
+ }
+
+ final String digest = computeDigest(assetFile);
+ final Asset asset = new StandardAsset(assetId, connectorId, assetName, assetFile, digest);
+ assets.put(assetId, asset);
+ return asset;
+ }
+
+ private void deleteFile(final File file) {
+ try {
+ Files.delete(file.toPath());
+ } catch (final IOException e) {
+ logger.warn("Failed to delete [{}]", file.getAbsolutePath(), e);
+ }
+ }
+
+ private String generateAssetId() {
+ return UUID.randomUUID().toString();
+ }
+
+ private File getFile(final String connectorId, final String assetId, final String assetName) {
+ final Path parentPath = assetStorageLocation.toPath().normalize();
+ final Path assetPath = Paths.get(connectorId, assetId, assetName).normalize();
+ final Path fullPath = parentPath.resolve(assetPath);
+ return fullPath.toFile();
+ }
+
+ private String getStorageLocation(final AssetManagerInitializationContext initializationContext) {
+ final String storageLocation = initializationContext.getProperties().get(ASSET_STORAGE_LOCATION_PROPERTY);
+ return storageLocation == null ? DEFAULT_ASSET_STORAGE_LOCATION : storageLocation;
+ }
+
+ private void recoverLocalAssets() throws IOException {
+ final File[] connectorDirectories = assetStorageLocation.listFiles();
+ if (connectorDirectories == null) {
+ throw new IOException("Unable to list files for connector asset storage location %s".formatted(assetStorageLocation.getAbsolutePath()));
+ }
+
+ for (final File connectorDirectory : connectorDirectories) {
+ if (!connectorDirectory.isDirectory()) {
+ continue;
+ }
+
+ final String connectorId = connectorDirectory.getName();
+ final File[] assetIdDirectories = connectorDirectory.listFiles();
+ if (assetIdDirectories == null) {
+ logger.warn("Unable to list directory [{}]", connectorDirectory.getAbsolutePath());
+ continue;
+ }
+
+ for (final File assetIdDirectory : assetIdDirectories) {
+ final String assetId = assetIdDirectory.getName();
+
+ final File[] assetFiles = assetIdDirectory.listFiles();
+ if (assetFiles == null) {
+ logger.warn("Unable to list directory [{}]", assetIdDirectory.getAbsolutePath());
+ continue;
+ }
+
+ for (final File assetFile : assetFiles) {
+ final String assetName = assetFile.getName();
+ final String digest = computeDigest(assetFile);
Review Comment:
I just noticed this, but so long as we are computing hash digests on every write operation anyway, maybe we want to use content addressable storage to avoid saving duplicate-copies of assets, so instead of the local file path being connectorId/assetId/filename, it would just be the hash of the content? this could work across multiple connectors too, e.g., maybe many instances of database connector that all need the same driver binary or something. Just an idea.
##########
nifi-framework-bundle/nifi-framework/nifi-framework-components/src/main/java/org/apache/nifi/asset/StandardConnectorAssetManager.java:
##########
@@ -0,0 +1,225 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.asset;
+
+import org.apache.nifi.nar.FileDigestUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.File;
+import java.io.IOException;
+import java.io.InputStream;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.nio.file.Paths;
+import java.nio.file.StandardCopyOption;
+import java.util.ArrayList;
+import java.util.HexFormat;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.UUID;
+import java.util.concurrent.ConcurrentHashMap;
+
+/**
+ * AssetManager implementation for Connectors. This differs from the StandardAssetManager used for Parameter Contexts in that
+ * Connectors want every asset added to be treated as a new asset, even if the same filename is being saved to the same Connector.
+ * This is because all changes to the Connector needs to be stored separately in the working configuration until applied and can
+ * not interfere with the active configuration.
+ */
+public class StandardConnectorAssetManager implements AssetManager {
+
+ private static final Logger logger = LoggerFactory.getLogger(StandardConnectorAssetManager.class);
+
+ public static final String ASSET_STORAGE_LOCATION_PROPERTY = "directory";
+ public static final String DEFAULT_ASSET_STORAGE_LOCATION = "./connector-assets";
+
+ private volatile File assetStorageLocation;
+ private final Map<String, Asset> assets = new ConcurrentHashMap<>();
+
+ @Override
+ public void initialize(final AssetManagerInitializationContext context) {
+ final String storageLocation = getStorageLocation(context);
+
+ assetStorageLocation = new File(storageLocation);
+ if (!assetStorageLocation.exists()) {
+ try {
+ Files.createDirectories(assetStorageLocation.toPath());
+ } catch (IOException e) {
+ throw new RuntimeException("The Connector Asset Manager's [%s] property is set to [%s] but the directory does not exist and cannot be created"
+ .formatted(ASSET_STORAGE_LOCATION_PROPERTY, storageLocation), e);
+ }
+ }
+
+ try {
+ recoverLocalAssets();
+ } catch (final IOException e) {
+ throw new RuntimeException("Unable to access connector assets", e);
+ }
+ }
+
+ @Override
+ public Asset createAsset(final String connectorId, final String assetName, final InputStream contents) throws IOException {
+ final String assetId = generateAssetId();
+ final File assetFile = getFile(connectorId, assetId, assetName);
+ return saveAsset(connectorId, assetId, assetName, contents, assetFile);
+ }
+
+ @Override
+ public Asset saveAsset(final String connectorId, final String assetId, final String assetName, final InputStream contents) throws IOException {
+ final File assetFile = getFile(connectorId, assetId, assetName);
+ return saveAsset(connectorId, assetId, assetName, contents, assetFile);
+ }
+
+ @Override
+ public Optional<Asset> getAsset(final String id) {
+ return Optional.ofNullable(assets.get(id));
+ }
+
+ @Override
+ public List<Asset> getAssets(final String connectorId) {
+ final List<Asset> allAssets = new ArrayList<>(assets.values());
+ final List<Asset> ownerAssets = new ArrayList<>();
+ for (final Asset asset : allAssets) {
+ if (asset.getOwnerIdentifier().equals(connectorId)) {
+ ownerAssets.add(asset);
+ }
+ }
+ return ownerAssets;
+ }
+
+ @Override
+ public Asset createMissingAsset(final String connectorId, final String assetName) {
+ final String assetId = generateAssetId();
+ final File file = getFile(connectorId, assetId, assetName);
+ final Asset asset = new StandardAsset(assetId, connectorId, assetName, file, null);
+ assets.put(assetId, asset);
+ return asset;
+ }
+
+ @Override
+ public Optional<Asset> deleteAsset(final String id) {
+ final Asset removed = assets.remove(id);
+ if (removed == null) {
+ return Optional.empty();
+ }
+
+ final File assetFile = removed.getFile();
+ if (assetFile.exists()) {
+ deleteFile(assetFile);
+
+ final File assetIdDir = assetFile.getParentFile();
+ deleteFile(assetIdDir);
+
+ final File connectorDir = assetIdDir.getParentFile();
+ final File[] children = connectorDir.listFiles();
+ if (children != null && children.length == 0) {
+ deleteFile(connectorDir);
+ }
+ }
+
+ return Optional.of(removed);
+ }
+
+ private Asset saveAsset(final String connectorId, final String assetId, final String assetName, final InputStream contents, final File assetFile) throws IOException {
+ final File dir = assetFile.getParentFile();
+ if (!dir.exists()) {
+ try {
+ Files.createDirectories(dir.toPath());
+ } catch (final IOException ioe) {
+ throw new IOException("Could not create directory in order to store connector asset", ioe);
+ }
+ }
+
+ try {
+ Files.copy(contents, assetFile.toPath(), StandardCopyOption.REPLACE_EXISTING);
+ } catch (final Exception e) {
+ throw new IOException("Failed to write connector asset to file " + assetFile.getAbsolutePath(), e);
+ }
+
+ final String digest = computeDigest(assetFile);
Review Comment:
I just noticed this, but so long as we are computing hash digests on every write operation anyway, maybe we want to use content addressable storage to avoid saving duplicate-copies of assets, so instead of the local file path being `connectorId/assetId/filename`, it would just be the hash of the content? this could work across multiple connectors too, e.g., maybe many instances of database connector that all need the same driver binary or something. Just an idea.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] NIFI-15315 Add support for assets in connectors [nifi]
Posted by "kevdoran (via GitHub)" <gi...@apache.org>.
kevdoran commented on code in PR #10647:
URL: https://github.com/apache/nifi/pull/10647#discussion_r2628700961
##########
nifi-framework-bundle/nifi-framework/nifi-framework-components/src/main/java/org/apache/nifi/asset/StandardConnectorAssetManager.java:
##########
@@ -0,0 +1,225 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.asset;
+
+import org.apache.nifi.nar.FileDigestUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.File;
+import java.io.IOException;
+import java.io.InputStream;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.nio.file.Paths;
+import java.nio.file.StandardCopyOption;
+import java.util.ArrayList;
+import java.util.HexFormat;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.UUID;
+import java.util.concurrent.ConcurrentHashMap;
+
+/**
+ * AssetManager implementation for Connectors. This differs from the StandardAssetManager used for Parameter Contexts in that
+ * Connectors want every asset added to be treated as a new asset, even if the same filename is being saved to the same Connector.
+ * This is because all changes to the Connector needs to be stored separately in the working configuration until applied and can
+ * not interfere with the active configuration.
+ */
+public class StandardConnectorAssetManager implements AssetManager {
+
+ private static final Logger logger = LoggerFactory.getLogger(StandardConnectorAssetManager.class);
+
+ public static final String ASSET_STORAGE_LOCATION_PROPERTY = "directory";
+ public static final String DEFAULT_ASSET_STORAGE_LOCATION = "./connector-assets";
+
+ private volatile File assetStorageLocation;
+ private final Map<String, Asset> assets = new ConcurrentHashMap<>();
+
+ @Override
+ public void initialize(final AssetManagerInitializationContext context) {
+ final String storageLocation = getStorageLocation(context);
+
+ assetStorageLocation = new File(storageLocation);
+ if (!assetStorageLocation.exists()) {
+ try {
+ Files.createDirectories(assetStorageLocation.toPath());
+ } catch (IOException e) {
+ throw new RuntimeException("The Connector Asset Manager's [%s] property is set to [%s] but the directory does not exist and cannot be created"
+ .formatted(ASSET_STORAGE_LOCATION_PROPERTY, storageLocation), e);
+ }
+ }
+
+ try {
+ recoverLocalAssets();
+ } catch (final IOException e) {
+ throw new RuntimeException("Unable to access connector assets", e);
+ }
+ }
+
+ @Override
+ public Asset createAsset(final String connectorId, final String assetName, final InputStream contents) throws IOException {
+ final String assetId = generateAssetId();
+ final File assetFile = getFile(connectorId, assetId, assetName);
+ return saveAsset(connectorId, assetId, assetName, contents, assetFile);
+ }
+
+ @Override
+ public Asset saveAsset(final String connectorId, final String assetId, final String assetName, final InputStream contents) throws IOException {
+ final File assetFile = getFile(connectorId, assetId, assetName);
+ return saveAsset(connectorId, assetId, assetName, contents, assetFile);
+ }
+
+ @Override
+ public Optional<Asset> getAsset(final String id) {
+ return Optional.ofNullable(assets.get(id));
+ }
+
+ @Override
+ public List<Asset> getAssets(final String connectorId) {
+ final List<Asset> allAssets = new ArrayList<>(assets.values());
+ final List<Asset> ownerAssets = new ArrayList<>();
+ for (final Asset asset : allAssets) {
+ if (asset.getOwnerIdentifier().equals(connectorId)) {
+ ownerAssets.add(asset);
+ }
+ }
+ return ownerAssets;
+ }
+
+ @Override
+ public Asset createMissingAsset(final String connectorId, final String assetName) {
+ final String assetId = generateAssetId();
+ final File file = getFile(connectorId, assetId, assetName);
+ final Asset asset = new StandardAsset(assetId, connectorId, assetName, file, null);
+ assets.put(assetId, asset);
+ return asset;
+ }
+
+ @Override
+ public Optional<Asset> deleteAsset(final String id) {
+ final Asset removed = assets.remove(id);
+ if (removed == null) {
+ return Optional.empty();
+ }
+
+ final File assetFile = removed.getFile();
+ if (assetFile.exists()) {
+ deleteFile(assetFile);
+
+ final File assetIdDir = assetFile.getParentFile();
+ deleteFile(assetIdDir);
+
+ final File connectorDir = assetIdDir.getParentFile();
+ final File[] children = connectorDir.listFiles();
+ if (children != null && children.length == 0) {
+ deleteFile(connectorDir);
+ }
+ }
+
+ return Optional.of(removed);
+ }
+
+ private Asset saveAsset(final String connectorId, final String assetId, final String assetName, final InputStream contents, final File assetFile) throws IOException {
+ final File dir = assetFile.getParentFile();
+ if (!dir.exists()) {
+ try {
+ Files.createDirectories(dir.toPath());
+ } catch (final IOException ioe) {
+ throw new IOException("Could not create directory in order to store connector asset", ioe);
+ }
+ }
+
+ try {
+ Files.copy(contents, assetFile.toPath(), StandardCopyOption.REPLACE_EXISTING);
+ } catch (final Exception e) {
+ throw new IOException("Failed to write connector asset to file " + assetFile.getAbsolutePath(), e);
+ }
+
+ final String digest = computeDigest(assetFile);
Review Comment:
I just noticed this, but so long as we are computing hash digests on every write operation anyway, maybe we want to use content addressable storage to avoid saving duplicate-copies of assets, so instead of the local file path being `connectorId/assetId/filename`, it would just be the hash of the content? this could work across multiple connectors too, e.g., maybe many instances of database connector that all need the same driver binary or something. Just an idea.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] NIFI-15315 Add support for assets in connectors [nifi]
Posted by "markap14 (via GitHub)" <gi...@apache.org>.
markap14 commented on code in PR #10647:
URL: https://github.com/apache/nifi/pull/10647#discussion_r2627259972
##########
nifi-framework-bundle/nifi-framework/nifi-framework-components/src/main/java/org/apache/nifi/asset/StandardAssetManager.java:
##########
@@ -155,14 +139,43 @@ public Optional<Asset> deleteAsset(final String id) {
return Optional.of(removed);
}
- private String createAssetId(final String parameterContextId, final String assetName) {
- final String seed = parameterContextId + "/" + assetName;
+ private Asset saveAsset(final String ownerId, final String assetId, final String assetName, final InputStream contents, final File assetFile) throws IOException {
+ final File dir = assetFile.getParentFile();
+ if (!dir.exists()) {
+ try {
+ Files.createDirectories(dir.toPath());
+ } catch (final IOException ioe) {
+ throw new IOException("Could not create directory in order to store asset", ioe);
+ }
+ }
+
+ // Write contents to a temporary file, then move it to the final location.
+ // This allows us to avoid a situation where we upload a file, then we attempt to overwrite it but fail, leaving a corrupt asset.
+ final File tempFile = new File(dir, assetFile.getName() + ".tmp");
+ logger.debug("Writing temp asset file [{}]", tempFile.getAbsolutePath());
+
+ try {
+ Files.copy(contents, tempFile.toPath(), StandardCopyOption.REPLACE_EXISTING);
+ } catch (final Exception e) {
+ throw new IOException("Failed to write asset to file " + tempFile.getAbsolutePath(), e);
+ }
+
+ Files.move(tempFile.toPath(), assetFile.toPath(), StandardCopyOption.REPLACE_EXISTING);
+
+ final String digest = computeDigest(assetFile);
Review Comment:
Not crucial for this PR, but if the Asset is large, this could be pretty expensive, re-reading the content from disk to compute the digest. Would be worth considering in a follow-on PR, perhaps, to just wrap the `InputStream` with a `DigestInputStream`.
##########
nifi-framework-bundle/nifi-framework/nifi-framework-components/src/main/java/org/apache/nifi/asset/StandardConnectorAssetManager.java:
##########
@@ -0,0 +1,225 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.asset;
+
+import org.apache.nifi.nar.FileDigestUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.File;
+import java.io.IOException;
+import java.io.InputStream;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.nio.file.Paths;
+import java.nio.file.StandardCopyOption;
+import java.util.ArrayList;
+import java.util.HexFormat;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.UUID;
+import java.util.concurrent.ConcurrentHashMap;
+
+/**
+ * AssetManager implementation for Connectors. This differs from the StandardAssetManager used for Parameter Contexts in that
+ * Connectors want every asset added to be treated as a new asset, even if the same filename is being saved to the same Connector.
+ * This is because all changes to the Connector needs to be stored separately in the working configuration until applied and can
+ * not interfere with the active configuration.
+ */
+public class StandardConnectorAssetManager implements AssetManager {
+
+ private static final Logger logger = LoggerFactory.getLogger(StandardConnectorAssetManager.class);
+
+ public static final String ASSET_STORAGE_LOCATION_PROPERTY = "directory";
+ public static final String DEFAULT_ASSET_STORAGE_LOCATION = "./connector-assets";
+
+ private volatile File assetStorageLocation;
+ private final Map<String, Asset> assets = new ConcurrentHashMap<>();
+
+ @Override
+ public void initialize(final AssetManagerInitializationContext context) {
+ final String storageLocation = getStorageLocation(context);
+
+ assetStorageLocation = new File(storageLocation);
+ if (!assetStorageLocation.exists()) {
+ try {
+ Files.createDirectories(assetStorageLocation.toPath());
+ } catch (IOException e) {
+ throw new RuntimeException("The Connector Asset Manager's [%s] property is set to [%s] but the directory does not exist and cannot be created"
+ .formatted(ASSET_STORAGE_LOCATION_PROPERTY, storageLocation), e);
+ }
+ }
+
+ try {
+ recoverLocalAssets();
+ } catch (final IOException e) {
+ throw new RuntimeException("Unable to access connector assets", e);
+ }
+ }
+
+ @Override
+ public Asset createAsset(final String connectorId, final String assetName, final InputStream contents) throws IOException {
+ final String assetId = generateAssetId();
+ final File assetFile = getFile(connectorId, assetId, assetName);
+ return saveAsset(connectorId, assetId, assetName, contents, assetFile);
+ }
+
+ @Override
+ public Asset saveAsset(final String connectorId, final String assetId, final String assetName, final InputStream contents) throws IOException {
+ final File assetFile = getFile(connectorId, assetId, assetName);
+ return saveAsset(connectorId, assetId, assetName, contents, assetFile);
+ }
+
+ @Override
+ public Optional<Asset> getAsset(final String id) {
+ return Optional.ofNullable(assets.get(id));
+ }
+
+ @Override
+ public List<Asset> getAssets(final String connectorId) {
+ final List<Asset> allAssets = new ArrayList<>(assets.values());
+ final List<Asset> ownerAssets = new ArrayList<>();
+ for (final Asset asset : allAssets) {
+ if (asset.getOwnerIdentifier().equals(connectorId)) {
+ ownerAssets.add(asset);
+ }
+ }
+ return ownerAssets;
+ }
+
+ @Override
+ public Asset createMissingAsset(final String connectorId, final String assetName) {
+ final String assetId = generateAssetId();
+ final File file = getFile(connectorId, assetId, assetName);
+ final Asset asset = new StandardAsset(assetId, connectorId, assetName, file, null);
+ assets.put(assetId, asset);
+ return asset;
+ }
+
+ @Override
+ public Optional<Asset> deleteAsset(final String id) {
+ final Asset removed = assets.remove(id);
+ if (removed == null) {
+ return Optional.empty();
+ }
+
+ final File assetFile = removed.getFile();
+ if (assetFile.exists()) {
+ deleteFile(assetFile);
+
+ final File assetIdDir = assetFile.getParentFile();
+ deleteFile(assetIdDir);
+
+ final File connectorDir = assetIdDir.getParentFile();
+ final File[] children = connectorDir.listFiles();
+ if (children != null && children.length == 0) {
+ deleteFile(connectorDir);
+ }
+ }
+
+ return Optional.of(removed);
+ }
+
+ private Asset saveAsset(final String connectorId, final String assetId, final String assetName, final InputStream contents, final File assetFile) throws IOException {
+ final File dir = assetFile.getParentFile();
+ if (!dir.exists()) {
+ try {
+ Files.createDirectories(dir.toPath());
+ } catch (final IOException ioe) {
+ throw new IOException("Could not create directory in order to store connector asset", ioe);
+ }
+ }
+
+ try {
+ Files.copy(contents, assetFile.toPath(), StandardCopyOption.REPLACE_EXISTING);
+ } catch (final Exception e) {
+ throw new IOException("Failed to write connector asset to file " + assetFile.getAbsolutePath(), e);
+ }
+
+ final String digest = computeDigest(assetFile);
+ final Asset asset = new StandardAsset(assetId, connectorId, assetName, assetFile, digest);
+ assets.put(assetId, asset);
+ return asset;
+ }
+
+ private void deleteFile(final File file) {
+ try {
+ Files.delete(file.toPath());
+ } catch (final IOException e) {
+ logger.warn("Failed to delete [{}]", file.getAbsolutePath(), e);
+ }
+ }
+
+ private String generateAssetId() {
+ return UUID.randomUUID().toString();
+ }
+
+ private File getFile(final String connectorId, final String assetId, final String assetName) {
+ final Path parentPath = assetStorageLocation.toPath().normalize();
+ final Path assetPath = Paths.get(connectorId, assetId, assetName).normalize();
+ final Path fullPath = parentPath.resolve(assetPath);
+ return fullPath.toFile();
+ }
+
+ private String getStorageLocation(final AssetManagerInitializationContext initializationContext) {
+ final String storageLocation = initializationContext.getProperties().get(ASSET_STORAGE_LOCATION_PROPERTY);
+ return storageLocation == null ? DEFAULT_ASSET_STORAGE_LOCATION : storageLocation;
+ }
+
+ private void recoverLocalAssets() throws IOException {
+ final File[] connectorDirectories = assetStorageLocation.listFiles();
+ if (connectorDirectories == null) {
+ throw new IOException("Unable to list files for connector asset storage location %s".formatted(assetStorageLocation.getAbsolutePath()));
+ }
+
+ for (final File connectorDirectory : connectorDirectories) {
+ if (!connectorDirectory.isDirectory()) {
+ continue;
+ }
+
+ final String connectorId = connectorDirectory.getName();
+ final File[] assetIdDirectories = connectorDirectory.listFiles();
+ if (assetIdDirectories == null) {
+ logger.warn("Unable to list directory [{}]", connectorDirectory.getAbsolutePath());
+ continue;
+ }
+
+ for (final File assetIdDirectory : assetIdDirectories) {
+ final String assetId = assetIdDirectory.getName();
+
+ final File[] assetFiles = assetIdDirectory.listFiles();
+ if (assetFiles == null) {
+ logger.warn("Unable to list directory [{}]", assetIdDirectory.getAbsolutePath());
+ continue;
+ }
+
+ for (final File assetFile : assetFiles) {
+ final String assetName = assetFile.getName();
+ final String digest = computeDigest(assetFile);
Review Comment:
If we expect a bunch of assets, especially large ones, it may also be worth considering adding the digest to the filename so that it's not necessary to compute on startup every time. Again, not necessary here but worth considering for a future PR.
##########
nifi-framework-bundle/nifi-framework/nifi-framework-components/src/main/java/org/apache/nifi/asset/StandardConnectorAssetManager.java:
##########
@@ -0,0 +1,225 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.asset;
+
+import org.apache.nifi.nar.FileDigestUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.File;
+import java.io.IOException;
+import java.io.InputStream;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.nio.file.Paths;
+import java.nio.file.StandardCopyOption;
+import java.util.ArrayList;
+import java.util.HexFormat;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.UUID;
+import java.util.concurrent.ConcurrentHashMap;
+
+/**
+ * AssetManager implementation for Connectors. This differs from the StandardAssetManager used for Parameter Contexts in that
+ * Connectors want every asset added to be treated as a new asset, even if the same filename is being saved to the same Connector.
+ * This is because all changes to the Connector needs to be stored separately in the working configuration until applied and can
+ * not interfere with the active configuration.
+ */
+public class StandardConnectorAssetManager implements AssetManager {
+
+ private static final Logger logger = LoggerFactory.getLogger(StandardConnectorAssetManager.class);
+
+ public static final String ASSET_STORAGE_LOCATION_PROPERTY = "directory";
+ public static final String DEFAULT_ASSET_STORAGE_LOCATION = "./connector-assets";
+
+ private volatile File assetStorageLocation;
+ private final Map<String, Asset> assets = new ConcurrentHashMap<>();
+
+ @Override
+ public void initialize(final AssetManagerInitializationContext context) {
+ final String storageLocation = getStorageLocation(context);
+
+ assetStorageLocation = new File(storageLocation);
+ if (!assetStorageLocation.exists()) {
+ try {
+ Files.createDirectories(assetStorageLocation.toPath());
+ } catch (IOException e) {
+ throw new RuntimeException("The Connector Asset Manager's [%s] property is set to [%s] but the directory does not exist and cannot be created"
+ .formatted(ASSET_STORAGE_LOCATION_PROPERTY, storageLocation), e);
+ }
+ }
+
+ try {
+ recoverLocalAssets();
+ } catch (final IOException e) {
+ throw new RuntimeException("Unable to access connector assets", e);
+ }
+ }
+
+ @Override
+ public Asset createAsset(final String connectorId, final String assetName, final InputStream contents) throws IOException {
+ final String assetId = generateAssetId();
+ final File assetFile = getFile(connectorId, assetId, assetName);
+ return saveAsset(connectorId, assetId, assetName, contents, assetFile);
+ }
+
+ @Override
+ public Asset saveAsset(final String connectorId, final String assetId, final String assetName, final InputStream contents) throws IOException {
+ final File assetFile = getFile(connectorId, assetId, assetName);
+ return saveAsset(connectorId, assetId, assetName, contents, assetFile);
+ }
+
+ @Override
+ public Optional<Asset> getAsset(final String id) {
+ return Optional.ofNullable(assets.get(id));
+ }
+
+ @Override
+ public List<Asset> getAssets(final String connectorId) {
+ final List<Asset> allAssets = new ArrayList<>(assets.values());
+ final List<Asset> ownerAssets = new ArrayList<>();
+ for (final Asset asset : allAssets) {
+ if (asset.getOwnerIdentifier().equals(connectorId)) {
+ ownerAssets.add(asset);
+ }
+ }
+ return ownerAssets;
+ }
+
+ @Override
+ public Asset createMissingAsset(final String connectorId, final String assetName) {
+ final String assetId = generateAssetId();
+ final File file = getFile(connectorId, assetId, assetName);
+ final Asset asset = new StandardAsset(assetId, connectorId, assetName, file, null);
+ assets.put(assetId, asset);
+ return asset;
+ }
+
+ @Override
+ public Optional<Asset> deleteAsset(final String id) {
+ final Asset removed = assets.remove(id);
+ if (removed == null) {
+ return Optional.empty();
+ }
+
+ final File assetFile = removed.getFile();
+ if (assetFile.exists()) {
+ deleteFile(assetFile);
+
+ final File assetIdDir = assetFile.getParentFile();
+ deleteFile(assetIdDir);
+
+ final File connectorDir = assetIdDir.getParentFile();
+ final File[] children = connectorDir.listFiles();
+ if (children != null && children.length == 0) {
+ deleteFile(connectorDir);
+ }
+ }
+
+ return Optional.of(removed);
+ }
+
+ private Asset saveAsset(final String connectorId, final String assetId, final String assetName, final InputStream contents, final File assetFile) throws IOException {
+ final File dir = assetFile.getParentFile();
+ if (!dir.exists()) {
+ try {
+ Files.createDirectories(dir.toPath());
+ } catch (final IOException ioe) {
+ throw new IOException("Could not create directory in order to store connector asset", ioe);
+ }
+ }
+
+ try {
+ Files.copy(contents, assetFile.toPath(), StandardCopyOption.REPLACE_EXISTING);
+ } catch (final Exception e) {
+ throw new IOException("Failed to write connector asset to file " + assetFile.getAbsolutePath(), e);
+ }
+
+ final String digest = computeDigest(assetFile);
+ final Asset asset = new StandardAsset(assetId, connectorId, assetName, assetFile, digest);
+ assets.put(assetId, asset);
+ return asset;
+ }
+
+ private void deleteFile(final File file) {
+ try {
+ Files.delete(file.toPath());
+ } catch (final IOException e) {
+ logger.warn("Failed to delete [{}]", file.getAbsolutePath(), e);
+ }
+ }
+
+ private String generateAssetId() {
+ return UUID.randomUUID().toString();
+ }
+
+ private File getFile(final String connectorId, final String assetId, final String assetName) {
+ final Path parentPath = assetStorageLocation.toPath().normalize();
+ final Path assetPath = Paths.get(connectorId, assetId, assetName).normalize();
+ final Path fullPath = parentPath.resolve(assetPath);
+ return fullPath.toFile();
+ }
+
+ private String getStorageLocation(final AssetManagerInitializationContext initializationContext) {
+ final String storageLocation = initializationContext.getProperties().get(ASSET_STORAGE_LOCATION_PROPERTY);
+ return storageLocation == null ? DEFAULT_ASSET_STORAGE_LOCATION : storageLocation;
+ }
+
+ private void recoverLocalAssets() throws IOException {
+ final File[] connectorDirectories = assetStorageLocation.listFiles();
+ if (connectorDirectories == null) {
+ throw new IOException("Unable to list files for connector asset storage location %s".formatted(assetStorageLocation.getAbsolutePath()));
+ }
+
+ for (final File connectorDirectory : connectorDirectories) {
+ if (!connectorDirectory.isDirectory()) {
+ continue;
+ }
+
+ final String connectorId = connectorDirectory.getName();
+ final File[] assetIdDirectories = connectorDirectory.listFiles();
+ if (assetIdDirectories == null) {
+ logger.warn("Unable to list directory [{}]", connectorDirectory.getAbsolutePath());
+ continue;
+ }
+
+ for (final File assetIdDirectory : assetIdDirectories) {
+ final String assetId = assetIdDirectory.getName();
+
+ final File[] assetFiles = assetIdDirectory.listFiles();
+ if (assetFiles == null) {
+ logger.warn("Unable to list directory [{}]", assetIdDirectory.getAbsolutePath());
+ continue;
+ }
+
+ for (final File assetFile : assetFiles) {
+ final String assetName = assetFile.getName();
+ final String digest = computeDigest(assetFile);
Review Comment:
Would also mean that comparing assets would get simpler across cluster. If the local asset has the same filename, you know it also has the same hash because the hash is built into the filename.
##########
nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/asset/StandardConnectorAssetSynchronizer.java:
##########
@@ -0,0 +1,245 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.asset;
+
+import org.apache.commons.lang3.exception.ExceptionUtils;
+import org.apache.nifi.client.NiFiRestApiRetryableException;
+import org.apache.nifi.cluster.coordination.ClusterCoordinator;
+import org.apache.nifi.cluster.protocol.NodeIdentifier;
+import org.apache.nifi.components.connector.ConnectorNode;
+import org.apache.nifi.controller.FlowController;
+import org.apache.nifi.controller.flow.FlowManager;
+import org.apache.nifi.util.NiFiProperties;
+import org.apache.nifi.web.api.dto.AssetDTO;
+import org.apache.nifi.web.api.entity.AssetEntity;
+import org.apache.nifi.web.api.entity.AssetsEntity;
+import org.apache.nifi.web.client.api.WebClientService;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.time.Duration;
+import java.time.Instant;
+import java.util.Collection;
+import java.util.List;
+import java.util.Map;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+
+/**
+ * Synchronizes connector assets from the cluster coordinator using the NiFi REST API.
+ */
+public class StandardConnectorAssetSynchronizer implements AssetSynchronizer {
+
+ private static final Logger logger = LoggerFactory.getLogger(StandardConnectorAssetSynchronizer.class);
+
+ private static final Duration CLUSTER_COORDINATOR_RETRY_DURATION = Duration.ofSeconds(60);
+ private static final Duration LIST_ASSETS_RETRY_DURATION = Duration.ofMinutes(5);
+ private static final Duration SYNC_ASSET_RETRY_DURATION = Duration.ofSeconds(30);
+
+ private final AssetManager assetManager;
+ private final FlowManager flowManager;
+ private final ClusterCoordinator clusterCoordinator;
+ private final WebClientService webClientService;
+ private final NiFiProperties properties;
+
+ public StandardConnectorAssetSynchronizer(final FlowController flowController,
+ final ClusterCoordinator clusterCoordinator,
+ final WebClientService webClientService,
+ final NiFiProperties properties) {
+ this.assetManager = flowController.getConnectorAssetManager();
+ this.flowManager = flowController.getFlowManager();
Review Comment:
If we only need the Connector AssetManager and the FlowManager from FlowController, probably best to just pass those in, rather than FlowController?
##########
nifi-framework-bundle/nifi-framework/nifi-framework-components/src/main/java/org/apache/nifi/asset/StandardConnectorAssetManager.java:
##########
@@ -0,0 +1,225 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.asset;
+
+import org.apache.nifi.nar.FileDigestUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.File;
+import java.io.IOException;
+import java.io.InputStream;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.nio.file.Paths;
+import java.nio.file.StandardCopyOption;
+import java.util.ArrayList;
+import java.util.HexFormat;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.UUID;
+import java.util.concurrent.ConcurrentHashMap;
+
+/**
+ * AssetManager implementation for Connectors. This differs from the StandardAssetManager used for Parameter Contexts in that
+ * Connectors want every asset added to be treated as a new asset, even if the same filename is being saved to the same Connector.
+ * This is because all changes to the Connector needs to be stored separately in the working configuration until applied and can
+ * not interfere with the active configuration.
+ */
+public class StandardConnectorAssetManager implements AssetManager {
+
+ private static final Logger logger = LoggerFactory.getLogger(StandardConnectorAssetManager.class);
+
+ public static final String ASSET_STORAGE_LOCATION_PROPERTY = "directory";
+ public static final String DEFAULT_ASSET_STORAGE_LOCATION = "./connector-assets";
+
+ private volatile File assetStorageLocation;
+ private final Map<String, Asset> assets = new ConcurrentHashMap<>();
+
+ @Override
+ public void initialize(final AssetManagerInitializationContext context) {
+ final String storageLocation = getStorageLocation(context);
+
+ assetStorageLocation = new File(storageLocation);
+ if (!assetStorageLocation.exists()) {
+ try {
+ Files.createDirectories(assetStorageLocation.toPath());
+ } catch (IOException e) {
+ throw new RuntimeException("The Connector Asset Manager's [%s] property is set to [%s] but the directory does not exist and cannot be created"
+ .formatted(ASSET_STORAGE_LOCATION_PROPERTY, storageLocation), e);
+ }
+ }
+
+ try {
+ recoverLocalAssets();
+ } catch (final IOException e) {
+ throw new RuntimeException("Unable to access connector assets", e);
+ }
+ }
+
+ @Override
+ public Asset createAsset(final String connectorId, final String assetName, final InputStream contents) throws IOException {
+ final String assetId = generateAssetId();
+ final File assetFile = getFile(connectorId, assetId, assetName);
+ return saveAsset(connectorId, assetId, assetName, contents, assetFile);
+ }
+
+ @Override
+ public Asset saveAsset(final String connectorId, final String assetId, final String assetName, final InputStream contents) throws IOException {
+ final File assetFile = getFile(connectorId, assetId, assetName);
+ return saveAsset(connectorId, assetId, assetName, contents, assetFile);
+ }
+
+ @Override
+ public Optional<Asset> getAsset(final String id) {
+ return Optional.ofNullable(assets.get(id));
+ }
+
+ @Override
+ public List<Asset> getAssets(final String connectorId) {
+ final List<Asset> allAssets = new ArrayList<>(assets.values());
+ final List<Asset> ownerAssets = new ArrayList<>();
+ for (final Asset asset : allAssets) {
+ if (asset.getOwnerIdentifier().equals(connectorId)) {
+ ownerAssets.add(asset);
+ }
+ }
+ return ownerAssets;
+ }
+
+ @Override
+ public Asset createMissingAsset(final String connectorId, final String assetName) {
+ final String assetId = generateAssetId();
+ final File file = getFile(connectorId, assetId, assetName);
+ final Asset asset = new StandardAsset(assetId, connectorId, assetName, file, null);
+ assets.put(assetId, asset);
+ return asset;
+ }
+
+ @Override
+ public Optional<Asset> deleteAsset(final String id) {
+ final Asset removed = assets.remove(id);
+ if (removed == null) {
+ return Optional.empty();
+ }
+
+ final File assetFile = removed.getFile();
+ if (assetFile.exists()) {
+ deleteFile(assetFile);
+
+ final File assetIdDir = assetFile.getParentFile();
+ deleteFile(assetIdDir);
+
+ final File connectorDir = assetIdDir.getParentFile();
+ final File[] children = connectorDir.listFiles();
+ if (children != null && children.length == 0) {
+ deleteFile(connectorDir);
+ }
+ }
+
+ return Optional.of(removed);
+ }
+
+ private Asset saveAsset(final String connectorId, final String assetId, final String assetName, final InputStream contents, final File assetFile) throws IOException {
+ final File dir = assetFile.getParentFile();
+ if (!dir.exists()) {
+ try {
+ Files.createDirectories(dir.toPath());
+ } catch (final IOException ioe) {
+ throw new IOException("Could not create directory in order to store connector asset", ioe);
+ }
+ }
+
+ try {
+ Files.copy(contents, assetFile.toPath(), StandardCopyOption.REPLACE_EXISTING);
+ } catch (final Exception e) {
+ throw new IOException("Failed to write connector asset to file " + assetFile.getAbsolutePath(), e);
+ }
+
+ final String digest = computeDigest(assetFile);
Review Comment:
As with StandardAssetManager might be worth a follow-on using a `DigestInputStream` to save re-reading the file contents just to calculate hash
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] NIFI-15315 Add support for assets in connectors [nifi]
Posted by "kevdoran (via GitHub)" <gi...@apache.org>.
kevdoran commented on code in PR #10647:
URL: https://github.com/apache/nifi/pull/10647#discussion_r2627673429
##########
nifi-commons/nifi-properties/src/main/java/org/apache/nifi/util/NiFiProperties.java:
##########
@@ -124,6 +124,9 @@ public class NiFiProperties extends ApplicationProperties {
public static final String ASSET_MANAGER_IMPLEMENTATION = "nifi.asset.manager.implementation";
public static final String ASSET_MANAGER_PREFIX = "nifi.asset.manager.properties.";
+ public static final String CONNECTOR_ASSET_MANAGER_IMPLEMENTATION = "nifi.connector.asset.manager.implementation";
+ public static final String CONNECTOR_ASSET_MANAGER_PREFIX = "nifi.connector.asset.manager.properties.";
+
Review Comment:
We probably want to co-locate this with the `CONNECTOR_REPOSITORY_IMPLEMENTATION` below and use a common prefix for the property names:
```
// Connector Repository properties
public static final String CONNECTOR_REPOSITORY_IMPLEMENTATION = "nifi.components.connectors.repository.implementation";
```
I'm good with `nifi.connector.*` as the connector prefix. I can adjust that in a separate PR as it is unrelated to this, but if we are using `nifi.components.connector(s).*` then this PR should probably be updated.
##########
nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/components/connector/StandardConnectorAssetRepository.java:
##########
@@ -0,0 +1,63 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.components.connector;
+
+import org.apache.nifi.asset.Asset;
+import org.apache.nifi.asset.AssetManager;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.util.List;
+import java.util.Optional;
+
+public class StandardConnectorAssetRepository implements ConnectorAssetRepository {
+
+ private final AssetManager assetManager;
+
+ public StandardConnectorAssetRepository(final AssetManager assetManager) {
+ this.assetManager = assetManager;
+ }
+
+ @Override
+ public Asset storeAsset(final String connectorId, final String assetId, final String assetName, final InputStream content) throws IOException {
+ return assetManager.saveAsset(connectorId, assetId, assetName, content);
+ }
+
+ @Override
+ public Optional<Asset> getAsset(String assetId) {
+ return assetManager.getAsset(assetId);
+ }
+
+ @Override
+ public List<Asset> getAssets(final String connectorId) {
+ return assetManager.getAssets(connectorId);
+ }
+
+ @Override
+ public void deleteAsset(final String assetId) {
+ assetManager.deleteAsset(assetId);
+ }
+
+ @Override
+ public void deleteAssets(final String connectorId) {
+ final List<Asset> assets = assetManager.getAssets(connectorId);
+ for (final Asset asset : assets) {
+ assetManager.deleteAsset(asset.getIdentifier());
+ }
+ }
+
+}
Review Comment:
I'm not sure it is a blocker for this PR, but I have the feeling we are going to have to refactor the relationships of some of these types:
```
ConnectorRepository
ConnectorAssetRepository
ConnectorAssetManager
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] NIFI-15315 Add support for assets in connectors [nifi]
Posted by "kevdoran (via GitHub)" <gi...@apache.org>.
kevdoran merged PR #10647:
URL: https://github.com/apache/nifi/pull/10647
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org