You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@nifi.apache.org by GitBox <gi...@apache.org> on 2020/01/10 17:13:02 UTC
[GitHub] [nifi] turcsanyip opened a new pull request #3979: NIFI-7009: Atlas reporting task retrieves only the active flow compon…
turcsanyip opened a new pull request #3979: NIFI-7009: Atlas reporting task retrieves only the active flow compon…
URL: https://github.com/apache/nifi/pull/3979
…ents
Filter out the deleted components before querying them, instead of retrieving
all the components before filtering.
https://issues.apache.org/jira/browse/NIFI-7009
Thank you for submitting a contribution to Apache NiFi.
Please provide a short description of the PR here:
#### Description of PR
_Enables X functionality; fixes bug NIFI-YYYY._
In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:
### For all changes:
- [ ] Is there a JIRA ticket associated with this PR? Is it referenced
in the commit message?
- [ ] Does your PR title start with **NIFI-XXXX** where XXXX is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character.
- [ ] Has your PR been rebased against the latest commit within the target branch (typically `master`)?
- [ ] Is your initial contribution a single, squashed commit? _Additional commits in response to PR reviewer feedback should be made on this branch and pushed to allow change tracking. Do not `squash` or use `--force` when pushing to allow for clean monitoring of changes._
### For code changes:
- [ ] Have you ensured that the full suite of tests is executed via `mvn -Pcontrib-check clean install` at the root `nifi` folder?
- [ ] Have you written or updated unit tests to verify your changes?
- [ ] Have you verified that the full build is successful on both JDK 8 and JDK 11?
- [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] If applicable, have you updated the `LICENSE` file, including the main `LICENSE` file under `nifi-assembly`?
- [ ] If applicable, have you updated the `NOTICE` file, including the main `NOTICE` file found under `nifi-assembly`?
- [ ] If adding new Properties, have you added `.displayName` in addition to .name (programmatic access) for each of the new properties?
### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in which it is rendered?
### Note:
Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [nifi] turcsanyip commented on a change in pull request #3979: NIFI-7009: Atlas reporting task retrieves only the active flow compon…
Posted by GitBox <gi...@apache.org>.
turcsanyip commented on a change in pull request #3979: NIFI-7009: Atlas reporting task retrieves only the active flow compon…
URL: https://github.com/apache/nifi/pull/3979#discussion_r367328847
##########
File path: nifi-nar-bundles/nifi-atlas-bundle/nifi-atlas-reporting-task/src/main/java/org/apache/nifi/atlas/NiFiAtlasClient.java
##########
@@ -230,6 +231,35 @@ public NiFiFlow fetchNiFiFlow(String rootProcessGroupId, String clusterName) thr
return nifiFlow;
}
+ /**
+ * Retrieves the flow components of type {@code componentType} from Atlas server.
+ * Deleted components will be filtered out before calling Atlas.
+ * Atlas object ids will be initialized with all the attributes (guid, type, unique attributes) in order to be able
+ * to match ids retrieved from Atlas (having guid) and ids created by the reporting task (not having guid yet).
+ *
+ * @param componentType Atlas type of the flow component (nifi_flow_path, nifi_queue, nifi_input_port, nifi_output_port)
+ * @param referredEntities referred entities of the flow entity (returned when the flow fetched) containing the basic data (id, status) of the flow components
+ * @return flow component entities mapped to their object ids
+ */
+ private Map<AtlasObjectId, AtlasEntity> fetchFlowComponents(String componentType, Map<String, AtlasEntity> referredEntities) {
+ return referredEntities.values().stream()
+ .filter(referredEntity -> referredEntity.getTypeName().equals(componentType))
+ .filter(referredEntity -> referredEntity.getStatus() == AtlasEntity.Status.ACTIVE)
+ .map(referredEntity -> {
+ final Map<String, Object> uniqueAttributes = Collections.singletonMap(ATTR_QUALIFIED_NAME, referredEntity.getAttribute(ATTR_QUALIFIED_NAME));
+ final AtlasObjectId id = new AtlasObjectId(referredEntity.getGuid(), componentType, uniqueAttributes);
+ try {
+ final AtlasEntity.AtlasEntityWithExtInfo fetchedEntityExt = searchEntityDef(id);
+ return new Tuple<>(id, fetchedEntityExt.getEntity());
+ } catch (AtlasServiceException e) {
+ logger.warn("Failed to search entity by id {}, due to {}", id, e);
+ return null;
Review comment:
It comes from the previous code. I saw it could be a problem, but I did want to change the existing behaviour in this ticket, only optimize it.
I'll create a jira for investigating it and finding a proper error handling for this case, if it is fine with you.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [nifi] turcsanyip commented on a change in pull request #3979: NIFI-7009: Atlas reporting task retrieves only the active flow compon…
Posted by GitBox <gi...@apache.org>.
turcsanyip commented on a change in pull request #3979: NIFI-7009: Atlas reporting task retrieves only the active flow compon…
URL: https://github.com/apache/nifi/pull/3979#discussion_r367378281
##########
File path: nifi-nar-bundles/nifi-atlas-bundle/nifi-atlas-reporting-task/src/main/java/org/apache/nifi/atlas/NiFiAtlasClient.java
##########
@@ -230,6 +231,35 @@ public NiFiFlow fetchNiFiFlow(String rootProcessGroupId, String clusterName) thr
return nifiFlow;
}
+ /**
+ * Retrieves the flow components of type {@code componentType} from Atlas server.
+ * Deleted components will be filtered out before calling Atlas.
+ * Atlas object ids will be initialized with all the attributes (guid, type, unique attributes) in order to be able
+ * to match ids retrieved from Atlas (having guid) and ids created by the reporting task (not having guid yet).
+ *
+ * @param componentType Atlas type of the flow component (nifi_flow_path, nifi_queue, nifi_input_port, nifi_output_port)
+ * @param referredEntities referred entities of the flow entity (returned when the flow fetched) containing the basic data (id, status) of the flow components
+ * @return flow component entities mapped to their object ids
+ */
+ private Map<AtlasObjectId, AtlasEntity> fetchFlowComponents(String componentType, Map<String, AtlasEntity> referredEntities) {
+ return referredEntities.values().stream()
+ .filter(referredEntity -> referredEntity.getTypeName().equals(componentType))
+ .filter(referredEntity -> referredEntity.getStatus() == AtlasEntity.Status.ACTIVE)
+ .map(referredEntity -> {
+ final Map<String, Object> uniqueAttributes = Collections.singletonMap(ATTR_QUALIFIED_NAME, referredEntity.getAttribute(ATTR_QUALIFIED_NAME));
+ final AtlasObjectId id = new AtlasObjectId(referredEntity.getGuid(), componentType, uniqueAttributes);
+ try {
+ final AtlasEntity.AtlasEntityWithExtInfo fetchedEntityExt = searchEntityDef(id);
Review comment:
Please see my reply to https://github.com/apache/nifi/pull/3979#discussion_r366927947.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [nifi] turcsanyip commented on a change in pull request #3979: NIFI-7009: Atlas reporting task retrieves only the active flow compon…
Posted by GitBox <gi...@apache.org>.
turcsanyip commented on a change in pull request #3979: NIFI-7009: Atlas reporting task retrieves only the active flow compon…
URL: https://github.com/apache/nifi/pull/3979#discussion_r367326456
##########
File path: nifi-nar-bundles/nifi-atlas-bundle/nifi-atlas-reporting-task/src/main/java/org/apache/nifi/atlas/NiFiAtlasClient.java
##########
@@ -201,12 +202,12 @@ public NiFiFlow fetchNiFiFlow(String rootProcessGroupId, String clusterName) thr
nifiFlow.setUrl(toStr(attributes.get(ATTR_URL)));
nifiFlow.setDescription(toStr(attributes.get(ATTR_DESCRIPTION)));
- nifiFlow.getQueues().putAll(toQualifiedNameIds(toAtlasObjectIds(nifiFlowEntity.getAttribute(ATTR_QUEUES))));
- nifiFlow.getRootInputPortEntities().putAll(toQualifiedNameIds(toAtlasObjectIds(nifiFlowEntity.getAttribute(ATTR_INPUT_PORTS))));
- nifiFlow.getRootOutputPortEntities().putAll(toQualifiedNameIds(toAtlasObjectIds(nifiFlowEntity.getAttribute(ATTR_OUTPUT_PORTS))));
+ nifiFlow.getQueues().putAll(fetchFlowComponents(TYPE_NIFI_QUEUE, nifiFlowReferredEntities));
+ nifiFlow.getRootInputPortEntities().putAll(fetchFlowComponents(TYPE_NIFI_INPUT_PORT, nifiFlowReferredEntities));
+ nifiFlow.getRootOutputPortEntities().putAll(fetchFlowComponents(TYPE_NIFI_OUTPUT_PORT, nifiFlowReferredEntities));
final Map<String, NiFiFlowPath> flowPaths = nifiFlow.getFlowPaths();
- final Map<AtlasObjectId, AtlasEntity> flowPathEntities = toQualifiedNameIds(toAtlasObjectIds(attributes.get(ATTR_FLOW_PATHS)));
+ final Map<AtlasObjectId, AtlasEntity> flowPathEntities = fetchFlowComponents(TYPE_NIFI_FLOW_PATH, nifiFlowReferredEntities);
for (AtlasEntity flowPathEntity : flowPathEntities.values()) {
Review comment:
Retrieving the flow and its components is composed of 3 steps:
1. get the flow
2. get the flow paths (and other subcomponents of the flow)
3. get the inputs / outputs of the flow paths
Input / output data was returned in all 3 steps which was definitely overkill.
[NIFI-6945](https://issues.apache.org/jira/browse/NIFI-6945) was about excluding the input/output data from step 1. There are 2 reasons for doing this: Getting all inputs/outputs of all flow paths in a single query can lead to a huge response json from Atlas (extremely high response time / timeout). Furthermore, there is no way to query only the ACTIVE `referredEntities` from Atlas. So in case of `minExtInfo=true`, all the deleted flow paths were retrieved with all their inputs/outputs unnecessarily (they will be filtered out in step 2). So I believe that having multiple but smaller requests in step 2 is better than having one huge request in step 1.
Your comment about falling back to the old logic (in step 3) is reasonable. Input / output data is still retrieved multiple times in step 2 and 3. My plan is to create a separate ticket for it. I would keep the scope of this ticket on the active/delete filtering of the flow paths only.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [nifi] tpalfy commented on a change in pull request #3979: NIFI-7009: Atlas reporting task retrieves only the active flow compon…
Posted by GitBox <gi...@apache.org>.
tpalfy commented on a change in pull request #3979: NIFI-7009: Atlas reporting task retrieves only the active flow compon…
URL: https://github.com/apache/nifi/pull/3979#discussion_r366927947
##########
File path: nifi-nar-bundles/nifi-atlas-bundle/nifi-atlas-reporting-task/src/main/java/org/apache/nifi/atlas/NiFiAtlasClient.java
##########
@@ -201,12 +202,12 @@ public NiFiFlow fetchNiFiFlow(String rootProcessGroupId, String clusterName) thr
nifiFlow.setUrl(toStr(attributes.get(ATTR_URL)));
nifiFlow.setDescription(toStr(attributes.get(ATTR_DESCRIPTION)));
- nifiFlow.getQueues().putAll(toQualifiedNameIds(toAtlasObjectIds(nifiFlowEntity.getAttribute(ATTR_QUEUES))));
- nifiFlow.getRootInputPortEntities().putAll(toQualifiedNameIds(toAtlasObjectIds(nifiFlowEntity.getAttribute(ATTR_INPUT_PORTS))));
- nifiFlow.getRootOutputPortEntities().putAll(toQualifiedNameIds(toAtlasObjectIds(nifiFlowEntity.getAttribute(ATTR_OUTPUT_PORTS))));
+ nifiFlow.getQueues().putAll(fetchFlowComponents(TYPE_NIFI_QUEUE, nifiFlowReferredEntities));
+ nifiFlow.getRootInputPortEntities().putAll(fetchFlowComponents(TYPE_NIFI_INPUT_PORT, nifiFlowReferredEntities));
+ nifiFlow.getRootOutputPortEntities().putAll(fetchFlowComponents(TYPE_NIFI_OUTPUT_PORT, nifiFlowReferredEntities));
final Map<String, NiFiFlowPath> flowPaths = nifiFlow.getFlowPaths();
- final Map<AtlasObjectId, AtlasEntity> flowPathEntities = toQualifiedNameIds(toAtlasObjectIds(attributes.get(ATTR_FLOW_PATHS)));
+ final Map<AtlasObjectId, AtlasEntity> flowPathEntities = fetchFlowComponents(TYPE_NIFI_FLOW_PATH, nifiFlowReferredEntities);
for (AtlasEntity flowPathEntity : flowPathEntities.values()) {
Review comment:
Feels strange to leave the handling of the flowpaths as it is. We retrieve only the `ACTIVE` ones, but we discard their `referredEntities`. So when it comes to their inputs/ouputs, we fall back to the old logic and retrieve them all and filter out the `DELETED` ones afterwards.
Even more, to me it seems if we actually didn't set `minExtInfo` to `true` but left it on `false`, we could get away with a single REST call, because in that case each of the `referredEntities` would contain `relationshipAttributes` that also have a `guid` and `entityStatus` (`ACTIVE` or `DELETED`) attribute.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [nifi] asfgit closed pull request #3979: NIFI-7009: Atlas reporting task retrieves only the active flow compon…
Posted by GitBox <gi...@apache.org>.
asfgit closed pull request #3979: NIFI-7009: Atlas reporting task retrieves only the active flow compon…
URL: https://github.com/apache/nifi/pull/3979
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [nifi] mcgilman commented on issue #3979: NIFI-7009: Atlas reporting task retrieves only the active flow compon…
Posted by GitBox <gi...@apache.org>.
mcgilman commented on issue #3979: NIFI-7009: Atlas reporting task retrieves only the active flow compon…
URL: https://github.com/apache/nifi/pull/3979#issuecomment-575234392
Thanks for the PR @turcsanyip. Thanks for the review @tpalfy. This has been merged to master.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [nifi] tpalfy commented on a change in pull request #3979: NIFI-7009: Atlas reporting task retrieves only the active flow compon…
Posted by GitBox <gi...@apache.org>.
tpalfy commented on a change in pull request #3979: NIFI-7009: Atlas reporting task retrieves only the active flow compon…
URL: https://github.com/apache/nifi/pull/3979#discussion_r366875871
##########
File path: nifi-nar-bundles/nifi-atlas-bundle/nifi-atlas-reporting-task/src/main/java/org/apache/nifi/atlas/NiFiAtlasClient.java
##########
@@ -230,6 +231,35 @@ public NiFiFlow fetchNiFiFlow(String rootProcessGroupId, String clusterName) thr
return nifiFlow;
}
+ /**
+ * Retrieves the flow components of type {@code componentType} from Atlas server.
+ * Deleted components will be filtered out before calling Atlas.
+ * Atlas object ids will be initialized with all the attributes (guid, type, unique attributes) in order to be able
+ * to match ids retrieved from Atlas (having guid) and ids created by the reporting task (not having guid yet).
+ *
+ * @param componentType Atlas type of the flow component (nifi_flow_path, nifi_queue, nifi_input_port, nifi_output_port)
+ * @param referredEntities referred entities of the flow entity (returned when the flow fetched) containing the basic data (id, status) of the flow components
+ * @return flow component entities mapped to their object ids
+ */
+ private Map<AtlasObjectId, AtlasEntity> fetchFlowComponents(String componentType, Map<String, AtlasEntity> referredEntities) {
+ return referredEntities.values().stream()
+ .filter(referredEntity -> referredEntity.getTypeName().equals(componentType))
+ .filter(referredEntity -> referredEntity.getStatus() == AtlasEntity.Status.ACTIVE)
+ .map(referredEntity -> {
+ final Map<String, Object> uniqueAttributes = Collections.singletonMap(ATTR_QUALIFIED_NAME, referredEntity.getAttribute(ATTR_QUALIFIED_NAME));
+ final AtlasObjectId id = new AtlasObjectId(referredEntity.getGuid(), componentType, uniqueAttributes);
+ try {
+ final AtlasEntity.AtlasEntityWithExtInfo fetchedEntityExt = searchEntityDef(id);
Review comment:
Wondering if we could set `ignoreRelationship` to `true` for this `atlasClient` call (in `searchEntityDef`).
Not sure if it matters that much.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [nifi] tpalfy commented on a change in pull request #3979: NIFI-7009: Atlas reporting task retrieves only the active flow compon…
Posted by GitBox <gi...@apache.org>.
tpalfy commented on a change in pull request #3979: NIFI-7009: Atlas reporting task retrieves only the active flow compon…
URL: https://github.com/apache/nifi/pull/3979#discussion_r366877708
##########
File path: nifi-nar-bundles/nifi-atlas-bundle/nifi-atlas-reporting-task/src/main/java/org/apache/nifi/atlas/NiFiAtlasClient.java
##########
@@ -230,6 +231,35 @@ public NiFiFlow fetchNiFiFlow(String rootProcessGroupId, String clusterName) thr
return nifiFlow;
}
+ /**
+ * Retrieves the flow components of type {@code componentType} from Atlas server.
+ * Deleted components will be filtered out before calling Atlas.
+ * Atlas object ids will be initialized with all the attributes (guid, type, unique attributes) in order to be able
+ * to match ids retrieved from Atlas (having guid) and ids created by the reporting task (not having guid yet).
+ *
+ * @param componentType Atlas type of the flow component (nifi_flow_path, nifi_queue, nifi_input_port, nifi_output_port)
+ * @param referredEntities referred entities of the flow entity (returned when the flow fetched) containing the basic data (id, status) of the flow components
+ * @return flow component entities mapped to their object ids
+ */
+ private Map<AtlasObjectId, AtlasEntity> fetchFlowComponents(String componentType, Map<String, AtlasEntity> referredEntities) {
+ return referredEntities.values().stream()
+ .filter(referredEntity -> referredEntity.getTypeName().equals(componentType))
+ .filter(referredEntity -> referredEntity.getStatus() == AtlasEntity.Status.ACTIVE)
+ .map(referredEntity -> {
+ final Map<String, Object> uniqueAttributes = Collections.singletonMap(ATTR_QUALIFIED_NAME, referredEntity.getAttribute(ATTR_QUALIFIED_NAME));
+ final AtlasObjectId id = new AtlasObjectId(referredEntity.getGuid(), componentType, uniqueAttributes);
+ try {
+ final AtlasEntity.AtlasEntityWithExtInfo fetchedEntityExt = searchEntityDef(id);
+ return new Tuple<>(id, fetchedEntityExt.getEntity());
+ } catch (AtlasServiceException e) {
+ logger.warn("Failed to search entity by id {}, due to {}", id, e);
+ return null;
Review comment:
We only WARN if a component cannot be retrieved from Atlas then move on. We don't end up with an incomplete flow because we will try to recreate the (only seemingly) missing components?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [nifi] turcsanyip commented on a change in pull request #3979: NIFI-7009: Atlas reporting task retrieves only the active flow compon…
Posted by GitBox <gi...@apache.org>.
turcsanyip commented on a change in pull request #3979: NIFI-7009: Atlas reporting task retrieves only the active flow compon…
URL: https://github.com/apache/nifi/pull/3979#discussion_r367326456
##########
File path: nifi-nar-bundles/nifi-atlas-bundle/nifi-atlas-reporting-task/src/main/java/org/apache/nifi/atlas/NiFiAtlasClient.java
##########
@@ -201,12 +202,12 @@ public NiFiFlow fetchNiFiFlow(String rootProcessGroupId, String clusterName) thr
nifiFlow.setUrl(toStr(attributes.get(ATTR_URL)));
nifiFlow.setDescription(toStr(attributes.get(ATTR_DESCRIPTION)));
- nifiFlow.getQueues().putAll(toQualifiedNameIds(toAtlasObjectIds(nifiFlowEntity.getAttribute(ATTR_QUEUES))));
- nifiFlow.getRootInputPortEntities().putAll(toQualifiedNameIds(toAtlasObjectIds(nifiFlowEntity.getAttribute(ATTR_INPUT_PORTS))));
- nifiFlow.getRootOutputPortEntities().putAll(toQualifiedNameIds(toAtlasObjectIds(nifiFlowEntity.getAttribute(ATTR_OUTPUT_PORTS))));
+ nifiFlow.getQueues().putAll(fetchFlowComponents(TYPE_NIFI_QUEUE, nifiFlowReferredEntities));
+ nifiFlow.getRootInputPortEntities().putAll(fetchFlowComponents(TYPE_NIFI_INPUT_PORT, nifiFlowReferredEntities));
+ nifiFlow.getRootOutputPortEntities().putAll(fetchFlowComponents(TYPE_NIFI_OUTPUT_PORT, nifiFlowReferredEntities));
final Map<String, NiFiFlowPath> flowPaths = nifiFlow.getFlowPaths();
- final Map<AtlasObjectId, AtlasEntity> flowPathEntities = toQualifiedNameIds(toAtlasObjectIds(attributes.get(ATTR_FLOW_PATHS)));
+ final Map<AtlasObjectId, AtlasEntity> flowPathEntities = fetchFlowComponents(TYPE_NIFI_FLOW_PATH, nifiFlowReferredEntities);
for (AtlasEntity flowPathEntity : flowPathEntities.values()) {
Review comment:
Retrieving the flow and its components is composed of 3 steps:
1. get the flow
2. get the flow paths (and other subcomponents of the flow)
3. get the inputs / outputs of the flow paths
Input / output data was returned in all 3 steps which was definitely overkill.
[NIFI-6945](https://issues.apache.org/jira/browse/NIFI-6945) was about excluding the input/output data from step 1. There are 2 reasons for doing this: Getting all inputs/outputs of all flow paths in a single query can lead to a huge response json from Atlas (extremely high response time / timeout). Furthermore, there is no way to query only the ACTIVE `referredEntities` from Atlas. So in case of `minExtInfo=true`, all the deleted flow paths were retrieved with all their inputs/outputs unnecessarily (they will be filtered out in step 2). So I believe that having multiple but smaller requests in step 2 is better than having one huge request in step 1.
Your comment about falling back to the old logic (in step 3) is reasonable. Input / output data is still retrieved multiple times in step 2 and 3. My plan is to create a separate ticket for it. I would keep the scope of this ticket on the active/delete filtering of the flow paths only.
Question of `ignoreRelationship=true` would belong to the new jira. It needs to be investigated which solution is better:
- keep the current step 3 (lots of very small requests), in this case `ignoreRelationship=true` can be set for step 2 which would decrease the size of that request significantly
- use the relationship data returned in step2 (one bigger request) and eliminate step 3 at all
The performance of these options needs to be tested with both the Simple Path / Complex Path lineage strategy settings of the reporting task.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services