You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@nifi.apache.org by GitBox <gi...@apache.org> on 2021/09/27 14:02:47 UTC
[GitHub] [nifi-minifi-cpp] lordgamez opened a new pull request #1180: MINIFICPP-1530 Put Kafka tests in order
lordgamez opened a new pull request #1180:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1180
- Removed obselete PublishKafkaOnScheduleTests
- PublishKafkaTests were not running previously because of the wrong parameters in the CMake file, but also would have required running Kafka broker. Moved the checks of the test to the Kafka integration tests.
- ConsumeKafkaTests mentioned in the Jira ticket were removed in a previous PR
https://issues.apache.org/jira/browse/MINIFICPP-1530
---------------------------------------------------------------------------------------------------
Thank you for submitting a contribution to Apache NiFi - MiNiFi C++.
In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:
### For all changes:
- [ ] Is there a JIRA ticket associated with this PR? Is it referenced
in the commit message?
- [ ] Does your PR title start with MINIFICPP-XXXX where XXXX is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character.
- [ ] Has your PR been rebased against the latest commit within the target branch (typically main)?
- [ ] Is your initial contribution a single, squashed commit?
### For code changes:
- [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] If applicable, have you updated the LICENSE file?
- [ ] If applicable, have you updated the NOTICE file?
### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in which it is rendered?
### Note:
Please ensure that once the PR is submitted, you check GitHub Actions CI results for build issues and submit an update to your PR as soon as possible.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi-minifi-cpp] lordgamez commented on a change in pull request #1180: MINIFICPP-1530 Put Kafka tests in order
Posted by GitBox <gi...@apache.org>.
lordgamez commented on a change in pull request #1180:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1180#discussion_r724809518
##########
File path: docker/test/integration/features/kafka.feature
##########
@@ -9,15 +9,41 @@ Feature: Sending data to using Kafka streaming platform using PublishKafka
Scenario: A MiNiFi instance transfers data to a kafka broker
Given a GetFile processor with the "Input Directory" property set to "/tmp/input"
And a file with the content "test" is present in "/tmp/input"
+ And a UpdateAttribute processor
+ And these processor properties are set:
+ | processor name | property name | property value |
+ | UpdateAttribute | kafka_require_num_acks | 1 |
+ | UpdateAttribute | kafka_message_key | unique_message_key_123 |
And a PublishKafka processor set up to communicate with a kafka broker instance
+ And these processor properties are set:
+ | processor name | property name | property value |
+ | PublishKafka | Topic Name | test |
+ | PublishKafka | Delivery Guarantee | ${kafka_require_num_acks} |
+ | PublishKafka | Request Timeout | 12 s |
+ | PublishKafka | Message Timeout | 13 s |
+ | PublishKafka | Known Brokers | kafka-broker:${literal(9000):plus(92)} |
+ | PublishKafka | Client Name | client_no_${literal(6):multiply(7)} |
+ | PublishKafka | Kafka Key | ${kafka_message_key} |
+ | PublishKafka | retry.backoff.ms | ${literal(2):multiply(25):multiply(3)} |
+ | PublishKafka | Message Key Field | kafka.key |
And a PutFile processor with the "Directory" property set to "/tmp/output"
- And the "success" relationship of the GetFile processor is connected to the PublishKafka
+ And the "success" relationship of the GetFile processor is connected to the UpdateAttribute
+ And the "success" relationship of the UpdateAttribute processor is connected to the PublishKafka
And the "success" relationship of the PublishKafka processor is connected to the PutFile
And a kafka broker is set up in correspondence with the PublishKafka
When both instances start up
Then a flowfile with the content "test" is placed in the monitored directory in less than 60 seconds
+ And the Minifi logs contain the following message: " is 'test'" in less than 60 seconds
+ And the Minifi logs contain the following message: "PublishKafka: request.required.acks [1]" in less than 60 seconds
+ And the Minifi logs contain the following message: "PublishKafka: request.timeout.ms [12000]" in less than 60 seconds
+ And the Minifi logs contain the following message: "PublishKafka: message.timeout.ms [13000]" in less than 60 seconds
+ And the Minifi logs contain the following message: "PublishKafka: bootstrap.servers [kafka-broker:9092]" in less than 60 seconds
+ And the Minifi logs contain the following message: "PublishKafka: client.id [client_no_42]" in less than 60 seconds
+ And the Minifi logs contain the following message: "PublishKafka: Message Key [unique_message_key_123]" in less than 60 seconds
+ And the Minifi logs contain the following message: "PublishKafka: DynamicProperty: [retry.backoff.ms] -> [150]" in less than 60 seconds
+ And the Minifi logs contain the following message: "The Message Key Field property is set. This property is DEPRECATED and has no effect; please use Kafka Key instead." in less than 60 seconds
Review comment:
You are right it was missing, I added the check in c8e330072c80476d03d3a84fb001f6a2d3d6a5a9 borrowing regex check functionality from #1172
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi-minifi-cpp] adamdebreceni commented on a change in pull request #1180: MINIFICPP-1530 Put Kafka tests in order
Posted by GitBox <gi...@apache.org>.
adamdebreceni commented on a change in pull request #1180:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1180#discussion_r724725347
##########
File path: docker/test/integration/features/kafka.feature
##########
@@ -9,15 +9,41 @@ Feature: Sending data to using Kafka streaming platform using PublishKafka
Scenario: A MiNiFi instance transfers data to a kafka broker
Given a GetFile processor with the "Input Directory" property set to "/tmp/input"
And a file with the content "test" is present in "/tmp/input"
+ And a UpdateAttribute processor
+ And these processor properties are set:
+ | processor name | property name | property value |
+ | UpdateAttribute | kafka_require_num_acks | 1 |
+ | UpdateAttribute | kafka_message_key | unique_message_key_123 |
And a PublishKafka processor set up to communicate with a kafka broker instance
+ And these processor properties are set:
+ | processor name | property name | property value |
+ | PublishKafka | Topic Name | test |
+ | PublishKafka | Delivery Guarantee | ${kafka_require_num_acks} |
+ | PublishKafka | Request Timeout | 12 s |
+ | PublishKafka | Message Timeout | 13 s |
+ | PublishKafka | Known Brokers | kafka-broker:${literal(9000):plus(92)} |
+ | PublishKafka | Client Name | client_no_${literal(6):multiply(7)} |
+ | PublishKafka | Kafka Key | ${kafka_message_key} |
+ | PublishKafka | retry.backoff.ms | ${literal(2):multiply(25):multiply(3)} |
+ | PublishKafka | Message Key Field | kafka.key |
And a PutFile processor with the "Directory" property set to "/tmp/output"
- And the "success" relationship of the GetFile processor is connected to the PublishKafka
+ And the "success" relationship of the GetFile processor is connected to the UpdateAttribute
+ And the "success" relationship of the UpdateAttribute processor is connected to the PublishKafka
And the "success" relationship of the PublishKafka processor is connected to the PutFile
And a kafka broker is set up in correspondence with the PublishKafka
When both instances start up
Then a flowfile with the content "test" is placed in the monitored directory in less than 60 seconds
+ And the Minifi logs contain the following message: " is 'test'" in less than 60 seconds
+ And the Minifi logs contain the following message: "PublishKafka: request.required.acks [1]" in less than 60 seconds
+ And the Minifi logs contain the following message: "PublishKafka: request.timeout.ms [12000]" in less than 60 seconds
+ And the Minifi logs contain the following message: "PublishKafka: message.timeout.ms [13000]" in less than 60 seconds
+ And the Minifi logs contain the following message: "PublishKafka: bootstrap.servers [kafka-broker:9092]" in less than 60 seconds
+ And the Minifi logs contain the following message: "PublishKafka: client.id [client_no_42]" in less than 60 seconds
+ And the Minifi logs contain the following message: "PublishKafka: Message Key [unique_message_key_123]" in less than 60 seconds
+ And the Minifi logs contain the following message: "PublishKafka: DynamicProperty: [retry.backoff.ms] -> [150]" in less than 60 seconds
+ And the Minifi logs contain the following message: "The Message Key Field property is set. This property is DEPRECATED and has no effect; please use Kafka Key instead." in less than 60 seconds
Review comment:
there was a test checking if we fallback to the flowfile's uuid if the Kafka Key is not set, is that replicated here somewhere?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi-minifi-cpp] adamdebreceni commented on a change in pull request #1180: MINIFICPP-1530 Put Kafka tests in order
Posted by GitBox <gi...@apache.org>.
adamdebreceni commented on a change in pull request #1180:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1180#discussion_r730636351
##########
File path: docker/test/integration/features/kafka.feature
##########
@@ -9,15 +9,41 @@ Feature: Sending data to using Kafka streaming platform using PublishKafka
Scenario: A MiNiFi instance transfers data to a kafka broker
Given a GetFile processor with the "Input Directory" property set to "/tmp/input"
And a file with the content "test" is present in "/tmp/input"
+ And a UpdateAttribute processor
+ And these processor properties are set:
+ | processor name | property name | property value |
+ | UpdateAttribute | kafka_require_num_acks | 1 |
+ | UpdateAttribute | kafka_message_key | unique_message_key_123 |
And a PublishKafka processor set up to communicate with a kafka broker instance
+ And these processor properties are set:
+ | processor name | property name | property value |
+ | PublishKafka | Topic Name | test |
+ | PublishKafka | Delivery Guarantee | ${kafka_require_num_acks} |
+ | PublishKafka | Request Timeout | 12 s |
+ | PublishKafka | Message Timeout | 13 s |
+ | PublishKafka | Known Brokers | kafka-broker:${literal(9000):plus(92)} |
+ | PublishKafka | Client Name | client_no_${literal(6):multiply(7)} |
+ | PublishKafka | Kafka Key | ${kafka_message_key} |
+ | PublishKafka | retry.backoff.ms | ${literal(2):multiply(25):multiply(3)} |
+ | PublishKafka | Message Key Field | kafka.key |
And a PutFile processor with the "Directory" property set to "/tmp/output"
- And the "success" relationship of the GetFile processor is connected to the PublishKafka
+ And the "success" relationship of the GetFile processor is connected to the UpdateAttribute
+ And the "success" relationship of the UpdateAttribute processor is connected to the PublishKafka
And the "success" relationship of the PublishKafka processor is connected to the PutFile
And a kafka broker is set up in correspondence with the PublishKafka
When both instances start up
Then a flowfile with the content "test" is placed in the monitored directory in less than 60 seconds
+ And the Minifi logs contain the following message: " is 'test'" in less than 60 seconds
+ And the Minifi logs contain the following message: "PublishKafka: request.required.acks [1]" in less than 60 seconds
+ And the Minifi logs contain the following message: "PublishKafka: request.timeout.ms [12000]" in less than 60 seconds
+ And the Minifi logs contain the following message: "PublishKafka: message.timeout.ms [13000]" in less than 60 seconds
+ And the Minifi logs contain the following message: "PublishKafka: bootstrap.servers [kafka-broker:9092]" in less than 60 seconds
+ And the Minifi logs contain the following message: "PublishKafka: client.id [client_no_42]" in less than 60 seconds
+ And the Minifi logs contain the following message: "PublishKafka: Message Key [unique_message_key_123]" in less than 60 seconds
+ And the Minifi logs contain the following message: "PublishKafka: DynamicProperty: [retry.backoff.ms] -> [150]" in less than 60 seconds
+ And the Minifi logs contain the following message: "The Message Key Field property is set. This property is DEPRECATED and has no effect; please use Kafka Key instead." in less than 60 seconds
Review comment:
agree that this functionally checks for the behavior, but I feel like it is not clear from just looking at this test, what is being checked here exactly
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi-minifi-cpp] lordgamez commented on a change in pull request #1180: MINIFICPP-1530 Put Kafka tests in order
Posted by GitBox <gi...@apache.org>.
lordgamez commented on a change in pull request #1180:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1180#discussion_r730645272
##########
File path: docker/test/integration/features/kafka.feature
##########
@@ -9,15 +9,41 @@ Feature: Sending data to using Kafka streaming platform using PublishKafka
Scenario: A MiNiFi instance transfers data to a kafka broker
Given a GetFile processor with the "Input Directory" property set to "/tmp/input"
And a file with the content "test" is present in "/tmp/input"
+ And a UpdateAttribute processor
+ And these processor properties are set:
+ | processor name | property name | property value |
+ | UpdateAttribute | kafka_require_num_acks | 1 |
+ | UpdateAttribute | kafka_message_key | unique_message_key_123 |
And a PublishKafka processor set up to communicate with a kafka broker instance
+ And these processor properties are set:
+ | processor name | property name | property value |
+ | PublishKafka | Topic Name | test |
+ | PublishKafka | Delivery Guarantee | ${kafka_require_num_acks} |
+ | PublishKafka | Request Timeout | 12 s |
+ | PublishKafka | Message Timeout | 13 s |
+ | PublishKafka | Known Brokers | kafka-broker:${literal(9000):plus(92)} |
+ | PublishKafka | Client Name | client_no_${literal(6):multiply(7)} |
+ | PublishKafka | Kafka Key | ${kafka_message_key} |
+ | PublishKafka | retry.backoff.ms | ${literal(2):multiply(25):multiply(3)} |
+ | PublishKafka | Message Key Field | kafka.key |
And a PutFile processor with the "Directory" property set to "/tmp/output"
- And the "success" relationship of the GetFile processor is connected to the PublishKafka
+ And the "success" relationship of the GetFile processor is connected to the UpdateAttribute
+ And the "success" relationship of the UpdateAttribute processor is connected to the PublishKafka
And the "success" relationship of the PublishKafka processor is connected to the PutFile
And a kafka broker is set up in correspondence with the PublishKafka
When both instances start up
Then a flowfile with the content "test" is placed in the monitored directory in less than 60 seconds
+ And the Minifi logs contain the following message: " is 'test'" in less than 60 seconds
+ And the Minifi logs contain the following message: "PublishKafka: request.required.acks [1]" in less than 60 seconds
+ And the Minifi logs contain the following message: "PublishKafka: request.timeout.ms [12000]" in less than 60 seconds
+ And the Minifi logs contain the following message: "PublishKafka: message.timeout.ms [13000]" in less than 60 seconds
+ And the Minifi logs contain the following message: "PublishKafka: bootstrap.servers [kafka-broker:9092]" in less than 60 seconds
+ And the Minifi logs contain the following message: "PublishKafka: client.id [client_no_42]" in less than 60 seconds
+ And the Minifi logs contain the following message: "PublishKafka: Message Key [unique_message_key_123]" in less than 60 seconds
+ And the Minifi logs contain the following message: "PublishKafka: DynamicProperty: [retry.backoff.ms] -> [150]" in less than 60 seconds
+ And the Minifi logs contain the following message: "The Message Key Field property is set. This property is DEPRECATED and has no effect; please use Kafka Key instead." in less than 60 seconds
Review comment:
It's possible to add comment lines. Added comments for fallback behavior in 8595d4da8b750b534d49a9e718205bad991af071
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi-minifi-cpp] adamdebreceni closed pull request #1180: MINIFICPP-1530 Put Kafka tests in order
Posted by GitBox <gi...@apache.org>.
adamdebreceni closed pull request #1180:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1180
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi-minifi-cpp] adamdebreceni commented on a change in pull request #1180: MINIFICPP-1530 Put Kafka tests in order
Posted by GitBox <gi...@apache.org>.
adamdebreceni commented on a change in pull request #1180:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1180#discussion_r730636631
##########
File path: docker/test/integration/features/kafka.feature
##########
@@ -9,15 +9,41 @@ Feature: Sending data to using Kafka streaming platform using PublishKafka
Scenario: A MiNiFi instance transfers data to a kafka broker
Given a GetFile processor with the "Input Directory" property set to "/tmp/input"
And a file with the content "test" is present in "/tmp/input"
+ And a UpdateAttribute processor
+ And these processor properties are set:
+ | processor name | property name | property value |
+ | UpdateAttribute | kafka_require_num_acks | 1 |
+ | UpdateAttribute | kafka_message_key | unique_message_key_123 |
And a PublishKafka processor set up to communicate with a kafka broker instance
+ And these processor properties are set:
+ | processor name | property name | property value |
+ | PublishKafka | Topic Name | test |
+ | PublishKafka | Delivery Guarantee | ${kafka_require_num_acks} |
+ | PublishKafka | Request Timeout | 12 s |
+ | PublishKafka | Message Timeout | 13 s |
+ | PublishKafka | Known Brokers | kafka-broker:${literal(9000):plus(92)} |
+ | PublishKafka | Client Name | client_no_${literal(6):multiply(7)} |
+ | PublishKafka | Kafka Key | ${kafka_message_key} |
+ | PublishKafka | retry.backoff.ms | ${literal(2):multiply(25):multiply(3)} |
+ | PublishKafka | Message Key Field | kafka.key |
And a PutFile processor with the "Directory" property set to "/tmp/output"
- And the "success" relationship of the GetFile processor is connected to the PublishKafka
+ And the "success" relationship of the GetFile processor is connected to the UpdateAttribute
+ And the "success" relationship of the UpdateAttribute processor is connected to the PublishKafka
And the "success" relationship of the PublishKafka processor is connected to the PutFile
And a kafka broker is set up in correspondence with the PublishKafka
When both instances start up
Then a flowfile with the content "test" is placed in the monitored directory in less than 60 seconds
+ And the Minifi logs contain the following message: " is 'test'" in less than 60 seconds
+ And the Minifi logs contain the following message: "PublishKafka: request.required.acks [1]" in less than 60 seconds
+ And the Minifi logs contain the following message: "PublishKafka: request.timeout.ms [12000]" in less than 60 seconds
+ And the Minifi logs contain the following message: "PublishKafka: message.timeout.ms [13000]" in less than 60 seconds
+ And the Minifi logs contain the following message: "PublishKafka: bootstrap.servers [kafka-broker:9092]" in less than 60 seconds
+ And the Minifi logs contain the following message: "PublishKafka: client.id [client_no_42]" in less than 60 seconds
+ And the Minifi logs contain the following message: "PublishKafka: Message Key [unique_message_key_123]" in less than 60 seconds
+ And the Minifi logs contain the following message: "PublishKafka: DynamicProperty: [retry.backoff.ms] -> [150]" in less than 60 seconds
+ And the Minifi logs contain the following message: "The Message Key Field property is set. This property is DEPRECATED and has no effect; please use Kafka Key instead." in less than 60 seconds
Review comment:
is it possible to add comments in feature files? (or is that an antipattern maybe)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org