You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@druid.apache.org by GitBox <gi...@apache.org> on 2020/04/21 03:40:10 UTC
[GitHub] [druid] maytasm opened a new pull request #9729: Test reading from empty kafka/kinesis partitions
maytasm opened a new pull request #9729:
URL: https://github.com/apache/druid/pull/9729
Add tests for reading from empty kafka/kinesis partitions
### Description
Add tests for reading from empty kafka/kinesis partitions. The new tests discovers that Kafka is working fine when reading from empty kafka partitions, and thus, supervisor shows healthy and tasks are running without failing when one or more partition is empty. However, Kinesis supervisor is showing unhealthy and task are not running when one or more partition is empty. This should be fix in a separate PR and subsequently these tests for Kinesis will have to be modify.
This PR has:
- [x] been self-reviewed.
- [ ] added documentation for new or modified features or behaviors.
- [ ] added Javadocs for most classes and all non-trivial methods. Linked related entities via Javadoc links.
- [ ] added or updated version, license, or notice information in [licenses.yaml](https://github.com/apache/druid/blob/master/licenses.yaml)
- [ ] added comments explaining the "why" and the intent of the code wherever would not be obvious for an unfamiliar reader.
- [x] added unit tests or modified existing tests to cover new code paths.
- [ ] added integration tests.
- [ ] been tested in a test Druid cluster.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] jihoonson commented on a change in pull request #9729: Test reading from empty kafka/kinesis partitions
Posted by GitBox <gi...@apache.org>.
jihoonson commented on a change in pull request #9729:
URL: https://github.com/apache/druid/pull/9729#discussion_r414264344
##########
File path: extensions-core/kinesis-indexing-service/src/main/java/org/apache/druid/indexing/kinesis/KinesisRecordSupplier.java
##########
@@ -780,15 +780,14 @@ private String getSequenceNumberInternal(StreamPartition<String> partition, Shar
private String getSequenceNumberInternal(StreamPartition<String> partition, String shardIterator)
{
long timeoutMillis = System.currentTimeMillis() + fetchSequenceNumberTimeout;
+ GetRecordsResult recordsResult = null;
Review comment:
This change doesn't seem doing anything. Is there some reason?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] maytasm commented on a change in pull request #9729: Test reading from empty kafka/kinesis partitions
Posted by GitBox <gi...@apache.org>.
maytasm commented on a change in pull request #9729:
URL: https://github.com/apache/druid/pull/9729#discussion_r414297759
##########
File path: extensions-core/kinesis-indexing-service/src/main/java/org/apache/druid/indexing/kinesis/KinesisRecordSupplier.java
##########
@@ -780,15 +780,14 @@ private String getSequenceNumberInternal(StreamPartition<String> partition, Shar
private String getSequenceNumberInternal(StreamPartition<String> partition, String shardIterator)
{
long timeoutMillis = System.currentTimeMillis() + fetchSequenceNumberTimeout;
+ GetRecordsResult recordsResult = null;
Review comment:
I want to access recordsResult outside of the while loop (so that I can print the millisBehindLatest in the warn logging line 834). Hence, I moved variable declaration of recordsResult to outside the while loop and we can check for null in the warn logging. If it is not null (meaning it was assign in the while loop when we getRecords from Kinesis) then we can log the millisBehindLatest. This will help us be sure that timeout is due to no data or not
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] jihoonson commented on a change in pull request #9729: Test reading from empty kafka/kinesis partitions
Posted by GitBox <gi...@apache.org>.
jihoonson commented on a change in pull request #9729:
URL: https://github.com/apache/druid/pull/9729#discussion_r414747884
##########
File path: extensions-core/kinesis-indexing-service/src/main/java/org/apache/druid/indexing/kinesis/KinesisRecordSupplier.java
##########
@@ -780,15 +780,14 @@ private String getSequenceNumberInternal(StreamPartition<String> partition, Shar
private String getSequenceNumberInternal(StreamPartition<String> partition, String shardIterator)
{
long timeoutMillis = System.currentTimeMillis() + fetchSequenceNumberTimeout;
+ GetRecordsResult recordsResult = null;
Review comment:
Got it. Thanks.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] maytasm commented on pull request #9729: Test reading from empty kafka/kinesis partitions
Posted by GitBox <gi...@apache.org>.
maytasm commented on pull request #9729:
URL: https://github.com/apache/druid/pull/9729#issuecomment-618806425
> > However, Kinesis supervisor is showing unhealthy and task are not running when one or more partition is empty. This should be fix in a separate PR and subsequently these tests for Kinesis will have to be modify.
>
> Can you please open an issue about it on Github?
https://github.com/apache/druid/issues/9763
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org