You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@druid.apache.org by GitBox <gi...@apache.org> on 2020/01/06 22:35:38 UTC
[GitHub] [druid] amishasthana opened a new issue #9133: Kinesis Indexing
throwing exception
amishasthana opened a new issue #9133: Kinesis Indexing throwing exception
URL: https://github.com/apache/druid/issues/9133
I am using Druid and druid kinesis plugin to get the kinesis stream data into my druid instance.
I am running "apache-druid-0.16.1-incubating" version.
Server started on local machine as micro service quick start.
The Kinesis plugin has been enabled.
I create a Json template, like this and create the Data store for kinesis.
{
"type": "kinesis",
"dataSchema": {
"dataSource": "<A data source name>",
"parser": {
"type": "string",
"parseSpec": {
"format": "json",
"timestampSpec": {
"column": "<a column of type timestamp>",
"format": "auto"
},
"dimensionsSpec": {
"dimensions": [],
"dimensionExclusions": [
"timestamp"
]
}
}
},
"granularitySpec": {
"type": "uniform",
"segmentGranularity": "HOUR",
"queryGranularity": "NONE"
}
},
"tuningConfig": {
"type": "kinesis",
"maxRowsPerSegment": 5000000,
"resetOffsetAutomatically": "true",
"useEarliestOffset" : "false"
},
"ioConfig": {
"stream": "stream-name which exist in AWS",
"endpoint": "kinesis.us-east-1.amazonaws.com",
"taskCount": 1,
"replicas": 1,
"taskDuration": "PT1H",
"recordsPerFetch": 2000,
"fetchDelayMillis": 1000
}
}
The data source gets created.
I have added the following aws key/secret key in file "conf/druid/single-server/micro-quickstart/_common/common.runtime.properties".
druid.kinesis.accessKey=<The key>
druid.kinesis.secretKey=<The secret key>
Do note that with these values I can connect from this local machine to said AWS kinesis stream using KCL classes.
When I start I see the following exceptions:
### 2020-01-06T21:17:03,581 WARN [KinesisSupervisor-am-kinesis-matched-v7] org.apache.druid.indexing.kinesis.KinesisRecordSupplier - timed out while trying to fetch position for shard[shardId-000000000000], likely no more records in shard
2020-01-06T21:17:03,587 WARN [KinesisSupervisor-am-kinesis-matched-v7] org.apache.druid.indexing.seekablestream.supervisor.SeekableStreamSupervisor - Exception in supervisor run loop for dataSource [am-kinesis-matched-v7]
org.apache.druid.java.util.common.ISE: unable to fetch sequence number for partition[shardId-000000000000] from stream
at org.apache.druid.indexing.seekablestream.supervisor.SeekableStreamSupervisor.getOffsetFromStorageForPartition(SeekableStreamSupervisor.java:2559) ~[druid-indexing-service-0.16.1-incubating.jar:0.16.1-incubating]
at org.apache.druid.indexing.seekablestream.supervisor.SeekableStreamSupervisor.generateStartingSequencesForPartitionGroup(SeekableStreamSupervisor.java:2499) ~[druid-indexing-service-0.16.1-incubating.jar:0.16.1-incubating]
at org.apache.druid.indexing.seekablestream.supervisor.SeekableStreamSupervisor.createNewTasks(SeekableStreamSupervisor.java:2397) ~[druid-indexing-service-0.16.1-incubating.jar:0.16.1-incubating]
at org.apache.druid.indexing.seekablestream.supervisor.SeekableStreamSupervisor.runInternal(SeekableStreamSupervisor.java:1066) ~[druid-indexing-service-0.16.1-incubating.jar:0.16.1-incubating]
at org.apache.druid.indexing.seekablestream.supervisor.SeekableStreamSupervisor$RunNotice.handle(SeekableStreamSupervisor.java:293) ~[druid-indexing-service-0.16.1-incubating.jar:0.16.1-incubating]
at org.apache.druid.indexing.seekablestream.supervisor.SeekableStreamSupervisor.lambda$tryInit$3(SeekableStreamSupervisor.java:749) ~[druid-indexing-service-0.16.1-incubating.jar:0.16.1-incubating]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_232]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_232]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_232]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_232]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_232]
One another point to note, when there are new records in stream, I do see that Druid task is getting initiated. In other words it seem to be connected and aware of changes in kinesis stream.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] amishasthana commented on issue #9133: Kinesis Indexing
throwing exception
Posted by GitBox <gi...@apache.org>.
amishasthana commented on issue #9133: Kinesis Indexing throwing exception
URL: https://github.com/apache/druid/issues/9133#issuecomment-571355215
This error is repeating continuously:
2020-01-06T23:05:23,157 INFO [Coordinator-Exec--0] org.apache.druid.server.coordinator.helper.DruidCoordinatorSegmentInfoLoader - Starting coordination. Getting used segments.
2020-01-06T23:05:23,157 INFO [Coordinator-Exec--0] org.apache.druid.server.coordinator.helper.DruidCoordinatorSegmentInfoLoader - Found [0] used segments.
2020-01-06T23:05:23,158 INFO [Coordinator-Exec--0] org.apache.druid.server.coordinator.ReplicationThrottler - [_default_tier]: Replicant create queue is empty.
2020-01-06T23:05:23,158 INFO [Coordinator-Exec--0] org.apache.druid.server.coordinator.helper.DruidCoordinatorBalancer - Metadata segments are not available. Cannot balance.
2020-01-06T23:05:23,158 INFO [Coordinator-Exec--0] org.apache.druid.server.coordinator.helper.DruidCoordinatorLogger - Load Queues:
2020-01-06T23:05:23,158 INFO [Coordinator-Exec--0] org.apache.druid.server.coordinator.helper.DruidCoordinatorLogger - Server[localhost:8083, historical, _default_tier] has 0 left to load, 0 left to drop, 0 bytes queued, 0 bytes served.
2020-01-06T23:05:28,162 INFO [Coordinator-Exec--0] org.apache.druid.server.coordinator.helper.DruidCoordinatorSegmentInfoLoader - Starting coordination. Getting used segments.
2020-01-06T23:05:28,162 INFO [Coordinator-Exec--0] org.apache.druid.server.coordinator.helper.DruidCoordinatorSegmentInfoLoader - Found [0] used segments.
2020-01-06T23:05:28,163 INFO [Coordinator-Exec--0] org.apache.druid.server.coordinator.ReplicationThrottler - [_default_tier]: Replicant create queue is empty.
2020-01-06T23:05:28,163 INFO [Coordinator-Exec--0] org.apache.druid.server.coordinator.helper.DruidCoordinatorBalancer - Metadata segments are not available. Cannot balance.
2020-01-06T23:05:28,163 INFO [Coordinator-Exec--0] org.apache.druid.server.coordinator.helper.DruidCoordinatorLogger - Load Queues:
2020-01-06T23:05:28,163 INFO [Coordinator-Exec--0] org.apache.druid.server.coordinator.helper.DruidCoordinatorLogger - Server[localhost:8083, historical, _default_tier] has 0 left to load, 0 left to drop, 0 bytes queued, 0 bytes served.
2020-01-06T23:05:29,650 WARN [KinesisSupervisor-am-kinesis-matched-v8] org.apache.druid.indexing.kinesis.KinesisRecordSupplier - timed out while trying to fetch position for shard[shardId-000000000000], likely no more records in shard
2020-01-06T23:05:29,651 WARN [KinesisSupervisor-am-kinesis-matched-v8] org.apache.druid.indexing.seekablestream.supervisor.SeekableStreamSupervisor - Exception in supervisor run loop for dataSource [am-kinesis-matched-v8]
org.apache.druid.java.util.common.ISE: unable to fetch sequence number for partition[shardId-000000000000] from stream
at org.apache.druid.indexing.seekablestream.supervisor.SeekableStreamSupervisor.getOffsetFromStorageForPartition(SeekableStreamSupervisor.java:2559) ~[druid-indexing-service-0.16.1-incubating.jar:0.16.1-incubating]
at org.apache.druid.indexing.seekablestream.supervisor.SeekableStreamSupervisor.generateStartingSequencesForPartitionGroup(SeekableStreamSupervisor.java:2499) ~[druid-indexing-service-0.16.1-incubating.jar:0.16.1-incubating]
at org.apache.druid.indexing.seekablestream.supervisor.SeekableStreamSupervisor.createNewTasks(SeekableStreamSupervisor.java:2397) ~[druid-indexing-service-0.16.1-incubating.jar:0.16.1-incubating]
at org.apache.druid.indexing.seekablestream.supervisor.SeekableStreamSupervisor.runInternal(SeekableStreamSupervisor.java:1066) ~[druid-indexing-service-0.16.1-incubating.jar:0.16.1-incubating]
at org.apache.druid.indexing.seekablestream.supervisor.SeekableStreamSupervisor$RunNotice.handle(SeekableStreamSupervisor.java:293) ~[druid-indexing-service-0.16.1-incubating.jar:0.16.1-incubating]
at org.apache.druid.indexing.seekablestream.supervisor.SeekableStreamSupervisor.lambda$tryInit$3(SeekableStreamSupervisor.java:749) ~[druid-indexing-service-0.16.1-incubating.jar:0.16.1-incubating]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_232]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_232]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_232]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_232]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_232]
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] jon-wei commented on issue #9133: Kinesis Indexing throwing
exception
Posted by GitBox <gi...@apache.org>.
jon-wei commented on issue #9133: Kinesis Indexing throwing exception
URL: https://github.com/apache/druid/issues/9133#issuecomment-572232556
For this error:
> 2020-01-06T23:05:29,650 WARN [KinesisSupervisor-am-kinesis-matched-v8] org.apache.druid.indexing.kinesis.KinesisRecordSupplier - timed out while trying to fetch position for shard[shardId-000000000000], likely no more records in shard
This is ultimately handled by `KinesisRecordSupplier.getSequenceNumberInternal`, if this timeout occurs it means the supervisor was able to get a shard iterator but never saw any records from that shard iterator within the configured `fetchSequenceNumberTimeout` (set in the tuningConfig).
Some suggestions:
- Try increasing `fetchSequenceNumberTimeout`
- Try adding more debug logs to that method to see if you can get any more info on why there's a timeout.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] jon-wei commented on issue #9133: Kinesis Indexing throwing
exception
Posted by GitBox <gi...@apache.org>.
jon-wei commented on issue #9133: Kinesis Indexing throwing exception
URL: https://github.com/apache/druid/issues/9133#issuecomment-571347030
This is a known issue, you'd need to ensure some records are written to that shard: https://github.com/apache/druid/issues/7600
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] amishasthana commented on issue #9133: Kinesis Indexing
throwing exception
Posted by GitBox <gi...@apache.org>.
amishasthana commented on issue #9133: Kinesis Indexing throwing exception
URL: https://github.com/apache/druid/issues/9133#issuecomment-571348401
@jon-wei I do have bunch of data in that shard. If I open KCL I can read bunch of these. Or did you mean something else?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] amishasthana commented on issue #9133: Kinesis Indexing
throwing exception
Posted by GitBox <gi...@apache.org>.
amishasthana commented on issue #9133: Kinesis Indexing throwing exception
URL: https://github.com/apache/druid/issues/9133#issuecomment-571354816
Continuously pumping data in stream. Same issue.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] amishasthana commented on issue #9133: Kinesis Indexing
throwing exception
Posted by GitBox <gi...@apache.org>.
amishasthana commented on issue #9133: Kinesis Indexing throwing exception
URL: https://github.com/apache/druid/issues/9133#issuecomment-571805578
I have been trying to build from main. However being new its not clear to me how to run it. I have not been able to build a distributed tar.gz type which will be equivalent. Is there some docs which I should follow.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] amishasthana commented on issue #9133: Kinesis Indexing
throwing exception
Posted by GitBox <gi...@apache.org>.
amishasthana commented on issue #9133: Kinesis Indexing throwing exception
URL: https://github.com/apache/druid/issues/9133#issuecomment-572218527
Was able to build and try. Same issue. Its still giving the same error.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] jon-wei commented on issue #9133: Kinesis Indexing throwing
exception
Posted by GitBox <gi...@apache.org>.
jon-wei commented on issue #9133: Kinesis Indexing throwing exception
URL: https://github.com/apache/druid/issues/9133#issuecomment-571355571
Can you try using the current master and see if the same thing happens?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] jon-wei commented on issue #9133: Kinesis Indexing throwing
exception
Posted by GitBox <gi...@apache.org>.
jon-wei commented on issue #9133: Kinesis Indexing throwing exception
URL: https://github.com/apache/druid/issues/9133#issuecomment-571817176
There are build instructions at https://druid.apache.org/docs/latest/development/build.html
`mvn clean install -DskipTests -Pdist` should work, the tarball will be under `distribution/target`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] jon-wei commented on issue #9133: Kinesis Indexing throwing
exception
Posted by GitBox <gi...@apache.org>.
jon-wei commented on issue #9133: Kinesis Indexing throwing exception
URL: https://github.com/apache/druid/issues/9133#issuecomment-571349266
How often is data written to that shard? You may be hitting the third issue fixed by this PR: https://github.com/apache/druid/pull/8671
> Adjusts checkSequenceNumberAvailability (renamed to checkOffsetAvailability), only the earliest available offset needs to be compared against the target offset. Because the "LATEST" Kinesis shard iterator only provides access to the seq no after the most recent record (and there is no mode to get the latest readable record), the "get latest offset" operation in Kinesis runs in a polling loop. When data is not being continuously written to a shard, this can result in a Kinesis supervisor making extremely slow progress as it waits for the polling loop to timeout for each shard that has no recent data. Removing the latest offset check restricts this problem situation to only when a Kinesis supervisor first discovers a shard that's empty (after data is written to that shard once, the earliest seq no check can return immediately)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org