You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@druid.apache.org by GitBox <gi...@apache.org> on 2020/02/27 04:06:52 UTC
[GitHub] [druid] abossert opened a new pull request #9428: Adding request
logging support to the Kafka-Emitter extension
abossert opened a new pull request #9428: Adding request logging support to the Kafka-Emitter extension
URL: https://github.com/apache/druid/pull/9428
This is an effort to revive #6493 , which seemed to have fizzled out some time ago. I managed to miss #6493 during my initial work...I had created a very similar implementation that I have been using locally that simply changed the `toMap()` method of `DefaultRequestLogEvent` to output a map of the event properties, rather than an empty map.
I have attempted to address the concerns that @gianm raised in his review of #6493 in that I removed the `request.topic` as a mandatory field and I also implemented the simpler code that @gianm suggested in the `org.apache.druid.emitter.kafka.KafkaEmitter` class.
This PR has:
- [x] been self-reviewed.
- [x] added documentation for new or modified features or behaviors.
- [ ] added Javadocs for most classes and all non-trivial methods. Linked related entities via Javadoc links.
- [ ] added or updated version, license, or notice information in [licenses.yaml](https://github.com/apache/druid/blob/master/licenses.yaml)
- [x] added comments explaining the "why" and the intent of the code wherever would not be obvious for an unfamiliar reader.
- [x] added unit tests or modified existing tests to cover new code paths.
- [ ] added integration tests.
- [x] been tested in a test Druid cluster.
<hr>
##### Key changed/added classes in this PR
* `org.apache.druid.emitter.kafka.KafkaEmitterConfigTest`
* `org.apache.druid.emitter.kafka.KafkaEmitterConfig`
* `org.apache.druid.emitter.kafka.KafkaEmitter`
* `org.apache.druid.emitter.kafka.KafkaEmitter`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] abossert commented on issue #9428: Adding request logging
support to the Kafka-Emitter extension
Posted by GitBox <gi...@apache.org>.
abossert commented on issue #9428: Adding request logging support to the Kafka-Emitter extension
URL: https://github.com/apache/druid/pull/9428#issuecomment-592011644
I saw that the Travis CI job failed. When I checked the reason for the error, it seems like it might be some sort of problem not associated with my code, perhaps? The error relates to a failure to bind to a port for Kafka. Here is the relevant section of the Travis log. Perhaps you can re-run the Travis job?
```
[ERROR] Tests run: 2, Failures: 0, Errors: 2, Skipped: 0, Time elapsed: 3.175 s <<< FAILURE! - in org.apache.druid.indexing.kafka.KafkaIndexTaskTest
[ERROR] org.apache.druid.indexing.kafka.KafkaIndexTaskTest Time elapsed: 3.154 s <<< ERROR!
org.apache.kafka.common.KafkaException: Socket server failed to bind to 0.0.0.0:10010: Address already in use.
at kafka.network.Acceptor.openServerSocket(SocketServer.scala:573)
at kafka.network.Acceptor.<init>(SocketServer.scala:451)
at kafka.network.SocketServer.createAcceptor(SocketServer.scala:245)
at kafka.network.SocketServer.$anonfun$createDataPlaneAcceptorsAndProcessors$1(SocketServer.scala:215)
at kafka.network.SocketServer.$anonfun$createDataPlaneAcceptorsAndProcessors$1$adapted(SocketServer.scala:214)
at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:58)
at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:51)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)
at kafka.network.SocketServer.createDataPlaneAcceptorsAndProcessors(SocketServer.scala:214)
at kafka.network.SocketServer.startup(SocketServer.scala:114)
at kafka.server.KafkaServer.startup(KafkaServer.scala:253)
at org.apache.druid.indexing.kafka.test.TestBroker.start(TestBroker.java:93)
at org.apache.druid.indexing.kafka.KafkaIndexTaskTest.setupClass(KafkaIndexTaskTest.java:271)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:24)
at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
at org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345)
at org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126)
at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418)
Caused by: java.net.BindException: Address already in use
at sun.nio.ch.Net.bind0(Native Method)
at sun.nio.ch.Net.bind(Net.java:433)
at sun.nio.ch.Net.bind(Net.java:425)
at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:220)
at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:85)
at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:78)
at kafka.network.Acceptor.openServerSocket(SocketServer.scala:569)
... 30 more
[ERROR] org.apache.druid.indexing.kafka.KafkaIndexTaskTest Time elapsed: 3.168 s <<< ERROR!
java.lang.NullPointerException
at org.apache.druid.indexing.kafka.KafkaIndexTaskTest.tearDownClass(KafkaIndexTaskTest.java:313)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:33)
at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
at org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345)
at org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126)
at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418)
[INFO]
[INFO] Results:
[INFO]
[ERROR] Errors:
[ERROR] org.apache.druid.indexing.kafka.KafkaIndexTaskTest.org.apache.druid.indexing.kafka.KafkaIndexTaskTest
[ERROR] Run 1: KafkaIndexTaskTest.setupClass:271 ยป Kafka Socket server failed to bind to 0.0....
[ERROR] Run 2: KafkaIndexTaskTest.tearDownClass:313 NullPointer
[INFO]
[INFO]
[ERROR] Tests run: 189, Failures: 0, Errors: 1, Skipped: 0
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary for druid-indexing-hadoop 0.18.0-SNAPSHOT:
[INFO]
[INFO] druid-indexing-hadoop .............................. SUCCESS [01:00 min]
[INFO] druid-indexing-service ............................. SUCCESS [07:56 min]
[INFO] druid-kafka-indexing-service ....................... FAILURE [01:50 min]
[INFO] druid-kinesis-indexing-service ..................... SKIPPED
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 10:48 min
[INFO] Finished at: 2020-02-27T04:24:35Z
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org