You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Claudia Wegmann <c....@kasasi.de> on 2018/12/17 10:10:59 UTC

Configuration of log compaction

Dear kafka users,

I've got a problem on one of my kafka clusters. I use this cluster with kafka streams applications. Some of this stream apps use a kafka state store. Therefore a changelog topic is created for those stores with cleanup policy "compact". One of these topics is running wild for some time now and seems to grow indefinitely. When I check the  log file of the first segment, there is a lot of data in it, that should have been compacted already.

So I guess I did not configure everything correctly for log compaction to work as expected. What config parameters do have influence on log compaction? And how to set them, when I want data older than 4 hours to be compacted?

Thanks in advance.

Best,
Claudia

RE: Configuration of log compaction

Posted by Komal Babu <ko...@qbs.co.uk>.
Hi,

Can you please remove me from this mail box.

Regards, 

Komal Babu
Sourcing Administrator
komal@qbs.co.uk
020 8733 7139
http://www.qbsd.co.uk

QBS Distribution, 
7 Wharfside, 
Rosemont Road, 
Wembley, HA0 4QB
United Kingdom


Banner

-----Original Message-----
From: Claudia Wegmann <c....@kasasi.de> 
Sent: 17 December 2018 14:18
To: users@kafka.apache.org
Subject: AW: Configuration of log compaction

Hi,

thanks for the quick response.

My problem is not, that no new segments are created, but that segments with old data do not get compacted.
I had to restart one broker because there was no diskspace left. After recreating all indexes etc. the broker recognized the old data and compacted it correctly. I had to restart all other brokers of the cluster, too, for them to also recognize the old data and start compacting.

So I guess, before restarting the brokers where to busy to compact/delete old data? Is there a configuration to ensure compaction after a certain amount of time or something?

Best,
Claudia

-----Ursprüngliche Nachricht-----
Von: Spico Florin <sp...@gmail.com>
Gesendet: Montag, 17. Dezember 2018 14:28
An: users@kafka.apache.org
Betreff: Re: Configuration of log compaction

Hello!
  Please check whether the segment.ms configuration on topic will help you to solve your problem.

https://kafka.apache.org/documentation/

https://stackoverflow.com/questions/41048041/kafka-deletes-segments-even-before-segment-size-is-reached

Regards,
 Florin

segment.ms This configuration controls the period of time after which Kafka will force the log to roll even if the segment file isn't full to ensure that retention can delete or compact old data. long 604800000 [1,...] log.roll.ms medium

On Mon, Dec 17, 2018 at 12:28 PM Claudia Wegmann <c....@kasasi.de>
wrote:

> Dear kafka users,
>
> I've got a problem on one of my kafka clusters. I use this cluster 
> with kafka streams applications. Some of this stream apps use a kafka 
> state store. Therefore a changelog topic is created for those stores 
> with cleanup policy "compact". One of these topics is running wild for 
> some time now and seems to grow indefinitely. When I check the  log 
> file of the first segment, there is a lot of data in it, that should 
> have been compacted already.
>
> So I guess I did not configure everything correctly for log compaction 
> to work as expected. What config parameters do have influence on log 
> compaction? And how to set them, when I want data older than 4 hours 
> to be compacted?
>
> Thanks in advance.
>
> Best,
> Claudia
>

AW: Configuration of log compaction

Posted by Claudia Wegmann <c....@kasasi.de>.
Hi kafka users,

I still have the problem of the log cleaner dying regularly. Yesterday I restarted 3 of my brokers in a 5-broker cluster. On two of them the log cleaner threads already died again before they were able to clean up most of the old data.

A general question in this regard:
1.) What is the exact purpose of the log cleaner? Is it just responsible for cleaning up internal topics?
Eventhough the log cleaner shut down, the broker still logs lines such as
"[Log partition=resultGatewayDataDispatcher-11, dir=/data/kafka-logs] Found deletable segments with base offsets [42976352] due to retention time 14400000ms breach (kafka.log.Log)"
and then cleans the stated segments.
But internal topics (for state stores) or the consumer-offset topic do not get compacted/cleaned up.

2.) What can I do to stabilize the log cleaning? Is it easier for the log cleaner to clean up small segments but many? Up until now I use the default configurations for log segment size. Should I reduce it?

Anyone any other ideas?

Best,
Claudia


-----Ursprüngliche Nachricht-----
Von: Claudia Wegmann <c....@kasasi.de> 
Gesendet: Mittwoch, 19. Dezember 2018 08:50
An: users@kafka.apache.org
Betreff: AW: Configuration of log compaction

Hi Liam,

My brokers are running in version 2.0.0 Restarting was not a problem, it just seemed "a lot" for just restarting the log cleaner. I will think about piping the logs of the log cleaner to elk. And I will definitely keep an eye on the logs to see if the error occurs again.

Thanks again!

Best,
Claudia

-----Ursprüngliche Nachricht-----
Von: Liam Clarke <li...@adscale.co.nz>
Gesendet: Dienstag, 18. Dezember 2018 23:05
An: users@kafka.apache.org
Betreff: Re: Configuration of log compaction

Kia ora Claudia,

What version Kafka are you running? I can't find the specified code in Kafka 1.0.0.

With regards to your questions:

1) I don't think so, but I'm just guessing here.
2) The log cleaner logs, if piped into something like Elasticsearch could be used to monitor it?
3) No, it's a thread started by the broker. I imagine you're using an old broker version, around 0.8.2 perhaps? We've found that rolling broker restarts with 0.11 are rather easy and not to be feared.

Kind regards,

Liam Clarke

On Tue, Dec 18, 2018 at 10:43 PM Claudia Wegmann <c....@kasasi.de>
wrote:

> Hi Liam,
>
> thanks for the pointer. I found out, that the log cleaner on all kafka 
> brokers died with the following error message:
>
> [2018-12-04 15:33:24,886] INFO Cleaner 0: Caught segment overflow 
> error during cleaning: Detected offset overflow at offset -1 in 
> segment LogSegment(baseOffset=3605669, size=6970326)
> (kafka.log.LogCleaner)
> [2018-12-04 15:33:24,958] ERROR [kafka-log-cleaner-thread-0]: Error 
> due to
> (kafka.log.LogCleaner)
> java.lang.IllegalArgumentException: requirement failed: Split 
> operation is only permitted for segments with overflow
>         at scala.Predef$.require(Predef.scala:277)
>         at kafka.log.Log.splitOverflowedSegment(Log.scala:1873)
>         at kafka.log.Cleaner.cleanSegments(LogCleaner.scala:517)
>         at kafka.log.Cleaner.$anonfun$doClean$6(LogCleaner.scala:465)
>         at
> kafka.log.Cleaner.$anonfun$doClean$6$adapted(LogCleaner.scala:464)
>         at scala.collection.immutable.List.foreach(List.scala:389)
>         at kafka.log.Cleaner.doClean(LogCleaner.scala:464)
>         at kafka.log.Cleaner.clean(LogCleaner.scala:442)
>         at
> kafka.log.LogCleaner$CleanerThread.cleanOrSleep(LogCleaner.scala:303)
>         at kafka.log.LogCleaner$CleanerThread.doWork(LogCleaner.scala:289)
>         at
> kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:82)
> [2018-12-04 15:33:24,959] INFO [kafka-log-cleaner-thread-0]: Stopped
> (kafka.log.LogCleaner)
>
> This brought up some new questions:
> 1.) How do I avoid this error? Did I misconfigure the creation of segments?
> 2.) Is there a way to monitor the log cleaner?
> 3.) Can I restart the log cleaner manually without restarting the 
> whole broker?
>
> Thanks for your help.
>
> Best,
> Claudia
>
> -----Ursprüngliche Nachricht-----
> Von: Liam Clarke <li...@adscale.co.nz>
> Gesendet: Montag, 17. Dezember 2018 23:06
> An: users@kafka.apache.org
> Betreff: Re: Configuration of log compaction
>
> Hi Claudia,
>
> Anything useful in the log cleaner log files?
>
> Cheers,
>
> Liam Clarke
>
> On Tue, 18 Dec. 2018, 3:18 am Claudia Wegmann <c.wegmann@kasasi.de wrote:
>
> > Hi,
> >
> > thanks for the quick response.
> >
> > My problem is not, that no new segments are created, but that 
> > segments with old data do not get compacted.
> > I had to restart one broker because there was no diskspace left. 
> > After recreating all indexes etc. the broker recognized the old data 
> > and compacted it correctly. I had to restart all other brokers of 
> > the cluster, too, for them to also recognize the old data and start
> compacting.
> >
> > So I guess, before restarting the brokers where to busy to 
> > compact/delete old data? Is there a configuration to ensure 
> > compaction after a certain amount of time or something?
> >
> > Best,
> > Claudia
> >
> > -----Ursprüngliche Nachricht-----
> > Von: Spico Florin <sp...@gmail.com>
> > Gesendet: Montag, 17. Dezember 2018 14:28
> > An: users@kafka.apache.org
> > Betreff: Re: Configuration of log compaction
> >
> > Hello!
> >   Please check whether the segment.ms configuration on topic will 
> > help you to solve your problem.
> >
> > https://kafka.apache.org/documentation/
> >
> >
> > https://stackoverflow.com/questions/41048041/kafka-deletes-segments-
> > ev
> > en-before-segment-size-is-reached
> >
> > Regards,
> >  Florin
> >
> > segment.ms This configuration controls the period of time after 
> > which Kafka will force the log to roll even if the segment file 
> > isn't full to ensure that retention can delete or compact old data.
> > long
> > 604800000 [1,...] log.roll.ms medium
> >
> > On Mon, Dec 17, 2018 at 12:28 PM Claudia Wegmann 
> > <c....@kasasi.de>
> > wrote:
> >
> > > Dear kafka users,
> > >
> > > I've got a problem on one of my kafka clusters. I use this cluster 
> > > with kafka streams applications. Some of this stream apps use a 
> > > kafka state store. Therefore a changelog topic is created for 
> > > those stores with cleanup policy "compact". One of these topics is 
> > > running wild for some time now and seems to grow indefinitely.
> > > When I check the  log file of the first segment, there is a lot of 
> > > data in it, that should have been compacted already.
> > >
> > > So I guess I did not configure everything correctly for log 
> > > compaction to work as expected. What config parameters do have 
> > > influence on log compaction? And how to set them, when I want data 
> > > older than 4 hours to be compacted?
> > >
> > > Thanks in advance.
> > >
> > > Best,
> > > Claudia
> > >
> >
>

AW: Configuration of log compaction

Posted by Claudia Wegmann <c....@kasasi.de>.
Hi Liam,

My brokers are running in version 2.0.0 Restarting was not a problem, it just seemed "a lot" for just restarting the log cleaner. I will think about piping the logs of the log cleaner to elk. And I will definitely keep an eye on the logs to see if the error occurs again.

Thanks again!

Best,
Claudia

-----Ursprüngliche Nachricht-----
Von: Liam Clarke <li...@adscale.co.nz> 
Gesendet: Dienstag, 18. Dezember 2018 23:05
An: users@kafka.apache.org
Betreff: Re: Configuration of log compaction

Kia ora Claudia,

What version Kafka are you running? I can't find the specified code in Kafka 1.0.0.

With regards to your questions:

1) I don't think so, but I'm just guessing here.
2) The log cleaner logs, if piped into something like Elasticsearch could be used to monitor it?
3) No, it's a thread started by the broker. I imagine you're using an old broker version, around 0.8.2 perhaps? We've found that rolling broker restarts with 0.11 are rather easy and not to be feared.

Kind regards,

Liam Clarke

On Tue, Dec 18, 2018 at 10:43 PM Claudia Wegmann <c....@kasasi.de>
wrote:

> Hi Liam,
>
> thanks for the pointer. I found out, that the log cleaner on all kafka 
> brokers died with the following error message:
>
> [2018-12-04 15:33:24,886] INFO Cleaner 0: Caught segment overflow 
> error during cleaning: Detected offset overflow at offset -1 in 
> segment LogSegment(baseOffset=3605669, size=6970326) 
> (kafka.log.LogCleaner)
> [2018-12-04 15:33:24,958] ERROR [kafka-log-cleaner-thread-0]: Error 
> due to
> (kafka.log.LogCleaner)
> java.lang.IllegalArgumentException: requirement failed: Split 
> operation is only permitted for segments with overflow
>         at scala.Predef$.require(Predef.scala:277)
>         at kafka.log.Log.splitOverflowedSegment(Log.scala:1873)
>         at kafka.log.Cleaner.cleanSegments(LogCleaner.scala:517)
>         at kafka.log.Cleaner.$anonfun$doClean$6(LogCleaner.scala:465)
>         at
> kafka.log.Cleaner.$anonfun$doClean$6$adapted(LogCleaner.scala:464)
>         at scala.collection.immutable.List.foreach(List.scala:389)
>         at kafka.log.Cleaner.doClean(LogCleaner.scala:464)
>         at kafka.log.Cleaner.clean(LogCleaner.scala:442)
>         at
> kafka.log.LogCleaner$CleanerThread.cleanOrSleep(LogCleaner.scala:303)
>         at kafka.log.LogCleaner$CleanerThread.doWork(LogCleaner.scala:289)
>         at 
> kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:82)
> [2018-12-04 15:33:24,959] INFO [kafka-log-cleaner-thread-0]: Stopped
> (kafka.log.LogCleaner)
>
> This brought up some new questions:
> 1.) How do I avoid this error? Did I misconfigure the creation of segments?
> 2.) Is there a way to monitor the log cleaner?
> 3.) Can I restart the log cleaner manually without restarting the 
> whole broker?
>
> Thanks for your help.
>
> Best,
> Claudia
>
> -----Ursprüngliche Nachricht-----
> Von: Liam Clarke <li...@adscale.co.nz>
> Gesendet: Montag, 17. Dezember 2018 23:06
> An: users@kafka.apache.org
> Betreff: Re: Configuration of log compaction
>
> Hi Claudia,
>
> Anything useful in the log cleaner log files?
>
> Cheers,
>
> Liam Clarke
>
> On Tue, 18 Dec. 2018, 3:18 am Claudia Wegmann <c.wegmann@kasasi.de wrote:
>
> > Hi,
> >
> > thanks for the quick response.
> >
> > My problem is not, that no new segments are created, but that 
> > segments with old data do not get compacted.
> > I had to restart one broker because there was no diskspace left. 
> > After recreating all indexes etc. the broker recognized the old data 
> > and compacted it correctly. I had to restart all other brokers of 
> > the cluster, too, for them to also recognize the old data and start
> compacting.
> >
> > So I guess, before restarting the brokers where to busy to 
> > compact/delete old data? Is there a configuration to ensure 
> > compaction after a certain amount of time or something?
> >
> > Best,
> > Claudia
> >
> > -----Ursprüngliche Nachricht-----
> > Von: Spico Florin <sp...@gmail.com>
> > Gesendet: Montag, 17. Dezember 2018 14:28
> > An: users@kafka.apache.org
> > Betreff: Re: Configuration of log compaction
> >
> > Hello!
> >   Please check whether the segment.ms configuration on topic will 
> > help you to solve your problem.
> >
> > https://kafka.apache.org/documentation/
> >
> >
> > https://stackoverflow.com/questions/41048041/kafka-deletes-segments-
> > ev
> > en-before-segment-size-is-reached
> >
> > Regards,
> >  Florin
> >
> > segment.ms This configuration controls the period of time after 
> > which Kafka will force the log to roll even if the segment file 
> > isn't full to ensure that retention can delete or compact old data. 
> > long
> > 604800000 [1,...] log.roll.ms medium
> >
> > On Mon, Dec 17, 2018 at 12:28 PM Claudia Wegmann 
> > <c....@kasasi.de>
> > wrote:
> >
> > > Dear kafka users,
> > >
> > > I've got a problem on one of my kafka clusters. I use this cluster 
> > > with kafka streams applications. Some of this stream apps use a 
> > > kafka state store. Therefore a changelog topic is created for 
> > > those stores with cleanup policy "compact". One of these topics is 
> > > running wild for some time now and seems to grow indefinitely. 
> > > When I check the  log file of the first segment, there is a lot of 
> > > data in it, that should have been compacted already.
> > >
> > > So I guess I did not configure everything correctly for log 
> > > compaction to work as expected. What config parameters do have 
> > > influence on log compaction? And how to set them, when I want data 
> > > older than 4 hours to be compacted?
> > >
> > > Thanks in advance.
> > >
> > > Best,
> > > Claudia
> > >
> >
>

Re: Configuration of log compaction

Posted by Liam Clarke <li...@adscale.co.nz>.
Kia ora Claudia,

What version Kafka are you running? I can't find the specified code in
Kafka 1.0.0.

With regards to your questions:

1) I don't think so, but I'm just guessing here.
2) The log cleaner logs, if piped into something like Elasticsearch could
be used to monitor it?
3) No, it's a thread started by the broker. I imagine you're using an old
broker version, around 0.8.2 perhaps? We've found that rolling broker
restarts with 0.11 are rather easy and not to be feared.

Kind regards,

Liam Clarke

On Tue, Dec 18, 2018 at 10:43 PM Claudia Wegmann <c....@kasasi.de>
wrote:

> Hi Liam,
>
> thanks for the pointer. I found out, that the log cleaner on all kafka
> brokers died with the following error message:
>
> [2018-12-04 15:33:24,886] INFO Cleaner 0: Caught segment overflow error
> during cleaning: Detected offset overflow at offset -1 in segment
> LogSegment(baseOffset=3605669, size=6970326) (kafka.log.LogCleaner)
> [2018-12-04 15:33:24,958] ERROR [kafka-log-cleaner-thread-0]: Error due to
> (kafka.log.LogCleaner)
> java.lang.IllegalArgumentException: requirement failed: Split operation is
> only permitted for segments with overflow
>         at scala.Predef$.require(Predef.scala:277)
>         at kafka.log.Log.splitOverflowedSegment(Log.scala:1873)
>         at kafka.log.Cleaner.cleanSegments(LogCleaner.scala:517)
>         at kafka.log.Cleaner.$anonfun$doClean$6(LogCleaner.scala:465)
>         at
> kafka.log.Cleaner.$anonfun$doClean$6$adapted(LogCleaner.scala:464)
>         at scala.collection.immutable.List.foreach(List.scala:389)
>         at kafka.log.Cleaner.doClean(LogCleaner.scala:464)
>         at kafka.log.Cleaner.clean(LogCleaner.scala:442)
>         at
> kafka.log.LogCleaner$CleanerThread.cleanOrSleep(LogCleaner.scala:303)
>         at kafka.log.LogCleaner$CleanerThread.doWork(LogCleaner.scala:289)
>         at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:82)
> [2018-12-04 15:33:24,959] INFO [kafka-log-cleaner-thread-0]: Stopped
> (kafka.log.LogCleaner)
>
> This brought up some new questions:
> 1.) How do I avoid this error? Did I misconfigure the creation of segments?
> 2.) Is there a way to monitor the log cleaner?
> 3.) Can I restart the log cleaner manually without restarting the whole
> broker?
>
> Thanks for your help.
>
> Best,
> Claudia
>
> -----Ursprüngliche Nachricht-----
> Von: Liam Clarke <li...@adscale.co.nz>
> Gesendet: Montag, 17. Dezember 2018 23:06
> An: users@kafka.apache.org
> Betreff: Re: Configuration of log compaction
>
> Hi Claudia,
>
> Anything useful in the log cleaner log files?
>
> Cheers,
>
> Liam Clarke
>
> On Tue, 18 Dec. 2018, 3:18 am Claudia Wegmann <c.wegmann@kasasi.de wrote:
>
> > Hi,
> >
> > thanks for the quick response.
> >
> > My problem is not, that no new segments are created, but that segments
> > with old data do not get compacted.
> > I had to restart one broker because there was no diskspace left. After
> > recreating all indexes etc. the broker recognized the old data and
> > compacted it correctly. I had to restart all other brokers of the
> > cluster, too, for them to also recognize the old data and start
> compacting.
> >
> > So I guess, before restarting the brokers where to busy to
> > compact/delete old data? Is there a configuration to ensure compaction
> > after a certain amount of time or something?
> >
> > Best,
> > Claudia
> >
> > -----Ursprüngliche Nachricht-----
> > Von: Spico Florin <sp...@gmail.com>
> > Gesendet: Montag, 17. Dezember 2018 14:28
> > An: users@kafka.apache.org
> > Betreff: Re: Configuration of log compaction
> >
> > Hello!
> >   Please check whether the segment.ms configuration on topic will help
> > you to solve your problem.
> >
> > https://kafka.apache.org/documentation/
> >
> >
> > https://stackoverflow.com/questions/41048041/kafka-deletes-segments-ev
> > en-before-segment-size-is-reached
> >
> > Regards,
> >  Florin
> >
> > segment.ms This configuration controls the period of time after which
> > Kafka will force the log to roll even if the segment file isn't full
> > to ensure that retention can delete or compact old data. long
> > 604800000 [1,...] log.roll.ms medium
> >
> > On Mon, Dec 17, 2018 at 12:28 PM Claudia Wegmann <c....@kasasi.de>
> > wrote:
> >
> > > Dear kafka users,
> > >
> > > I've got a problem on one of my kafka clusters. I use this cluster
> > > with kafka streams applications. Some of this stream apps use a
> > > kafka state store. Therefore a changelog topic is created for those
> > > stores with cleanup policy "compact". One of these topics is running
> > > wild for some time now and seems to grow indefinitely. When I check
> > > the  log file of the first segment, there is a lot of data in it,
> > > that should have been compacted already.
> > >
> > > So I guess I did not configure everything correctly for log
> > > compaction to work as expected. What config parameters do have
> > > influence on log compaction? And how to set them, when I want data
> > > older than 4 hours to be compacted?
> > >
> > > Thanks in advance.
> > >
> > > Best,
> > > Claudia
> > >
> >
>

AW: Configuration of log compaction

Posted by Claudia Wegmann <c....@kasasi.de>.
Hi Liam,

thanks for the pointer. I found out, that the log cleaner on all kafka brokers died with the following error message:

[2018-12-04 15:33:24,886] INFO Cleaner 0: Caught segment overflow error during cleaning: Detected offset overflow at offset -1 in segment LogSegment(baseOffset=3605669, size=6970326) (kafka.log.LogCleaner)
[2018-12-04 15:33:24,958] ERROR [kafka-log-cleaner-thread-0]: Error due to (kafka.log.LogCleaner)
java.lang.IllegalArgumentException: requirement failed: Split operation is only permitted for segments with overflow
        at scala.Predef$.require(Predef.scala:277)
        at kafka.log.Log.splitOverflowedSegment(Log.scala:1873)
        at kafka.log.Cleaner.cleanSegments(LogCleaner.scala:517)
        at kafka.log.Cleaner.$anonfun$doClean$6(LogCleaner.scala:465)
        at kafka.log.Cleaner.$anonfun$doClean$6$adapted(LogCleaner.scala:464)
        at scala.collection.immutable.List.foreach(List.scala:389)
        at kafka.log.Cleaner.doClean(LogCleaner.scala:464)
        at kafka.log.Cleaner.clean(LogCleaner.scala:442)
        at kafka.log.LogCleaner$CleanerThread.cleanOrSleep(LogCleaner.scala:303)
        at kafka.log.LogCleaner$CleanerThread.doWork(LogCleaner.scala:289)
        at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:82)
[2018-12-04 15:33:24,959] INFO [kafka-log-cleaner-thread-0]: Stopped (kafka.log.LogCleaner)

This brought up some new questions:
1.) How do I avoid this error? Did I misconfigure the creation of segments?
2.) Is there a way to monitor the log cleaner?
3.) Can I restart the log cleaner manually without restarting the whole broker?

Thanks for your help.

Best,
Claudia

-----Ursprüngliche Nachricht-----
Von: Liam Clarke <li...@adscale.co.nz> 
Gesendet: Montag, 17. Dezember 2018 23:06
An: users@kafka.apache.org
Betreff: Re: Configuration of log compaction

Hi Claudia,

Anything useful in the log cleaner log files?

Cheers,

Liam Clarke

On Tue, 18 Dec. 2018, 3:18 am Claudia Wegmann <c.wegmann@kasasi.de wrote:

> Hi,
>
> thanks for the quick response.
>
> My problem is not, that no new segments are created, but that segments 
> with old data do not get compacted.
> I had to restart one broker because there was no diskspace left. After 
> recreating all indexes etc. the broker recognized the old data and 
> compacted it correctly. I had to restart all other brokers of the 
> cluster, too, for them to also recognize the old data and start compacting.
>
> So I guess, before restarting the brokers where to busy to 
> compact/delete old data? Is there a configuration to ensure compaction 
> after a certain amount of time or something?
>
> Best,
> Claudia
>
> -----Ursprüngliche Nachricht-----
> Von: Spico Florin <sp...@gmail.com>
> Gesendet: Montag, 17. Dezember 2018 14:28
> An: users@kafka.apache.org
> Betreff: Re: Configuration of log compaction
>
> Hello!
>   Please check whether the segment.ms configuration on topic will help 
> you to solve your problem.
>
> https://kafka.apache.org/documentation/
>
>
> https://stackoverflow.com/questions/41048041/kafka-deletes-segments-ev
> en-before-segment-size-is-reached
>
> Regards,
>  Florin
>
> segment.ms This configuration controls the period of time after which 
> Kafka will force the log to roll even if the segment file isn't full 
> to ensure that retention can delete or compact old data. long 
> 604800000 [1,...] log.roll.ms medium
>
> On Mon, Dec 17, 2018 at 12:28 PM Claudia Wegmann <c....@kasasi.de>
> wrote:
>
> > Dear kafka users,
> >
> > I've got a problem on one of my kafka clusters. I use this cluster 
> > with kafka streams applications. Some of this stream apps use a 
> > kafka state store. Therefore a changelog topic is created for those 
> > stores with cleanup policy "compact". One of these topics is running 
> > wild for some time now and seems to grow indefinitely. When I check 
> > the  log file of the first segment, there is a lot of data in it, 
> > that should have been compacted already.
> >
> > So I guess I did not configure everything correctly for log 
> > compaction to work as expected. What config parameters do have 
> > influence on log compaction? And how to set them, when I want data 
> > older than 4 hours to be compacted?
> >
> > Thanks in advance.
> >
> > Best,
> > Claudia
> >
>

Re: maven dependency problems

Posted by big data <bi...@outlook.com>.
Solved this problem! Because of SpringBoot dependency in parent POM, it  
depends lots of components...


在 2018/12/18 上午10:54, big data 写道:
> When I remove/comment parent dependency like <!--<parent>xxxx</parent> -->   in Module A pom.xml, it seems ok, and streaming kafka's depends on kafka_2.11:0.8.2.1.
>
> The Modula A's tree:
>
> [INFO] +- org.apache.spark:spark-core_2.11:jar:2.1.0:provided
> [INFO] |  +- org.apache.avro:avro-mapred:jar:hadoop2:1.7.7:provided
> [INFO] |  |  +- org.apache.avro:avro-ipc:jar:1.7.7:provided
> [INFO] |  |  +- org.apache.avro:avro-ipc:jar:tests:1.7.7:provided
> [INFO] |  |  \- org.codehaus.jackson:jackson-core-asl:jar:1.9.13:compile
> [INFO] |  +- com.twitter:chill_2.11:jar:0.8.0:provided
> [INFO] |  |  \- com.esotericsoftware:kryo-shaded:jar:3.0.3:provided
> [INFO] |  |     +- com.esotericsoftware:minlog:jar:1.3.0:provided
> [INFO] |  |     \- org.objenesis:objenesis:jar:2.1:provided
> [INFO] |  +- com.twitter:chill-java:jar:0.8.0:provided
> [INFO] |  +- org.apache.xbean:xbean-asm5-shaded:jar:4.4:provided
> [INFO] |  +- org.apache.hadoop:hadoop-client:jar:2.2.0:provided
> [INFO] |  |  +- org.apache.hadoop:hadoop-hdfs:jar:2.2.0:provided
> [INFO] |  |  +- org.apache.hadoop:hadoop-mapreduce-client-app:jar:2.2.0:provided
> [INFO] |  |  |  +- org.apache.hadoop:hadoop-mapreduce-client-common:jar:2.2.0:provided
> [INFO] |  |  |  |  +- org.apache.hadoop:hadoop-yarn-client:jar:2.2.0:provided
> [INFO] |  |  |  |  |  \- com.google.inject:guice:jar:3.0:provided
> [INFO] |  |  |  |  |     +- javax.inject:javax.inject:jar:1:provided
> [INFO] |  |  |  |  |     \- aopalliance:aopalliance:jar:1.0:provided
> [INFO] |  |  |  |  \- org.apache.hadoop:hadoop-yarn-server-common:jar:2.2.0:provided
> [INFO] |  |  |  \- org.apache.hadoop:hadoop-mapreduce-client-shuffle:jar:2.2.0:provided
> [INFO] |  |  +- org.apache.hadoop:hadoop-yarn-api:jar:2.2.0:provided
> [INFO] |  |  +- org.apache.hadoop:hadoop-mapreduce-client-core:jar:2.2.0:provided
> [INFO] |  |  |  \- org.apache.hadoop:hadoop-yarn-common:jar:2.2.0:provided
> [INFO] |  |  \- org.apache.hadoop:hadoop-mapreduce-client-jobclient:jar:2.2.0:provided
> [INFO] |  +- org.apache.spark:spark-launcher_2.11:jar:2.1.0:provided
> [INFO] |  +- org.apache.spark:spark-network-common_2.11:jar:2.1.0:provided
> [INFO] |  |  +- org.fusesource.leveldbjni:leveldbjni-all:jar:1.8:provided
> [INFO] |  |  \- com.fasterxml.jackson.core:jackson-annotations:jar:2.6.5:provided
> [INFO] |  +- org.apache.spark:spark-network-shuffle_2.11:jar:2.1.0:provided
> [INFO] |  +- org.apache.spark:spark-unsafe_2.11:jar:2.1.0:provided
> [INFO] |  +- net.java.dev.jets3t:jets3t:jar:0.7.1:compile
> [INFO] |  +- org.apache.curator:curator-recipes:jar:2.4.0:compile
> [INFO] |  |  \- org.apache.curator:curator-framework:jar:2.4.0:compile
> [INFO] |  +- javax.servlet:javax.servlet-api:jar:3.1.0:provided
> [INFO] |  +- org.apache.commons:commons-lang3:jar:3.5:provided
> [INFO] |  +- org.apache.commons:commons-math3:jar:3.4.1:compile
> [INFO] |  +- com.google.code.findbugs:jsr305:jar:1.3.9:compile
> [INFO] |  +- org.slf4j:slf4j-api:jar:1.7.16:compile
> [INFO] |  +- org.slf4j:jul-to-slf4j:jar:1.7.16:provided
> [INFO] |  +- org.slf4j:jcl-over-slf4j:jar:1.7.16:provided
> [INFO] |  +- log4j:log4j:jar:1.2.17:compile
> [INFO] |  +- org.slf4j:slf4j-log4j12:jar:1.7.16:compile
> [INFO] |  +- com.ning:compress-lzf:jar:1.0.3:provided
> [INFO] |  +- org.xerial.snappy:snappy-java:jar:1.1.2.6:compile
> [INFO] |  +- net.jpountz.lz4:lz4:jar:1.3.0:compile
> [INFO] |  +- org.roaringbitmap:RoaringBitmap:jar:0.5.11:provided
> [INFO] |  +- commons-net:commons-net:jar:2.2:compile
> [INFO] |  +- org.json4s:json4s-jackson_2.11:jar:3.2.11:provided
> [INFO] |  |  \- org.json4s:json4s-core_2.11:jar:3.2.11:provided
> [INFO] |  |     +- org.json4s:json4s-ast_2.11:jar:3.2.11:provided
> [INFO] |  |     \- org.scala-lang:scalap:jar:2.11.0:provided
> [INFO] |  |        \- org.scala-lang:scala-compiler:jar:2.11.0:provided
> [INFO] |  +- org.glassfish.jersey.core:jersey-client:jar:2.22.2:provided
> [INFO] |  |  +- javax.ws.rs:javax.ws.rs-api:jar:2.0.1:provided
> [INFO] |  |  +- org.glassfish.hk2:hk2-api:jar:2.4.0-b34:provided
> [INFO] |  |  |  +- org.glassfish.hk2:hk2-utils:jar:2.4.0-b34:provided
> [INFO] |  |  |  \- org.glassfish.hk2.external:aopalliance-repackaged:jar:2.4.0-b34:provided
> [INFO] |  |  +- org.glassfish.hk2.external:javax.inject:jar:2.4.0-b34:provided
> [INFO] |  |  \- org.glassfish.hk2:hk2-locator:jar:2.4.0-b34:provided
> [INFO] |  |     \- org.javassist:javassist:jar:3.18.1-GA:provided
> [INFO] |  +- org.glassfish.jersey.core:jersey-common:jar:2.22.2:provided
> [INFO] |  |  +- javax.annotation:javax.annotation-api:jar:1.2:provided
> [INFO] |  |  +- org.glassfish.jersey.bundles.repackaged:jersey-guava:jar:2.22.2:provided
> [INFO] |  |  \- org.glassfish.hk2:osgi-resource-locator:jar:1.0.1:provided
> [INFO] |  +- org.glassfish.jersey.core:jersey-server:jar:2.22.2:provided
> [INFO] |  |  +- org.glassfish.jersey.media:jersey-media-jaxb:jar:2.22.2:provided
> [INFO] |  |  \- javax.validation:validation-api:jar:1.1.0.Final:provided
> [INFO] |  +- org.glassfish.jersey.containers:jersey-container-servlet:jar:2.22.2:provided
> [INFO] |  +- org.glassfish.jersey.containers:jersey-container-servlet-core:jar:2.22.2:provided
> [INFO] |  +- io.netty:netty-all:jar:4.0.42.Final:compile
> [INFO] |  +- io.netty:netty:jar:3.8.0.Final:provided
> [INFO] |  +- com.clearspring.analytics:stream:jar:2.7.0:provided
> [INFO] |  +- io.dropwizard.metrics:metrics-core:jar:3.1.2:provided
> [INFO] |  +- io.dropwizard.metrics:metrics-jvm:jar:3.1.2:provided
> [INFO] |  +- io.dropwizard.metrics:metrics-json:jar:3.1.2:provided
> [INFO] |  +- io.dropwizard.metrics:metrics-graphite:jar:3.1.2:provided
> [INFO] |  +- com.fasterxml.jackson.core:jackson-databind:jar:2.6.5:provided
> [INFO] |  |  \- com.fasterxml.jackson.core:jackson-core:jar:2.6.5:provided
> [INFO] |  +- com.fasterxml.jackson.module:jackson-module-scala_2.11:jar:2.6.5:provided
> [INFO] |  |  \- com.fasterxml.jackson.module:jackson-module-paranamer:jar:2.6.5:provided
> [INFO] |  +- org.apache.ivy:ivy:jar:2.4.0:provided
> [INFO] |  +- oro:oro:jar:2.0.8:provided
> [INFO] |  +- net.razorvine:pyrolite:jar:4.13:provided
> [INFO] |  +- net.sf.py4j:py4j:jar:0.10.4:provided
> [INFO] |  +- org.apache.spark:spark-tags_2.11:jar:2.1.0:provided
> [INFO] |  +- org.apache.commons:commons-crypto:jar:1.0.0:provided
> [INFO] |  \- org.spark-project.spark:unused:jar:1.0.0:compile
> [INFO] +- org.apache.spark:spark-streaming_2.11:jar:2.1.0:provided
> [INFO] +- org.apache.spark:spark-sql_2.11:jar:2.1.0:provided
> [INFO] |  +- com.univocity:univocity-parsers:jar:2.2.1:provided
> [INFO] |  +- org.apache.spark:spark-sketch_2.11:jar:2.1.0:provided
> [INFO] |  +- org.apache.spark:spark-catalyst_2.11:jar:2.1.0:provided
> [INFO] |  |  +- org.codehaus.janino:janino:jar:3.0.0:provided
> [INFO] |  |  +- org.codehaus.janino:commons-compiler:jar:3.0.0:provided
> [INFO] |  |  \- org.antlr:antlr4-runtime:jar:4.5.3:provided
> [INFO] |  +- org.apache.parquet:parquet-column:jar:1.8.1:provided
> [INFO] |  |  +- org.apache.parquet:parquet-common:jar:1.8.1:provided
> [INFO] |  |  \- org.apache.parquet:parquet-encoding:jar:1.8.1:provided
> [INFO] |  \- org.apache.parquet:parquet-hadoop:jar:1.8.1:provided
> [INFO] |     +- org.apache.parquet:parquet-format:jar:2.3.0-incubating:provided
> [INFO] |     \- org.apache.parquet:parquet-jackson:jar:1.8.1:provided
> [INFO] +- org.apache.spark:spark-hive_2.11:jar:2.1.0:provided
> [INFO] |  +- com.twitter:parquet-hadoop-bundle:jar:1.6.0:provided
> [INFO] |  +- org.spark-project.hive:hive-exec:jar:1.2.1.spark2:provided
> [INFO] |  |  +- javolution:javolution:jar:5.5.1:provided
> [INFO] |  |  +- log4j:apache-log4j-extras:jar:1.2.17:provided
> [INFO] |  |  +- org.antlr:antlr-runtime:jar:3.4:provided
> [INFO] |  |  |  +- org.antlr:stringtemplate:jar:3.2.1:provided
> [INFO] |  |  |  \- antlr:antlr:jar:2.7.7:provided
> [INFO] |  |  +- org.antlr:ST4:jar:4.0.4:provided
> [INFO] |  |  +- org.apache.commons:commons-compress:jar:1.4.1:compile
> [INFO] |  |  |  \- org.tukaani:xz:jar:1.0:compile
> [INFO] |  |  +- com.googlecode.javaewah:JavaEWAH:jar:0.3.2:provided
> [INFO] |  |  +- org.iq80.snappy:snappy:jar:0.2:provided
> [INFO] |  |  +- stax:stax-api:jar:1.0.1:provided
> [INFO] |  |  \- net.sf.opencsv:opencsv:jar:2.3:provided
> [INFO] |  +- org.spark-project.hive:hive-metastore:jar:1.2.1.spark2:provided
> [INFO] |  |  +- com.jolbox:bonecp:jar:0.8.0.RELEASE:provided
> [INFO] |  |  +- commons-cli:commons-cli:jar:1.2:compile
> [INFO] |  |  +- org.apache.derby:derby:jar:10.10.2.0:provided
> [INFO] |  |  +- org.datanucleus:datanucleus-api-jdo:jar:3.2.6:provided
> [INFO] |  |  +- org.datanucleus:datanucleus-rdbms:jar:3.2.9:provided
> [INFO] |  |  +- commons-pool:commons-pool:jar:1.5.4:provided
> [INFO] |  |  +- commons-dbcp:commons-dbcp:jar:1.4:provided
> [INFO] |  |  \- javax.jdo:jdo-api:jar:3.0.1:provided
> [INFO] |  |     \- javax.transaction:jta:jar:1.1:provided
> [INFO] |  +- org.apache.avro:avro:jar:1.7.7:compile
> [INFO] |  |  \- com.thoughtworks.paranamer:paranamer:jar:2.3:compile
> [INFO] |  +- commons-httpclient:commons-httpclient:jar:3.1:compile
> [INFO] |  +- org.apache.calcite:calcite-avatica:jar:1.2.0-incubating:provided
> [INFO] |  +- org.apache.calcite:calcite-core:jar:1.2.0-incubating:provided
> [INFO] |  |  +- org.apache.calcite:calcite-linq4j:jar:1.2.0-incubating:provided
> [INFO] |  |  \- net.hydromatic:eigenbase-properties:jar:1.1.5:provided
> [INFO] |  +- org.apache.httpcomponents:httpclient:jar:4.5.2:compile
> [INFO] |  |  \- org.apache.httpcomponents:httpcore:jar:4.4.4:compile
> [INFO] |  +- org.codehaus.jackson:jackson-mapper-asl:jar:1.9.13:compile
> [INFO] |  +- commons-codec:commons-codec:jar:1.10:compile
> [INFO] |  +- joda-time:joda-time:jar:2.9.3:provided
> [INFO] |  +- org.jodd:jodd-core:jar:3.5.2:provided
> [INFO] |  +- org.datanucleus:datanucleus-core:jar:3.2.10:provided
> [INFO] |  +- org.apache.thrift:libthrift:jar:0.9.2:provided
> [INFO] |  \- org.apache.thrift:libfb303:jar:0.9.2:provided
> [INFO] +- org.apache.spark:spark-streaming-kafka_2.11:jar:1.6.3:compile
> [INFO] |  \- org.apache.kafka:kafka_2.11:jar:0.8.2.1:compile
> [INFO] |     +- org.scala-lang.modules:scala-parser-combinators_2.11:jar:1.0.2:compile
> [INFO] |     +- com.101tec:zkclient:jar:0.3:compile
> [INFO] |     \- org.apache.kafka:kafka-clients:jar:0.8.2.1:compile
> [INFO] +- org.scala-lang:scala-library:jar:2.11.8:provided
> [INFO] +- org.scala-lang:scala-actors:jar:2.11.8:provided
> [INFO] +- org.scalatest:scalatest_2.11:jar:2.2.1:test
> [INFO] |  +- org.scala-lang:scala-reflect:jar:2.11.2:provided
> [INFO] |  \- org.scala-lang.modules:scala-xml_2.11:jar:1.0.2:compile
> [INFO] +- org.apache.hbase:hbase-client:jar:1.2.0-cdh5.15.0:compile
> [INFO] |  +- org.apache.hbase:hbase-annotations:jar:1.2.0-cdh5.15.0:compile
> [INFO] |  +- org.apache.hbase:hbase-common:jar:1.2.0-cdh5.15.0:compile
> [INFO] |  |  +- commons-collections:commons-collections:jar:3.2.2:compile
> [INFO] |  |  +- org.mortbay.jetty:jetty-util:jar:6.1.26.cloudera.4:compile
> [INFO] |  |  \- org.apache.hadoop:hadoop-core:jar:2.6.0-mr1-cdh5.15.0:compile
> [INFO] |  |     +- tomcat:jasper-runtime:jar:5.5.23:compile
> [INFO] |  |     +- tomcat:jasper-compiler:jar:5.5.23:compile
> [INFO] |  |     +- javax.servlet.jsp:jsp-api:jar:2.1:compile
> [INFO] |  |     +- javax.servlet:servlet-api:jar:2.5:compile
> [INFO] |  |     +- hsqldb:hsqldb:jar:1.8.0.10:compile
> [INFO] |  |     \- org.eclipse.jdt:core:jar:3.1.1:compile
> [INFO] |  +- org.apache.hbase:hbase-protocol:jar:1.2.0-cdh5.15.0:compile
> [INFO] |  +- commons-io:commons-io:jar:2.4:compile
> [INFO] |  +- commons-lang:commons-lang:jar:2.6:compile
> [INFO] |  +- commons-logging:commons-logging:jar:1.2:compile
> [INFO] |  +- com.google.guava:guava:jar:12.0.1:compile
> [INFO] |  +- com.google.protobuf:protobuf-java:jar:2.5.0:compile
> [INFO] |  +- org.apache.zookeeper:zookeeper:jar:3.4.5-cdh5.15.0:compile
> [INFO] |  +- org.apache.htrace:htrace-core:jar:3.2.0-incubating:compile
> [INFO] |  +- org.jruby.jcodings:jcodings:jar:1.0.8:compile
> [INFO] |  +- org.jruby.joni:joni:jar:2.1.2:compile
> [INFO] |  +- com.yammer.metrics:metrics-core:jar:2.2.0:compile
> [INFO] |  +- org.apache.hadoop:hadoop-common:jar:2.6.0-cdh5.15.0:compile
> [INFO] |  |  +- xmlenc:xmlenc:jar:0.52:compile
> [INFO] |  |  +- org.mortbay.jetty:jetty:jar:6.1.26.cloudera.4:compile
> [INFO] |  |  +- com.sun.jersey:jersey-core:jar:1.9:compile
> [INFO] |  |  +- com.sun.jersey:jersey-json:jar:1.9:compile
> [INFO] |  |  |  +- org.codehaus.jettison:jettison:jar:1.1:compile
> [INFO] |  |  |  +- com.sun.xml.bind:jaxb-impl:jar:2.2.3-1:compile
> [INFO] |  |  |  |  \- javax.xml.bind:jaxb-api:jar:2.2.2:compile
> [INFO] |  |  |  |     +- javax.xml.stream:stax-api:jar:1.0-2:compile
> [INFO] |  |  |  |     \- javax.activation:activation:jar:1.1:compile
> [INFO] |  |  |  +- org.codehaus.jackson:jackson-jaxrs:jar:1.8.3:compile
> [INFO] |  |  |  \- org.codehaus.jackson:jackson-xc:jar:1.8.3:compile
> [INFO] |  |  +- commons-el:commons-el:jar:1.0:compile
> [INFO] |  |  +- commons-configuration:commons-configuration:jar:1.6:compile
> [INFO] |  |  |  +- commons-digester:commons-digester:jar:1.8:compile
> [INFO] |  |  |  |  \- commons-beanutils:commons-beanutils:jar:1.7.0:compile
> [INFO] |  |  |  \- commons-beanutils:commons-beanutils-core:jar:1.8.0:compile
> [INFO] |  |  +- com.google.code.gson:gson:jar:2.2.4:compile
> [INFO] |  |  +- com.jcraft:jsch:jar:0.1.42:compile
> [INFO] |  |  +- org.apache.curator:curator-client:jar:2.7.1:compile
> [INFO] |  |  \- org.apache.htrace:htrace-core4:jar:4.0.1-incubating:compile
> [INFO] |  +- org.apache.hadoop:hadoop-annotations:jar:2.6.0-cdh5.15.0:compile
> [INFO] |  +- org.apache.hadoop:hadoop-auth:jar:2.6.0-cdh5.15.0:compile
> [INFO] |  |  \- org.apache.directory.server:apacheds-kerberos-codec:jar:2.0.0-M15:compile
> [INFO] |  |     +- org.apache.directory.server:apacheds-i18n:jar:2.0.0-M15:compile
> [INFO] |  |     +- org.apache.directory.api:api-asn1-api:jar:1.0.0-M20:compile
> [INFO] |  |     \- org.apache.directory.api:api-util:jar:1.0.0-M20:compile
> [INFO] |  +- com.github.stephenc.findbugs:findbugs-annotations:jar:1.3.9-1:compile
> [INFO] |  \- junit:junit:jar:4.12:compile
> [INFO] |     \- org.hamcrest:hamcrest-core:jar:1.3:compile
> [INFO] \- com.oracle.jdbc:ojdbc6:jar:11.1.0.6.0:compile
>
>
> But If I open <parent>xxx</parent>, and Module A's tree
>
> [INFO] +- org.apache.spark:spark-core_2.11:jar:2.1.0:provided
> [INFO] |  +- org.apache.avro:avro-mapred:jar:hadoop2:1.7.7:provided
> [INFO] |  |  +- org.apache.avro:avro-ipc:jar:1.7.7:provided
> [INFO] |  |  +- org.apache.avro:avro-ipc:jar:tests:1.7.7:provided
> [INFO] |  |  \- org.codehaus.jackson:jackson-core-asl:jar:1.9.13:compile
> [INFO] |  +- com.twitter:chill_2.11:jar:0.8.0:provided
> [INFO] |  |  \- com.esotericsoftware:kryo-shaded:jar:3.0.3:provided
> [INFO] |  |     \- com.esotericsoftware:minlog:jar:1.3.0:provided
> [INFO] |  +- com.twitter:chill-java:jar:0.8.0:provided
> [INFO] |  +- org.apache.xbean:xbean-asm5-shaded:jar:4.4:provided
> [INFO] |  +- org.apache.hadoop:hadoop-client:jar:2.2.0:provided
> [INFO] |  |  +- org.apache.hadoop:hadoop-hdfs:jar:2.2.0:provided
> [INFO] |  |  +- org.apache.hadoop:hadoop-mapreduce-client-app:jar:2.2.0:provided
> [INFO] |  |  |  +- org.apache.hadoop:hadoop-mapreduce-client-common:jar:2.2.0:provided
> [INFO] |  |  |  |  +- org.apache.hadoop:hadoop-yarn-client:jar:2.2.0:provided
> [INFO] |  |  |  |  |  \- com.google.inject:guice:jar:3.0:provided
> [INFO] |  |  |  |  |     +- javax.inject:javax.inject:jar:1:provided
> [INFO] |  |  |  |  |     \- aopalliance:aopalliance:jar:1.0:provided
> [INFO] |  |  |  |  \- org.apache.hadoop:hadoop-yarn-server-common:jar:2.2.0:provided
> [INFO] |  |  |  \- org.apache.hadoop:hadoop-mapreduce-client-shuffle:jar:2.2.0:provided
> [INFO] |  |  +- org.apache.hadoop:hadoop-yarn-api:jar:2.2.0:provided
> [INFO] |  |  +- org.apache.hadoop:hadoop-mapreduce-client-core:jar:2.2.0:provided
> [INFO] |  |  |  \- org.apache.hadoop:hadoop-yarn-common:jar:2.2.0:provided
> [INFO] |  |  \- org.apache.hadoop:hadoop-mapreduce-client-jobclient:jar:2.2.0:provided
> [INFO] |  +- org.apache.spark:spark-launcher_2.11:jar:2.1.0:provided
> [INFO] |  +- org.apache.spark:spark-network-common_2.11:jar:2.1.0:provided
> [INFO] |  |  +- org.fusesource.leveldbjni:leveldbjni-all:jar:1.8:provided
> [INFO] |  |  \- com.fasterxml.jackson.core:jackson-annotations:jar:2.9.0:compile
> [INFO] |  +- org.apache.spark:spark-network-shuffle_2.11:jar:2.1.0:provided
> [INFO] |  +- org.apache.spark:spark-unsafe_2.11:jar:2.1.0:provided
> [INFO] |  +- net.java.dev.jets3t:jets3t:jar:0.7.1:compile
> [INFO] |  +- org.apache.curator:curator-recipes:jar:2.4.0:compile
> [INFO] |  |  \- org.apache.curator:curator-framework:jar:2.4.0:compile
> [INFO] |  +- javax.servlet:javax.servlet-api:jar:3.1.0:provided
> [INFO] |  +- org.apache.commons:commons-lang3:jar:3.7:provided
> [INFO] |  +- org.apache.commons:commons-math3:jar:3.4.1:compile
> [INFO] |  +- com.google.code.findbugs:jsr305:jar:1.3.9:compile
> [INFO] |  +- org.slf4j:slf4j-api:jar:1.7.25:compile
> [INFO] |  +- org.slf4j:jul-to-slf4j:jar:1.7.25:provided
> [INFO] |  +- org.slf4j:jcl-over-slf4j:jar:1.7.25:provided
> [INFO] |  +- log4j:log4j:jar:1.2.17:compile
> [INFO] |  +- org.slf4j:slf4j-log4j12:jar:1.7.25:compile
> [INFO] |  +- com.ning:compress-lzf:jar:1.0.3:provided
> [INFO] |  +- org.xerial.snappy:snappy-java:jar:1.1.2.6:compile
> [INFO] |  +- net.jpountz.lz4:lz4:jar:1.3.0:provided
> [INFO] |  +- org.roaringbitmap:RoaringBitmap:jar:0.5.11:provided
> [INFO] |  +- commons-net:commons-net:jar:2.2:compile
> [INFO] |  +- org.json4s:json4s-jackson_2.11:jar:3.2.11:provided
> [INFO] |  |  \- org.json4s:json4s-core_2.11:jar:3.2.11:provided
> [INFO] |  |     +- org.json4s:json4s-ast_2.11:jar:3.2.11:provided
> [INFO] |  |     \- org.scala-lang:scalap:jar:2.11.0:provided
> [INFO] |  |        \- org.scala-lang:scala-compiler:jar:2.11.0:provided
> [INFO] |  |           \- org.scala-lang.modules:scala-parser-combinators_2.11:jar:1.0.1:provided
> [INFO] |  +- org.glassfish.jersey.core:jersey-client:jar:2.26:provided
> [INFO] |  |  +- javax.ws.rs:javax.ws.rs-api:jar:2.1:provided
> [INFO] |  |  \- org.glassfish.hk2.external:javax.inject:jar:2.5.0-b42:provided
> [INFO] |  +- org.glassfish.jersey.core:jersey-common:jar:2.26:provided
> [INFO] |  |  +- javax.annotation:javax.annotation-api:jar:1.3.2:provided
> [INFO] |  |  \- org.glassfish.hk2:osgi-resource-locator:jar:1.0.1:provided
> [INFO] |  +- org.glassfish.jersey.core:jersey-server:jar:2.26:provided
> [INFO] |  |  +- org.glassfish.jersey.media:jersey-media-jaxb:jar:2.26:provided
> [INFO] |  |  \- javax.validation:validation-api:jar:2.0.1.Final:provided
> [INFO] |  +- org.glassfish.jersey.containers:jersey-container-servlet:jar:2.26:provided
> [INFO] |  +- org.glassfish.jersey.containers:jersey-container-servlet-core:jar:2.26:provided
> [INFO] |  +- io.netty:netty-all:jar:4.1.27.Final:compile
> [INFO] |  +- io.netty:netty:jar:3.8.0.Final:provided
> [INFO] |  +- com.clearspring.analytics:stream:jar:2.7.0:provided
> [INFO] |  +- io.dropwizard.metrics:metrics-core:jar:3.2.6:provided
> [INFO] |  +- io.dropwizard.metrics:metrics-jvm:jar:3.2.6:provided
> [INFO] |  +- io.dropwizard.metrics:metrics-json:jar:3.2.6:provided
> [INFO] |  +- io.dropwizard.metrics:metrics-graphite:jar:3.2.6:provided
> [INFO] |  +- com.fasterxml.jackson.core:jackson-databind:jar:2.9.6:compile
> [INFO] |  |  \- com.fasterxml.jackson.core:jackson-core:jar:2.9.6:compile
> [INFO] |  +- com.fasterxml.jackson.module:jackson-module-scala_2.11:jar:2.9.6:provided
> [INFO] |  |  \- com.fasterxml.jackson.module:jackson-module-paranamer:jar:2.9.6:provided
> [INFO] |  +- org.apache.ivy:ivy:jar:2.4.0:provided
> [INFO] |  +- oro:oro:jar:2.0.8:provided
> [INFO] |  +- net.razorvine:pyrolite:jar:4.13:provided
> [INFO] |  +- net.sf.py4j:py4j:jar:0.10.4:provided
> [INFO] |  +- org.apache.spark:spark-tags_2.11:jar:2.1.0:provided
> [INFO] |  +- org.apache.commons:commons-crypto:jar:1.0.0:provided
> [INFO] |  \- org.spark-project.spark:unused:jar:1.0.0:compile
> [INFO] +- org.apache.spark:spark-streaming_2.11:jar:2.1.0:provided
> [INFO] +- org.apache.spark:spark-sql_2.11:jar:2.1.0:provided
> [INFO] |  +- com.univocity:univocity-parsers:jar:2.2.1:provided
> [INFO] |  +- org.apache.spark:spark-sketch_2.11:jar:2.1.0:provided
> [INFO] |  +- org.apache.spark:spark-catalyst_2.11:jar:2.1.0:provided
> [INFO] |  |  +- org.codehaus.janino:janino:jar:3.0.8:provided
> [INFO] |  |  +- org.codehaus.janino:commons-compiler:jar:3.0.0:provided
> [INFO] |  |  \- org.antlr:antlr4-runtime:jar:4.5.3:provided
> [INFO] |  +- org.apache.parquet:parquet-column:jar:1.8.1:provided
> [INFO] |  |  +- org.apache.parquet:parquet-common:jar:1.8.1:provided
> [INFO] |  |  \- org.apache.parquet:parquet-encoding:jar:1.8.1:provided
> [INFO] |  \- org.apache.parquet:parquet-hadoop:jar:1.8.1:provided
> [INFO] |     +- org.apache.parquet:parquet-format:jar:2.3.0-incubating:provided
> [INFO] |     \- org.apache.parquet:parquet-jackson:jar:1.8.1:provided
> [INFO] +- org.apache.spark:spark-hive_2.11:jar:2.1.0:provided
> [INFO] |  +- com.twitter:parquet-hadoop-bundle:jar:1.6.0:provided
> [INFO] |  +- org.spark-project.hive:hive-exec:jar:1.2.1.spark2:provided
> [INFO] |  |  +- javolution:javolution:jar:5.5.1:provided
> [INFO] |  |  +- log4j:apache-log4j-extras:jar:1.2.17:provided
> [INFO] |  |  +- org.antlr:antlr-runtime:jar:3.4:provided
> [INFO] |  |  |  +- org.antlr:stringtemplate:jar:3.2.1:provided
> [INFO] |  |  |  \- antlr:antlr:jar:2.7.7:provided
> [INFO] |  |  +- org.antlr:ST4:jar:4.0.4:provided
> [INFO] |  |  +- org.apache.commons:commons-compress:jar:1.4.1:compile
> [INFO] |  |  |  \- org.tukaani:xz:jar:1.0:compile
> [INFO] |  |  +- com.googlecode.javaewah:JavaEWAH:jar:0.3.2:provided
> [INFO] |  |  +- org.iq80.snappy:snappy:jar:0.2:provided
> [INFO] |  |  +- stax:stax-api:jar:1.0.1:provided
> [INFO] |  |  \- net.sf.opencsv:opencsv:jar:2.3:provided
> [INFO] |  +- org.spark-project.hive:hive-metastore:jar:1.2.1.spark2:provided
> [INFO] |  |  +- com.jolbox:bonecp:jar:0.8.0.RELEASE:provided
> [INFO] |  |  +- commons-cli:commons-cli:jar:1.2:compile
> [INFO] |  |  +- org.apache.derby:derby:jar:10.14.1.0:provided
> [INFO] |  |  +- org.datanucleus:datanucleus-api-jdo:jar:3.2.6:provided
> [INFO] |  |  +- org.datanucleus:datanucleus-rdbms:jar:3.2.9:provided
> [INFO] |  |  +- commons-pool:commons-pool:jar:1.6:provided
> [INFO] |  |  +- commons-dbcp:commons-dbcp:jar:1.4:provided
> [INFO] |  |  \- javax.jdo:jdo-api:jar:3.0.1:provided
> [INFO] |  |     \- javax.transaction:jta:jar:1.1:provided
> [INFO] |  +- org.apache.avro:avro:jar:1.7.7:compile
> [INFO] |  |  \- com.thoughtworks.paranamer:paranamer:jar:2.3:compile
> [INFO] |  +- commons-httpclient:commons-httpclient:jar:3.1:compile
> [INFO] |  +- org.apache.calcite:calcite-avatica:jar:1.2.0-incubating:provided
> [INFO] |  +- org.apache.calcite:calcite-core:jar:1.2.0-incubating:provided
> [INFO] |  |  +- org.apache.calcite:calcite-linq4j:jar:1.2.0-incubating:provided
> [INFO] |  |  \- net.hydromatic:eigenbase-properties:jar:1.1.5:provided
> [INFO] |  +- org.apache.httpcomponents:httpclient:jar:4.5.6:compile
> [INFO] |  |  \- org.apache.httpcomponents:httpcore:jar:4.4.10:compile
> [INFO] |  +- org.codehaus.jackson:jackson-mapper-asl:jar:1.9.13:compile
> [INFO] |  +- commons-codec:commons-codec:jar:1.11:compile
> [INFO] |  +- joda-time:joda-time:jar:2.9.9:provided
> [INFO] |  +- org.jodd:jodd-core:jar:3.5.2:provided
> [INFO] |  +- org.datanucleus:datanucleus-core:jar:3.2.10:provided
> [INFO] |  +- org.apache.thrift:libthrift:jar:0.9.2:provided
> [INFO] |  \- org.apache.thrift:libfb303:jar:0.9.2:provided
> [INFO] +- org.apache.spark:spark-streaming-kafka_2.11:jar:1.6.3:compile
> [INFO] |  \- org.apache.kafka:kafka_2.11:jar:1.0.2:compile
> [INFO] |     +- org.apache.kafka:kafka-clients:jar:1.0.2:compile
> [INFO] |     |  \- org.lz4:lz4-java:jar:1.4:compile
> [INFO] |     \- com.101tec:zkclient:jar:0.10:compile
> [INFO] +- org.scala-lang:scala-library:jar:2.11.8:provided
> [INFO] +- org.scala-lang:scala-actors:jar:2.11.8:provided
> [INFO] +- org.scalatest:scalatest_2.11:jar:2.2.1:test
> [INFO] |  +- org.scala-lang:scala-reflect:jar:2.11.2:provided
> [INFO] |  \- org.scala-lang.modules:scala-xml_2.11:jar:1.0.2:provided
> [INFO] +- org.apache.hbase:hbase-client:jar:1.2.0-cdh5.15.0:compile
> [INFO] |  +- org.apache.hbase:hbase-annotations:jar:1.2.0-cdh5.15.0:compile
> [INFO] |  +- org.apache.hbase:hbase-common:jar:1.2.0-cdh5.15.0:compile
> [INFO] |  |  +- commons-collections:commons-collections:jar:3.2.2:compile
> [INFO] |  |  +- org.mortbay.jetty:jetty-util:jar:6.1.26.cloudera.4:compile
> [INFO] |  |  \- org.apache.hadoop:hadoop-core:jar:2.6.0-mr1-cdh5.15.0:compile
> [INFO] |  |     +- tomcat:jasper-runtime:jar:5.5.23:compile
> [INFO] |  |     +- tomcat:jasper-compiler:jar:5.5.23:compile
> [INFO] |  |     +- javax.servlet.jsp:jsp-api:jar:2.1:compile
> [INFO] |  |     +- javax.servlet:servlet-api:jar:2.5:compile
> [INFO] |  |     +- hsqldb:hsqldb:jar:1.8.0.10:compile
> [INFO] |  |     \- org.eclipse.jdt:core:jar:3.1.1:compile
> [INFO] |  +- org.apache.hbase:hbase-protocol:jar:1.2.0-cdh5.15.0:compile
> [INFO] |  +- commons-io:commons-io:jar:2.4:compile
> [INFO] |  +- commons-lang:commons-lang:jar:2.6:compile
> [INFO] |  +- commons-logging:commons-logging:jar:1.2:compile
> [INFO] |  +- com.google.guava:guava:jar:12.0.1:compile
> [INFO] |  +- com.google.protobuf:protobuf-java:jar:2.5.0:compile
> [INFO] |  +- org.apache.zookeeper:zookeeper:jar:3.4.5-cdh5.15.0:compile
> [INFO] |  +- org.apache.htrace:htrace-core:jar:3.2.0-incubating:compile
> [INFO] |  +- org.jruby.jcodings:jcodings:jar:1.0.8:compile
> [INFO] |  +- org.jruby.joni:joni:jar:2.1.2:compile
> [INFO] |  +- com.yammer.metrics:metrics-core:jar:2.2.0:compile
> [INFO] |  +- org.apache.hadoop:hadoop-common:jar:2.6.0-cdh5.15.0:compile
> [INFO] |  |  +- xmlenc:xmlenc:jar:0.52:compile
> [INFO] |  |  +- org.mortbay.jetty:jetty:jar:6.1.26.cloudera.4:compile
> [INFO] |  |  +- com.sun.jersey:jersey-core:jar:1.9:compile
> [INFO] |  |  +- com.sun.jersey:jersey-json:jar:1.9:compile
> [INFO] |  |  |  +- org.codehaus.jettison:jettison:jar:1.1:compile
> [INFO] |  |  |  +- com.sun.xml.bind:jaxb-impl:jar:2.2.3-1:compile
> [INFO] |  |  |  |  \- javax.xml.bind:jaxb-api:jar:2.3.0:compile
> [INFO] |  |  |  +- org.codehaus.jackson:jackson-jaxrs:jar:1.8.3:compile
> [INFO] |  |  |  \- org.codehaus.jackson:jackson-xc:jar:1.8.3:compile
> [INFO] |  |  +- commons-el:commons-el:jar:1.0:compile
> [INFO] |  |  +- commons-configuration:commons-configuration:jar:1.6:compile
> [INFO] |  |  |  +- commons-digester:commons-digester:jar:1.8:compile
> [INFO] |  |  |  |  \- commons-beanutils:commons-beanutils:jar:1.7.0:compile
> [INFO] |  |  |  \- commons-beanutils:commons-beanutils-core:jar:1.8.0:compile
> [INFO] |  |  +- com.google.code.gson:gson:jar:2.8.5:compile
> [INFO] |  |  +- com.jcraft:jsch:jar:0.1.42:compile
> [INFO] |  |  +- org.apache.curator:curator-client:jar:2.7.1:compile
> [INFO] |  |  \- org.apache.htrace:htrace-core4:jar:4.0.1-incubating:compile
> [INFO] |  +- org.apache.hadoop:hadoop-annotations:jar:2.6.0-cdh5.15.0:compile
> [INFO] |  +- org.apache.hadoop:hadoop-auth:jar:2.6.0-cdh5.15.0:compile
> [INFO] |  |  \- org.apache.directory.server:apacheds-kerberos-codec:jar:2.0.0-M15:compile
> [INFO] |  |     +- org.apache.directory.server:apacheds-i18n:jar:2.0.0-M15:compile
> [INFO] |  |     +- org.apache.directory.api:api-asn1-api:jar:1.0.0-M20:compile
> [INFO] |  |     \- org.apache.directory.api:api-util:jar:1.0.0-M20:compile
> [INFO] |  \- com.github.stephenc.findbugs:findbugs-annotations:jar:1.3.9-1:compile
> [INFO] +- com.oracle.jdbc:ojdbc6:jar:11.1.0.6.0:compile
> [INFO] +- ch.qos.logback:logback-classic:jar:1.2.3:compile
> [INFO] |  \- ch.qos.logback:logback-core:jar:1.2.3:compile
> [INFO] +- junit:junit:jar:4.12:test
> [INFO] |  \- org.hamcrest:hamcrest-core:jar:1.3:test
> [INFO] +- org.powermock:powermock-module-junit4:jar:1.7.1:test
> [INFO] |  \- org.powermock:powermock-module-junit4-common:jar:1.7.1:test
> [INFO] |     +- org.powermock:powermock-reflect:jar:1.7.1:test
> [INFO] |     \- org.powermock:powermock-core:jar:1.7.1:test
> [INFO] |        \- org.javassist:javassist:jar:3.21.0-GA:test
> [INFO] \- org.powermock:powermock-api-mockito2:jar:1.7.1:test
> [INFO]    +- org.powermock:powermock-api-mockito-common:jar:1.7.1:test
> [INFO]    |  \- org.powermock:powermock-api-support:jar:1.7.1:test
> [INFO]    \- org.mockito:mockito-core:jar:2.15.0:test
> [INFO]       +- net.bytebuddy:byte-buddy:jar:1.7.11:test
> [INFO]       +- net.bytebuddy:byte-buddy-agent:jar:1.7.11:test
> [INFO]       \- org.objenesis:objenesis:jar:2.6:provided
>
>
> The parent pom's dependency tree:(parent pom only has few dependency, same as before)
>
> [INFO] +- ch.qos.logback:logback-classic:jar:1.2.3:compile
> [INFO] |  +- ch.qos.logback:logback-core:jar:1.2.3:compile
> [INFO] |  \- org.slf4j:slf4j-api:jar:1.7.25:compile
> [INFO] +- junit:junit:jar:4.12:test
> [INFO] |  \- org.hamcrest:hamcrest-core:jar:1.3:test
> [INFO] +- org.powermock:powermock-module-junit4:jar:1.7.1:test
> [INFO] |  \- org.powermock:powermock-module-junit4-common:jar:1.7.1:test
> [INFO] |     +- org.powermock:powermock-reflect:jar:1.7.1:test
> [INFO] |     \- org.powermock:powermock-core:jar:1.7.1:test
> [INFO] |        \- org.javassist:javassist:jar:3.21.0-GA:test
> [INFO] \- org.powermock:powermock-api-mockito2:jar:1.7.1:test
> [INFO]    +- org.powermock:powermock-api-mockito-common:jar:1.7.1:test
> [INFO]    |  \- org.powermock:powermock-api-support:jar:1.7.1:test
> [INFO]    \- org.mockito:mockito-core:jar:2.15.0:test
> [INFO]       +- net.bytebuddy:byte-buddy:jar:1.7.11:test
> [INFO]       +- net.bytebuddy:byte-buddy-agent:jar:1.7.11:test
> [INFO]       \- org.objenesis:objenesis:jar:2.6:test
>
>
> Other module trees do not have kafka's dependency, so I don't post it.
>
> thanks.
>
> 在 2018/12/18 上午10:42, Liam Clarke 写道:
>
> Can you please post the full output of maven dependency:tree from the
> parent POM in both scenarios?
>
>
> Thanks,
>
> Liam Clarke
>
> On Tue, 18 Dec. 2018, 3:26 pm big data <bi...@outlook.com> wrote:
>
>
>
> Hi,
>
> No other dependency include kafka's jar.
>
> The project structure is:
>
> pom.xml
>
> |---Module A
>
>     |--pom.xml
>
> |---Module B
>
> |---Module C
>
> When spark-streaming-kafka_2.11:1.6.3 in in Module A's pom.xml, from
> dependency tree, we can see in depends on kafka_2.11:0.8.2.1
> When we move spark-streaming-kafka_2.11:1.6.3 dependency to the parent
> pom.xml, from tree, it depends on kafka_2.11:1.0.2
>
> In whoel project, only one dependency about streaming-kafka, and no
> other dependency about kafka_2.11.
>
>
> 在 2018/12/18 上午10:16, Liam Clarke 写道:
>
>
> Hi,
>
> Is it a transitive dependency of any of your other dependencies?
>
> Cheers,
>
> Liam Clarke
>
> On Tue, 18 Dec. 2018, 2:57 pm big data <bi...@outlook.com> wrote:
>
>
>
> Hi,
>     our project includes this dependency by:
>
> <dependency>    <groupId>org.apache.spark</groupId>
>
>
> <artifactId>spark-streaming-kafka_2.11</artifactId>
> <version>1.6.3</version></dependency>
>
>
>
>   From dependency tree, we can see it dependency kafka_2.11:0.8.2.1
>
>
> verson.
>
>
>
>
> But when we move this dependency to project's parent pom file, the
> dependency "kafka_2.11" version is changed automatically. Why?
>
>
> You can see Kafka_2.11 version is changed to 1.0.2?  It's so strange, in
> out project we DO NOT dependency this jar in other places.
>
>
>
>
>
>
>
>
>
>


Re: maven dependency problems

Posted by big data <bi...@outlook.com>.
When I remove/comment parent dependency like <!--<parent>xxxx</parent> -->   in Module A pom.xml, it seems ok, and streaming kafka's depends on kafka_2.11:0.8.2.1.

The Modula A's tree:

[INFO] +- org.apache.spark:spark-core_2.11:jar:2.1.0:provided
[INFO] |  +- org.apache.avro:avro-mapred:jar:hadoop2:1.7.7:provided
[INFO] |  |  +- org.apache.avro:avro-ipc:jar:1.7.7:provided
[INFO] |  |  +- org.apache.avro:avro-ipc:jar:tests:1.7.7:provided
[INFO] |  |  \- org.codehaus.jackson:jackson-core-asl:jar:1.9.13:compile
[INFO] |  +- com.twitter:chill_2.11:jar:0.8.0:provided
[INFO] |  |  \- com.esotericsoftware:kryo-shaded:jar:3.0.3:provided
[INFO] |  |     +- com.esotericsoftware:minlog:jar:1.3.0:provided
[INFO] |  |     \- org.objenesis:objenesis:jar:2.1:provided
[INFO] |  +- com.twitter:chill-java:jar:0.8.0:provided
[INFO] |  +- org.apache.xbean:xbean-asm5-shaded:jar:4.4:provided
[INFO] |  +- org.apache.hadoop:hadoop-client:jar:2.2.0:provided
[INFO] |  |  +- org.apache.hadoop:hadoop-hdfs:jar:2.2.0:provided
[INFO] |  |  +- org.apache.hadoop:hadoop-mapreduce-client-app:jar:2.2.0:provided
[INFO] |  |  |  +- org.apache.hadoop:hadoop-mapreduce-client-common:jar:2.2.0:provided
[INFO] |  |  |  |  +- org.apache.hadoop:hadoop-yarn-client:jar:2.2.0:provided
[INFO] |  |  |  |  |  \- com.google.inject:guice:jar:3.0:provided
[INFO] |  |  |  |  |     +- javax.inject:javax.inject:jar:1:provided
[INFO] |  |  |  |  |     \- aopalliance:aopalliance:jar:1.0:provided
[INFO] |  |  |  |  \- org.apache.hadoop:hadoop-yarn-server-common:jar:2.2.0:provided
[INFO] |  |  |  \- org.apache.hadoop:hadoop-mapreduce-client-shuffle:jar:2.2.0:provided
[INFO] |  |  +- org.apache.hadoop:hadoop-yarn-api:jar:2.2.0:provided
[INFO] |  |  +- org.apache.hadoop:hadoop-mapreduce-client-core:jar:2.2.0:provided
[INFO] |  |  |  \- org.apache.hadoop:hadoop-yarn-common:jar:2.2.0:provided
[INFO] |  |  \- org.apache.hadoop:hadoop-mapreduce-client-jobclient:jar:2.2.0:provided
[INFO] |  +- org.apache.spark:spark-launcher_2.11:jar:2.1.0:provided
[INFO] |  +- org.apache.spark:spark-network-common_2.11:jar:2.1.0:provided
[INFO] |  |  +- org.fusesource.leveldbjni:leveldbjni-all:jar:1.8:provided
[INFO] |  |  \- com.fasterxml.jackson.core:jackson-annotations:jar:2.6.5:provided
[INFO] |  +- org.apache.spark:spark-network-shuffle_2.11:jar:2.1.0:provided
[INFO] |  +- org.apache.spark:spark-unsafe_2.11:jar:2.1.0:provided
[INFO] |  +- net.java.dev.jets3t:jets3t:jar:0.7.1:compile
[INFO] |  +- org.apache.curator:curator-recipes:jar:2.4.0:compile
[INFO] |  |  \- org.apache.curator:curator-framework:jar:2.4.0:compile
[INFO] |  +- javax.servlet:javax.servlet-api:jar:3.1.0:provided
[INFO] |  +- org.apache.commons:commons-lang3:jar:3.5:provided
[INFO] |  +- org.apache.commons:commons-math3:jar:3.4.1:compile
[INFO] |  +- com.google.code.findbugs:jsr305:jar:1.3.9:compile
[INFO] |  +- org.slf4j:slf4j-api:jar:1.7.16:compile
[INFO] |  +- org.slf4j:jul-to-slf4j:jar:1.7.16:provided
[INFO] |  +- org.slf4j:jcl-over-slf4j:jar:1.7.16:provided
[INFO] |  +- log4j:log4j:jar:1.2.17:compile
[INFO] |  +- org.slf4j:slf4j-log4j12:jar:1.7.16:compile
[INFO] |  +- com.ning:compress-lzf:jar:1.0.3:provided
[INFO] |  +- org.xerial.snappy:snappy-java:jar:1.1.2.6:compile
[INFO] |  +- net.jpountz.lz4:lz4:jar:1.3.0:compile
[INFO] |  +- org.roaringbitmap:RoaringBitmap:jar:0.5.11:provided
[INFO] |  +- commons-net:commons-net:jar:2.2:compile
[INFO] |  +- org.json4s:json4s-jackson_2.11:jar:3.2.11:provided
[INFO] |  |  \- org.json4s:json4s-core_2.11:jar:3.2.11:provided
[INFO] |  |     +- org.json4s:json4s-ast_2.11:jar:3.2.11:provided
[INFO] |  |     \- org.scala-lang:scalap:jar:2.11.0:provided
[INFO] |  |        \- org.scala-lang:scala-compiler:jar:2.11.0:provided
[INFO] |  +- org.glassfish.jersey.core:jersey-client:jar:2.22.2:provided
[INFO] |  |  +- javax.ws.rs:javax.ws.rs-api:jar:2.0.1:provided
[INFO] |  |  +- org.glassfish.hk2:hk2-api:jar:2.4.0-b34:provided
[INFO] |  |  |  +- org.glassfish.hk2:hk2-utils:jar:2.4.0-b34:provided
[INFO] |  |  |  \- org.glassfish.hk2.external:aopalliance-repackaged:jar:2.4.0-b34:provided
[INFO] |  |  +- org.glassfish.hk2.external:javax.inject:jar:2.4.0-b34:provided
[INFO] |  |  \- org.glassfish.hk2:hk2-locator:jar:2.4.0-b34:provided
[INFO] |  |     \- org.javassist:javassist:jar:3.18.1-GA:provided
[INFO] |  +- org.glassfish.jersey.core:jersey-common:jar:2.22.2:provided
[INFO] |  |  +- javax.annotation:javax.annotation-api:jar:1.2:provided
[INFO] |  |  +- org.glassfish.jersey.bundles.repackaged:jersey-guava:jar:2.22.2:provided
[INFO] |  |  \- org.glassfish.hk2:osgi-resource-locator:jar:1.0.1:provided
[INFO] |  +- org.glassfish.jersey.core:jersey-server:jar:2.22.2:provided
[INFO] |  |  +- org.glassfish.jersey.media:jersey-media-jaxb:jar:2.22.2:provided
[INFO] |  |  \- javax.validation:validation-api:jar:1.1.0.Final:provided
[INFO] |  +- org.glassfish.jersey.containers:jersey-container-servlet:jar:2.22.2:provided
[INFO] |  +- org.glassfish.jersey.containers:jersey-container-servlet-core:jar:2.22.2:provided
[INFO] |  +- io.netty:netty-all:jar:4.0.42.Final:compile
[INFO] |  +- io.netty:netty:jar:3.8.0.Final:provided
[INFO] |  +- com.clearspring.analytics:stream:jar:2.7.0:provided
[INFO] |  +- io.dropwizard.metrics:metrics-core:jar:3.1.2:provided
[INFO] |  +- io.dropwizard.metrics:metrics-jvm:jar:3.1.2:provided
[INFO] |  +- io.dropwizard.metrics:metrics-json:jar:3.1.2:provided
[INFO] |  +- io.dropwizard.metrics:metrics-graphite:jar:3.1.2:provided
[INFO] |  +- com.fasterxml.jackson.core:jackson-databind:jar:2.6.5:provided
[INFO] |  |  \- com.fasterxml.jackson.core:jackson-core:jar:2.6.5:provided
[INFO] |  +- com.fasterxml.jackson.module:jackson-module-scala_2.11:jar:2.6.5:provided
[INFO] |  |  \- com.fasterxml.jackson.module:jackson-module-paranamer:jar:2.6.5:provided
[INFO] |  +- org.apache.ivy:ivy:jar:2.4.0:provided
[INFO] |  +- oro:oro:jar:2.0.8:provided
[INFO] |  +- net.razorvine:pyrolite:jar:4.13:provided
[INFO] |  +- net.sf.py4j:py4j:jar:0.10.4:provided
[INFO] |  +- org.apache.spark:spark-tags_2.11:jar:2.1.0:provided
[INFO] |  +- org.apache.commons:commons-crypto:jar:1.0.0:provided
[INFO] |  \- org.spark-project.spark:unused:jar:1.0.0:compile
[INFO] +- org.apache.spark:spark-streaming_2.11:jar:2.1.0:provided
[INFO] +- org.apache.spark:spark-sql_2.11:jar:2.1.0:provided
[INFO] |  +- com.univocity:univocity-parsers:jar:2.2.1:provided
[INFO] |  +- org.apache.spark:spark-sketch_2.11:jar:2.1.0:provided
[INFO] |  +- org.apache.spark:spark-catalyst_2.11:jar:2.1.0:provided
[INFO] |  |  +- org.codehaus.janino:janino:jar:3.0.0:provided
[INFO] |  |  +- org.codehaus.janino:commons-compiler:jar:3.0.0:provided
[INFO] |  |  \- org.antlr:antlr4-runtime:jar:4.5.3:provided
[INFO] |  +- org.apache.parquet:parquet-column:jar:1.8.1:provided
[INFO] |  |  +- org.apache.parquet:parquet-common:jar:1.8.1:provided
[INFO] |  |  \- org.apache.parquet:parquet-encoding:jar:1.8.1:provided
[INFO] |  \- org.apache.parquet:parquet-hadoop:jar:1.8.1:provided
[INFO] |     +- org.apache.parquet:parquet-format:jar:2.3.0-incubating:provided
[INFO] |     \- org.apache.parquet:parquet-jackson:jar:1.8.1:provided
[INFO] +- org.apache.spark:spark-hive_2.11:jar:2.1.0:provided
[INFO] |  +- com.twitter:parquet-hadoop-bundle:jar:1.6.0:provided
[INFO] |  +- org.spark-project.hive:hive-exec:jar:1.2.1.spark2:provided
[INFO] |  |  +- javolution:javolution:jar:5.5.1:provided
[INFO] |  |  +- log4j:apache-log4j-extras:jar:1.2.17:provided
[INFO] |  |  +- org.antlr:antlr-runtime:jar:3.4:provided
[INFO] |  |  |  +- org.antlr:stringtemplate:jar:3.2.1:provided
[INFO] |  |  |  \- antlr:antlr:jar:2.7.7:provided
[INFO] |  |  +- org.antlr:ST4:jar:4.0.4:provided
[INFO] |  |  +- org.apache.commons:commons-compress:jar:1.4.1:compile
[INFO] |  |  |  \- org.tukaani:xz:jar:1.0:compile
[INFO] |  |  +- com.googlecode.javaewah:JavaEWAH:jar:0.3.2:provided
[INFO] |  |  +- org.iq80.snappy:snappy:jar:0.2:provided
[INFO] |  |  +- stax:stax-api:jar:1.0.1:provided
[INFO] |  |  \- net.sf.opencsv:opencsv:jar:2.3:provided
[INFO] |  +- org.spark-project.hive:hive-metastore:jar:1.2.1.spark2:provided
[INFO] |  |  +- com.jolbox:bonecp:jar:0.8.0.RELEASE:provided
[INFO] |  |  +- commons-cli:commons-cli:jar:1.2:compile
[INFO] |  |  +- org.apache.derby:derby:jar:10.10.2.0:provided
[INFO] |  |  +- org.datanucleus:datanucleus-api-jdo:jar:3.2.6:provided
[INFO] |  |  +- org.datanucleus:datanucleus-rdbms:jar:3.2.9:provided
[INFO] |  |  +- commons-pool:commons-pool:jar:1.5.4:provided
[INFO] |  |  +- commons-dbcp:commons-dbcp:jar:1.4:provided
[INFO] |  |  \- javax.jdo:jdo-api:jar:3.0.1:provided
[INFO] |  |     \- javax.transaction:jta:jar:1.1:provided
[INFO] |  +- org.apache.avro:avro:jar:1.7.7:compile
[INFO] |  |  \- com.thoughtworks.paranamer:paranamer:jar:2.3:compile
[INFO] |  +- commons-httpclient:commons-httpclient:jar:3.1:compile
[INFO] |  +- org.apache.calcite:calcite-avatica:jar:1.2.0-incubating:provided
[INFO] |  +- org.apache.calcite:calcite-core:jar:1.2.0-incubating:provided
[INFO] |  |  +- org.apache.calcite:calcite-linq4j:jar:1.2.0-incubating:provided
[INFO] |  |  \- net.hydromatic:eigenbase-properties:jar:1.1.5:provided
[INFO] |  +- org.apache.httpcomponents:httpclient:jar:4.5.2:compile
[INFO] |  |  \- org.apache.httpcomponents:httpcore:jar:4.4.4:compile
[INFO] |  +- org.codehaus.jackson:jackson-mapper-asl:jar:1.9.13:compile
[INFO] |  +- commons-codec:commons-codec:jar:1.10:compile
[INFO] |  +- joda-time:joda-time:jar:2.9.3:provided
[INFO] |  +- org.jodd:jodd-core:jar:3.5.2:provided
[INFO] |  +- org.datanucleus:datanucleus-core:jar:3.2.10:provided
[INFO] |  +- org.apache.thrift:libthrift:jar:0.9.2:provided
[INFO] |  \- org.apache.thrift:libfb303:jar:0.9.2:provided
[INFO] +- org.apache.spark:spark-streaming-kafka_2.11:jar:1.6.3:compile
[INFO] |  \- org.apache.kafka:kafka_2.11:jar:0.8.2.1:compile
[INFO] |     +- org.scala-lang.modules:scala-parser-combinators_2.11:jar:1.0.2:compile
[INFO] |     +- com.101tec:zkclient:jar:0.3:compile
[INFO] |     \- org.apache.kafka:kafka-clients:jar:0.8.2.1:compile
[INFO] +- org.scala-lang:scala-library:jar:2.11.8:provided
[INFO] +- org.scala-lang:scala-actors:jar:2.11.8:provided
[INFO] +- org.scalatest:scalatest_2.11:jar:2.2.1:test
[INFO] |  +- org.scala-lang:scala-reflect:jar:2.11.2:provided
[INFO] |  \- org.scala-lang.modules:scala-xml_2.11:jar:1.0.2:compile
[INFO] +- org.apache.hbase:hbase-client:jar:1.2.0-cdh5.15.0:compile
[INFO] |  +- org.apache.hbase:hbase-annotations:jar:1.2.0-cdh5.15.0:compile
[INFO] |  +- org.apache.hbase:hbase-common:jar:1.2.0-cdh5.15.0:compile
[INFO] |  |  +- commons-collections:commons-collections:jar:3.2.2:compile
[INFO] |  |  +- org.mortbay.jetty:jetty-util:jar:6.1.26.cloudera.4:compile
[INFO] |  |  \- org.apache.hadoop:hadoop-core:jar:2.6.0-mr1-cdh5.15.0:compile
[INFO] |  |     +- tomcat:jasper-runtime:jar:5.5.23:compile
[INFO] |  |     +- tomcat:jasper-compiler:jar:5.5.23:compile
[INFO] |  |     +- javax.servlet.jsp:jsp-api:jar:2.1:compile
[INFO] |  |     +- javax.servlet:servlet-api:jar:2.5:compile
[INFO] |  |     +- hsqldb:hsqldb:jar:1.8.0.10:compile
[INFO] |  |     \- org.eclipse.jdt:core:jar:3.1.1:compile
[INFO] |  +- org.apache.hbase:hbase-protocol:jar:1.2.0-cdh5.15.0:compile
[INFO] |  +- commons-io:commons-io:jar:2.4:compile
[INFO] |  +- commons-lang:commons-lang:jar:2.6:compile
[INFO] |  +- commons-logging:commons-logging:jar:1.2:compile
[INFO] |  +- com.google.guava:guava:jar:12.0.1:compile
[INFO] |  +- com.google.protobuf:protobuf-java:jar:2.5.0:compile
[INFO] |  +- org.apache.zookeeper:zookeeper:jar:3.4.5-cdh5.15.0:compile
[INFO] |  +- org.apache.htrace:htrace-core:jar:3.2.0-incubating:compile
[INFO] |  +- org.jruby.jcodings:jcodings:jar:1.0.8:compile
[INFO] |  +- org.jruby.joni:joni:jar:2.1.2:compile
[INFO] |  +- com.yammer.metrics:metrics-core:jar:2.2.0:compile
[INFO] |  +- org.apache.hadoop:hadoop-common:jar:2.6.0-cdh5.15.0:compile
[INFO] |  |  +- xmlenc:xmlenc:jar:0.52:compile
[INFO] |  |  +- org.mortbay.jetty:jetty:jar:6.1.26.cloudera.4:compile
[INFO] |  |  +- com.sun.jersey:jersey-core:jar:1.9:compile
[INFO] |  |  +- com.sun.jersey:jersey-json:jar:1.9:compile
[INFO] |  |  |  +- org.codehaus.jettison:jettison:jar:1.1:compile
[INFO] |  |  |  +- com.sun.xml.bind:jaxb-impl:jar:2.2.3-1:compile
[INFO] |  |  |  |  \- javax.xml.bind:jaxb-api:jar:2.2.2:compile
[INFO] |  |  |  |     +- javax.xml.stream:stax-api:jar:1.0-2:compile
[INFO] |  |  |  |     \- javax.activation:activation:jar:1.1:compile
[INFO] |  |  |  +- org.codehaus.jackson:jackson-jaxrs:jar:1.8.3:compile
[INFO] |  |  |  \- org.codehaus.jackson:jackson-xc:jar:1.8.3:compile
[INFO] |  |  +- commons-el:commons-el:jar:1.0:compile
[INFO] |  |  +- commons-configuration:commons-configuration:jar:1.6:compile
[INFO] |  |  |  +- commons-digester:commons-digester:jar:1.8:compile
[INFO] |  |  |  |  \- commons-beanutils:commons-beanutils:jar:1.7.0:compile
[INFO] |  |  |  \- commons-beanutils:commons-beanutils-core:jar:1.8.0:compile
[INFO] |  |  +- com.google.code.gson:gson:jar:2.2.4:compile
[INFO] |  |  +- com.jcraft:jsch:jar:0.1.42:compile
[INFO] |  |  +- org.apache.curator:curator-client:jar:2.7.1:compile
[INFO] |  |  \- org.apache.htrace:htrace-core4:jar:4.0.1-incubating:compile
[INFO] |  +- org.apache.hadoop:hadoop-annotations:jar:2.6.0-cdh5.15.0:compile
[INFO] |  +- org.apache.hadoop:hadoop-auth:jar:2.6.0-cdh5.15.0:compile
[INFO] |  |  \- org.apache.directory.server:apacheds-kerberos-codec:jar:2.0.0-M15:compile
[INFO] |  |     +- org.apache.directory.server:apacheds-i18n:jar:2.0.0-M15:compile
[INFO] |  |     +- org.apache.directory.api:api-asn1-api:jar:1.0.0-M20:compile
[INFO] |  |     \- org.apache.directory.api:api-util:jar:1.0.0-M20:compile
[INFO] |  +- com.github.stephenc.findbugs:findbugs-annotations:jar:1.3.9-1:compile
[INFO] |  \- junit:junit:jar:4.12:compile
[INFO] |     \- org.hamcrest:hamcrest-core:jar:1.3:compile
[INFO] \- com.oracle.jdbc:ojdbc6:jar:11.1.0.6.0:compile


But If I open <parent>xxx</parent>, and Module A's tree

[INFO] +- org.apache.spark:spark-core_2.11:jar:2.1.0:provided
[INFO] |  +- org.apache.avro:avro-mapred:jar:hadoop2:1.7.7:provided
[INFO] |  |  +- org.apache.avro:avro-ipc:jar:1.7.7:provided
[INFO] |  |  +- org.apache.avro:avro-ipc:jar:tests:1.7.7:provided
[INFO] |  |  \- org.codehaus.jackson:jackson-core-asl:jar:1.9.13:compile
[INFO] |  +- com.twitter:chill_2.11:jar:0.8.0:provided
[INFO] |  |  \- com.esotericsoftware:kryo-shaded:jar:3.0.3:provided
[INFO] |  |     \- com.esotericsoftware:minlog:jar:1.3.0:provided
[INFO] |  +- com.twitter:chill-java:jar:0.8.0:provided
[INFO] |  +- org.apache.xbean:xbean-asm5-shaded:jar:4.4:provided
[INFO] |  +- org.apache.hadoop:hadoop-client:jar:2.2.0:provided
[INFO] |  |  +- org.apache.hadoop:hadoop-hdfs:jar:2.2.0:provided
[INFO] |  |  +- org.apache.hadoop:hadoop-mapreduce-client-app:jar:2.2.0:provided
[INFO] |  |  |  +- org.apache.hadoop:hadoop-mapreduce-client-common:jar:2.2.0:provided
[INFO] |  |  |  |  +- org.apache.hadoop:hadoop-yarn-client:jar:2.2.0:provided
[INFO] |  |  |  |  |  \- com.google.inject:guice:jar:3.0:provided
[INFO] |  |  |  |  |     +- javax.inject:javax.inject:jar:1:provided
[INFO] |  |  |  |  |     \- aopalliance:aopalliance:jar:1.0:provided
[INFO] |  |  |  |  \- org.apache.hadoop:hadoop-yarn-server-common:jar:2.2.0:provided
[INFO] |  |  |  \- org.apache.hadoop:hadoop-mapreduce-client-shuffle:jar:2.2.0:provided
[INFO] |  |  +- org.apache.hadoop:hadoop-yarn-api:jar:2.2.0:provided
[INFO] |  |  +- org.apache.hadoop:hadoop-mapreduce-client-core:jar:2.2.0:provided
[INFO] |  |  |  \- org.apache.hadoop:hadoop-yarn-common:jar:2.2.0:provided
[INFO] |  |  \- org.apache.hadoop:hadoop-mapreduce-client-jobclient:jar:2.2.0:provided
[INFO] |  +- org.apache.spark:spark-launcher_2.11:jar:2.1.0:provided
[INFO] |  +- org.apache.spark:spark-network-common_2.11:jar:2.1.0:provided
[INFO] |  |  +- org.fusesource.leveldbjni:leveldbjni-all:jar:1.8:provided
[INFO] |  |  \- com.fasterxml.jackson.core:jackson-annotations:jar:2.9.0:compile
[INFO] |  +- org.apache.spark:spark-network-shuffle_2.11:jar:2.1.0:provided
[INFO] |  +- org.apache.spark:spark-unsafe_2.11:jar:2.1.0:provided
[INFO] |  +- net.java.dev.jets3t:jets3t:jar:0.7.1:compile
[INFO] |  +- org.apache.curator:curator-recipes:jar:2.4.0:compile
[INFO] |  |  \- org.apache.curator:curator-framework:jar:2.4.0:compile
[INFO] |  +- javax.servlet:javax.servlet-api:jar:3.1.0:provided
[INFO] |  +- org.apache.commons:commons-lang3:jar:3.7:provided
[INFO] |  +- org.apache.commons:commons-math3:jar:3.4.1:compile
[INFO] |  +- com.google.code.findbugs:jsr305:jar:1.3.9:compile
[INFO] |  +- org.slf4j:slf4j-api:jar:1.7.25:compile
[INFO] |  +- org.slf4j:jul-to-slf4j:jar:1.7.25:provided
[INFO] |  +- org.slf4j:jcl-over-slf4j:jar:1.7.25:provided
[INFO] |  +- log4j:log4j:jar:1.2.17:compile
[INFO] |  +- org.slf4j:slf4j-log4j12:jar:1.7.25:compile
[INFO] |  +- com.ning:compress-lzf:jar:1.0.3:provided
[INFO] |  +- org.xerial.snappy:snappy-java:jar:1.1.2.6:compile
[INFO] |  +- net.jpountz.lz4:lz4:jar:1.3.0:provided
[INFO] |  +- org.roaringbitmap:RoaringBitmap:jar:0.5.11:provided
[INFO] |  +- commons-net:commons-net:jar:2.2:compile
[INFO] |  +- org.json4s:json4s-jackson_2.11:jar:3.2.11:provided
[INFO] |  |  \- org.json4s:json4s-core_2.11:jar:3.2.11:provided
[INFO] |  |     +- org.json4s:json4s-ast_2.11:jar:3.2.11:provided
[INFO] |  |     \- org.scala-lang:scalap:jar:2.11.0:provided
[INFO] |  |        \- org.scala-lang:scala-compiler:jar:2.11.0:provided
[INFO] |  |           \- org.scala-lang.modules:scala-parser-combinators_2.11:jar:1.0.1:provided
[INFO] |  +- org.glassfish.jersey.core:jersey-client:jar:2.26:provided
[INFO] |  |  +- javax.ws.rs:javax.ws.rs-api:jar:2.1:provided
[INFO] |  |  \- org.glassfish.hk2.external:javax.inject:jar:2.5.0-b42:provided
[INFO] |  +- org.glassfish.jersey.core:jersey-common:jar:2.26:provided
[INFO] |  |  +- javax.annotation:javax.annotation-api:jar:1.3.2:provided
[INFO] |  |  \- org.glassfish.hk2:osgi-resource-locator:jar:1.0.1:provided
[INFO] |  +- org.glassfish.jersey.core:jersey-server:jar:2.26:provided
[INFO] |  |  +- org.glassfish.jersey.media:jersey-media-jaxb:jar:2.26:provided
[INFO] |  |  \- javax.validation:validation-api:jar:2.0.1.Final:provided
[INFO] |  +- org.glassfish.jersey.containers:jersey-container-servlet:jar:2.26:provided
[INFO] |  +- org.glassfish.jersey.containers:jersey-container-servlet-core:jar:2.26:provided
[INFO] |  +- io.netty:netty-all:jar:4.1.27.Final:compile
[INFO] |  +- io.netty:netty:jar:3.8.0.Final:provided
[INFO] |  +- com.clearspring.analytics:stream:jar:2.7.0:provided
[INFO] |  +- io.dropwizard.metrics:metrics-core:jar:3.2.6:provided
[INFO] |  +- io.dropwizard.metrics:metrics-jvm:jar:3.2.6:provided
[INFO] |  +- io.dropwizard.metrics:metrics-json:jar:3.2.6:provided
[INFO] |  +- io.dropwizard.metrics:metrics-graphite:jar:3.2.6:provided
[INFO] |  +- com.fasterxml.jackson.core:jackson-databind:jar:2.9.6:compile
[INFO] |  |  \- com.fasterxml.jackson.core:jackson-core:jar:2.9.6:compile
[INFO] |  +- com.fasterxml.jackson.module:jackson-module-scala_2.11:jar:2.9.6:provided
[INFO] |  |  \- com.fasterxml.jackson.module:jackson-module-paranamer:jar:2.9.6:provided
[INFO] |  +- org.apache.ivy:ivy:jar:2.4.0:provided
[INFO] |  +- oro:oro:jar:2.0.8:provided
[INFO] |  +- net.razorvine:pyrolite:jar:4.13:provided
[INFO] |  +- net.sf.py4j:py4j:jar:0.10.4:provided
[INFO] |  +- org.apache.spark:spark-tags_2.11:jar:2.1.0:provided
[INFO] |  +- org.apache.commons:commons-crypto:jar:1.0.0:provided
[INFO] |  \- org.spark-project.spark:unused:jar:1.0.0:compile
[INFO] +- org.apache.spark:spark-streaming_2.11:jar:2.1.0:provided
[INFO] +- org.apache.spark:spark-sql_2.11:jar:2.1.0:provided
[INFO] |  +- com.univocity:univocity-parsers:jar:2.2.1:provided
[INFO] |  +- org.apache.spark:spark-sketch_2.11:jar:2.1.0:provided
[INFO] |  +- org.apache.spark:spark-catalyst_2.11:jar:2.1.0:provided
[INFO] |  |  +- org.codehaus.janino:janino:jar:3.0.8:provided
[INFO] |  |  +- org.codehaus.janino:commons-compiler:jar:3.0.0:provided
[INFO] |  |  \- org.antlr:antlr4-runtime:jar:4.5.3:provided
[INFO] |  +- org.apache.parquet:parquet-column:jar:1.8.1:provided
[INFO] |  |  +- org.apache.parquet:parquet-common:jar:1.8.1:provided
[INFO] |  |  \- org.apache.parquet:parquet-encoding:jar:1.8.1:provided
[INFO] |  \- org.apache.parquet:parquet-hadoop:jar:1.8.1:provided
[INFO] |     +- org.apache.parquet:parquet-format:jar:2.3.0-incubating:provided
[INFO] |     \- org.apache.parquet:parquet-jackson:jar:1.8.1:provided
[INFO] +- org.apache.spark:spark-hive_2.11:jar:2.1.0:provided
[INFO] |  +- com.twitter:parquet-hadoop-bundle:jar:1.6.0:provided
[INFO] |  +- org.spark-project.hive:hive-exec:jar:1.2.1.spark2:provided
[INFO] |  |  +- javolution:javolution:jar:5.5.1:provided
[INFO] |  |  +- log4j:apache-log4j-extras:jar:1.2.17:provided
[INFO] |  |  +- org.antlr:antlr-runtime:jar:3.4:provided
[INFO] |  |  |  +- org.antlr:stringtemplate:jar:3.2.1:provided
[INFO] |  |  |  \- antlr:antlr:jar:2.7.7:provided
[INFO] |  |  +- org.antlr:ST4:jar:4.0.4:provided
[INFO] |  |  +- org.apache.commons:commons-compress:jar:1.4.1:compile
[INFO] |  |  |  \- org.tukaani:xz:jar:1.0:compile
[INFO] |  |  +- com.googlecode.javaewah:JavaEWAH:jar:0.3.2:provided
[INFO] |  |  +- org.iq80.snappy:snappy:jar:0.2:provided
[INFO] |  |  +- stax:stax-api:jar:1.0.1:provided
[INFO] |  |  \- net.sf.opencsv:opencsv:jar:2.3:provided
[INFO] |  +- org.spark-project.hive:hive-metastore:jar:1.2.1.spark2:provided
[INFO] |  |  +- com.jolbox:bonecp:jar:0.8.0.RELEASE:provided
[INFO] |  |  +- commons-cli:commons-cli:jar:1.2:compile
[INFO] |  |  +- org.apache.derby:derby:jar:10.14.1.0:provided
[INFO] |  |  +- org.datanucleus:datanucleus-api-jdo:jar:3.2.6:provided
[INFO] |  |  +- org.datanucleus:datanucleus-rdbms:jar:3.2.9:provided
[INFO] |  |  +- commons-pool:commons-pool:jar:1.6:provided
[INFO] |  |  +- commons-dbcp:commons-dbcp:jar:1.4:provided
[INFO] |  |  \- javax.jdo:jdo-api:jar:3.0.1:provided
[INFO] |  |     \- javax.transaction:jta:jar:1.1:provided
[INFO] |  +- org.apache.avro:avro:jar:1.7.7:compile
[INFO] |  |  \- com.thoughtworks.paranamer:paranamer:jar:2.3:compile
[INFO] |  +- commons-httpclient:commons-httpclient:jar:3.1:compile
[INFO] |  +- org.apache.calcite:calcite-avatica:jar:1.2.0-incubating:provided
[INFO] |  +- org.apache.calcite:calcite-core:jar:1.2.0-incubating:provided
[INFO] |  |  +- org.apache.calcite:calcite-linq4j:jar:1.2.0-incubating:provided
[INFO] |  |  \- net.hydromatic:eigenbase-properties:jar:1.1.5:provided
[INFO] |  +- org.apache.httpcomponents:httpclient:jar:4.5.6:compile
[INFO] |  |  \- org.apache.httpcomponents:httpcore:jar:4.4.10:compile
[INFO] |  +- org.codehaus.jackson:jackson-mapper-asl:jar:1.9.13:compile
[INFO] |  +- commons-codec:commons-codec:jar:1.11:compile
[INFO] |  +- joda-time:joda-time:jar:2.9.9:provided
[INFO] |  +- org.jodd:jodd-core:jar:3.5.2:provided
[INFO] |  +- org.datanucleus:datanucleus-core:jar:3.2.10:provided
[INFO] |  +- org.apache.thrift:libthrift:jar:0.9.2:provided
[INFO] |  \- org.apache.thrift:libfb303:jar:0.9.2:provided
[INFO] +- org.apache.spark:spark-streaming-kafka_2.11:jar:1.6.3:compile
[INFO] |  \- org.apache.kafka:kafka_2.11:jar:1.0.2:compile
[INFO] |     +- org.apache.kafka:kafka-clients:jar:1.0.2:compile
[INFO] |     |  \- org.lz4:lz4-java:jar:1.4:compile
[INFO] |     \- com.101tec:zkclient:jar:0.10:compile
[INFO] +- org.scala-lang:scala-library:jar:2.11.8:provided
[INFO] +- org.scala-lang:scala-actors:jar:2.11.8:provided
[INFO] +- org.scalatest:scalatest_2.11:jar:2.2.1:test
[INFO] |  +- org.scala-lang:scala-reflect:jar:2.11.2:provided
[INFO] |  \- org.scala-lang.modules:scala-xml_2.11:jar:1.0.2:provided
[INFO] +- org.apache.hbase:hbase-client:jar:1.2.0-cdh5.15.0:compile
[INFO] |  +- org.apache.hbase:hbase-annotations:jar:1.2.0-cdh5.15.0:compile
[INFO] |  +- org.apache.hbase:hbase-common:jar:1.2.0-cdh5.15.0:compile
[INFO] |  |  +- commons-collections:commons-collections:jar:3.2.2:compile
[INFO] |  |  +- org.mortbay.jetty:jetty-util:jar:6.1.26.cloudera.4:compile
[INFO] |  |  \- org.apache.hadoop:hadoop-core:jar:2.6.0-mr1-cdh5.15.0:compile
[INFO] |  |     +- tomcat:jasper-runtime:jar:5.5.23:compile
[INFO] |  |     +- tomcat:jasper-compiler:jar:5.5.23:compile
[INFO] |  |     +- javax.servlet.jsp:jsp-api:jar:2.1:compile
[INFO] |  |     +- javax.servlet:servlet-api:jar:2.5:compile
[INFO] |  |     +- hsqldb:hsqldb:jar:1.8.0.10:compile
[INFO] |  |     \- org.eclipse.jdt:core:jar:3.1.1:compile
[INFO] |  +- org.apache.hbase:hbase-protocol:jar:1.2.0-cdh5.15.0:compile
[INFO] |  +- commons-io:commons-io:jar:2.4:compile
[INFO] |  +- commons-lang:commons-lang:jar:2.6:compile
[INFO] |  +- commons-logging:commons-logging:jar:1.2:compile
[INFO] |  +- com.google.guava:guava:jar:12.0.1:compile
[INFO] |  +- com.google.protobuf:protobuf-java:jar:2.5.0:compile
[INFO] |  +- org.apache.zookeeper:zookeeper:jar:3.4.5-cdh5.15.0:compile
[INFO] |  +- org.apache.htrace:htrace-core:jar:3.2.0-incubating:compile
[INFO] |  +- org.jruby.jcodings:jcodings:jar:1.0.8:compile
[INFO] |  +- org.jruby.joni:joni:jar:2.1.2:compile
[INFO] |  +- com.yammer.metrics:metrics-core:jar:2.2.0:compile
[INFO] |  +- org.apache.hadoop:hadoop-common:jar:2.6.0-cdh5.15.0:compile
[INFO] |  |  +- xmlenc:xmlenc:jar:0.52:compile
[INFO] |  |  +- org.mortbay.jetty:jetty:jar:6.1.26.cloudera.4:compile
[INFO] |  |  +- com.sun.jersey:jersey-core:jar:1.9:compile
[INFO] |  |  +- com.sun.jersey:jersey-json:jar:1.9:compile
[INFO] |  |  |  +- org.codehaus.jettison:jettison:jar:1.1:compile
[INFO] |  |  |  +- com.sun.xml.bind:jaxb-impl:jar:2.2.3-1:compile
[INFO] |  |  |  |  \- javax.xml.bind:jaxb-api:jar:2.3.0:compile
[INFO] |  |  |  +- org.codehaus.jackson:jackson-jaxrs:jar:1.8.3:compile
[INFO] |  |  |  \- org.codehaus.jackson:jackson-xc:jar:1.8.3:compile
[INFO] |  |  +- commons-el:commons-el:jar:1.0:compile
[INFO] |  |  +- commons-configuration:commons-configuration:jar:1.6:compile
[INFO] |  |  |  +- commons-digester:commons-digester:jar:1.8:compile
[INFO] |  |  |  |  \- commons-beanutils:commons-beanutils:jar:1.7.0:compile
[INFO] |  |  |  \- commons-beanutils:commons-beanutils-core:jar:1.8.0:compile
[INFO] |  |  +- com.google.code.gson:gson:jar:2.8.5:compile
[INFO] |  |  +- com.jcraft:jsch:jar:0.1.42:compile
[INFO] |  |  +- org.apache.curator:curator-client:jar:2.7.1:compile
[INFO] |  |  \- org.apache.htrace:htrace-core4:jar:4.0.1-incubating:compile
[INFO] |  +- org.apache.hadoop:hadoop-annotations:jar:2.6.0-cdh5.15.0:compile
[INFO] |  +- org.apache.hadoop:hadoop-auth:jar:2.6.0-cdh5.15.0:compile
[INFO] |  |  \- org.apache.directory.server:apacheds-kerberos-codec:jar:2.0.0-M15:compile
[INFO] |  |     +- org.apache.directory.server:apacheds-i18n:jar:2.0.0-M15:compile
[INFO] |  |     +- org.apache.directory.api:api-asn1-api:jar:1.0.0-M20:compile
[INFO] |  |     \- org.apache.directory.api:api-util:jar:1.0.0-M20:compile
[INFO] |  \- com.github.stephenc.findbugs:findbugs-annotations:jar:1.3.9-1:compile
[INFO] +- com.oracle.jdbc:ojdbc6:jar:11.1.0.6.0:compile
[INFO] +- ch.qos.logback:logback-classic:jar:1.2.3:compile
[INFO] |  \- ch.qos.logback:logback-core:jar:1.2.3:compile
[INFO] +- junit:junit:jar:4.12:test
[INFO] |  \- org.hamcrest:hamcrest-core:jar:1.3:test
[INFO] +- org.powermock:powermock-module-junit4:jar:1.7.1:test
[INFO] |  \- org.powermock:powermock-module-junit4-common:jar:1.7.1:test
[INFO] |     +- org.powermock:powermock-reflect:jar:1.7.1:test
[INFO] |     \- org.powermock:powermock-core:jar:1.7.1:test
[INFO] |        \- org.javassist:javassist:jar:3.21.0-GA:test
[INFO] \- org.powermock:powermock-api-mockito2:jar:1.7.1:test
[INFO]    +- org.powermock:powermock-api-mockito-common:jar:1.7.1:test
[INFO]    |  \- org.powermock:powermock-api-support:jar:1.7.1:test
[INFO]    \- org.mockito:mockito-core:jar:2.15.0:test
[INFO]       +- net.bytebuddy:byte-buddy:jar:1.7.11:test
[INFO]       +- net.bytebuddy:byte-buddy-agent:jar:1.7.11:test
[INFO]       \- org.objenesis:objenesis:jar:2.6:provided


The parent pom's dependency tree:(parent pom only has few dependency, same as before)

[INFO] +- ch.qos.logback:logback-classic:jar:1.2.3:compile
[INFO] |  +- ch.qos.logback:logback-core:jar:1.2.3:compile
[INFO] |  \- org.slf4j:slf4j-api:jar:1.7.25:compile
[INFO] +- junit:junit:jar:4.12:test
[INFO] |  \- org.hamcrest:hamcrest-core:jar:1.3:test
[INFO] +- org.powermock:powermock-module-junit4:jar:1.7.1:test
[INFO] |  \- org.powermock:powermock-module-junit4-common:jar:1.7.1:test
[INFO] |     +- org.powermock:powermock-reflect:jar:1.7.1:test
[INFO] |     \- org.powermock:powermock-core:jar:1.7.1:test
[INFO] |        \- org.javassist:javassist:jar:3.21.0-GA:test
[INFO] \- org.powermock:powermock-api-mockito2:jar:1.7.1:test
[INFO]    +- org.powermock:powermock-api-mockito-common:jar:1.7.1:test
[INFO]    |  \- org.powermock:powermock-api-support:jar:1.7.1:test
[INFO]    \- org.mockito:mockito-core:jar:2.15.0:test
[INFO]       +- net.bytebuddy:byte-buddy:jar:1.7.11:test
[INFO]       +- net.bytebuddy:byte-buddy-agent:jar:1.7.11:test
[INFO]       \- org.objenesis:objenesis:jar:2.6:test


Other module trees do not have kafka's dependency, so I don't post it.

thanks.

在 2018/12/18 上午10:42, Liam Clarke 写道:

Can you please post the full output of maven dependency:tree from the
parent POM in both scenarios?


Thanks,

Liam Clarke

On Tue, 18 Dec. 2018, 3:26 pm big data <bi...@outlook.com> wrote:



Hi,

No other dependency include kafka's jar.

The project structure is:

pom.xml

|---Module A

   |--pom.xml

|---Module B

|---Module C

When spark-streaming-kafka_2.11:1.6.3 in in Module A's pom.xml, from
dependency tree, we can see in depends on kafka_2.11:0.8.2.1
When we move spark-streaming-kafka_2.11:1.6.3 dependency to the parent
pom.xml, from tree, it depends on kafka_2.11:1.0.2

In whoel project, only one dependency about streaming-kafka, and no
other dependency about kafka_2.11.


在 2018/12/18 上午10:16, Liam Clarke 写道:


Hi,

Is it a transitive dependency of any of your other dependencies?

Cheers,

Liam Clarke

On Tue, 18 Dec. 2018, 2:57 pm big data <bi...@outlook.com> wrote:



Hi,
   our project includes this dependency by:

<dependency>    <groupId>org.apache.spark</groupId>


<artifactId>spark-streaming-kafka_2.11</artifactId>
<version>1.6.3</version></dependency>



 From dependency tree, we can see it dependency kafka_2.11:0.8.2.1


verson.




But when we move this dependency to project's parent pom file, the
dependency "kafka_2.11" version is changed automatically. Why?


You can see Kafka_2.11 version is changed to 1.0.2?  It's so strange, in
out project we DO NOT dependency this jar in other places.











Re: maven dependency problems

Posted by Liam Clarke <li...@adscale.co.nz>.
Can you please post the full output of maven dependency:tree from the
parent POM in both scenarios?


Thanks,

Liam Clarke

On Tue, 18 Dec. 2018, 3:26 pm big data <bigdatabase@outlook.com wrote:

> Hi,
>
> No other dependency include kafka's jar.
>
> The project structure is:
>
> pom.xml
>
> |---Module A
>
>    |--pom.xml
>
> |---Module B
>
> |---Module C
>
> When spark-streaming-kafka_2.11:1.6.3 in in Module A's pom.xml, from
> dependency tree, we can see in depends on kafka_2.11:0.8.2.1
> When we move spark-streaming-kafka_2.11:1.6.3 dependency to the parent
> pom.xml, from tree, it depends on kafka_2.11:1.0.2
>
> In whoel project, only one dependency about streaming-kafka, and no
> other dependency about kafka_2.11.
>
>
> 在 2018/12/18 上午10:16, Liam Clarke 写道:
> > Hi,
> >
> > Is it a transitive dependency of any of your other dependencies?
> >
> > Cheers,
> >
> > Liam Clarke
> >
> > On Tue, 18 Dec. 2018, 2:57 pm big data <bigdatabase@outlook.com wrote:
> >
> >> Hi,
> >>    our project includes this dependency by:
> >>
> >> <dependency>    <groupId>org.apache.spark</groupId>
> <artifactId>spark-streaming-kafka_2.11</artifactId>
> <version>1.6.3</version></dependency>
> >>
> >>  From dependency tree, we can see it dependency kafka_2.11:0.8.2.1
> verson.
> >>
> >>
> >> But when we move this dependency to project's parent pom file, the
> >> dependency "kafka_2.11" version is changed automatically. Why?
> >>
> >>
> >> You can see Kafka_2.11 version is changed to 1.0.2?  It's so strange, in
> >> out project we DO NOT dependency this jar in other places.
> >>
>
>

Re: maven dependency problems

Posted by big data <bi...@outlook.com>.
Hi,

No other dependency include kafka's jar.

The project structure is:

pom.xml

|---Module A

   |--pom.xml

|---Module B

|---Module C

When spark-streaming-kafka_2.11:1.6.3 in in Module A's pom.xml, from 
dependency tree, we can see in depends on kafka_2.11:0.8.2.1
When we move spark-streaming-kafka_2.11:1.6.3 dependency to the parent 
pom.xml, from tree, it depends on kafka_2.11:1.0.2

In whoel project, only one dependency about streaming-kafka, and no 
other dependency about kafka_2.11.


在 2018/12/18 上午10:16, Liam Clarke 写道:
> Hi,
>
> Is it a transitive dependency of any of your other dependencies?
>
> Cheers,
>
> Liam Clarke
>
> On Tue, 18 Dec. 2018, 2:57 pm big data <bigdatabase@outlook.com wrote:
>
>> Hi,
>>    our project includes this dependency by:
>>
>> <dependency>    <groupId>org.apache.spark</groupId>    <artifactId>spark-streaming-kafka_2.11</artifactId>    <version>1.6.3</version></dependency>
>>
>>  From dependency tree, we can see it dependency kafka_2.11:0.8.2.1 verson.
>>
>>
>> But when we move this dependency to project's parent pom file, the
>> dependency "kafka_2.11" version is changed automatically. Why?
>>
>>
>> You can see Kafka_2.11 version is changed to 1.0.2?  It's so strange, in
>> out project we DO NOT dependency this jar in other places.
>>


Re: maven dependency problems

Posted by Liam Clarke <li...@adscale.co.nz>.
Hi,

Is it a transitive dependency of any of your other dependencies?

Cheers,

Liam Clarke

On Tue, 18 Dec. 2018, 2:57 pm big data <bigdatabase@outlook.com wrote:

> Hi,
>   our project includes this dependency by:
>
> <dependency>    <groupId>org.apache.spark</groupId>    <artifactId>spark-streaming-kafka_2.11</artifactId>    <version>1.6.3</version></dependency>
>
> From dependency tree, we can see it dependency kafka_2.11:0.8.2.1 verson.
>
>
> But when we move this dependency to project's parent pom file, the
> dependency "kafka_2.11" version is changed automatically. Why?
>
>
> You can see Kafka_2.11 version is changed to 1.0.2?  It's so strange, in
> out project we DO NOT dependency this jar in other places.
>

maven dependency problems

Posted by big data <bi...@outlook.com>.
Hi,
  our project includes this dependency by:

<dependency>
    <groupId>org.apache.spark</groupId>
    <artifactId>spark-streaming-kafka_2.11</artifactId>
    <version>1.6.3</version>
</dependency>

From dependency tree, we can see it dependency kafka_2.11:0.8.2.1 verson.
[cid:part1.6C4724D2.B06C4F81@outlook.com]

But when we move this dependency to project's parent pom file, the dependency "kafka_2.11" version is changed automatically. Why?
[cid:part2.913D9571.5BD40CBB@outlook.com]

You can see Kafka_2.11 version is changed to 1.0.2?  It's so strange, in out project we DO NOT dependency this jar in other places.

Re: Configuration of log compaction

Posted by Liam Clarke <li...@adscale.co.nz>.
Hi Claudia,

Anything useful in the log cleaner log files?

Cheers,

Liam Clarke

On Tue, 18 Dec. 2018, 3:18 am Claudia Wegmann <c.wegmann@kasasi.de wrote:

> Hi,
>
> thanks for the quick response.
>
> My problem is not, that no new segments are created, but that segments
> with old data do not get compacted.
> I had to restart one broker because there was no diskspace left. After
> recreating all indexes etc. the broker recognized the old data and
> compacted it correctly. I had to restart all other brokers of the cluster,
> too, for them to also recognize the old data and start compacting.
>
> So I guess, before restarting the brokers where to busy to compact/delete
> old data? Is there a configuration to ensure compaction after a certain
> amount of time or something?
>
> Best,
> Claudia
>
> -----Ursprüngliche Nachricht-----
> Von: Spico Florin <sp...@gmail.com>
> Gesendet: Montag, 17. Dezember 2018 14:28
> An: users@kafka.apache.org
> Betreff: Re: Configuration of log compaction
>
> Hello!
>   Please check whether the segment.ms configuration on topic will help
> you to solve your problem.
>
> https://kafka.apache.org/documentation/
>
>
> https://stackoverflow.com/questions/41048041/kafka-deletes-segments-even-before-segment-size-is-reached
>
> Regards,
>  Florin
>
> segment.ms This configuration controls the period of time after which
> Kafka will force the log to roll even if the segment file isn't full to
> ensure that retention can delete or compact old data. long 604800000
> [1,...] log.roll.ms medium
>
> On Mon, Dec 17, 2018 at 12:28 PM Claudia Wegmann <c....@kasasi.de>
> wrote:
>
> > Dear kafka users,
> >
> > I've got a problem on one of my kafka clusters. I use this cluster
> > with kafka streams applications. Some of this stream apps use a kafka
> > state store. Therefore a changelog topic is created for those stores
> > with cleanup policy "compact". One of these topics is running wild for
> > some time now and seems to grow indefinitely. When I check the  log
> > file of the first segment, there is a lot of data in it, that should
> > have been compacted already.
> >
> > So I guess I did not configure everything correctly for log compaction
> > to work as expected. What config parameters do have influence on log
> > compaction? And how to set them, when I want data older than 4 hours
> > to be compacted?
> >
> > Thanks in advance.
> >
> > Best,
> > Claudia
> >
>

AW: Configuration of log compaction

Posted by Claudia Wegmann <c....@kasasi.de>.
Hi,

thanks for the quick response.

My problem is not, that no new segments are created, but that segments with old data do not get compacted.
I had to restart one broker because there was no diskspace left. After recreating all indexes etc. the broker recognized the old data and compacted it correctly. I had to restart all other brokers of the cluster, too, for them to also recognize the old data and start compacting.

So I guess, before restarting the brokers where to busy to compact/delete old data? Is there a configuration to ensure compaction after a certain amount of time or something?

Best,
Claudia

-----Ursprüngliche Nachricht-----
Von: Spico Florin <sp...@gmail.com> 
Gesendet: Montag, 17. Dezember 2018 14:28
An: users@kafka.apache.org
Betreff: Re: Configuration of log compaction

Hello!
  Please check whether the segment.ms configuration on topic will help you to solve your problem.

https://kafka.apache.org/documentation/

https://stackoverflow.com/questions/41048041/kafka-deletes-segments-even-before-segment-size-is-reached

Regards,
 Florin

segment.ms This configuration controls the period of time after which Kafka will force the log to roll even if the segment file isn't full to ensure that retention can delete or compact old data. long 604800000 [1,...] log.roll.ms medium

On Mon, Dec 17, 2018 at 12:28 PM Claudia Wegmann <c....@kasasi.de>
wrote:

> Dear kafka users,
>
> I've got a problem on one of my kafka clusters. I use this cluster 
> with kafka streams applications. Some of this stream apps use a kafka 
> state store. Therefore a changelog topic is created for those stores 
> with cleanup policy "compact". One of these topics is running wild for 
> some time now and seems to grow indefinitely. When I check the  log 
> file of the first segment, there is a lot of data in it, that should 
> have been compacted already.
>
> So I guess I did not configure everything correctly for log compaction 
> to work as expected. What config parameters do have influence on log 
> compaction? And how to set them, when I want data older than 4 hours 
> to be compacted?
>
> Thanks in advance.
>
> Best,
> Claudia
>

Re: Configuration of log compaction

Posted by Spico Florin <sp...@gmail.com>.
Hello!
  Please check whether the segment.ms configuration on topic will help you
to solve your problem.

https://kafka.apache.org/documentation/

https://stackoverflow.com/questions/41048041/kafka-deletes-segments-even-before-segment-size-is-reached

Regards,
 Florin

segment.ms This configuration controls the period of time after which Kafka
will force the log to roll even if the segment file isn't full to ensure
that retention can delete or compact old data. long 604800000 [1,...]
log.roll.ms medium

On Mon, Dec 17, 2018 at 12:28 PM Claudia Wegmann <c....@kasasi.de>
wrote:

> Dear kafka users,
>
> I've got a problem on one of my kafka clusters. I use this cluster with
> kafka streams applications. Some of this stream apps use a kafka state
> store. Therefore a changelog topic is created for those stores with cleanup
> policy "compact". One of these topics is running wild for some time now and
> seems to grow indefinitely. When I check the  log file of the first
> segment, there is a lot of data in it, that should have been compacted
> already.
>
> So I guess I did not configure everything correctly for log compaction to
> work as expected. What config parameters do have influence on log
> compaction? And how to set them, when I want data older than 4 hours to be
> compacted?
>
> Thanks in advance.
>
> Best,
> Claudia
>