You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@camel.apache.org by Marc Carter <dr...@gmail.com> on 2019/02/09 11:01:36 UTC
DefaultTimeoutMap vs Caffeine
TimeoutMap (primarily used by AggregateProcessor with completionTimeout
) is not well suited to large aggregation maps as it does an O(n) scan
across all registered timeouts every 1000ms. I intended to take a look
at improving this but it's also not great example of open-close either,
all users must extend DefaultTimeoutMap to get any use at all meaning
it's difficult to substitute an experimental implementation. The
further the thought process goes the more it turns into "why not use
Caffeine"
* Already in use in camel-core (LRUCache etc)
* Synchronous eviction listeners -
https://github.com/ben-manes/caffeine/wiki/Writer
* Supports O(1) per-entry timeouts -
https://github.com/ben-manes/caffeine/issues/114
* Has active eviction/refresh capabilities which can (probably) be used
to trigger EIP completed-by-timeout
Thoughts?
Marc
Re: DefaultTimeoutMap vs Caffeine
Posted by Marc Carter <dr...@gmail.com>.
Thanks for passing that on! Is Netty also being removed as a core
dependency?
I have been looking into this and Caffeine is _not_ currently a great
fit as it *doesn't* yet support synchronous eviction listeners (*).
Instead, I now have a pared-down but fully working implementation using
CHM memory-consistency guarantees in place of the explicit Lock,
io.netty.util.HashedWheelTimer to implement evictions and a few JMH
benchmarks for #put(K,V) cost. Remaining work:
* Align with AggregationProcessor locking strategies (this one might be
tricky)
* Figure out if/how to compose suitable scenario tests for something
with asynchronous behaviour and amortising costs.
* Compare Netty HashedWheelTimer vs Caffeine TimerWheel
Marc
(*) Below Issue is exactly the scenario Camel wants. Linked issues
update the accidentally misleading docs:
https://github.com/ben-manes/caffeine/issues/195
On 3/25/19 11:42 AM, Guillaume Nodet wrote:
> I'm currently in the process of making caffeine optional by externalizing
> it from camel-core into its own module, similar to camel-headersmap. It
> will be aumtically wired if in the class path using a custom
> META-INF/services file. So please make sure it actually says optional.
>
> Le sam. 9 févr. 2019 à 12:01, Marc Carter <dr...@gmail.com> a écrit :
>
>> TimeoutMap (primarily used by AggregateProcessor with completionTimeout
>> ) is not well suited to large aggregation maps as it does an O(n) scan
>> across all registered timeouts every 1000ms. I intended to take a look
>> at improving this but it's also not great example of open-close either,
>> all users must extend DefaultTimeoutMap to get any use at all meaning
>> it's difficult to substitute an experimental implementation. The
>> further the thought process goes the more it turns into "why not use
>> Caffeine"
>>
>> * Already in use in camel-core (LRUCache etc)
>>
>> * Synchronous eviction listeners -
>> https://github.com/ben-manes/caffeine/wiki/Writer
>>
>> * Supports O(1) per-entry timeouts -
>> https://github.com/ben-manes/caffeine/issues/114
>>
>> * Has active eviction/refresh capabilities which can (probably) be used
>> to trigger EIP completed-by-timeout
>>
>> Thoughts?
>>
>> Marc
>>
>>
>>
Re: DefaultTimeoutMap vs Caffeine
Posted by Guillaume Nodet <gn...@apache.org>.
I'm currently in the process of making caffeine optional by externalizing
it from camel-core into its own module, similar to camel-headersmap. It
will be aumtically wired if in the class path using a custom
META-INF/services file. So please make sure it actually says optional.
Le sam. 9 févr. 2019 à 12:01, Marc Carter <dr...@gmail.com> a écrit :
> TimeoutMap (primarily used by AggregateProcessor with completionTimeout
> ) is not well suited to large aggregation maps as it does an O(n) scan
> across all registered timeouts every 1000ms. I intended to take a look
> at improving this but it's also not great example of open-close either,
> all users must extend DefaultTimeoutMap to get any use at all meaning
> it's difficult to substitute an experimental implementation. The
> further the thought process goes the more it turns into "why not use
> Caffeine"
>
> * Already in use in camel-core (LRUCache etc)
>
> * Synchronous eviction listeners -
> https://github.com/ben-manes/caffeine/wiki/Writer
>
> * Supports O(1) per-entry timeouts -
> https://github.com/ben-manes/caffeine/issues/114
>
> * Has active eviction/refresh capabilities which can (probably) be used
> to trigger EIP completed-by-timeout
>
> Thoughts?
>
> Marc
>
>
>
--
------------------------
Guillaume Nodet
Re: DefaultTimeoutMap vs Caffeine
Posted by Marc Carter <dr...@gmail.com>.
General preparations for this started with tidying TimeoutMap interface
which exposes too much of its first implementation (PR2794 / CAMEL-13262)
When replacing extend-DefaultTimeoutMap with listen-to-TimeoutMap, there
are two problematic areas uncovered
ITEM1 (in the PR)
Implicit in a _post-event_ listener is the removal of the (unused)
ability to veto an expiry though TimeoutMap.onEviction(). I presume
this was intended for error handling? If anything ever did veto, the map
would simply retry every (100ms) cycle.
ITEM2 (not required for PR - discussion point only for now)
AggregateProcessor.AggregationTimeoutMap#purge locks the whole
AggregateProcessor (when in pessimistic mode) before the periodic scan
for expired entries. I also see, as of today, it can skip the purge
completely during startup.
I think that Caffeine does have a similar hook so the behaviour would be
retained but a generic TimeoutMap should not support this
implementation-specific detail (#1 reason for looking at this code is to
remove that periodic scan). The lock could only be claimed per-evictee
after eviction (and possibly after the same key has been re-added which
is where this becomes 'interesting'!).
Marc
PS. average map sizes are usually only in the 1's/10's/100's though
recovery scenarios might need 1000's/10000's as historic events flood
in. I was looking at "(how quickly) can we replay a full day's traffic"
when noticing this.
On 2/11/19 5:45 AM, Claus Ibsen wrote:
> Hi Marc
>
> Just curious how large aggregation maps do you have?
>
> You are surely welcome to work on an experiment, and log a JIRA ticket.
> Mind that we are working on Camel 3 development, and such a change
> would need to be applicable there, and this change is potentially only
> being a candidate for 3.x, as 2.x development is slowing down, and 2.x
> has been using the current way always.
>
>
> On Sat, Feb 9, 2019 at 12:01 PM Marc Carter <dr...@gmail.com> wrote:
>> TimeoutMap (primarily used by AggregateProcessor with completionTimeout
>> ) is not well suited to large aggregation maps as it does an O(n) scan
>> across all registered timeouts every 1000ms. I intended to take a look
>> at improving this but it's also not great example of open-close either,
>> all users must extend DefaultTimeoutMap to get any use at all meaning
>> it's difficult to substitute an experimental implementation. The
>> further the thought process goes the more it turns into "why not use
>> Caffeine"
>>
>> * Already in use in camel-core (LRUCache etc)
>>
>> * Synchronous eviction listeners -
>> https://github.com/ben-manes/caffeine/wiki/Writer
>>
>> * Supports O(1) per-entry timeouts -
>> https://github.com/ben-manes/caffeine/issues/114
>>
>> * Has active eviction/refresh capabilities which can (probably) be used
>> to trigger EIP completed-by-timeout
>>
>> Thoughts?
>>
>> Marc
>>
>>
>
Re: DefaultTimeoutMap vs Caffeine
Posted by Claus Ibsen <cl...@gmail.com>.
Hi Marc
Just curious how large aggregation maps do you have?
You are surely welcome to work on an experiment, and log a JIRA ticket.
Mind that we are working on Camel 3 development, and such a change
would need to be applicable there, and this change is potentially only
being a candidate for 3.x, as 2.x development is slowing down, and 2.x
has been using the current way always.
On Sat, Feb 9, 2019 at 12:01 PM Marc Carter <dr...@gmail.com> wrote:
>
> TimeoutMap (primarily used by AggregateProcessor with completionTimeout
> ) is not well suited to large aggregation maps as it does an O(n) scan
> across all registered timeouts every 1000ms. I intended to take a look
> at improving this but it's also not great example of open-close either,
> all users must extend DefaultTimeoutMap to get any use at all meaning
> it's difficult to substitute an experimental implementation. The
> further the thought process goes the more it turns into "why not use
> Caffeine"
>
> * Already in use in camel-core (LRUCache etc)
>
> * Synchronous eviction listeners -
> https://github.com/ben-manes/caffeine/wiki/Writer
>
> * Supports O(1) per-entry timeouts -
> https://github.com/ben-manes/caffeine/issues/114
>
> * Has active eviction/refresh capabilities which can (probably) be used
> to trigger EIP completed-by-timeout
>
> Thoughts?
>
> Marc
>
>
--
Claus Ibsen
-----------------
http://davsclaus.com @davsclaus
Camel in Action 2: https://www.manning.com/ibsen2
R: DefaultTimeoutMap vs Caffeine
Posted by Andrea Cosentino <an...@yahoo.com.INVALID>.
We have also a Caffeine component btw
Inviato da Yahoo Mail su Android
Il sab, 9 feb, 2019 alle 12:01, Marc Carter<dr...@gmail.com> ha scritto: TimeoutMap (primarily used by AggregateProcessor with completionTimeout
) is not well suited to large aggregation maps as it does an O(n) scan
across all registered timeouts every 1000ms. I intended to take a look
at improving this but it's also not great example of open-close either,
all users must extend DefaultTimeoutMap to get any use at all meaning
it's difficult to substitute an experimental implementation. The
further the thought process goes the more it turns into "why not use
Caffeine"
* Already in use in camel-core (LRUCache etc)
* Synchronous eviction listeners -
https://github.com/ben-manes/caffeine/wiki/Writer
* Supports O(1) per-entry timeouts -
https://github.com/ben-manes/caffeine/issues/114
* Has active eviction/refresh capabilities which can (probably) be used
to trigger EIP completed-by-timeout
Thoughts?
Marc
Re: IntelliJ IDEA setup for Apache Camel
Posted by Zoran Regvart <zo...@regvart.com>.
Hi Cameleers,
I've changed the headers (from `/**` to `/*`), make sure that you have
a freshly built buildingtools module installed in your local Maven
repository so all the configuration (license, checkstyle) lines up.
Thanks Marc for bringing attention to this and pushing it through :)
zoran
On Mon, Mar 18, 2019 at 4:23 PM Zoran Regvart <zo...@regvart.com> wrote:
>
> Hi Marc,
> I can do that not an issue :) I'll open an issue[1] first and let
> anyone else to chime in if they wish.
>
> zoran
>
> [1] https://issues.apache.org/jira/browse/CAMEL-13336
>
> On Mon, Mar 18, 2019 at 2:19 PM Marc Carter <dr...@gmail.com> wrote:
> >
> > Ok. Not sure if I should take that on via non-committer public PR of
> > 18000 Java files?
> >
> > On 3/18/19 12:29 PM, Zoran Regvart wrote:
> > > Hi Marc,
> > > I would not mind to touch every Java file to fix the headers, that
> > > should also be easily automated via `mvn license:format` after
> > > changing the `header-java.txt`[1], we should also make sure that
> > > Checkstyle and RAT configuration matches.
> > >
> > > zoran
> > >
> > > [1] https://github.com/apache/camel/blob/master/buildingtools/src/main/resources/header-java.txt
> > >
> > > On Mon, Mar 18, 2019 at 12:01 PM Marc Carter <dr...@gmail.com> wrote:
> > >> Interesting one perhaps Apache knows about at a general level...
> > >> Per comments on IntelliJ issue I raised:
> > >> https://youtrack.jetbrains.com/issue/IDEA-208439 Java formatting amends OSS license headers as JavaDoc
> > >>
> > >> Camel (* and apparently many others) is mistakenly applying the *.java license header as a /** */ dangling Javadoc instead of a /* */ multi-line comment. Eclipse has a "workaround" in its formatter to skip header-comments. Options:
> > >> (a) make no change
> > >> I will add explicit IntellIJ formatter rules to ignore all "dangling javadocs" (overly broad but not too evil provided CheckStyle backs us up by enabling the same as a read-only verification)
> > >> (b) change the header file and let Java files change "organically"
> > >> Probably a terrible idea...
> > >> (c) touch every source file
> > >> I wouldn't have bothered raising this normally but, right now, (c) is not dissimilar to the major directory restructuring occurring in 3.0.0
> > >>
> > >> Marc
> > >>
> > >> (*) FYI
> > >> https://issues.apache.org/jira/browse/HIVE-17952 Fix license headers to avoid dangling javadoc warnings
> > >> https://github.com/palantir/atlasdb/issues/1693Copyright text in code should be in comment and not JavaDoc
> > >> https://github.com/mycila/license-maven-plugin/issues/118 license
> > >> headers for java files https://github.com/yegor256/eo/issues/109 Qulice
> > >> incorrectly requires that the license text is in a Javadoc comment
> > >>
> > >> On 3/6/19 9:24 AM, Zoran Regvart wrote:
> > >>> Hi Marc,
> > >>> yes, the files in `buildingtools/src/main/resources` are used when we
> > >>> run Checkstyle as a part of our build on Jenkins and during the
> > >>> release.
> > >>>
> > >>> You can also cleanup the duplicate files, not sure why those remained there.
> > >>>
> > >>> Thanks :)
> > >>>
> > >>> zoran
> > >>>
> > >>> On Wed, Mar 6, 2019 at 10:22 AM Marc Carter <dr...@gmail.com> wrote:
> > >>>> So I'm hearing that `buildingtools/src/main/resources` is canonical
> > >>>> (including all dupe files in `buildingtools`). They have slightly
> > >>>> different content and even different change histories - probably because
> > >>>> of this confusion. I'm not going to check the fine detail of all of
> > >>>> these though. If unofficial ones are not part of the builds, they should
> > >>>> be deleted?
> > >>>>
> > >>>> I'm going to PR the result of these conversations along with IJ setup
> > >>>> when ready
> > >>>>
> > >>>> Marc
> > >>>>
> > >>>> PS. Raised an issue on IJ code formatting as we cannot yet just "import
> > >>>> the Eclipse format" because of a limitation
> > >>>>
> > >>>> https://youtrack.jetbrains.com/issue/IDEA-208439 Java formatting amends
> > >>>> OSS license headers
> > >>>>
> > >>>> On 3/6/19 8:53 AM, Zoran Regvart wrote:
> > >>>>> Hi Cameleers,
> > >>>>> On Wed, Mar 6, 2019 at 2:03 AM Willem Jiang <wi...@gmail.com> wrote:
> > >>>>>> But I cannot find the checkstyle plugin setting with the building
> > >>>>>> tools on the master branch.
> > >>>>>>
> > >>>>>> [1]https://github.com/apache/camel/blob/84c003e039e00622313d0328270c5f139b6a8821/pom.xml#L407
> > >>>>> I think that points to a classpath location[1] which in turn is
> > >>>>> contained in the `camel-buildtools` module[2].
> > >>>>>
> > >>>>> I usually configure Eclipse using M2E and there's a Checkstyle
> > >>>>> extension[3] that picks up the configuration from the pom.xml file, so
> > >>>>> no need for manual setup via Ant...
> > >>>>>
> > >>>>> zoran
> > >>>>>
> > >>>>> [1] https://github.com/apache/camel/blob/84c003e039e00622313d0328270c5f139b6a8821/pom.xml#L399
> > >>>>> [2] https://github.com/apache/camel/tree/1697ba6074aea80fe341dd07baac4fd85165dd74/buildingtools
> > >>>>> [3] https://m2e-code-quality.github.io/m2e-code-quality/
> > >>>
> > >
> > >
>
>
>
> --
> Zoran Regvart
--
Zoran Regvart
Re: IntelliJ IDEA setup for Apache Camel
Posted by Zoran Regvart <zo...@regvart.com>.
Hi Marc,
I can do that not an issue :) I'll open an issue[1] first and let
anyone else to chime in if they wish.
zoran
[1] https://issues.apache.org/jira/browse/CAMEL-13336
On Mon, Mar 18, 2019 at 2:19 PM Marc Carter <dr...@gmail.com> wrote:
>
> Ok. Not sure if I should take that on via non-committer public PR of
> 18000 Java files?
>
> On 3/18/19 12:29 PM, Zoran Regvart wrote:
> > Hi Marc,
> > I would not mind to touch every Java file to fix the headers, that
> > should also be easily automated via `mvn license:format` after
> > changing the `header-java.txt`[1], we should also make sure that
> > Checkstyle and RAT configuration matches.
> >
> > zoran
> >
> > [1] https://github.com/apache/camel/blob/master/buildingtools/src/main/resources/header-java.txt
> >
> > On Mon, Mar 18, 2019 at 12:01 PM Marc Carter <dr...@gmail.com> wrote:
> >> Interesting one perhaps Apache knows about at a general level...
> >> Per comments on IntelliJ issue I raised:
> >> https://youtrack.jetbrains.com/issue/IDEA-208439 Java formatting amends OSS license headers as JavaDoc
> >>
> >> Camel (* and apparently many others) is mistakenly applying the *.java license header as a /** */ dangling Javadoc instead of a /* */ multi-line comment. Eclipse has a "workaround" in its formatter to skip header-comments. Options:
> >> (a) make no change
> >> I will add explicit IntellIJ formatter rules to ignore all "dangling javadocs" (overly broad but not too evil provided CheckStyle backs us up by enabling the same as a read-only verification)
> >> (b) change the header file and let Java files change "organically"
> >> Probably a terrible idea...
> >> (c) touch every source file
> >> I wouldn't have bothered raising this normally but, right now, (c) is not dissimilar to the major directory restructuring occurring in 3.0.0
> >>
> >> Marc
> >>
> >> (*) FYI
> >> https://issues.apache.org/jira/browse/HIVE-17952 Fix license headers to avoid dangling javadoc warnings
> >> https://github.com/palantir/atlasdb/issues/1693Copyright text in code should be in comment and not JavaDoc
> >> https://github.com/mycila/license-maven-plugin/issues/118 license
> >> headers for java files https://github.com/yegor256/eo/issues/109 Qulice
> >> incorrectly requires that the license text is in a Javadoc comment
> >>
> >> On 3/6/19 9:24 AM, Zoran Regvart wrote:
> >>> Hi Marc,
> >>> yes, the files in `buildingtools/src/main/resources` are used when we
> >>> run Checkstyle as a part of our build on Jenkins and during the
> >>> release.
> >>>
> >>> You can also cleanup the duplicate files, not sure why those remained there.
> >>>
> >>> Thanks :)
> >>>
> >>> zoran
> >>>
> >>> On Wed, Mar 6, 2019 at 10:22 AM Marc Carter <dr...@gmail.com> wrote:
> >>>> So I'm hearing that `buildingtools/src/main/resources` is canonical
> >>>> (including all dupe files in `buildingtools`). They have slightly
> >>>> different content and even different change histories - probably because
> >>>> of this confusion. I'm not going to check the fine detail of all of
> >>>> these though. If unofficial ones are not part of the builds, they should
> >>>> be deleted?
> >>>>
> >>>> I'm going to PR the result of these conversations along with IJ setup
> >>>> when ready
> >>>>
> >>>> Marc
> >>>>
> >>>> PS. Raised an issue on IJ code formatting as we cannot yet just "import
> >>>> the Eclipse format" because of a limitation
> >>>>
> >>>> https://youtrack.jetbrains.com/issue/IDEA-208439 Java formatting amends
> >>>> OSS license headers
> >>>>
> >>>> On 3/6/19 8:53 AM, Zoran Regvart wrote:
> >>>>> Hi Cameleers,
> >>>>> On Wed, Mar 6, 2019 at 2:03 AM Willem Jiang <wi...@gmail.com> wrote:
> >>>>>> But I cannot find the checkstyle plugin setting with the building
> >>>>>> tools on the master branch.
> >>>>>>
> >>>>>> [1]https://github.com/apache/camel/blob/84c003e039e00622313d0328270c5f139b6a8821/pom.xml#L407
> >>>>> I think that points to a classpath location[1] which in turn is
> >>>>> contained in the `camel-buildtools` module[2].
> >>>>>
> >>>>> I usually configure Eclipse using M2E and there's a Checkstyle
> >>>>> extension[3] that picks up the configuration from the pom.xml file, so
> >>>>> no need for manual setup via Ant...
> >>>>>
> >>>>> zoran
> >>>>>
> >>>>> [1] https://github.com/apache/camel/blob/84c003e039e00622313d0328270c5f139b6a8821/pom.xml#L399
> >>>>> [2] https://github.com/apache/camel/tree/1697ba6074aea80fe341dd07baac4fd85165dd74/buildingtools
> >>>>> [3] https://m2e-code-quality.github.io/m2e-code-quality/
> >>>
> >
> >
--
Zoran Regvart
Re: IntelliJ IDEA setup for Apache Camel
Posted by Marc Carter <dr...@gmail.com>.
Ok. Not sure if I should take that on via non-committer public PR of
18000 Java files?
On 3/18/19 12:29 PM, Zoran Regvart wrote:
> Hi Marc,
> I would not mind to touch every Java file to fix the headers, that
> should also be easily automated via `mvn license:format` after
> changing the `header-java.txt`[1], we should also make sure that
> Checkstyle and RAT configuration matches.
>
> zoran
>
> [1] https://github.com/apache/camel/blob/master/buildingtools/src/main/resources/header-java.txt
>
> On Mon, Mar 18, 2019 at 12:01 PM Marc Carter <dr...@gmail.com> wrote:
>> Interesting one perhaps Apache knows about at a general level...
>> Per comments on IntelliJ issue I raised:
>> https://youtrack.jetbrains.com/issue/IDEA-208439 Java formatting amends OSS license headers as JavaDoc
>>
>> Camel (* and apparently many others) is mistakenly applying the *.java license header as a /** */ dangling Javadoc instead of a /* */ multi-line comment. Eclipse has a "workaround" in its formatter to skip header-comments. Options:
>> (a) make no change
>> I will add explicit IntellIJ formatter rules to ignore all "dangling javadocs" (overly broad but not too evil provided CheckStyle backs us up by enabling the same as a read-only verification)
>> (b) change the header file and let Java files change "organically"
>> Probably a terrible idea...
>> (c) touch every source file
>> I wouldn't have bothered raising this normally but, right now, (c) is not dissimilar to the major directory restructuring occurring in 3.0.0
>>
>> Marc
>>
>> (*) FYI
>> https://issues.apache.org/jira/browse/HIVE-17952 Fix license headers to avoid dangling javadoc warnings
>> https://github.com/palantir/atlasdb/issues/1693Copyright text in code should be in comment and not JavaDoc
>> https://github.com/mycila/license-maven-plugin/issues/118 license
>> headers for java files https://github.com/yegor256/eo/issues/109 Qulice
>> incorrectly requires that the license text is in a Javadoc comment
>>
>> On 3/6/19 9:24 AM, Zoran Regvart wrote:
>>> Hi Marc,
>>> yes, the files in `buildingtools/src/main/resources` are used when we
>>> run Checkstyle as a part of our build on Jenkins and during the
>>> release.
>>>
>>> You can also cleanup the duplicate files, not sure why those remained there.
>>>
>>> Thanks :)
>>>
>>> zoran
>>>
>>> On Wed, Mar 6, 2019 at 10:22 AM Marc Carter <dr...@gmail.com> wrote:
>>>> So I'm hearing that `buildingtools/src/main/resources` is canonical
>>>> (including all dupe files in `buildingtools`). They have slightly
>>>> different content and even different change histories - probably because
>>>> of this confusion. I'm not going to check the fine detail of all of
>>>> these though. If unofficial ones are not part of the builds, they should
>>>> be deleted?
>>>>
>>>> I'm going to PR the result of these conversations along with IJ setup
>>>> when ready
>>>>
>>>> Marc
>>>>
>>>> PS. Raised an issue on IJ code formatting as we cannot yet just "import
>>>> the Eclipse format" because of a limitation
>>>>
>>>> https://youtrack.jetbrains.com/issue/IDEA-208439 Java formatting amends
>>>> OSS license headers
>>>>
>>>> On 3/6/19 8:53 AM, Zoran Regvart wrote:
>>>>> Hi Cameleers,
>>>>> On Wed, Mar 6, 2019 at 2:03 AM Willem Jiang <wi...@gmail.com> wrote:
>>>>>> But I cannot find the checkstyle plugin setting with the building
>>>>>> tools on the master branch.
>>>>>>
>>>>>> [1]https://github.com/apache/camel/blob/84c003e039e00622313d0328270c5f139b6a8821/pom.xml#L407
>>>>> I think that points to a classpath location[1] which in turn is
>>>>> contained in the `camel-buildtools` module[2].
>>>>>
>>>>> I usually configure Eclipse using M2E and there's a Checkstyle
>>>>> extension[3] that picks up the configuration from the pom.xml file, so
>>>>> no need for manual setup via Ant...
>>>>>
>>>>> zoran
>>>>>
>>>>> [1] https://github.com/apache/camel/blob/84c003e039e00622313d0328270c5f139b6a8821/pom.xml#L399
>>>>> [2] https://github.com/apache/camel/tree/1697ba6074aea80fe341dd07baac4fd85165dd74/buildingtools
>>>>> [3] https://m2e-code-quality.github.io/m2e-code-quality/
>>>
>
>
Re: IntelliJ IDEA setup for Apache Camel
Posted by Zoran Regvart <zo...@regvart.com>.
Hi Marc,
I would not mind to touch every Java file to fix the headers, that
should also be easily automated via `mvn license:format` after
changing the `header-java.txt`[1], we should also make sure that
Checkstyle and RAT configuration matches.
zoran
[1] https://github.com/apache/camel/blob/master/buildingtools/src/main/resources/header-java.txt
On Mon, Mar 18, 2019 at 12:01 PM Marc Carter <dr...@gmail.com> wrote:
>
> Interesting one perhaps Apache knows about at a general level...
> Per comments on IntelliJ issue I raised:
> https://youtrack.jetbrains.com/issue/IDEA-208439 Java formatting amends OSS license headers as JavaDoc
>
> Camel (* and apparently many others) is mistakenly applying the *.java license header as a /** */ dangling Javadoc instead of a /* */ multi-line comment. Eclipse has a "workaround" in its formatter to skip header-comments. Options:
> (a) make no change
> I will add explicit IntellIJ formatter rules to ignore all "dangling javadocs" (overly broad but not too evil provided CheckStyle backs us up by enabling the same as a read-only verification)
> (b) change the header file and let Java files change "organically"
> Probably a terrible idea...
> (c) touch every source file
> I wouldn't have bothered raising this normally but, right now, (c) is not dissimilar to the major directory restructuring occurring in 3.0.0
>
> Marc
>
> (*) FYI
> https://issues.apache.org/jira/browse/HIVE-17952 Fix license headers to avoid dangling javadoc warnings
> https://github.com/palantir/atlasdb/issues/1693Copyright text in code should be in comment and not JavaDoc
> https://github.com/mycila/license-maven-plugin/issues/118 license
> headers for java files https://github.com/yegor256/eo/issues/109 Qulice
> incorrectly requires that the license text is in a Javadoc comment
>
> On 3/6/19 9:24 AM, Zoran Regvart wrote:
> > Hi Marc,
> > yes, the files in `buildingtools/src/main/resources` are used when we
> > run Checkstyle as a part of our build on Jenkins and during the
> > release.
> >
> > You can also cleanup the duplicate files, not sure why those remained there.
> >
> > Thanks :)
> >
> > zoran
> >
> > On Wed, Mar 6, 2019 at 10:22 AM Marc Carter <dr...@gmail.com> wrote:
> >> So I'm hearing that `buildingtools/src/main/resources` is canonical
> >> (including all dupe files in `buildingtools`). They have slightly
> >> different content and even different change histories - probably because
> >> of this confusion. I'm not going to check the fine detail of all of
> >> these though. If unofficial ones are not part of the builds, they should
> >> be deleted?
> >>
> >> I'm going to PR the result of these conversations along with IJ setup
> >> when ready
> >>
> >> Marc
> >>
> >> PS. Raised an issue on IJ code formatting as we cannot yet just "import
> >> the Eclipse format" because of a limitation
> >>
> >> https://youtrack.jetbrains.com/issue/IDEA-208439 Java formatting amends
> >> OSS license headers
> >>
> >> On 3/6/19 8:53 AM, Zoran Regvart wrote:
> >>> Hi Cameleers,
> >>> On Wed, Mar 6, 2019 at 2:03 AM Willem Jiang <wi...@gmail.com> wrote:
> >>>> But I cannot find the checkstyle plugin setting with the building
> >>>> tools on the master branch.
> >>>>
> >>>> [1]https://github.com/apache/camel/blob/84c003e039e00622313d0328270c5f139b6a8821/pom.xml#L407
> >>> I think that points to a classpath location[1] which in turn is
> >>> contained in the `camel-buildtools` module[2].
> >>>
> >>> I usually configure Eclipse using M2E and there's a Checkstyle
> >>> extension[3] that picks up the configuration from the pom.xml file, so
> >>> no need for manual setup via Ant...
> >>>
> >>> zoran
> >>>
> >>> [1] https://github.com/apache/camel/blob/84c003e039e00622313d0328270c5f139b6a8821/pom.xml#L399
> >>> [2] https://github.com/apache/camel/tree/1697ba6074aea80fe341dd07baac4fd85165dd74/buildingtools
> >>> [3] https://m2e-code-quality.github.io/m2e-code-quality/
> >
> >
--
Zoran Regvart
Re: IntelliJ IDEA setup for Apache Camel
Posted by Marc Carter <dr...@gmail.com>.
Interesting one perhaps Apache knows about at a general level...
Per comments on IntelliJ issue I raised:
https://youtrack.jetbrains.com/issue/IDEA-208439 Java formatting amends OSS license headers as JavaDoc
Camel (* and apparently many others) is mistakenly applying the *.java license header as a /** */ dangling Javadoc instead of a /* */ multi-line comment. Eclipse has a "workaround" in its formatter to skip header-comments. Options:
(a) make no change
I will add explicit IntellIJ formatter rules to ignore all "dangling javadocs" (overly broad but not too evil provided CheckStyle backs us up by enabling the same as a read-only verification)
(b) change the header file and let Java files change "organically"
Probably a terrible idea...
(c) touch every source file
I wouldn't have bothered raising this normally but, right now, (c) is not dissimilar to the major directory restructuring occurring in 3.0.0
Marc
(*) FYI
https://issues.apache.org/jira/browse/HIVE-17952 Fix license headers to avoid dangling javadoc warnings
https://github.com/palantir/atlasdb/issues/1693Copyright text in code should be in comment and not JavaDoc
https://github.com/mycila/license-maven-plugin/issues/118 license
headers for java files https://github.com/yegor256/eo/issues/109 Qulice
incorrectly requires that the license text is in a Javadoc comment
On 3/6/19 9:24 AM, Zoran Regvart wrote:
> Hi Marc,
> yes, the files in `buildingtools/src/main/resources` are used when we
> run Checkstyle as a part of our build on Jenkins and during the
> release.
>
> You can also cleanup the duplicate files, not sure why those remained there.
>
> Thanks :)
>
> zoran
>
> On Wed, Mar 6, 2019 at 10:22 AM Marc Carter <dr...@gmail.com> wrote:
>> So I'm hearing that `buildingtools/src/main/resources` is canonical
>> (including all dupe files in `buildingtools`). They have slightly
>> different content and even different change histories - probably because
>> of this confusion. I'm not going to check the fine detail of all of
>> these though. If unofficial ones are not part of the builds, they should
>> be deleted?
>>
>> I'm going to PR the result of these conversations along with IJ setup
>> when ready
>>
>> Marc
>>
>> PS. Raised an issue on IJ code formatting as we cannot yet just "import
>> the Eclipse format" because of a limitation
>>
>> https://youtrack.jetbrains.com/issue/IDEA-208439 Java formatting amends
>> OSS license headers
>>
>> On 3/6/19 8:53 AM, Zoran Regvart wrote:
>>> Hi Cameleers,
>>> On Wed, Mar 6, 2019 at 2:03 AM Willem Jiang <wi...@gmail.com> wrote:
>>>> But I cannot find the checkstyle plugin setting with the building
>>>> tools on the master branch.
>>>>
>>>> [1]https://github.com/apache/camel/blob/84c003e039e00622313d0328270c5f139b6a8821/pom.xml#L407
>>> I think that points to a classpath location[1] which in turn is
>>> contained in the `camel-buildtools` module[2].
>>>
>>> I usually configure Eclipse using M2E and there's a Checkstyle
>>> extension[3] that picks up the configuration from the pom.xml file, so
>>> no need for manual setup via Ant...
>>>
>>> zoran
>>>
>>> [1] https://github.com/apache/camel/blob/84c003e039e00622313d0328270c5f139b6a8821/pom.xml#L399
>>> [2] https://github.com/apache/camel/tree/1697ba6074aea80fe341dd07baac4fd85165dd74/buildingtools
>>> [3] https://m2e-code-quality.github.io/m2e-code-quality/
>
>
Re: IntelliJ IDEA setup for Apache Camel
Posted by Zoran Regvart <zo...@regvart.com>.
Hi Marc,
yes, the files in `buildingtools/src/main/resources` are used when we
run Checkstyle as a part of our build on Jenkins and during the
release.
You can also cleanup the duplicate files, not sure why those remained there.
Thanks :)
zoran
On Wed, Mar 6, 2019 at 10:22 AM Marc Carter <dr...@gmail.com> wrote:
>
> So I'm hearing that `buildingtools/src/main/resources` is canonical
> (including all dupe files in `buildingtools`). They have slightly
> different content and even different change histories - probably because
> of this confusion. I'm not going to check the fine detail of all of
> these though. If unofficial ones are not part of the builds, they should
> be deleted?
>
> I'm going to PR the result of these conversations along with IJ setup
> when ready
>
> Marc
>
> PS. Raised an issue on IJ code formatting as we cannot yet just "import
> the Eclipse format" because of a limitation
>
> https://youtrack.jetbrains.com/issue/IDEA-208439 Java formatting amends
> OSS license headers
>
> On 3/6/19 8:53 AM, Zoran Regvart wrote:
> > Hi Cameleers,
> > On Wed, Mar 6, 2019 at 2:03 AM Willem Jiang <wi...@gmail.com> wrote:
> >> But I cannot find the checkstyle plugin setting with the building
> >> tools on the master branch.
> >>
> >> [1]https://github.com/apache/camel/blob/84c003e039e00622313d0328270c5f139b6a8821/pom.xml#L407
> > I think that points to a classpath location[1] which in turn is
> > contained in the `camel-buildtools` module[2].
> >
> > I usually configure Eclipse using M2E and there's a Checkstyle
> > extension[3] that picks up the configuration from the pom.xml file, so
> > no need for manual setup via Ant...
> >
> > zoran
> >
> > [1] https://github.com/apache/camel/blob/84c003e039e00622313d0328270c5f139b6a8821/pom.xml#L399
> > [2] https://github.com/apache/camel/tree/1697ba6074aea80fe341dd07baac4fd85165dd74/buildingtools
> > [3] https://m2e-code-quality.github.io/m2e-code-quality/
--
Zoran Regvart
Re: IntelliJ IDEA setup for Apache Camel
Posted by Marc Carter <dr...@gmail.com>.
So I'm hearing that `buildingtools/src/main/resources` is canonical
(including all dupe files in `buildingtools`). They have slightly
different content and even different change histories - probably because
of this confusion. I'm not going to check the fine detail of all of
these though. If unofficial ones are not part of the builds, they should
be deleted?
I'm going to PR the result of these conversations along with IJ setup
when ready
Marc
PS. Raised an issue on IJ code formatting as we cannot yet just "import
the Eclipse format" because of a limitation
https://youtrack.jetbrains.com/issue/IDEA-208439 Java formatting amends
OSS license headers
On 3/6/19 8:53 AM, Zoran Regvart wrote:
> Hi Cameleers,
> On Wed, Mar 6, 2019 at 2:03 AM Willem Jiang <wi...@gmail.com> wrote:
>> But I cannot find the checkstyle plugin setting with the building
>> tools on the master branch.
>>
>> [1]https://github.com/apache/camel/blob/84c003e039e00622313d0328270c5f139b6a8821/pom.xml#L407
> I think that points to a classpath location[1] which in turn is
> contained in the `camel-buildtools` module[2].
>
> I usually configure Eclipse using M2E and there's a Checkstyle
> extension[3] that picks up the configuration from the pom.xml file, so
> no need for manual setup via Ant...
>
> zoran
>
> [1] https://github.com/apache/camel/blob/84c003e039e00622313d0328270c5f139b6a8821/pom.xml#L399
> [2] https://github.com/apache/camel/tree/1697ba6074aea80fe341dd07baac4fd85165dd74/buildingtools
> [3] https://m2e-code-quality.github.io/m2e-code-quality/
Re: IntelliJ IDEA setup for Apache Camel
Posted by Zoran Regvart <zo...@regvart.com>.
Hi Cameleers,
On Wed, Mar 6, 2019 at 2:03 AM Willem Jiang <wi...@gmail.com> wrote:
> But I cannot find the checkstyle plugin setting with the building
> tools on the master branch.
>
> [1]https://github.com/apache/camel/blob/84c003e039e00622313d0328270c5f139b6a8821/pom.xml#L407
I think that points to a classpath location[1] which in turn is
contained in the `camel-buildtools` module[2].
I usually configure Eclipse using M2E and there's a Checkstyle
extension[3] that picks up the configuration from the pom.xml file, so
no need for manual setup via Ant...
zoran
[1] https://github.com/apache/camel/blob/84c003e039e00622313d0328270c5f139b6a8821/pom.xml#L399
[2] https://github.com/apache/camel/tree/1697ba6074aea80fe341dd07baac4fd85165dd74/buildingtools
[3] https://m2e-code-quality.github.io/m2e-code-quality/
--
Zoran Regvart
Re: IntelliJ IDEA setup for Apache Camel
Posted by Willem Jiang <wi...@gmail.com>.
Willem JiangTwitter: willemjiangWeibo: 姜宁willem
On Wed, Mar 6, 2019 at 12:39 AM Marc Carter <dr...@gmail.com> wrote:
>
> I took a look at that and it's helped a bit but this does need a
> retelling. For the sake of sanity, CheckStyle and Eclipse rules should
> be golden - IntelliJ has decent import system for both that can be (re)
> documented. However (isn't there always something...)
>
> a) etc/eclipse/CamelCodeFormatter.xml contains an indent of 8 which is
> obviously incorrect (clashes with CS rules). Is this the correct file?
>
> <setting id="org.eclipse.jdt.core.formatter.indentation.size" value="8"/>
>
It's wrong, do you mind send a PR for it.
> b) buildingtools/camel-checkstyle.xml differs from
> buildingtools/src/main/resources/camel-checkstyle.xml
>
> Which is correct?
It's confuse the user.
The check style file in the resource directory[1] is used in setting
up the eclipse workspace.
But I cannot find the checkstyle plugin setting with the building
tools on the master branch.
[1]https://github.com/apache/camel/blob/84c003e039e00622313d0328270c5f139b6a8821/pom.xml#L407
>
> On 3/5/19 11:45 AM, Onder SEZGIN wrote:
> > http://camel.apache.org/set-up-your-ide.html
> >
> > see this file mentioned below is on older branches on github.
> >
> > *etc/idea/settings.jar*
> >
> > On Tue, Mar 5, 2019 at 1:42 PM Marc Carter <dr...@gmail.com> wrote:
> >
> >> Official instructions for IntelliJ(*) point to files that no longer
> >> exist. I'm sure there's some history to this but it does mean I (and
> >> anyone else not using Eclipse) is missing the checkstyle and format
> >> rules that prevent poor quality checkins.
> >>
> >> I have learnt the command-line version but it's no substitute for your
> >> IDE getting the details correct as you type!
> >>
> >> ./mvnw -TC1 -pl :camel-module1,:camel-module2 -Pfastinstall -Psourcecheck
> >>
> >> Are there any newer IJ instructions or should I set this up myself?
> >>
> >> Marc
> >>
> >> (*) http://camel.apache.org/set-up-your-ide.html
> >>
> >>
> >>
> >>
Re: IntelliJ IDEA setup for Apache Camel
Posted by Marc Carter <dr...@gmail.com>.
I took a look at that and it's helped a bit but this does need a
retelling. For the sake of sanity, CheckStyle and Eclipse rules should
be golden - IntelliJ has decent import system for both that can be (re)
documented. However (isn't there always something...)
a) etc/eclipse/CamelCodeFormatter.xml contains an indent of 8 which is
obviously incorrect (clashes with CS rules). Is this the correct file?
<setting id="org.eclipse.jdt.core.formatter.indentation.size" value="8"/>
b) buildingtools/camel-checkstyle.xml differs from
buildingtools/src/main/resources/camel-checkstyle.xml
Which is correct?
On 3/5/19 11:45 AM, Onder SEZGIN wrote:
> http://camel.apache.org/set-up-your-ide.html
>
> see this file mentioned below is on older branches on github.
>
> *etc/idea/settings.jar*
>
> On Tue, Mar 5, 2019 at 1:42 PM Marc Carter <dr...@gmail.com> wrote:
>
>> Official instructions for IntelliJ(*) point to files that no longer
>> exist. I'm sure there's some history to this but it does mean I (and
>> anyone else not using Eclipse) is missing the checkstyle and format
>> rules that prevent poor quality checkins.
>>
>> I have learnt the command-line version but it's no substitute for your
>> IDE getting the details correct as you type!
>>
>> ./mvnw -TC1 -pl :camel-module1,:camel-module2 -Pfastinstall -Psourcecheck
>>
>> Are there any newer IJ instructions or should I set this up myself?
>>
>> Marc
>>
>> (*) http://camel.apache.org/set-up-your-ide.html
>>
>>
>>
>>
Re: IntelliJ IDEA setup for Apache Camel
Posted by Onder SEZGIN <on...@apache.org>.
http://camel.apache.org/set-up-your-ide.html
see this file mentioned below is on older branches on github.
*etc/idea/settings.jar*
On Tue, Mar 5, 2019 at 1:42 PM Marc Carter <dr...@gmail.com> wrote:
> Official instructions for IntelliJ(*) point to files that no longer
> exist. I'm sure there's some history to this but it does mean I (and
> anyone else not using Eclipse) is missing the checkstyle and format
> rules that prevent poor quality checkins.
>
> I have learnt the command-line version but it's no substitute for your
> IDE getting the details correct as you type!
>
> ./mvnw -TC1 -pl :camel-module1,:camel-module2 -Pfastinstall -Psourcecheck
>
> Are there any newer IJ instructions or should I set this up myself?
>
> Marc
>
> (*) http://camel.apache.org/set-up-your-ide.html
>
>
>
>
IntelliJ IDEA setup for Apache Camel
Posted by Marc Carter <dr...@gmail.com>.
Official instructions for IntelliJ(*) point to files that no longer
exist. I'm sure there's some history to this but it does mean I (and
anyone else not using Eclipse) is missing the checkstyle and format
rules that prevent poor quality checkins.
I have learnt the command-line version but it's no substitute for your
IDE getting the details correct as you type!
./mvnw -TC1 -pl :camel-module1,:camel-module2 -Pfastinstall -Psourcecheck
Are there any newer IJ instructions or should I set this up myself?
Marc
(*) http://camel.apache.org/set-up-your-ide.html