You are viewing a plain text version of this content. The canonical link for it is here.
Posted to server-dev@james.apache.org by ma...@gmail.com on 2020/06/08 08:48:38 UTC
Implementing more things in scala
https://github.com/linagora/james-project/pull/3309,
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
Re: Implementing more things in scala
Posted by David Leangen <ap...@leangen.net>.
Hi Matthieu,
> I'm very happy having people to give their opinions, it means, to me,
> we have some kind of community.
Yes, I agree. It must remain respectful and cordial, though. I hope I was able to do that.
>> I totally agree with the point: one should not confuse “complex” with
>> “familiar”. I suppose the only thing I don’t agree with in this
>> sentence is the choice of character ( ` ) for a quote. That goes
>> against everything I have ever learned, and is forbidden by my
>> religion.
>
> You are right, it's just lazyness because I changed from an AZERTY
> keyboard to a QWERTY one and back-quote is simpler to type. I won't do
> that in the future.
Hahahaha.
Just to be sure: that was my attempt at humour to try to lighten the situation because I didn’t know how you would react to my critique. (Don’t worry, my wife never laughs at my jokes, either.)
But your reply was even funnier to me because I completely understand. 😂
> And so on. And not adopting any new thing basically makes a obsolete
> and/or dead project.
You make some good points, but just to be very clear: I am not against things that are “new”. I am *only* against unnecessary complexity.
>> (Actually, if somebody really hates Java that much, if the entire
>> code base were changed to a different language, like Scala, perhaps
>> THAT could be a good thing!)
>
> But no software of this size can be shifted overnight to another
> language.
I agree. I am only trying to reiterate that I have a particular compulsion to try to reduce complexity. I think I have been traumatized too much in the past. 😫
> I can list here some things that we really benefits from adopting
> Scala:
Thanks for the explanation.
Again, I stand by my point emphasizing the word *unnecessary*. If an analysis was done and the addition of Scala in the way it was done was deemed to be necessary, then so be it. I don’t know enough about the project or its history to second guess the decision.
Cheers,
=David
Re: Implementing more things in scala
Posted by Matthieu Baechler <ma...@apache.org>.
Hi David,
I'm very happy having people to give their opinions, it means, to me,
we have some kind of community.
On Tue, 2020-06-09 at 06:22 +0900, David Leangen wrote:
> > > My point is that (1) the “core” should remain in Java alone, not
> > > because Java is so awesome but simply just to avoid unnecessary
> > > complexity,
> >
> > Well, one should not confuse `complex` with `familiar`. Java is
> > `familiar` to many people but is way more complex than Scala in
> > many
> > ways.
>
> I totally agree with the point: one should not confuse “complex” with
> “familiar”. I suppose the only thing I don’t agree with in this
> sentence is the choice of character ( ` ) for a quote. That goes
> against everything I have ever learned, and is forbidden by my
> religion.
You are right, it's just lazyness because I changed from an AZERTY
keyboard to a QWERTY one and back-quote is simpler to type. I won't do
that in the future.
> Seriously though, my assumption was that we were talking about the
> development domain. I was thinking about “complex” as described so
> well by Rich Hickey:
>
> —> https://www.infoq.com/presentations/Simple-Made-Easy/
>
> If I need one language, which comes with a syntax, a set of rules, a
> compiler/interpreter, a set of tools… there is one (big) “thing”. But
> then if I need to add another language which also has its own syntax,
> rules, quirks, tools, compiler/interpreter… now there are two (big)
> “things”. It seems to me that changing from one thing to two things
> is by definition more “complex”.
>
> I think it is possible in my description to change the words “Java”
> and “Scala” to “language A” and “language B” and the idea shouldn’t
> change.
>
> Heck, you could even switch the two around and I think it would still
> express what I intend. Would the answer be the same if most of the
> code was in Scala and somebody was proposing to add more Java?
I pretty much agree with "the more you add the harder it is for
somebody new to the project to be ready to work on the codebase".
But if you follow that principle stricly:
* you don't go from java 5 to java 8 because people are not ready to
write functional-ish java (the knowledge shift is huge)
* you don't use async APIs because it's harder than synchronous APIs
* you don't adopt reactive programming because it's yet another complex
piece of knowledge
And so on. And not adopting any new thing basically makes a obsolete
and/or dead project.
> (Actually, if somebody really hates Java that much, if the entire
> code base were changed to a different language, like Scala, perhaps
> THAT could be a good thing!)
But no software of this size can be shifted overnight to another
language.
> My answer was intended to be agnostic to __which__ language is used,
> but was more about being cautious about adding complexity
> unnecessarily. Or if there is value to adding complexity, to take
> appropriate steps to at least mitigate the complexity.
>
> The idea of “familiar” should be applied both ways: both to anyone
> who resists because it is not familiar enough, and also to anyone who
> is excited because it is so much better than Java.
>
>
> So `yes`, I totally agree with this point!!
>
We did resist for years, actually.
I can list here some things that we really benefits from adopting
Scala:
* POJOs (datastructure) are hundreds of lines today with potential
errors and we write builders everywhere to workaround the lack of named
parameters in method calls. It's really a waste of time. We could have
adopted Lombok so solve that but it's harder to setup than integrating
Scala and Scala `case class` are way better then Java libraries trying
to provide that feature. BTW, I know that Java is getting records in
Java 14 with an experimental flag but it mean waiting for the next LTS
before using it (java 17?) and switch the James runtime. We are talking
about a several years delay
* Algebraic Data Typdes (ADT): Java doesn't offer sealed hierarchies
nor pattern-matching on datastructures. If Java did a lot to spread OOP
in the computer science world, OOP does not fit all problems. If you
look at the PR https://github.com/linagora/james-project/pull/3309 that
triggered that thread, you'll see it's about implementing search.
Search is a tree of conditions with a static grammar: an ADT with
pattern-matching is without a doubt more appropriate than OOP for that.
* When working with Java streams or reactive streams, because of
checked exception in Java, you spend much time cheating the compiler to
use checked exception in lambdas, something Scala doesn't require.
* Again with streams, we spend so much time calling `.stream()` then
`.collect()` and copying immutable structures where Scala offers
persistent datastructure where you can just call `map` on a structure
and be done with it. We also thought about including vavr library for
that in the past and refrained because of too small return on
investment of it.
* Json serialization is a pain in Java as soon as you want separation
of concerns. You can't use annotation on your POJOs so you either
create DTO classes with lots of annotations or you can use mix-ins with
Jackson. In both cases you end with a huge pile of boiler plate. Type-
classes in Scala solve that issue very eleganlty (see what we do we
play-json for instance).
In conclusion: we have a lot of reasons to switch the James codebase to
a more modern language (and keep in mind that java is adopting most of
Scala features with a 10 years latency before answering Java is
modern). We did study some candidates and proposed an ADR with
rationales. The complexity of James project is so high that I doubt
using a second language is really a problem with respect to the
essential complexity of the project and if it can help making the code
more readable and express concepts in a more obvious way, it could
balance having to learn this new thing.
I think that this debate is interesting nonetheless and I thank you
again for your email.
Cheers,
-- Matthieu Baechler
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
Re: Implementing more things in scala
Posted by Antoine Duprat <ad...@apache.org>.
Hi,
I'm not really skilled in Scala.
But I don't think that it's a mistake to go ahead for such a language, I
will not enumerate the advantages of Scala Vs Java.
One point is that James can attract another population of developers.
In order to don't lose none-Scala developers, I think it will be great to
have a document summarizing the Scala patterns used in James.
Antoine
Le mar. 9 juin 2020 à 14:46, Matthieu Baechler <ma...@apache.org> a
écrit :
> Hi Eugen,
>
> On Tue, 2020-06-09 at 00:52 +0300, Eugen Stan wrote:
> > I'm also against adding more complexity in James.
>
> I guess we all are.
>
> > My arguments against adding scala to James are:
> >
> > - It adds another language that is more complex that Java - operator
> > overloading, much more dense language (easier to write, harder to
> > read).
>
> My experience with Scala is the opposite: you can write code that
> express the problem you are solving instead of a lot of boiler plate
> that divert the reader from the intent.
>
> I'm not saying Scala is perfect but I would not say it's more complex
> than Java (no checked exception, support for immutable datastructure,
> easy separation of infrastructure and data, no primitive mess, easy
> data type definition, etc).
>
> > - Extra build and runtime dependencies. Scala is another compiler
> > step
> > and if I remember quite slow (might have changed).
>
> It's definitely slower than Java.
>
> > It also requires adds
> > the library to the applications.
>
> That's also true
>
> > More bytes to push, more surface area
> > to atack. Ideally we should have NO outside dependencies in James
> > core.
> > Hard but doable - Lucene does it.
>
> Depending on what is the 'core', I may agree. However scala stdlib is
> like the jdk: do you see the jdk as a too-big dependency? We have to
> balance number of dependencies with not re-implementing everything
> (with more bugs) from scratch.
>
> >
> >
> > I do believe we should make James easy to consume from other
> > languages
> > but I don't believe we should add more things to James. We should
> > strip
> > them off, starting with dependencies.
>
> Depending on what you define as being James, the goal is opposite as
> what is done currently. James intent is to store things in RDBMS or
> distributed datastores, we can't do that without dependencies.
>
> From what I understand you would at least want a central part of James
> to keep dependencies to the minimum. The ADR we wrote is about writing
> components and extensions in Scala. So I don't think it opposes this
> vision as component are swap-able, having one in Scala should not be a
> problem. What do you think?
>
> >
> > Right now I'm trying to "mvn compile" master and I can't. mvn clean
> > package also fails after running for 30-45 minutes on a Ryzen 7. I
> > feel
> > that does not work.
>
> Ok, there are two problems here:
> 1. we have a lot of tests and they are run by default
> 2. the build process is not as reliable as it should
>
> > The project has 408244 lines of Java and takes longer to compile
> > than
> > the linux kernel [1]. I believe we are doing something wrong here.
>
> I can build James in less than 3 minutes and my computer is slower than
> yours. Most of the time is spent in Maven and not in actual code
> compilation.
>
> Here you compare the time our testsuite takes to run to compile C code
> to a binary: it's not a fair comparison.
>
> > As a developer I should be able to work on James and build it
> > locally,
> > otherwise things are not going to move forward.
>
> Agree.
>
> > Every new tool and external dependency adds complexity.
>
> Not exactly: when using a tool that prevent us from writing a lot of
> code, it actually removes complexity. All technologies are not
> equivalent: some are better in some cases and help keeping code simple,
> etc.
>
> I would give a single example here: do you think splitting strings for
> doing protocol parsing is less complex than using a typesafe PEG parser
> library? In the later case: we can stick to ABNF grammar and everything
> is obvious at the cost of learning the tool. In the former case: it
> looks like raw Java so people feel at home but the grammar is lost and
> it's probably very buggy.
>
> > I believe we should avoid adding more languages to James.
> >
> > I personally prefer Clojure but I will not add Clojure to James core
> > for
> > the same reason. I will build apps on top of James with Clojure for
> > sure.
> >
>
> As somebody who write code for this project, I'm probably more willing
> to use modern tool. It's very usual: users want things to be stable
> because they don't want to work for keeping their service online.
> Developers want things that make their work easier or funnier.
>
> It's one reason I asked to the community and I'm glad you answered.
>
> I didn't changed my mind (yet) but at least I evaluated my opinion
> against others and it's always valuable. Thank you for the time you
> took to write that answer.
>
> Cheers,
>
> -- Matthieu Baechler
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
> For additional commands, e-mail: server-dev-help@james.apache.org
>
>
Re: Implementing more things in scala
Posted by David Leangen <ap...@leangen.net>.
> Sure. I'll give it a try. My version of a library definition is:
I would like to add that a GOOD library that will be LOVED by the community:
* Should have NO transient dependencies
* Should have a clear and minimalistic API
* Should have implementation code in a separate package from the API package
* The API should NEVER depend on the implementation code
* Should follow semantic versioning
Cheers,
=David
Re: Implementing more things in scala
Posted by Eugen Stan <eu...@netdava.com>.
Hi,
Please use Eugen. That is my given name. Stan is my family name.
La 11.06.2020 11:27, Tellier Benoit a scris:
> On 10/06/2020 15:19, Eugen Stan wrote:
>> I'm mostly referring to libraries. Having a library bring a dependency
>> like scala is a no-no on my part.
>>
>> Ideally the lower parts of James should not bring any dependencies.
>> Guava is also big and I would like to see that gone as well.
>>
>> [...]
> Hello Stan,
>
> It is unclear to me what you actually call a library.
>
> Could you provide us with an explanation? Maybe with a list of what you
> consider being libraries within the James project?
>
> I think sharing your thoughts here would help reaching a common vision.
Sure. I'll give it a try. My version of a library definition is:
"A library is a piece of code that is designed to used by other pieces
of code (libraries or applications).
It encapsulates some behaviors that are specific to an area.
It's designed to be reused.
It targets developers."
Examples of libraries from James:
- mime4j
- the mailbox implementations
- the protocol implementations: smtp, lmtp, etc
There can be libraries internal to a project and not very useful outside
of the project, because they are specific.
I think james-core fits this profile and some others.
Libraries can depend on internal libraries - since they form a closed
world (of course the other libraries should have no or few external
dependencies).
For an application:
Something that is meant to be used "as is" by end users (download and run).
It can be customized and tweaked by end user (branding, change
implementations, etc).
Examples of applications in James:
- James Server - all the flavors
- mbox parser
- mpt
Please keep in mind that a single piece of source code can be packaged
in many forms for delivery.
Taking the mailbox import/export functionality as an example we could
have it part of the James server and we could have it packaged as a CLI
application that is distributed separately .
The core functionality of import/export in the above examples is the
same, however each use case comes with it's specifics and might (will)
bring in other dependencies and steps: the CLI will probably require an
argument parsing library. The binaries for the CLI could be built for
arm64, amd64, etc.
When talking about libraries, we don't usually discuss all those points.
Libraries should be small and focused. They should have minimal / no
dependencies (maybe logging).
Libraries can be used to compose one or many applications (via guice
modules or spring beans) as we do for James Server.
IMO the composability part should not be in the library (no spring
annotations like @Component or @Autowired, probably not @Inject either).
Libraries following the above guidelines/rules are usually very stable
and very easy to upgrade.
They compose easily.
Please share your feedback on this.
The list of libraries is not complete.
Regards,
--
Eugen Stan
+40720 898 747 / netdava.com
Re: Implementing more things in scala
Posted by Tellier Benoit <bt...@apache.org>.
On 10/06/2020 15:19, Eugen Stan wrote:
> I'm mostly referring to libraries. Having a library bring a dependency
> like scala is a no-no on my part.
>
> Ideally the lower parts of James should not bring any dependencies.
> Guava is also big and I would like to see that gone as well.
>
> [...]
Hello Stan,
It is unclear to me what you actually call a library.
Could you provide us with an explanation? Maybe with a list of what you
consider being libraries within the James project?
I think sharing your thoughts here would help reaching a common vision.
Thanks!
Benoit
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
Re: Implementing more things in scala
Posted by Matthieu Baechler <ma...@apache.org>.
Hi,
I tried to reply to your message, see below.
On Wed, 2020-06-10 at 11:19 +0300, Eugen Stan wrote:
> Hello Matthieu,
>
> La 09.06.2020 15:46, Matthieu Baechler a scris:
> > My experience with Scala is the opposite: you can write code that
> > express the problem you are solving instead of a lot of boiler
> > plate
> > that divert the reader from the intent.
> >
> > I'm not saying Scala is perfect but I would not say it's more
> > complex
> > than Java (no checked exception, support for immutable
> > datastructure,
> > easy separation of infrastructure and data, no primitive mess, easy
> > data type definition, etc).
> There are immutable collections since Java 9
> https://docs.oracle.com/javase/9/core/creating-immutable-lists-sets-and-maps.htm#JSCOR-GUID-202D195E-6E18-41F6-90C0-7423B2C9B381
The thing about immutable datastructure is not to throw at runtime
but to know by the type that data won't change. Sadly they didn't
choose that solution in the JDK and it makes these Lists almost
useless.
>
>
> Some of the other you mentioned are coming to Java as well. Slower
> for
> sure.
>
> JDK 15 (release in september) comes with text blocks, records (second
> preview), sealed classes preview, pattern matching preview etc.
>
> IMO in 1 year they will be mainline and just hopefully that will be
> the
> next Java LTS ( september 2021).
>
> I do believe (and hope) value types are going to come in the next 2
> years. Let's see if my predictions are accurate :).
>
> https://openjdk.java.net/projects/jdk/15/
>
> The benefits of having that in Java is the bigger ecosystem. It does
> look like a lot of languages will lose some of their appeal.
I agree with all that but I've been sell java 8 lambdas years before we
finally can use it and I see this pattern again: for sure the next LTS
will be great, but we are talking about at least 2 years before
adoption in James if we are optimistic and 10 years later than Scala.
Why should we use a lagging language when a better and mature one is
just here, ready to be used?
> > Depending on what is the 'core', I may agree. However scala stdlib
> > is
> > like the jdk: do you see the jdk as a too-big dependency? We have
> > to
> > balance number of dependencies with not re-implementing everything
> > (with more bugs) from scratch.
> >
> I'm mostly referring to libraries. Having a library bring a
> dependency
> like scala is a no-no on my part.
Agree. However I don't see much libraries into James code base (I mean
valuable parts that can be used in any other project).
> Ideally the lower parts of James should not bring any dependencies.
> Guava is also big and I would like to see that gone as well.
>
> It's not reasonable to bring 1+MB of data for 2-3 classes like
> Preconditions and immutables. Especially since most of that is
> available
> in the newer JDK.
You should look closer to what we use in Guava. JDK doesn't provide
immutable datastructure as good as Guava. Agree that Precondition is
not worth it. However, Range, Cache, FileBackedInputStream and a lot of
others are really valuable and it would be a waste to try to re-
implement similar features to avoid that dependency.
> There are a lot of projects that use James libraries. Asking them to
> bring an extra 5.6 MB for scala is also not reasonable.
I'm not aware of that, please tell me which projects!
> (I checked the size of this:
> https://search.maven.org/artifact/org.scala-lang/scala-library/2.13.2/jar
> . I might be wrong.)
>
> I mean the James libraries have less then a few K of data and bring
> in
> 10 MB of extra dependencies that we can avoid by being mindful about
> our
> choices.
>
> Those classes take disk space, take time to load and process, take
> memory and also maintenance on the security side - something else to
> worry about.
>
> I would also like to see some native images for James with GraalVM.
> Again things add up.
>
> I'm going to stop ranting about that for now because I do want to see
> other languages use James.
>
>
> I do think it's very valuable to consume James libraries from other
> languages and to build applications on top of James.
>
> Using multiple programing paradigmes will improve our API and
> developer
> reach.
>
> I would like to write mailets in Clojure for example and make James
> more
> appealing in the Clojure community.
>
> And of course I won't stop anyone for adding a new library in their
> own
> preferred language to James.
>
> > Depending on what you define as being James, the goal is opposite
> > as
> > what is done currently. James intent is to store things in RDBMS or
> > distributed datastores, we can't do that without dependencies.
> >
> > From what I understand you would at least want a central part of
> > James
> > to keep dependencies to the minimum. The ADR we wrote is about
> > writing
> > components and extensions in Scala. So I don't think it opposes
> > this
> > vision as component are swap-able, having one in Scala should not
> > be a
> > problem. What do you think?
>
> Yes the central part of James that we should keep dependency free are
> the libraries.
>
> The pieces of James that tend to be reused by other people in their
> projects.
>
> If we want a broad reach, we need to make as few assumptions and use
> the
> least amount of dependencies as we can.
For now we assumed James is not used much that way and focused on James
being a good mail server. However, prove us wrong and we'll reconsider
that aspect of the project.
> For extending James - bring as much examples as possible in as many
> languages as possible.
>
> I am all for extending and having implementations in other languages.
>
> We should be mindful that software is easy to write and hard to
> maintain. Maintenance is going to fall on the whole team.
Doing maintenance on James for 5 or 6 years, I know what you mean.
> > I can build James in less than 3 minutes and my computer is slower
> > than
> > yours. Most of the time is spent in Maven and not in actual code
> > compilation.
> >
> > Here you compare the time our testsuite takes to run to compile C
> > code
> > to a binary: it's not a fair comparison.
>
> I'm sold :). Please share with me how. I ran "mvn clean package
> -DskipTests" and it took 10 min.
>
> [INFO] BUILD SUCCESS
> [INFO] Total time: 10:40 min
Just add `-T 1C` to have parallel build that uses all your cores.
> > Not exactly: when using a tool that prevent us from writing a lot
> > of
> > code, it actually removes complexity. All technologies are not
> > equivalent: some are better in some cases and help keeping code
> > simple,
> > etc.
> >
> > I would give a single example here: do you think splitting strings
> > for
> > doing protocol parsing is less complex than using a typesafe PEG
> > parser
> > library? In the later case: we can stick to ABNF grammar and
> > everything
> > is obvious at the cost of learning the tool. In the former case: it
> > looks like raw Java so people feel at home but the grammar is lost
> > and
> > it's probably very buggy.
>
> Have you tried finding an alternative implementation in Java for
> grammars?
Yes, it's what we used actually: parboild 1 for Java.
> I agree, all tools are not born equal. Like I said above I am open to
> adding alternative implementations in new languages.
>
> The users can choose based on the features which one they use.
>
> Please keep in mind that each piece will add overhead:
>
> - more time to build for everyone in the project
Relying on dependencies instead of having our own implementations
actually reduce build time. It's not as simple.
> - extra things to take care of during release, bug triaging,
> maintenance
> when API's are changed.
>
> I'm asking for the sake of discussion:
>
> How do you think we should handle a case when we need to change an
> API.
> An implementation, written in another language needs to be updated
> but
> the original maintainer is not here/not responsive and we don't know
> the
> specific language / tool well enough ?
We are not including several niche languages here: most of the current
active developers know enough Scala to be able to adapt to an API
change for now.
The downside is a higher entry-level for a developer that want to
change an important API. However, most java developer would have some
difficulties with the James code: look at how we use lambda for example
or how hard it is to implement reactive code.
What we try very hard to do is: make the easy things easy. I mean,
writing a mailet should be easy, writing a Mailbox Listener should be
easy.
Even writing an implementation of the Mailbox API is probably not very
hard given we provide a very easy way to test the implemetation for
free.
However a modern mail-server is not something anyone will be able to
contribute from day one and we should use the best tools when dealing
with these parts.
>
> > As somebody who write code for this project, I'm probably more
> > willing
> > to use modern tool. It's very usual: users want things to be stable
> > because they don't want to work for keeping their service online.
> > Developers want things that make their work easier or funnier.
> >
> > It's one reason I asked to the community and I'm glad you
> > answered.
> >
> > I didn't changed my mind (yet) but at least I evaluated my opinion
> > against others and it's always valuable. Thank you for the time you
> > took to write that answer.
>
> I'm also glad you opened the discussion. I'm also glad you did not
> change your mind. It's not my intention. We need to find solutions on
> how to do this in a way that has a small impact on the others.
>
> The only way we can do that is by having a discussion.
>
> On my part I can say this (as a conclusion):
>
> - I will be very resistant to adding dependencies to James libraries.
> It's my default. Also to James Apps, but less so.
> - I will do my best to remove dependencies from those libraries (and
> I
> hope you will support me)
Not sure yet what they are but I suspect I agree.
>
> - I support consuming James libraries from other languages - with
> examples
>
> - I support extending James API's in other languages and/or tools -
> with
> the caveat that we will have to maintain or drop that if no one steps
> in
> to maintain it
>
> - I support adding NEW implementations for protocols, API's /
> libraries
> in other languages - with the caveat of maintenance. This should be
> justified since it will dilute our development efforts.
>
Thank you for your long and detailed response.
--
Matthieu Baechler <ma...@apache.org>
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
Re: Implementing more things in scala
Posted by Eugen Stan <eu...@netdava.com>.
Hello Matthieu,
La 09.06.2020 15:46, Matthieu Baechler a scris:
> My experience with Scala is the opposite: you can write code that
> express the problem you are solving instead of a lot of boiler plate
> that divert the reader from the intent.
>
> I'm not saying Scala is perfect but I would not say it's more complex
> than Java (no checked exception, support for immutable datastructure,
> easy separation of infrastructure and data, no primitive mess, easy
> data type definition, etc).
There are immutable collections since Java 9
https://docs.oracle.com/javase/9/core/creating-immutable-lists-sets-and-maps.htm#JSCOR-GUID-202D195E-6E18-41F6-90C0-7423B2C9B381
.
Some of the other you mentioned are coming to Java as well. Slower for
sure.
JDK 15 (release in september) comes with text blocks, records (second
preview), sealed classes preview, pattern matching preview etc.
IMO in 1 year they will be mainline and just hopefully that will be the
next Java LTS ( september 2021).
I do believe (and hope) value types are going to come in the next 2
years. Let's see if my predictions are accurate :).
https://openjdk.java.net/projects/jdk/15/
The benefits of having that in Java is the bigger ecosystem. It does
look like a lot of languages will lose some of their appeal.
> Depending on what is the 'core', I may agree. However scala stdlib is
> like the jdk: do you see the jdk as a too-big dependency? We have to
> balance number of dependencies with not re-implementing everything
> (with more bugs) from scratch.
>
I'm mostly referring to libraries. Having a library bring a dependency
like scala is a no-no on my part.
Ideally the lower parts of James should not bring any dependencies.
Guava is also big and I would like to see that gone as well.
It's not reasonable to bring 1+MB of data for 2-3 classes like
Preconditions and immutables. Especially since most of that is available
in the newer JDK.
There are a lot of projects that use James libraries. Asking them to
bring an extra 5.6 MB for scala is also not reasonable.
(I checked the size of this:
https://search.maven.org/artifact/org.scala-lang/scala-library/2.13.2/jar
. I might be wrong.)
I mean the James libraries have less then a few K of data and bring in
10 MB of extra dependencies that we can avoid by being mindful about our
choices.
Those classes take disk space, take time to load and process, take
memory and also maintenance on the security side - something else to
worry about.
I would also like to see some native images for James with GraalVM.
Again things add up.
I'm going to stop ranting about that for now because I do want to see
other languages use James.
I do think it's very valuable to consume James libraries from other
languages and to build applications on top of James.
Using multiple programing paradigmes will improve our API and developer
reach.
I would like to write mailets in Clojure for example and make James more
appealing in the Clojure community.
And of course I won't stop anyone for adding a new library in their own
preferred language to James.
> Depending on what you define as being James, the goal is opposite as
> what is done currently. James intent is to store things in RDBMS or
> distributed datastores, we can't do that without dependencies.
>
> From what I understand you would at least want a central part of James
> to keep dependencies to the minimum. The ADR we wrote is about writing
> components and extensions in Scala. So I don't think it opposes this
> vision as component are swap-able, having one in Scala should not be a
> problem. What do you think?
Yes the central part of James that we should keep dependency free are
the libraries.
The pieces of James that tend to be reused by other people in their
projects.
If we want a broad reach, we need to make as few assumptions and use the
least amount of dependencies as we can.
For extending James - bring as much examples as possible in as many
languages as possible.
I am all for extending and having implementations in other languages.
We should be mindful that software is easy to write and hard to
maintain. Maintenance is going to fall on the whole team.
> I can build James in less than 3 minutes and my computer is slower than
> yours. Most of the time is spent in Maven and not in actual code
> compilation.
>
> Here you compare the time our testsuite takes to run to compile C code
> to a binary: it's not a fair comparison.
I'm sold :). Please share with me how. I ran "mvn clean package
-DskipTests" and it took 10 min.
[INFO] BUILD SUCCESS
[INFO] Total time: 10:40 min
> Not exactly: when using a tool that prevent us from writing a lot of
> code, it actually removes complexity. All technologies are not
> equivalent: some are better in some cases and help keeping code simple,
> etc.
>
> I would give a single example here: do you think splitting strings for
> doing protocol parsing is less complex than using a typesafe PEG parser
> library? In the later case: we can stick to ABNF grammar and everything
> is obvious at the cost of learning the tool. In the former case: it
> looks like raw Java so people feel at home but the grammar is lost and
> it's probably very buggy.
Have you tried finding an alternative implementation in Java for grammars?
I agree, all tools are not born equal. Like I said above I am open to
adding alternative implementations in new languages.
The users can choose based on the features which one they use.
Please keep in mind that each piece will add overhead:
- more time to build for everyone in the project
- extra things to take care of during release, bug triaging, maintenance
when API's are changed.
I'm asking for the sake of discussion:
How do you think we should handle a case when we need to change an API.
An implementation, written in another language needs to be updated but
the original maintainer is not here/not responsive and we don't know the
specific language / tool well enough ?
> As somebody who write code for this project, I'm probably more willing
> to use modern tool. It's very usual: users want things to be stable
> because they don't want to work for keeping their service online.
> Developers want things that make their work easier or funnier.
>
> It's one reason I asked to the community and I'm glad you answered.
>
> I didn't changed my mind (yet) but at least I evaluated my opinion
> against others and it's always valuable. Thank you for the time you
> took to write that answer.
I'm also glad you opened the discussion. I'm also glad you did not
change your mind. It's not my intention. We need to find solutions on
how to do this in a way that has a small impact on the others.
The only way we can do that is by having a discussion.
On my part I can say this (as a conclusion):
- I will be very resistant to adding dependencies to James libraries.
It's my default. Also to James Apps, but less so.
- I will do my best to remove dependencies from those libraries (and I
hope you will support me)
- I support consuming James libraries from other languages - with examples
- I support extending James API's in other languages and/or tools - with
the caveat that we will have to maintain or drop that if no one steps in
to maintain it
- I support adding NEW implementations for protocols, API's / libraries
in other languages - with the caveat of maintenance. This should be
justified since it will dilute our development efforts.
Regards,
--
Eugen Stan
+40720 898 747 / netdava.com
Re: Implementing more things in scala
Posted by Matthieu Baechler <ma...@apache.org>.
Hi Eugen,
On Tue, 2020-06-09 at 00:52 +0300, Eugen Stan wrote:
> I'm also against adding more complexity in James.
I guess we all are.
> My arguments against adding scala to James are:
>
> - It adds another language that is more complex that Java - operator
> overloading, much more dense language (easier to write, harder to
> read).
My experience with Scala is the opposite: you can write code that
express the problem you are solving instead of a lot of boiler plate
that divert the reader from the intent.
I'm not saying Scala is perfect but I would not say it's more complex
than Java (no checked exception, support for immutable datastructure,
easy separation of infrastructure and data, no primitive mess, easy
data type definition, etc).
> - Extra build and runtime dependencies. Scala is another compiler
> step
> and if I remember quite slow (might have changed).
It's definitely slower than Java.
> It also requires adds
> the library to the applications.
That's also true
> More bytes to push, more surface area
> to atack. Ideally we should have NO outside dependencies in James
> core.
> Hard but doable - Lucene does it.
Depending on what is the 'core', I may agree. However scala stdlib is
like the jdk: do you see the jdk as a too-big dependency? We have to
balance number of dependencies with not re-implementing everything
(with more bugs) from scratch.
>
>
> I do believe we should make James easy to consume from other
> languages
> but I don't believe we should add more things to James. We should
> strip
> them off, starting with dependencies.
Depending on what you define as being James, the goal is opposite as
what is done currently. James intent is to store things in RDBMS or
distributed datastores, we can't do that without dependencies.
From what I understand you would at least want a central part of James
to keep dependencies to the minimum. The ADR we wrote is about writing
components and extensions in Scala. So I don't think it opposes this
vision as component are swap-able, having one in Scala should not be a
problem. What do you think?
>
> Right now I'm trying to "mvn compile" master and I can't. mvn clean
> package also fails after running for 30-45 minutes on a Ryzen 7. I
> feel
> that does not work.
Ok, there are two problems here:
1. we have a lot of tests and they are run by default
2. the build process is not as reliable as it should
> The project has 408244 lines of Java and takes longer to compile
> than
> the linux kernel [1]. I believe we are doing something wrong here.
I can build James in less than 3 minutes and my computer is slower than
yours. Most of the time is spent in Maven and not in actual code
compilation.
Here you compare the time our testsuite takes to run to compile C code
to a binary: it's not a fair comparison.
> As a developer I should be able to work on James and build it
> locally,
> otherwise things are not going to move forward.
Agree.
> Every new tool and external dependency adds complexity.
Not exactly: when using a tool that prevent us from writing a lot of
code, it actually removes complexity. All technologies are not
equivalent: some are better in some cases and help keeping code simple,
etc.
I would give a single example here: do you think splitting strings for
doing protocol parsing is less complex than using a typesafe PEG parser
library? In the later case: we can stick to ABNF grammar and everything
is obvious at the cost of learning the tool. In the former case: it
looks like raw Java so people feel at home but the grammar is lost and
it's probably very buggy.
> I believe we should avoid adding more languages to James.
>
> I personally prefer Clojure but I will not add Clojure to James core
> for
> the same reason. I will build apps on top of James with Clojure for
> sure.
>
As somebody who write code for this project, I'm probably more willing
to use modern tool. It's very usual: users want things to be stable
because they don't want to work for keeping their service online.
Developers want things that make their work easier or funnier.
It's one reason I asked to the community and I'm glad you answered.
I didn't changed my mind (yet) but at least I evaluated my opinion
against others and it's always valuable. Thank you for the time you
took to write that answer.
Cheers,
-- Matthieu Baechler
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
Re: Implementing more things in scala
Posted by Eugen Stan <eu...@netdava.com>.
I'm also against adding more complexity in James.
My arguments against adding scala to James are:
- It adds another language that is more complex that Java - operator
overloading, much more dense language (easier to write, harder to read).
- Extra build and runtime dependencies. Scala is another compiler step
and if I remember quite slow (might have changed). It also requires adds
the library to the applications. More bytes to push, more surface area
to atack. Ideally we should have NO outside dependencies in James core.
Hard but doable - Lucene does it.
I do believe we should make James easy to consume from other languages
but I don't believe we should add more things to James. We should strip
them off, starting with dependencies.
Right now I'm trying to "mvn compile" master and I can't. mvn clean
package also fails after running for 30-45 minutes on a Ryzen 7. I feel
that does not work.
The project has 408244 lines of Java and takes longer to compile than
the linux kernel [1]. I believe we are doing something wrong here.
As a developer I should be able to work on James and build it locally,
otherwise things are not going to move forward.
Every new tool and external dependency adds complexity.
I believe we should avoid adding more languages to James.
I personally prefer Clojure but I will not add Clojure to James core for
the same reason. I will build apps on top of James with Clojure for sure.
-----
[INFO] Total time: 11.296 s
[INFO] Finished at: 2020-06-09T00:30:17+03:00
[INFO]
------------------------------------------------------------------------
[ERROR] Failed to execute goal
net.alchim31.maven:scala-maven-plugin:3.4.4:compile
(scala-compile-first) on project event-sourcing-event-store-api: wrap:
org.apache.maven.shared.dependency.graph.DependencyGraphBuilderException:
Could not resolve following dependencies:
[org.apache.james:event-sourcing-pojo:jar:tests:3.6.0-SNAPSHOT (test)]:
Could not resolve dependencies for project
org.apache.james:event-sourcing-event-store-api:jar:3.6.0-SNAPSHOT:
Failure to find
org.apache.james:event-sourcing-pojo:jar:tests:3.6.0-SNAPSHOT in
https://jcenter.bintray.com was cached in the local repository,
resolution will not be reattempted until the update interval of central
has elapsed or updates are forced -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the
-e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions,
please read the following articles:
[ERROR] [Help 1]
http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the
command
[ERROR] mvn <args> -rf :event-sourcing-event-store-api
-----
[1] https://phoronix.com/scan.php?page=article&item=ryzen-2600x-2700x&num=4
La 09.06.2020 00:22, David Leangen a scris:
>>> My point is that (1) the “core” should remain in Java alone, not
>>> because Java is so awesome but simply just to avoid unnecessary
>>> complexity,
>> Well, one should not confuse `complex` with `familiar`. Java is
>> `familiar` to many people but is way more complex than Scala in many
>> ways.
> I totally agree with the point: one should not confuse “complex” with “familiar”. I suppose the only thing I don’t agree with in this sentence is the choice of character ( ` ) for a quote. That goes against everything I have ever learned, and is forbidden by my religion.
>
> Seriously though, my assumption was that we were talking about the development domain. I was thinking about “complex” as described so well by Rich Hickey:
>
> —> https://www.infoq.com/presentations/Simple-Made-Easy/
>
> If I need one language, which comes with a syntax, a set of rules, a compiler/interpreter, a set of tools… there is one (big) “thing”. But then if I need to add another language which also has its own syntax, rules, quirks, tools, compiler/interpreter… now there are two (big) “things”. It seems to me that changing from one thing to two things is by definition more “complex”.
>
> I think it is possible in my description to change the words “Java” and “Scala” to “language A” and “language B” and the idea shouldn’t change.
>
> Heck, you could even switch the two around and I think it would still express what I intend. Would the answer be the same if most of the code was in Scala and somebody was proposing to add more Java?
>
> (Actually, if somebody really hates Java that much, if the entire code base were changed to a different language, like Scala, perhaps THAT could be a good thing!)
>
> My answer was intended to be agnostic to __which__ language is used, but was more about being cautious about adding complexity unnecessarily. Or if there is value to adding complexity, to take appropriate steps to at least mitigate the complexity.
>
> The idea of “familiar” should be applied both ways: both to anyone who resists because it is not familiar enough, and also to anyone who is excited because it is so much better than Java.
>
>
> So `yes`, I totally agree with this point!!
>
>
> Cheers,
> =David
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
> For additional commands, e-mail: server-dev-help@james.apache.org
>
--
Eugen Stan
+40720 898 747 / netdava.com
Re: Implementing more things in scala
Posted by David Leangen <ap...@leangen.net>.
>> My point is that (1) the “core” should remain in Java alone, not
>> because Java is so awesome but simply just to avoid unnecessary
>> complexity,
>
> Well, one should not confuse `complex` with `familiar`. Java is
> `familiar` to many people but is way more complex than Scala in many
> ways.
I totally agree with the point: one should not confuse “complex” with “familiar”. I suppose the only thing I don’t agree with in this sentence is the choice of character ( ` ) for a quote. That goes against everything I have ever learned, and is forbidden by my religion.
Seriously though, my assumption was that we were talking about the development domain. I was thinking about “complex” as described so well by Rich Hickey:
—> https://www.infoq.com/presentations/Simple-Made-Easy/
If I need one language, which comes with a syntax, a set of rules, a compiler/interpreter, a set of tools… there is one (big) “thing”. But then if I need to add another language which also has its own syntax, rules, quirks, tools, compiler/interpreter… now there are two (big) “things”. It seems to me that changing from one thing to two things is by definition more “complex”.
I think it is possible in my description to change the words “Java” and “Scala” to “language A” and “language B” and the idea shouldn’t change.
Heck, you could even switch the two around and I think it would still express what I intend. Would the answer be the same if most of the code was in Scala and somebody was proposing to add more Java?
(Actually, if somebody really hates Java that much, if the entire code base were changed to a different language, like Scala, perhaps THAT could be a good thing!)
My answer was intended to be agnostic to __which__ language is used, but was more about being cautious about adding complexity unnecessarily. Or if there is value to adding complexity, to take appropriate steps to at least mitigate the complexity.
The idea of “familiar” should be applied both ways: both to anyone who resists because it is not familiar enough, and also to anyone who is excited because it is so much better than Java.
So `yes`, I totally agree with this point!!
Cheers,
=David
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
Re: Implementing more things in scala
Posted by Matthieu Baechler <ma...@apache.org>.
On Mon, 2020-06-08 at 21:28 +0900, David Leangen wrote:
> > > […] it's a good occasion to gather people opinions about that and
> > > move forward with James as a project.
> > >
> > > So, what do you think ?
>
> My 2 yen:
>
> My impression of James is that it is already much too
> overcomplicated.
Agree
> It seems to me that a major refactoring ought to take place at some
> point.
Well, we could argue we are in a several years long refactoring now.
You should look at code from 6 years old, you'll understand what I
mean.
Don't wait for the refactoring-will-make-everything-easier to happen:
it won't anytime soon.
>
> Scala could be introduced, but it should be done so very cautiously.
> I have nothing at all against Scala or any other language. I do have
> a lot of grudges against unnecessary complications, though.
>
> <tangent>
> At this time in the code base, there are java classes that depend on
> some Scala classes. This means that I am **forced** to get Scala
> working just so I can compile James. However, at this time I am
> unable to do it in Eclipse, despite several hours of investigation,
> due to the current weird state of Scala development. Not good for me.
>
> If it were the opposite (Scala depended on the Java code base), then
> I could simply ignore the Scala, get the Java code working in my IDE,
> and at least be able to do *something* with the code base.
> </tangent>
Well, I understand that IDE support is lacking but Scala rather remove
complexity otherwise we would not start using it.
I agree that it's not welcoming to people when IDE is not working in a
few steps but it's something we can expect to get solved in the near
future.
> My point is that (1) the “core” should remain in Java alone, not
> because Java is so awesome but simply just to avoid unnecessary
> complexity,
Well, one should not confuse `complex` with `familiar`. Java is
`familiar` to many people but is way more complex than Scala in many
ways.
> and (2) any other JVM language could happy co-exist, but should not
> create a dependency (i.e. it could happy sit atop the core without
> disrupting anybody).
>
For now we settled for Scala in component and extensions, not in core.
Thank you for your feedback.
Cheers,
-- Matthieu
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
Re: Implementing more things in scala
Posted by David Leangen <ap...@leangen.net>.
>> […] it's a good occasion to gather people opinions about that and move forward with James as a project.
>>
>> So, what do you think ?
My 2 yen:
My impression of James is that it is already much too overcomplicated. It seems to me that a major refactoring ought to take place at some point.
Scala could be introduced, but it should be done so very cautiously. I have nothing at all against Scala or any other language. I do have a lot of grudges against unnecessary complications, though.
<tangent>
At this time in the code base, there are java classes that depend on some Scala classes. This means that I am **forced** to get Scala working just so I can compile James. However, at this time I am unable to do it in Eclipse, despite several hours of investigation, due to the current weird state of Scala development. Not good for me.
If it were the opposite (Scala depended on the Java code base), then I could simply ignore the Scala, get the Java code working in my IDE, and at least be able to do *something* with the code base.
</tangent>
My point is that (1) the “core” should remain in Java alone, not because Java is so awesome but simply just to avoid unnecessary complexity, and (2) any other JVM language could happy co-exist, but should not create a dependency (i.e. it could happy sit atop the core without disrupting anybody).
Cheers,
=David
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
Re: Implementing more things in scala
Posted by Tellier Benoit <bt...@apache.org>.
To be honest, given the exchange on the pull request and the content of
the ADR I came to this formulation
(https://github.com/linagora/james-project/pull/3309#issuecomment-620957935)
'that I am not hostile to it', but would also like a broader feedback
(thanks for asking for it!).
To give my own opinion:
I personally agree for Scala local usage, in implementations, or some
reasonably isolated APIs (eg: Event Sourcing), as long as Scala usage is
not required for James overall understanding. If it helps making the
code more readable and maintainable, reduce the complexity of some
specific pieces of codes, then it likely is benefic.
I think the pull request you refer falls in this category.
I am more concerned when we start using Scala in some core API (like
mailbox-api), as it forces all developer to understand another language
and can makes future contributions harder by increasing the entry
barrier. (As detailed here:
https://github.com/linagora/james-project/pull/3336)
Best regards,
Benoit
On 08/06/2020 15:58, Matthieu Baechler wrote:
> Hi,
>
> Sorry for the previous mail that I sent by error.
>
> We specified that we wanted to be polyglot-friendly in James and also
> that for the current people writing code we would start using Scala in
> some specific context.
>
> It's what is written in adr 24 here :
> https://github.com/apache/james-project/blob/master/src/adr/0024-polyglot-strategy.md
>
> I personnally like Scala very much and refactored IMAP search which is
> a very verbose piece of code today in Java to a more readable Scala
> version: https://github.com/linagora/james-project/pull/3309
>
> Benoit expressed some doubts about whether that important component
> should be written in scala or not here
> https://github.com/linagora/james-project/pull/3309#issuecomment-616367598
>
> I don't know what to think about that: do people will be afraid of
> reading and/or writing Scala? Will it be a way to make people more
> interested because Java feels something from the past?
>
> I don't really care if this code get merged or not but it's a good
> occasion to gather people opinions about that and move forward with
> James as a project.
>
> So, what do you think ?
>
> Cheers,
>
> -- Matthieu Baechler
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
> For additional commands, e-mail: server-dev-help@james.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
Re: Implementing more things in scala
Posted by Matthieu Baechler <ma...@apache.org>.
Hi,
Sorry for the previous mail that I sent by error.
We specified that we wanted to be polyglot-friendly in James and also
that for the current people writing code we would start using Scala in
some specific context.
It's what is written in adr 24 here :
https://github.com/apache/james-project/blob/master/src/adr/0024-polyglot-strategy.md
I personnally like Scala very much and refactored IMAP search which is
a very verbose piece of code today in Java to a more readable Scala
version: https://github.com/linagora/james-project/pull/3309
Benoit expressed some doubts about whether that important component
should be written in scala or not here
https://github.com/linagora/james-project/pull/3309#issuecomment-616367598
I don't know what to think about that: do people will be afraid of
reading and/or writing Scala? Will it be a way to make people more
interested because Java feels something from the past?
I don't really care if this code get merged or not but it's a good
occasion to gather people opinions about that and move forward with
James as a project.
So, what do you think ?
Cheers,
-- Matthieu Baechler
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org