You are viewing a plain text version of this content. The canonical link for it is here.
Posted to server-dev@james.apache.org by Tellier Benoit <bt...@apache.org> on 2020/07/23 01:55:41 UTC

Antora documentation: thoughts about mailetdoc-maven-plugin and Antora includes

Hello,

I am making good progress on Antora documentation migration for the
Distributed Server, only a couple of pages are left.

I am encountering difficulties to document mailet/matcher an operator
can currently use.

Nowaday we rely on the mailet doc plugin to compile a website page from
the javaDoc of such mailets.

This page currently is in html format and will not be reusable for
Antora. Also the maven plugin will be an obstacle for the graddle
migration. Furthermore, not having to compile James server code when we
build the website seems desirable.

As such I propose to retire the maven-doc plugin, and migrate the
javadoc that exist today as Antora Documentation.

Also, now that I almost finished the Distributed server documentation, I
start thinking of my next step: documenting the other servers. I believe
we should keep documentation writing centralize to ease its maintenance
while keeping each server documentation complete for the reader path to
be straightforward. I believe Antora "include" directives here [1] are
the way to go, and I am thinking of putting that in practice as soon as
I will start documenting another server.

[1] https://docs.antora.org/antora/2.0/asciidoc/include-page/
[2] https://docs.antora.org/antora/2.2/asciidoc/include-partial/

Cheers,

Benoit

---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org

Re: Antora documentation: thoughts about mailetdoc-maven-plugin and Antora includes

Posted by David Leangen <ap...@leangen.net>.
> Some of the topics in the last few days:
> - how to run the SMTP server
> - how to process the emails
> - how to configure James

I think if we stick to the concept of Basic Server and Extendable Server, those use cases will naturally be taken care of. I explain more below.


> I prefer documenting manually each mailet/matcher individually so far,
> based on the existing javadoc.

This is also required for the Extendable Server, so I will help out here.



The original plan was:

 * Get the Basic Server working and document it
 * Get the Extendable Server working and document it

My assumption is:

 * The Distributed Server should describe most of the same concepts as the Extendable Server, except
     —> It needs to talk about distributed concepts (Cassandra etc.)

I have been using the project that Eugen set up:

    —> https://github.com/ieugen/james-self-hosting-sandbox/

Also thanks to Eugen’s work with gradle, I am able to compile the code and actually run it in debug mode in IntelliJ.

I am making progress, but it is very slow progress, partly because I am new to IntelliJ, but mostly because I keep running into road blocks. For instance, right now I am trying to figure out how persistence works so I can save state between executions. It is not at all obvious to me yet why the data resets each time. Then I need to figure out why I’m not able to authenticate when I start an SMTP session, but determining that is also not obvious, so I am trying to step through the code to understand what is happening.

This is just one issue. I keep hitting wall after wall after wall. As a newbie and without a clear path, it takes me a lot of time to figure these things out. The advantage, though, is that it gives me the material I need to write the docs, and the suggestions for how the Basic and Extendable Servers ought to work.

If it were possible to get more help getting the Basic Server and Extendable Server working, I could make much faster progress with the docs, and we would have better answers for the users that ask the questions that Eugen was pointing out. :-)


Cheers,
=David



---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org

Re: Antora documentation: thoughts about mailetdoc-maven-plugin and Antora includes

Posted by Tellier Benoit <bt...@apache.org>.
Le 23/07/2020 à 15:57, Eugen Stan a écrit :
> La 23.07.2020 11:17, David Leangen a scris:
>
> It's great to see progress on this.
> I've also been busy with client/internal work.
>
> I'll make some time to finish up the documentation publishing over the
> weekend probably.
>
> Regarding the javadocs and mailet docs I believe we can make them as
> part of the automated build but not necessarily as part of Antora build.
I think the intent is not exactly the right one here.

I don't want operators to have to go through javadoc to have information
about their mailet / matcher (which was the original intent of my
proposition)
>
> Gradle generates sources and javadocs as part of java builds and publish.
> We can reference the javadoc jar in gradle.
> We can unpack it and push it to some folders as static files.
> Gradle is very powerful and in Jenkinsfile we can call maven as well
> to generate the mailetdocs.
> It's not pretty to use many tools but it should do the job until we
> find better options.
>
> I would stay away from migrating the mailet-docs right now.
> Too much work for too little value IMO.
Me too yes.

I prefer documenting manually each mailet/matcher individually so far,
based on the existing javadoc.
>
> Let's avoid this as much as we can and focus on getting the website up
> so we can feel good about it :).
+1
>
>
> Do you think it's possible to push for a website ready for review by
> 1st august?
I think the "Distributed Server" of the website would be mostly ready.
It could serve as an inspiration while writing other servers
documentation, and reviewing what had been done there first will be
valuable IMO.

I won't have time to work on other parts.

Also, I will be away from keyboard from 1st to 9th August ( vacations :-) )

Regards,

Benoit

---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org

Re: Antora documentation: thoughts about mailetdoc-maven-plugin and Antora includes

Posted by David Leangen <ap...@leangen.net>.
> It's great to see progress on this.

Yes indeed!

> Do you think it's possible to push for a website ready for review by 1st august?

For me at least, this is too ambitious. Perhaps a few more (say, weekly?) milestones and a final review by the end of August?

At least from my perspective, there is still lots and lots of work to do. Personally, I think we should have an initially working site, rather than something half-baked.

But then, maybe we haven’t yet completely clarified the objective(s) of the site?? It could very well be that you have different objectives from me.


Cheers,
=David


---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org

Re: Antora documentation: thoughts about mailetdoc-maven-plugin and Antora includes

Posted by Eugen Stan <eu...@netdava.com>.
La 23.07.2020 11:17, David Leangen a scris:

It's great to see progress on this.
I've also been busy with client/internal work.

I'll make some time to finish up the documentation publishing over the 
weekend probably.

Regarding the javadocs and mailet docs I believe we can make them as 
part of the automated build but not necessarily as part of Antora build.

Gradle generates sources and javadocs as part of java builds and publish.
We can reference the javadoc jar in gradle.
We can unpack it and push it to some folders as static files.
Gradle is very powerful and in Jenkinsfile we can call maven as well to 
generate the mailetdocs.
It's not pretty to use many tools but it should do the job until we find 
better options.

I would stay away from migrating the mailet-docs right now.
Too much work for too little value IMO.

Let's avoid this as much as we can and focus on getting the website up 
so we can feel good about it :).


Do you think it's possible to push for a website ready for review by 1st 
august?


Regards,
-- 
Eugen Stan
+40720 898 747 / netdava.com

Re: Antora documentation: thoughts about mailetdoc-maven-plugin and Antora includes

Posted by Eugen Stan <eu...@netdava.com>.
La 23.07.2020 11:56, David Leangen a scris:
>> For Mailet/Matchers, I would port existing documentation from javadoc to antora (and use includes to not duplicate the sources).
> 
> Do you mean transform from Javadoc format to asciidoc format (either manually or with a conversation tool)?
> 
> Or do you mean build a tool to extract the Javadoc and transform it to asciidoc?
> 
> 
>> JavaDoc would be more for targeting "developers" and is a nice to have too. Especially for API projects (mailet-api, mailbox-api, mailqueue-api, etc...) where Javadoc is both maintained and valuable.
> 
> If somebody can integrate the javadoc into the build (if it isn’t already), then I can look into integrating the javadocs into Antora.

Gradle and maven both generate javadoc jars = jar files that contain 
HTML pages.

We don't need to convert anything to AsciiDoc - we need to publish the 
HTML files so people have the documentation online.

Gradle can download, pack / unpack all of the javadoc jars and push them 
as html pages to be available online.

We can add a link in AsciiDoc pages to the place where we will push the 
javadoc html files.

The only issue I see is that we have a gazillion modules in James, each 
production jars and javadocs.

IMO that is an issue that we will need to address as part of the 
https://issues.apache.org/jira/browse/JAMES-3259 .


-- 
Eugen Stan
+40720 898 747 / netdava.com

Re: Antora documentation: thoughts about mailetdoc-maven-plugin and Antora includes

Posted by David Leangen <ap...@leangen.net>.
> For Mailet/Matchers, I would port existing documentation from javadoc to antora (and use includes to not duplicate the sources).

Do you mean transform from Javadoc format to asciidoc format (either manually or with a conversation tool)?

Or do you mean build a tool to extract the Javadoc and transform it to asciidoc?


> JavaDoc would be more for targeting "developers" and is a nice to have too. Especially for API projects (mailet-api, mailbox-api, mailqueue-api, etc...) where Javadoc is both maintained and valuable.

If somebody can integrate the javadoc into the build (if it isn’t already), then I can look into integrating the javadocs into Antora.


Cheers,
=David


---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org

Re: Antora documentation: thoughts about mailetdoc-maven-plugin and Antora includes

Posted by Tellier Benoit <bt...@apache.org>.
Le 23/07/2020 à 15:17, David Leangen a écrit :
>
>>> [...] I propose to retire the maven-doc plugin, and migrate the
>>> javadoc that exist today as Antora Documentation.
>>
>> I am happy to spend time looking into integrating with Javadoc. I
>> recall reading in the Antora manual that since it reads from the git
>> repository, we can insert actual code snippets and javadoc into the
>> documentation. I remember this because I thought it was a very cool
>> feature. However, I have not yet looked into it in any detail so I
>> can’t say any more at this time. If I can, I will try to take a look
>> today.
>
> It seems that I am mistaken about directly referencing the source
> code. Apparently, I had misread the Antora docs in the past,
> unfortunately. If someone develops an extension, it could be possible,
> but I did not find any existing Antora extension that would allow us
> to directly read the source code. That is unfortunate because I think
> that would have been a really cool feature. I doubt there is enough
> value for us to have somebody make that effort, so we’ll have to live
> without it.
Yes, I had a look at this too this lunch, and came to the same conclusion.

For Mailet/Matchers, I would port existing documentation from javadoc to
antora (and use includes to not duplicate the sources).
>
> There is however an Antora extension available that allows us to
> integrate Javadocs.
+1 to integrate javadoc however it is not the goal here.

The idea of the mailet plugin was to derive "operator" documentation
from the javadoc and integrate/format it into the website.

JavaDoc would be more for targeting "developers" and is a nice to have
too. Especially for API projects (mailet-api, mailbox-api,
mailqueue-api, etc...) where Javadoc is both maintained and valuable.
>
> The way it works is:
>
>  * Javadocs first need to be made available somehow
>     - either as a dependent build task (i.e. that gets executed before
> the Antora task)
>     - or we could reference something online like this:
>      
>  —>  https://search.maven.org/remotecontent?filepath=org/apache/felix/org.apache.felix.scr/2.1.16/org.apache.felix.scr-2.1.16-javadoc.jar
>          (I think it would be better to have a build task and
> reference the local javadocs…)
>   * The Antora extension transforms the html to make it “useable” in
> the Antora site
>      - Antora needs to be run as an npm script and must require a few
> extra dependencies
>   * There is some work that will have to be done to the bundle-ui.zip
> package
>       (“Our” ui does not work, but it does work if I use the ui from
> the example)
>
>
> Here is a snapshot of the prototype. I just used the example given,
> which uses the Apache Felix Javadoc:
>
This sounds promissing to me <3

We may consider adding these javaDocs in the developer guide?

Looking at the sources here are the api maven module which would benefit
from javadoc in my opinion:

 - apache-james-mailbox-api
 - apache-mailet-api
 - protocols-* projects
 - james-server-filesystem-api
 - james-server-data-api
 - james-server-dnsservice-api
 - james-server-mailetcontainer-api
 - james-server-mailrepository-api
 - james-server-queue-api

This would allow both documenting the Java interface an extension
developer can use in its extensions as well as provide more information
concerning the extensions themselves

Cheers,

Benoit

Re: Antora documentation: thoughts about mailetdoc-maven-plugin and Antora includes

Posted by David Leangen <ap...@leangen.net>.
>> [...] I propose to retire the maven-doc plugin, and migrate the
>> javadoc that exist today as Antora Documentation.
> 
> I am happy to spend time looking into integrating with Javadoc. I recall reading in the Antora manual that since it reads from the git repository, we can insert actual code snippets and javadoc into the documentation. I remember this because I thought it was a very cool feature. However, I have not yet looked into it in any detail so I can’t say any more at this time. If I can, I will try to take a look today.

It seems that I am mistaken about directly referencing the source code. Apparently, I had misread the Antora docs in the past, unfortunately. If someone develops an extension, it could be possible, but I did not find any existing Antora extension that would allow us to directly read the source code. That is unfortunate because I think that would have been a really cool feature. I doubt there is enough value for us to have somebody make that effort, so we’ll have to live without it.

There is however an Antora extension available that allows us to integrate Javadocs.

The way it works is:

 * Javadocs first need to be made available somehow
    - either as a dependent build task (i.e. that gets executed before the Antora task)
    - or we could reference something online like this:
       —>  https://search.maven.org/remotecontent?filepath=org/apache/felix/org.apache.felix.scr/2.1.16/org.apache.felix.scr-2.1.16-javadoc.jar
         (I think it would be better to have a build task and reference the local javadocs…)
  * The Antora extension transforms the html to make it “useable” in the Antora site
     - Antora needs to be run as an npm script and must require a few extra dependencies
  * There is some work that will have to be done to the bundle-ui.zip package
      (“Our” ui does not work, but it does work if I use the ui from the example)


Here is a snapshot of the prototype. I just used the example given, which uses the Apache Felix Javadoc:




Cheers,
=David

Re: Antora documentation: thoughts about mailetdoc-maven-plugin and Antora includes

Posted by David Leangen <ap...@leangen.net>.
>> I will need a bit more time.
> Take it :-)

I will reply directly in the code by means of a PR (either by making changes to the code that we can revise before merging or by adding comments).

Cheers,
=David


---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org

Re: Antora documentation: thoughts about mailetdoc-maven-plugin and Antora includes

Posted by Eugen Stan <eu...@netdava.com>.
La 23.07.2020 11:55, Tellier Benoit a scris:

>> I notice that effectively you mention:
>>
>>   * Running
>>   * Configuring
>>   * Operating
>>   * Extending
>>
>>  From our User Model, we have:
>>
>> 	• User —> Probably not relevant
>> 	• Operator —> Relevant
>> 	• Integrator —> Relevant
>> 	• Developer —> ????
>> 	• Contributor —> ????
>> 	• Committer —> ????
>> 	• PMC Member —> Probably not relevant
>>
>> What were your thoughts regarding the target audience of these docs? Was it your intention that:
> 
> I'm targeting operators and integrators. This doc is not relevant for
> others.
> 

We get a lot of feedback on the gitter channel from users trying to use 
James.

I believe we should document those use cases.
When people ask, we can just hand them a link with the documentation.
If we can do that for most of the cases, we are in a good place.

Some of the topics in the last few days:
- how to run the SMTP server
- how to process the emails
- how to configure James

-- 
Eugen Stan
+40720 898 747 / netdava.com

Re: Antora documentation: thoughts about mailetdoc-maven-plugin and Antora includes

Posted by Tellier Benoit <bt...@apache.org>.
Le 23/07/2020 à 15:44, David Leangen a écrit :
>>>> Cool! I will try to take some time today to comment more thoroughly, likely in the form of a PR.
>>> There is quite a lot of text there and a lot of concepts, so it will take me a while to go through it. :-)
>> Maybe more importantly than going through the entire text, the important
>> question for me would be:
>>
>>  - Which concepts do you miss to understand this doc?
> I will need a bit more time.
Take it :-)

It is not a problem if feedback comes a bit late.
>
>>  - Can we address that with a better structure?
> Ok, if I take a very quick top-down view, here are my immediate comments, though I don’t know how useful they are. 
>
> I notice that effectively you mention:
>
>  * Running
>  * Configuring
>  * Operating
>  * Extending
>
> From our User Model, we have:
>
> 	• User —> Probably not relevant
> 	• Operator —> Relevant
> 	• Integrator —> Relevant
> 	• Developer —> ????
> 	• Contributor —> ????
> 	• Committer —> ????
> 	• PMC Member —> Probably not relevant
>
> What were your thoughts regarding the target audience of these docs? Was it your intention that:

I'm targeting operators and integrators. This doc is not relevant for
others.

>
>  * Running —> Operator??
Yes. An operator needs to run the software.

As guice servers are not (yet) available for download, an operator have
either so far to compile the software or use docker images.

See https://github.com/apache/james-project/pull/221 for adding a ZIP
distribution of Guice servers that could be advertized on the dowload page.
>  * Configuring —> Operator??
Yes.
>  * Operating —> Operator (this one is clear)
>  * Extending —> Integrator (this one is clear, too)
Here we are more targeting operators that happen to be also extension
developers.

Maybe I need to explain clearly here who the target audience is.
>
> Or did you have some other structure in mind? It’s difficult for me to grasp exactly what you are intending.
>
> I would have thought that “Operating” includes Installing, Configuring, Running. An Operator (as we define in the User Model) would naturally go to the “Operating” section to figure out how to Operate the server.
>
>
> Like I said, those are just my initial gut reactions. I’ll try to dig in more next week.
I believe the "Operating" session is misnamed then. Maybe I mean
"monitoring & administrating". Would it address your concerns?
>> I'm really looking forward into this review ;-)
> Whew, pressure! Hope I don’t disappoint. 😜
>
>
>> Also note that I did not merge yet the most recent works and
>> improvements for the Distributed Server doc... Do you want me to merge
>> it before your review, to make it easier for you?
> Please, yes! Thank you.
Will do ;-)
>
> Cheers,
> =David
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
> For additional commands, e-mail: server-dev-help@james.apache.org
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org

Re: Antora documentation: thoughts about mailetdoc-maven-plugin and Antora includes

Posted by David Leangen <ap...@leangen.net>.
>>> Cool! I will try to take some time today to comment more thoroughly, likely in the form of a PR.

>> There is quite a lot of text there and a lot of concepts, so it will take me a while to go through it. :-)

> Maybe more importantly than going through the entire text, the important
> question for me would be:
> 
>  - Which concepts do you miss to understand this doc?

I will need a bit more time.

>  - Can we address that with a better structure?

Ok, if I take a very quick top-down view, here are my immediate comments, though I don’t know how useful they are. 

I notice that effectively you mention:

 * Running
 * Configuring
 * Operating
 * Extending

From our User Model, we have:

	• User —> Probably not relevant
	• Operator —> Relevant
	• Integrator —> Relevant
	• Developer —> ????
	• Contributor —> ????
	• Committer —> ????
	• PMC Member —> Probably not relevant

What were your thoughts regarding the target audience of these docs? Was it your intention that:

 * Running —> Operator??
 * Configuring —> Operator??
 * Operating —> Operator (this one is clear)
 * Extending —> Integrator (this one is clear, too)

Or did you have some other structure in mind? It’s difficult for me to grasp exactly what you are intending.

I would have thought that “Operating” includes Installing, Configuring, Running. An Operator (as we define in the User Model) would naturally go to the “Operating” section to figure out how to Operate the server.


Like I said, those are just my initial gut reactions. I’ll try to dig in more next week.


> I'm really looking forward into this review ;-)

Whew, pressure! Hope I don’t disappoint. 😜


> Also note that I did not merge yet the most recent works and
> improvements for the Distributed Server doc... Do you want me to merge
> it before your review, to make it easier for you?

Please, yes! Thank you.

Cheers,
=David


---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org

Re: Antora documentation: thoughts about mailetdoc-maven-plugin and Antora includes

Posted by Tellier Benoit <bt...@apache.org>.
Le 23/07/2020 à 14:27, David Leangen a écrit :
>> Cool! I will try to take some time today to comment more thoroughly, likely in the form of a PR.
> There is quite a lot of text there and a lot of concepts, so it will take me a while to go through it. :-)
Maybe more importantly than going through the entire text, the important
question for me would be:

 - Which concepts do you miss to understand this doc?
 - Can we address that with a better structure?

I think fine grained page-per-page review is important, but is less
critical.

I'm really looking forward into this review ;-)

Also note that I did not merge yet the most recent works and
improvements for the Distributed Server doc... Do you want me to merge
it before your review, to make it easier for you?

Cheers,

Benoit

---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org

Re: Antora documentation: thoughts about mailetdoc-maven-plugin and Antora includes

Posted by David Leangen <ap...@leangen.net>.
> Cool! I will try to take some time today to comment more thoroughly, likely in the form of a PR.

There is quite a lot of text there and a lot of concepts, so it will take me a while to go through it. :-)


---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org

Re: Antora documentation: thoughts about mailetdoc-maven-plugin and Antora includes

Posted by David Leangen <ap...@leangen.net>.
Hi Benoit,

Thank you for giving this so much thought.

(By the way, I am very sorry that I have been less available these days, and will continue to be so for the next few weeks, but I will still try to contribute to this as much as possible.)

> I am making good progress on Antora documentation migration for the
> Distributed Server, only a couple of pages are left.

Cool! I will try to take some time today to comment more thoroughly, likely in the form of a PR.


> [...] I propose to retire the maven-doc plugin, and migrate the
> javadoc that exist today as Antora Documentation.

I am happy to spend time looking into integrating with Javadoc. I recall reading in the Antora manual that since it reads from the git repository, we can insert actual code snippets and javadoc into the documentation. I remember this because I thought it was a very cool feature. However, I have not yet looked into it in any detail so I can’t say any more at this time. If I can, I will try to take a look today.



> Also, now that I almost finished the Distributed server documentation, I
> start thinking of my next step: documenting the other servers.

There is already a PR outstanding for the Basic Server, but it is still in draft mode because I am not yet done:

  —> https://github.com/apache/james-project/pull/206


Regarding the Extendable Server, I am currently trying to run it and use it so I can understand it. I have managed to get the system compiling and can finally run in debug mode in IntelliJ, but because I have never used this IDE before, my progress is slow. 😅 I have filed a few issues to help me make progress. Maybe I’ll bump those since it is easy for them to get lost (or if you have a suggestion for a better way for me to communicate my issues, I will follow your suggestion).

Once I can understand the Extendable Server better, I am planning on writing about it.


Of course, the concepts for these servers are based on what is in the “Concepts” section, which still has many, many holes.


> I believe we should keep documentation writing centralize to ease its maintenance
> while keeping each server documentation complete for the reader path to
> be straightforward.

I am not quite sure I understand what you mean by this point.

Regarding Antora includes and such, yes you are right: we should definitely use these when appropriate.

I am also happy that you are thinking about this from the reader’s perspective. We also have to remember that “reader” can be one of: 

	• User
	• Operator
	• Integrator
	• Developer

and the docs should be geared toward whichever role the reader happens to be playing.



Cheers,
=David


---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org