You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@camel.apache.org by David Jencks <da...@gmail.com> on 2021/09/01 06:42:22 UTC

Re: [Website] Possible change in components page generation (preview available)

With  both the CI build for my PR and locally, `yarn build-all` is building the UI after the main build command.  Obviously this doesn’t work.

Can someone point to a situation in which the UI is built before it is used?  Any ideas why yarn is picking this dysfunctional workspace build order?

David Jencks

> On Aug 30, 2021, at 1:23 AM, Claus Ibsen <cl...@gmail.com> wrote:
> 
> On Sun, Aug 29, 2021 at 10:20 AM David Jencks <david.a.jencks@gmail.com <ma...@gmail.com>> wrote:
>> 
>> The current source state is now visible.
>> 
>> - I npm-published my asciidoctor-jsonpath extension
>> - The site content changes are at https://github.com/djencks/camel.git jsonpath-options branch. I’ve put all the generated changes as the last commit, so it should be possible to update the branch by dropping the last commit, rebasing on main, and regenerating the source with the maven build.
>> - The camel-website changes are at https://github.com/djencks/camel-website.git issue-16854-jsonpath-options branch
>> 
>> There’s a PR https://github.com/apache/camel-website/pull/614.
>> 
>> I don’t understand what the patch-sitemap.js does and was having dependency problems so removed it from the command line.  What does it do and why?
>> 
>> The PR’s build fails because of missing ui bundle.  Is this expected?  I like to have the built UI bundle available somewhere: I often check in the built bundle.  Locally I can’t build the UI, it complains somehow about the helpers for component sorting/hiding.
>> 
>> I set up the build to fail on warnings, and there are quite a few warnings.
>> 
>> There are several source problems I don’t know the proper fix for as it requires domain knowledge.  One is these warnings:
>> 
>> [00:47:14.821] WARN (asciidoctor): skipping reference to missing attribute: apisyntax
>>    file: docs/components/modules/ROOT/pages/google-calendar-stream-component.adoc
>>    source: https://github.com/djencks/camel.git (refname: jsonpath-options, start path: docs/components)
>> [00:47:15.979] WARN (asciidoctor): skipping reference to missing attribute: apisyntax
>>    file: docs/components/modules/ROOT/pages/google-mail-stream-component.adoc
>>    source: https://github.com/djencks/camel.git (refname: jsonpath-options, start path: docs/components)
>> [00:47:16.442] WARN (asciidoctor): skipping reference to missing attribute: apisyntax
>>    file: docs/components/modules/ROOT/pages/google-sheets-stream-component.adoc
>>    source: https://github.com/djencks/camel.git (refname: jsonpath-options, start path: docs/components)
>> 
>> 
>> These three components are missing an apisyntax entry in their json files.  The problem shows up in the current site as the literal string ’null’.
>> 
> 
> Ah yeah it looks like the -stream components are not API based, I am
> going to fix this for Camel 3.12.
> 
> 
>> There are some inconsistencies and mysteries in the gulpfile.js. I’ve commented on some.
>> 
>> It would be great to know if anyone else can build the site from the PR branch.
>> 
>> David Jencks
>> 
>>> On Aug 23, 2021, at 6:00 AM, Claus Ibsen <cl...@gmail.com> wrote:
>>> 
>>> Hi
>>> 
>>> Ah yeah those were an idea to include the full page documentation in
>>> case tooling may be able to use that for something useable.
>>> However the tooling uses all the other bits, so we have just marked
>>> those apis as deprecated.
>>> 
>>> So we can remove the files from the camel-catalog.
>>> I have created a ticket to remove them
>>> https://issues.apache.org/jira/browse/CAMEL-16881
>>> 
>>> On Sun, Aug 22, 2021 at 9:48 AM Zoran Regvart <zo...@regvart.com> wrote:
>>>> 
>>>> Hi David,
>>>> 
>>>> On Sat, Aug 21, 2021 at 10:10 PM David Jencks <da...@gmail.com> wrote:
>>>>> 
>>>>> I have a question about the purpose of the .adoc files in the catalog.  The changes proposed here will remove the tables of options from these copies of the component .adoc files.  These copies are already quite skimpy as they don’t successfully include the spring-boot information.
>>>>> 
>>>>> What are these copies of the .adoc files supposed to be useful for?
>>>> 
>>>> I think the catalog contains those so that an alternative UI can show
>>>> them, think tooltips or inline help in an IDE. I think, though I'm not
>>>> 100% sure that VSCode tooling is using that...
>>>> 
>>>>> If there’s a desire to make them more complete, one strategy would be to build the components module of the website using a custom UI that has nothing in it, so we just get plain undecorated html pages, and putting those in the catalog.  It would require some investigation, but it might conceivably be possible to arrange so links out of the components module go to the website rather than just be broken.
>>>> 
>>>> I'd keep it as simple as it can be, so plain .adoc files, or if we
>>>> find out they're not used just remove them...
>>>> 
>>>> 2c
>>>> 
>>>> zoran
>>>> --
>>>> Zoran Regvart
>>> 
>>> 
>>> 
>>> --
>>> Claus Ibsen
>>> -----------------
>>> http://davsclaus.com @davsclaus
>>> Camel in Action 2: https://www.manning.com/ibsen2
>> 
> 
> 
> -- 
> Claus Ibsen
> -----------------
> http://davsclaus.com <http://davsclaus.com/> @davsclaus
> Camel in Action 2: https://www.manning.com/ibsen2 <https://www.manning.com/ibsen2>

Re: [Website] Possible change in components page generation (preview available)

Posted by David Jencks <da...@gmail.com>.
I’ve been thinking of this as monolithic across repositories (camel and camel-website at least) but perhaps a more sensible path is to put the attribute fixes in camel main into a separate PR, and then update camel-website to use Antora 3, and configure it to make @djencks/asciidoctor-jsonpath available.  Then we can update camel to actually use the extension.

David Jencks

> On Sep 2, 2021, at 11:21 PM, David Jencks <da...@gmail.com> wrote:
> 
> I think this work is complete enough to be used.  There’s now a Ci built preview at https://pr-614--camel.netlify.app <https://pr-614--camel.netlify.app/>
> 
> The main work is on https://github.com/apache/camel/pull/6040 <https://github.com/apache/camel/pull/6040>. Possibly the commits should be squashed into fewer.  Once this is merged the camel-website PR can be updated to use the normal repos/branches.
> 
> Subsidiary PRs fixing warnings that would now break the build: (these can be merged before the main work)
> 
> https://github.com/apache/camel-kamelets/pull/480 <https://github.com/apache/camel-kamelets/pull/480> (no error from Antora). This is the bizarre looking Kamelets catalog page fix.  I have no idea why the previous adoc worked, but this works on both Antora 2 and 3
> 
> https://github.com/apache/camel-quarkus/pull/3064 <https://github.com/apache/camel-quarkus/pull/3064>
> https://github.com/apache/camel-quarkus/pull/3065 <https://github.com/apache/camel-quarkus/pull/3065>
> 
> https://github.com/apache/camel/pull/6039 <https://github.com/apache/camel/pull/6039>
> 
> comments:
> 
> - I removed the patch-sitemap.js use.  I can’t figure out why this is needed.  If it really is needed, it should be done with a pipeline extension.
> - yarn workspace  —topological-dev is not building the UI before it’s used.  I can’t figure out why; it is for other PRs.  I replaced this by just calling the UI build directly.
> - If nothing else is using the model building in UpdateReadmeMojo it can probably be simplified further.
> - The camel-website node dependencies have changed dramatically.  It looks like the yarn cache is checked into  git; I haven’t checked in the changes.  Should I?
> 
> known problems:
> 
> - When the first row contains a description with a list, the list isn’t rendered properly.  cf. https://pr-614--camel.netlify.app/components/latest/activemq-component.html#_path_parameters_2_parameters <https://pr-614--camel.netlify.app/components/latest/activemq-component.html#_path_parameters_2_parameters>
> - All the names in the generated tables have ids so they can be linked to but there’s no easy way to find out the id. The sections have a link to themselves which makes it easy to copy the URL.  I haven’t looked into whether this can be done with  the table entries.
> 
> Updating the main PR to track changes in camel is somewhat time consuming so I’d appreciate it if we could move this towards resolution quickly.
> 
> David Jencks
> 
> 
>> On Aug 31, 2021, at 11:42 PM, David Jencks <david.a.jencks@gmail.com <ma...@gmail.com>> wrote:
>> 
>> With  both the CI build for my PR and locally, `yarn build-all` is building the UI after the main build command.  Obviously this doesn’t work.
>> 
>> Can someone point to a situation in which the UI is built before it is used?  Any ideas why yarn is picking this dysfunctional workspace build order?
>> 
>> David Jencks
>> 
>>> On Aug 30, 2021, at 1:23 AM, Claus Ibsen <claus.ibsen@gmail.com <ma...@gmail.com>> wrote:
>>> 
>>> On Sun, Aug 29, 2021 at 10:20 AM David Jencks <david.a.jencks@gmail.com <ma...@gmail.com>> wrote:
>>>> 
>>>> The current source state is now visible.
>>>> 
>>>> - I npm-published my asciidoctor-jsonpath extension
>>>> - The site content changes are at https://github.com/djencks/camel.git <https://github.com/djencks/camel.git> jsonpath-options branch. I’ve put all the generated changes as the last commit, so it should be possible to update the branch by dropping the last commit, rebasing on main, and regenerating the source with the maven build.
>>>> - The camel-website changes are at https://github.com/djencks/camel-website.git <https://github.com/djencks/camel-website.git> issue-16854-jsonpath-options branch
>>>> 
>>>> There’s a PR https://github.com/apache/camel-website/pull/614 <https://github.com/apache/camel-website/pull/614>.
>>>> 
>>>> I don’t understand what the patch-sitemap.js does and was having dependency problems so removed it from the command line.  What does it do and why?
>>>> 
>>>> The PR’s build fails because of missing ui bundle.  Is this expected?  I like to have the built UI bundle available somewhere: I often check in the built bundle.  Locally I can’t build the UI, it complains somehow about the helpers for component sorting/hiding.
>>>> 
>>>> I set up the build to fail on warnings, and there are quite a few warnings.
>>>> 
>>>> There are several source problems I don’t know the proper fix for as it requires domain knowledge.  One is these warnings:
>>>> 
>>>> [00:47:14.821] WARN (asciidoctor): skipping reference to missing attribute: apisyntax
>>>>    file: docs/components/modules/ROOT/pages/google-calendar-stream-component.adoc
>>>>    source: https://github.com/djencks/camel.git <https://github.com/djencks/camel.git> (refname: jsonpath-options, start path: docs/components)
>>>> [00:47:15.979] WARN (asciidoctor): skipping reference to missing attribute: apisyntax
>>>>    file: docs/components/modules/ROOT/pages/google-mail-stream-component.adoc
>>>>    source: https://github.com/djencks/camel.git <https://github.com/djencks/camel.git> (refname: jsonpath-options, start path: docs/components)
>>>> [00:47:16.442] WARN (asciidoctor): skipping reference to missing attribute: apisyntax
>>>>    file: docs/components/modules/ROOT/pages/google-sheets-stream-component.adoc
>>>>    source: https://github.com/djencks/camel.git <https://github.com/djencks/camel.git> (refname: jsonpath-options, start path: docs/components)
>>>> 
>>>> 
>>>> These three components are missing an apisyntax entry in their json files.  The problem shows up in the current site as the literal string ’null’.
>>>> 
>>> 
>>> Ah yeah it looks like the -stream components are not API based, I am
>>> going to fix this for Camel 3.12.
>>> 
>>> 
>>>> There are some inconsistencies and mysteries in the gulpfile.js. I’ve commented on some.
>>>> 
>>>> It would be great to know if anyone else can build the site from the PR branch.
>>>> 
>>>> David Jencks
>>>> 
>>>>> On Aug 23, 2021, at 6:00 AM, Claus Ibsen <claus.ibsen@gmail.com <ma...@gmail.com>> wrote:
>>>>> 
>>>>> Hi
>>>>> 
>>>>> Ah yeah those were an idea to include the full page documentation in
>>>>> case tooling may be able to use that for something useable.
>>>>> However the tooling uses all the other bits, so we have just marked
>>>>> those apis as deprecated.
>>>>> 
>>>>> So we can remove the files from the camel-catalog.
>>>>> I have created a ticket to remove them
>>>>> https://issues.apache.org/jira/browse/CAMEL-16881 <https://issues.apache.org/jira/browse/CAMEL-16881>
>>>>> 
>>>>> On Sun, Aug 22, 2021 at 9:48 AM Zoran Regvart <zoran@regvart.com <ma...@regvart.com>> wrote:
>>>>>> 
>>>>>> Hi David,
>>>>>> 
>>>>>> On Sat, Aug 21, 2021 at 10:10 PM David Jencks <david.a.jencks@gmail.com <ma...@gmail.com>> wrote:
>>>>>>> 
>>>>>>> I have a question about the purpose of the .adoc files in the catalog.  The changes proposed here will remove the tables of options from these copies of the component .adoc files.  These copies are already quite skimpy as they don’t successfully include the spring-boot information.
>>>>>>> 
>>>>>>> What are these copies of the .adoc files supposed to be useful for?
>>>>>> 
>>>>>> I think the catalog contains those so that an alternative UI can show
>>>>>> them, think tooltips or inline help in an IDE. I think, though I'm not
>>>>>> 100% sure that VSCode tooling is using that...
>>>>>> 
>>>>>>> If there’s a desire to make them more complete, one strategy would be to build the components module of the website using a custom UI that has nothing in it, so we just get plain undecorated html pages, and putting those in the catalog.  It would require some investigation, but it might conceivably be possible to arrange so links out of the components module go to the website rather than just be broken.
>>>>>> 
>>>>>> I'd keep it as simple as it can be, so plain .adoc files, or if we
>>>>>> find out they're not used just remove them...
>>>>>> 
>>>>>> 2c
>>>>>> 
>>>>>> zoran
>>>>>> --
>>>>>> Zoran Regvart
>>>>> 
>>>>> 
>>>>> 
>>>>> --
>>>>> Claus Ibsen
>>>>> -----------------
>>>>> http://davsclaus.com <http://davsclaus.com/> @davsclaus
>>>>> Camel in Action 2: https://www.manning.com/ibsen2 <https://www.manning.com/ibsen2>
>>>> 
>>> 
>>> 
>>> -- 
>>> Claus Ibsen
>>> -----------------
>>> http://davsclaus.com <http://davsclaus.com/> @davsclaus
>>> Camel in Action 2: https://www.manning.com/ibsen2 <https://www.manning.com/ibsen2>
>

Re: [Website] Possible change in components page generation (preview available)

Posted by David Jencks <da...@gmail.com>.
I’ve addressed all the problems I know about…. now we can find more :-)

At this point I think we could apply the PRs and work from there.  In more detail:

(1) https://github.com/apache/camel-kamelets/pull/487 fixes the Kamelet catalog rendering and AFAICT doesn’t break the Antora 2 build. I’d merge this but I’m not sure I know all the places to look for problems :-)
(2) [website] Issue 16854 antora 3 <https://github.com/apache/camel-website/pull/617> Preview at https://pr-617--camel.netlify.app <https://pr-617--camel.netlify.app/>. This uses a camel-website setup with the latest Antora 3.0.0-alpha.9 ready to build the next set of changes: the preview is against the current content state.  The only problem I know of is the Kamelet catalog which would be fixed by the preceding PR.
(3) [Website] Use @djencks/asciidoctor-jsonpath to generate component doc content <https://github.com/apache/camel/pull/6040>. Preview at https://pr-614--camel.netlify.app <https://pr-614--camel.netlify.app/> (with the Kamelet catalog fix) (using a customized playbook).

(3) should be rebased on main and the generated changes regenerated just before merging.

Issues (all with (2)):
- I’ve committed the results of running yarn in antora-ui-camel and root, but I’m not at all sure this is correct.  The build seems to work in CI, but I’m on MacOS.  In particular I sometimes seem to end up with a 50Mb+ file for the UI that I don’t see in the current state.
- Using yarn workspaces topology/dependency detection to build the UI first doesn’t work for me, locally or in CI.  I’ve hard-coded building the UI first.  I think this is more reliable so a good idea, but perhaps I did something wrong.
- I removed patch-sitemap.js usage.  I can’t tell what it’s supposed to be doing or why and, even if it’s needed, it should be implemented as a pipeline extension.

AFAIK Zoran put all these in place so perhaps we need his input.

Comments:
- I think that the EIP doc pages haven’t been updated for some time: the UpdateReadmeMojo appeared to be looking in the wrong directory.  Certainly the previous version of my work wasn’t using the jsonpath eip template I wrote.
- The first row of generated tables is now treated the same as following rows: this fixes the enum list display in descriptions and the bold name problem.
- I think I’ve excluded the same rows from all tables as the preceding approach.
- The main options tables for all kinds have the same columns but not always in the same order.  Perhaps we should make the order the same?
components: activemq-component.html <https://pr-614--camel.netlify.app/components/latest/activemq-component.html#_component_options> | Name | Description | Default | Type
dataformats: any23-dataformat.html <https://pr-614--camel.netlify.app/components/latest/dataformats/any23-dataformat.html#_any23_options> | Name | Default | Java Type | Description
languages: bean-language.html <https://pr-614--camel.netlify.app/components/latest/languages/bean-language.html#_bean_language_options> | Name | Default | Java Type | Description
eips: aggregate-eip.html <https://pr-614--camel.netlify.app/components/latest/eips/aggregate-eip.html#_aggregator_options>  | Name | Description | Default | Type

We’re always showing the java type short name, so we should also decide between Type and Java Type.

- UpdateReadmeMojo can be further simplified, and I may get to it shortly, but that can also be done later.
- docs/gulpfile.js can be (IMO) considerably simplified by driving more of the symlink generation from data: I already did this for the new json symlinks. The disadvantage is that the task is then anonymous so you can’t tell what gulp is doing as easily.
- I think I’m creating a lot of unused json symlinks for at least dataformats and eips because I can’t figure out how to find only the relevant ones.  We should probably deal with  this later.
- There are some inconsistencies in naming and directory layout for some components that we should probably deal with later if at all.

David Jencks


> On Sep 3, 2021, at 1:46 AM, Claus Ibsen <cl...@gmail.com> wrote:
> 
> Hi
> 
> Great work David.
> 
> I noticed the data formats table, then the 1st row is in bold, and the
> others are not, and the styling of the javaType is also different
> https://pr-614--camel.netlify.app/components/latest/dataformats/thrift-dataformat.html <https://pr-614--camel.netlify.app/components/latest/dataformats/thrift-dataformat.html>
> 
> And for languages then we should skip the "expression" row as its not
> really an option but the actual language code being used (implied).
> https://pr-614--camel.netlify.app/components/latest/languages/jsonpath-language.html <https://pr-614--camel.netlify.app/components/latest/languages/jsonpath-language.html>
> 
> You can compare these pages with the regular website to see the subtle
> differences.
> 
> 
> For EIP patterns then maybe we should style the Type column to be same
> as for the others (blue color, lower case for boolean, string, int
> etc.)
> https://pr-614--camel.netlify.app/components/latest/eips/claimCheck-eip.html <https://pr-614--camel.netlify.app/components/latest/eips/claimCheck-eip.html>
> 
> 
> On Fri, Sep 3, 2021 at 8:21 AM David Jencks <david.a.jencks@gmail.com <ma...@gmail.com>> wrote:
>> 
>> I think this work is complete enough to be used.  There’s now a Ci built preview at https://pr-614--camel.netlify.app
>> 
>> The main work is on https://github.com/apache/camel/pull/6040. Possibly the commits should be squashed into fewer.  Once this is merged the camel-website PR can be updated to use the normal repos/branches.
>> 
>> Subsidiary PRs fixing warnings that would now break the build: (these can be merged before the main work)
>> 
>> https://github.com/apache/camel-kamelets/pull/480 (no error from Antora). This is the bizarre looking Kamelets catalog page fix.  I have no idea why the previous adoc worked, but this works on both Antora 2 and 3
>> 
>> https://github.com/apache/camel-quarkus/pull/3064
>> https://github.com/apache/camel-quarkus/pull/3065
>> 
>> https://github.com/apache/camel/pull/6039
>> 
>> comments:
>> 
>> - I removed the patch-sitemap.js use.  I can’t figure out why this is needed.  If it really is needed, it should be done with a pipeline extension.
>> - yarn workspace  —topological-dev is not building the UI before it’s used.  I can’t figure out why; it is for other PRs.  I replaced this by just calling the UI build directly.
>> - If nothing else is using the model building in UpdateReadmeMojo it can probably be simplified further.
>> - The camel-website node dependencies have changed dramatically.  It looks like the yarn cache is checked into  git; I haven’t checked in the changes.  Should I?
>> 
>> known problems:
>> 
>> - When the first row contains a description with a list, the list isn’t rendered properly.  cf. https://pr-614--camel.netlify.app/components/latest/activemq-component.html#_path_parameters_2_parameters <https://pr-614--camel.netlify.app/components/latest/activemq-component.html#_path_parameters_2_parameters> <https://pr-614--camel.netlify.app/components/latest/activemq-component.html#_path_parameters_2_parameters <https://pr-614--camel.netlify.app/components/latest/activemq-component.html#_path_parameters_2_parameters>>
>> - All the names in the generated tables have ids so they can be linked to but there’s no easy way to find out the id. The sections have a link to themselves which makes it easy to copy the URL.  I haven’t looked into whether this can be done with  the table entries.
>> 
>> Updating the main PR to track changes in camel is somewhat time consuming so I’d appreciate it if we could move this towards resolution quickly.
>> 
>> David Jencks
>> 
>> 
>>> On Aug 31, 2021, at 11:42 PM, David Jencks <david.a.jencks@gmail.com <ma...@gmail.com>> wrote:
>>> 
>>> With  both the CI build for my PR and locally, `yarn build-all` is building the UI after the main build command.  Obviously this doesn’t work.
>>> 
>>> Can someone point to a situation in which the UI is built before it is used?  Any ideas why yarn is picking this dysfunctional workspace build order?
>>> 
>>> David Jencks
>>> 
>>>> On Aug 30, 2021, at 1:23 AM, Claus Ibsen <claus.ibsen@gmail.com <ma...@gmail.com> <mailto:claus.ibsen@gmail.com <ma...@gmail.com>>> wrote:
>>>> 
>>>> On Sun, Aug 29, 2021 at 10:20 AM David Jencks <david.a.jencks@gmail.com <ma...@gmail.com> <mailto:david.a.jencks@gmail.com <ma...@gmail.com>>> wrote:
>>>>> 
>>>>> The current source state is now visible.
>>>>> 
>>>>> - I npm-published my asciidoctor-jsonpath extension
>>>>> - The site content changes are at https://github.com/djencks/camel.git <https://github.com/djencks/camel.git> <https://github.com/djencks/camel.git <https://github.com/djencks/camel.git>> jsonpath-options branch. I’ve put all the generated changes as the last commit, so it should be possible to update the branch by dropping the last commit, rebasing on main, and regenerating the source with the maven build.
>>>>> - The camel-website changes are at https://github.com/djencks/camel-website.git <https://github.com/djencks/camel-website.git> <https://github.com/djencks/camel-website.git <https://github.com/djencks/camel-website.git>> issue-16854-jsonpath-options branch
>>>>> 
>>>>> There’s a PR https://github.com/apache/camel-website/pull/614 <https://github.com/apache/camel-website/pull/614> <https://github.com/apache/camel-website/pull/614 <https://github.com/apache/camel-website/pull/614>>.
>>>>> 
>>>>> I don’t understand what the patch-sitemap.js does and was having dependency problems so removed it from the command line.  What does it do and why?
>>>>> 
>>>>> The PR’s build fails because of missing ui bundle.  Is this expected?  I like to have the built UI bundle available somewhere: I often check in the built bundle.  Locally I can’t build the UI, it complains somehow about the helpers for component sorting/hiding.
>>>>> 
>>>>> I set up the build to fail on warnings, and there are quite a few warnings.
>>>>> 
>>>>> There are several source problems I don’t know the proper fix for as it requires domain knowledge.  One is these warnings:
>>>>> 
>>>>> [00:47:14.821] WARN (asciidoctor): skipping reference to missing attribute: apisyntax
>>>>>   file: docs/components/modules/ROOT/pages/google-calendar-stream-component.adoc
>>>>>   source: https://github.com/djencks/camel.git <https://github.com/djencks/camel.git> <https://github.com/djencks/camel.git <https://github.com/djencks/camel.git>> (refname: jsonpath-options, start path: docs/components)
>>>>> [00:47:15.979] WARN (asciidoctor): skipping reference to missing attribute: apisyntax
>>>>>   file: docs/components/modules/ROOT/pages/google-mail-stream-component.adoc
>>>>>   source: https://github.com/djencks/camel.git <https://github.com/djencks/camel.git> <https://github.com/djencks/camel.git <https://github.com/djencks/camel.git>> (refname: jsonpath-options, start path: docs/components)
>>>>> [00:47:16.442] WARN (asciidoctor): skipping reference to missing attribute: apisyntax
>>>>>   file: docs/components/modules/ROOT/pages/google-sheets-stream-component.adoc
>>>>>   source: https://github.com/djencks/camel.git <https://github.com/djencks/camel.git> <https://github.com/djencks/camel.git <https://github.com/djencks/camel.git>> (refname: jsonpath-options, start path: docs/components)
>>>>> 
>>>>> 
>>>>> These three components are missing an apisyntax entry in their json files.  The problem shows up in the current site as the literal string ’null’.
>>>>> 
>>>> 
>>>> Ah yeah it looks like the -stream components are not API based, I am
>>>> going to fix this for Camel 3.12.
>>>> 
>>>> 
>>>>> There are some inconsistencies and mysteries in the gulpfile.js. I’ve commented on some.
>>>>> 
>>>>> It would be great to know if anyone else can build the site from the PR branch.
>>>>> 
>>>>> David Jencks
>>>>> 
>>>>>> On Aug 23, 2021, at 6:00 AM, Claus Ibsen <claus.ibsen@gmail.com <ma...@gmail.com> <mailto:claus.ibsen@gmail.com <ma...@gmail.com>>> wrote:
>>>>>> 
>>>>>> Hi
>>>>>> 
>>>>>> Ah yeah those were an idea to include the full page documentation in
>>>>>> case tooling may be able to use that for something useable.
>>>>>> However the tooling uses all the other bits, so we have just marked
>>>>>> those apis as deprecated.
>>>>>> 
>>>>>> So we can remove the files from the camel-catalog.
>>>>>> I have created a ticket to remove them
>>>>>> https://issues.apache.org/jira/browse/CAMEL-16881 <https://issues.apache.org/jira/browse/CAMEL-16881> <https://issues.apache.org/jira/browse/CAMEL-16881 <https://issues.apache.org/jira/browse/CAMEL-16881>>
>>>>>> 
>>>>>> On Sun, Aug 22, 2021 at 9:48 AM Zoran Regvart <zoran@regvart.com <ma...@regvart.com>> wrote:
>>>>>>> 
>>>>>>> Hi David,
>>>>>>> 
>>>>>>> On Sat, Aug 21, 2021 at 10:10 PM David Jencks <david.a.jencks@gmail.com <ma...@gmail.com>> wrote:
>>>>>>>> 
>>>>>>>> I have a question about the purpose of the .adoc files in the catalog.  The changes proposed here will remove the tables of options from these copies of the component .adoc files.  These copies are already quite skimpy as they don’t successfully include the spring-boot information.
>>>>>>>> 
>>>>>>>> What are these copies of the .adoc files supposed to be useful for?
>>>>>>> 
>>>>>>> I think the catalog contains those so that an alternative UI can show
>>>>>>> them, think tooltips or inline help in an IDE. I think, though I'm not
>>>>>>> 100% sure that VSCode tooling is using that...
>>>>>>> 
>>>>>>>> If there’s a desire to make them more complete, one strategy would be to build the components module of the website using a custom UI that has nothing in it, so we just get plain undecorated html pages, and putting those in the catalog.  It would require some investigation, but it might conceivably be possible to arrange so links out of the components module go to the website rather than just be broken.
>>>>>>> 
>>>>>>> I'd keep it as simple as it can be, so plain .adoc files, or if we
>>>>>>> find out they're not used just remove them...
>>>>>>> 
>>>>>>> 2c
>>>>>>> 
>>>>>>> zoran
>>>>>>> --
>>>>>>> Zoran Regvart
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> --
>>>>>> Claus Ibsen
>>>>>> -----------------
>>>>>> http://davsclaus.com <http://davsclaus.com/> @davsclaus
>>>>>> Camel in Action 2: https://www.manning.com/ibsen2 <https://www.manning.com/ibsen2>
>>>>> 
>>>> 
>>>> 
>>>> --
>>>> Claus Ibsen
>>>> -----------------
>>>> http://davsclaus.com <http://davsclaus.com/> <http://davsclaus.com/ <http://davsclaus.com/>> @davsclaus
>>>> Camel in Action 2: https://www.manning.com/ibsen2 <https://www.manning.com/ibsen2> <https://www.manning.com/ibsen2 <https://www.manning.com/ibsen2>>
>> 
> 
> 
> -- 
> Claus Ibsen
> -----------------
> http://davsclaus.com <http://davsclaus.com/> @davsclaus
> Camel in Action 2: https://www.manning.com/ibsen2 <https://www.manning.com/ibsen2>

Re: [Website] Possible change in components page generation (preview available)

Posted by Claus Ibsen <cl...@gmail.com>.
Hi

Great work David.

I noticed the data formats table, then the 1st row is in bold, and the
others are not, and the styling of the javaType is also different
https://pr-614--camel.netlify.app/components/latest/dataformats/thrift-dataformat.html

And for languages then we should skip the "expression" row as its not
really an option but the actual language code being used (implied).
https://pr-614--camel.netlify.app/components/latest/languages/jsonpath-language.html

You can compare these pages with the regular website to see the subtle
differences.


For EIP patterns then maybe we should style the Type column to be same
as for the others (blue color, lower case for boolean, string, int
etc.)
https://pr-614--camel.netlify.app/components/latest/eips/claimCheck-eip.html


On Fri, Sep 3, 2021 at 8:21 AM David Jencks <da...@gmail.com> wrote:
>
> I think this work is complete enough to be used.  There’s now a Ci built preview at https://pr-614--camel.netlify.app
>
> The main work is on https://github.com/apache/camel/pull/6040. Possibly the commits should be squashed into fewer.  Once this is merged the camel-website PR can be updated to use the normal repos/branches.
>
> Subsidiary PRs fixing warnings that would now break the build: (these can be merged before the main work)
>
> https://github.com/apache/camel-kamelets/pull/480 (no error from Antora). This is the bizarre looking Kamelets catalog page fix.  I have no idea why the previous adoc worked, but this works on both Antora 2 and 3
>
> https://github.com/apache/camel-quarkus/pull/3064
> https://github.com/apache/camel-quarkus/pull/3065
>
> https://github.com/apache/camel/pull/6039
>
> comments:
>
> - I removed the patch-sitemap.js use.  I can’t figure out why this is needed.  If it really is needed, it should be done with a pipeline extension.
> - yarn workspace  —topological-dev is not building the UI before it’s used.  I can’t figure out why; it is for other PRs.  I replaced this by just calling the UI build directly.
> - If nothing else is using the model building in UpdateReadmeMojo it can probably be simplified further.
> - The camel-website node dependencies have changed dramatically.  It looks like the yarn cache is checked into  git; I haven’t checked in the changes.  Should I?
>
> known problems:
>
> - When the first row contains a description with a list, the list isn’t rendered properly.  cf. https://pr-614--camel.netlify.app/components/latest/activemq-component.html#_path_parameters_2_parameters <https://pr-614--camel.netlify.app/components/latest/activemq-component.html#_path_parameters_2_parameters>
> - All the names in the generated tables have ids so they can be linked to but there’s no easy way to find out the id. The sections have a link to themselves which makes it easy to copy the URL.  I haven’t looked into whether this can be done with  the table entries.
>
> Updating the main PR to track changes in camel is somewhat time consuming so I’d appreciate it if we could move this towards resolution quickly.
>
> David Jencks
>
>
> > On Aug 31, 2021, at 11:42 PM, David Jencks <da...@gmail.com> wrote:
> >
> > With  both the CI build for my PR and locally, `yarn build-all` is building the UI after the main build command.  Obviously this doesn’t work.
> >
> > Can someone point to a situation in which the UI is built before it is used?  Any ideas why yarn is picking this dysfunctional workspace build order?
> >
> > David Jencks
> >
> >> On Aug 30, 2021, at 1:23 AM, Claus Ibsen <claus.ibsen@gmail.com <ma...@gmail.com>> wrote:
> >>
> >> On Sun, Aug 29, 2021 at 10:20 AM David Jencks <david.a.jencks@gmail.com <ma...@gmail.com>> wrote:
> >>>
> >>> The current source state is now visible.
> >>>
> >>> - I npm-published my asciidoctor-jsonpath extension
> >>> - The site content changes are at https://github.com/djencks/camel.git <https://github.com/djencks/camel.git> jsonpath-options branch. I’ve put all the generated changes as the last commit, so it should be possible to update the branch by dropping the last commit, rebasing on main, and regenerating the source with the maven build.
> >>> - The camel-website changes are at https://github.com/djencks/camel-website.git <https://github.com/djencks/camel-website.git> issue-16854-jsonpath-options branch
> >>>
> >>> There’s a PR https://github.com/apache/camel-website/pull/614 <https://github.com/apache/camel-website/pull/614>.
> >>>
> >>> I don’t understand what the patch-sitemap.js does and was having dependency problems so removed it from the command line.  What does it do and why?
> >>>
> >>> The PR’s build fails because of missing ui bundle.  Is this expected?  I like to have the built UI bundle available somewhere: I often check in the built bundle.  Locally I can’t build the UI, it complains somehow about the helpers for component sorting/hiding.
> >>>
> >>> I set up the build to fail on warnings, and there are quite a few warnings.
> >>>
> >>> There are several source problems I don’t know the proper fix for as it requires domain knowledge.  One is these warnings:
> >>>
> >>> [00:47:14.821] WARN (asciidoctor): skipping reference to missing attribute: apisyntax
> >>>    file: docs/components/modules/ROOT/pages/google-calendar-stream-component.adoc
> >>>    source: https://github.com/djencks/camel.git <https://github.com/djencks/camel.git> (refname: jsonpath-options, start path: docs/components)
> >>> [00:47:15.979] WARN (asciidoctor): skipping reference to missing attribute: apisyntax
> >>>    file: docs/components/modules/ROOT/pages/google-mail-stream-component.adoc
> >>>    source: https://github.com/djencks/camel.git <https://github.com/djencks/camel.git> (refname: jsonpath-options, start path: docs/components)
> >>> [00:47:16.442] WARN (asciidoctor): skipping reference to missing attribute: apisyntax
> >>>    file: docs/components/modules/ROOT/pages/google-sheets-stream-component.adoc
> >>>    source: https://github.com/djencks/camel.git <https://github.com/djencks/camel.git> (refname: jsonpath-options, start path: docs/components)
> >>>
> >>>
> >>> These three components are missing an apisyntax entry in their json files.  The problem shows up in the current site as the literal string ’null’.
> >>>
> >>
> >> Ah yeah it looks like the -stream components are not API based, I am
> >> going to fix this for Camel 3.12.
> >>
> >>
> >>> There are some inconsistencies and mysteries in the gulpfile.js. I’ve commented on some.
> >>>
> >>> It would be great to know if anyone else can build the site from the PR branch.
> >>>
> >>> David Jencks
> >>>
> >>>> On Aug 23, 2021, at 6:00 AM, Claus Ibsen <claus.ibsen@gmail.com <ma...@gmail.com>> wrote:
> >>>>
> >>>> Hi
> >>>>
> >>>> Ah yeah those were an idea to include the full page documentation in
> >>>> case tooling may be able to use that for something useable.
> >>>> However the tooling uses all the other bits, so we have just marked
> >>>> those apis as deprecated.
> >>>>
> >>>> So we can remove the files from the camel-catalog.
> >>>> I have created a ticket to remove them
> >>>> https://issues.apache.org/jira/browse/CAMEL-16881 <https://issues.apache.org/jira/browse/CAMEL-16881>
> >>>>
> >>>> On Sun, Aug 22, 2021 at 9:48 AM Zoran Regvart <zo...@regvart.com> wrote:
> >>>>>
> >>>>> Hi David,
> >>>>>
> >>>>> On Sat, Aug 21, 2021 at 10:10 PM David Jencks <da...@gmail.com> wrote:
> >>>>>>
> >>>>>> I have a question about the purpose of the .adoc files in the catalog.  The changes proposed here will remove the tables of options from these copies of the component .adoc files.  These copies are already quite skimpy as they don’t successfully include the spring-boot information.
> >>>>>>
> >>>>>> What are these copies of the .adoc files supposed to be useful for?
> >>>>>
> >>>>> I think the catalog contains those so that an alternative UI can show
> >>>>> them, think tooltips or inline help in an IDE. I think, though I'm not
> >>>>> 100% sure that VSCode tooling is using that...
> >>>>>
> >>>>>> If there’s a desire to make them more complete, one strategy would be to build the components module of the website using a custom UI that has nothing in it, so we just get plain undecorated html pages, and putting those in the catalog.  It would require some investigation, but it might conceivably be possible to arrange so links out of the components module go to the website rather than just be broken.
> >>>>>
> >>>>> I'd keep it as simple as it can be, so plain .adoc files, or if we
> >>>>> find out they're not used just remove them...
> >>>>>
> >>>>> 2c
> >>>>>
> >>>>> zoran
> >>>>> --
> >>>>> Zoran Regvart
> >>>>
> >>>>
> >>>>
> >>>> --
> >>>> Claus Ibsen
> >>>> -----------------
> >>>> http://davsclaus.com @davsclaus
> >>>> Camel in Action 2: https://www.manning.com/ibsen2
> >>>
> >>
> >>
> >> --
> >> Claus Ibsen
> >> -----------------
> >> http://davsclaus.com <http://davsclaus.com/> @davsclaus
> >> Camel in Action 2: https://www.manning.com/ibsen2 <https://www.manning.com/ibsen2>
>


-- 
Claus Ibsen
-----------------
http://davsclaus.com @davsclaus
Camel in Action 2: https://www.manning.com/ibsen2

Re: [Website] Possible change in components page generation (preview available)

Posted by Zoran Regvart <zo...@regvart.com>.
Hi David,

On Fri, Sep 10, 2021 at 7:36 PM David Jencks <da...@gmail.com> wrote:
...snip...
> I certainly agree build speed is important.  However, I also think that we should make it difficult for people to edit the docs and not look at the results: this is analogous to changing the java code and not checking whether it compiles.  The idea behind Dan’s site-manifest idea is that a site build can produce a json file listing the files or resources in the Antora catalog and then another site build can read that in to the catalog to resolve xrefs etc.  In the case of a component or similar project, the idea would be to build just the single doc page for that component, but have the xrefs resolved to the main site.  Since we’d only be building one page, it should be very quick.
>
> This is definitely not a solution available out-of-the-box now but perhaps something to think about and aim for for the future.  Among other things we’d have to make the UI bundle accessible and the main site site-manifest accessible.
>
> If we can get this to work then the question opens of whether validating the .adoc files is better done in javascript as part of the (1-page, here) site build or in a maven plugin.

Oh, that seems really neat, so let's watch out for the site manifests
and try to get them working when it's shipped in Antora. With ASF
policy on how websites are built/distributed I'm not sure if we can
leverage distributed website build, but that might be something we can
look into.

zoran
-- 
Zoran Regvart

Re: [Website] Possible change in components page generation (preview available)

Posted by David Jencks <da...@gmail.com>.

> On Sep 10, 2021, at 12:57 AM, Zoran Regvart <zo...@regvart.com> wrote:
> 
> Hi David,
> 
> On Thu, Sep 9, 2021 at 6:36 PM David Jencks <da...@gmail.com> wrote:
>> I wonder if we could have, as part of the maven build of a component, perhaps in a profile, Antora build just that component's doc page. It might be possible to make this visible as part of the whole Antora site using the site-manifest idea Dan’s been working on.  I turned this into a (non-standard, so far) pipeline extension: antora-site-manifest <https://gitlab.com/djencks/antora-site-manifest>
> 
> Up for debate really. My 2c on this is that we invested a lot of
> effort in build performance and if we add more processing to the build
> this goes against that. Also as any .adoc can link via xref to any
> other .adoc wouldn't we build a non-trivial subset of pages for every
> component .adoc? That would severely impact the build times. I much
> rather have a linter go over the .adoc file with the context of the
> component metadata and check for things we know cause issues (missing
> JSON files, missing attributes, broken xrefs...), and improve to add
> more checks as we find more common issues...

I certainly agree build speed is important.  However, I also think that we should make it difficult for people to edit the docs and not look at the results: this is analogous to changing the java code and not checking whether it compiles.  The idea behind Dan’s site-manifest idea is that a site build can produce a json file listing the files or resources in the Antora catalog and then another site build can read that in to the catalog to resolve xrefs etc.  In the case of a component or similar project, the idea would be to build just the single doc page for that component, but have the xrefs resolved to the main site.  Since we’d only be building one page, it should be very quick.  

This is definitely not a solution available out-of-the-box now but perhaps something to think about and aim for for the future.  Among other things we’d have to make the UI bundle accessible and the main site site-manifest accessible.

If we can get this to work then the question opens of whether validating the .adoc files is better done in javascript as part of the (1-page, here) site build or in a maven plugin.

David Jencks

Re: [Website] Possible change in components page generation (preview available)

Posted by Zoran Regvart <zo...@regvart.com>.
Hi David,

On Thu, Sep 9, 2021 at 6:36 PM David Jencks <da...@gmail.com> wrote:
> I wonder if we could have, as part of the maven build of a component, perhaps in a profile, Antora build just that component's doc page. It might be possible to make this visible as part of the whole Antora site using the site-manifest idea Dan’s been working on.  I turned this into a (non-standard, so far) pipeline extension: antora-site-manifest <https://gitlab.com/djencks/antora-site-manifest>

Up for debate really. My 2c on this is that we invested a lot of
effort in build performance and if we add more processing to the build
this goes against that. Also as any .adoc can link via xref to any
other .adoc wouldn't we build a non-trivial subset of pages for every
component .adoc? That would severely impact the build times. I much
rather have a linter go over the .adoc file with the context of the
component metadata and check for things we know cause issues (missing
JSON files, missing attributes, broken xrefs...), and improve to add
more checks as we find more common issues...

zoran
-- 
Zoran Regvart

Re: [Website] Possible change in components page generation (preview available)

Posted by David Jencks <da...@gmail.com>.

> On Sep 8, 2021, at 11:22 AM, Zoran Regvart <zo...@regvart.com> wrote:
> 
> Hi David & Cameleers,
> 
> On Wed, Sep 8, 2021 at 4:48 PM David Jencks <david.a.jencks@gmail.com <ma...@gmail.com>> wrote:
>> 
>>> <snip>
>>>>>> - If nothing else is using the model building in UpdateReadmeMojo it can probably be simplified further.
>>>>> 
>>>>> From what I understand we only need the JSON files generated now,
>>>>> perhaps the we don't need the UpdateReadmeMojo at all any more?
>>>> 
>>>> Perhaps.  It still generates the doc header.  If Allow asciidoctor extensions when processing nav files and headers <https://gitlab.com/antora/antora/-/issues/592> were implemented, the header generation could be done by the jsonpath extension, but I’m not sure how much of an improvement that would be.  In any case, UpdateReadmeMojo is now more of a sanity checker than content generator :-).
>>> 
>>> I kinda think that we should have a prescribed attributes that any
>>> document needs to be published and if there are attributes missing or
>>> values don't make sense then that needs to be corrected. I saw this
>>> work as a good way to reduce the amount of regen commits and moving
>>> this to a responsibility of the author.
>>> 
>>> Perhaps we can change the role of the UpdateReadmeMojo (and rename
>>> it), to be a linter for the documentation? Any thoughts
>>> Claus/Andrea/others?
>> 
>> If we agree it should be only a checker, the functionality could also be implemented as an Antora pipeline extension. This would fail the site build rather than doing something during the java build. I don’t know whether or not this would be a good idea, but it’s a possibility :-)
> 
> I think the website build is far removed from the other projects, so
> by the time folk see the issue the context has already been forgotten.
> Especially for new contributors, contributing a new component and doc
> improvements, they might find it confusing that everything was built
> okay, but the page on the website, or even the website build is
> broken.

Currently indeed that is a problem, but on the other hand I think that we don’t want people changing the documentation without looking at the result in the site, at least the result for the page(s) they changed.

I wonder if we could have, as part of the maven build of a component, perhaps in a profile, Antora build just that component's doc page. It might be possible to make this visible as part of the whole Antora site using the site-manifest idea Dan’s been working on.  I turned this into a (non-standard, so far) pipeline extension: antora-site-manifest <https://gitlab.com/djencks/antora-site-manifest>

David Jencks

Re: [Website] Possible change in components page generation (preview available)

Posted by Zoran Regvart <zo...@regvart.com>.
Hi David & Cameleers,
after much arguing with Gulp I created a pull request for the next
step in this: to use symbolic links instead of file copies for the
documentation. The pull request is up for review and I very much
welcome feedback, for example I did not test this other than on a Unix
(Linux) machine, so there _might_ be some issues on Windows.

https://github.com/apache/camel/pull/6072

I'm also very interested in usability feedback, I do realize that for
non-JavaScript/Gulp/Asciidoc experts this might be a difficult process
to understand. I did try to put in as much information in comments to
help with understanding. So if there is anything that is not clear I'd
like to improve on that.

Next step would be to enable creation of symbolic links for Camel JSON
metadata files and apply changes from David's[1] PR that use the
jsonpath macro to generate tables of options instead of the Maven
UpdateReadme mojo.

Again, big thanks to David that started this work, this is merely a
refinement of his ideas taken piece by piece to prevent a big bang
rollout...

zoran

[1] https://github.com/apache/camel/pull/6040
-- 
Zoran Regvart

Re: [Website] Possible change in components page generation (preview available)

Posted by Zoran Regvart <zo...@regvart.com>.
Hi David & Cameleers,

On Thu, Sep 9, 2021 at 6:57 PM David Jencks <da...@gmail.com> wrote:
>
> Overall I like the gulp file changes, I left one comment.

Thanks for that, good point, I think for future maintenance it's
important to have the Gulp build as understandable as possible.
Otherwise we'll end up with black magic spaghetti. I'll try to make
stuff more apparent or add more comments so it can be understood a bit
better.

> I discovered another anomaly you might like to investigate:
> Is components/camel-disruptor/src/main/docs/disruptor-vm-component.adoc supposed to exist?  It doesn’t appear to have a json file for properties etc, and it’s rendering in the current site looks slightly odd (with the empty “Options” section).

Most likely the glob for inclusion did not match it, on the
`pr/docs-use-symlinks` branch on my fork I just run the build and I
did get all the files needed:

$ cd docs
$ yarn gulp
$ find . -name '*disruptor*.*'
./components/modules/ROOT/pages/disruptor-component.adoc
./components/modules/ROOT/pages/disruptor-vm-component.adoc
./components/modules/ROOT/examples/json/disruptor-vm.json
./components/modules/ROOT/examples/json/disruptor.json

But another discrepancy I noticed was in the Bindy component, we have
one .adoc file and three JSON files:

$ find . -name '*bindy*.*'
./components/modules/dataformats/pages/bindy-dataformat.adoc
./components/modules/dataformats/examples/json/bindy-csv.json
./components/modules/dataformats/examples/json/bindy-fixed.json
./components/modules/dataformats/examples/json/bindy-kvp.json

That will be a problem, I'll investigate and then create a followup
JIRA issue if I can't find a solution.

I think this underlines the piecemeal approach I'm trying to take,
we'll switch to using symlinks first and then we can change to use the
jsonpath macros and JSON files when we see that everything is in
place...

zoran
-- 
Zoran Regvart

Re: [Website] Possible change in components page generation (preview available)

Posted by David Jencks <da...@gmail.com>.
Overall I like the gulp file changes, I left one comment.

I discovered another anomaly you might like to investigate:
Is components/camel-disruptor/src/main/docs/disruptor-vm-component.adoc supposed to exist?  It doesn’t appear to have a json file for properties etc, and it’s rendering in the current site looks slightly odd (with the empty “Options” section).

Thanks!
David Jencks

> On Sep 9, 2021, at 8:15 AM, Zoran Regvart <zo...@regvart.com> wrote:
> 
> Hi David & Cameleers,
> I've made some progress in the main Camel repository focusing on the
> Gulp tasks to generate symlinks and maintain nav.adoc files. Much
> inspired by the work David did on the JSON files and trying to
> generate Gulp tasks from data rather than having multiple, mostly
> same, functionality in separate Gulp function tasks. This is still
> very much a work in progress and a bit clunky at the moment, so I did
> not create a PR at this time. But if someone wishes to have a look at
> how this is shaping up, the code is here:
> 
> https://github.com/zregvart/camel/commit/f753ed1099a5b93c4c5880f7ee220a20d66bdb58
> 
> I'll rework that much more later today/tomorrow. Again the focus,
> right now is to use symbolic links instead of copying .adoc files and
> to simplify the Gulp build by having it data driven, hopefully this
> will make it easier to maintain in the future.
> 
> I've found the same issues David did, so the comments David left in
> the PR #6040[1] still stand -- we're working around some
> inconsistencies in location and naming of .adoc files, look for
> FIXME/IMPORTANT comments in gulpfile.js. I'll try to resolve those so
> we don't carry too much baggage forward, and I might have some more
> questions later.
> 
> I've also prepared a bit (still incomplete) the second part of David's
> changes, to get the component/endpoint option tables generated from
> generated catalog JSON files, I think that I'll split that off into
> two changes one for Gulp refactoring & symlinks and another for that.
> 
> Again, very much a work in progress so feedback at this stage is very welcome...
> 
> zoran
> 
> [1] https://github.com/apache/camel/pull/6040
> -- 
> Zoran Regvart

Re: [Website] Possible change in components page generation (preview available)

Posted by Zoran Regvart <zo...@regvart.com>.
Hi David & Cameleers,
I've made some progress in the main Camel repository focusing on the
Gulp tasks to generate symlinks and maintain nav.adoc files. Much
inspired by the work David did on the JSON files and trying to
generate Gulp tasks from data rather than having multiple, mostly
same, functionality in separate Gulp function tasks. This is still
very much a work in progress and a bit clunky at the moment, so I did
not create a PR at this time. But if someone wishes to have a look at
how this is shaping up, the code is here:

https://github.com/zregvart/camel/commit/f753ed1099a5b93c4c5880f7ee220a20d66bdb58

I'll rework that much more later today/tomorrow. Again the focus,
right now is to use symbolic links instead of copying .adoc files and
to simplify the Gulp build by having it data driven, hopefully this
will make it easier to maintain in the future.

I've found the same issues David did, so the comments David left in
the PR #6040[1] still stand -- we're working around some
inconsistencies in location and naming of .adoc files, look for
FIXME/IMPORTANT comments in gulpfile.js. I'll try to resolve those so
we don't carry too much baggage forward, and I might have some more
questions later.

I've also prepared a bit (still incomplete) the second part of David's
changes, to get the component/endpoint option tables generated from
generated catalog JSON files, I think that I'll split that off into
two changes one for Gulp refactoring & symlinks and another for that.

Again, very much a work in progress so feedback at this stage is very welcome...

zoran

[1] https://github.com/apache/camel/pull/6040
-- 
Zoran Regvart

Re: [Website] Possible change in components page generation (preview available)

Posted by Zoran Regvart <zo...@regvart.com>.
Hi David & Cameleers,

On Wed, Sep 8, 2021 at 4:48 PM David Jencks <da...@gmail.com> wrote:
>
> > <snip>
> >>>> - If nothing else is using the model building in UpdateReadmeMojo it can probably be simplified further.
> >>>
> >>> From what I understand we only need the JSON files generated now,
> >>> perhaps the we don't need the UpdateReadmeMojo at all any more?
> >>
> >> Perhaps.  It still generates the doc header.  If Allow asciidoctor extensions when processing nav files and headers <https://gitlab.com/antora/antora/-/issues/592> were implemented, the header generation could be done by the jsonpath extension, but I’m not sure how much of an improvement that would be.  In any case, UpdateReadmeMojo is now more of a sanity checker than content generator :-).
> >
> > I kinda think that we should have a prescribed attributes that any
> > document needs to be published and if there are attributes missing or
> > values don't make sense then that needs to be corrected. I saw this
> > work as a good way to reduce the amount of regen commits and moving
> > this to a responsibility of the author.
> >
> > Perhaps we can change the role of the UpdateReadmeMojo (and rename
> > it), to be a linter for the documentation? Any thoughts
> > Claus/Andrea/others?
>
> If we agree it should be only a checker, the functionality could also be implemented as an Antora pipeline extension. This would fail the site build rather than doing something during the java build. I don’t know whether or not this would be a good idea, but it’s a possibility :-)

I think the website build is far removed from the other projects, so
by the time folk see the issue the context has already been forgotten.
Especially for new contributors, contributing a new component and doc
improvements, they might find it confusing that everything was built
okay, but the page on the website, or even the website build is
broken.

zoran
-- 
Zoran Regvart

Re: [Website] Possible change in components page generation (preview available)

Posted by David Jencks <da...@gmail.com>.
> <snip>
>>>> - If nothing else is using the model building in UpdateReadmeMojo it can probably be simplified further.
>>> 
>>> From what I understand we only need the JSON files generated now,
>>> perhaps the we don't need the UpdateReadmeMojo at all any more?
>> 
>> Perhaps.  It still generates the doc header.  If Allow asciidoctor extensions when processing nav files and headers <https://gitlab.com/antora/antora/-/issues/592> were implemented, the header generation could be done by the jsonpath extension, but I’m not sure how much of an improvement that would be.  In any case, UpdateReadmeMojo is now more of a sanity checker than content generator :-).
> 
> I kinda think that we should have a prescribed attributes that any
> document needs to be published and if there are attributes missing or
> values don't make sense then that needs to be corrected. I saw this
> work as a good way to reduce the amount of regen commits and moving
> this to a responsibility of the author.
> 
> Perhaps we can change the role of the UpdateReadmeMojo (and rename
> it), to be a linter for the documentation? Any thoughts
> Claus/Andrea/others?

If we agree it should be only a checker, the functionality could also be implemented as an Antora pipeline extension. This would fail the site build rather than doing something during the java build. I don’t know whether or not this would be a good idea, but it’s a possibility :-)

David Jencks
> 
> <snip>
> zoran
> -- 
> Zoran Regvart

Re: [Website] Possible change in components page generation (preview available)

Posted by Zoran Regvart <zo...@regvart.com>.
Hi David,
snipping a bunch of text, trying to keep this short for other folk to follow...

On Tue, Sep 7, 2021 at 5:22 PM David Jencks <da...@gmail.com> wrote:
> > On Fri, Sep 3, 2021 at 8:21 AM David Jencks <da...@gmail.com> wrote:
> >> comments:
> >>
> >> - I removed the patch-sitemap.js use.  I can’t figure out why this is needed.  If it really is needed, it should be done with a pipeline extension.
> >
> > I think we'll need to restore this, otherwise we'll lose the sitemap
> > generation which in turn leaves us with outdated links on the search
> > engines, and as we change the links when we add or remove new
> > documentation versions this becomes an issue.
> >
>
> I’m rather confused by this since my experience and Dan’s repeated statements are that Antora always generates relative links within the site.  It’s possible to write the UI to generate either absolute or relative links, but that’s not something Antora is doing itself.  Getting Antora to generate absolute links in page bodies (not the UI) requires major surgery :-)  So, I’ll compare generating with absolute and ‘/‘ siteUrl and come back with more definite info.

We should check if the sitemaps are generated correctly, I think if
it's not absolute they will not be generated at all...

> >> - yarn workspace  —topological-dev is not building the UI before it’s used.  I can’t figure out why; it is for other PRs.  I replaced this by just calling the UI build directly.
> >
> > That's odd, I did not notice this issue before, I'll try to reproduce
> > it and see if there is something we can do about this, this should be
> > one of the benefits in using workspaces.
>
> It doesn’t seem to afflict other people’s PRs :-), but I have not yet had any build use the correct order with —topological-dev.  One run showing the problem is https://github.com/apache/camel-website/runs/3453859096?check_suite_focus=true.  I wonder if it’s related to checking in the wrong .yarn contents: I think in this particular run I hadn’t checked in any .yarn changes, but later I have been checking in such changes and still get the wrong order.

Hmm, I'll try to have a better look, for sure we want this to be
consistent, it could be the .yarn, but it might be something else,
I'll try to investigate.

Since you're doing this on a Mac, the link checker we use uses a Linux
binary, so that won't work for sure...

> >> - If nothing else is using the model building in UpdateReadmeMojo it can probably be simplified further.
> >
> > From what I understand we only need the JSON files generated now,
> > perhaps the we don't need the UpdateReadmeMojo at all any more?
>
> Perhaps.  It still generates the doc header.  If Allow asciidoctor extensions when processing nav files and headers <https://gitlab.com/antora/antora/-/issues/592> were implemented, the header generation could be done by the jsonpath extension, but I’m not sure how much of an improvement that would be.  In any case, UpdateReadmeMojo is now more of a sanity checker than content generator :-).

I kinda think that we should have a prescribed attributes that any
document needs to be published and if there are attributes missing or
values don't make sense then that needs to be corrected. I saw this
work as a good way to reduce the amount of regen commits and moving
this to a responsibility of the author.

Perhaps we can change the role of the UpdateReadmeMojo (and rename
it), to be a linter for the documentation? Any thoughts
Claus/Andrea/others?

> >> - The camel-website node dependencies have changed dramatically.  It looks like the yarn cache is checked into  git; I haven’t checked in the changes.  Should I?
> >
>
> Later on I started checking in .yarn, but I’m really not sure if I’m getting appropriate content.  I think that previously the UI was not in the main .yarn, but at least sometimes I get it, and it’s > 50 MB and I get a warning.

I haven't checked this, GitHub workflow runs `yarn workspaces foreach
install --check-cache` -- so if that's happy it should be okay to push
50MB cache changes.

> Note that I have 2 camel-website PRs: the “antora 3” one is the one that should be merged, and after the camel jsonpath-options PR is merged it will build with all the new stuff.  The other camel-website PR is just to build the jsonpath-options branch.

Noted :)

zoran
-- 
Zoran Regvart

Re: [Website] Possible change in components page generation (preview available)

Posted by David Jencks <da...@gmail.com>.
OK I think everything’s ready to merge…. we’ll see if CI agrees :-)

David Jencks

> On Sep 20, 2021, at 11:48 PM, Zoran Regvart <zo...@regvart.com> wrote:
> 
> Hi David,
> big thanks for cleaning that up!
> 
> On Tue, Sep 21, 2021 at 6:30 AM David Jencks <da...@gmail.com> wrote:
>> 
>> I think I’ve covered all the work related to this in a final PR, https://github.com/apache/camel/pull/6124.  It was supposed to be simple but I discovered quite a few problems…
>> 
>> I suspect it would be fairly simple to back port these changes to 3.11.x.  Shall I look into doing that?  3.7.x would probably be harder, but I could look into that also. 3.4.x is about to be removed so I won’t consider it, nor 2.x
> 
> I'd squash those changes and point to the JIRA issue from the commit
> title. I don't think we need to backport these, the docs end up mostly
> the same and will fall off the website eventually...
> 
> zoran
> -- 
> Zoran Regvart

Re: [Website] Possible change in components page generation (preview available)

Posted by Zoran Regvart <zo...@regvart.com>.
Hi David,
big thanks for cleaning that up!

On Tue, Sep 21, 2021 at 6:30 AM David Jencks <da...@gmail.com> wrote:
>
> I think I’ve covered all the work related to this in a final PR, https://github.com/apache/camel/pull/6124.  It was supposed to be simple but I discovered quite a few problems…
>
> I suspect it would be fairly simple to back port these changes to 3.11.x.  Shall I look into doing that?  3.7.x would probably be harder, but I could look into that also. 3.4.x is about to be removed so I won’t consider it, nor 2.x

I'd squash those changes and point to the JIRA issue from the commit
title. I don't think we need to backport these, the docs end up mostly
the same and will fall off the website eventually...

zoran
-- 
Zoran Regvart

Re: [Website] Possible change in components page generation (preview available)

Posted by Claus Ibsen <cl...@gmail.com>.
On Tue, Sep 21, 2021 at 6:30 AM David Jencks <da...@gmail.com> wrote:
>
> I think I’ve covered all the work related to this in a final PR, https://github.com/apache/camel/pull/6124.  It was supposed to be simple but I discovered quite a few problems…
>
> I suspect it would be fairly simple to back port these changes to 3.11.x.  Shall I look into doing that?  3.7.x would probably be harder, but I could look into that also. 3.4.x is about to be removed so I won’t consider it, nor 2.x
>

Hmm I would keep it for main only - there are too much changes
already. We should keep the LTS branches stable and only do changes if
its a bug or CVE etc.


> David Jencks
>
> > On Sep 13, 2021, at 9:05 AM, David Jencks <da...@gmail.com> wrote:
> >
> >
> >
> >> On Sep 13, 2021, at 8:05 AM, Zoran Regvart <zo...@regvart.com> wrote:
> >>
> >> Hi Cameleers,
> >> some of these changes are in, and for the last one explicitly stated
> >> in the $subject I'll create a pull request soon.
> >>
> >> You will notice changes in the workflow and in the git repository.
> >> We're now using symbolic links and not copying files over to the docs
> >> directory, with the jsonpath macro, that's in the works now, the
> >> configuration tables will not be generated with the Maven plugin.
> >>
> >> So far we did not have major issues with this, with jsonpath macro
> >> change I do anticipate some issues as the assumption there is a
> >> one-to-one relationship between the .adoc file and the .json metadata
> >> file. And that is not true for some cases. Right now I'm thinking in
> >> those cases we keep the Maven plugin, but I have to see if that can be
> >> avoided.
> >
> > I thought I’d taken care of these, but I could easily have missed some.  The name of the json file for a component/dataformat/etc page is in the `shortname` attribute so I think we can deal with these cases by setting that attribute explicitly in the maven plugin when we write out the header.  I remember doing this explicitly for mail/imap/smtp/etc and I thought bindy just worked. The bindy results look OK to me…
> >
> > One possibility for the future relates to the annotations docs that I think are only in bindy at the moment.  Architecturally I don’t like having the maven doc construction plugin scanning the source for annotations. I think it would be more appropriate to move this scanning code to whatever is constructing the json metadata file and put the results into the json file.  That’s definitely not part of this current effort :-)
> >
> > Thanks for all the work!  It definitely makes sense to roll this out in the small chunks you are creating, although that certainly wasn’t how I developed it!  Having more eyes is wonderful :-)
> >
> > David Jencks
> >
> >>
> >> Anyhow, happy to hear comments on this and do reach out if you notice
> >> any faults on the website or issues in your workflow...
> >>
> >> zoran
> >> --
> >> Zoran Regvart
>


-- 
Claus Ibsen
-----------------
http://davsclaus.com @davsclaus
Camel in Action 2: https://www.manning.com/ibsen2

Re: [Website] Possible change in components page generation (preview available)

Posted by David Jencks <da...@gmail.com>.
I think I’ve covered all the work related to this in a final PR, https://github.com/apache/camel/pull/6124.  It was supposed to be simple but I discovered quite a few problems…

I suspect it would be fairly simple to back port these changes to 3.11.x.  Shall I look into doing that?  3.7.x would probably be harder, but I could look into that also. 3.4.x is about to be removed so I won’t consider it, nor 2.x

David Jencks

> On Sep 13, 2021, at 9:05 AM, David Jencks <da...@gmail.com> wrote:
> 
> 
> 
>> On Sep 13, 2021, at 8:05 AM, Zoran Regvart <zo...@regvart.com> wrote:
>> 
>> Hi Cameleers,
>> some of these changes are in, and for the last one explicitly stated
>> in the $subject I'll create a pull request soon.
>> 
>> You will notice changes in the workflow and in the git repository.
>> We're now using symbolic links and not copying files over to the docs
>> directory, with the jsonpath macro, that's in the works now, the
>> configuration tables will not be generated with the Maven plugin.
>> 
>> So far we did not have major issues with this, with jsonpath macro
>> change I do anticipate some issues as the assumption there is a
>> one-to-one relationship between the .adoc file and the .json metadata
>> file. And that is not true for some cases. Right now I'm thinking in
>> those cases we keep the Maven plugin, but I have to see if that can be
>> avoided.
> 
> I thought I’d taken care of these, but I could easily have missed some.  The name of the json file for a component/dataformat/etc page is in the `shortname` attribute so I think we can deal with these cases by setting that attribute explicitly in the maven plugin when we write out the header.  I remember doing this explicitly for mail/imap/smtp/etc and I thought bindy just worked. The bindy results look OK to me…
> 
> One possibility for the future relates to the annotations docs that I think are only in bindy at the moment.  Architecturally I don’t like having the maven doc construction plugin scanning the source for annotations. I think it would be more appropriate to move this scanning code to whatever is constructing the json metadata file and put the results into the json file.  That’s definitely not part of this current effort :-)
> 
> Thanks for all the work!  It definitely makes sense to roll this out in the small chunks you are creating, although that certainly wasn’t how I developed it!  Having more eyes is wonderful :-)
> 
> David Jencks
> 
>> 
>> Anyhow, happy to hear comments on this and do reach out if you notice
>> any faults on the website or issues in your workflow...
>> 
>> zoran
>> -- 
>> Zoran Regvart

Re: [Website] Possible change in components page generation (preview available)

Posted by David Jencks <da...@gmail.com>.

> On Sep 13, 2021, at 8:05 AM, Zoran Regvart <zo...@regvart.com> wrote:
> 
> Hi Cameleers,
> some of these changes are in, and for the last one explicitly stated
> in the $subject I'll create a pull request soon.
> 
> You will notice changes in the workflow and in the git repository.
> We're now using symbolic links and not copying files over to the docs
> directory, with the jsonpath macro, that's in the works now, the
> configuration tables will not be generated with the Maven plugin.
> 
> So far we did not have major issues with this, with jsonpath macro
> change I do anticipate some issues as the assumption there is a
> one-to-one relationship between the .adoc file and the .json metadata
> file. And that is not true for some cases. Right now I'm thinking in
> those cases we keep the Maven plugin, but I have to see if that can be
> avoided.

I thought I’d taken care of these, but I could easily have missed some.  The name of the json file for a component/dataformat/etc page is in the `shortname` attribute so I think we can deal with these cases by setting that attribute explicitly in the maven plugin when we write out the header.  I remember doing this explicitly for mail/imap/smtp/etc and I thought bindy just worked. The bindy results look OK to me…

One possibility for the future relates to the annotations docs that I think are only in bindy at the moment.  Architecturally I don’t like having the maven doc construction plugin scanning the source for annotations. I think it would be more appropriate to move this scanning code to whatever is constructing the json metadata file and put the results into the json file.  That’s definitely not part of this current effort :-)

Thanks for all the work!  It definitely makes sense to roll this out in the small chunks you are creating, although that certainly wasn’t how I developed it!  Having more eyes is wonderful :-)

David Jencks

> 
> Anyhow, happy to hear comments on this and do reach out if you notice
> any faults on the website or issues in your workflow...
> 
> zoran
> -- 
> Zoran Regvart

Re: [Website] Possible change in components page generation (preview available)

Posted by Zoran Regvart <zo...@regvart.com>.
Hi Cameleers,
some of these changes are in, and for the last one explicitly stated
in the $subject I'll create a pull request soon.

You will notice changes in the workflow and in the git repository.
We're now using symbolic links and not copying files over to the docs
directory, with the jsonpath macro, that's in the works now, the
configuration tables will not be generated with the Maven plugin.

So far we did not have major issues with this, with jsonpath macro
change I do anticipate some issues as the assumption there is a
one-to-one relationship between the .adoc file and the .json metadata
file. And that is not true for some cases. Right now I'm thinking in
those cases we keep the Maven plugin, but I have to see if that can be
avoided.

Anyhow, happy to hear comments on this and do reach out if you notice
any faults on the website or issues in your workflow...

zoran
-- 
Zoran Regvart

Re: [Website] Possible change in components page generation (preview available)

Posted by Zoran Regvart <zo...@regvart.com>.
On Sat, Sep 11, 2021 at 11:06 PM David Jencks <da...@gmail.com> wrote:
> I was inspired:-) and came up with https://github.com/apache/camel-website/pull/624

Awesome, thanks, merged :)

zoran
-- 
Zoran Regvart

Re: [Website] Possible change in components page generation (preview available)

Posted by David Jencks <da...@gmail.com>.
I was inspired:-) and came up with https://github.com/apache/camel-website/pull/624

David Jencks

> On Sep 8, 2021, at 11:19 AM, Zoran Regvart <zo...@regvart.com> wrote:
> 
> Hi David,
> 
> On Wed, Sep 8, 2021 at 4:42 PM David Jencks <da...@gmail.com> wrote:
>> I’m going to leave checking the sitemap generation to you because AFAICT they are correct when generated with the site:url.
>> 
>> Would you like to make the UI changes you suggest or shall I write a PR?
> 
> I can definitively have a go, I was going to try what Dan suggested
> anyhow, that should not stop you if you're inspired though :)
> 
> I'll probably get to that when we get the jsonpath/symlink changes
> done, I feel that should take priority...
> 
> zoran
> -- 
> Zoran Regvart

Re: [Website] Possible change in components page generation (preview available)

Posted by Zoran Regvart <zo...@regvart.com>.
Hi David,

On Wed, Sep 8, 2021 at 4:42 PM David Jencks <da...@gmail.com> wrote:
> I’m going to leave checking the sitemap generation to you because AFAICT they are correct when generated with the site:url.
>
> Would you like to make the UI changes you suggest or shall I write a PR?

I can definitively have a go, I was going to try what Dan suggested
anyhow, that should not stop you if you're inspired though :)

I'll probably get to that when we get the jsonpath/symlink changes
done, I feel that should take priority...

zoran
-- 
Zoran Regvart

Re: [Website] Possible change in components page generation (preview available)

Posted by David Jencks <da...@gmail.com>.
Hi Zoran!

I’m going to leave checking the sitemap generation to you because AFAICT they are correct when generated with the site:url.

Would you like to make the UI changes you suggest or shall I write a PR?

Thanks!
David Jencks

> On Sep 8, 2021, at 2:26 AM, Zoran Regvart <zo...@regvart.com> wrote:
> 
> Hi David,
> 
> On Tue, Sep 7, 2021 at 7:32 PM David Jencks <da...@gmail.com> wrote:
>> 
>> I investigated the patch-sitemap.js question a bit.
>> 
>> Using my `issue-16854-jsonpath-options` camel-website branch, I built the site twice, with site:url set to `https://camel.apache.org` and set to `/`.  I didn’t look at every page, but diffing the generated sites seems to consistently show two differences between the results:
>> 
>> - with `https://camel.apache.org` a head <link> element is included such as
>> 
>>    <link rel="canonical" href="https://camel.apache.org/components/latest/ironmq-component.html">
>> 
>> It’s omitted as expected with `/`.  My understanding is that this needs to be an absolute URI and that it’s function is to help search engines.  However, if we don’t want it, it’s trivial to modify the UI to not generate it.
> 
> The Hugo built bits also contain the `<link rel="canonical>`[1]. When
> used, recommendation is to place absolute URLs[2]. Now since we use
> 301 redirects, have sitemap(s), I'm not entirely convinced we need
> `<link rel="canonical>` at all. I suggest we remove it (from Hugo
> layout and Antora UI).
> 
>> - with  `https://camel.apache.org` a footer micro data script is plausible rather than meaningless (the `url:`  entry):
> 
> I think the URLs in JSON-LD microdata need to be absolute, I can't
> find a definite reference on that, but if I test[3] with relative URL
> I get "http://example-test.site/" for URL, might be a placeholder...
> 
>> In both cases I’d expect that to be usable the logo should be an absolute URI?
> 
> Yeah, I think all URLs need to be absolute in JSON-LD, but I'm not
> 100% on that...
> 
>> I note that the next bit of micro data, BreadcrumbList, is always generated with absolute URIs with https://camel.apache.org. Shouldn’t this be generated from the site:url?
> 
> Well, I'm guessing that was a pragmatic choice when we did that
> initially, I do remember some back and forth on that but the context
> escapes me
> 
>> My conclusions are:
>> 
>> - There is no need for patch-sitemap.js and that the site needs to be generated with the correct site:url.
> 
> Let's check sitemaps and JSON-LD microdata if they are all generated
> and contain absolute URLs first. otherwise what Dan suggested on #772
> could be a good way to go...
> 
>> - If inclusion of the <link> element causes a problem it can be removed from the UI.
> 
> +1, not sure if we need it at all...
> 
>> - The Organization micro data needs it’s logo URL fixed to be absolute based on site:url
> 
> +1
> 
>> - The BreadcrumbList micro data needs to be generated based on site:url.
> 
> +1
> 
>> Have I missed something?
> 
> We need to double check the XML sitemaps...
> 
> zoran
> 
> [1] https://github.com/apache/camel-website/blob/ad0b8c6efcea9943ae8e690ec020f5589c227a54/layouts/partials/header.html#L28
> [2] https://developers.google.com/search/docs/advanced/crawling/consolidate-duplicate-urls#rel-canonical-link-method
> [3] https://search.google.com/test/rich-results?id=CaNl08DpmzTpYqlM7Cg1kA
> -- 
> Zoran Regvart

Re: [Website] Possible change in components page generation (preview available)

Posted by Zoran Regvart <zo...@regvart.com>.
Hi David,

On Tue, Sep 7, 2021 at 7:32 PM David Jencks <da...@gmail.com> wrote:
>
> I investigated the patch-sitemap.js question a bit.
>
> Using my `issue-16854-jsonpath-options` camel-website branch, I built the site twice, with site:url set to `https://camel.apache.org` and set to `/`.  I didn’t look at every page, but diffing the generated sites seems to consistently show two differences between the results:
>
> - with `https://camel.apache.org` a head <link> element is included such as
>
>     <link rel="canonical" href="https://camel.apache.org/components/latest/ironmq-component.html">
>
> It’s omitted as expected with `/`.  My understanding is that this needs to be an absolute URI and that it’s function is to help search engines.  However, if we don’t want it, it’s trivial to modify the UI to not generate it.

The Hugo built bits also contain the `<link rel="canonical>`[1]. When
used, recommendation is to place absolute URLs[2]. Now since we use
301 redirects, have sitemap(s), I'm not entirely convinced we need
`<link rel="canonical>` at all. I suggest we remove it (from Hugo
layout and Antora UI).

> - with  `https://camel.apache.org` a footer micro data script is plausible rather than meaningless (the `url:`  entry):

I think the URLs in JSON-LD microdata need to be absolute, I can't
find a definite reference on that, but if I test[3] with relative URL
I get "http://example-test.site/" for URL, might be a placeholder...

> In both cases I’d expect that to be usable the logo should be an absolute URI?

Yeah, I think all URLs need to be absolute in JSON-LD, but I'm not
100% on that...

> I note that the next bit of micro data, BreadcrumbList, is always generated with absolute URIs with https://camel.apache.org. Shouldn’t this be generated from the site:url?

Well, I'm guessing that was a pragmatic choice when we did that
initially, I do remember some back and forth on that but the context
escapes me

> My conclusions are:
>
> - There is no need for patch-sitemap.js and that the site needs to be generated with the correct site:url.

Let's check sitemaps and JSON-LD microdata if they are all generated
and contain absolute URLs first. otherwise what Dan suggested on #772
could be a good way to go...

> - If inclusion of the <link> element causes a problem it can be removed from the UI.

+1, not sure if we need it at all...

> - The Organization micro data needs it’s logo URL fixed to be absolute based on site:url

+1

> - The BreadcrumbList micro data needs to be generated based on site:url.

+1

> Have I missed something?

We need to double check the XML sitemaps...

zoran

[1] https://github.com/apache/camel-website/blob/ad0b8c6efcea9943ae8e690ec020f5589c227a54/layouts/partials/header.html#L28
[2] https://developers.google.com/search/docs/advanced/crawling/consolidate-duplicate-urls#rel-canonical-link-method
[3] https://search.google.com/test/rich-results?id=CaNl08DpmzTpYqlM7Cg1kA
-- 
Zoran Regvart

Re: [Website] Possible change in components page generation (preview available)

Posted by David Jencks <da...@gmail.com>.
I investigated the patch-sitemap.js question a bit.

Using my `issue-16854-jsonpath-options` camel-website branch, I built the site twice, with site:url set to `https://camel.apache.org` and set to `/`.  I didn’t look at every page, but diffing the generated sites seems to consistently show two differences between the results:

- with `https://camel.apache.org` a head <link> element is included such as 

    <link rel="canonical" href="https://camel.apache.org/components/latest/ironmq-component.html">

It’s omitted as expected with `/`.  My understanding is that this needs to be an absolute URI and that it’s function is to help search engines.  However, if we don’t want it, it’s trivial to modify the UI to not generate it.

- with  `https://camel.apache.org` a footer micro data script is plausible rather than meaningless (the `url:`  entry):

        <script type='application/ld+json'>
{
  "@context": "http://schema.org",
  "@type": "Organization",
  "name": "Apache Camel",
  "url": "https://camel.apache.org",
  "sameAs": [
     "https://twitter.com/ApacheCamel"
  ],
  "logo": "../../_/img/logo-d.svg",
  "description": "Apache Camel ™ is a versatile open-source integration framework based on known Enterprise Integration Patterns. Camel empowers you to define routing and mediation rules in a variety of domain-specific languages, including a Java-based Fluent API, Spring or Blueprint XML Configuration files, and a Scala DSL."
}

versus

        <script type='application/ld+json'>
{
  "@context": "http://schema.org",
  "@type": "Organization",
  "name": "Apache Camel",
  "url": "/",
  "sameAs": [
     "https://twitter.com/ApacheCamel"
  ],
  "logo": "../../_/img/logo-d.svg",
  "description": "Apache Camel ™ is a versatile open-source integration framework based on known Enterprise Integration Patterns. Camel empowers you to define routing and mediation rules in a variety of domain-specific languages, including a Java-based Fluent API, Spring or Blueprint XML Configuration files, and a Scala DSL."
}
</script>

In both cases I’d expect that to be usable the logo should be an absolute URI?

I note that the next bit of micro data, BreadcrumbList, is always generated with absolute URIs with https://camel.apache.org. Shouldn’t this be generated from the site:url?

My conclusions are:

- There is no need for patch-sitemap.js and that the site needs to be generated with the correct site:url.
- If inclusion of the <link> element causes a problem it can be removed from the UI.
- The Organization micro data needs it’s logo URL fixed to be absolute based on site:url
- The BreadcrumbList micro data needs to be generated based on site:url.

Have I missed something?

David Jencks

> On Sep 7, 2021, at 8:22 AM, David Jencks <da...@gmail.com> wrote:
> 
> Hi Zoran,
> 
> I just pushed some more fixes, I hope this isn’t making reviewing too hard.
> 
> Starting far back in the email chain is probably pretty confusing, since I or others have fixed a lot of the problems noted.  I’ve tried to summarize all the current problems in each email: I think I should have also noted which ones got fixed.
> 
> Some comments inline.
> 
>> On Sep 7, 2021, at 4:42 AM, Zoran Regvart <zoran@regvart.com <ma...@regvart.com>> wrote:
>> 
>> Hi David,
>> working through the changes and emails, some more comments here...
>> 
>> On Fri, Sep 3, 2021 at 8:21 AM David Jencks <david.a.jencks@gmail.com <ma...@gmail.com>> wrote:
>>> comments:
>>> 
>>> - I removed the patch-sitemap.js use.  I can’t figure out why this is needed.  If it really is needed, it should be done with a pipeline extension.
>> 
>> I think we'll need to restore this, otherwise we'll lose the sitemap
>> generation which in turn leaves us with outdated links on the search
>> engines, and as we change the links when we add or remove new
>> documentation versions this becomes an issue.
>> 
> 
> I’m rather confused by this since my experience and Dan’s repeated statements are that Antora always generates relative links within the site.  It’s possible to write the UI to generate either absolute or relative links, but that’s not something Antora is doing itself.  Getting Antora to generate absolute links in page bodies (not the UI) requires major surgery :-)  So, I’ll compare generating with absolute and ‘/‘ siteUrl and come back with more definite info.
> 
>>> - yarn workspace  —topological-dev is not building the UI before it’s used.  I can’t figure out why; it is for other PRs.  I replaced this by just calling the UI build directly.
>> 
>> That's odd, I did not notice this issue before, I'll try to reproduce
>> it and see if there is something we can do about this, this should be
>> one of the benefits in using workspaces.
> 
> It doesn’t seem to afflict other people’s PRs :-), but I have not yet had any build use the correct order with —topological-dev.  One run showing the problem is https://github.com/apache/camel-website/runs/3453859096?check_suite_focus=true <https://github.com/apache/camel-website/runs/3453859096?check_suite_focus=true>.  I wonder if it’s related to checking in the wrong .yarn contents: I think in this particular run I hadn’t checked in any .yarn changes, but later I have been checking in such changes and still get the wrong order.
>> 
>>> - If nothing else is using the model building in UpdateReadmeMojo it can probably be simplified further.
>> 
>> From what I understand we only need the JSON files generated now,
>> perhaps the we don't need the UpdateReadmeMojo at all any more?
> 
> Perhaps.  It still generates the doc header.  If Allow asciidoctor extensions when processing nav files and headers <https://gitlab.com/antora/antora/-/issues/592> were implemented, the header generation could be done by the jsonpath extension, but I’m not sure how much of an improvement that would be.  In any case, UpdateReadmeMojo is now more of a sanity checker than content generator :-).
>> 
>>> - The camel-website node dependencies have changed dramatically.  It looks like the yarn cache is checked into  git; I haven’t checked in the changes.  Should I?
>> 
> 
> Later on I started checking in .yarn, but I’m really not sure if I’m getting appropriate content.  I think that previously the UI was not in the main .yarn, but at least sometimes I get it, and it’s > 50 MB and I get a warning.
> 
> Note that I have 2 camel-website PRs: the “antora 3” one is the one that should be merged, and after the camel jsonpath-options PR is merged it will build with all the new stuff.  The other camel-website PR is just to build the jsonpath-options branch.
>> 
>>> known problems:
>>> 
>>> - When the first row contains a description with a list, the list isn’t rendered properly.  cf. https://pr-614--camel.netlify.app/components/latest/activemq-component.html#_path_parameters_2_parameters <https://pr-614--camel.netlify.app/components/latest/activemq-component.html#_path_parameters_2_parameters> <https://pr-614--camel.netlify.app/components/latest/activemq-component.html#_path_parameters_2_parameters <https://pr-614--camel.netlify.app/components/latest/activemq-component.html#_path_parameters_2_parameters>>
>> 
>> I guess this is fixed now, looks okay to me...
> 
> Yes, this is one of the many problems now fixed :-)
>> 
>>> - All the names in the generated tables have ids so they can be linked to but there’s no easy way to find out the id. The sections have a link to themselves which makes it easy to copy the URL.  I haven’t looked into whether this can be done with  the table entries.
>> 
>> I think this is a nice to have, so we can definitely merge this
>> without this and followup after...
>> 
>>> Updating the main PR to track changes in camel is somewhat time consuming so I’d appreciate it if we could move this towards resolution quickly.
>> 
>> Yes, I'm looking into this right now, my goal is to get this merged _very_ soon!
> 
> Thanks for the review!!
> 
> David Jencks
>> 
>> zoran
>> -- 
>> Zoran Regvart

Re: [Website] Possible change in components page generation (preview available)

Posted by David Jencks <da...@gmail.com>.
Hi Zoran,

I just pushed some more fixes, I hope this isn’t making reviewing too hard.

Starting far back in the email chain is probably pretty confusing, since I or others have fixed a lot of the problems noted.  I’ve tried to summarize all the current problems in each email: I think I should have also noted which ones got fixed.

Some comments inline.

> On Sep 7, 2021, at 4:42 AM, Zoran Regvart <zo...@regvart.com> wrote:
> 
> Hi David,
> working through the changes and emails, some more comments here...
> 
> On Fri, Sep 3, 2021 at 8:21 AM David Jencks <da...@gmail.com> wrote:
>> comments:
>> 
>> - I removed the patch-sitemap.js use.  I can’t figure out why this is needed.  If it really is needed, it should be done with a pipeline extension.
> 
> I think we'll need to restore this, otherwise we'll lose the sitemap
> generation which in turn leaves us with outdated links on the search
> engines, and as we change the links when we add or remove new
> documentation versions this becomes an issue.
> 

I’m rather confused by this since my experience and Dan’s repeated statements are that Antora always generates relative links within the site.  It’s possible to write the UI to generate either absolute or relative links, but that’s not something Antora is doing itself.  Getting Antora to generate absolute links in page bodies (not the UI) requires major surgery :-)  So, I’ll compare generating with absolute and ‘/‘ siteUrl and come back with more definite info.

>> - yarn workspace  —topological-dev is not building the UI before it’s used.  I can’t figure out why; it is for other PRs.  I replaced this by just calling the UI build directly.
> 
> That's odd, I did not notice this issue before, I'll try to reproduce
> it and see if there is something we can do about this, this should be
> one of the benefits in using workspaces.

It doesn’t seem to afflict other people’s PRs :-), but I have not yet had any build use the correct order with —topological-dev.  One run showing the problem is https://github.com/apache/camel-website/runs/3453859096?check_suite_focus=true.  I wonder if it’s related to checking in the wrong .yarn contents: I think in this particular run I hadn’t checked in any .yarn changes, but later I have been checking in such changes and still get the wrong order.
> 
>> - If nothing else is using the model building in UpdateReadmeMojo it can probably be simplified further.
> 
> From what I understand we only need the JSON files generated now,
> perhaps the we don't need the UpdateReadmeMojo at all any more?

Perhaps.  It still generates the doc header.  If Allow asciidoctor extensions when processing nav files and headers <https://gitlab.com/antora/antora/-/issues/592> were implemented, the header generation could be done by the jsonpath extension, but I’m not sure how much of an improvement that would be.  In any case, UpdateReadmeMojo is now more of a sanity checker than content generator :-).
> 
>> - The camel-website node dependencies have changed dramatically.  It looks like the yarn cache is checked into  git; I haven’t checked in the changes.  Should I?
> 

Later on I started checking in .yarn, but I’m really not sure if I’m getting appropriate content.  I think that previously the UI was not in the main .yarn, but at least sometimes I get it, and it’s > 50 MB and I get a warning.

Note that I have 2 camel-website PRs: the “antora 3” one is the one that should be merged, and after the camel jsonpath-options PR is merged it will build with all the new stuff.  The other camel-website PR is just to build the jsonpath-options branch.
> 
>> known problems:
>> 
>> - When the first row contains a description with a list, the list isn’t rendered properly.  cf. https://pr-614--camel.netlify.app/components/latest/activemq-component.html#_path_parameters_2_parameters <https://pr-614--camel.netlify.app/components/latest/activemq-component.html#_path_parameters_2_parameters>
> 
> I guess this is fixed now, looks okay to me...

Yes, this is one of the many problems now fixed :-)
> 
>> - All the names in the generated tables have ids so they can be linked to but there’s no easy way to find out the id. The sections have a link to themselves which makes it easy to copy the URL.  I haven’t looked into whether this can be done with  the table entries.
> 
> I think this is a nice to have, so we can definitely merge this
> without this and followup after...
> 
>> Updating the main PR to track changes in camel is somewhat time consuming so I’d appreciate it if we could move this towards resolution quickly.
> 
> Yes, I'm looking into this right now, my goal is to get this merged _very_ soon!

Thanks for the review!!

David Jencks
> 
> zoran
> -- 
> Zoran Regvart

Re: [Website] Possible change in components page generation (preview available)

Posted by Zoran Regvart <zo...@regvart.com>.
Hi David,
working through the changes and emails, some more comments here...

On Fri, Sep 3, 2021 at 8:21 AM David Jencks <da...@gmail.com> wrote:
> comments:
>
> - I removed the patch-sitemap.js use.  I can’t figure out why this is needed.  If it really is needed, it should be done with a pipeline extension.

I think we'll need to restore this, otherwise we'll lose the sitemap
generation which in turn leaves us with outdated links on the search
engines, and as we change the links when we add or remove new
documentation versions this becomes an issue.

> - yarn workspace  —topological-dev is not building the UI before it’s used.  I can’t figure out why; it is for other PRs.  I replaced this by just calling the UI build directly.

That's odd, I did not notice this issue before, I'll try to reproduce
it and see if there is something we can do about this, this should be
one of the benefits in using workspaces.

> - If nothing else is using the model building in UpdateReadmeMojo it can probably be simplified further.

From what I understand we only need the JSON files generated now,
perhaps the we don't need the UpdateReadmeMojo at all any more?

> - The camel-website node dependencies have changed dramatically.  It looks like the yarn cache is checked into  git; I haven’t checked in the changes.  Should I?

Yes, we have a check in the CI that makes sure the cache is correct.

> known problems:
>
> - When the first row contains a description with a list, the list isn’t rendered properly.  cf. https://pr-614--camel.netlify.app/components/latest/activemq-component.html#_path_parameters_2_parameters <https://pr-614--camel.netlify.app/components/latest/activemq-component.html#_path_parameters_2_parameters>

I guess this is fixed now, looks okay to me...

> - All the names in the generated tables have ids so they can be linked to but there’s no easy way to find out the id. The sections have a link to themselves which makes it easy to copy the URL.  I haven’t looked into whether this can be done with  the table entries.

I think this is a nice to have, so we can definitely merge this
without this and followup after...

> Updating the main PR to track changes in camel is somewhat time consuming so I’d appreciate it if we could move this towards resolution quickly.

Yes, I'm looking into this right now, my goal is to get this merged _very_ soon!

zoran
-- 
Zoran Regvart

Re: [Website] Possible change in components page generation (preview available)

Posted by David Jencks <da...@gmail.com>.
I think this work is complete enough to be used.  There’s now a Ci built preview at https://pr-614--camel.netlify.app

The main work is on https://github.com/apache/camel/pull/6040. Possibly the commits should be squashed into fewer.  Once this is merged the camel-website PR can be updated to use the normal repos/branches.

Subsidiary PRs fixing warnings that would now break the build: (these can be merged before the main work)

https://github.com/apache/camel-kamelets/pull/480 (no error from Antora). This is the bizarre looking Kamelets catalog page fix.  I have no idea why the previous adoc worked, but this works on both Antora 2 and 3

https://github.com/apache/camel-quarkus/pull/3064
https://github.com/apache/camel-quarkus/pull/3065

https://github.com/apache/camel/pull/6039

comments:

- I removed the patch-sitemap.js use.  I can’t figure out why this is needed.  If it really is needed, it should be done with a pipeline extension.
- yarn workspace  —topological-dev is not building the UI before it’s used.  I can’t figure out why; it is for other PRs.  I replaced this by just calling the UI build directly.
- If nothing else is using the model building in UpdateReadmeMojo it can probably be simplified further.
- The camel-website node dependencies have changed dramatically.  It looks like the yarn cache is checked into  git; I haven’t checked in the changes.  Should I?

known problems:

- When the first row contains a description with a list, the list isn’t rendered properly.  cf. https://pr-614--camel.netlify.app/components/latest/activemq-component.html#_path_parameters_2_parameters <https://pr-614--camel.netlify.app/components/latest/activemq-component.html#_path_parameters_2_parameters>
- All the names in the generated tables have ids so they can be linked to but there’s no easy way to find out the id. The sections have a link to themselves which makes it easy to copy the URL.  I haven’t looked into whether this can be done with  the table entries.

Updating the main PR to track changes in camel is somewhat time consuming so I’d appreciate it if we could move this towards resolution quickly.

David Jencks


> On Aug 31, 2021, at 11:42 PM, David Jencks <da...@gmail.com> wrote:
> 
> With  both the CI build for my PR and locally, `yarn build-all` is building the UI after the main build command.  Obviously this doesn’t work.
> 
> Can someone point to a situation in which the UI is built before it is used?  Any ideas why yarn is picking this dysfunctional workspace build order?
> 
> David Jencks
> 
>> On Aug 30, 2021, at 1:23 AM, Claus Ibsen <claus.ibsen@gmail.com <ma...@gmail.com>> wrote:
>> 
>> On Sun, Aug 29, 2021 at 10:20 AM David Jencks <david.a.jencks@gmail.com <ma...@gmail.com>> wrote:
>>> 
>>> The current source state is now visible.
>>> 
>>> - I npm-published my asciidoctor-jsonpath extension
>>> - The site content changes are at https://github.com/djencks/camel.git <https://github.com/djencks/camel.git> jsonpath-options branch. I’ve put all the generated changes as the last commit, so it should be possible to update the branch by dropping the last commit, rebasing on main, and regenerating the source with the maven build.
>>> - The camel-website changes are at https://github.com/djencks/camel-website.git <https://github.com/djencks/camel-website.git> issue-16854-jsonpath-options branch
>>> 
>>> There’s a PR https://github.com/apache/camel-website/pull/614 <https://github.com/apache/camel-website/pull/614>.
>>> 
>>> I don’t understand what the patch-sitemap.js does and was having dependency problems so removed it from the command line.  What does it do and why?
>>> 
>>> The PR’s build fails because of missing ui bundle.  Is this expected?  I like to have the built UI bundle available somewhere: I often check in the built bundle.  Locally I can’t build the UI, it complains somehow about the helpers for component sorting/hiding.
>>> 
>>> I set up the build to fail on warnings, and there are quite a few warnings.
>>> 
>>> There are several source problems I don’t know the proper fix for as it requires domain knowledge.  One is these warnings:
>>> 
>>> [00:47:14.821] WARN (asciidoctor): skipping reference to missing attribute: apisyntax
>>>    file: docs/components/modules/ROOT/pages/google-calendar-stream-component.adoc
>>>    source: https://github.com/djencks/camel.git <https://github.com/djencks/camel.git> (refname: jsonpath-options, start path: docs/components)
>>> [00:47:15.979] WARN (asciidoctor): skipping reference to missing attribute: apisyntax
>>>    file: docs/components/modules/ROOT/pages/google-mail-stream-component.adoc
>>>    source: https://github.com/djencks/camel.git <https://github.com/djencks/camel.git> (refname: jsonpath-options, start path: docs/components)
>>> [00:47:16.442] WARN (asciidoctor): skipping reference to missing attribute: apisyntax
>>>    file: docs/components/modules/ROOT/pages/google-sheets-stream-component.adoc
>>>    source: https://github.com/djencks/camel.git <https://github.com/djencks/camel.git> (refname: jsonpath-options, start path: docs/components)
>>> 
>>> 
>>> These three components are missing an apisyntax entry in their json files.  The problem shows up in the current site as the literal string ’null’.
>>> 
>> 
>> Ah yeah it looks like the -stream components are not API based, I am
>> going to fix this for Camel 3.12.
>> 
>> 
>>> There are some inconsistencies and mysteries in the gulpfile.js. I’ve commented on some.
>>> 
>>> It would be great to know if anyone else can build the site from the PR branch.
>>> 
>>> David Jencks
>>> 
>>>> On Aug 23, 2021, at 6:00 AM, Claus Ibsen <claus.ibsen@gmail.com <ma...@gmail.com>> wrote:
>>>> 
>>>> Hi
>>>> 
>>>> Ah yeah those were an idea to include the full page documentation in
>>>> case tooling may be able to use that for something useable.
>>>> However the tooling uses all the other bits, so we have just marked
>>>> those apis as deprecated.
>>>> 
>>>> So we can remove the files from the camel-catalog.
>>>> I have created a ticket to remove them
>>>> https://issues.apache.org/jira/browse/CAMEL-16881 <https://issues.apache.org/jira/browse/CAMEL-16881>
>>>> 
>>>> On Sun, Aug 22, 2021 at 9:48 AM Zoran Regvart <zo...@regvart.com> wrote:
>>>>> 
>>>>> Hi David,
>>>>> 
>>>>> On Sat, Aug 21, 2021 at 10:10 PM David Jencks <da...@gmail.com> wrote:
>>>>>> 
>>>>>> I have a question about the purpose of the .adoc files in the catalog.  The changes proposed here will remove the tables of options from these copies of the component .adoc files.  These copies are already quite skimpy as they don’t successfully include the spring-boot information.
>>>>>> 
>>>>>> What are these copies of the .adoc files supposed to be useful for?
>>>>> 
>>>>> I think the catalog contains those so that an alternative UI can show
>>>>> them, think tooltips or inline help in an IDE. I think, though I'm not
>>>>> 100% sure that VSCode tooling is using that...
>>>>> 
>>>>>> If there’s a desire to make them more complete, one strategy would be to build the components module of the website using a custom UI that has nothing in it, so we just get plain undecorated html pages, and putting those in the catalog.  It would require some investigation, but it might conceivably be possible to arrange so links out of the components module go to the website rather than just be broken.
>>>>> 
>>>>> I'd keep it as simple as it can be, so plain .adoc files, or if we
>>>>> find out they're not used just remove them...
>>>>> 
>>>>> 2c
>>>>> 
>>>>> zoran
>>>>> --
>>>>> Zoran Regvart
>>>> 
>>>> 
>>>> 
>>>> --
>>>> Claus Ibsen
>>>> -----------------
>>>> http://davsclaus.com @davsclaus
>>>> Camel in Action 2: https://www.manning.com/ibsen2
>>> 
>> 
>> 
>> -- 
>> Claus Ibsen
>> -----------------
>> http://davsclaus.com <http://davsclaus.com/> @davsclaus
>> Camel in Action 2: https://www.manning.com/ibsen2 <https://www.manning.com/ibsen2>

Re: [Website] Possible change in components page generation (preview available)

Posted by David Jencks <da...@gmail.com>.
FWIW all the changes I’ve proposed to repos other than camel main and camel-website have already been merged.  In any case this is an excellent strategy :-)

David Jencks

> On Sep 8, 2021, at 5:59 AM, Zoran Regvart <zo...@regvart.com> wrote:
> 
> Hi David & Cameleers,
> I've decided to pick smaller changes into separate PRs, starting first
> with the camel-website ones, I think we can get those merged and then
> piecemeal merge changes to the non-website repositories. Reason behind
> this is to try to scope the changes being merged, allowing for easier
> review and revert if things go wrong. And I think we can make changes
> to the camel-website first as those will have little effect if the
> jsonpath macro is not used (i.e. camel#6040 is not merged).
> 
> First up is the Antora 3.0.0-alpha9 upgrade that removes the xref
> checker and fails the build for any warnings logged (including broken
> xref links):
> 
> https://github.com/apache/camel-website/pull/620
> 
> Please have a look, I intend to do a PR for the
> @djencks/asciidoctor-jsonpath bits from #617 next based on #620 next.
> 
> Happy to receive feedback here or on the PRs...
> 
> zoran
> -- 
> Zoran Regvart

Re: [Website] Possible change in components page generation (preview available)

Posted by Zoran Regvart <zo...@regvart.com>.
Hi David & Cameleers,
I've decided to pick smaller changes into separate PRs, starting first
with the camel-website ones, I think we can get those merged and then
piecemeal merge changes to the non-website repositories. Reason behind
this is to try to scope the changes being merged, allowing for easier
review and revert if things go wrong. And I think we can make changes
to the camel-website first as those will have little effect if the
jsonpath macro is not used (i.e. camel#6040 is not merged).

First up is the Antora 3.0.0-alpha9 upgrade that removes the xref
checker and fails the build for any warnings logged (including broken
xref links):

https://github.com/apache/camel-website/pull/620

Please have a look, I intend to do a PR for the
@djencks/asciidoctor-jsonpath bits from #617 next based on #620 next.

Happy to receive feedback here or on the PRs...

zoran
-- 
Zoran Regvart

Re: [Website] Possible change in components page generation (preview available)

Posted by Zoran Regvart <zo...@regvart.com>.
Hi David, and Cameleers trying to follow this while David and myself geek out :)

On Tue, Sep 7, 2021 at 1:12 PM Zoran Regvart <zo...@regvart.com> wrote:
> On Wed, Sep 1, 2021 at 8:42 AM David Jencks <da...@gmail.com> wrote:
> >
> > With  both the CI build for my PR and locally, `yarn build-all` is building the UI after the main build command.  Obviously this doesn’t work.
>
> Can you point to the CI build that does that so I can have a look?
>
> > Can someone point to a situation in which the UI is built before it is used?  Any ideas why yarn is picking this dysfunctional workspace build order?
>
> The `build-all` runs `yarn workspaces foreach --topological-dev run
> build`, the `--topological-dev`[1] should guarantee proper order given
> that the `antora-ui-camel` is a devDependency of
> `apache-camel-website`. Or am I missing something?

On my end this was fixed by reverting this change:

https://github.com/apache/camel-website/pull/617/commits/d70796f3b680e9986ca149444d35c163e01ff9de#diff-7ae45ad102eab3b6d7e7896acd08c427a9b25b346470d7bc6507b6481575d519L36

i.e. for the `--topological-dev` to function the `antora-ui-camel`
dependency needs to be `workspace:antora-ui-camel` not
`./antora-ui-camel`, the project relative range is experimental and
folk are advised not to use it at the moment[1]

zoran
[1] https://yarnpkg.com/features/workspaces/#workspace-ranges-workspace
-- 
Zoran Regvart

Re: [Website] Possible change in components page generation (preview available)

Posted by Zoran Regvart <zo...@regvart.com>.
Hi David,

On Wed, Sep 1, 2021 at 8:42 AM David Jencks <da...@gmail.com> wrote:
>
> With  both the CI build for my PR and locally, `yarn build-all` is building the UI after the main build command.  Obviously this doesn’t work.

Can you point to the CI build that does that so I can have a look?

> Can someone point to a situation in which the UI is built before it is used?  Any ideas why yarn is picking this dysfunctional workspace build order?

The `build-all` runs `yarn workspaces foreach --topological-dev run
build`, the `--topological-dev`[1] should guarantee proper order given
that the `antora-ui-camel` is a devDependency of
`apache-camel-website`. Or am I missing something?

zoran

[1] https://yarnpkg.com/cli/workspaces/foreach#options-topological-dev
-- 
Zoran Regvart