You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@systemml.apache.org by "Baunsgaard, Sebastian" <ba...@tugraz.at> on 2020/06/04 10:44:28 UTC
Re: [DISCUSSION] Website stack `systemml.apache.org`. Thanks.
Hi dev team!
Update / suggestion.
First of all, i think it is great that we have people working on our documentation. It is really important. But I can't help to note that the improvements, are going in an unmaintainable direction where we have to maintain the documentation at multiple locations.
For instance our new toOneHot function already had some basic documentation inside its build-in script already [1], it would be much nicer if we modified that and then had some automation to generate the docs made in commit [2]. Now we have to maintain two locations, resulting in high risk of either being outdated.
We need to ask ourselves where do we want to make the edit of documentation. I would argue that it should be located as close to the internal definition of a function, because then when making or modifying that function the documentation is also reviewed.
To achieve this i suggest that we instead relocate the documentation source and generation to an appropriately named file inside [3], that simply loops through all function definitions from [4], and generate markdown files for our /docs/ folder.
Shifting to such an implementation enables us to systematically cover all functions defined in the system. It also is a full list of the systems current build-in functions thereby giving us the exact file paths to extract the source documentation from the comments and generate the docs for those files.
In the future it could be extended to verify and detecting which input and output types each function definition has, such that we have even less manual documentation needs.
Best regards
Sebastian
[1] https://github.com/apache/systemml/blob/master/scripts/builtin/toOneHot.dml
[2] https://github.com/apache/systemml/commit/859ad3a72906c67b7300c9980251da4cde9ed8f8
[3] https://github.com/apache/systemml/tree/master/src/main/java/org/apache/sysds/common
[4] https://github.com/apache/systemml/blob/master/src/main/java/org/apache/sysds/common/Builtins.java
________________________________
From: arnab phani <ph...@gmail.com>
Sent: Sunday, May 31, 2020 3:15:44 PM
To: dev@systemml.apache.org
Subject: Re: [DISCUSSION] Website stack `systemml.apache.org`. Thanks.
Hi,
Another advantage of markdown syntax is that it gives more freedom to add a
description as needed. Java methods and builtins might not fall in the same
bucket and different builtins might need different ways to describe it.
For example, glm needs more details than other built-ins. Too much
standardization can be a bad idea.
Regards,
Arnab.
On Sun, May 31, 2020 at 9:55 AM Baunsgaard, Sebastian <ba...@tugraz.at>
wrote:
> Hi Janardhan,
>
>
> That would be one option, lets call it option 1, I have another suggestion:
>
>
> 1. Markdown syntax with out commented individual lines.
> 2. Jdoc like syntax with annotations
>
> Pros and cons
>
> * 1 Pros
> * Easy Syntax most ppl know it.
> * Easy to implement.
> * 1 Cons
> * Enables custom docs (everyone make their own distinct format
> again)
> * No standard way of verifying correctness of the docs (correct
> parameter names etc.)
> * No standard way of marking Input, return parameters.
> * Requires modified Syntax with # at each line. where # normally
> means header in markdown
> * 2 Pros
> * Easy Syntax Java ppl know it
> * Standard way of making input
> * Do not have to change any syntax at all from Jdoc
> * Already supported syntax in DML
> * 2 Cons
> * Harder to implement since one would have to look into java doc
> extraction and what is needed to support that, maybe we have to make such a
> thing ourselves?
>
> I personally like option 2 with Jdoc more and then extending into
> automatically parsing and making the either markdown files or HTML files
> for the docs that you either way have to do in option 1.
>
> Best regards
> Sebastian
> ________________________________
> From: Janardhan <ja...@gmail.com>
> Sent: Sunday, May 31, 2020 9:25:43 AM
> To: dev@systemml.apache.org
> Subject: Re: [DISCUSSION] Website stack `systemml.apache.org`. Thanks.
>
> Hi Sebastian, :)
>
> Since the builtin DML files do not have full documentation yet, Can we
> write
> markdown syntax with the following heading in each dml file itself!
>
> 1. Function description
> 2. Usage
> 3. Arguments
> 4. Returns
> 5. Usage
>
> After that, as you've mentioned can be parsed with only removing # at the
> start of each line, and write
> all this data to one `builtins-reference.md` file.
>
> Thank you,
> Janardhan
>
> On Tue, May 26, 2020 at 5:05 PM Baunsgaard, Sebastian <
> baunsgaard@tugraz.at>
> wrote:
>
> >
> > If this is meant as a question then, yes from me (this probably calls for
> > a vote?), if:
> >
> >
> > 1. someone is able to change the settings for the repository, to use
> > the docs folder as the GitHub docs folder.
> > 2. someone wants to make a PR containing the content of the gh-pages
> > branch.
> >
> > Best regards
> > Sebastian
> > ________________________________
> > From: Janardhan <ja...@gmail.com>
> > Sent: Tuesday, May 26, 2020 1:22:11 PM
> > To: dev@systemml.apache.org
> > Subject: Re: [DISCUSSION] Website stack `systemml.apache.org`. Thanks.
> >
> > Hi Sebastian,
> >
> > Can we move the `docs` folder in the existing `master` branch
> > and merge the `gh-pages` branch to `docs` folder in the top-level
> > directory. :)
> >
> > - Janardhan
> >
> > On Sun, May 24, 2020 at 3:28 PM Baunsgaard, Sebastian <
> > baunsgaard@tugraz.at>
> > wrote:
> >
> > > Hi Janardhan,
> > >
> > >
> > > Reworking the documentation seems like a really good idea!
> > >
> > >
> > > A first step in my opinion is to start in the main repository.
> > >
> > >
> > > I would suggest the following:
> > >
> > >
> > > - Merge the current gh-pages branch into master docs folder.
> > >
> > > - Change the documentations page : http://apache.github.io/systemml/
SystemDS Documentation - SystemDS<http://apache.github.io/systemml/>
apache.github.io
SystemDS Documentation
> to
> > > use the docs folder for documentation. Should be doable inside the
> > settings
> > > on the repository.
> > >
> > > - Delete the gh-pages branch
> > >
> > >
> > > I want this change because then new PRs can change the documentation in
> > > one PR rather than two (which to be honest is not going to happen most
> of
> > > the time).
> > >
> > > The downside (we have to mention this) is that when you clone the
> > > repository, you have slightly larger downloads, but i think the
> trade-off
> > > is fair. If you see any other issues please mention these in this
> thread.
> > >
> > >
> > > Next task I would like to suggest to include:
> > >
> > >
> > > - Java docs generated and
> > >
> > > - Python docs generated to also be included as a sub-folder in docs, so
> > > that the current webpage contains the docs for that to.
> > >
> > >
> > > How to include these in a sensible manner is currently a good question,
> > > and has to be seriously well considered. Including these will also
> enable
> > > us to move documentation closer to the individual code parts, making it
> > > again more likely to get the documentation done.
> > >
> > >
> > > Another more programmatic task is to make a parser for our DML files,
> > such
> > > that we can generate webpage documentation based on the documentation
> > > syntax used (much like the python and java docs generator). This would
> be
> > > great since it will remove the duplication of documentation in the
> > > individual DML files, and the documentation we have already in files
> such
> > > as: docs/dml-language-reference.md.
> > >
> > > This also gives us an excuse to clean up that documentation/scripts
> which
> > > has very different implementations in our scripts folder:
> > >
> > >
> > > examples:
> > >
> > >
> > > - <https://github.com/apache/systemml/tree/master/scripts>
[https://avatars3.githubusercontent.com/u/47359?s=400&v=4]<https://github.com/apache/systemml/tree/master/scripts>
systemml/scripts at master · apache/systemml · GitHub<https://github.com/apache/systemml/tree/master/scripts>
github.com
Mirror of Apache SystemML. Contribute to apache/systemml development by creating an account on GitHub.
> > >
> >
> https://github.com/apache/systemml/blob/master/scripts/nn/layers/batch_norm1d.dml
[https://avatars3.githubusercontent.com/u/47359?s=400&v=4]<https://github.com/apache/systemml/blob/master/scripts/nn/layers/batch_norm1d.dml>
systemml/batch_norm1d.dml at master · apache/systemml · GitHub<https://github.com/apache/systemml/blob/master/scripts/nn/layers/batch_norm1d.dml>
github.com
Mirror of Apache SystemML. Contribute to apache/systemml development by creating an account on GitHub.
> > > <
> > >
> >
> https://github.com/Baunsgaard/systemml/blob/master/scripts/nn/layers/batch_norm2d.dml
[https://avatars2.githubusercontent.com/u/9947148?s=400&v=4]<https://github.com/Baunsgaard/systemml/blob/master/scripts/nn/layers/batch_norm2d.dml>
systemml/batch_norm2d.dml at master · Baunsgaard/systemml · GitHub<https://github.com/Baunsgaard/systemml/blob/master/scripts/nn/layers/batch_norm2d.dml>
github.com
Mirror of Apache SystemML. Contribute to Baunsgaard/systemml development by creating an account on GitHub.
> > > >
> > >
> > > - <
> > >
> >
> https://github.com/Baunsgaard/systemml/blob/master/scripts/builtin/glm.dml
[https://avatars2.githubusercontent.com/u/9947148?s=400&v=4]<https://github.com/Baunsgaard/systemml/blob/master/scripts/builtin/glm.dml>
systemml/glm.dml at master · Baunsgaard/systemml · GitHub<https://github.com/Baunsgaard/systemml/blob/master/scripts/builtin/glm.dml>
github.com
Mirror of Apache SystemML. Contribute to Baunsgaard/systemml development by creating an account on GitHub.
> > >
> > > <https://github.com/apache/systemml/tree/master/scripts>
[https://avatars3.githubusercontent.com/u/47359?s=400&v=4]<https://github.com/apache/systemml/tree/master/scripts>
systemml/scripts at master · apache/systemml · GitHub<https://github.com/apache/systemml/tree/master/scripts>
github.com
Mirror of Apache SystemML. Contribute to apache/systemml development by creating an account on GitHub.
> > >
> >
> https://github.com/apache/systemml/blob/master/scripts/algorithms/Cox.dml
[https://avatars3.githubusercontent.com/u/47359?s=400&v=4]<https://github.com/apache/systemml/blob/master/scripts/algorithms/Cox.dml>
systemml/Cox.dml at master · apache/systemml · GitHub<https://github.com/apache/systemml/blob/master/scripts/algorithms/Cox.dml>
github.com
Mirror of Apache SystemML. Contribute to apache/systemml development by creating an account on GitHub.
> > >
> > >
> > >
> > > Furthermore, I would suggest to postpone changing the main website:
> > >
> > > https://github.com/apache/systemml-website
[https://avatars3.githubusercontent.com/u/47359?s=400&v=4]<https://github.com/apache/systemml-website>
GitHub - apache/systemml-website: Mirror of Apache SystemML site<https://github.com/apache/systemml-website>
github.com
Mirror of Apache SystemML site. Contribute to apache/systemml-website development by creating an account on GitHub.
> > >
> > > But that task has to be started when the next release is scheduled,
> > > because then we need to find out how to copy the current documentation
> > to a
> > > archive list of static webpages located:
> > >
> > > https://systemml.apache.org/documentation
Documentation<https://systemml.apache.org/documentation>
systemml.apache.org
Project Documentation Page
> > >
> > >
> > > best regards
> > >
> > > Sebastian
> > >
> > > ________________________________
> > > From: Janardhan <ja...@apache.org>
> > > Sent: Sunday, May 24, 2020 11:02:53 AM
> > > To: dev@systemml.apache.org
> > > Subject: [DISCUSSION] Website stack `systemml.apache.org`. Thanks.
> > >
> > > Hi Sebastian,
> > >
> > > In our discussion[1][2] a while ago, there was a topic about changing
> > > website
> > > stack.
> > >
> > > Let us start a discussion about this in this thread.
> > >
> > > We are planning to work on this, after a feasibility study and tech
> stack
> > > vetting.
> > >
> > > Anybody would like to give suggestions would be great.
> > >
> > > [1] https://github.com/apache/systemml/pull/877
[https://avatars3.githubusercontent.com/u/47359?s=400&v=4]<https://github.com/apache/systemml/pull/877>
[MINOR][DOC] Name refactor SystemML to SystemDS in Documentation by Baunsgaard · Pull Request #877 · apache/systemml · GitHub<https://github.com/apache/systemml/pull/877>
github.com
Brute change of documentation page to reflect name change. To fully enforce this two more steps are needed: https://github.com/apache/systemml-website Access to SystemML website server on https://systemml.apache.org/ Furthermore, we might want to consider starting fresh in the documentation.
> > > [2] https://github.com/apache/systemml-website/pull/66
[https://avatars3.githubusercontent.com/u/47359?s=400&v=4]<https://github.com/apache/systemml-website/pull/66>
[SYSTEMML-2539][BUILD] Upgrade build configuration by j143 · Pull Request #66 · apache/systemml-website · GitHub<https://github.com/apache/systemml-website/pull/66>
github.com
Upgrade gulp to 4.0 Since the task api is deprecated, fuctions were used, and exported as tasks. Documentation for gulpfile.js. Stick to Susy2, since mixins are not supported in 3.0 Upgrade npm dependencies to state-of-art. Commit package-lock.json file to version control.
> > >
> > > Thank you,
> > > Janardhan
> > >
> >
>
Re: [DISCUSSION] Website stack `systemml.apache.org`. Thanks.
Posted by Janardhan <ja...@gmail.com>.
Anyways the changeset looks good to me. Including the pyhton api, and java.
- Janardhan
On Friday, June 19, 2020, Matthias Boehm <mb...@gmail.com> wrote:
> don't worry about - it's all fine. Leaving it like that (for a single
> commit) is MUCH better than trying to forcefully rewrite the history.
>
> Regards,
> Matthias
>
> On 6/18/2020 5:35 PM, Baunsgaard, Sebastian wrote:
>
>> Hi Dev,
>>
>>
>> I merged in the documentation PR, this was an slight oversight, while
>> trying to merge something else. That said nothing bad happened except you
>> did not get a chance to look through the PR, and it is progress in the
>> direction discussed in this thread.
>>
>>
>> Best regards
>>
>> Sebastian
>>
>>
>> ________________________________
>> From: Matthias Boehm <mb...@gmail.com>
>> Sent: Saturday, June 6, 2020 7:03:19 PM
>> To: dev@systemml.apache.org
>> Subject: Re: [DISCUSSION] Website stack `systemml.apache.org`. Thanks.
>>
>> from my perspective it would be very important to have all builtin
>> functions in a single markdown file to allow users to search for things
>> and users don't care how a builtin function is internally implemented.
>> So we might want to use this opportunity to consolidate the already
>> documented builtin functions into the new list and add everything that
>> is not yet documented as I'm sure many functions are missing.
>>
>> Btw, in the INFRA ticket for renaming our repositories [1], I also asked
>> to disable sending all these github comments and PR interactions to our
>> dev list as I'm sure people get overwhelmed by these auto-generated
>> emails which leads to important messages being missed. Related to
>> documentation, we might want to group categories of builtin functions
>> into larger PRs instead of having a PR per builtin function as its
>> primarily a copy of existing documentation so they can be reviewed more
>> efficiently together to avoid unnecessary overhead.
>>
>> [1] https://issues.apache.org/jira/browse/INFRA-20362
>>
>> Regards,
>> Matthias
>>
>> On 6/6/2020 5:23 PM, Janardhan wrote:
>>
>>> Hi Arnab,
>>>
>>> We have all the Java-based builtin functions documented at [1].
>>> Most of these functions are already documented!
>>>
>>> [1]
>>> http://apache.github.io/systemml/dml-language-reference.
>>> html#built-in-functions
>>>
>>>
>>> Thank you,
>>> Janardhan
>>>
>>> On Fri, Jun 5, 2020 at 4:37 PM arnab phani <ph...@gmail.com> wrote:
>>>
>>> That's a good suggestion. One thing I want to point out that
>>>> Builtins.java
>>>> [1] contains all the dml-bodied and java-bodied built-ins. For the dml
>>>> built-ins, it is easier to parse the corresponding dml script (if
>>>> documentation is available there), but for other built-ins, I am not
>>>> aware
>>>> of any particular source today where they are documented -- a few of
>>>> them
>>>> are documented as a part of commit message but not all.
>>>> Though it is certainly possible to slice the dml builtins from [1] out,
>>>> but
>>>> it will be awesome to generate docs for all of those.
>>>>
>>>> [1]
>>>>
>>>> https://github.com/apache/systemml/blob/master/src/main/java
>>>> /org/apache/sysds/common/Builtins.java
>>>>
>>>> Regards,
>>>> Arnab..
>>>>
>>>> On Thu, Jun 4, 2020 at 12:44 PM Baunsgaard, Sebastian <
>>>> baunsgaard@tugraz.at>
>>>> wrote:
>>>>
>>>> Hi dev team!
>>>>>
>>>>>
>>>>> Update / suggestion.
>>>>>
>>>>>
>>>>> First of all, i think it is great that we have people working on our
>>>>> documentation. It is really important. But I can't help to note that
>>>>> the
>>>>> improvements, are going in an unmaintainable direction where we have to
>>>>> maintain the documentation at multiple locations.
>>>>>
>>>>>
>>>>> For instance our new toOneHot function already had some basic
>>>>> documentation inside its build-in script already [1], it would be much
>>>>> nicer if we modified that and then had some automation to generate the
>>>>>
>>>> docs
>>>>
>>>>> made in commit [2]. Now we have to maintain two locations, resulting in
>>>>> high risk of either being outdated.
>>>>>
>>>>>
>>>>> We need to ask ourselves where do we want to make the edit of
>>>>> documentation. I would argue that it should be located as close to the
>>>>> internal definition of a function, because then when making or
>>>>> modifying
>>>>> that function the documentation is also reviewed.
>>>>>
>>>>>
>>>>> To achieve this i suggest that we instead relocate the documentation
>>>>> source and generation to an appropriately named file inside [3], that
>>>>> simply loops through all function definitions from [4], and generate
>>>>> markdown files for our /docs/ folder.
>>>>>
>>>>>
>>>>> Shifting to such an implementation enables us to systematically cover
>>>>> all
>>>>> functions defined in the system. It also is a full list of the systems
>>>>> current build-in functions thereby giving us the exact file paths to
>>>>> extract the source documentation from the comments and generate the
>>>>> docs
>>>>> for those files.
>>>>>
>>>>> In the future it could be extended to verify and detecting which input
>>>>>
>>>> and
>>>>
>>>>> output types each function definition has, such that we have even less
>>>>> manual documentation needs.
>>>>>
>>>>>
>>>>> Best regards
>>>>>
>>>>> Sebastian
>>>>>
>>>>>
>>>>> [1]
>>>>>
>>>>> https://github.com/apache/systemml/blob/master/scripts/built
>>>> in/toOneHot.dml
>>>>
>>>>> [2]
>>>>>
>>>>> https://github.com/apache/systemml/commit/859ad3a72906c67b73
>>>> 00c9980251da4cde9ed8f8
>>>>
>>>>> [3]
>>>>>
>>>>> https://github.com/apache/systemml/tree/master/src/main/java
>>>> /org/apache/sysds/common
>>>>
>>>>> [4]
>>>>>
>>>>> https://github.com/apache/systemml/blob/master/src/main/java
>>>> /org/apache/sysds/common/Builtins.java
>>>>
>>>>>
>>>>> ________________________________
>>>>> From: arnab phani <ph...@gmail.com>
>>>>> Sent: Sunday, May 31, 2020 3:15:44 PM
>>>>> To: dev@systemml.apache.org
>>>>> Subject: Re: [DISCUSSION] Website stack `systemml.apache.org`. Thanks.
>>>>>
>>>>> Hi,
>>>>>
>>>>> Another advantage of markdown syntax is that it gives more freedom to
>>>>>
>>>> add a
>>>>
>>>>> description as needed. Java methods and builtins might not fall in the
>>>>>
>>>> same
>>>>
>>>>> bucket and different builtins might need different ways to describe it.
>>>>> For example, glm needs more details than other built-ins. Too much
>>>>> standardization can be a bad idea.
>>>>>
>>>>> Regards,
>>>>> Arnab.
>>>>>
>>>>> On Sun, May 31, 2020 at 9:55 AM Baunsgaard, Sebastian <
>>>>> baunsgaard@tugraz.at>
>>>>> wrote:
>>>>>
>>>>> Hi Janardhan,
>>>>>>
>>>>>>
>>>>>> That would be one option, lets call it option 1, I have another
>>>>>>
>>>>> suggestion:
>>>>>
>>>>>>
>>>>>> 1. Markdown syntax with out commented individual lines.
>>>>>> 2. Jdoc like syntax with annotations
>>>>>>
>>>>>> Pros and cons
>>>>>>
>>>>>> * 1 Pros
>>>>>> * Easy Syntax most ppl know it.
>>>>>> * Easy to implement.
>>>>>> * 1 Cons
>>>>>> * Enables custom docs (everyone make their own distinct
>>>>>> format
>>>>>> again)
>>>>>> * No standard way of verifying correctness of the docs
>>>>>> (correct
>>>>>> parameter names etc.)
>>>>>> * No standard way of marking Input, return parameters.
>>>>>> * Requires modified Syntax with # at each line. where #
>>>>>> normally
>>>>>> means header in markdown
>>>>>> * 2 Pros
>>>>>> * Easy Syntax Java ppl know it
>>>>>> * Standard way of making input
>>>>>> * Do not have to change any syntax at all from Jdoc
>>>>>> * Already supported syntax in DML
>>>>>> * 2 Cons
>>>>>> * Harder to implement since one would have to look into java
>>>>>> doc
>>>>>> extraction and what is needed to support that, maybe we have to make
>>>>>>
>>>>> such a
>>>>>
>>>>>> thing ourselves?
>>>>>>
>>>>>> I personally like option 2 with Jdoc more and then extending into
>>>>>> automatically parsing and making the either markdown files or HTML
>>>>>>
>>>>> files
>>>>
>>>>> for the docs that you either way have to do in option 1.
>>>>>>
>>>>>> Best regards
>>>>>> Sebastian
>>>>>> ________________________________
>>>>>> From: Janardhan <ja...@gmail.com>
>>>>>> Sent: Sunday, May 31, 2020 9:25:43 AM
>>>>>> To: dev@systemml.apache.org
>>>>>> Subject: Re: [DISCUSSION] Website stack `systemml.apache.org`.
>>>>>> Thanks.
>>>>>>
>>>>>> Hi Sebastian, :)
>>>>>>
>>>>>> Since the builtin DML files do not have full documentation yet, Can we
>>>>>> write
>>>>>> markdown syntax with the following heading in each dml file itself!
>>>>>>
>>>>>> 1. Function description
>>>>>> 2. Usage
>>>>>> 3. Arguments
>>>>>> 4. Returns
>>>>>> 5. Usage
>>>>>>
>>>>>> After that, as you've mentioned can be parsed with only removing # at
>>>>>>
>>>>> the
>>>>
>>>>> start of each line, and write
>>>>>> all this data to one `builtins-reference.md` file.
>>>>>>
>>>>>> Thank you,
>>>>>> Janardhan
>>>>>>
>>>>>> On Tue, May 26, 2020 at 5:05 PM Baunsgaard, Sebastian <
>>>>>> baunsgaard@tugraz.at>
>>>>>> wrote:
>>>>>>
>>>>>> If this is meant as a question then, yes from me (this probably calls
>>>>>>>
>>>>>> for
>>>>>
>>>>>> a vote?), if:
>>>>>>>
>>>>>>>
>>>>>>> 1. someone is able to change the settings for the repository, to
>>>>>>>
>>>>>> use
>>>>
>>>>> the docs folder as the GitHub docs folder.
>>>>>>> 2. someone wants to make a PR containing the content of the
>>>>>>>
>>>>>> gh-pages
>>>>
>>>>> branch.
>>>>>>>
>>>>>>> Best regards
>>>>>>> Sebastian
>>>>>>> ________________________________
>>>>>>> From: Janardhan <ja...@gmail.com>
>>>>>>> Sent: Tuesday, May 26, 2020 1:22:11 PM
>>>>>>> To: dev@systemml.apache.org
>>>>>>> Subject: Re: [DISCUSSION] Website stack `systemml.apache.org`.
>>>>>>>
>>>>>> Thanks.
>>>>
>>>>> Hi Sebastian,
>>>>>>>
>>>>>>> Can we move the `docs` folder in the existing `master` branch
>>>>>>> and merge the `gh-pages` branch to `docs` folder in the top-level
>>>>>>> directory. :)
>>>>>>>
>>>>>>> - Janardhan
>>>>>>>
>>>>>>> On Sun, May 24, 2020 at 3:28 PM Baunsgaard, Sebastian <
>>>>>>> baunsgaard@tugraz.at>
>>>>>>> wrote:
>>>>>>>
>>>>>>> Hi Janardhan,
>>>>>>>>
>>>>>>>>
>>>>>>>> Reworking the documentation seems like a really good idea!
>>>>>>>>
>>>>>>>>
>>>>>>>> A first step in my opinion is to start in the main repository.
>>>>>>>>
>>>>>>>>
>>>>>>>> I would suggest the following:
>>>>>>>>
>>>>>>>>
>>>>>>>> - Merge the current gh-pages branch into master docs folder.
>>>>>>>>
>>>>>>>> - Change the documentations page :
>>>>>>>>
>>>>>>> http://apache.github.io/systemml/
>>>>
>>>>> SystemDS Documentation - SystemDS<http://apache.github.io/systemml/>
>>>>> apache.github.io
>>>>> SystemDS Documentation
>>>>>
>>>>>
>>>>> to
>>>>>>
>>>>>>> use the docs folder for documentation. Should be doable inside the
>>>>>>>>
>>>>>>> settings
>>>>>>>
>>>>>>>> on the repository.
>>>>>>>>
>>>>>>>> - Delete the gh-pages branch
>>>>>>>>
>>>>>>>>
>>>>>>>> I want this change because then new PRs can change the
>>>>>>>>
>>>>>>> documentation
>>>>
>>>>> in
>>>>>
>>>>>> one PR rather than two (which to be honest is not going to happen
>>>>>>>>
>>>>>>> most
>>>>>
>>>>>> of
>>>>>>
>>>>>>> the time).
>>>>>>>>
>>>>>>>> The downside (we have to mention this) is that when you clone the
>>>>>>>> repository, you have slightly larger downloads, but i think the
>>>>>>>>
>>>>>>> trade-off
>>>>>>
>>>>>>> is fair. If you see any other issues please mention these in this
>>>>>>>>
>>>>>>> thread.
>>>>>>
>>>>>>>
>>>>>>>> Next task I would like to suggest to include:
>>>>>>>>
>>>>>>>>
>>>>>>>> - Java docs generated and
>>>>>>>>
>>>>>>>> - Python docs generated to also be included as a sub-folder in
>>>>>>>>
>>>>>>> docs,
>>>>
>>>>> so
>>>>>
>>>>>> that the current webpage contains the docs for that to.
>>>>>>>>
>>>>>>>>
>>>>>>>> How to include these in a sensible manner is currently a good
>>>>>>>>
>>>>>>> question,
>>>>>
>>>>>> and has to be seriously well considered. Including these will also
>>>>>>>>
>>>>>>> enable
>>>>>>
>>>>>>> us to move documentation closer to the individual code parts,
>>>>>>>>
>>>>>>> making
>>>>
>>>>> it
>>>>>
>>>>>> again more likely to get the documentation done.
>>>>>>>>
>>>>>>>>
>>>>>>>> Another more programmatic task is to make a parser for our DML
>>>>>>>>
>>>>>>> files,
>>>>
>>>>> such
>>>>>>>
>>>>>>>> that we can generate webpage documentation based on the
>>>>>>>>
>>>>>>> documentation
>>>>
>>>>> syntax used (much like the python and java docs generator). This
>>>>>>>>
>>>>>>> would
>>>>>
>>>>>> be
>>>>>>
>>>>>>> great since it will remove the duplication of documentation in the
>>>>>>>> individual DML files, and the documentation we have already in
>>>>>>>>
>>>>>>> files
>>>>
>>>>> such
>>>>>>
>>>>>>> as: docs/dml-language-reference.md.
>>>>>>>>
>>>>>>>> This also gives us an excuse to clean up that documentation/scripts
>>>>>>>>
>>>>>>> which
>>>>>>
>>>>>>> has very different implementations in our scripts folder:
>>>>>>>>
>>>>>>>>
>>>>>>>> examples:
>>>>>>>>
>>>>>>>>
>>>>>>>> - <https://github.com/apache/systemml/tree/master/scripts>
>>>>>>>>
>>>>>>> [https://avatars3.githubusercontent.com/u/47359?s=400&v=4]<
>>>>> https://github.com/apache/systemml/tree/master/scripts>
>>>>>
>>>>> systemml/scripts at master · apache/systemml · GitHub<
>>>>> https://github.com/apache/systemml/tree/master/scripts>
>>>>> github.com
>>>>> Mirror of Apache SystemML. Contribute to apache/systemml development by
>>>>> creating an account on GitHub.
>>>>>
>>>>>
>>>>> https://github.com/apache/systemml/blob/master/scripts/nn/
>>>> layers/batch_norm1d.dml
>>>>
>>>>> [https://avatars3.githubusercontent.com/u/47359?s=400&v=4]<
>>>>>
>>>>> https://github.com/apache/systemml/blob/master/scripts/nn/
>>>> layers/batch_norm1d.dml
>>>>
>>>>> systemml/batch_norm1d.dml at master · apache/systemml · GitHub<
>>>>>
>>>>> https://github.com/apache/systemml/blob/master/scripts/nn/
>>>> layers/batch_norm1d.dml
>>>>
>>>>> github.com
>>>>> Mirror of Apache SystemML. Contribute to apache/systemml development by
>>>>> creating an account on GitHub.
>>>>>
>>>>>
>>>>> <
>>>>>>>>
>>>>>>>> https://github.com/Baunsgaard/systemml/blob/master/scripts/n
>>>> n/layers/batch_norm2d.dml
>>>>
>>>>> [https://avatars2.githubusercontent.com/u/9947148?s=400&v=4]<
>>>>>
>>>>> https://github.com/Baunsgaard/systemml/blob/master/scripts/n
>>>> n/layers/batch_norm2d.dml
>>>>
>>>>> systemml/batch_norm2d.dml at master · Baunsgaard/systemml · GitHub<
>>>>>
>>>>> https://github.com/Baunsgaard/systemml/blob/master/scripts/n
>>>> n/layers/batch_norm2d.dml
>>>>
>>>>> github.com
>>>>> Mirror of Apache SystemML. Contribute to Baunsgaard/systemml
>>>>> development
>>>>> by creating an account on GitHub.
>>>>>
>>>>>
>>>>> - <
>>>>>>>>
>>>>>>>> https://github.com/Baunsgaard/systemml/blob/master/scripts/b
>>>> uiltin/glm.dml
>>>>
>>>>> [https://avatars2.githubusercontent.com/u/9947148?s=400&v=4]<
>>>>>
>>>>> https://github.com/Baunsgaard/systemml/blob/master/scripts/b
>>>> uiltin/glm.dml
>>>>
>>>>> systemml/glm.dml at master · Baunsgaard/systemml · GitHub<
>>>>>
>>>>> https://github.com/Baunsgaard/systemml/blob/master/scripts/b
>>>> uiltin/glm.dml
>>>>
>>>>> github.com
>>>>> Mirror of Apache SystemML. Contribute to Baunsgaard/systemml
>>>>> development
>>>>> by creating an account on GitHub.
>>>>>
>>>>>
>>>>> <https://github.com/apache/systemml/tree/master/scripts>
>>>>>>>>
>>>>>>> [https://avatars3.githubusercontent.com/u/47359?s=400&v=4]<
>>>>> https://github.com/apache/systemml/tree/master/scripts>
>>>>>
>>>>> systemml/scripts at master · apache/systemml · GitHub<
>>>>> https://github.com/apache/systemml/tree/master/scripts>
>>>>> github.com
>>>>> Mirror of Apache SystemML. Contribute to apache/systemml development by
>>>>> creating an account on GitHub.
>>>>>
>>>>>
>>>>> https://github.com/apache/systemml/blob/master/scripts/algor
>>>> ithms/Cox.dml
>>>>
>>>>> [https://avatars3.githubusercontent.com/u/47359?s=400&v=4]<
>>>>>
>>>>> https://github.com/apache/systemml/blob/master/scripts/algor
>>>> ithms/Cox.dml>
>>>>
>>>>> systemml/Cox.dml at master · apache/systemml · GitHub<
>>>>>
>>>>> https://github.com/apache/systemml/blob/master/scripts/algor
>>>> ithms/Cox.dml>
>>>>
>>>>> github.com
>>>>> Mirror of Apache SystemML. Contribute to apache/systemml development by
>>>>> creating an account on GitHub.
>>>>>
>>>>>
>>>>>
>>>>>>>>
>>>>>>>> Furthermore, I would suggest to postpone changing the main website:
>>>>>>>>
>>>>>>>> https://github.com/apache/systemml-website
>>>>>>>>
>>>>>>> [https://avatars3.githubusercontent.com/u/47359?s=400&v=4]<
>>>>> https://github.com/apache/systemml-website>
>>>>>
>>>>> GitHub - apache/systemml-website: Mirror of Apache SystemML site<
>>>>> https://github.com/apache/systemml-website>
>>>>> github.com
>>>>> Mirror of Apache SystemML site. Contribute to apache/systemml-website
>>>>> development by creating an account on GitHub.
>>>>>
>>>>>
>>>>> But that task has to be started when the next release is scheduled,
>>>>>>>> because then we need to find out how to copy the current
>>>>>>>>
>>>>>>> documentation
>>>>>
>>>>>> to a
>>>>>>>
>>>>>>>> archive list of static webpages located:
>>>>>>>>
>>>>>>>> https://systemml.apache.org/documentation
>>>>>>>>
>>>>>>> Documentation<https://systemml.apache.org/documentation>
>>>>> systemml.apache.org
>>>>> Project Documentation Page
>>>>>
>>>>>
>>>>>
>>>>>>>> best regards
>>>>>>>>
>>>>>>>> Sebastian
>>>>>>>>
>>>>>>>> ________________________________
>>>>>>>> From: Janardhan <ja...@apache.org>
>>>>>>>> Sent: Sunday, May 24, 2020 11:02:53 AM
>>>>>>>> To: dev@systemml.apache.org
>>>>>>>> Subject: [DISCUSSION] Website stack `systemml.apache.org`. Thanks.
>>>>>>>>
>>>>>>>> Hi Sebastian,
>>>>>>>>
>>>>>>>> In our discussion[1][2] a while ago, there was a topic about
>>>>>>>>
>>>>>>> changing
>>>>
>>>>> website
>>>>>>>> stack.
>>>>>>>>
>>>>>>>> Let us start a discussion about this in this thread.
>>>>>>>>
>>>>>>>> We are planning to work on this, after a feasibility study and tech
>>>>>>>>
>>>>>>> stack
>>>>>>
>>>>>>> vetting.
>>>>>>>>
>>>>>>>> Anybody would like to give suggestions would be great.
>>>>>>>>
>>>>>>>> [1] https://github.com/apache/systemml/pull/877
>>>>>>>>
>>>>>>> [https://avatars3.githubusercontent.com/u/47359?s=400&v=4]<
>>>>> https://github.com/apache/systemml/pull/877>
>>>>>
>>>>> [MINOR][DOC] Name refactor SystemML to SystemDS in Documentation by
>>>>> Baunsgaard · Pull Request #877 · apache/systemml · GitHub<
>>>>> https://github.com/apache/systemml/pull/877>
>>>>> github.com
>>>>> Brute change of documentation page to reflect name change. To fully
>>>>> enforce this two more steps are needed:
>>>>> https://github.com/apache/systemml-website Access to SystemML website
>>>>> server on https://systemml.apache.org/ Furthermore, we might want to
>>>>> consider starting fresh in the documentation.
>>>>>
>>>>>
>>>>> [2] https://github.com/apache/systemml-website/pull/66
>>>>>>>>
>>>>>>> [https://avatars3.githubusercontent.com/u/47359?s=400&v=4]<
>>>>> https://github.com/apache/systemml-website/pull/66>
>>>>>
>>>>> [SYSTEMML-2539][BUILD] Upgrade build configuration by j143 · Pull
>>>>> Request
>>>>> #66 · apache/systemml-website · GitHub<
>>>>> https://github.com/apache/systemml-website/pull/66>
>>>>> github.com
>>>>> Upgrade gulp to 4.0 Since the task api is deprecated, fuctions were
>>>>> used,
>>>>> and exported as tasks. Documentation for gulpfile.js. Stick to Susy2,
>>>>>
>>>> since
>>>>
>>>>> mixins are not supported in 3.0 Upgrade npm dependencies to
>>>>> state-of-art.
>>>>> Commit package-lock.json file to version control.
>>>>>
>>>>>
>>>>> Thank you,
>>>>>>>> Janardhan
>>>>>>>>
>>>>>>>>
>
Re: [DISCUSSION] Website stack `systemml.apache.org`. Thanks.
Posted by Matthias Boehm <mb...@gmail.com>.
don't worry about - it's all fine. Leaving it like that (for a single
commit) is MUCH better than trying to forcefully rewrite the history.
Regards,
Matthias
On 6/18/2020 5:35 PM, Baunsgaard, Sebastian wrote:
> Hi Dev,
>
>
> I merged in the documentation PR, this was an slight oversight, while trying to merge something else. That said nothing bad happened except you did not get a chance to look through the PR, and it is progress in the direction discussed in this thread.
>
>
> Best regards
>
> Sebastian
>
>
> ________________________________
> From: Matthias Boehm <mb...@gmail.com>
> Sent: Saturday, June 6, 2020 7:03:19 PM
> To: dev@systemml.apache.org
> Subject: Re: [DISCUSSION] Website stack `systemml.apache.org`. Thanks.
>
> from my perspective it would be very important to have all builtin
> functions in a single markdown file to allow users to search for things
> and users don't care how a builtin function is internally implemented.
> So we might want to use this opportunity to consolidate the already
> documented builtin functions into the new list and add everything that
> is not yet documented as I'm sure many functions are missing.
>
> Btw, in the INFRA ticket for renaming our repositories [1], I also asked
> to disable sending all these github comments and PR interactions to our
> dev list as I'm sure people get overwhelmed by these auto-generated
> emails which leads to important messages being missed. Related to
> documentation, we might want to group categories of builtin functions
> into larger PRs instead of having a PR per builtin function as its
> primarily a copy of existing documentation so they can be reviewed more
> efficiently together to avoid unnecessary overhead.
>
> [1] https://issues.apache.org/jira/browse/INFRA-20362
>
> Regards,
> Matthias
>
> On 6/6/2020 5:23 PM, Janardhan wrote:
>> Hi Arnab,
>>
>> We have all the Java-based builtin functions documented at [1].
>> Most of these functions are already documented!
>>
>> [1]
>> http://apache.github.io/systemml/dml-language-reference.html#built-in-functions
>>
>>
>> Thank you,
>> Janardhan
>>
>> On Fri, Jun 5, 2020 at 4:37 PM arnab phani <ph...@gmail.com> wrote:
>>
>>> That's a good suggestion. One thing I want to point out that Builtins.java
>>> [1] contains all the dml-bodied and java-bodied built-ins. For the dml
>>> built-ins, it is easier to parse the corresponding dml script (if
>>> documentation is available there), but for other built-ins, I am not aware
>>> of any particular source today where they are documented -- a few of them
>>> are documented as a part of commit message but not all.
>>> Though it is certainly possible to slice the dml builtins from [1] out, but
>>> it will be awesome to generate docs for all of those.
>>>
>>> [1]
>>>
>>> https://github.com/apache/systemml/blob/master/src/main/java/org/apache/sysds/common/Builtins.java
>>>
>>> Regards,
>>> Arnab..
>>>
>>> On Thu, Jun 4, 2020 at 12:44 PM Baunsgaard, Sebastian <
>>> baunsgaard@tugraz.at>
>>> wrote:
>>>
>>>> Hi dev team!
>>>>
>>>>
>>>> Update / suggestion.
>>>>
>>>>
>>>> First of all, i think it is great that we have people working on our
>>>> documentation. It is really important. But I can't help to note that the
>>>> improvements, are going in an unmaintainable direction where we have to
>>>> maintain the documentation at multiple locations.
>>>>
>>>>
>>>> For instance our new toOneHot function already had some basic
>>>> documentation inside its build-in script already [1], it would be much
>>>> nicer if we modified that and then had some automation to generate the
>>> docs
>>>> made in commit [2]. Now we have to maintain two locations, resulting in
>>>> high risk of either being outdated.
>>>>
>>>>
>>>> We need to ask ourselves where do we want to make the edit of
>>>> documentation. I would argue that it should be located as close to the
>>>> internal definition of a function, because then when making or modifying
>>>> that function the documentation is also reviewed.
>>>>
>>>>
>>>> To achieve this i suggest that we instead relocate the documentation
>>>> source and generation to an appropriately named file inside [3], that
>>>> simply loops through all function definitions from [4], and generate
>>>> markdown files for our /docs/ folder.
>>>>
>>>>
>>>> Shifting to such an implementation enables us to systematically cover all
>>>> functions defined in the system. It also is a full list of the systems
>>>> current build-in functions thereby giving us the exact file paths to
>>>> extract the source documentation from the comments and generate the docs
>>>> for those files.
>>>>
>>>> In the future it could be extended to verify and detecting which input
>>> and
>>>> output types each function definition has, such that we have even less
>>>> manual documentation needs.
>>>>
>>>>
>>>> Best regards
>>>>
>>>> Sebastian
>>>>
>>>>
>>>> [1]
>>>>
>>> https://github.com/apache/systemml/blob/master/scripts/builtin/toOneHot.dml
>>>> [2]
>>>>
>>> https://github.com/apache/systemml/commit/859ad3a72906c67b7300c9980251da4cde9ed8f8
>>>> [3]
>>>>
>>> https://github.com/apache/systemml/tree/master/src/main/java/org/apache/sysds/common
>>>> [4]
>>>>
>>> https://github.com/apache/systemml/blob/master/src/main/java/org/apache/sysds/common/Builtins.java
>>>>
>>>> ________________________________
>>>> From: arnab phani <ph...@gmail.com>
>>>> Sent: Sunday, May 31, 2020 3:15:44 PM
>>>> To: dev@systemml.apache.org
>>>> Subject: Re: [DISCUSSION] Website stack `systemml.apache.org`. Thanks.
>>>>
>>>> Hi,
>>>>
>>>> Another advantage of markdown syntax is that it gives more freedom to
>>> add a
>>>> description as needed. Java methods and builtins might not fall in the
>>> same
>>>> bucket and different builtins might need different ways to describe it.
>>>> For example, glm needs more details than other built-ins. Too much
>>>> standardization can be a bad idea.
>>>>
>>>> Regards,
>>>> Arnab.
>>>>
>>>> On Sun, May 31, 2020 at 9:55 AM Baunsgaard, Sebastian <
>>>> baunsgaard@tugraz.at>
>>>> wrote:
>>>>
>>>>> Hi Janardhan,
>>>>>
>>>>>
>>>>> That would be one option, lets call it option 1, I have another
>>>> suggestion:
>>>>>
>>>>> 1. Markdown syntax with out commented individual lines.
>>>>> 2. Jdoc like syntax with annotations
>>>>>
>>>>> Pros and cons
>>>>>
>>>>> * 1 Pros
>>>>> * Easy Syntax most ppl know it.
>>>>> * Easy to implement.
>>>>> * 1 Cons
>>>>> * Enables custom docs (everyone make their own distinct format
>>>>> again)
>>>>> * No standard way of verifying correctness of the docs (correct
>>>>> parameter names etc.)
>>>>> * No standard way of marking Input, return parameters.
>>>>> * Requires modified Syntax with # at each line. where # normally
>>>>> means header in markdown
>>>>> * 2 Pros
>>>>> * Easy Syntax Java ppl know it
>>>>> * Standard way of making input
>>>>> * Do not have to change any syntax at all from Jdoc
>>>>> * Already supported syntax in DML
>>>>> * 2 Cons
>>>>> * Harder to implement since one would have to look into java doc
>>>>> extraction and what is needed to support that, maybe we have to make
>>>> such a
>>>>> thing ourselves?
>>>>>
>>>>> I personally like option 2 with Jdoc more and then extending into
>>>>> automatically parsing and making the either markdown files or HTML
>>> files
>>>>> for the docs that you either way have to do in option 1.
>>>>>
>>>>> Best regards
>>>>> Sebastian
>>>>> ________________________________
>>>>> From: Janardhan <ja...@gmail.com>
>>>>> Sent: Sunday, May 31, 2020 9:25:43 AM
>>>>> To: dev@systemml.apache.org
>>>>> Subject: Re: [DISCUSSION] Website stack `systemml.apache.org`. Thanks.
>>>>>
>>>>> Hi Sebastian, :)
>>>>>
>>>>> Since the builtin DML files do not have full documentation yet, Can we
>>>>> write
>>>>> markdown syntax with the following heading in each dml file itself!
>>>>>
>>>>> 1. Function description
>>>>> 2. Usage
>>>>> 3. Arguments
>>>>> 4. Returns
>>>>> 5. Usage
>>>>>
>>>>> After that, as you've mentioned can be parsed with only removing # at
>>> the
>>>>> start of each line, and write
>>>>> all this data to one `builtins-reference.md` file.
>>>>>
>>>>> Thank you,
>>>>> Janardhan
>>>>>
>>>>> On Tue, May 26, 2020 at 5:05 PM Baunsgaard, Sebastian <
>>>>> baunsgaard@tugraz.at>
>>>>> wrote:
>>>>>
>>>>>> If this is meant as a question then, yes from me (this probably calls
>>>> for
>>>>>> a vote?), if:
>>>>>>
>>>>>>
>>>>>> 1. someone is able to change the settings for the repository, to
>>> use
>>>>>> the docs folder as the GitHub docs folder.
>>>>>> 2. someone wants to make a PR containing the content of the
>>> gh-pages
>>>>>> branch.
>>>>>>
>>>>>> Best regards
>>>>>> Sebastian
>>>>>> ________________________________
>>>>>> From: Janardhan <ja...@gmail.com>
>>>>>> Sent: Tuesday, May 26, 2020 1:22:11 PM
>>>>>> To: dev@systemml.apache.org
>>>>>> Subject: Re: [DISCUSSION] Website stack `systemml.apache.org`.
>>> Thanks.
>>>>>> Hi Sebastian,
>>>>>>
>>>>>> Can we move the `docs` folder in the existing `master` branch
>>>>>> and merge the `gh-pages` branch to `docs` folder in the top-level
>>>>>> directory. :)
>>>>>>
>>>>>> - Janardhan
>>>>>>
>>>>>> On Sun, May 24, 2020 at 3:28 PM Baunsgaard, Sebastian <
>>>>>> baunsgaard@tugraz.at>
>>>>>> wrote:
>>>>>>
>>>>>>> Hi Janardhan,
>>>>>>>
>>>>>>>
>>>>>>> Reworking the documentation seems like a really good idea!
>>>>>>>
>>>>>>>
>>>>>>> A first step in my opinion is to start in the main repository.
>>>>>>>
>>>>>>>
>>>>>>> I would suggest the following:
>>>>>>>
>>>>>>>
>>>>>>> - Merge the current gh-pages branch into master docs folder.
>>>>>>>
>>>>>>> - Change the documentations page :
>>> http://apache.github.io/systemml/
>>>> SystemDS Documentation - SystemDS<http://apache.github.io/systemml/>
>>>> apache.github.io
>>>> SystemDS Documentation
>>>>
>>>>
>>>>> to
>>>>>>> use the docs folder for documentation. Should be doable inside the
>>>>>> settings
>>>>>>> on the repository.
>>>>>>>
>>>>>>> - Delete the gh-pages branch
>>>>>>>
>>>>>>>
>>>>>>> I want this change because then new PRs can change the
>>> documentation
>>>> in
>>>>>>> one PR rather than two (which to be honest is not going to happen
>>>> most
>>>>> of
>>>>>>> the time).
>>>>>>>
>>>>>>> The downside (we have to mention this) is that when you clone the
>>>>>>> repository, you have slightly larger downloads, but i think the
>>>>> trade-off
>>>>>>> is fair. If you see any other issues please mention these in this
>>>>> thread.
>>>>>>>
>>>>>>> Next task I would like to suggest to include:
>>>>>>>
>>>>>>>
>>>>>>> - Java docs generated and
>>>>>>>
>>>>>>> - Python docs generated to also be included as a sub-folder in
>>> docs,
>>>> so
>>>>>>> that the current webpage contains the docs for that to.
>>>>>>>
>>>>>>>
>>>>>>> How to include these in a sensible manner is currently a good
>>>> question,
>>>>>>> and has to be seriously well considered. Including these will also
>>>>> enable
>>>>>>> us to move documentation closer to the individual code parts,
>>> making
>>>> it
>>>>>>> again more likely to get the documentation done.
>>>>>>>
>>>>>>>
>>>>>>> Another more programmatic task is to make a parser for our DML
>>> files,
>>>>>> such
>>>>>>> that we can generate webpage documentation based on the
>>> documentation
>>>>>>> syntax used (much like the python and java docs generator). This
>>>> would
>>>>> be
>>>>>>> great since it will remove the duplication of documentation in the
>>>>>>> individual DML files, and the documentation we have already in
>>> files
>>>>> such
>>>>>>> as: docs/dml-language-reference.md.
>>>>>>>
>>>>>>> This also gives us an excuse to clean up that documentation/scripts
>>>>> which
>>>>>>> has very different implementations in our scripts folder:
>>>>>>>
>>>>>>>
>>>>>>> examples:
>>>>>>>
>>>>>>>
>>>>>>> - <https://github.com/apache/systemml/tree/master/scripts>
>>>> [https://avatars3.githubusercontent.com/u/47359?s=400&v=4]<
>>>> https://github.com/apache/systemml/tree/master/scripts>
>>>>
>>>> systemml/scripts at master · apache/systemml · GitHub<
>>>> https://github.com/apache/systemml/tree/master/scripts>
>>>> github.com
>>>> Mirror of Apache SystemML. Contribute to apache/systemml development by
>>>> creating an account on GitHub.
>>>>
>>>>
>>> https://github.com/apache/systemml/blob/master/scripts/nn/layers/batch_norm1d.dml
>>>> [https://avatars3.githubusercontent.com/u/47359?s=400&v=4]<
>>>>
>>> https://github.com/apache/systemml/blob/master/scripts/nn/layers/batch_norm1d.dml
>>>> systemml/batch_norm1d.dml at master · apache/systemml · GitHub<
>>>>
>>> https://github.com/apache/systemml/blob/master/scripts/nn/layers/batch_norm1d.dml
>>>> github.com
>>>> Mirror of Apache SystemML. Contribute to apache/systemml development by
>>>> creating an account on GitHub.
>>>>
>>>>
>>>>>>> <
>>>>>>>
>>> https://github.com/Baunsgaard/systemml/blob/master/scripts/nn/layers/batch_norm2d.dml
>>>> [https://avatars2.githubusercontent.com/u/9947148?s=400&v=4]<
>>>>
>>> https://github.com/Baunsgaard/systemml/blob/master/scripts/nn/layers/batch_norm2d.dml
>>>> systemml/batch_norm2d.dml at master · Baunsgaard/systemml · GitHub<
>>>>
>>> https://github.com/Baunsgaard/systemml/blob/master/scripts/nn/layers/batch_norm2d.dml
>>>> github.com
>>>> Mirror of Apache SystemML. Contribute to Baunsgaard/systemml development
>>>> by creating an account on GitHub.
>>>>
>>>>
>>>>>>> - <
>>>>>>>
>>> https://github.com/Baunsgaard/systemml/blob/master/scripts/builtin/glm.dml
>>>> [https://avatars2.githubusercontent.com/u/9947148?s=400&v=4]<
>>>>
>>> https://github.com/Baunsgaard/systemml/blob/master/scripts/builtin/glm.dml
>>>> systemml/glm.dml at master · Baunsgaard/systemml · GitHub<
>>>>
>>> https://github.com/Baunsgaard/systemml/blob/master/scripts/builtin/glm.dml
>>>> github.com
>>>> Mirror of Apache SystemML. Contribute to Baunsgaard/systemml development
>>>> by creating an account on GitHub.
>>>>
>>>>
>>>>>>> <https://github.com/apache/systemml/tree/master/scripts>
>>>> [https://avatars3.githubusercontent.com/u/47359?s=400&v=4]<
>>>> https://github.com/apache/systemml/tree/master/scripts>
>>>>
>>>> systemml/scripts at master · apache/systemml · GitHub<
>>>> https://github.com/apache/systemml/tree/master/scripts>
>>>> github.com
>>>> Mirror of Apache SystemML. Contribute to apache/systemml development by
>>>> creating an account on GitHub.
>>>>
>>>>
>>> https://github.com/apache/systemml/blob/master/scripts/algorithms/Cox.dml
>>>> [https://avatars3.githubusercontent.com/u/47359?s=400&v=4]<
>>>>
>>> https://github.com/apache/systemml/blob/master/scripts/algorithms/Cox.dml>
>>>> systemml/Cox.dml at master · apache/systemml · GitHub<
>>>>
>>> https://github.com/apache/systemml/blob/master/scripts/algorithms/Cox.dml>
>>>> github.com
>>>> Mirror of Apache SystemML. Contribute to apache/systemml development by
>>>> creating an account on GitHub.
>>>>
>>>>
>>>>>>>
>>>>>>>
>>>>>>> Furthermore, I would suggest to postpone changing the main website:
>>>>>>>
>>>>>>> https://github.com/apache/systemml-website
>>>> [https://avatars3.githubusercontent.com/u/47359?s=400&v=4]<
>>>> https://github.com/apache/systemml-website>
>>>>
>>>> GitHub - apache/systemml-website: Mirror of Apache SystemML site<
>>>> https://github.com/apache/systemml-website>
>>>> github.com
>>>> Mirror of Apache SystemML site. Contribute to apache/systemml-website
>>>> development by creating an account on GitHub.
>>>>
>>>>
>>>>>>> But that task has to be started when the next release is scheduled,
>>>>>>> because then we need to find out how to copy the current
>>>> documentation
>>>>>> to a
>>>>>>> archive list of static webpages located:
>>>>>>>
>>>>>>> https://systemml.apache.org/documentation
>>>> Documentation<https://systemml.apache.org/documentation>
>>>> systemml.apache.org
>>>> Project Documentation Page
>>>>
>>>>
>>>>>>>
>>>>>>> best regards
>>>>>>>
>>>>>>> Sebastian
>>>>>>>
>>>>>>> ________________________________
>>>>>>> From: Janardhan <ja...@apache.org>
>>>>>>> Sent: Sunday, May 24, 2020 11:02:53 AM
>>>>>>> To: dev@systemml.apache.org
>>>>>>> Subject: [DISCUSSION] Website stack `systemml.apache.org`. Thanks.
>>>>>>>
>>>>>>> Hi Sebastian,
>>>>>>>
>>>>>>> In our discussion[1][2] a while ago, there was a topic about
>>> changing
>>>>>>> website
>>>>>>> stack.
>>>>>>>
>>>>>>> Let us start a discussion about this in this thread.
>>>>>>>
>>>>>>> We are planning to work on this, after a feasibility study and tech
>>>>> stack
>>>>>>> vetting.
>>>>>>>
>>>>>>> Anybody would like to give suggestions would be great.
>>>>>>>
>>>>>>> [1] https://github.com/apache/systemml/pull/877
>>>> [https://avatars3.githubusercontent.com/u/47359?s=400&v=4]<
>>>> https://github.com/apache/systemml/pull/877>
>>>>
>>>> [MINOR][DOC] Name refactor SystemML to SystemDS in Documentation by
>>>> Baunsgaard · Pull Request #877 · apache/systemml · GitHub<
>>>> https://github.com/apache/systemml/pull/877>
>>>> github.com
>>>> Brute change of documentation page to reflect name change. To fully
>>>> enforce this two more steps are needed:
>>>> https://github.com/apache/systemml-website Access to SystemML website
>>>> server on https://systemml.apache.org/ Furthermore, we might want to
>>>> consider starting fresh in the documentation.
>>>>
>>>>
>>>>>>> [2] https://github.com/apache/systemml-website/pull/66
>>>> [https://avatars3.githubusercontent.com/u/47359?s=400&v=4]<
>>>> https://github.com/apache/systemml-website/pull/66>
>>>>
>>>> [SYSTEMML-2539][BUILD] Upgrade build configuration by j143 · Pull Request
>>>> #66 · apache/systemml-website · GitHub<
>>>> https://github.com/apache/systemml-website/pull/66>
>>>> github.com
>>>> Upgrade gulp to 4.0 Since the task api is deprecated, fuctions were used,
>>>> and exported as tasks. Documentation for gulpfile.js. Stick to Susy2,
>>> since
>>>> mixins are not supported in 3.0 Upgrade npm dependencies to state-of-art.
>>>> Commit package-lock.json file to version control.
>>>>
>>>>
>>>>>>> Thank you,
>>>>>>> Janardhan
>>>>>>>
Re: [DISCUSSION] Website stack `systemml.apache.org`. Thanks.
Posted by "Baunsgaard, Sebastian" <ba...@tugraz.at>.
Hi Dev,
I merged in the documentation PR, this was an slight oversight, while trying to merge something else. That said nothing bad happened except you did not get a chance to look through the PR, and it is progress in the direction discussed in this thread.
Best regards
Sebastian
________________________________
From: Matthias Boehm <mb...@gmail.com>
Sent: Saturday, June 6, 2020 7:03:19 PM
To: dev@systemml.apache.org
Subject: Re: [DISCUSSION] Website stack `systemml.apache.org`. Thanks.
from my perspective it would be very important to have all builtin
functions in a single markdown file to allow users to search for things
and users don't care how a builtin function is internally implemented.
So we might want to use this opportunity to consolidate the already
documented builtin functions into the new list and add everything that
is not yet documented as I'm sure many functions are missing.
Btw, in the INFRA ticket for renaming our repositories [1], I also asked
to disable sending all these github comments and PR interactions to our
dev list as I'm sure people get overwhelmed by these auto-generated
emails which leads to important messages being missed. Related to
documentation, we might want to group categories of builtin functions
into larger PRs instead of having a PR per builtin function as its
primarily a copy of existing documentation so they can be reviewed more
efficiently together to avoid unnecessary overhead.
[1] https://issues.apache.org/jira/browse/INFRA-20362
Regards,
Matthias
On 6/6/2020 5:23 PM, Janardhan wrote:
> Hi Arnab,
>
> We have all the Java-based builtin functions documented at [1].
> Most of these functions are already documented!
>
> [1]
> http://apache.github.io/systemml/dml-language-reference.html#built-in-functions
>
>
> Thank you,
> Janardhan
>
> On Fri, Jun 5, 2020 at 4:37 PM arnab phani <ph...@gmail.com> wrote:
>
>> That's a good suggestion. One thing I want to point out that Builtins.java
>> [1] contains all the dml-bodied and java-bodied built-ins. For the dml
>> built-ins, it is easier to parse the corresponding dml script (if
>> documentation is available there), but for other built-ins, I am not aware
>> of any particular source today where they are documented -- a few of them
>> are documented as a part of commit message but not all.
>> Though it is certainly possible to slice the dml builtins from [1] out, but
>> it will be awesome to generate docs for all of those.
>>
>> [1]
>>
>> https://github.com/apache/systemml/blob/master/src/main/java/org/apache/sysds/common/Builtins.java
>>
>> Regards,
>> Arnab..
>>
>> On Thu, Jun 4, 2020 at 12:44 PM Baunsgaard, Sebastian <
>> baunsgaard@tugraz.at>
>> wrote:
>>
>>> Hi dev team!
>>>
>>>
>>> Update / suggestion.
>>>
>>>
>>> First of all, i think it is great that we have people working on our
>>> documentation. It is really important. But I can't help to note that the
>>> improvements, are going in an unmaintainable direction where we have to
>>> maintain the documentation at multiple locations.
>>>
>>>
>>> For instance our new toOneHot function already had some basic
>>> documentation inside its build-in script already [1], it would be much
>>> nicer if we modified that and then had some automation to generate the
>> docs
>>> made in commit [2]. Now we have to maintain two locations, resulting in
>>> high risk of either being outdated.
>>>
>>>
>>> We need to ask ourselves where do we want to make the edit of
>>> documentation. I would argue that it should be located as close to the
>>> internal definition of a function, because then when making or modifying
>>> that function the documentation is also reviewed.
>>>
>>>
>>> To achieve this i suggest that we instead relocate the documentation
>>> source and generation to an appropriately named file inside [3], that
>>> simply loops through all function definitions from [4], and generate
>>> markdown files for our /docs/ folder.
>>>
>>>
>>> Shifting to such an implementation enables us to systematically cover all
>>> functions defined in the system. It also is a full list of the systems
>>> current build-in functions thereby giving us the exact file paths to
>>> extract the source documentation from the comments and generate the docs
>>> for those files.
>>>
>>> In the future it could be extended to verify and detecting which input
>> and
>>> output types each function definition has, such that we have even less
>>> manual documentation needs.
>>>
>>>
>>> Best regards
>>>
>>> Sebastian
>>>
>>>
>>> [1]
>>>
>> https://github.com/apache/systemml/blob/master/scripts/builtin/toOneHot.dml
>>>
>>> [2]
>>>
>> https://github.com/apache/systemml/commit/859ad3a72906c67b7300c9980251da4cde9ed8f8
>>>
>>> [3]
>>>
>> https://github.com/apache/systemml/tree/master/src/main/java/org/apache/sysds/common
>>>
>>> [4]
>>>
>> https://github.com/apache/systemml/blob/master/src/main/java/org/apache/sysds/common/Builtins.java
>>>
>>>
>>> ________________________________
>>> From: arnab phani <ph...@gmail.com>
>>> Sent: Sunday, May 31, 2020 3:15:44 PM
>>> To: dev@systemml.apache.org
>>> Subject: Re: [DISCUSSION] Website stack `systemml.apache.org`. Thanks.
>>>
>>> Hi,
>>>
>>> Another advantage of markdown syntax is that it gives more freedom to
>> add a
>>> description as needed. Java methods and builtins might not fall in the
>> same
>>> bucket and different builtins might need different ways to describe it.
>>> For example, glm needs more details than other built-ins. Too much
>>> standardization can be a bad idea.
>>>
>>> Regards,
>>> Arnab.
>>>
>>> On Sun, May 31, 2020 at 9:55 AM Baunsgaard, Sebastian <
>>> baunsgaard@tugraz.at>
>>> wrote:
>>>
>>>> Hi Janardhan,
>>>>
>>>>
>>>> That would be one option, lets call it option 1, I have another
>>> suggestion:
>>>>
>>>>
>>>> 1. Markdown syntax with out commented individual lines.
>>>> 2. Jdoc like syntax with annotations
>>>>
>>>> Pros and cons
>>>>
>>>> * 1 Pros
>>>> * Easy Syntax most ppl know it.
>>>> * Easy to implement.
>>>> * 1 Cons
>>>> * Enables custom docs (everyone make their own distinct format
>>>> again)
>>>> * No standard way of verifying correctness of the docs (correct
>>>> parameter names etc.)
>>>> * No standard way of marking Input, return parameters.
>>>> * Requires modified Syntax with # at each line. where # normally
>>>> means header in markdown
>>>> * 2 Pros
>>>> * Easy Syntax Java ppl know it
>>>> * Standard way of making input
>>>> * Do not have to change any syntax at all from Jdoc
>>>> * Already supported syntax in DML
>>>> * 2 Cons
>>>> * Harder to implement since one would have to look into java doc
>>>> extraction and what is needed to support that, maybe we have to make
>>> such a
>>>> thing ourselves?
>>>>
>>>> I personally like option 2 with Jdoc more and then extending into
>>>> automatically parsing and making the either markdown files or HTML
>> files
>>>> for the docs that you either way have to do in option 1.
>>>>
>>>> Best regards
>>>> Sebastian
>>>> ________________________________
>>>> From: Janardhan <ja...@gmail.com>
>>>> Sent: Sunday, May 31, 2020 9:25:43 AM
>>>> To: dev@systemml.apache.org
>>>> Subject: Re: [DISCUSSION] Website stack `systemml.apache.org`. Thanks.
>>>>
>>>> Hi Sebastian, :)
>>>>
>>>> Since the builtin DML files do not have full documentation yet, Can we
>>>> write
>>>> markdown syntax with the following heading in each dml file itself!
>>>>
>>>> 1. Function description
>>>> 2. Usage
>>>> 3. Arguments
>>>> 4. Returns
>>>> 5. Usage
>>>>
>>>> After that, as you've mentioned can be parsed with only removing # at
>> the
>>>> start of each line, and write
>>>> all this data to one `builtins-reference.md` file.
>>>>
>>>> Thank you,
>>>> Janardhan
>>>>
>>>> On Tue, May 26, 2020 at 5:05 PM Baunsgaard, Sebastian <
>>>> baunsgaard@tugraz.at>
>>>> wrote:
>>>>
>>>>>
>>>>> If this is meant as a question then, yes from me (this probably calls
>>> for
>>>>> a vote?), if:
>>>>>
>>>>>
>>>>> 1. someone is able to change the settings for the repository, to
>> use
>>>>> the docs folder as the GitHub docs folder.
>>>>> 2. someone wants to make a PR containing the content of the
>> gh-pages
>>>>> branch.
>>>>>
>>>>> Best regards
>>>>> Sebastian
>>>>> ________________________________
>>>>> From: Janardhan <ja...@gmail.com>
>>>>> Sent: Tuesday, May 26, 2020 1:22:11 PM
>>>>> To: dev@systemml.apache.org
>>>>> Subject: Re: [DISCUSSION] Website stack `systemml.apache.org`.
>> Thanks.
>>>>>
>>>>> Hi Sebastian,
>>>>>
>>>>> Can we move the `docs` folder in the existing `master` branch
>>>>> and merge the `gh-pages` branch to `docs` folder in the top-level
>>>>> directory. :)
>>>>>
>>>>> - Janardhan
>>>>>
>>>>> On Sun, May 24, 2020 at 3:28 PM Baunsgaard, Sebastian <
>>>>> baunsgaard@tugraz.at>
>>>>> wrote:
>>>>>
>>>>>> Hi Janardhan,
>>>>>>
>>>>>>
>>>>>> Reworking the documentation seems like a really good idea!
>>>>>>
>>>>>>
>>>>>> A first step in my opinion is to start in the main repository.
>>>>>>
>>>>>>
>>>>>> I would suggest the following:
>>>>>>
>>>>>>
>>>>>> - Merge the current gh-pages branch into master docs folder.
>>>>>>
>>>>>> - Change the documentations page :
>> http://apache.github.io/systemml/
>>> SystemDS Documentation - SystemDS<http://apache.github.io/systemml/>
>>> apache.github.io
>>> SystemDS Documentation
>>>
>>>
>>>> to
>>>>>> use the docs folder for documentation. Should be doable inside the
>>>>> settings
>>>>>> on the repository.
>>>>>>
>>>>>> - Delete the gh-pages branch
>>>>>>
>>>>>>
>>>>>> I want this change because then new PRs can change the
>> documentation
>>> in
>>>>>> one PR rather than two (which to be honest is not going to happen
>>> most
>>>> of
>>>>>> the time).
>>>>>>
>>>>>> The downside (we have to mention this) is that when you clone the
>>>>>> repository, you have slightly larger downloads, but i think the
>>>> trade-off
>>>>>> is fair. If you see any other issues please mention these in this
>>>> thread.
>>>>>>
>>>>>>
>>>>>> Next task I would like to suggest to include:
>>>>>>
>>>>>>
>>>>>> - Java docs generated and
>>>>>>
>>>>>> - Python docs generated to also be included as a sub-folder in
>> docs,
>>> so
>>>>>> that the current webpage contains the docs for that to.
>>>>>>
>>>>>>
>>>>>> How to include these in a sensible manner is currently a good
>>> question,
>>>>>> and has to be seriously well considered. Including these will also
>>>> enable
>>>>>> us to move documentation closer to the individual code parts,
>> making
>>> it
>>>>>> again more likely to get the documentation done.
>>>>>>
>>>>>>
>>>>>> Another more programmatic task is to make a parser for our DML
>> files,
>>>>> such
>>>>>> that we can generate webpage documentation based on the
>> documentation
>>>>>> syntax used (much like the python and java docs generator). This
>>> would
>>>> be
>>>>>> great since it will remove the duplication of documentation in the
>>>>>> individual DML files, and the documentation we have already in
>> files
>>>> such
>>>>>> as: docs/dml-language-reference.md.
>>>>>>
>>>>>> This also gives us an excuse to clean up that documentation/scripts
>>>> which
>>>>>> has very different implementations in our scripts folder:
>>>>>>
>>>>>>
>>>>>> examples:
>>>>>>
>>>>>>
>>>>>> - <https://github.com/apache/systemml/tree/master/scripts>
>>> [https://avatars3.githubusercontent.com/u/47359?s=400&v=4]<
>>> https://github.com/apache/systemml/tree/master/scripts>
>>>
>>> systemml/scripts at master · apache/systemml · GitHub<
>>> https://github.com/apache/systemml/tree/master/scripts>
>>> github.com
>>> Mirror of Apache SystemML. Contribute to apache/systemml development by
>>> creating an account on GitHub.
>>>
>>>
>>>>>>
>>>>>
>>>>
>>>
>> https://github.com/apache/systemml/blob/master/scripts/nn/layers/batch_norm1d.dml
>>> [https://avatars3.githubusercontent.com/u/47359?s=400&v=4]<
>>>
>> https://github.com/apache/systemml/blob/master/scripts/nn/layers/batch_norm1d.dml
>>>>
>>>
>>> systemml/batch_norm1d.dml at master · apache/systemml · GitHub<
>>>
>> https://github.com/apache/systemml/blob/master/scripts/nn/layers/batch_norm1d.dml
>>>>
>>> github.com
>>> Mirror of Apache SystemML. Contribute to apache/systemml development by
>>> creating an account on GitHub.
>>>
>>>
>>>>>> <
>>>>>>
>>>>>
>>>>
>>>
>> https://github.com/Baunsgaard/systemml/blob/master/scripts/nn/layers/batch_norm2d.dml
>>> [https://avatars2.githubusercontent.com/u/9947148?s=400&v=4]<
>>>
>> https://github.com/Baunsgaard/systemml/blob/master/scripts/nn/layers/batch_norm2d.dml
>>>>
>>>
>>> systemml/batch_norm2d.dml at master · Baunsgaard/systemml · GitHub<
>>>
>> https://github.com/Baunsgaard/systemml/blob/master/scripts/nn/layers/batch_norm2d.dml
>>>>
>>> github.com
>>> Mirror of Apache SystemML. Contribute to Baunsgaard/systemml development
>>> by creating an account on GitHub.
>>>
>>>
>>>>>>>
>>>>>>
>>>>>> - <
>>>>>>
>>>>>
>>>>
>>>
>> https://github.com/Baunsgaard/systemml/blob/master/scripts/builtin/glm.dml
>>> [https://avatars2.githubusercontent.com/u/9947148?s=400&v=4]<
>>>
>> https://github.com/Baunsgaard/systemml/blob/master/scripts/builtin/glm.dml
>>>>
>>>
>>> systemml/glm.dml at master · Baunsgaard/systemml · GitHub<
>>>
>> https://github.com/Baunsgaard/systemml/blob/master/scripts/builtin/glm.dml
>>>>
>>> github.com
>>> Mirror of Apache SystemML. Contribute to Baunsgaard/systemml development
>>> by creating an account on GitHub.
>>>
>>>
>>>>>>
>>>>>> <https://github.com/apache/systemml/tree/master/scripts>
>>> [https://avatars3.githubusercontent.com/u/47359?s=400&v=4]<
>>> https://github.com/apache/systemml/tree/master/scripts>
>>>
>>> systemml/scripts at master · apache/systemml · GitHub<
>>> https://github.com/apache/systemml/tree/master/scripts>
>>> github.com
>>> Mirror of Apache SystemML. Contribute to apache/systemml development by
>>> creating an account on GitHub.
>>>
>>>
>>>>>>
>>>>>
>>>>
>>>
>> https://github.com/apache/systemml/blob/master/scripts/algorithms/Cox.dml
>>> [https://avatars3.githubusercontent.com/u/47359?s=400&v=4]<
>>>
>> https://github.com/apache/systemml/blob/master/scripts/algorithms/Cox.dml>
>>>
>>> systemml/Cox.dml at master · apache/systemml · GitHub<
>>>
>> https://github.com/apache/systemml/blob/master/scripts/algorithms/Cox.dml>
>>> github.com
>>> Mirror of Apache SystemML. Contribute to apache/systemml development by
>>> creating an account on GitHub.
>>>
>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> Furthermore, I would suggest to postpone changing the main website:
>>>>>>
>>>>>> https://github.com/apache/systemml-website
>>> [https://avatars3.githubusercontent.com/u/47359?s=400&v=4]<
>>> https://github.com/apache/systemml-website>
>>>
>>> GitHub - apache/systemml-website: Mirror of Apache SystemML site<
>>> https://github.com/apache/systemml-website>
>>> github.com
>>> Mirror of Apache SystemML site. Contribute to apache/systemml-website
>>> development by creating an account on GitHub.
>>>
>>>
>>>>>>
>>>>>> But that task has to be started when the next release is scheduled,
>>>>>> because then we need to find out how to copy the current
>>> documentation
>>>>> to a
>>>>>> archive list of static webpages located:
>>>>>>
>>>>>> https://systemml.apache.org/documentation
>>> Documentation<https://systemml.apache.org/documentation>
>>> systemml.apache.org
>>> Project Documentation Page
>>>
>>>
>>>>>>
>>>>>>
>>>>>> best regards
>>>>>>
>>>>>> Sebastian
>>>>>>
>>>>>> ________________________________
>>>>>> From: Janardhan <ja...@apache.org>
>>>>>> Sent: Sunday, May 24, 2020 11:02:53 AM
>>>>>> To: dev@systemml.apache.org
>>>>>> Subject: [DISCUSSION] Website stack `systemml.apache.org`. Thanks.
>>>>>>
>>>>>> Hi Sebastian,
>>>>>>
>>>>>> In our discussion[1][2] a while ago, there was a topic about
>> changing
>>>>>> website
>>>>>> stack.
>>>>>>
>>>>>> Let us start a discussion about this in this thread.
>>>>>>
>>>>>> We are planning to work on this, after a feasibility study and tech
>>>> stack
>>>>>> vetting.
>>>>>>
>>>>>> Anybody would like to give suggestions would be great.
>>>>>>
>>>>>> [1] https://github.com/apache/systemml/pull/877
>>> [https://avatars3.githubusercontent.com/u/47359?s=400&v=4]<
>>> https://github.com/apache/systemml/pull/877>
>>>
>>> [MINOR][DOC] Name refactor SystemML to SystemDS in Documentation by
>>> Baunsgaard · Pull Request #877 · apache/systemml · GitHub<
>>> https://github.com/apache/systemml/pull/877>
>>> github.com
>>> Brute change of documentation page to reflect name change. To fully
>>> enforce this two more steps are needed:
>>> https://github.com/apache/systemml-website Access to SystemML website
>>> server on https://systemml.apache.org/ Furthermore, we might want to
>>> consider starting fresh in the documentation.
>>>
>>>
>>>>>> [2] https://github.com/apache/systemml-website/pull/66
>>> [https://avatars3.githubusercontent.com/u/47359?s=400&v=4]<
>>> https://github.com/apache/systemml-website/pull/66>
>>>
>>> [SYSTEMML-2539][BUILD] Upgrade build configuration by j143 · Pull Request
>>> #66 · apache/systemml-website · GitHub<
>>> https://github.com/apache/systemml-website/pull/66>
>>> github.com
>>> Upgrade gulp to 4.0 Since the task api is deprecated, fuctions were used,
>>> and exported as tasks. Documentation for gulpfile.js. Stick to Susy2,
>> since
>>> mixins are not supported in 3.0 Upgrade npm dependencies to state-of-art.
>>> Commit package-lock.json file to version control.
>>>
>>>
>>>>>>
>>>>>> Thank you,
>>>>>> Janardhan
>>>>>>
>>>>>
>>>>
>>>
>>
>
Re: [DISCUSSION] Website stack `systemml.apache.org`. Thanks.
Posted by Matthias Boehm <mb...@gmail.com>.
from my perspective it would be very important to have all builtin
functions in a single markdown file to allow users to search for things
and users don't care how a builtin function is internally implemented.
So we might want to use this opportunity to consolidate the already
documented builtin functions into the new list and add everything that
is not yet documented as I'm sure many functions are missing.
Btw, in the INFRA ticket for renaming our repositories [1], I also asked
to disable sending all these github comments and PR interactions to our
dev list as I'm sure people get overwhelmed by these auto-generated
emails which leads to important messages being missed. Related to
documentation, we might want to group categories of builtin functions
into larger PRs instead of having a PR per builtin function as its
primarily a copy of existing documentation so they can be reviewed more
efficiently together to avoid unnecessary overhead.
[1] https://issues.apache.org/jira/browse/INFRA-20362
Regards,
Matthias
On 6/6/2020 5:23 PM, Janardhan wrote:
> Hi Arnab,
>
> We have all the Java-based builtin functions documented at [1].
> Most of these functions are already documented!
>
> [1]
> http://apache.github.io/systemml/dml-language-reference.html#built-in-functions
>
>
> Thank you,
> Janardhan
>
> On Fri, Jun 5, 2020 at 4:37 PM arnab phani <ph...@gmail.com> wrote:
>
>> That's a good suggestion. One thing I want to point out that Builtins.java
>> [1] contains all the dml-bodied and java-bodied built-ins. For the dml
>> built-ins, it is easier to parse the corresponding dml script (if
>> documentation is available there), but for other built-ins, I am not aware
>> of any particular source today where they are documented -- a few of them
>> are documented as a part of commit message but not all.
>> Though it is certainly possible to slice the dml builtins from [1] out, but
>> it will be awesome to generate docs for all of those.
>>
>> [1]
>>
>> https://github.com/apache/systemml/blob/master/src/main/java/org/apache/sysds/common/Builtins.java
>>
>> Regards,
>> Arnab..
>>
>> On Thu, Jun 4, 2020 at 12:44 PM Baunsgaard, Sebastian <
>> baunsgaard@tugraz.at>
>> wrote:
>>
>>> Hi dev team!
>>>
>>>
>>> Update / suggestion.
>>>
>>>
>>> First of all, i think it is great that we have people working on our
>>> documentation. It is really important. But I can't help to note that the
>>> improvements, are going in an unmaintainable direction where we have to
>>> maintain the documentation at multiple locations.
>>>
>>>
>>> For instance our new toOneHot function already had some basic
>>> documentation inside its build-in script already [1], it would be much
>>> nicer if we modified that and then had some automation to generate the
>> docs
>>> made in commit [2]. Now we have to maintain two locations, resulting in
>>> high risk of either being outdated.
>>>
>>>
>>> We need to ask ourselves where do we want to make the edit of
>>> documentation. I would argue that it should be located as close to the
>>> internal definition of a function, because then when making or modifying
>>> that function the documentation is also reviewed.
>>>
>>>
>>> To achieve this i suggest that we instead relocate the documentation
>>> source and generation to an appropriately named file inside [3], that
>>> simply loops through all function definitions from [4], and generate
>>> markdown files for our /docs/ folder.
>>>
>>>
>>> Shifting to such an implementation enables us to systematically cover all
>>> functions defined in the system. It also is a full list of the systems
>>> current build-in functions thereby giving us the exact file paths to
>>> extract the source documentation from the comments and generate the docs
>>> for those files.
>>>
>>> In the future it could be extended to verify and detecting which input
>> and
>>> output types each function definition has, such that we have even less
>>> manual documentation needs.
>>>
>>>
>>> Best regards
>>>
>>> Sebastian
>>>
>>>
>>> [1]
>>>
>> https://github.com/apache/systemml/blob/master/scripts/builtin/toOneHot.dml
>>>
>>> [2]
>>>
>> https://github.com/apache/systemml/commit/859ad3a72906c67b7300c9980251da4cde9ed8f8
>>>
>>> [3]
>>>
>> https://github.com/apache/systemml/tree/master/src/main/java/org/apache/sysds/common
>>>
>>> [4]
>>>
>> https://github.com/apache/systemml/blob/master/src/main/java/org/apache/sysds/common/Builtins.java
>>>
>>>
>>> ________________________________
>>> From: arnab phani <ph...@gmail.com>
>>> Sent: Sunday, May 31, 2020 3:15:44 PM
>>> To: dev@systemml.apache.org
>>> Subject: Re: [DISCUSSION] Website stack `systemml.apache.org`. Thanks.
>>>
>>> Hi,
>>>
>>> Another advantage of markdown syntax is that it gives more freedom to
>> add a
>>> description as needed. Java methods and builtins might not fall in the
>> same
>>> bucket and different builtins might need different ways to describe it.
>>> For example, glm needs more details than other built-ins. Too much
>>> standardization can be a bad idea.
>>>
>>> Regards,
>>> Arnab.
>>>
>>> On Sun, May 31, 2020 at 9:55 AM Baunsgaard, Sebastian <
>>> baunsgaard@tugraz.at>
>>> wrote:
>>>
>>>> Hi Janardhan,
>>>>
>>>>
>>>> That would be one option, lets call it option 1, I have another
>>> suggestion:
>>>>
>>>>
>>>> 1. Markdown syntax with out commented individual lines.
>>>> 2. Jdoc like syntax with annotations
>>>>
>>>> Pros and cons
>>>>
>>>> * 1 Pros
>>>> * Easy Syntax most ppl know it.
>>>> * Easy to implement.
>>>> * 1 Cons
>>>> * Enables custom docs (everyone make their own distinct format
>>>> again)
>>>> * No standard way of verifying correctness of the docs (correct
>>>> parameter names etc.)
>>>> * No standard way of marking Input, return parameters.
>>>> * Requires modified Syntax with # at each line. where # normally
>>>> means header in markdown
>>>> * 2 Pros
>>>> * Easy Syntax Java ppl know it
>>>> * Standard way of making input
>>>> * Do not have to change any syntax at all from Jdoc
>>>> * Already supported syntax in DML
>>>> * 2 Cons
>>>> * Harder to implement since one would have to look into java doc
>>>> extraction and what is needed to support that, maybe we have to make
>>> such a
>>>> thing ourselves?
>>>>
>>>> I personally like option 2 with Jdoc more and then extending into
>>>> automatically parsing and making the either markdown files or HTML
>> files
>>>> for the docs that you either way have to do in option 1.
>>>>
>>>> Best regards
>>>> Sebastian
>>>> ________________________________
>>>> From: Janardhan <ja...@gmail.com>
>>>> Sent: Sunday, May 31, 2020 9:25:43 AM
>>>> To: dev@systemml.apache.org
>>>> Subject: Re: [DISCUSSION] Website stack `systemml.apache.org`. Thanks.
>>>>
>>>> Hi Sebastian, :)
>>>>
>>>> Since the builtin DML files do not have full documentation yet, Can we
>>>> write
>>>> markdown syntax with the following heading in each dml file itself!
>>>>
>>>> 1. Function description
>>>> 2. Usage
>>>> 3. Arguments
>>>> 4. Returns
>>>> 5. Usage
>>>>
>>>> After that, as you've mentioned can be parsed with only removing # at
>> the
>>>> start of each line, and write
>>>> all this data to one `builtins-reference.md` file.
>>>>
>>>> Thank you,
>>>> Janardhan
>>>>
>>>> On Tue, May 26, 2020 at 5:05 PM Baunsgaard, Sebastian <
>>>> baunsgaard@tugraz.at>
>>>> wrote:
>>>>
>>>>>
>>>>> If this is meant as a question then, yes from me (this probably calls
>>> for
>>>>> a vote?), if:
>>>>>
>>>>>
>>>>> 1. someone is able to change the settings for the repository, to
>> use
>>>>> the docs folder as the GitHub docs folder.
>>>>> 2. someone wants to make a PR containing the content of the
>> gh-pages
>>>>> branch.
>>>>>
>>>>> Best regards
>>>>> Sebastian
>>>>> ________________________________
>>>>> From: Janardhan <ja...@gmail.com>
>>>>> Sent: Tuesday, May 26, 2020 1:22:11 PM
>>>>> To: dev@systemml.apache.org
>>>>> Subject: Re: [DISCUSSION] Website stack `systemml.apache.org`.
>> Thanks.
>>>>>
>>>>> Hi Sebastian,
>>>>>
>>>>> Can we move the `docs` folder in the existing `master` branch
>>>>> and merge the `gh-pages` branch to `docs` folder in the top-level
>>>>> directory. :)
>>>>>
>>>>> - Janardhan
>>>>>
>>>>> On Sun, May 24, 2020 at 3:28 PM Baunsgaard, Sebastian <
>>>>> baunsgaard@tugraz.at>
>>>>> wrote:
>>>>>
>>>>>> Hi Janardhan,
>>>>>>
>>>>>>
>>>>>> Reworking the documentation seems like a really good idea!
>>>>>>
>>>>>>
>>>>>> A first step in my opinion is to start in the main repository.
>>>>>>
>>>>>>
>>>>>> I would suggest the following:
>>>>>>
>>>>>>
>>>>>> - Merge the current gh-pages branch into master docs folder.
>>>>>>
>>>>>> - Change the documentations page :
>> http://apache.github.io/systemml/
>>> SystemDS Documentation - SystemDS<http://apache.github.io/systemml/>
>>> apache.github.io
>>> SystemDS Documentation
>>>
>>>
>>>> to
>>>>>> use the docs folder for documentation. Should be doable inside the
>>>>> settings
>>>>>> on the repository.
>>>>>>
>>>>>> - Delete the gh-pages branch
>>>>>>
>>>>>>
>>>>>> I want this change because then new PRs can change the
>> documentation
>>> in
>>>>>> one PR rather than two (which to be honest is not going to happen
>>> most
>>>> of
>>>>>> the time).
>>>>>>
>>>>>> The downside (we have to mention this) is that when you clone the
>>>>>> repository, you have slightly larger downloads, but i think the
>>>> trade-off
>>>>>> is fair. If you see any other issues please mention these in this
>>>> thread.
>>>>>>
>>>>>>
>>>>>> Next task I would like to suggest to include:
>>>>>>
>>>>>>
>>>>>> - Java docs generated and
>>>>>>
>>>>>> - Python docs generated to also be included as a sub-folder in
>> docs,
>>> so
>>>>>> that the current webpage contains the docs for that to.
>>>>>>
>>>>>>
>>>>>> How to include these in a sensible manner is currently a good
>>> question,
>>>>>> and has to be seriously well considered. Including these will also
>>>> enable
>>>>>> us to move documentation closer to the individual code parts,
>> making
>>> it
>>>>>> again more likely to get the documentation done.
>>>>>>
>>>>>>
>>>>>> Another more programmatic task is to make a parser for our DML
>> files,
>>>>> such
>>>>>> that we can generate webpage documentation based on the
>> documentation
>>>>>> syntax used (much like the python and java docs generator). This
>>> would
>>>> be
>>>>>> great since it will remove the duplication of documentation in the
>>>>>> individual DML files, and the documentation we have already in
>> files
>>>> such
>>>>>> as: docs/dml-language-reference.md.
>>>>>>
>>>>>> This also gives us an excuse to clean up that documentation/scripts
>>>> which
>>>>>> has very different implementations in our scripts folder:
>>>>>>
>>>>>>
>>>>>> examples:
>>>>>>
>>>>>>
>>>>>> - <https://github.com/apache/systemml/tree/master/scripts>
>>> [https://avatars3.githubusercontent.com/u/47359?s=400&v=4]<
>>> https://github.com/apache/systemml/tree/master/scripts>
>>>
>>> systemml/scripts at master · apache/systemml · GitHub<
>>> https://github.com/apache/systemml/tree/master/scripts>
>>> github.com
>>> Mirror of Apache SystemML. Contribute to apache/systemml development by
>>> creating an account on GitHub.
>>>
>>>
>>>>>>
>>>>>
>>>>
>>>
>> https://github.com/apache/systemml/blob/master/scripts/nn/layers/batch_norm1d.dml
>>> [https://avatars3.githubusercontent.com/u/47359?s=400&v=4]<
>>>
>> https://github.com/apache/systemml/blob/master/scripts/nn/layers/batch_norm1d.dml
>>>>
>>>
>>> systemml/batch_norm1d.dml at master · apache/systemml · GitHub<
>>>
>> https://github.com/apache/systemml/blob/master/scripts/nn/layers/batch_norm1d.dml
>>>>
>>> github.com
>>> Mirror of Apache SystemML. Contribute to apache/systemml development by
>>> creating an account on GitHub.
>>>
>>>
>>>>>> <
>>>>>>
>>>>>
>>>>
>>>
>> https://github.com/Baunsgaard/systemml/blob/master/scripts/nn/layers/batch_norm2d.dml
>>> [https://avatars2.githubusercontent.com/u/9947148?s=400&v=4]<
>>>
>> https://github.com/Baunsgaard/systemml/blob/master/scripts/nn/layers/batch_norm2d.dml
>>>>
>>>
>>> systemml/batch_norm2d.dml at master · Baunsgaard/systemml · GitHub<
>>>
>> https://github.com/Baunsgaard/systemml/blob/master/scripts/nn/layers/batch_norm2d.dml
>>>>
>>> github.com
>>> Mirror of Apache SystemML. Contribute to Baunsgaard/systemml development
>>> by creating an account on GitHub.
>>>
>>>
>>>>>>>
>>>>>>
>>>>>> - <
>>>>>>
>>>>>
>>>>
>>>
>> https://github.com/Baunsgaard/systemml/blob/master/scripts/builtin/glm.dml
>>> [https://avatars2.githubusercontent.com/u/9947148?s=400&v=4]<
>>>
>> https://github.com/Baunsgaard/systemml/blob/master/scripts/builtin/glm.dml
>>>>
>>>
>>> systemml/glm.dml at master · Baunsgaard/systemml · GitHub<
>>>
>> https://github.com/Baunsgaard/systemml/blob/master/scripts/builtin/glm.dml
>>>>
>>> github.com
>>> Mirror of Apache SystemML. Contribute to Baunsgaard/systemml development
>>> by creating an account on GitHub.
>>>
>>>
>>>>>>
>>>>>> <https://github.com/apache/systemml/tree/master/scripts>
>>> [https://avatars3.githubusercontent.com/u/47359?s=400&v=4]<
>>> https://github.com/apache/systemml/tree/master/scripts>
>>>
>>> systemml/scripts at master · apache/systemml · GitHub<
>>> https://github.com/apache/systemml/tree/master/scripts>
>>> github.com
>>> Mirror of Apache SystemML. Contribute to apache/systemml development by
>>> creating an account on GitHub.
>>>
>>>
>>>>>>
>>>>>
>>>>
>>>
>> https://github.com/apache/systemml/blob/master/scripts/algorithms/Cox.dml
>>> [https://avatars3.githubusercontent.com/u/47359?s=400&v=4]<
>>>
>> https://github.com/apache/systemml/blob/master/scripts/algorithms/Cox.dml>
>>>
>>> systemml/Cox.dml at master · apache/systemml · GitHub<
>>>
>> https://github.com/apache/systemml/blob/master/scripts/algorithms/Cox.dml>
>>> github.com
>>> Mirror of Apache SystemML. Contribute to apache/systemml development by
>>> creating an account on GitHub.
>>>
>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> Furthermore, I would suggest to postpone changing the main website:
>>>>>>
>>>>>> https://github.com/apache/systemml-website
>>> [https://avatars3.githubusercontent.com/u/47359?s=400&v=4]<
>>> https://github.com/apache/systemml-website>
>>>
>>> GitHub - apache/systemml-website: Mirror of Apache SystemML site<
>>> https://github.com/apache/systemml-website>
>>> github.com
>>> Mirror of Apache SystemML site. Contribute to apache/systemml-website
>>> development by creating an account on GitHub.
>>>
>>>
>>>>>>
>>>>>> But that task has to be started when the next release is scheduled,
>>>>>> because then we need to find out how to copy the current
>>> documentation
>>>>> to a
>>>>>> archive list of static webpages located:
>>>>>>
>>>>>> https://systemml.apache.org/documentation
>>> Documentation<https://systemml.apache.org/documentation>
>>> systemml.apache.org
>>> Project Documentation Page
>>>
>>>
>>>>>>
>>>>>>
>>>>>> best regards
>>>>>>
>>>>>> Sebastian
>>>>>>
>>>>>> ________________________________
>>>>>> From: Janardhan <ja...@apache.org>
>>>>>> Sent: Sunday, May 24, 2020 11:02:53 AM
>>>>>> To: dev@systemml.apache.org
>>>>>> Subject: [DISCUSSION] Website stack `systemml.apache.org`. Thanks.
>>>>>>
>>>>>> Hi Sebastian,
>>>>>>
>>>>>> In our discussion[1][2] a while ago, there was a topic about
>> changing
>>>>>> website
>>>>>> stack.
>>>>>>
>>>>>> Let us start a discussion about this in this thread.
>>>>>>
>>>>>> We are planning to work on this, after a feasibility study and tech
>>>> stack
>>>>>> vetting.
>>>>>>
>>>>>> Anybody would like to give suggestions would be great.
>>>>>>
>>>>>> [1] https://github.com/apache/systemml/pull/877
>>> [https://avatars3.githubusercontent.com/u/47359?s=400&v=4]<
>>> https://github.com/apache/systemml/pull/877>
>>>
>>> [MINOR][DOC] Name refactor SystemML to SystemDS in Documentation by
>>> Baunsgaard · Pull Request #877 · apache/systemml · GitHub<
>>> https://github.com/apache/systemml/pull/877>
>>> github.com
>>> Brute change of documentation page to reflect name change. To fully
>>> enforce this two more steps are needed:
>>> https://github.com/apache/systemml-website Access to SystemML website
>>> server on https://systemml.apache.org/ Furthermore, we might want to
>>> consider starting fresh in the documentation.
>>>
>>>
>>>>>> [2] https://github.com/apache/systemml-website/pull/66
>>> [https://avatars3.githubusercontent.com/u/47359?s=400&v=4]<
>>> https://github.com/apache/systemml-website/pull/66>
>>>
>>> [SYSTEMML-2539][BUILD] Upgrade build configuration by j143 · Pull Request
>>> #66 · apache/systemml-website · GitHub<
>>> https://github.com/apache/systemml-website/pull/66>
>>> github.com
>>> Upgrade gulp to 4.0 Since the task api is deprecated, fuctions were used,
>>> and exported as tasks. Documentation for gulpfile.js. Stick to Susy2,
>> since
>>> mixins are not supported in 3.0 Upgrade npm dependencies to state-of-art.
>>> Commit package-lock.json file to version control.
>>>
>>>
>>>>>>
>>>>>> Thank you,
>>>>>> Janardhan
>>>>>>
>>>>>
>>>>
>>>
>>
>
Re: [DISCUSSION] Website stack `systemml.apache.org`. Thanks.
Posted by Janardhan <ja...@gmail.com>.
Hi Arnab,
We have all the Java-based builtin functions documented at [1].
Most of these functions are already documented!
[1]
http://apache.github.io/systemml/dml-language-reference.html#built-in-functions
Thank you,
Janardhan
On Fri, Jun 5, 2020 at 4:37 PM arnab phani <ph...@gmail.com> wrote:
> That's a good suggestion. One thing I want to point out that Builtins.java
> [1] contains all the dml-bodied and java-bodied built-ins. For the dml
> built-ins, it is easier to parse the corresponding dml script (if
> documentation is available there), but for other built-ins, I am not aware
> of any particular source today where they are documented -- a few of them
> are documented as a part of commit message but not all.
> Though it is certainly possible to slice the dml builtins from [1] out, but
> it will be awesome to generate docs for all of those.
>
> [1]
>
> https://github.com/apache/systemml/blob/master/src/main/java/org/apache/sysds/common/Builtins.java
>
> Regards,
> Arnab..
>
> On Thu, Jun 4, 2020 at 12:44 PM Baunsgaard, Sebastian <
> baunsgaard@tugraz.at>
> wrote:
>
> > Hi dev team!
> >
> >
> > Update / suggestion.
> >
> >
> > First of all, i think it is great that we have people working on our
> > documentation. It is really important. But I can't help to note that the
> > improvements, are going in an unmaintainable direction where we have to
> > maintain the documentation at multiple locations.
> >
> >
> > For instance our new toOneHot function already had some basic
> > documentation inside its build-in script already [1], it would be much
> > nicer if we modified that and then had some automation to generate the
> docs
> > made in commit [2]. Now we have to maintain two locations, resulting in
> > high risk of either being outdated.
> >
> >
> > We need to ask ourselves where do we want to make the edit of
> > documentation. I would argue that it should be located as close to the
> > internal definition of a function, because then when making or modifying
> > that function the documentation is also reviewed.
> >
> >
> > To achieve this i suggest that we instead relocate the documentation
> > source and generation to an appropriately named file inside [3], that
> > simply loops through all function definitions from [4], and generate
> > markdown files for our /docs/ folder.
> >
> >
> > Shifting to such an implementation enables us to systematically cover all
> > functions defined in the system. It also is a full list of the systems
> > current build-in functions thereby giving us the exact file paths to
> > extract the source documentation from the comments and generate the docs
> > for those files.
> >
> > In the future it could be extended to verify and detecting which input
> and
> > output types each function definition has, such that we have even less
> > manual documentation needs.
> >
> >
> > Best regards
> >
> > Sebastian
> >
> >
> > [1]
> >
> https://github.com/apache/systemml/blob/master/scripts/builtin/toOneHot.dml
> >
> > [2]
> >
> https://github.com/apache/systemml/commit/859ad3a72906c67b7300c9980251da4cde9ed8f8
> >
> > [3]
> >
> https://github.com/apache/systemml/tree/master/src/main/java/org/apache/sysds/common
> >
> > [4]
> >
> https://github.com/apache/systemml/blob/master/src/main/java/org/apache/sysds/common/Builtins.java
> >
> >
> > ________________________________
> > From: arnab phani <ph...@gmail.com>
> > Sent: Sunday, May 31, 2020 3:15:44 PM
> > To: dev@systemml.apache.org
> > Subject: Re: [DISCUSSION] Website stack `systemml.apache.org`. Thanks.
> >
> > Hi,
> >
> > Another advantage of markdown syntax is that it gives more freedom to
> add a
> > description as needed. Java methods and builtins might not fall in the
> same
> > bucket and different builtins might need different ways to describe it.
> > For example, glm needs more details than other built-ins. Too much
> > standardization can be a bad idea.
> >
> > Regards,
> > Arnab.
> >
> > On Sun, May 31, 2020 at 9:55 AM Baunsgaard, Sebastian <
> > baunsgaard@tugraz.at>
> > wrote:
> >
> > > Hi Janardhan,
> > >
> > >
> > > That would be one option, lets call it option 1, I have another
> > suggestion:
> > >
> > >
> > > 1. Markdown syntax with out commented individual lines.
> > > 2. Jdoc like syntax with annotations
> > >
> > > Pros and cons
> > >
> > > * 1 Pros
> > > * Easy Syntax most ppl know it.
> > > * Easy to implement.
> > > * 1 Cons
> > > * Enables custom docs (everyone make their own distinct format
> > > again)
> > > * No standard way of verifying correctness of the docs (correct
> > > parameter names etc.)
> > > * No standard way of marking Input, return parameters.
> > > * Requires modified Syntax with # at each line. where # normally
> > > means header in markdown
> > > * 2 Pros
> > > * Easy Syntax Java ppl know it
> > > * Standard way of making input
> > > * Do not have to change any syntax at all from Jdoc
> > > * Already supported syntax in DML
> > > * 2 Cons
> > > * Harder to implement since one would have to look into java doc
> > > extraction and what is needed to support that, maybe we have to make
> > such a
> > > thing ourselves?
> > >
> > > I personally like option 2 with Jdoc more and then extending into
> > > automatically parsing and making the either markdown files or HTML
> files
> > > for the docs that you either way have to do in option 1.
> > >
> > > Best regards
> > > Sebastian
> > > ________________________________
> > > From: Janardhan <ja...@gmail.com>
> > > Sent: Sunday, May 31, 2020 9:25:43 AM
> > > To: dev@systemml.apache.org
> > > Subject: Re: [DISCUSSION] Website stack `systemml.apache.org`. Thanks.
> > >
> > > Hi Sebastian, :)
> > >
> > > Since the builtin DML files do not have full documentation yet, Can we
> > > write
> > > markdown syntax with the following heading in each dml file itself!
> > >
> > > 1. Function description
> > > 2. Usage
> > > 3. Arguments
> > > 4. Returns
> > > 5. Usage
> > >
> > > After that, as you've mentioned can be parsed with only removing # at
> the
> > > start of each line, and write
> > > all this data to one `builtins-reference.md` file.
> > >
> > > Thank you,
> > > Janardhan
> > >
> > > On Tue, May 26, 2020 at 5:05 PM Baunsgaard, Sebastian <
> > > baunsgaard@tugraz.at>
> > > wrote:
> > >
> > > >
> > > > If this is meant as a question then, yes from me (this probably calls
> > for
> > > > a vote?), if:
> > > >
> > > >
> > > > 1. someone is able to change the settings for the repository, to
> use
> > > > the docs folder as the GitHub docs folder.
> > > > 2. someone wants to make a PR containing the content of the
> gh-pages
> > > > branch.
> > > >
> > > > Best regards
> > > > Sebastian
> > > > ________________________________
> > > > From: Janardhan <ja...@gmail.com>
> > > > Sent: Tuesday, May 26, 2020 1:22:11 PM
> > > > To: dev@systemml.apache.org
> > > > Subject: Re: [DISCUSSION] Website stack `systemml.apache.org`.
> Thanks.
> > > >
> > > > Hi Sebastian,
> > > >
> > > > Can we move the `docs` folder in the existing `master` branch
> > > > and merge the `gh-pages` branch to `docs` folder in the top-level
> > > > directory. :)
> > > >
> > > > - Janardhan
> > > >
> > > > On Sun, May 24, 2020 at 3:28 PM Baunsgaard, Sebastian <
> > > > baunsgaard@tugraz.at>
> > > > wrote:
> > > >
> > > > > Hi Janardhan,
> > > > >
> > > > >
> > > > > Reworking the documentation seems like a really good idea!
> > > > >
> > > > >
> > > > > A first step in my opinion is to start in the main repository.
> > > > >
> > > > >
> > > > > I would suggest the following:
> > > > >
> > > > >
> > > > > - Merge the current gh-pages branch into master docs folder.
> > > > >
> > > > > - Change the documentations page :
> http://apache.github.io/systemml/
> > SystemDS Documentation - SystemDS<http://apache.github.io/systemml/>
> > apache.github.io
> > SystemDS Documentation
> >
> >
> > > to
> > > > > use the docs folder for documentation. Should be doable inside the
> > > > settings
> > > > > on the repository.
> > > > >
> > > > > - Delete the gh-pages branch
> > > > >
> > > > >
> > > > > I want this change because then new PRs can change the
> documentation
> > in
> > > > > one PR rather than two (which to be honest is not going to happen
> > most
> > > of
> > > > > the time).
> > > > >
> > > > > The downside (we have to mention this) is that when you clone the
> > > > > repository, you have slightly larger downloads, but i think the
> > > trade-off
> > > > > is fair. If you see any other issues please mention these in this
> > > thread.
> > > > >
> > > > >
> > > > > Next task I would like to suggest to include:
> > > > >
> > > > >
> > > > > - Java docs generated and
> > > > >
> > > > > - Python docs generated to also be included as a sub-folder in
> docs,
> > so
> > > > > that the current webpage contains the docs for that to.
> > > > >
> > > > >
> > > > > How to include these in a sensible manner is currently a good
> > question,
> > > > > and has to be seriously well considered. Including these will also
> > > enable
> > > > > us to move documentation closer to the individual code parts,
> making
> > it
> > > > > again more likely to get the documentation done.
> > > > >
> > > > >
> > > > > Another more programmatic task is to make a parser for our DML
> files,
> > > > such
> > > > > that we can generate webpage documentation based on the
> documentation
> > > > > syntax used (much like the python and java docs generator). This
> > would
> > > be
> > > > > great since it will remove the duplication of documentation in the
> > > > > individual DML files, and the documentation we have already in
> files
> > > such
> > > > > as: docs/dml-language-reference.md.
> > > > >
> > > > > This also gives us an excuse to clean up that documentation/scripts
> > > which
> > > > > has very different implementations in our scripts folder:
> > > > >
> > > > >
> > > > > examples:
> > > > >
> > > > >
> > > > > - <https://github.com/apache/systemml/tree/master/scripts>
> > [https://avatars3.githubusercontent.com/u/47359?s=400&v=4]<
> > https://github.com/apache/systemml/tree/master/scripts>
> >
> > systemml/scripts at master · apache/systemml · GitHub<
> > https://github.com/apache/systemml/tree/master/scripts>
> > github.com
> > Mirror of Apache SystemML. Contribute to apache/systemml development by
> > creating an account on GitHub.
> >
> >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/systemml/blob/master/scripts/nn/layers/batch_norm1d.dml
> > [https://avatars3.githubusercontent.com/u/47359?s=400&v=4]<
> >
> https://github.com/apache/systemml/blob/master/scripts/nn/layers/batch_norm1d.dml
> > >
> >
> > systemml/batch_norm1d.dml at master · apache/systemml · GitHub<
> >
> https://github.com/apache/systemml/blob/master/scripts/nn/layers/batch_norm1d.dml
> > >
> > github.com
> > Mirror of Apache SystemML. Contribute to apache/systemml development by
> > creating an account on GitHub.
> >
> >
> > > > > <
> > > > >
> > > >
> > >
> >
> https://github.com/Baunsgaard/systemml/blob/master/scripts/nn/layers/batch_norm2d.dml
> > [https://avatars2.githubusercontent.com/u/9947148?s=400&v=4]<
> >
> https://github.com/Baunsgaard/systemml/blob/master/scripts/nn/layers/batch_norm2d.dml
> > >
> >
> > systemml/batch_norm2d.dml at master · Baunsgaard/systemml · GitHub<
> >
> https://github.com/Baunsgaard/systemml/blob/master/scripts/nn/layers/batch_norm2d.dml
> > >
> > github.com
> > Mirror of Apache SystemML. Contribute to Baunsgaard/systemml development
> > by creating an account on GitHub.
> >
> >
> > > > > >
> > > > >
> > > > > - <
> > > > >
> > > >
> > >
> >
> https://github.com/Baunsgaard/systemml/blob/master/scripts/builtin/glm.dml
> > [https://avatars2.githubusercontent.com/u/9947148?s=400&v=4]<
> >
> https://github.com/Baunsgaard/systemml/blob/master/scripts/builtin/glm.dml
> > >
> >
> > systemml/glm.dml at master · Baunsgaard/systemml · GitHub<
> >
> https://github.com/Baunsgaard/systemml/blob/master/scripts/builtin/glm.dml
> > >
> > github.com
> > Mirror of Apache SystemML. Contribute to Baunsgaard/systemml development
> > by creating an account on GitHub.
> >
> >
> > > > >
> > > > > <https://github.com/apache/systemml/tree/master/scripts>
> > [https://avatars3.githubusercontent.com/u/47359?s=400&v=4]<
> > https://github.com/apache/systemml/tree/master/scripts>
> >
> > systemml/scripts at master · apache/systemml · GitHub<
> > https://github.com/apache/systemml/tree/master/scripts>
> > github.com
> > Mirror of Apache SystemML. Contribute to apache/systemml development by
> > creating an account on GitHub.
> >
> >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/systemml/blob/master/scripts/algorithms/Cox.dml
> > [https://avatars3.githubusercontent.com/u/47359?s=400&v=4]<
> >
> https://github.com/apache/systemml/blob/master/scripts/algorithms/Cox.dml>
> >
> > systemml/Cox.dml at master · apache/systemml · GitHub<
> >
> https://github.com/apache/systemml/blob/master/scripts/algorithms/Cox.dml>
> > github.com
> > Mirror of Apache SystemML. Contribute to apache/systemml development by
> > creating an account on GitHub.
> >
> >
> > > > >
> > > > >
> > > > >
> > > > > Furthermore, I would suggest to postpone changing the main website:
> > > > >
> > > > > https://github.com/apache/systemml-website
> > [https://avatars3.githubusercontent.com/u/47359?s=400&v=4]<
> > https://github.com/apache/systemml-website>
> >
> > GitHub - apache/systemml-website: Mirror of Apache SystemML site<
> > https://github.com/apache/systemml-website>
> > github.com
> > Mirror of Apache SystemML site. Contribute to apache/systemml-website
> > development by creating an account on GitHub.
> >
> >
> > > > >
> > > > > But that task has to be started when the next release is scheduled,
> > > > > because then we need to find out how to copy the current
> > documentation
> > > > to a
> > > > > archive list of static webpages located:
> > > > >
> > > > > https://systemml.apache.org/documentation
> > Documentation<https://systemml.apache.org/documentation>
> > systemml.apache.org
> > Project Documentation Page
> >
> >
> > > > >
> > > > >
> > > > > best regards
> > > > >
> > > > > Sebastian
> > > > >
> > > > > ________________________________
> > > > > From: Janardhan <ja...@apache.org>
> > > > > Sent: Sunday, May 24, 2020 11:02:53 AM
> > > > > To: dev@systemml.apache.org
> > > > > Subject: [DISCUSSION] Website stack `systemml.apache.org`. Thanks.
> > > > >
> > > > > Hi Sebastian,
> > > > >
> > > > > In our discussion[1][2] a while ago, there was a topic about
> changing
> > > > > website
> > > > > stack.
> > > > >
> > > > > Let us start a discussion about this in this thread.
> > > > >
> > > > > We are planning to work on this, after a feasibility study and tech
> > > stack
> > > > > vetting.
> > > > >
> > > > > Anybody would like to give suggestions would be great.
> > > > >
> > > > > [1] https://github.com/apache/systemml/pull/877
> > [https://avatars3.githubusercontent.com/u/47359?s=400&v=4]<
> > https://github.com/apache/systemml/pull/877>
> >
> > [MINOR][DOC] Name refactor SystemML to SystemDS in Documentation by
> > Baunsgaard · Pull Request #877 · apache/systemml · GitHub<
> > https://github.com/apache/systemml/pull/877>
> > github.com
> > Brute change of documentation page to reflect name change. To fully
> > enforce this two more steps are needed:
> > https://github.com/apache/systemml-website Access to SystemML website
> > server on https://systemml.apache.org/ Furthermore, we might want to
> > consider starting fresh in the documentation.
> >
> >
> > > > > [2] https://github.com/apache/systemml-website/pull/66
> > [https://avatars3.githubusercontent.com/u/47359?s=400&v=4]<
> > https://github.com/apache/systemml-website/pull/66>
> >
> > [SYSTEMML-2539][BUILD] Upgrade build configuration by j143 · Pull Request
> > #66 · apache/systemml-website · GitHub<
> > https://github.com/apache/systemml-website/pull/66>
> > github.com
> > Upgrade gulp to 4.0 Since the task api is deprecated, fuctions were used,
> > and exported as tasks. Documentation for gulpfile.js. Stick to Susy2,
> since
> > mixins are not supported in 3.0 Upgrade npm dependencies to state-of-art.
> > Commit package-lock.json file to version control.
> >
> >
> > > > >
> > > > > Thank you,
> > > > > Janardhan
> > > > >
> > > >
> > >
> >
>
Re: [DISCUSSION] Website stack `systemml.apache.org`. Thanks.
Posted by arnab phani <ph...@gmail.com>.
That's a good suggestion. One thing I want to point out that Builtins.java
[1] contains all the dml-bodied and java-bodied built-ins. For the dml
built-ins, it is easier to parse the corresponding dml script (if
documentation is available there), but for other built-ins, I am not aware
of any particular source today where they are documented -- a few of them
are documented as a part of commit message but not all.
Though it is certainly possible to slice the dml builtins from [1] out, but
it will be awesome to generate docs for all of those.
[1]
https://github.com/apache/systemml/blob/master/src/main/java/org/apache/sysds/common/Builtins.java
Regards,
Arnab..
On Thu, Jun 4, 2020 at 12:44 PM Baunsgaard, Sebastian <ba...@tugraz.at>
wrote:
> Hi dev team!
>
>
> Update / suggestion.
>
>
> First of all, i think it is great that we have people working on our
> documentation. It is really important. But I can't help to note that the
> improvements, are going in an unmaintainable direction where we have to
> maintain the documentation at multiple locations.
>
>
> For instance our new toOneHot function already had some basic
> documentation inside its build-in script already [1], it would be much
> nicer if we modified that and then had some automation to generate the docs
> made in commit [2]. Now we have to maintain two locations, resulting in
> high risk of either being outdated.
>
>
> We need to ask ourselves where do we want to make the edit of
> documentation. I would argue that it should be located as close to the
> internal definition of a function, because then when making or modifying
> that function the documentation is also reviewed.
>
>
> To achieve this i suggest that we instead relocate the documentation
> source and generation to an appropriately named file inside [3], that
> simply loops through all function definitions from [4], and generate
> markdown files for our /docs/ folder.
>
>
> Shifting to such an implementation enables us to systematically cover all
> functions defined in the system. It also is a full list of the systems
> current build-in functions thereby giving us the exact file paths to
> extract the source documentation from the comments and generate the docs
> for those files.
>
> In the future it could be extended to verify and detecting which input and
> output types each function definition has, such that we have even less
> manual documentation needs.
>
>
> Best regards
>
> Sebastian
>
>
> [1]
> https://github.com/apache/systemml/blob/master/scripts/builtin/toOneHot.dml
>
> [2]
> https://github.com/apache/systemml/commit/859ad3a72906c67b7300c9980251da4cde9ed8f8
>
> [3]
> https://github.com/apache/systemml/tree/master/src/main/java/org/apache/sysds/common
>
> [4]
> https://github.com/apache/systemml/blob/master/src/main/java/org/apache/sysds/common/Builtins.java
>
>
> ________________________________
> From: arnab phani <ph...@gmail.com>
> Sent: Sunday, May 31, 2020 3:15:44 PM
> To: dev@systemml.apache.org
> Subject: Re: [DISCUSSION] Website stack `systemml.apache.org`. Thanks.
>
> Hi,
>
> Another advantage of markdown syntax is that it gives more freedom to add a
> description as needed. Java methods and builtins might not fall in the same
> bucket and different builtins might need different ways to describe it.
> For example, glm needs more details than other built-ins. Too much
> standardization can be a bad idea.
>
> Regards,
> Arnab.
>
> On Sun, May 31, 2020 at 9:55 AM Baunsgaard, Sebastian <
> baunsgaard@tugraz.at>
> wrote:
>
> > Hi Janardhan,
> >
> >
> > That would be one option, lets call it option 1, I have another
> suggestion:
> >
> >
> > 1. Markdown syntax with out commented individual lines.
> > 2. Jdoc like syntax with annotations
> >
> > Pros and cons
> >
> > * 1 Pros
> > * Easy Syntax most ppl know it.
> > * Easy to implement.
> > * 1 Cons
> > * Enables custom docs (everyone make their own distinct format
> > again)
> > * No standard way of verifying correctness of the docs (correct
> > parameter names etc.)
> > * No standard way of marking Input, return parameters.
> > * Requires modified Syntax with # at each line. where # normally
> > means header in markdown
> > * 2 Pros
> > * Easy Syntax Java ppl know it
> > * Standard way of making input
> > * Do not have to change any syntax at all from Jdoc
> > * Already supported syntax in DML
> > * 2 Cons
> > * Harder to implement since one would have to look into java doc
> > extraction and what is needed to support that, maybe we have to make
> such a
> > thing ourselves?
> >
> > I personally like option 2 with Jdoc more and then extending into
> > automatically parsing and making the either markdown files or HTML files
> > for the docs that you either way have to do in option 1.
> >
> > Best regards
> > Sebastian
> > ________________________________
> > From: Janardhan <ja...@gmail.com>
> > Sent: Sunday, May 31, 2020 9:25:43 AM
> > To: dev@systemml.apache.org
> > Subject: Re: [DISCUSSION] Website stack `systemml.apache.org`. Thanks.
> >
> > Hi Sebastian, :)
> >
> > Since the builtin DML files do not have full documentation yet, Can we
> > write
> > markdown syntax with the following heading in each dml file itself!
> >
> > 1. Function description
> > 2. Usage
> > 3. Arguments
> > 4. Returns
> > 5. Usage
> >
> > After that, as you've mentioned can be parsed with only removing # at the
> > start of each line, and write
> > all this data to one `builtins-reference.md` file.
> >
> > Thank you,
> > Janardhan
> >
> > On Tue, May 26, 2020 at 5:05 PM Baunsgaard, Sebastian <
> > baunsgaard@tugraz.at>
> > wrote:
> >
> > >
> > > If this is meant as a question then, yes from me (this probably calls
> for
> > > a vote?), if:
> > >
> > >
> > > 1. someone is able to change the settings for the repository, to use
> > > the docs folder as the GitHub docs folder.
> > > 2. someone wants to make a PR containing the content of the gh-pages
> > > branch.
> > >
> > > Best regards
> > > Sebastian
> > > ________________________________
> > > From: Janardhan <ja...@gmail.com>
> > > Sent: Tuesday, May 26, 2020 1:22:11 PM
> > > To: dev@systemml.apache.org
> > > Subject: Re: [DISCUSSION] Website stack `systemml.apache.org`. Thanks.
> > >
> > > Hi Sebastian,
> > >
> > > Can we move the `docs` folder in the existing `master` branch
> > > and merge the `gh-pages` branch to `docs` folder in the top-level
> > > directory. :)
> > >
> > > - Janardhan
> > >
> > > On Sun, May 24, 2020 at 3:28 PM Baunsgaard, Sebastian <
> > > baunsgaard@tugraz.at>
> > > wrote:
> > >
> > > > Hi Janardhan,
> > > >
> > > >
> > > > Reworking the documentation seems like a really good idea!
> > > >
> > > >
> > > > A first step in my opinion is to start in the main repository.
> > > >
> > > >
> > > > I would suggest the following:
> > > >
> > > >
> > > > - Merge the current gh-pages branch into master docs folder.
> > > >
> > > > - Change the documentations page : http://apache.github.io/systemml/
> SystemDS Documentation - SystemDS<http://apache.github.io/systemml/>
> apache.github.io
> SystemDS Documentation
>
>
> > to
> > > > use the docs folder for documentation. Should be doable inside the
> > > settings
> > > > on the repository.
> > > >
> > > > - Delete the gh-pages branch
> > > >
> > > >
> > > > I want this change because then new PRs can change the documentation
> in
> > > > one PR rather than two (which to be honest is not going to happen
> most
> > of
> > > > the time).
> > > >
> > > > The downside (we have to mention this) is that when you clone the
> > > > repository, you have slightly larger downloads, but i think the
> > trade-off
> > > > is fair. If you see any other issues please mention these in this
> > thread.
> > > >
> > > >
> > > > Next task I would like to suggest to include:
> > > >
> > > >
> > > > - Java docs generated and
> > > >
> > > > - Python docs generated to also be included as a sub-folder in docs,
> so
> > > > that the current webpage contains the docs for that to.
> > > >
> > > >
> > > > How to include these in a sensible manner is currently a good
> question,
> > > > and has to be seriously well considered. Including these will also
> > enable
> > > > us to move documentation closer to the individual code parts, making
> it
> > > > again more likely to get the documentation done.
> > > >
> > > >
> > > > Another more programmatic task is to make a parser for our DML files,
> > > such
> > > > that we can generate webpage documentation based on the documentation
> > > > syntax used (much like the python and java docs generator). This
> would
> > be
> > > > great since it will remove the duplication of documentation in the
> > > > individual DML files, and the documentation we have already in files
> > such
> > > > as: docs/dml-language-reference.md.
> > > >
> > > > This also gives us an excuse to clean up that documentation/scripts
> > which
> > > > has very different implementations in our scripts folder:
> > > >
> > > >
> > > > examples:
> > > >
> > > >
> > > > - <https://github.com/apache/systemml/tree/master/scripts>
> [https://avatars3.githubusercontent.com/u/47359?s=400&v=4]<
> https://github.com/apache/systemml/tree/master/scripts>
>
> systemml/scripts at master · apache/systemml · GitHub<
> https://github.com/apache/systemml/tree/master/scripts>
> github.com
> Mirror of Apache SystemML. Contribute to apache/systemml development by
> creating an account on GitHub.
>
>
> > > >
> > >
> >
> https://github.com/apache/systemml/blob/master/scripts/nn/layers/batch_norm1d.dml
> [https://avatars3.githubusercontent.com/u/47359?s=400&v=4]<
> https://github.com/apache/systemml/blob/master/scripts/nn/layers/batch_norm1d.dml
> >
>
> systemml/batch_norm1d.dml at master · apache/systemml · GitHub<
> https://github.com/apache/systemml/blob/master/scripts/nn/layers/batch_norm1d.dml
> >
> github.com
> Mirror of Apache SystemML. Contribute to apache/systemml development by
> creating an account on GitHub.
>
>
> > > > <
> > > >
> > >
> >
> https://github.com/Baunsgaard/systemml/blob/master/scripts/nn/layers/batch_norm2d.dml
> [https://avatars2.githubusercontent.com/u/9947148?s=400&v=4]<
> https://github.com/Baunsgaard/systemml/blob/master/scripts/nn/layers/batch_norm2d.dml
> >
>
> systemml/batch_norm2d.dml at master · Baunsgaard/systemml · GitHub<
> https://github.com/Baunsgaard/systemml/blob/master/scripts/nn/layers/batch_norm2d.dml
> >
> github.com
> Mirror of Apache SystemML. Contribute to Baunsgaard/systemml development
> by creating an account on GitHub.
>
>
> > > > >
> > > >
> > > > - <
> > > >
> > >
> >
> https://github.com/Baunsgaard/systemml/blob/master/scripts/builtin/glm.dml
> [https://avatars2.githubusercontent.com/u/9947148?s=400&v=4]<
> https://github.com/Baunsgaard/systemml/blob/master/scripts/builtin/glm.dml
> >
>
> systemml/glm.dml at master · Baunsgaard/systemml · GitHub<
> https://github.com/Baunsgaard/systemml/blob/master/scripts/builtin/glm.dml
> >
> github.com
> Mirror of Apache SystemML. Contribute to Baunsgaard/systemml development
> by creating an account on GitHub.
>
>
> > > >
> > > > <https://github.com/apache/systemml/tree/master/scripts>
> [https://avatars3.githubusercontent.com/u/47359?s=400&v=4]<
> https://github.com/apache/systemml/tree/master/scripts>
>
> systemml/scripts at master · apache/systemml · GitHub<
> https://github.com/apache/systemml/tree/master/scripts>
> github.com
> Mirror of Apache SystemML. Contribute to apache/systemml development by
> creating an account on GitHub.
>
>
> > > >
> > >
> >
> https://github.com/apache/systemml/blob/master/scripts/algorithms/Cox.dml
> [https://avatars3.githubusercontent.com/u/47359?s=400&v=4]<
> https://github.com/apache/systemml/blob/master/scripts/algorithms/Cox.dml>
>
> systemml/Cox.dml at master · apache/systemml · GitHub<
> https://github.com/apache/systemml/blob/master/scripts/algorithms/Cox.dml>
> github.com
> Mirror of Apache SystemML. Contribute to apache/systemml development by
> creating an account on GitHub.
>
>
> > > >
> > > >
> > > >
> > > > Furthermore, I would suggest to postpone changing the main website:
> > > >
> > > > https://github.com/apache/systemml-website
> [https://avatars3.githubusercontent.com/u/47359?s=400&v=4]<
> https://github.com/apache/systemml-website>
>
> GitHub - apache/systemml-website: Mirror of Apache SystemML site<
> https://github.com/apache/systemml-website>
> github.com
> Mirror of Apache SystemML site. Contribute to apache/systemml-website
> development by creating an account on GitHub.
>
>
> > > >
> > > > But that task has to be started when the next release is scheduled,
> > > > because then we need to find out how to copy the current
> documentation
> > > to a
> > > > archive list of static webpages located:
> > > >
> > > > https://systemml.apache.org/documentation
> Documentation<https://systemml.apache.org/documentation>
> systemml.apache.org
> Project Documentation Page
>
>
> > > >
> > > >
> > > > best regards
> > > >
> > > > Sebastian
> > > >
> > > > ________________________________
> > > > From: Janardhan <ja...@apache.org>
> > > > Sent: Sunday, May 24, 2020 11:02:53 AM
> > > > To: dev@systemml.apache.org
> > > > Subject: [DISCUSSION] Website stack `systemml.apache.org`. Thanks.
> > > >
> > > > Hi Sebastian,
> > > >
> > > > In our discussion[1][2] a while ago, there was a topic about changing
> > > > website
> > > > stack.
> > > >
> > > > Let us start a discussion about this in this thread.
> > > >
> > > > We are planning to work on this, after a feasibility study and tech
> > stack
> > > > vetting.
> > > >
> > > > Anybody would like to give suggestions would be great.
> > > >
> > > > [1] https://github.com/apache/systemml/pull/877
> [https://avatars3.githubusercontent.com/u/47359?s=400&v=4]<
> https://github.com/apache/systemml/pull/877>
>
> [MINOR][DOC] Name refactor SystemML to SystemDS in Documentation by
> Baunsgaard · Pull Request #877 · apache/systemml · GitHub<
> https://github.com/apache/systemml/pull/877>
> github.com
> Brute change of documentation page to reflect name change. To fully
> enforce this two more steps are needed:
> https://github.com/apache/systemml-website Access to SystemML website
> server on https://systemml.apache.org/ Furthermore, we might want to
> consider starting fresh in the documentation.
>
>
> > > > [2] https://github.com/apache/systemml-website/pull/66
> [https://avatars3.githubusercontent.com/u/47359?s=400&v=4]<
> https://github.com/apache/systemml-website/pull/66>
>
> [SYSTEMML-2539][BUILD] Upgrade build configuration by j143 · Pull Request
> #66 · apache/systemml-website · GitHub<
> https://github.com/apache/systemml-website/pull/66>
> github.com
> Upgrade gulp to 4.0 Since the task api is deprecated, fuctions were used,
> and exported as tasks. Documentation for gulpfile.js. Stick to Susy2, since
> mixins are not supported in 3.0 Upgrade npm dependencies to state-of-art.
> Commit package-lock.json file to version control.
>
>
> > > >
> > > > Thank you,
> > > > Janardhan
> > > >
> > >
> >
>
Re: [DISCUSSION] Website stack `systemml.apache.org`. Thanks.
Posted by Janardhan <ja...@gmail.com>.
Thank you, Sebastian. I've noticed toOneHot previously, only this function
seems to be duplicated!
Right now, there are five contributors who will be here only for the next
five days. For this reason we are working on plain md files, which further
can be transferred to corresponding DML files if I we want to parse with
one of the options we have previously discussed (taking into account what
Arnab has mentioned about md!)
Let us continue our discussion about parsing DML files, shall we start with
any already implemented parsers at other projects?
Thank you,
Janardhan
On Thu, 4 Jun, 2020, 16:14 Baunsgaard, Sebastian, <ba...@tugraz.at>
wrote:
> Hi dev team!
>
>
> Update / suggestion.
>
>
> First of all, i think it is great that we have people working on our
> documentation. It is really important. But I can't help to note that the
> improvements, are going in an unmaintainable direction where we have to
> maintain the documentation at multiple locations.
>
>
> For instance our new toOneHot function already had some basic
> documentation inside its build-in script already [1], it would be much
> nicer if we modified that and then had some automation to generate the docs
> made in commit [2]. Now we have to maintain two locations, resulting in
> high risk of either being outdated.
>
>
> We need to ask ourselves where do we want to make the edit of
> documentation. I would argue that it should be located as close to the
> internal definition of a function, because then when making or modifying
> that function the documentation is also reviewed.
>
>
> To achieve this i suggest that we instead relocate the documentation
> source and generation to an appropriately named file inside [3], that
> simply loops through all function definitions from [4], and generate
> markdown files for our /docs/ folder.
>
>
> Shifting to such an implementation enables us to systematically cover all
> functions defined in the system. It also is a full list of the systems
> current build-in functions thereby giving us the exact file paths to
> extract the source documentation from the comments and generate the docs
> for those files.
>
> In the future it could be extended to verify and detecting which input and
> output types each function definition has, such that we have even less
> manual documentation needs.
>
>
> Best regards
>
> Sebastian
>
>
> [1]
> https://github.com/apache/systemml/blob/master/scripts/builtin/toOneHot.dml
>
> [2]
> https://github.com/apache/systemml/commit/859ad3a72906c67b7300c9980251da4cde9ed8f8
>
> [3]
> https://github.com/apache/systemml/tree/master/src/main/java/org/apache/sysds/common
>
> [4]
> https://github.com/apache/systemml/blob/master/src/main/java/org/apache/sysds/common/Builtins.java
>
>
> ________________________________
> From: arnab phani <ph...@gmail.com>
> Sent: Sunday, May 31, 2020 3:15:44 PM
> To: dev@systemml.apache.org
> Subject: Re: [DISCUSSION] Website stack `systemml.apache.org`. Thanks.
>
> Hi,
>
> Another advantage of markdown syntax is that it gives more freedom to add a
> description as needed. Java methods and builtins might not fall in the same
> bucket and different builtins might need different ways to describe it.
> For example, glm needs more details than other built-ins. Too much
> standardization can be a bad idea.
>
> Regards,
> Arnab.
>
> On Sun, May 31, 2020 at 9:55 AM Baunsgaard, Sebastian <
> baunsgaard@tugraz.at>
> wrote:
>
> > Hi Janardhan,
> >
> >
> > That would be one option, lets call it option 1, I have another
> suggestion:
> >
> >
> > 1. Markdown syntax with out commented individual lines.
> > 2. Jdoc like syntax with annotations
> >
> > Pros and cons
> >
> > * 1 Pros
> > * Easy Syntax most ppl know it.
> > * Easy to implement.
> > * 1 Cons
> > * Enables custom docs (everyone make their own distinct format
> > again)
> > * No standard way of verifying correctness of the docs (correct
> > parameter names etc.)
> > * No standard way of marking Input, return parameters.
> > * Requires modified Syntax with # at each line. where # normally
> > means header in markdown
> > * 2 Pros
> > * Easy Syntax Java ppl know it
> > * Standard way of making input
> > * Do not have to change any syntax at all from Jdoc
> > * Already supported syntax in DML
> > * 2 Cons
> > * Harder to implement since one would have to look into java doc
> > extraction and what is needed to support that, maybe we have to make
> such a
> > thing ourselves?
> >
> > I personally like option 2 with Jdoc more and then extending into
> > automatically parsing and making the either markdown files or HTML files
> > for the docs that you either way have to do in option 1.
> >
> > Best regards
> > Sebastian
> > ________________________________
> > From: Janardhan <ja...@gmail.com>
> > Sent: Sunday, May 31, 2020 9:25:43 AM
> > To: dev@systemml.apache.org
> > Subject: Re: [DISCUSSION] Website stack `systemml.apache.org`. Thanks.
> >
> > Hi Sebastian, :)
> >
> > Since the builtin DML files do not have full documentation yet, Can we
> > write
> > markdown syntax with the following heading in each dml file itself!
> >
> > 1. Function description
> > 2. Usage
> > 3. Arguments
> > 4. Returns
> > 5. Usage
> >
> > After that, as you've mentioned can be parsed with only removing # at the
> > start of each line, and write
> > all this data to one `builtins-reference.md` file.
> >
> > Thank you,
> > Janardhan
> >
> > On Tue, May 26, 2020 at 5:05 PM Baunsgaard, Sebastian <
> > baunsgaard@tugraz.at>
> > wrote:
> >
> > >
> > > If this is meant as a question then, yes from me (this probably calls
> for
> > > a vote?), if:
> > >
> > >
> > > 1. someone is able to change the settings for the repository, to use
> > > the docs folder as the GitHub docs folder.
> > > 2. someone wants to make a PR containing the content of the gh-pages
> > > branch.
> > >
> > > Best regards
> > > Sebastian
> > > ________________________________
> > > From: Janardhan <ja...@gmail.com>
> > > Sent: Tuesday, May 26, 2020 1:22:11 PM
> > > To: dev@systemml.apache.org
> > > Subject: Re: [DISCUSSION] Website stack `systemml.apache.org`. Thanks.
> > >
> > > Hi Sebastian,
> > >
> > > Can we move the `docs` folder in the existing `master` branch
> > > and merge the `gh-pages` branch to `docs` folder in the top-level
> > > directory. :)
> > >
> > > - Janardhan
> > >
> > > On Sun, May 24, 2020 at 3:28 PM Baunsgaard, Sebastian <
> > > baunsgaard@tugraz.at>
> > > wrote:
> > >
> > > > Hi Janardhan,
> > > >
> > > >
> > > > Reworking the documentation seems like a really good idea!
> > > >
> > > >
> > > > A first step in my opinion is to start in the main repository.
> > > >
> > > >
> > > > I would suggest the following:
> > > >
> > > >
> > > > - Merge the current gh-pages branch into master docs folder.
> > > >
> > > > - Change the documentations page : http://apache.github.io/systemml/
> SystemDS Documentation - SystemDS<http://apache.github.io/systemml/>
> apache.github.io
> SystemDS Documentation
>
>
> > to
> > > > use the docs folder for documentation. Should be doable inside the
> > > settings
> > > > on the repository.
> > > >
> > > > - Delete the gh-pages branch
> > > >
> > > >
> > > > I want this change because then new PRs can change the documentation
> in
> > > > one PR rather than two (which to be honest is not going to happen
> most
> > of
> > > > the time).
> > > >
> > > > The downside (we have to mention this) is that when you clone the
> > > > repository, you have slightly larger downloads, but i think the
> > trade-off
> > > > is fair. If you see any other issues please mention these in this
> > thread.
> > > >
> > > >
> > > > Next task I would like to suggest to include:
> > > >
> > > >
> > > > - Java docs generated and
> > > >
> > > > - Python docs generated to also be included as a sub-folder in docs,
> so
> > > > that the current webpage contains the docs for that to.
> > > >
> > > >
> > > > How to include these in a sensible manner is currently a good
> question,
> > > > and has to be seriously well considered. Including these will also
> > enable
> > > > us to move documentation closer to the individual code parts, making
> it
> > > > again more likely to get the documentation done.
> > > >
> > > >
> > > > Another more programmatic task is to make a parser for our DML files,
> > > such
> > > > that we can generate webpage documentation based on the documentation
> > > > syntax used (much like the python and java docs generator). This
> would
> > be
> > > > great since it will remove the duplication of documentation in the
> > > > individual DML files, and the documentation we have already in files
> > such
> > > > as: docs/dml-language-reference.md.
> > > >
> > > > This also gives us an excuse to clean up that documentation/scripts
> > which
> > > > has very different implementations in our scripts folder:
> > > >
> > > >
> > > > examples:
> > > >
> > > >
> > > > - <https://github.com/apache/systemml/tree/master/scripts>
> [https://avatars3.githubusercontent.com/u/47359?s=400&v=4]<
> https://github.com/apache/systemml/tree/master/scripts>
>
> systemml/scripts at master · apache/systemml · GitHub<
> https://github.com/apache/systemml/tree/master/scripts>
> github.com
> Mirror of Apache SystemML. Contribute to apache/systemml development by
> creating an account on GitHub.
>
>
> > > >
> > >
> >
> https://github.com/apache/systemml/blob/master/scripts/nn/layers/batch_norm1d.dml
> [https://avatars3.githubusercontent.com/u/47359?s=400&v=4]<
> https://github.com/apache/systemml/blob/master/scripts/nn/layers/batch_norm1d.dml
> >
>
> systemml/batch_norm1d.dml at master · apache/systemml · GitHub<
> https://github.com/apache/systemml/blob/master/scripts/nn/layers/batch_norm1d.dml
> >
> github.com
> Mirror of Apache SystemML. Contribute to apache/systemml development by
> creating an account on GitHub.
>
>
> > > > <
> > > >
> > >
> >
> https://github.com/Baunsgaard/systemml/blob/master/scripts/nn/layers/batch_norm2d.dml
> [https://avatars2.githubusercontent.com/u/9947148?s=400&v=4]<
> https://github.com/Baunsgaard/systemml/blob/master/scripts/nn/layers/batch_norm2d.dml
> >
>
> systemml/batch_norm2d.dml at master · Baunsgaard/systemml · GitHub<
> https://github.com/Baunsgaard/systemml/blob/master/scripts/nn/layers/batch_norm2d.dml
> >
> github.com
> Mirror of Apache SystemML. Contribute to Baunsgaard/systemml development
> by creating an account on GitHub.
>
>
> > > > >
> > > >
> > > > - <
> > > >
> > >
> >
> https://github.com/Baunsgaard/systemml/blob/master/scripts/builtin/glm.dml
> [https://avatars2.githubusercontent.com/u/9947148?s=400&v=4]<
> https://github.com/Baunsgaard/systemml/blob/master/scripts/builtin/glm.dml
> >
>
> systemml/glm.dml at master · Baunsgaard/systemml · GitHub<
> https://github.com/Baunsgaard/systemml/blob/master/scripts/builtin/glm.dml
> >
> github.com
> Mirror of Apache SystemML. Contribute to Baunsgaard/systemml development
> by creating an account on GitHub.
>
>
> > > >
> > > > <https://github.com/apache/systemml/tree/master/scripts>
> [https://avatars3.githubusercontent.com/u/47359?s=400&v=4]<
> https://github.com/apache/systemml/tree/master/scripts>
>
> systemml/scripts at master · apache/systemml · GitHub<
> https://github.com/apache/systemml/tree/master/scripts>
> github.com
> Mirror of Apache SystemML. Contribute to apache/systemml development by
> creating an account on GitHub.
>
>
> > > >
> > >
> >
> https://github.com/apache/systemml/blob/master/scripts/algorithms/Cox.dml
> [https://avatars3.githubusercontent.com/u/47359?s=400&v=4]<
> https://github.com/apache/systemml/blob/master/scripts/algorithms/Cox.dml>
>
> systemml/Cox.dml at master · apache/systemml · GitHub<
> https://github.com/apache/systemml/blob/master/scripts/algorithms/Cox.dml>
> github.com
> Mirror of Apache SystemML. Contribute to apache/systemml development by
> creating an account on GitHub.
>
>
> > > >
> > > >
> > > >
> > > > Furthermore, I would suggest to postpone changing the main website:
> > > >
> > > > https://github.com/apache/systemml-website
> [https://avatars3.githubusercontent.com/u/47359?s=400&v=4]<
> https://github.com/apache/systemml-website>
>
> GitHub - apache/systemml-website: Mirror of Apache SystemML site<
> https://github.com/apache/systemml-website>
> github.com
> Mirror of Apache SystemML site. Contribute to apache/systemml-website
> development by creating an account on GitHub.
>
>
> > > >
> > > > But that task has to be started when the next release is scheduled,
> > > > because then we need to find out how to copy the current
> documentation
> > > to a
> > > > archive list of static webpages located:
> > > >
> > > > https://systemml.apache.org/documentation
> Documentation<https://systemml.apache.org/documentation>
> systemml.apache.org
> Project Documentation Page
>
>
> > > >
> > > >
> > > > best regards
> > > >
> > > > Sebastian
> > > >
> > > > ________________________________
> > > > From: Janardhan <ja...@apache.org>
> > > > Sent: Sunday, May 24, 2020 11:02:53 AM
> > > > To: dev@systemml.apache.org
> > > > Subject: [DISCUSSION] Website stack `systemml.apache.org`. Thanks.
> > > >
> > > > Hi Sebastian,
> > > >
> > > > In our discussion[1][2] a while ago, there was a topic about changing
> > > > website
> > > > stack.
> > > >
> > > > Let us start a discussion about this in this thread.
> > > >
> > > > We are planning to work on this, after a feasibility study and tech
> > stack
> > > > vetting.
> > > >
> > > > Anybody would like to give suggestions would be great.
> > > >
> > > > [1] https://github.com/apache/systemml/pull/877
> [https://avatars3.githubusercontent.com/u/47359?s=400&v=4]<
> https://github.com/apache/systemml/pull/877>
>
> [MINOR][DOC] Name refactor SystemML to SystemDS in Documentation by
> Baunsgaard · Pull Request #877 · apache/systemml · GitHub<
> https://github.com/apache/systemml/pull/877>
> github.com
> Brute change of documentation page to reflect name change. To fully
> enforce this two more steps are needed:
> https://github.com/apache/systemml-website Access to SystemML website
> server on https://systemml.apache.org/ Furthermore, we might want to
> consider starting fresh in the documentation.
>
>
> > > > [2] https://github.com/apache/systemml-website/pull/66
> [https://avatars3.githubusercontent.com/u/47359?s=400&v=4]<
> https://github.com/apache/systemml-website/pull/66>
>
> [SYSTEMML-2539][BUILD] Upgrade build configuration by j143 · Pull Request
> #66 · apache/systemml-website · GitHub<
> https://github.com/apache/systemml-website/pull/66>
> github.com
> Upgrade gulp to 4.0 Since the task api is deprecated, fuctions were used,
> and exported as tasks. Documentation for gulpfile.js. Stick to Susy2, since
> mixins are not supported in 3.0 Upgrade npm dependencies to state-of-art.
> Commit package-lock.json file to version control.
>
>
> > > >
> > > > Thank you,
> > > > Janardhan
> > > >
> > >
> >
>