Thoughts on Release 3.0

[Niclas Hedhman <ni...@hedhman.org>]

1. Polygene generator doesn't produce functioning projects. It is quite
close, but there are issues in security and perhaps in Configuration
handling. I get indexer complaining on restarts that Configuration type is
not visible and other odd things. Part of the issues is in the generator,
but I think part of it is problems in at least the SQL Indexer. I suggest
that we leave out the tools/generator-polygene in the 3.0 release and
target 3.1 instead.

2. Mentioned above; I am suspicious about the SQL Indexer and its "reindex"
on start up. I have frequent exceptions on that when working on the
generator, and although one bug was fixed (thanks Stan) I think there are
others in the type lookup. One hypothesis I have, which is not confirmed,
is that "Module" is written into the SQL database, which I think is used on
reindexing, and during reindexing that "Module" field becomes the indexer's
module, and on the second restart and its indexing things are messed up. I
suggest that we do not include the SQL indexer in 3.0

3. Paul has previously brought up that Extensions are not unified and
implements different amounts of features, and it is difficult to know what
is supported. Solr indexer is a example of something that doesn't support
much of the Indexing SPI, and is actually addressing a different concern;
Text Search, which I think should be a separate API in a new Query API
which I would like to work on for Release 3.2. I suggest that Solr Indexer
is dropped from 3.0

4. There will soon be a new SQL Entity Store, and I don't think there is
any point in keeping the existing one. I suggest (and prefer) that it is
dropped from 3.0, or at least renamed to something like
entitystore-legacysql

5. Roadmap. Below is what I would like to do and the goals I would like to
have and willing to work on.

    July 2017 - Release 3.0

    Oct 2017 - Release 3.1 : The new SQL Entity Store, Polygene Generator
and Kotlin showcasing.

    Jan 2018 - Release 3.2 : New indexing system and new Query API.

    April 2018 - Release 3.3 : Timeseries support


WDYAT?

Cheers
-- 
Niclas Hedhman, Software Developer
http://polygene.apache.org - New Energy for Java

Re: Thoughts on Release 3.0

[Kent Sølvsten <ke...@gmail.com>]

I think you make a lot of sense, and i wholeheartedly agree on the first 2
points.

Rearding the solr indexer my impression was that it is fully functional,
but more or less supports another API (native queries) instead of the
"standard" Query API. That being said, i cannot see what we lose by keeping
it inside 3.0 (provided it is working) - since i expect we would probably
need to make changes to the Query API anyway, and Thus consider the issue
of backward compatibility.

+1 to renaming to entitystore-legacysql.

I hope to be able to return to the project post 3.0 - so let's get this
thing out of the door!

/Kent



On Tue, Jul 11, 2017 at 9:58 AM, Niclas Hedhman <ni...@hedhman.org> wrote:

> 1. Polygene generator doesn't produce functioning projects. It is quite
> close, but there are issues in security and perhaps in Configuration
> handling. I get indexer complaining on restarts that Configuration type is
> not visible and other odd things. Part of the issues is in the generator,
> but I think part of it is problems in at least the SQL Indexer. I suggest
> that we leave out the tools/generator-polygene in the 3.0 release and
> target 3.1 instead.
>
> 2. Mentioned above; I am suspicious about the SQL Indexer and its "reindex"
> on start up. I have frequent exceptions on that when working on the
> generator, and although one bug was fixed (thanks Stan) I think there are
> others in the type lookup. One hypothesis I have, which is not confirmed,
> is that "Module" is written into the SQL database, which I think is used on
> reindexing, and during reindexing that "Module" field becomes the indexer's
> module, and on the second restart and its indexing things are messed up. I
> suggest that we do not include the SQL indexer in 3.0
>
> 3. Paul has previously brought up that Extensions are not unified and
> implements different amounts of features, and it is difficult to know what
> is supported. Solr indexer is a example of something that doesn't support
> much of the Indexing SPI, and is actually addressing a different concern;
> Text Search, which I think should be a separate API in a new Query API
> which I would like to work on for Release 3.2. I suggest that Solr Indexer
> is dropped from 3.0
>
> 4. There will soon be a new SQL Entity Store, and I don't think there is
> any point in keeping the existing one. I suggest (and prefer) that it is
> dropped from 3.0, or at least renamed to something like
> entitystore-legacysql
>
> 5. Roadmap. Below is what I would like to do and the goals I would like to
> have and willing to work on.
>
>     July 2017 - Release 3.0
>
>     Oct 2017 - Release 3.1 : The new SQL Entity Store, Polygene Generator
> and Kotlin showcasing.
>
>     Jan 2018 - Release 3.2 : New indexing system and new Query API.
>
>     April 2018 - Release 3.3 : Timeseries support
>
>
> WDYAT?
>
> Cheers
> --
> Niclas Hedhman, Software Developer
> http://polygene.apache.org - New Energy for Java
>

Re: Thoughts on Release 3.0

[Paul Merlin <pa...@apache.org>]

Le 2017-07-14 02:19, Niclas Hedhman a écrit :
>> 3.1
>> - a suite of Gradle plugins: polygene-library, polygene-extension,
>>   polygene-application .. to make it damn easy to setup the build of
>>   Polygene apps
> 
> Ok, that would simply fold into the generator? Or is it something to be
> published separately?

The Gradle plugins would be published on their own and usable directly.
The generator would then be able to consume these plugins to simplify 
the generated builds.


>> 3.2
>> - Polygene runs on Jigsaw
> 
> Is Oracle still going ahead with Jigsaw? I thought the EC shot it down?

The EC voted against Jigsaw at some point.
Recent changes like "--permit-illegal-access" being the default in 9 
made the subsequent vote pass.
See https://www.jcp.org/en/jsr/results?id=6016


>> - Performance testing, some guarantee that we don't introduce
>> regressions would be awesome
> 
> I will offer my new company's resources for this, and it should be
> available around 3.2 time frame.

Excellent!
Basically we'll need to regularly run performance measurements on a well 
known non-changing, non-busy machine, for the latest release and the 
head of develop, record the results, assert we don't regress, handle 
flakiness etc...
Then we'll need some nice graph to look at, something like: 
https://builds.gradle.org/repository/download/Gradle_Check_PerformanceHistoricalCoordinator/3675790:id/results/performance/build/performance-tests/report/tests/clean-assemble-on-largeJavaMultiProject-with-local-cache-(parallel:-true).html 
("Login as Guest")


>> - Collaboration on the new indexing system / query api is appealing!
> 
> I doubt that I can manage a new indexing/querying on my own, and 
> looking
> forward to heavy collaboration on that.

\o/

Re: Thoughts on Release 3.0

[Niclas Hedhman <ni...@hedhman.org>]

On Fri, Jul 14, 2017 at 12:13 AM, Paul Merlin <pa...@apache.org> wrote:

> Niclas Hedhman a écrit :
> > 2. Mentioned above; I am suspicious about the SQL Indexer and its
> "reindex"
> > on start up. I have frequent exceptions on that when working on the
> > generator, and although one bug was fixed (thanks Stan) I think there are
> > others in the type lookup. One hypothesis I have, which is not confirmed,
> > is that "Module" is written into the SQL database, which I think is used
> on
> > reindexing, and during reindexing that "Module" field becomes the
> indexer's
> > module, and on the second restart and its indexing things are messed up.
> I
> > suggest that we do not include the SQL indexer in 3.0
>
> +1
> @Niclas: could you create an issue with this?
>

I don't have it nailed down and getting so frustrated with it, since I
think it is on the 3 start or something like that.


> > 3. Paul has previously brought up that Extensions are not unified and
> > implements different amounts of features, and it is difficult to know
> what
> > is supported. Solr indexer is a example of something that doesn't support
> > much of the Indexing SPI, and is actually addressing a different concern;
> > Text Search, which I think should be a separate API in a new Query API
> > which I would like to work on for Release 3.2. I suggest that Solr
> Indexer
> > is dropped from 3.0
>
> I'm with Kent on this, the Solr index/query extension has value and its
> documentation states clearly what is supported and what is not:
> http://polygene.apache.org/java/develop/extension-index-solr.html.
> I see no reason to not ship it with 3.0.
>

Ok, we keep it in.


> 4. There will soon be a new SQL Entity Store, and I don't think there is
> > any point in keeping the existing one. I suggest (and prefer) that it is
> > dropped from 3.0, or at least renamed to something like
> > entitystore-legacysql
>
> What about entitystore-map-sql instead?
> It is a MapEntityStore.
> We can deprecate and retire it at some point, but the word "legacy"
> makes me itchy.
>

Ok...
It is funny that only in the computing is "legacy" something bad. Every
other industry and field of achievements the word "legacy" is something to
strive for. :-)



> Here are some areas I would like to work on next with the tentative
> calendar above in mind:
>
> 3.1
>
> - Tooling
>     - a suite of Gradle plugins: polygene-library, polygene-extension,
> polygene-application .. to make it damn easy to setup the build of
> Polygene apps
>     - formalize Extensions vs. Features (reporting, documentation,
> testing?)
>     - reinstate checkstyle checks now that it supports Java 8
>

Ok, that would simply fold into the generator? Or is it something to be
published separately?


3.2
>
> - Polygene runs on Jigsaw
>

Is Oracle still going ahead with Jigsaw? I thought the EC shot it down?


> - Performance testing, some guarantee that we don't introduce
> regressions would be awesome
>

I will offer my new company's resources for this, and it should be
available around 3.2 time frame.


> - Performance optimisations (e.g. known hot spots in serialization
> fixable once Johnzon supports JSR-374)
> - Collaboration on the new indexing system / query api is appealing!
>

I doubt that I can manage a new indexing/querying on my own, and looking
forward to heavy collaboration on that.


I'll take a look at 'develop' later today.


Cheers
-- 
Niclas Hedhman, Software Developer
http://polygene.apache.org - New Energy for Java

Re: Thoughts on Release 3.0

[Paul Merlin <pa...@apache.org>]

Hey folks,

Niclas Hedhman a écrit :
> 1. Polygene generator doesn't produce functioning projects. It is quite
> close, but there are issues in security and perhaps in Configuration
> handling. I get indexer complaining on restarts that Configuration type is
> not visible and other odd things. Part of the issues is in the generator,
> but I think part of it is problems in at least the SQL Indexer. I suggest
> that we leave out the tools/generator-polygene in the 3.0 release and
> target 3.1 instead.

+1

> 2. Mentioned above; I am suspicious about the SQL Indexer and its "reindex"
> on start up. I have frequent exceptions on that when working on the
> generator, and although one bug was fixed (thanks Stan) I think there are
> others in the type lookup. One hypothesis I have, which is not confirmed,
> is that "Module" is written into the SQL database, which I think is used on
> reindexing, and during reindexing that "Module" field becomes the indexer's
> module, and on the second restart and its indexing things are messed up. I
> suggest that we do not include the SQL indexer in 3.0

+1
@Niclas: could you create an issue with this?

> 3. Paul has previously brought up that Extensions are not unified and
> implements different amounts of features, and it is difficult to know what
> is supported. Solr indexer is a example of something that doesn't support
> much of the Indexing SPI, and is actually addressing a different concern;
> Text Search, which I think should be a separate API in a new Query API
> which I would like to work on for Release 3.2. I suggest that Solr Indexer
> is dropped from 3.0

I'm with Kent on this, the Solr index/query extension has value and its
documentation states clearly what is supported and what is not:
http://polygene.apache.org/java/develop/extension-index-solr.html.
I see no reason to not ship it with 3.0.

> 4. There will soon be a new SQL Entity Store, and I don't think there is
> any point in keeping the existing one. I suggest (and prefer) that it is
> dropped from 3.0, or at least renamed to something like
> entitystore-legacysql

What about entitystore-map-sql instead?
It is a MapEntityStore.
We can deprecate and retire it at some point, but the word "legacy"
makes me itchy.

> 5. Roadmap. Below is what I would like to do and the goals I would like to
> have and willing to work on.
>
>     July 2017 - Release 3.0
>
>     Oct 2017 - Release 3.1 : The new SQL Entity Store, Polygene Generator
> and Kotlin showcasing.
>
>     Jan 2018 - Release 3.2 : New indexing system and new Query API.
>
>     April 2018 - Release 3.3 : Timeseries support
>
>
> WDYAT?

This makes a lot of sense to me.

I already started on Kotlin and have a library with Kotlin extensions to
the core & bootstrap APIs. I'll push it on a branch right after 3.0 and
some polishing, we'll go from there.

Here are some areas I would like to work on next with the tentative
calendar above in mind:

3.1

- Kotlin, see above
- Tooling
    - a suite of Gradle plugins: polygene-library, polygene-extension,
polygene-application .. to make it damn easy to setup the build of
Polygene apps
    - formalize Extensions vs. Features (reporting, documentation, testing?)
    - reinstate checkstyle checks now that it supports Java 8

3.2

- Polygene runs on Jigsaw
- Performance testing, some guarantee that we don't introduce
regressions would be awesome
- Performance optimisations (e.g. known hot spots in serialization
fixable once Johnzon supports JSR-374)
- Collaboration on the new indexing system / query api is appealing!
 
3.3
- Don't know yet


I'll start rolling 3.0 during the weekend, or early Monday my-morning.

Cheers

/Paul