You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@polygene.apache.org by Paul Merlin <pa...@apache.org> on 2017/06/03 12:24:26 UTC

Re: yeoman-work back to develop

Hey,

I took a quick look at the project generator integration tests.

First thing, I just tried on a computer where I don't have docker
installed and the tests present in generated projects simply fail with a
not so obvious error.

We should not require docker for a successful build of generated
projects. This would mean that the tests should be skiped/ignored if a
docker service is not available.

We should also not use `latest` tags for docker images otherwise we
can't guarantee that the generated projects will work in the future.
About Memcached, I noticed that the docker image used is
`memcached:latest`. The extension tests (as opposed to tests of
generated projects) use the Alpine Linux variant instead, see
https://github.com/apache/polygene-java/blob/develop/dependencies.gradle#L281,
maybe there's something different here.

Lastly, the docker rule for MongoDBEntityStore is missing. Note that the
extension tests use an embedded version of MongoDB.

I've been quite busy this week. I'll have some time early next week to
try to fix stuff and roll a second release candidate for 3.0 if we are
ready.

Given that the tests take quite a long time to run it's unpleasant to
not be able to easily run a single permutation while working on the
underlying plumbing. Even the reduced set of tests run by
:tools:generator:npmTest takes several minutes. Plus I miss proper test
reports with output and involved paths. Moreover, these tests are
unfortunately currently disabled on CI. We can look at these problems
post 3.0.

Cheers

/Paul


Niclas Hedhman a écrit :
> Gang,
> I just merged back all the work that has been done on yeoman-work branch.
>
> I think the build is Ok, the "checkDist" worked ok locally, think I tried
> with clean local setup.
>
> Well, if it doesn't I will deal with it tomorrow, unless someone beats me
> to it.
>
>
> The main thing outstanding is that generated testcase is marked @Ignore,
> until I can figure out all the remaining Docker starts needed. For
> instance, Memchched is used for most tests, and if I start an memcached
> server locally, many of the tests works, but with the Docker start that I
> am attempting, something is not right.
>
> Other thing that was fixed was the Configuration bootstrapping, now using
> the Module of the ConfigurationComposite for any Deserializer service
> needed.
>
> The stacktrace addition is there as well.
>
> And probably more that I have forgotten about. For instance, some name
> changes to try to stay unified and simplified code generation.
>
>
> All-in-all, I think the code generation facility is a good step towards a
> really great way to get started, without the initial struggles that we all
> face in the beginning to get structure working.
>
>
> Cheers

Re: yeoman-work back to develop

Posted by Paul Merlin <pa...@apache.org>.
So, :tools:generator:npmTestAll eventually finished.

BUILD FAILED
Total time: 12 hrs 39 mins 22.934 secs

6530 tests

1115 failures
The JMX library assemblies do not compile!

252 timeouts
Some tests are rather fast, <5sec, a few are are rather long, >30sec.
Offenders use Cassandra ES, MongoDB ES, Riak ES that take a bit of time 
to start.
Need to raise the test timeout.


Re: yeoman-work back to develop

Posted by Niclas Hedhman <ni...@hedhman.org>.
Really glad that you have dug into this work. Felt a bit lonely for a
while... :-D


Below is a model.json file that I have been using to get additional bits
and pieces working beyond the bare bone skeleton. Feel free to use as a
starting point for some more elaborate generations.


{
  "name": "Flood",
  "packageName": "com.sensetif.sink",
  "applicationtype": "Rest API",
  "features": [
    "jmx",
    "security"
  ],
  "entitystore": "Cassandra",
  "indexing": "Rdf",
  "caching": "Memcache",
  "metrics": "Codahale",
  "modules": {
    "user": {
      "cruds": [
        {"name": "User"},
        {"name": "Users" },
        {"name": "Role"},
        {"name": "Roles"},
        {"name": "Permission"},
        {"name": "Permissions"},
        {"name": "Group"},
        {"name": "Groups"}
      ],
      "services": [
        {
          "name": "AuthService",
          "visibility": "application",
          "configuration" : [
            { "name": "backend", "type": "java.lang.String",
"default": "ldap", "description": [ "Type of backend authentication
and authorization system" ] },
            { "name": "connectString", "type": "java.lang.String",
"description": ["The connection string to the authentication and
authorization backend system."] }
          ]
        }
      ],
      "plainTypes": [
        { "name": "AuthBackend", "type": "enum", "values":
"ldap,kerberos,entitystore" }
      ]
    },
    "organization": {
      "cruds": [
        {
          "name": "Organization",
          "properties": [
            { "name": "name", "type": "java.lang.String" },
            { "name": "created", "type": "java.time.Instant" }
          ]
        },
        { "name": "Project" },
        { "name": "Contract" },
        { "name": "ContractPart" },
        { "name": "Order" },
        { "name": "OrderConfirmation" },
        { "name": "Invoice" }
      ],
      "entities": [
      ],
      "values": [
        {
          "name": "CreditLimit",
          "properties" : [
            { "name" : "amount", "type" : "java.math.BigDecimal" },
            { "name" : "currency", "type" : "java.lang.String" }
          ]
        },
        { "name": "Label" }
      ],
      "transients": [
        { "name": "PaypalNotification" }
      ]
    },
    "sensor": {
      "cruds": [
        { "name": "SensorDetails",
          "properties": [
            { "name": "name", "type": "java.lang.String" },
            { "name": "location", "type": "java.lang.String" },
            { "name": "description", "type": "java.lang.String" }
          ],
          "associations": [
            { "name": "project", "type":
"com.sensetif.sink.model.organization.Project" }
          ],
          "manyassociations": [
          ]
        }
      ],
      "entities": [
        { "name": "SensorPoll" },
        { "name": "Access",
          "properties": [
            { "name": "type", "type":
"com.sensetif.sink.model.sensor.AccessType"},
            { "name": "address", "type":
"com.sensetif.sink.model.sensor.Address"},
            { "name": "selector", "type":
"com.sensetif.sink.model.sensor.Selector"}
          ]
        }
      ],
      "plainTypes": [
        { "name": "AccessType", "type": "enum" },
        { "name": "Address", "type": "interface" },
        { "name": "Selector", "type": "interface" }
      ],
      "values": [
        { "name": "JsonDocumentAddress" },
        { "name": "XmlDocumentAddress" },
        { "name": "ModbusDeviceAddress" },
        { "name": "JsonPathSelector" },
        { "name": "XPathSelector" },
        { "name": "ModbusSelector" },
        { "name": "PollSchedule" }
      ]
    }
  }
}


On Tue, Jun 6, 2017 at 12:48 AM, Paul Merlin <pa...@apache.org> wrote:

> Le 2017-06-05 18:38, Niclas Hedhman a écrit :
>
>> Yeah, good point... I will fix that asap.
>>
>
> I had the change handy so I pushed it.
>
>
>
> ./gradlew :tools:generator:npmTest
>
> BUILD SUCCESSFUL
>
> Total time: 7.118 secs
>
>
>
> That is with the docker tests.
>
> \o/
>
> The tools/generator-polygene/build/reports/tests/npmTest/generator-polygene.xml
> JUnit formatted test report is handy.
> It lacks captured test output but it's better than nothing.
>
>
> ./gradlew :tools:generator:npmTestAll
>
> ... running ... it may take a while, I may also just kill it at some point
> :)
>
>
>
>
>
> On Tue, Jun 6, 2017 at 12:13 AM, Paul Merlin <pa...@apache.org>
>> wrote:
>>
>> So, I took a stab at this and made some progress.
>>>
>>> The generated tests needing Docker will now automatically be skipped if
>>> no
>>> docker service is available.
>>> IOW, the generated projects aren't broken anymore for people that do not
>>> run docker.
>>>
>>> I fixed Memcache, MongoDB, Riak, Redis, JClouds, H2SQL docker
>>> integration.
>>>
>>> I disabled tests in generated projects when MySQL or PostgreSQL is
>>> involved because I couldn't get the former to work (timing issue?) and
>>> the
>>> latter requires a custom docker image. We can fix that post 3.0.
>>>
>>> So, in the reduced set of tests, only 2 fail now, the ones using XML or
>>> msgpack serialization.
>>>
>>> I think there's a hiccup with how Serialization is assembled in the
>>> generated applications.
>>> The generator asks for which serialization subsystem to use, json, xml or
>>> msgpack, and assemble it in infra/serialization with an application
>>> visibility, the infra layer is used by all other layers except 'config'.
>>>
>>> But, what is the intent really?
>>>
>>> All JSONMapEntityStores uses JSON, and requires a JsonSerialization
>>> service.
>>> RdfIndexing uses some JSON under the hood, and requires a
>>> JsonSerialization service.
>>> The generated projects fail to bootstrap.
>>>
>>> Values toString() method uses the Serialization visible from the value's
>>> module.
>>> What happen with XML or msgpack?
>>> It will respectively return XML or Base64 encoded binary.
>>>
>>> The rest library assumes JSON Serialization but it should require a
>>> JsonSerialization service explicitly.
>>> What happen with XML or msgpack?
>>> HTTP requests are expected in application/json but parsed as XML / Base64
>>> encoded binary.
>>> HTTP responses are advertised as application/json but contains XML /
>>> Base64 encoded binary.
>>>
>>> In this context, a HTTP/JSON service, XML and MessagePack serializations
>>> aren't that useful.
>>>
>>> The rest library could be enhanced to provide XML, even binary,
>>> representations at some point.
>>> We could use Message Pack in the configuration Memory ES instead of JSON
>>> to save some memory, maybe.
>>> The user might implement e.g. a custom service that need XML or binary
>>> serialization.
>>>
>>> I would remove the choice of serialization from the generator for 3.0 and
>>> assemble default JSON serialization where appropriate instead. Any
>>> objections ?
>>>
>>> /Paul
>>>
>>>
>>>
>>>
>>>
>>> Le 2017-06-04 11:09, Paul Merlin a écrit :
>>>
>>> Le 2017-06-04 02:54, Niclas Hedhman a écrit :
>>>>
>>>> As for Docker;
>>>>>
>>>>> Don't forget that this is code generation and it is EXPECTED that
>>>>> people
>>>>> tailor it to their needs. I chose ":latest" so that we don't have to
>>>>> constantly maintain the version as those dependencies are likely to
>>>>> upgrade
>>>>> faster than we do. It is not important whether the generated project
>>>>> will
>>>>> break long-term for using ":latest".
>>>>>
>>>>>
>>>> `:latest` could eventually start breaking the day after we release
>>>> 3.0. We do already have the list of qualified docker images in a
>>>> central place. It's like any other dependency. "maintaining" boils
>>>> down to not repeat ourselves.
>>>>
>>>> BTW, it's the same with java dependencies that are also already
>>>> duplicated in the templates. We should reuse those defined in
>>>> ~/dependencies.gradle at some point. That can be post 3.0.
>>>>
>>>> Docker being present is a similar thing. If you don't want Docker, kill
>>>>
>>>>> the
>>>>> DockerRule and change to connect to the external system available
>>>>> through
>>>>> other means.
>>>>>
>>>>> I think the main point is; We don't need to tailor for all possible
>>>>> situations, as generated code is not 'final' in any shape, way or form.
>>>>> But
>>>>> perhaps the generator should "check for Docker" and don't generate a
>>>>> Dockerrule and disable the test if it is not present.
>>>>>
>>>>>
>>>> Well, I get your point but I strongly think that the generated
>>>> projects should not be broken. They are if they fail to build/run. And
>>>> users will think the same way I suppose. The generated projects are
>>>> just scaffoldings yes, but getting started with a broken build (as in
>>>> not building/running) will confuse users and produce noise.
>>>>
>>>>
>


-- 
Niclas Hedhman, Software Developer
http://polygene.apache.org - New Energy for Java

Re: yeoman-work back to develop

Posted by Paul Merlin <pa...@apache.org>.
Le 2017-06-05 18:38, Niclas Hedhman a écrit :
> Yeah, good point... I will fix that asap.

I had the change handy so I pushed it.



./gradlew :tools:generator:npmTest

BUILD SUCCESSFUL

Total time: 7.118 secs



That is with the docker tests.

\o/

The 
tools/generator-polygene/build/reports/tests/npmTest/generator-polygene.xml 
JUnit formatted test report is handy.
It lacks captured test output but it's better than nothing.


./gradlew :tools:generator:npmTestAll

... running ... it may take a while, I may also just kill it at some 
point :)




> On Tue, Jun 6, 2017 at 12:13 AM, Paul Merlin <pa...@apache.org> 
> wrote:
> 
>> So, I took a stab at this and made some progress.
>> 
>> The generated tests needing Docker will now automatically be skipped 
>> if no
>> docker service is available.
>> IOW, the generated projects aren't broken anymore for people that do 
>> not
>> run docker.
>> 
>> I fixed Memcache, MongoDB, Riak, Redis, JClouds, H2SQL docker 
>> integration.
>> 
>> I disabled tests in generated projects when MySQL or PostgreSQL is
>> involved because I couldn't get the former to work (timing issue?) and 
>> the
>> latter requires a custom docker image. We can fix that post 3.0.
>> 
>> So, in the reduced set of tests, only 2 fail now, the ones using XML 
>> or
>> msgpack serialization.
>> 
>> I think there's a hiccup with how Serialization is assembled in the
>> generated applications.
>> The generator asks for which serialization subsystem to use, json, xml 
>> or
>> msgpack, and assemble it in infra/serialization with an application
>> visibility, the infra layer is used by all other layers except 
>> 'config'.
>> 
>> But, what is the intent really?
>> 
>> All JSONMapEntityStores uses JSON, and requires a JsonSerialization
>> service.
>> RdfIndexing uses some JSON under the hood, and requires a
>> JsonSerialization service.
>> The generated projects fail to bootstrap.
>> 
>> Values toString() method uses the Serialization visible from the 
>> value's
>> module.
>> What happen with XML or msgpack?
>> It will respectively return XML or Base64 encoded binary.
>> 
>> The rest library assumes JSON Serialization but it should require a
>> JsonSerialization service explicitly.
>> What happen with XML or msgpack?
>> HTTP requests are expected in application/json but parsed as XML / 
>> Base64
>> encoded binary.
>> HTTP responses are advertised as application/json but contains XML /
>> Base64 encoded binary.
>> 
>> In this context, a HTTP/JSON service, XML and MessagePack 
>> serializations
>> aren't that useful.
>> 
>> The rest library could be enhanced to provide XML, even binary,
>> representations at some point.
>> We could use Message Pack in the configuration Memory ES instead of 
>> JSON
>> to save some memory, maybe.
>> The user might implement e.g. a custom service that need XML or binary
>> serialization.
>> 
>> I would remove the choice of serialization from the generator for 3.0 
>> and
>> assemble default JSON serialization where appropriate instead. Any
>> objections ?
>> 
>> /Paul
>> 
>> 
>> 
>> 
>> 
>> Le 2017-06-04 11:09, Paul Merlin a écrit :
>> 
>>> Le 2017-06-04 02:54, Niclas Hedhman a écrit :
>>> 
>>>> As for Docker;
>>>> 
>>>> Don't forget that this is code generation and it is EXPECTED that 
>>>> people
>>>> tailor it to their needs. I chose ":latest" so that we don't have to
>>>> constantly maintain the version as those dependencies are likely to
>>>> upgrade
>>>> faster than we do. It is not important whether the generated project 
>>>> will
>>>> break long-term for using ":latest".
>>>> 
>>> 
>>> `:latest` could eventually start breaking the day after we release
>>> 3.0. We do already have the list of qualified docker images in a
>>> central place. It's like any other dependency. "maintaining" boils
>>> down to not repeat ourselves.
>>> 
>>> BTW, it's the same with java dependencies that are also already
>>> duplicated in the templates. We should reuse those defined in
>>> ~/dependencies.gradle at some point. That can be post 3.0.
>>> 
>>> Docker being present is a similar thing. If you don't want Docker, 
>>> kill
>>>> the
>>>> DockerRule and change to connect to the external system available 
>>>> through
>>>> other means.
>>>> 
>>>> I think the main point is; We don't need to tailor for all possible
>>>> situations, as generated code is not 'final' in any shape, way or 
>>>> form.
>>>> But
>>>> perhaps the generator should "check for Docker" and don't generate a
>>>> Dockerrule and disable the test if it is not present.
>>>> 
>>> 
>>> Well, I get your point but I strongly think that the generated
>>> projects should not be broken. They are if they fail to build/run. 
>>> And
>>> users will think the same way I suppose. The generated projects are
>>> just scaffoldings yes, but getting started with a broken build (as in
>>> not building/running) will confuse users and produce noise.
>>> 


Re: yeoman-work back to develop

Posted by Niclas Hedhman <ni...@hedhman.org>.
Yeah, good point... I will fix that asap.

On Tue, Jun 6, 2017 at 12:13 AM, Paul Merlin <pa...@apache.org> wrote:

> So, I took a stab at this and made some progress.
>
> The generated tests needing Docker will now automatically be skipped if no
> docker service is available.
> IOW, the generated projects aren't broken anymore for people that do not
> run docker.
>
> I fixed Memcache, MongoDB, Riak, Redis, JClouds, H2SQL docker integration.
>
> I disabled tests in generated projects when MySQL or PostgreSQL is
> involved because I couldn't get the former to work (timing issue?) and the
> latter requires a custom docker image. We can fix that post 3.0.
>
> So, in the reduced set of tests, only 2 fail now, the ones using XML or
> msgpack serialization.
>
> I think there's a hiccup with how Serialization is assembled in the
> generated applications.
> The generator asks for which serialization subsystem to use, json, xml or
> msgpack, and assemble it in infra/serialization with an application
> visibility, the infra layer is used by all other layers except 'config'.
>
> But, what is the intent really?
>
> All JSONMapEntityStores uses JSON, and requires a JsonSerialization
> service.
> RdfIndexing uses some JSON under the hood, and requires a
> JsonSerialization service.
> The generated projects fail to bootstrap.
>
> Values toString() method uses the Serialization visible from the value's
> module.
> What happen with XML or msgpack?
> It will respectively return XML or Base64 encoded binary.
>
> The rest library assumes JSON Serialization but it should require a
> JsonSerialization service explicitly.
> What happen with XML or msgpack?
> HTTP requests are expected in application/json but parsed as XML / Base64
> encoded binary.
> HTTP responses are advertised as application/json but contains XML /
> Base64 encoded binary.
>
> In this context, a HTTP/JSON service, XML and MessagePack serializations
> aren't that useful.
>
> The rest library could be enhanced to provide XML, even binary,
> representations at some point.
> We could use Message Pack in the configuration Memory ES instead of JSON
> to save some memory, maybe.
> The user might implement e.g. a custom service that need XML or binary
> serialization.
>
> I would remove the choice of serialization from the generator for 3.0 and
> assemble default JSON serialization where appropriate instead. Any
> objections ?
>
> /Paul
>
>
>
>
>
> Le 2017-06-04 11:09, Paul Merlin a écrit :
>
>> Le 2017-06-04 02:54, Niclas Hedhman a écrit :
>>
>>> As for Docker;
>>>
>>> Don't forget that this is code generation and it is EXPECTED that people
>>> tailor it to their needs. I chose ":latest" so that we don't have to
>>> constantly maintain the version as those dependencies are likely to
>>> upgrade
>>> faster than we do. It is not important whether the generated project will
>>> break long-term for using ":latest".
>>>
>>
>> `:latest` could eventually start breaking the day after we release
>> 3.0. We do already have the list of qualified docker images in a
>> central place. It's like any other dependency. "maintaining" boils
>> down to not repeat ourselves.
>>
>> BTW, it's the same with java dependencies that are also already
>> duplicated in the templates. We should reuse those defined in
>> ~/dependencies.gradle at some point. That can be post 3.0.
>>
>> Docker being present is a similar thing. If you don't want Docker, kill
>>> the
>>> DockerRule and change to connect to the external system available through
>>> other means.
>>>
>>> I think the main point is; We don't need to tailor for all possible
>>> situations, as generated code is not 'final' in any shape, way or form.
>>> But
>>> perhaps the generator should "check for Docker" and don't generate a
>>> Dockerrule and disable the test if it is not present.
>>>
>>
>> Well, I get your point but I strongly think that the generated
>> projects should not be broken. They are if they fail to build/run. And
>> users will think the same way I suppose. The generated projects are
>> just scaffoldings yes, but getting started with a broken build (as in
>> not building/running) will confuse users and produce noise.
>>
>
>


-- 
Niclas Hedhman, Software Developer
http://polygene.apache.org - New Energy for Java

Re: yeoman-work back to develop

Posted by Paul Merlin <pa...@apache.org>.
So, I took a stab at this and made some progress.

The generated tests needing Docker will now automatically be skipped if 
no docker service is available.
IOW, the generated projects aren't broken anymore for people that do not 
run docker.

I fixed Memcache, MongoDB, Riak, Redis, JClouds, H2SQL docker 
integration.

I disabled tests in generated projects when MySQL or PostgreSQL is 
involved because I couldn't get the former to work (timing issue?) and 
the latter requires a custom docker image. We can fix that post 3.0.

So, in the reduced set of tests, only 2 fail now, the ones using XML or 
msgpack serialization.

I think there's a hiccup with how Serialization is assembled in the 
generated applications.
The generator asks for which serialization subsystem to use, json, xml 
or msgpack, and assemble it in infra/serialization with an application 
visibility, the infra layer is used by all other layers except 'config'.

But, what is the intent really?

All JSONMapEntityStores uses JSON, and requires a JsonSerialization 
service.
RdfIndexing uses some JSON under the hood, and requires a 
JsonSerialization service.
The generated projects fail to bootstrap.

Values toString() method uses the Serialization visible from the value's 
module.
What happen with XML or msgpack?
It will respectively return XML or Base64 encoded binary.

The rest library assumes JSON Serialization but it should require a 
JsonSerialization service explicitly.
What happen with XML or msgpack?
HTTP requests are expected in application/json but parsed as XML / 
Base64 encoded binary.
HTTP responses are advertised as application/json but contains XML / 
Base64 encoded binary.

In this context, a HTTP/JSON service, XML and MessagePack serializations 
aren't that useful.

The rest library could be enhanced to provide XML, even binary, 
representations at some point.
We could use Message Pack in the configuration Memory ES instead of JSON 
to save some memory, maybe.
The user might implement e.g. a custom service that need XML or binary 
serialization.

I would remove the choice of serialization from the generator for 3.0 
and assemble default JSON serialization where appropriate instead. Any 
objections ?

/Paul




Le 2017-06-04 11:09, Paul Merlin a écrit :
> Le 2017-06-04 02:54, Niclas Hedhman a écrit :
>> As for Docker;
>> 
>> Don't forget that this is code generation and it is EXPECTED that 
>> people
>> tailor it to their needs. I chose ":latest" so that we don't have to
>> constantly maintain the version as those dependencies are likely to 
>> upgrade
>> faster than we do. It is not important whether the generated project 
>> will
>> break long-term for using ":latest".
> 
> `:latest` could eventually start breaking the day after we release
> 3.0. We do already have the list of qualified docker images in a
> central place. It's like any other dependency. "maintaining" boils
> down to not repeat ourselves.
> 
> BTW, it's the same with java dependencies that are also already
> duplicated in the templates. We should reuse those defined in
> ~/dependencies.gradle at some point. That can be post 3.0.
> 
>> Docker being present is a similar thing. If you don't want Docker, 
>> kill the
>> DockerRule and change to connect to the external system available 
>> through
>> other means.
>> 
>> I think the main point is; We don't need to tailor for all possible
>> situations, as generated code is not 'final' in any shape, way or 
>> form. But
>> perhaps the generator should "check for Docker" and don't generate a
>> Dockerrule and disable the test if it is not present.
> 
> Well, I get your point but I strongly think that the generated
> projects should not be broken. They are if they fail to build/run. And
> users will think the same way I suppose. The generated projects are
> just scaffoldings yes, but getting started with a broken build (as in
> not building/running) will confuse users and produce noise.


Re: yeoman-work back to develop

Posted by Paul Merlin <pa...@apache.org>.
Le 2017-06-04 02:54, Niclas Hedhman a écrit :
> As for Docker;
> 
> Don't forget that this is code generation and it is EXPECTED that 
> people
> tailor it to their needs. I chose ":latest" so that we don't have to
> constantly maintain the version as those dependencies are likely to 
> upgrade
> faster than we do. It is not important whether the generated project 
> will
> break long-term for using ":latest".

`:latest` could eventually start breaking the day after we release 3.0. 
We do already have the list of qualified docker images in a central 
place. It's like any other dependency. "maintaining" boils down to not 
repeat ourselves.

BTW, it's the same with java dependencies that are also already 
duplicated in the templates. We should reuse those defined in 
~/dependencies.gradle at some point. That can be post 3.0.

> Docker being present is a similar thing. If you don't want Docker, kill 
> the
> DockerRule and change to connect to the external system available 
> through
> other means.
> 
> I think the main point is; We don't need to tailor for all possible
> situations, as generated code is not 'final' in any shape, way or form. 
> But
> perhaps the generator should "check for Docker" and don't generate a
> Dockerrule and disable the test if it is not present.

Well, I get your point but I strongly think that the generated projects 
should not be broken. They are if they fail to build/run. And users will 
think the same way I suppose. The generated projects are just 
scaffoldings yes, but getting started with a broken build (as in not 
building/running) will confuse users and produce noise.



Re: yeoman-work back to develop

Posted by Niclas Hedhman <ni...@hedhman.org>.
As for Docker;

Don't forget that this is code generation and it is EXPECTED that people
tailor it to their needs. I chose ":latest" so that we don't have to
constantly maintain the version as those dependencies are likely to upgrade
faster than we do. It is not important whether the generated project will
break long-term for using ":latest".

Docker being present is a similar thing. If you don't want Docker, kill the
DockerRule and change to connect to the external system available through
other means.

I think the main point is; We don't need to tailor for all possible
situations, as generated code is not 'final' in any shape, way or form. But
perhaps the generator should "check for Docker" and don't generate a
Dockerrule and disable the test if it is not present.


Cheers

On Sat, Jun 3, 2017 at 8:24 PM, Paul Merlin <pa...@apache.org> wrote:

> Hey,
>
> I took a quick look at the project generator integration tests.
>
> First thing, I just tried on a computer where I don't have docker
> installed and the tests present in generated projects simply fail with a
> not so obvious error.
>
> We should not require docker for a successful build of generated
> projects. This would mean that the tests should be skiped/ignored if a
> docker service is not available.
>
> We should also not use `latest` tags for docker images otherwise we
> can't guarantee that the generated projects will work in the future.
> About Memcached, I noticed that the docker image used is
> `memcached:latest`. The extension tests (as opposed to tests of
> generated projects) use the Alpine Linux variant instead, see
> https://github.com/apache/polygene-java/blob/develop/
> dependencies.gradle#L281,
> maybe there's something different here.
>
> Lastly, the docker rule for MongoDBEntityStore is missing. Note that the
> extension tests use an embedded version of MongoDB.
>
> I've been quite busy this week. I'll have some time early next week to
> try to fix stuff and roll a second release candidate for 3.0 if we are
> ready.
>
> Given that the tests take quite a long time to run it's unpleasant to
> not be able to easily run a single permutation while working on the
> underlying plumbing. Even the reduced set of tests run by
> :tools:generator:npmTest takes several minutes. Plus I miss proper test
> reports with output and involved paths. Moreover, these tests are
> unfortunately currently disabled on CI. We can look at these problems
> post 3.0.
>
> Cheers
>
> /Paul
>
>
> Niclas Hedhman a écrit :
> > Gang,
> > I just merged back all the work that has been done on yeoman-work branch.
> >
> > I think the build is Ok, the "checkDist" worked ok locally, think I tried
> > with clean local setup.
> >
> > Well, if it doesn't I will deal with it tomorrow, unless someone beats me
> > to it.
> >
> >
> > The main thing outstanding is that generated testcase is marked @Ignore,
> > until I can figure out all the remaining Docker starts needed. For
> > instance, Memchched is used for most tests, and if I start an memcached
> > server locally, many of the tests works, but with the Docker start that I
> > am attempting, something is not right.
> >
> > Other thing that was fixed was the Configuration bootstrapping, now using
> > the Module of the ConfigurationComposite for any Deserializer service
> > needed.
> >
> > The stacktrace addition is there as well.
> >
> > And probably more that I have forgotten about. For instance, some name
> > changes to try to stay unified and simplified code generation.
> >
> >
> > All-in-all, I think the code generation facility is a good step towards a
> > really great way to get started, without the initial struggles that we
> all
> > face in the beginning to get structure working.
> >
> >
> > Cheers
>



-- 
Niclas Hedhman, Software Developer
http://polygene.apache.org - New Energy for Java

Re: yeoman-work back to develop

Posted by Niclas Hedhman <ni...@hedhman.org>.
Thanks for this.

FYI, when I work on individual issues, I don't rely on test runs, because
they are generally too slow and as you said it isn't 'any' combination.

Instead I use a model json file, and run

    rm -rf * ; yo polygene --import=../model.json --noPrompt

which is mounted in IDE. There I can edit the generation to what I want it
to do, then paste in the resulting code back to the templates and run again.


But a few days ago, I was thinking about that format, and whether I should
change to what the toJson() in tools/model-detail/ generates. I think it is
the right thing to do, but it will take a while, and I am not convinced
that the toJson() will "remain", as I am kind of questioning the whole
visitor pattern and the extreme overlap with the Model itself, it feels
like repetition.


On Sat, Jun 3, 2017 at 8:24 PM, Paul Merlin <pa...@apache.org> wrote:

> Hey,
>
> I took a quick look at the project generator integration tests.
>
> First thing, I just tried on a computer where I don't have docker
> installed and the tests present in generated projects simply fail with a
> not so obvious error.
>
> We should not require docker for a successful build of generated
> projects. This would mean that the tests should be skiped/ignored if a
> docker service is not available.
>
> We should also not use `latest` tags for docker images otherwise we
> can't guarantee that the generated projects will work in the future.
> About Memcached, I noticed that the docker image used is
> `memcached:latest`. The extension tests (as opposed to tests of
> generated projects) use the Alpine Linux variant instead, see
> https://github.com/apache/polygene-java/blob/develop/
> dependencies.gradle#L281,
> maybe there's something different here.
>
> Lastly, the docker rule for MongoDBEntityStore is missing. Note that the
> extension tests use an embedded version of MongoDB.
>
> I've been quite busy this week. I'll have some time early next week to
> try to fix stuff and roll a second release candidate for 3.0 if we are
> ready.
>
> Given that the tests take quite a long time to run it's unpleasant to
> not be able to easily run a single permutation while working on the
> underlying plumbing. Even the reduced set of tests run by
> :tools:generator:npmTest takes several minutes. Plus I miss proper test
> reports with output and involved paths. Moreover, these tests are
> unfortunately currently disabled on CI. We can look at these problems
> post 3.0.
>
> Cheers
>
> /Paul
>
>
> Niclas Hedhman a écrit :
> > Gang,
> > I just merged back all the work that has been done on yeoman-work branch.
> >
> > I think the build is Ok, the "checkDist" worked ok locally, think I tried
> > with clean local setup.
> >
> > Well, if it doesn't I will deal with it tomorrow, unless someone beats me
> > to it.
> >
> >
> > The main thing outstanding is that generated testcase is marked @Ignore,
> > until I can figure out all the remaining Docker starts needed. For
> > instance, Memchched is used for most tests, and if I start an memcached
> > server locally, many of the tests works, but with the Docker start that I
> > am attempting, something is not right.
> >
> > Other thing that was fixed was the Configuration bootstrapping, now using
> > the Module of the ConfigurationComposite for any Deserializer service
> > needed.
> >
> > The stacktrace addition is there as well.
> >
> > And probably more that I have forgotten about. For instance, some name
> > changes to try to stay unified and simplified code generation.
> >
> >
> > All-in-all, I think the code generation facility is a good step towards a
> > really great way to get started, without the initial struggles that we
> all
> > face in the beginning to get structure working.
> >
> >
> > Cheers
>



-- 
Niclas Hedhman, Software Developer
http://polygene.apache.org - New Energy for Java