[jira] [Created] (RATIS-132) Support simple 'mvn clean compile' command (move shading before the compile phase)

["Elek, Marton (JIRA)" <ji...@apache.org>]

Elek, Marton created RATIS-132:
----------------------------------

             Summary: Support simple 'mvn clean compile' command (move shading before the compile phase) 
                 Key: RATIS-132
                 URL: https://issues.apache.org/jira/browse/RATIS-132
             Project: Ratis
          Issue Type: Bug
            Reporter: Elek, Marton
            Assignee: Elek, Marton


The current situation is described in the BUILDING.md:

{code}
When building Ratis the first time, shaded files need to be generated by the following command:
```
$ mvn package -DskipTests
```
After that, `mvn compile` or `mvn test` can be used as normal.
For example, we may run the basic tests by
```
{code}

In short: the shading phase is bound to the 'package' maven phase, therefore we can't use 'mvn clean install' just 'mvn clean package' because the package phase should be done before the compilation of ratis-proto-shaded.

This blocks the nightly build as yetus uses one 'mvn clean compile' or 'mvn clean test-compile' without invoking the package phase.

TLDR; in this patch I propose to fix this behaviour.

EXPLAIN THE PATCH:

1. The shading/proto generation should be done before the compile ratis-common in case of 'mvn clean compile' It means that the phases should be changed to 'compile' or something earlier (eg. generate-sources).

{code}
             <artifactId>maven-shade-plugin</artifactId>
             <executions>
               <execution>
-                <phase>package</phase>
+                <phase>process-resources</phase>
                 <goals>
{code}

Unfortunatelly with this modification it can not compile, because the following error:

{code}
The project main artifact does not exist. This could have the followin reason:
...
- You have bound the goal to a lifecycle phase before "package". Please remove this binding from your POM such that the goal will be run in the proper phase.
...
{code}

And it's true. In compile phase we don't have yet the project artifact which is registered by the 'jar:jar' plugin during package phase.

2. It could be fixed by moving a lot of plugin invocation before the compilation phase. Theoretically the jar:jar could be moved to earlier, but it doesn't solve the problem as we have also need a packaging after the compile.

A more easy workaround is to remove the project artifact from artifacct list which will be shaded.

{code}
                   <artifactSet>
                     <excludes>
+                      <exclude>org.apache.ratis:ratis-hadoop-shaded</exclude>
+
{code}

It is a safe step as we shade the sources and unpack them to the src/main/java. We are not interested about shading the binary project artifact.

Now we can execute the shade plugin during the compile phase, but still have a problem. The generated protobuf classes are not shaded any more and they use com.google.* packages instead of the org.apache.ratis.shaded.com.google.* packages.

3. I tried to solve this problem with moving the proto compilation to a separated project (ratis-proto), which could be used by the ratis-proto-shaded. Unfortunatelly the shade plugin doesn't support this. The artifacts should be a jar file for the shade plugin and can't be a target/generated-classes directory from an other project.

But the problem now is easy: the generated protobuf classes should use the shaded classes (which are generated to the src/main/java). It could be done with a simple search and replace:

{code}
+                  <tasks>
+                    <replace token="com.google." value="org.apache.ratis.shaded.com.google."
+                             dir="target/generated-sources/protobuf">
+                      <include name="**/*.java"/>
+                    </replace>
+                    <replace token="io.grpc." value="org.apache.ratis.shaded.io.grpc."
+                             dir="target/generated-sources/protobuf">
+                      <include name="**/*.java"/>
+                    </replace>
+                  </tasks>
{code}

And now the whole project could be rebuilt with a simple 'mvn clean compile'

TESTING:

Now the project could be compiled and the unit tests are running.

I also modified my hadoop (HDFS-7240 branch) to use 0.1.1-alpha-SNAPSHOT from ratis and start an ozone cluster with the new ratis and it worked well (I checked the Ratis JMX bean from the datanode web interface and it was registered).




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)