You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@yetus.apache.org by suraj acharya <su...@gmail.com> on 2016/09/10 18:48:59 UTC

Avro Pre-commit.

Hi,
I have been working on AVRO-1887
<https://issues.apache.org/jira/browse/AVRO-1887> to add pre-commit test to
avro.
I have been successful in adding the personality for java file using maven.
I having some questions about the other plugins.
Example :ant

The structure of avro is that there is a top level pom.
And then there is different langs in subdirectory below it.
So there is avro/lang/py
<https://github.com/apache/avro/tree/master/lang/py>which has python code
in it. The build tool for that is ant.
My understanding is running add_test ant will run ant test.
However, that doesn't actually run any tests. My assumption is that since
there is no build.xml present there it is unable to run any ant tests.

I tried cd to lang/py and run add_test ant from there. However, it throws
an error saying unable to find pom.xml.

Can someone give me some pointers on how to continue?


Thanks
-Suraj Acharya

Re: Avro Pre-commit.

Posted by Allen Wittenauer <aw...@effectivemachines.com>.

I've been thinking about this issue.  With a bit of work, I think the approach should probably be to:

	a) make a custom build tool that knows which dirs use which other tools
	b) add a new file in each dir that can tell precommit where to break for modules

For example, let's say we have a dir structure like this:

		build.sh
		c/
			Makefile
		java/
			pom.xml
		perl/
			Makefile.PL

Let's add a file called multibuild.sh that precommit can use:

		build.sh
		multibuild.sh
		c/
			Makefile
			multibuild.sh
		java/
			pom.xml
			multibuild.sh
		perl/
			Makefile.PL
			multibuild.sh


In our "multibuild" build tool plugin, we'd define

		function multibuild_buildfile
		{
			echo "multibuild.sh"
		}

	Now when precommit gets a patch, it will know that a change in c/ is a different module than a change in java/ or perl/ because of the presence of the multibuild.sh file.  This prevents the "build the world" problem for a largely diverse project like avro. 

	The place where this gets hard is our theoretical multibuild_executor function because precommit doesn't pass as a param the module that's currently being processed. (we should probably fix that) But we can still make it work and reality, it might have led us down a false path.  If we make this function look like this:

		function multibuild_executor
		{
			echo "multibuild.sh"
		}

and make sure that BUILDTOOLCWD=module (the default), then precommit will run our file that we placed to actually do the work.  This file can then take whatever parameters we're getting passed and convert it as appropriate to the build tool for that directory. Since this is a custom build tool, you can dictate what parameters to pass to which modules via _modules_workers.  For example, an easy route might be:

		function multibuild_modules_workers
		{
			modules_workers $2
		}

then multibuild.sh for each language becomes a series of case statements.  For C, it might look like:

		testname=$1
		shift

		case ${testname} in
			compile)
				make -Dcustomflag=1
			;;
			doc)
				make man
			unit)
				make -Danotherflag=1 test
			;;
			*)
			;;
		esac

	It's pretty hacky, but theoretically it would work until we get a chance to make precommit smarter about multiple build tools in a single project. (which would be a pretty hard undertaking, but something we should probably do)


> On Sep 10, 2016, at 6:32 PM, suraj acharya <su...@gmail.com> wrote:
> 
> Yes.
> The ./build.sh test
> <https://github.com/apache/avro/blob/master/build.sh#L40> command helps run
> all the tests from the top level.
> However, I am not sure if we need to run all the tests when a change is
> made?
> I was thinking of running language specific tests for changes in the
> specific directory.
> For example for changes in python will trigger ant test
> <https://github.com/apache/avro/blob/master/build.sh#L43>.
> If this is not a good idea let me know. Ill start another tread on dev@avro
> to see what the community thinks about it
> 
> Thanks
> 
> -Suraj Acharya
> 
> On Sat, Sep 10, 2016 at 5:45 PM, Sean Busbey <bu...@apache.org> wrote:
> 
>> my understanding of Avro's build system is that there are per-language
>> build tools and everything gets unified via shell scripts. is there
>> just the top level build.sh file, or do the individual components have
>> their own?
>> 
>> On Sat, Sep 10, 2016 at 11:48 AM, suraj acharya <su...@gmail.com>
>> wrote:
>>> Hi,
>>> I have been working on AVRO-1887
>>> <https://issues.apache.org/jira/browse/AVRO-1887> to add pre-commit
>> test to
>>> avro.
>>> I have been successful in adding the personality for java file using
>> maven.
>>> I having some questions about the other plugins.
>>> Example :ant
>>> 
>>> The structure of avro is that there is a top level pom.
>>> And then there is different langs in subdirectory below it.
>>> So there is avro/lang/py
>>> <https://github.com/apache/avro/tree/master/lang/py>which has python
>> code
>>> in it. The build tool for that is ant.
>>> My understanding is running add_test ant will run ant test.
>>> However, that doesn't actually run any tests. My assumption is that since
>>> there is no build.xml present there it is unable to run any ant tests.
>>> 
>>> I tried cd to lang/py and run add_test ant from there. However, it throws
>>> an error saying unable to find pom.xml.
>>> 
>>> Can someone give me some pointers on how to continue?
>>> 
>>> 
>>> Thanks
>>> -Suraj Acharya
>>

Re: Avro Pre-commit.

Posted by suraj acharya <su...@gmail.com>.

Yes.
The ./build.sh test
<https://github.com/apache/avro/blob/master/build.sh#L40> command helps run
all the tests from the top level.
However, I am not sure if we need to run all the tests when a change is
made?
I was thinking of running language specific tests for changes in the
specific directory.
For example for changes in python will trigger ant test
<https://github.com/apache/avro/blob/master/build.sh#L43>.
If this is not a good idea let me know. Ill start another tread on dev@avro
to see what the community thinks about it

Thanks

-Suraj Acharya

On Sat, Sep 10, 2016 at 5:45 PM, Sean Busbey <bu...@apache.org> wrote:

> my understanding of Avro's build system is that there are per-language
> build tools and everything gets unified via shell scripts. is there
> just the top level build.sh file, or do the individual components have
> their own?
>
> On Sat, Sep 10, 2016 at 11:48 AM, suraj acharya <su...@gmail.com>
> wrote:
> > Hi,
> > I have been working on AVRO-1887
> > <https://issues.apache.org/jira/browse/AVRO-1887> to add pre-commit
> test to
> > avro.
> > I have been successful in adding the personality for java file using
> maven.
> > I having some questions about the other plugins.
> > Example :ant
> >
> > The structure of avro is that there is a top level pom.
> > And then there is different langs in subdirectory below it.
> > So there is avro/lang/py
> > <https://github.com/apache/avro/tree/master/lang/py>which has python
> code
> > in it. The build tool for that is ant.
> > My understanding is running add_test ant will run ant test.
> > However, that doesn't actually run any tests. My assumption is that since
> > there is no build.xml present there it is unable to run any ant tests.
> >
> > I tried cd to lang/py and run add_test ant from there. However, it throws
> > an error saying unable to find pom.xml.
> >
> > Can someone give me some pointers on how to continue?
> >
> >
> > Thanks
> > -Suraj Acharya
>

Re: Avro Pre-commit.

Posted by Sean Busbey <bu...@apache.org>.

my understanding of Avro's build system is that there are per-language
build tools and everything gets unified via shell scripts. is there
just the top level build.sh file, or do the individual components have
their own?

On Sat, Sep 10, 2016 at 11:48 AM, suraj acharya <su...@gmail.com> wrote:
> Hi,
> I have been working on AVRO-1887
> <https://issues.apache.org/jira/browse/AVRO-1887> to add pre-commit test to
> avro.
> I have been successful in adding the personality for java file using maven.
> I having some questions about the other plugins.
> Example :ant
>
> The structure of avro is that there is a top level pom.
> And then there is different langs in subdirectory below it.
> So there is avro/lang/py
> <https://github.com/apache/avro/tree/master/lang/py>which has python code
> in it. The build tool for that is ant.
> My understanding is running add_test ant will run ant test.
> However, that doesn't actually run any tests. My assumption is that since
> there is no build.xml present there it is unable to run any ant tests.
>
> I tried cd to lang/py and run add_test ant from there. However, it throws
> an error saying unable to find pom.xml.
>
> Can someone give me some pointers on how to continue?
>
>
> Thanks
> -Suraj Acharya