You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spark.apache.org by dhruve ashar <dh...@gmail.com> on 2016/05/17 18:58:30 UTC
SBT doesn't pick resource file after clean
We are trying to pick the spark version automatically from pom instead of
manually modifying the files. This also includes richer pieces of
information like last commit, version, user who built the code etc to
better identify the framework running.
The setup is as follows :
- A shell script generates this piece of information and dumps it into a
properties file under *core/target/extra-resources - *we don't want to
pollute the source directory and hence we are generating this under target
as its dealing with build information.
- The shell script is invoked in both mvn and sbt.
The issue is that sbt doesn't pick up the generated properties file after
doing a clean. But it does pick it up in subsequent runs. Note, the
properties file is created before the classes are generated.
The code for this is available in the PR :
https://github.com/apache/spark/pull/13061
Does anybody have an idea about how we can achieve this in sbt?
Thanks,
Dhruve
Re: SBT doesn't pick resource file after clean
Posted by dhruve ashar <dh...@gmail.com>.
The issue is fixed.
Here's an explanation which interested people can read through:
For the earlier mail, the default resourceDirectory =>
core/src/main/resources didn't yield the expected result. By default all
the static resources placed under this directory are picked up and included
in the jar. But not the dynamic ones.
From running the build multiple times and trying out some basic combos, I
inferred that internally sbt prepares a list of files to be copied before
creating the jar and these files have to exist at the time of list
creation. Dynamically created files do not make it to the list and hence
are not copied.
So here's what was happening:
On the first run:
1. Perform clean - file doesn't exist
2. SBT prepares a list, file doesn't make it to the list.
3. Script invoked which generates the properties file.
4. jar created, but props file missing.
On subsequent runs:
1. props file exists from previous run.
2. SBT prepares a list, this time it appears in the list.
3. Script overwrites it with latest data.
4. jar created, with props file containing latest data.
So the whole idea was to inform sbt to add the dynamically generated file
in its list of files to be copied. So here's what we did:
resourceGenerators in Compile += {
1. Invoke shell script - this creates the props file.
2. Obtain a handle to the file
3. Return the file handle so that it gets included in the list and gets
picked up the first time as well.
}
There is no need of providing unmanagedResourceDirectories as it picks up
only static files. For dynamically generated files, you must return the
file handle to be included. It doesn't pick it up from the listed
directory.
Note: It was easy to miss this as the shell script was generating the file.
The behavior was also verified using a simple sbt project.
-Dhruve
On Fri, May 20, 2016 at 2:48 PM, Jakob Odersky <ja...@odersky.com> wrote:
> Ah, I think I see the issue. resourceManaged and core/src/resources
> aren't included in the classpath; to achieve that, you need to scope
> the setting to either "compile" or "test" (probably compile in your
> case). So, the simplest way to add the extra settings would be
> something like:
>
> resourceGenerators in Compile += {
> val file = //generate properties file
> IO.copy(file, (resourceManaged in Compile).value / "foo.properties")
>
> On Thu, May 19, 2016 at 7:21 PM, dhruve ashar <dh...@gmail.com>
> wrote:
> > Based on the conversation on PR, the intent was not to pollute the source
> > directory and hence we are placing the generated file outside it in the
> > target/extra-resources directory. I agree that the "sbt way" is to add
> the
> > generated resources under the resourceManaged setting which was
> essentially
> > the earlier approach implemented.
> >
> > However, even on generating the file under the default
> resourceDirectory =>
> > core/src/resources doesn't pick the file in jar after doing a clean. So
> this
> > seems to be a different issue.
> >
> >
> >
> >
> >
> > On Thu, May 19, 2016 at 4:17 PM, Jakob Odersky <ja...@odersky.com>
> wrote:
> >>
> >> To echo my comment on the PR: I think the "sbt way" to add extra,
> >> generated resources to the classpath is by adding a new task to the
> >> `resourceGenerators` setting. Also, the task should output any files
> >> into the directory specified by the `resourceManaged` setting. See
> >> http://www.scala-sbt.org/0.13/docs/Howto-Generating-Files.html. There
> >> shouldn't by any issues with clean if you follow the above
> >> conventions.
> >>
> >> On Tue, May 17, 2016 at 12:00 PM, Marcelo Vanzin <va...@cloudera.com>
> >> wrote:
> >> > Perhaps you need to make the "compile" task of the appropriate module
> >> > depend on the task that generates the resource file?
> >> >
> >> > Sorry but my knowledge of sbt doesn't really go too far.
> >> >
> >> > On Tue, May 17, 2016 at 11:58 AM, dhruve ashar <dhruveashar@gmail.com
> >
> >> > wrote:
> >> >> We are trying to pick the spark version automatically from pom
> instead
> >> >> of
> >> >> manually modifying the files. This also includes richer pieces of
> >> >> information like last commit, version, user who built the code etc to
> >> >> better
> >> >> identify the framework running.
> >> >>
> >> >> The setup is as follows :
> >> >> - A shell script generates this piece of information and dumps it
> into
> >> >> a
> >> >> properties file under core/target/extra-resources - we don't want to
> >> >> pollute
> >> >> the source directory and hence we are generating this under target as
> >> >> its
> >> >> dealing with build information.
> >> >> - The shell script is invoked in both mvn and sbt.
> >> >>
> >> >> The issue is that sbt doesn't pick up the generated properties file
> >> >> after
> >> >> doing a clean. But it does pick it up in subsequent runs. Note, the
> >> >> properties file is created before the classes are generated.
> >> >>
> >> >> The code for this is available in the PR :
> >> >> https://github.com/apache/spark/pull/13061
> >> >>
> >> >> Does anybody have an idea about how we can achieve this in sbt?
> >> >>
> >> >> Thanks,
> >> >> Dhruve
> >> >>
> >> >
> >> >
> >> >
> >> > --
> >> > Marcelo
> >> >
> >> > ---------------------------------------------------------------------
> >> > To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
> >> > For additional commands, e-mail: dev-help@spark.apache.org
> >> >
> >
> >
> >
> >
> > --
> > -Dhruve Ashar
> >
>
--
-Dhruve Ashar
Re: SBT doesn't pick resource file after clean
Posted by Jakob Odersky <ja...@odersky.com>.
Ah, I think I see the issue. resourceManaged and core/src/resources
aren't included in the classpath; to achieve that, you need to scope
the setting to either "compile" or "test" (probably compile in your
case). So, the simplest way to add the extra settings would be
something like:
resourceGenerators in Compile += {
val file = //generate properties file
IO.copy(file, (resourceManaged in Compile).value / "foo.properties")
On Thu, May 19, 2016 at 7:21 PM, dhruve ashar <dh...@gmail.com> wrote:
> Based on the conversation on PR, the intent was not to pollute the source
> directory and hence we are placing the generated file outside it in the
> target/extra-resources directory. I agree that the "sbt way" is to add the
> generated resources under the resourceManaged setting which was essentially
> the earlier approach implemented.
>
> However, even on generating the file under the default resourceDirectory =>
> core/src/resources doesn't pick the file in jar after doing a clean. So this
> seems to be a different issue.
>
>
>
>
>
> On Thu, May 19, 2016 at 4:17 PM, Jakob Odersky <ja...@odersky.com> wrote:
>>
>> To echo my comment on the PR: I think the "sbt way" to add extra,
>> generated resources to the classpath is by adding a new task to the
>> `resourceGenerators` setting. Also, the task should output any files
>> into the directory specified by the `resourceManaged` setting. See
>> http://www.scala-sbt.org/0.13/docs/Howto-Generating-Files.html. There
>> shouldn't by any issues with clean if you follow the above
>> conventions.
>>
>> On Tue, May 17, 2016 at 12:00 PM, Marcelo Vanzin <va...@cloudera.com>
>> wrote:
>> > Perhaps you need to make the "compile" task of the appropriate module
>> > depend on the task that generates the resource file?
>> >
>> > Sorry but my knowledge of sbt doesn't really go too far.
>> >
>> > On Tue, May 17, 2016 at 11:58 AM, dhruve ashar <dh...@gmail.com>
>> > wrote:
>> >> We are trying to pick the spark version automatically from pom instead
>> >> of
>> >> manually modifying the files. This also includes richer pieces of
>> >> information like last commit, version, user who built the code etc to
>> >> better
>> >> identify the framework running.
>> >>
>> >> The setup is as follows :
>> >> - A shell script generates this piece of information and dumps it into
>> >> a
>> >> properties file under core/target/extra-resources - we don't want to
>> >> pollute
>> >> the source directory and hence we are generating this under target as
>> >> its
>> >> dealing with build information.
>> >> - The shell script is invoked in both mvn and sbt.
>> >>
>> >> The issue is that sbt doesn't pick up the generated properties file
>> >> after
>> >> doing a clean. But it does pick it up in subsequent runs. Note, the
>> >> properties file is created before the classes are generated.
>> >>
>> >> The code for this is available in the PR :
>> >> https://github.com/apache/spark/pull/13061
>> >>
>> >> Does anybody have an idea about how we can achieve this in sbt?
>> >>
>> >> Thanks,
>> >> Dhruve
>> >>
>> >
>> >
>> >
>> > --
>> > Marcelo
>> >
>> > ---------------------------------------------------------------------
>> > To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
>> > For additional commands, e-mail: dev-help@spark.apache.org
>> >
>
>
>
>
> --
> -Dhruve Ashar
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
For additional commands, e-mail: dev-help@spark.apache.org
Re: SBT doesn't pick resource file after clean
Posted by dhruve ashar <dh...@gmail.com>.
Based on the conversation on PR, the intent was not to pollute the source
directory and hence we are placing the generated file outside it in the
target/extra-resources directory. I agree that the "sbt way" is to add the
generated resources under the resourceManaged setting which was essentially
the earlier approach implemented.
However, even on generating the file under the default resourceDirectory
=> core/src/resources doesn't pick the file in jar after doing a clean. So
this seems to be a different issue.
On Thu, May 19, 2016 at 4:17 PM, Jakob Odersky <ja...@odersky.com> wrote:
> To echo my comment on the PR: I think the "sbt way" to add extra,
> generated resources to the classpath is by adding a new task to the
> `resourceGenerators` setting. Also, the task should output any files
> into the directory specified by the `resourceManaged` setting. See
> http://www.scala-sbt.org/0.13/docs/Howto-Generating-Files.html. There
> shouldn't by any issues with clean if you follow the above
> conventions.
>
> On Tue, May 17, 2016 at 12:00 PM, Marcelo Vanzin <va...@cloudera.com>
> wrote:
> > Perhaps you need to make the "compile" task of the appropriate module
> > depend on the task that generates the resource file?
> >
> > Sorry but my knowledge of sbt doesn't really go too far.
> >
> > On Tue, May 17, 2016 at 11:58 AM, dhruve ashar <dh...@gmail.com>
> wrote:
> >> We are trying to pick the spark version automatically from pom instead
> of
> >> manually modifying the files. This also includes richer pieces of
> >> information like last commit, version, user who built the code etc to
> better
> >> identify the framework running.
> >>
> >> The setup is as follows :
> >> - A shell script generates this piece of information and dumps it into a
> >> properties file under core/target/extra-resources - we don't want to
> pollute
> >> the source directory and hence we are generating this under target as
> its
> >> dealing with build information.
> >> - The shell script is invoked in both mvn and sbt.
> >>
> >> The issue is that sbt doesn't pick up the generated properties file
> after
> >> doing a clean. But it does pick it up in subsequent runs. Note, the
> >> properties file is created before the classes are generated.
> >>
> >> The code for this is available in the PR :
> >> https://github.com/apache/spark/pull/13061
> >>
> >> Does anybody have an idea about how we can achieve this in sbt?
> >>
> >> Thanks,
> >> Dhruve
> >>
> >
> >
> >
> > --
> > Marcelo
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
> > For additional commands, e-mail: dev-help@spark.apache.org
> >
>
--
-Dhruve Ashar
Re: SBT doesn't pick resource file after clean
Posted by Jakob Odersky <ja...@odersky.com>.
To echo my comment on the PR: I think the "sbt way" to add extra,
generated resources to the classpath is by adding a new task to the
`resourceGenerators` setting. Also, the task should output any files
into the directory specified by the `resourceManaged` setting. See
http://www.scala-sbt.org/0.13/docs/Howto-Generating-Files.html. There
shouldn't by any issues with clean if you follow the above
conventions.
On Tue, May 17, 2016 at 12:00 PM, Marcelo Vanzin <va...@cloudera.com> wrote:
> Perhaps you need to make the "compile" task of the appropriate module
> depend on the task that generates the resource file?
>
> Sorry but my knowledge of sbt doesn't really go too far.
>
> On Tue, May 17, 2016 at 11:58 AM, dhruve ashar <dh...@gmail.com> wrote:
>> We are trying to pick the spark version automatically from pom instead of
>> manually modifying the files. This also includes richer pieces of
>> information like last commit, version, user who built the code etc to better
>> identify the framework running.
>>
>> The setup is as follows :
>> - A shell script generates this piece of information and dumps it into a
>> properties file under core/target/extra-resources - we don't want to pollute
>> the source directory and hence we are generating this under target as its
>> dealing with build information.
>> - The shell script is invoked in both mvn and sbt.
>>
>> The issue is that sbt doesn't pick up the generated properties file after
>> doing a clean. But it does pick it up in subsequent runs. Note, the
>> properties file is created before the classes are generated.
>>
>> The code for this is available in the PR :
>> https://github.com/apache/spark/pull/13061
>>
>> Does anybody have an idea about how we can achieve this in sbt?
>>
>> Thanks,
>> Dhruve
>>
>
>
>
> --
> Marcelo
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
> For additional commands, e-mail: dev-help@spark.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
For additional commands, e-mail: dev-help@spark.apache.org
Re: SBT doesn't pick resource file after clean
Posted by Marcelo Vanzin <va...@cloudera.com>.
Perhaps you need to make the "compile" task of the appropriate module
depend on the task that generates the resource file?
Sorry but my knowledge of sbt doesn't really go too far.
On Tue, May 17, 2016 at 11:58 AM, dhruve ashar <dh...@gmail.com> wrote:
> We are trying to pick the spark version automatically from pom instead of
> manually modifying the files. This also includes richer pieces of
> information like last commit, version, user who built the code etc to better
> identify the framework running.
>
> The setup is as follows :
> - A shell script generates this piece of information and dumps it into a
> properties file under core/target/extra-resources - we don't want to pollute
> the source directory and hence we are generating this under target as its
> dealing with build information.
> - The shell script is invoked in both mvn and sbt.
>
> The issue is that sbt doesn't pick up the generated properties file after
> doing a clean. But it does pick it up in subsequent runs. Note, the
> properties file is created before the classes are generated.
>
> The code for this is available in the PR :
> https://github.com/apache/spark/pull/13061
>
> Does anybody have an idea about how we can achieve this in sbt?
>
> Thanks,
> Dhruve
>
--
Marcelo
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
For additional commands, e-mail: dev-help@spark.apache.org