You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@subversion.apache.org by "Dymetman, Marc" <Ma...@xrce.xerox.com> on 2009/05/13 11:26:42 UTC
How to avoid cluttering the svn repository with several versions of
compiled files
Hi, I am a newbie to SVN, and have the following question:
Suppose that I am collaborating with several people on producing a book
using LaTeX.
I would like to use SVN not only for versioning the sources, but also I
would like to have in the repository an up-to-date version of the .pdf
(or .dvi, or .ps) compiled from these sources. The reason for that is
that, although the pdf is recoverable from the sources, it is convenient
for everybody to see the latest version without having to recompile the
sources.
I could compile the pdf in my working copy, and add it to the svn
repository, but then I would be cluttering the repository with many
versions of the pdf (for which SVN is probably not doing a good job at
incremental diffs).
Is there a way to tell SVN that the latest pdf file should *replace* the
previous one, that is, to force deletion of the previous revisions of
that file?
Of course I could save the pdf file somewhere else, not under version
control, but perhaps there is a way to do that inside svn?
(I guess my question would also make sense for binaries compiled from
programs, not only for Latex docs).
- Marc
------------------------------------------------------
http://subversion.tigris.org/ds/viewMessage.do?dsForumId=1065&dsMessageId=2235426
To unsubscribe from this discussion, e-mail: [users-unsubscribe@subversion.tigris.org].
RE: How to avoid cluttering the svn repository with several
versions of compiled files
Posted by we...@tigris.org.
Returning to Subversion-users after time away on other projects.
There's a basic flaw in Subversion: it has no graceful way to delete old files from the repository. The result is.... painful for situations like this, where the files are bulky and will clutter your source control repository. This is fundamentally a religious issue: should a source control system allow flushing old code? Subversion is clearly built with the idea of "no".
The key is to use a separate repository, accessed with 'svn:external'. Your build or commit process can commit to that separate repository, you can use subdirectories there to prevent the binary files from being stored in the same place and retaining history, and you can scrub the material from that external repository far more easily than in a location that stores both source code and binaries.
Or, switch to a source control system that handles it differently.
> Hi, I am a newbie to SVN, and have the following question:
>
>
>
> Suppose that I am collaborating with several people on producing a book
> using LaTeX.
>
> I would like to use SVN not only for versioning the sources, but also I
> would like to have in the repository an up-to-date version of the .pdf
> (or .dvi, or .ps) compiled from these sources. The reason for that is
> that, although the pdf is recoverable from the sources, it is convenient
> for everybody to see the latest version without having to recompile the
> sources.
>
> I could compile the pdf in my working copy, and add it to the svn
> repository, but then I would be cluttering the repository with many
> versions of the pdf (for which SVN is probably not doing a good job at
> incremental diffs).
>
>
>
> Is there a way to tell SVN that the latest pdf file should *replace* the
> previous one, that is, to force deletion of the previous revisions of
> that file?
>
> Of course I could save the pdf file somewhere else, not under version
> control, but perhaps there is a way to do that inside svn?
>
> (I guess my question would also make sense for binaries compiled from
> programs, not only for Latex docs).
>
>
>
> - Marc
------------------------------------------------------
http://subversion.tigris.org/ds/viewMessage.do?dsForumId=1065&dsMessageId=2282375
To unsubscribe from this discussion, e-mail: [users-unsubscribe@subversion.tigris.org].
RE: Re: How to avoid cluttering the svn repository with several
versions of compiled files
Posted by we...@tigris.org.
Thanks to Andreas and Andy for the detailed responses.
------------------------------------------------------
http://subversion.tigris.org/ds/viewMessage.do?dsForumId=1065&dsMessageId=2239089
To unsubscribe from this discussion, e-mail: [users-unsubscribe@subversion.tigris.org].
Re: How to avoid cluttering the svn repository with several
versions of compiled files
Posted by Andreas Schweigstill <an...@schweigstill.de>.
Hello!
Les Mikesell schrieb:
> So what's your plan for removing obsolete space-wasting binaries after
> the repository grows to a size that is inconvenient to maintain and back
> up?
There is not plan to remove such official software releases from the
repository. In this specific project the source part of the repository
is about 4GB. Every official release generated about 20MB of binary
files. There are about 10 official releases per year. So where is the
problem?
Storing such binary files on a file server also uses some space, about
the same amount as in a SVN repository. I have to admit that onn the
customer's site the siles are also stored on a file server for people
who don't have SVN accounts.
I know that there are many projects where the generated binaries are
much bigger than the source files, so it could really be a problem
to stuff them into SVN.
> Then you could periodically extract the few currently-useful versions
> and delete and rebuild the repository without bothering the one holding
> the source where you probably want the full history.
In this project the generated files have to be kept forever in order to
fulfill requirements of the certification authorities. And also I think
that it would really, really bad practice to delete any file which was
an important part of released products, especially for product which
have a life cycle of >=10 years.
Regards
Andreas Schweigstill
--
BITTE BEACHTEN SIE UNSERE NEUE TELEFONNUMMER!
Dipl.-Phys. Andreas Schweigstill
Schweigstill IT | Embedded Systems
Schauenburgerstraße 116, D-24118 Kiel, Germany
Phone: (+49) 431 53035-435, Fax: (+49) 431 53035-436
Mobile: (+49) 171 6921973, Web: http://www.schweigstill.de/
------------------------------------------------------
http://subversion.tigris.org/ds/viewMessage.do?dsForumId=1065&dsMessageId=2268532
To unsubscribe from this discussion, e-mail: [users-unsubscribe@subversion.tigris.org].
Re: How to avoid cluttering the svn repository with several
versions of compiled files
Posted by Les Mikesell <le...@gmail.com>.
Andreas Schweigstill wrote:
> Hello!
>
> Andy Levy schrieb:
>> Aside from space
>> efficiencies (which may not even be realized, due these being binary
>> files), there's not much to be gained by putting releases into
>> Subversion instead of a decently-structured archive directory on a
>> server.
>
> But this requires access to a file server which is not always given.
> For some customer projects I only have a VPN connection to the SVN
> server.
>
> And I think I make really a lot of sense to keep source and binaries
> for such purposes in a common location.
So what's your plan for removing obsolete space-wasting binaries after
the repository grows to a size that is inconvenient to maintain and back up?
> File server locations don't
> allow/force the user to put a message directly onto the stored files.
> It would be possible to create files like README.txt or RELEASENOTES.txt
> but they don't "stick" to the generated binaries.
File servers would let you put anything you want on them. You'd
probably want automated build tools for the project to ensure that the
binaries at a location actually matched the expected source
revision/tag. A compromise approach might be to maintain separate
subversion repositories for the binaries with a tag naming convention
for the contents. That would let you use the same set of tools for
remote access and with http(s) you could make the URL's look similar.
Then you could periodically extract the few currently-useful versions
and delete and rebuild the repository without bothering the one holding
the source where you probably want the full history.
--
Les Mikesell
lesmikesell@gmail.com
------------------------------------------------------
http://subversion.tigris.org/ds/viewMessage.do?dsForumId=1065&dsMessageId=2258805
To unsubscribe from this discussion, e-mail: [users-unsubscribe@subversion.tigris.org].
Re: How to avoid cluttering the svn repository with several
versions of compiled files
Posted by Andreas Schweigstill <an...@schweigstill.de>.
Hello!
Andy Levy schrieb:
> Aside from space
> efficiencies (which may not even be realized, due these being binary
> files), there's not much to be gained by putting releases into
> Subversion instead of a decently-structured archive directory on a
> server.
But this requires access to a file server which is not always given.
For some customer projects I only have a VPN connection to the SVN
server.
And I think I make really a lot of sense to keep source and binaries
for such purposes in a common location. File server locations don't
allow/force the user to put a message directly onto the stored files.
It would be possible to create files like README.txt or RELEASENOTES.txt
but they don't "stick" to the generated binaries.
Regards
Andreas Schweigstill
--
BITTE BEACHTEN SIE UNSERE NEUE TELEFONNUMMER!
Dipl.-Phys. Andreas Schweigstill
Schweigstill IT | Embedded Systems
Schauenburgerstraße 116, D-24118 Kiel, Germany
Phone: (+49) 431 53035-435, Fax: (+49) 431 53035-436
Mobile: (+49) 171 6921973, Web: http://www.schweigstill.de/
------------------------------------------------------
http://subversion.tigris.org/ds/viewMessage.do?dsForumId=1065&dsMessageId=2258217
To unsubscribe from this discussion, e-mail: [users-unsubscribe@subversion.tigris.org].
Re: How to avoid cluttering the svn repository with several
versions of compiled files
Posted by Andy Levy <an...@gmail.com>.
On Wed, May 13, 2009 at 11:31, Andreas Schweigstill
<an...@schweigstill.de> wrote:
> Hello!
>
> Andy Levy schrieb:
>> If you can produce your compiled item from resources in the repository
>> (sources, stylesheets, makefiles, etc.), then there's no real need to
>> store the compiled items themselves - you can reproduce them anytime
>> you want in the future. Most people *don't* keep compiled binaries in
>> their repositories.
>
> Hmm, a reproduction of old generated files is only possible if the build
> environment hasn't changed and if there are no other influences to the
> build process, e.g. timestamps or random generator data.
>
> When I do an offical software release I distinguish between "trunk",
> "tags", "branches" and "releases", so the repository has the following
> structure:
> Since the copied content of "trunk" is located in the "sources"
> subdirectory of "releases/version1" and not in the root it is possible
> to create multiple other directories, e.g. to store the generated
> binary files. Especially if the build process contains build steps which
> have to be asynchronously or seperated from the main build the
> "binaries" directory will also be used as a transfer directory for those
> special build steps.
>
> And also some kind of release documents which are created outside the
> software build process can be stored in their own directories below
> "releases/version1", e.g. release notes which have been created by an
> external certification authority.
>
> So there are lots of reasons to store generated documents in a
> Subversion repository. But it would be really bad practice to mix source
> files with generated files, e.g. put object files in the same
> directories as source files.
For this sort of setup, I wouldn't even put the compiled items into
the repository. I would create a separate "release archive" which is
built from a tag when a release is prepared. Aside from space
efficiencies (which may not even be realized, due these being binary
files), there's not much to be gained by putting releases into
Subversion instead of a decently-structured archive directory on a
server.
------------------------------------------------------
http://subversion.tigris.org/ds/viewMessage.do?dsForumId=1065&dsMessageId=2239396
To unsubscribe from this discussion, e-mail: [users-unsubscribe@subversion.tigris.org].
Re: How to avoid cluttering the svn repository with several
versions of compiled files
Posted by Andreas Schweigstill <an...@schweigstill.de>.
Hello!
Andy Levy schrieb:
> If you can produce your compiled item from resources in the repository
> (sources, stylesheets, makefiles, etc.), then there's no real need to
> store the compiled items themselves - you can reproduce them anytime
> you want in the future. Most people *don't* keep compiled binaries in
> their repositories.
Hmm, a reproduction of old generated files is only possible if the build
environment hasn't changed and if there are no other influences to the
build process, e.g. timestamps or random generator data.
When I do an offical software release I distinguish between "trunk",
"tags", "branches" and "releases", so the repository has the following
structure:
my_project
-- trunk
| -- dir1
| -- dir2
|
-- tags
|
-- branches
| -- branch1
| | -- dir1
| | -- dir2
| -- branch2
|
-- releases
-- version1
| -- sources
| -- dir1
| -- dir2
| -- binaries
| file1.bin
| filexy.elf
| build_sources_and_copy_to_binaries_dir.sh
| build_sources_and_copy_to_binaries_dir.log
-- version2
-- sources
-- dir1
-- dir2
-- binaries
file1.bin
filexy.elf
build_sources_and_copy_to_binaries_dir.sh
build_sources_and_copy_to_binaries_dir.log
The "tags" directory structure will only be used to store certain
milestones of a project which are not identical to official product
releases.
Since the copied content of "trunk" is located in the "sources"
subdirectory of "releases/version1" and not in the root it is possible
to create multiple other directories, e.g. to store the generated
binary files. Especially if the build process contains build steps which
have to be asynchronously or seperated from the main build the
"binaries" directory will also be used as a transfer directory for those
special build steps.
And also some kind of release documents which are created outside the
software build process can be stored in their own directories below
"releases/version1", e.g. release notes which have been created by an
external certification authority.
So there are lots of reasons to store generated documents in a
Subversion repository. But it would be really bad practice to mix source
files with generated files, e.g. put object files in the same
directories as source files.
Regards
Andreas Schweigstill
--
BITTE BEACHTEN SIE UNSERE NEUE TELEFONNUMMER!
Dipl.-Phys. Andreas Schweigstill
Schweigstill IT | Embedded Systems
Schauenburgerstraße 116, D-24118 Kiel, Germany
Phone: (+49) 431 53035-435, Fax: (+49) 431 53035-436
Mobile: (+49) 171 6921973, Web: http://www.schweigstill.de/
------------------------------------------------------
http://subversion.tigris.org/ds/viewMessage.do?dsForumId=1065&dsMessageId=2239351
To unsubscribe from this discussion, e-mail: [users-unsubscribe@subversion.tigris.org].
Re: How to avoid cluttering the svn repository with several
versions of compiled files
Posted by Andy Levy <an...@gmail.com>.
On Wed, May 13, 2009 at 07:26, Dymetman, Marc
<Ma...@xrce.xerox.com> wrote:
> Hi, I am a newbie to SVN, and have the following question:
>
>
>
> Suppose that I am collaborating with several people on producing a book
> using LaTeX.
>
> I would like to use SVN not only for versioning the sources, but also I
> would like to have in the repository an up-to-date version of the .pdf (or
> .dvi, or .ps) compiled from these sources. The reason for that is that,
> although the pdf is recoverable from the sources, it is convenient for
> everybody to see the latest version without having to recompile the sources.
>
> I could compile the pdf in my working copy, and add it to the svn
> repository, but then I would be cluttering the repository with many versions
> of the pdf (for which SVN is probably not doing a good job at incremental
> diffs).
>
>
>
> Is there a way to tell SVN that the latest pdf file should *replace* the
> previous one, that is, to force deletion of the previous revisions of that
> file?
>
> Of course I could save the pdf file somewhere else, not under version
> control, but perhaps there is a way to do that inside svn?
>
> (I guess my question would also make sense for binaries compiled from
> programs, not only for Latex docs).
If you can produce your compiled item from resources in the repository
(sources, stylesheets, makefiles, etc.), then there's no real need to
store the compiled items themselves - you can reproduce them anytime
you want in the future. Most people *don't* keep compiled binaries in
their repositories.
------------------------------------------------------
http://subversion.tigris.org/ds/viewMessage.do?dsForumId=1065&dsMessageId=2236698
To unsubscribe from this discussion, e-mail: [users-unsubscribe@subversion.tigris.org].
Re: How to avoid cluttering the svn repository with several
versions of compiled files
Posted by Andreas Schweigstill <an...@schweigstill.de>.
Hello!
Dymetman, Marc schrieb:
> Is there a way to tell SVN that the latest pdf file should **replace**
> the previous one, that is, to force deletion of the previous revisions
> of that file?
No. Due to the structure of the repository the content is stored on a
per-revision base and not on a per-file base. When using FSFS there will
*never* be peformed any modifications on already existing revision
files. This is a very important fact which will also be considered in
some backup szenarios.
Regarding deletion of old information look at several discussions about
a "svn obliterate" command which take place since several years. The
main reasons for such a command wouldn't be disk space savings but
legal requirements (accidentally checked in proprietary documents,
intellectual property, patents) or accidentally checked in *very big*
files.
> Of course I could save the pdf file somewhere else, not under version
> control, but perhaps there is a way to do that inside svn?
You could store your document in a revision property but I guess this
would be a really bad idea:
svn propset my:generated-files --revprop -r 123 `tar cf - file1.pdf \
file2.pdf | uuencode'
Revision properties themselves are not versioned, so their content can
be changed afterwards.
I don't know if there are some space limitations on revision properties.
With best regards
Andreas Schweigstill
--
BITTE BEACHTEN SIE UNSERE NEUE TELEFONNUMMER!
Dipl.-Phys. Andreas Schweigstill
Schweigstill IT | Embedded Systems
Schauenburgerstraße 116, D-24118 Kiel, Germany
Phone: (+49) 431 53035-435, Fax: (+49) 431 53035-436
Mobile: (+49) 171 6921973, Web: http://www.schweigstill.de/
------------------------------------------------------
http://subversion.tigris.org/ds/viewMessage.do?dsForumId=1065&dsMessageId=2236193
To unsubscribe from this discussion, e-mail: [users-unsubscribe@subversion.tigris.org].
RE: How to avoid cluttering the svn repository with several
versions of compiled files
Posted by "Bolstridge, Andrew" <an...@intergraph.com>.
> -----Original Message-----
> From: David Weintraub [mailto:qazwart@gmail.com]
> Sent: Thursday, May 14, 2009 4:41 PM
> To: Bolstridge, Andrew
> Cc: users@subversion.tigris.org
> Subject: Re: How to avoid cluttering the svn repository with several
> versions of compiled files
[snip]
> > Instead I'm hoping svn obliterate will appear sooner rather than
> later
> > and fix this problem for everyone :)
>
> The problem is that this issue is much more difficult to fix than you
> might realize.
>
> You make five file changes in revision #4556: one.gif, two.gif,
> three.gif, four.txt five.txt. Don't worry about the actual
> implementation. Think about these questions.
>
> * I want to remove two.gif at revision #4556. What about the other
> four files? Should I be forced to remove the entire revision, or can I
> simply remove on revision? What would svn log look like? What if
> there's a branch or tag that the revision depends upon? Remember in
> Subversion tags live forever even after they're deleted. How is a tag
> that uses that particular revision of the file affected?
>
> * If I delete revision two.gif at revision #4556, should these changes
> be merged into revision #4557? Probably not. What about four.txt? If I
> delete that, should those changes be placed into four.txt in revision
> 4557? Probably yes.
>
Personally, I would just replace the contents of that file's revision
delta (the deltas would just have the re-calculated from the next to the
previous revision - ie. If I have 3 revs, I obliterate rev 2, the delta
that used to be 3->2 is recalculated as 3->1 and stored in rev3's file,
and rev2's file becomes a delta of "no changes")
If you obliterate the contents of a tag's HEAD revision whilst removing
revisions from the original trunk, then that's your lookout - you'll now
see the wrong (ie the previous) revision - but that's what happens when
you delete data, you have to know what you're doing.
Obliterating contents that have subsequently changed in a branch or tag
may need to be obliterated there as well. (ie I commit customer data,
then branch the file and edit it. I now have to obliterate the original
and the branched revisions... I think.. maybe the recalculation of
deltas on the original file would handle that)
I'd also append to the log (or more likely add a property) to the
revision to note the files that were obliterated, by whom and when.
You'd still see all the change histories (unless there was an option to
ignore revisions that were now empty), your disk space would be reduced,
and confidential 'whoops I added that file' mistakes would be easily
corrected.
If you wanted to only keep the latest version of a file, you could put
the command in a post-commit hook to obliterate the last-but-x revision
(so you'd only ever have the latest x versions).
It's a tricky situation to get perfectly right, as you say, but
something that does the big requirements (confidential data and disk
usage) would satisfy nearly everyone. Unless I've got this completely
wrong in my thinking.
------------------------------------------------------
http://subversion.tigris.org/ds/viewMessage.do?dsForumId=1065&dsMessageId=2259940
To unsubscribe from this discussion, e-mail: [users-unsubscribe@subversion.tigris.org].
Re: How to avoid cluttering the svn repository with several
versions of compiled files
Posted by Brian Fox <br...@infinity.nu>.
A simple put is all that is required, the full webdav stack isn't needed
since the server is able to handle missing folders, etc. Nexus OSS supports
Maven2 and Maven1 out of the box, Pro adds P2 layouts. If I had to pick
one arbitrarily right now it would be Maven2 because that's the most widely
used and has the largest tool support.
On Fri, May 15, 2009 at 2:48 AM, Stephen Connolly <
stephen.alan.connolly@gmail.com> wrote:
> 2009/5/14 David Weintraub <qa...@gmail.com>
>
>> > David Weintraub wrote:
>> >> We use both Hudson and Nexus. Hudson is our continuous build engine,
>> >> and we store the daily builds on Hudson, and depending upon the
>> >> project, we store either the last 20 builds or at least three months
>> >> worth of builds(which ever is greater). Developers and QA simply
>> >> retrieve the builds they want from Hudson.
>>
>> On Thu, May 14, 2009 at 1:21 PM, Les Mikesell <le...@gmail.com>
>> wrote:
>> > Can you comment on how it might work for non-java projects? For
>> example,
>> > common libraries for other c or c++ projects or deployable executables -
>> > probably would be managed by Hudson, though.
>>
>> Nexus can store almost any binary type file, so storing executables
>> shouldn't be a problem. The big issue is adopting the Maven naming
>> conventions for releases. For example, you have a program called
>> Foo.exe. Maven puts the release number in the file name, so it will
>> become Foo-1.2.3.exe.
>>
>> The two issues would be adding a file into the Nexus repository and
>> retrieving a file from the Nexus repository.
>>
>> To add a file you can use the Nexus web interface to add a file, or
>> you can use the Maven deploy:deploy-file "mojo". That way, when you
>> build a DLL or .so, you can automatically deploy it into Nexus. We do
>> this via a shell script.
>>
>> To retrieve a file from Maven requires either curl or wget, both open
>> source tools available for Mac, Windows, or Linux. If you're using
>> Make, you could define a default rule that fetches the library from
>> the Nexus repository.
>>
>> There's probably better tools out there for non-Maven type tasks, but
>> since we use Nexus anyway, adopting it for other non-Maven projects
>> simply made sense.
>>
>
> CCing Jason and BrianF,
>
> FYI, you can also use Maven tasks for ANT to deploy to nexus...
>
> you can also deploy via HTTP... it may be as simple as a POST or else it's
> WEBDAV... you'd have to look into it... or ask Jason/BrianF ;-)
>
> Additionally, AFAIK there are other repository layouts supported by Nexus,
> and you might not even have to follow the Maven repository layout... you'd
> have to look into it... or ask Jason/BrianF ;-)
>
> -Stephen
>
>
>
------------------------------------------------------
http://subversion.tigris.org/ds/viewMessage.do?dsForumId=1065&dsMessageId=2271227
To unsubscribe from this discussion, e-mail: [users-unsubscribe@subversion.tigris.org].
RE: How to avoid cluttering the svn repository with several
versions of compiled files
Posted by "Bolstridge, Andrew" <an...@intergraph.com>.
From: Stephen Connolly [mailto:stephen.alan.connolly@gmail.com]
Sent: Friday, May 15, 2009 7:48 AM
To: David Weintraub
Cc: Les Mikesell; Bolstridge, Andrew; users@subversion.tigris.org; brianf@apache.org; Jason van Zyl
Subject: Re: How to avoid cluttering the svn repository with several versions of compiled files
2009/5/14 David Weintraub <qa...@gmail.com>
> David Weintraub wrote:
>> We use both Hudson and Nexus. Hudson is our continuous build engine,
>> and we store the daily builds on Hudson, and depending upon the
>> project, we store either the last 20 builds or at least three months
>> worth of builds(which ever is greater). Developers and QA simply
>> retrieve the builds they want from Hudson.
On Thu, May 14, 2009 at 1:21 PM, Les Mikesell <le...@gmail.com> wrote:
> Can you comment on how it might work for non-java projects? For example,
> common libraries for other c or c++ projects or deployable executables -
> probably would be managed by Hudson, though.
I took a quick look at Nexus and it does seem to be relatively tightly integrated with Maven, but there are others - Alfresco seems quite good, or KnowledgeTree, or Nuxeo (which lists a load of media awards)
It'd be ironic if any of them used SVN as a back-end data storage module :)
------------------------------------------------------
http://subversion.tigris.org/ds/viewMessage.do?dsForumId=1065&dsMessageId=2268466
To unsubscribe from this discussion, e-mail: [users-unsubscribe@subversion.tigris.org].
Re: How to avoid cluttering the svn repository with several
versions of compiled files
Posted by Stephen Connolly <st...@gmail.com>.
2009/5/14 David Weintraub <qa...@gmail.com>
> > David Weintraub wrote:
> >> We use both Hudson and Nexus. Hudson is our continuous build engine,
> >> and we store the daily builds on Hudson, and depending upon the
> >> project, we store either the last 20 builds or at least three months
> >> worth of builds(which ever is greater). Developers and QA simply
> >> retrieve the builds they want from Hudson.
>
> On Thu, May 14, 2009 at 1:21 PM, Les Mikesell <le...@gmail.com>
> wrote:
> > Can you comment on how it might work for non-java projects? For example,
> > common libraries for other c or c++ projects or deployable executables -
> > probably would be managed by Hudson, though.
>
> Nexus can store almost any binary type file, so storing executables
> shouldn't be a problem. The big issue is adopting the Maven naming
> conventions for releases. For example, you have a program called
> Foo.exe. Maven puts the release number in the file name, so it will
> become Foo-1.2.3.exe.
>
> The two issues would be adding a file into the Nexus repository and
> retrieving a file from the Nexus repository.
>
> To add a file you can use the Nexus web interface to add a file, or
> you can use the Maven deploy:deploy-file "mojo". That way, when you
> build a DLL or .so, you can automatically deploy it into Nexus. We do
> this via a shell script.
>
> To retrieve a file from Maven requires either curl or wget, both open
> source tools available for Mac, Windows, or Linux. If you're using
> Make, you could define a default rule that fetches the library from
> the Nexus repository.
>
> There's probably better tools out there for non-Maven type tasks, but
> since we use Nexus anyway, adopting it for other non-Maven projects
> simply made sense.
>
CCing Jason and BrianF,
FYI, you can also use Maven tasks for ANT to deploy to nexus...
you can also deploy via HTTP... it may be as simple as a POST or else it's
WEBDAV... you'd have to look into it... or ask Jason/BrianF ;-)
Additionally, AFAIK there are other repository layouts supported by Nexus,
and you might not even have to follow the Maven repository layout... you'd
have to look into it... or ask Jason/BrianF ;-)
-Stephen
------------------------------------------------------
http://subversion.tigris.org/ds/viewMessage.do?dsForumId=1065&dsMessageId=2267398
To unsubscribe from this discussion, e-mail: [users-unsubscribe@subversion.tigris.org].
Re: How to avoid cluttering the svn repository with several
versions of compiled files
Posted by David Weintraub <qa...@gmail.com>.
> David Weintraub wrote:
>> We use both Hudson and Nexus. Hudson is our continuous build engine,
>> and we store the daily builds on Hudson, and depending upon the
>> project, we store either the last 20 builds or at least three months
>> worth of builds(which ever is greater). Developers and QA simply
>> retrieve the builds they want from Hudson.
On Thu, May 14, 2009 at 1:21 PM, Les Mikesell <le...@gmail.com> wrote:
> Can you comment on how it might work for non-java projects? For example,
> common libraries for other c or c++ projects or deployable executables -
> probably would be managed by Hudson, though.
Nexus can store almost any binary type file, so storing executables
shouldn't be a problem. The big issue is adopting the Maven naming
conventions for releases. For example, you have a program called
Foo.exe. Maven puts the release number in the file name, so it will
become Foo-1.2.3.exe.
The two issues would be adding a file into the Nexus repository and
retrieving a file from the Nexus repository.
To add a file you can use the Nexus web interface to add a file, or
you can use the Maven deploy:deploy-file "mojo". That way, when you
build a DLL or .so, you can automatically deploy it into Nexus. We do
this via a shell script.
To retrieve a file from Maven requires either curl or wget, both open
source tools available for Mac, Windows, or Linux. If you're using
Make, you could define a default rule that fetches the library from
the Nexus repository.
There's probably better tools out there for non-Maven type tasks, but
since we use Nexus anyway, adopting it for other non-Maven projects
simply made sense.
--
David Weintraub
qazwart@gmail.com
------------------------------------------------------
http://subversion.tigris.org/ds/viewMessage.do?dsForumId=1065&dsMessageId=2261553
To unsubscribe from this discussion, e-mail: [users-unsubscribe@subversion.tigris.org].
Re: How to avoid cluttering the svn repository with several
versions of compiled files
Posted by Les Mikesell <le...@gmail.com>.
David Weintraub wrote:
>
>> That makes sense, but it's not that common a practice, and often
>> rebuilding old revisions can be a real nightmare when all you want is an
>> old release to test a bug or re-send to a customer.
>
> That's what the release repository is for: Storing the builds, so you
> don't have to rebuild them.
>
> We use both Hudson and Nexus. Hudson is our continuous build engine,
> and we store the daily builds on Hudson, and depending upon the
> project, we store either the last 20 builds or at least three months
> worth of builds(which ever is greater). Developers and QA simply
> retrieve the builds they want from Hudson.
>
> For releases, we use Nexus as our release repository. It is made for
> Maven, but it is possible to use it for non-Maven projects. I find it
> better storing even built jarfiles that other projects use on Nexus
> rather than in the Subversion repository. We can use a shell script
> and the Ant <get> to store and retrieve these jarfiles.
Can you comment on how it might work for non-java projects? For
example, common libraries for other c or c++ projects or deployable
executables - probably would be managed by Hudson, though.
>> Instead I'm hoping svn obliterate will appear sooner rather than later
>> and fix this problem for everyone :)
>
> The problem is that this issue is much more difficult to fix than you
> might realize.
>
> You make five file changes in revision #4556: one.gif, two.gif,
> three.gif, four.txt five.txt. Don't worry about the actual
> implementation. Think about these questions.
>
> * I want to remove two.gif at revision #4556. What about the other
> four files? Should I be forced to remove the entire revision, or can I
> simply remove on revision? What would svn log look like? What if
> there's a branch or tag that the revision depends upon? Remember in
> Subversion tags live forever even after they're deleted. How is a tag
> that uses that particular revision of the file affected?
>
> * If I delete revision two.gif at revision #4556, should these changes
> be merged into revision #4557? Probably not. What about four.txt? If I
> delete that, should those changes be placed into four.txt in revision
> 4557? Probably yes.
>
> There are all sorts of philosophical issues that have to be answered
> first. I would love to see the ability to remove a particular revision
> of any file in Subversion. There's a problem of people accidentally
> checking in information (like customer information) that shouldn't be
> in the repository. Removing that information is extremely important.
The same philosophical issues apply to doing it the hard way with dump
and filter. It's hard even to split things up keeping the history if
you have moved them around in the repository.
--
Les Mikesell
lesmikesell@gmail.com
------------------------------------------------------
http://subversion.tigris.org/ds/viewMessage.do?dsForumId=1065&dsMessageId=2260491
To unsubscribe from this discussion, e-mail: [users-unsubscribe@subversion.tigris.org].
Re: How to avoid cluttering the svn repository with several
versions of compiled files
Posted by David Weintraub <qa...@gmail.com>.
On Thu, May 14, 2009 at 5:20 AM, Bolstridge, Andrew
<an...@intergraph.com> wrote:
> That makes sense, but it's not that common a practice, and often
> rebuilding old revisions can be a real nightmare when all you want is an
> old release to test a bug or re-send to a customer.
That's what the release repository is for: Storing the builds, so you
don't have to rebuild them.
We use both Hudson and Nexus. Hudson is our continuous build engine,
and we store the daily builds on Hudson, and depending upon the
project, we store either the last 20 builds or at least three months
worth of builds(which ever is greater). Developers and QA simply
retrieve the builds they want from Hudson.
For releases, we use Nexus as our release repository. It is made for
Maven, but it is possible to use it for non-Maven projects. I find it
better storing even built jarfiles that other projects use on Nexus
rather than in the Subversion repository. We can use a shell script
and the Ant <get> to store and retrieve these jarfiles.
> Nobody complains about storing 'source' images in SVN, and they often
> clutter up the repository more than generated binaries, so I don't think
> the solution is to workaround it by not storing binaries.
Source files are stored in diff format in almost all version control
systems. That means if you change two or three lines in a source file,
only those changes are stored as the next revision.
In most revision control systems, the complete binary is stored --
sometimes in compressed format. That means if you make 1 dozen file
changes build a 500Mb project, the file changes probably take up a few
megabytes at the most while you're storing anywhere between 3 to
6Gigabytes for the built program. Subversion also attempts to store
binaries in "diff" format, but my experience (especially with
compressed file formats like zipped files, jars, jpgs, and gifs, even
the smallest changes end up pretty much storing the entire binary in
the repository.
> Instead I'm hoping svn obliterate will appear sooner rather than later
> and fix this problem for everyone :)
The problem is that this issue is much more difficult to fix than you
might realize.
You make five file changes in revision #4556: one.gif, two.gif,
three.gif, four.txt five.txt. Don't worry about the actual
implementation. Think about these questions.
* I want to remove two.gif at revision #4556. What about the other
four files? Should I be forced to remove the entire revision, or can I
simply remove on revision? What would svn log look like? What if
there's a branch or tag that the revision depends upon? Remember in
Subversion tags live forever even after they're deleted. How is a tag
that uses that particular revision of the file affected?
* If I delete revision two.gif at revision #4556, should these changes
be merged into revision #4557? Probably not. What about four.txt? If I
delete that, should those changes be placed into four.txt in revision
4557? Probably yes.
There are all sorts of philosophical issues that have to be answered
first. I would love to see the ability to remove a particular revision
of any file in Subversion. There's a problem of people accidentally
checking in information (like customer information) that shouldn't be
in the repository. Removing that information is extremely important.
--
David Weintraub
qazwart@gmail.com
------------------------------------------------------
http://subversion.tigris.org/ds/viewMessage.do?dsForumId=1065&dsMessageId=2258949
To unsubscribe from this discussion, e-mail: [users-unsubscribe@subversion.tigris.org].
RE: How to avoid cluttering the svn repository with several
versions of compiled files
Posted by "Bolstridge, Andrew" <an...@intergraph.com>.
> -----Original Message-----
> From: David Weintraub [mailto:qazwart@gmail.com]
> Sent: Wednesday, May 13, 2009 5:19 PM
> To: Dymetman, Marc
> Cc: users@subversion.tigris.org
> Subject: Re: How to avoid cluttering the svn repository with several
> versions of compiled files
>
> Don't bother putting the built items into Subversion. You can't easily
> remove revisions of files once they're in the Subversion repository.
> Thus, binary builds can clutter up the Subversion repository and
> greatly increase the amount of diskspace you need.
>
> Instead, setup a non-Subversion release repository where people can
> pull off these built versions.
>
That makes sense, but it's not that common a practice, and often
rebuilding old revisions can be a real nightmare when all you want is an
old release to test a bug or re-send to a customer.
Nobody complains about storing 'source' images in SVN, and they often
clutter up the repository more than generated binaries, so I don't think
the solution is to workaround it by not storing binaries.
Instead I'm hoping svn obliterate will appear sooner rather than later
and fix this problem for everyone :)
------------------------------------------------------
http://subversion.tigris.org/ds/viewMessage.do?dsForumId=1065&dsMessageId=2253546
To unsubscribe from this discussion, e-mail: [users-unsubscribe@subversion.tigris.org].
Re: How to avoid cluttering the svn repository with several
versions of compiled files
Posted by David Weintraub <qa...@gmail.com>.
Don't bother putting the built items into Subversion. You can't easily
remove revisions of files once they're in the Subversion repository.
Thus, binary builds can clutter up the Subversion repository and
greatly increase the amount of diskspace you need.
Instead, setup a non-Subversion release repository where people can
pull off these built versions.
We use Nexus as our release repository <http://nexus.sonatype.org/>.
Nexus is a Maven repository manager, but it works fine with non-Maven
projects as well. We put all of our releases from both Maven and
non-Maven project in our Nexus repository. This gives you the ability
to version your binary files while not clogging up your Subversion
repository with obsolete files. You can easily delete obsolete
revisions in Nexus, but not if you put them into Subversion.
If you have a non-Maven project, and you want to automatically add the
built revisions into your Nexus repository, you can use a shell script
like this one:
#! /usr/bin/ksh
# deploy.sh
########################################################################
########################################################################
# Name: David Weintraub
# Purpose: To deploy all files in the api-lib directory to our Maven
# repository.
#
########################################################################
########################################################################
# CONSTANTS
#
deployDir="_build/tmp/api-lib"
archiveDir="_build/archive"
#
########################################################################
########################################################################
# FIND CURRENT DIRECTORY
#
programName=$0 #Includes Directory
dirName=$(dirname $0) #Directory Executed
if [ -z "$dirName" ]
then
dirName=`pwd` #Current Working Directory
fi
deployDir="$dirName/$deployDir" #Prepend Directory Name
archiveDir="$dirName/$archiveDir"
#
########################################################################
########################################################################
# FIND REVISION NUMBER
#
if [ "$1" ]
then
version="$1"
else
version="5.3"
fi
#
########################################################################
########################################################################
# LOOP THROUGH EACH FILE IN THE DEPLOYMENT DIRECTORY
for file in ${deployDir}/*
do
#
# ####Strip the Name of the directory and suffix from the file
#
fileName=$(basename $file .jar)
#
# ####Deploy to Maven Repository
# ####NOTE: You need to setup the repository and accounts in
~/.m2/settings.xml file
#
mvn deploy:deploy-file \
-DrepositoryId="snapshot" \
-Durl="http://aladdin.solbright.com:8082/nexus/content/repositories/snapshots/"
\
-Dfile="$file" \
-DgroupId="com.solbright" \
-DartifactId="$fileName" \
-Dversion="$version-SNAPSHOT" \
-Dpackaging="jar"\
-DgeneratePom="true" \
-DuniqueVersion="false"
done
#
# END OF PROGRAM
########################################################################
On Wed, May 13, 2009 at 7:26 AM, Dymetman, Marc
<Ma...@xrce.xerox.com> wrote:
> Hi, I am a newbie to SVN, and have the following question:
>
>
>
> Suppose that I am collaborating with several people on producing a book
> using LaTeX.
>
> I would like to use SVN not only for versioning the sources, but also I
> would like to have in the repository an up-to-date version of the .pdf (or
> .dvi, or .ps) compiled from these sources. The reason for that is that,
> although the pdf is recoverable from the sources, it is convenient for
> everybody to see the latest version without having to recompile the sources.
>
> I could compile the pdf in my working copy, and add it to the svn
> repository, but then I would be cluttering the repository with many versions
> of the pdf (for which SVN is probably not doing a good job at incremental
> diffs).
>
>
>
> Is there a way to tell SVN that the latest pdf file should *replace* the
> previous one, that is, to force deletion of the previous revisions of that
> file?
>
> Of course I could save the pdf file somewhere else, not under version
> control, but perhaps there is a way to do that inside svn?
>
> (I guess my question would also make sense for binaries compiled from
> programs, not only for Latex docs).
>
>
>
> - Marc
--
David Weintraub
qazwart@gmail.com
------------------------------------------------------
http://subversion.tigris.org/ds/viewMessage.do?dsForumId=1065&dsMessageId=2239828
To unsubscribe from this discussion, e-mail: [users-unsubscribe@subversion.tigris.org].