You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@ofbiz.apache.org by Hans Bakker <h....@antwebsystems.com> on 2009/06/26 11:48:09 UTC
webslinger quick start guide?
I see all this webslinger stuff added and i really want to start using
it, is it possible to write a couple of sentences to get started?
a single example test with direction how to run it would do it.
regards,
Hans
--
http://www.antwebsystems.com :
Quality OFBiz support for competitive rates....
Re: webslinger quick start guide?
Posted by David E Jones <de...@me.com>.
If we were to use GIT for this it wouldn't be in that way, it would be
writing to the central repo in order to keep multiple clients
synchronized while each is making changes and wanting to see the
changes of others.
Perhaps it would be better to discuss for SVN and not for GIT at all,
and just keep it more simple.
As for the revision number: the whole point would be to have revision
control and do things like allow changes on other servers, or other
content records, while a specific revision is deployed to production
(just as is the case for development efforts).
The initial point that I mentioned was to provide a central repository
that is a good alternative to a database, and that may involve
branching and merging for more advanced users, but that wouldn't be
the main point.
-David
On Jul 1, 2009, at 5:08 PM, Adam Heath wrote:
> David E Jones wrote:
>>
>> This is an interesting overview and while I'm not sure why I hadn't
>> thought along these lines before, at least it's through my thick
>> skull
>> now...
>>
>> I asked Adam about how this would deploy on multiple servers with the
>> stuff in the filesystem versus the database, and I think what you've
>> written Ean is the answer.
>>
>> Why not treat a source repo (either plain SVN or something more
>> exotic
>> like GIT) like the database? Each app server would read from and
>> write
>> to the source repo just like it would a database record. If SVN or
>> GIT
>> support 2-phase commits we could probably even do write operations in
>> the a transaction that includes connections to both data stores.
>>
>> For performance reasons you'd want to cache content from the source
>> repo
>> just like you would content from a relational database. If it's
>> really
>> too terribly slow even doing that (ie reading directly from the
>> repo and
>> caching) you could cache it locally in the app server's file system,
>> though it would probably be best to never write directly to the local
>> filesystem and you'd want some sort of timeout or other logic to
>> invalidate the file system cache just like you'd do with the in
>> memory
>> cache (actually UtilCache supports this sort of thing, though now
>> with
>> straight files in the filesystem, just a sort of mini-database for
>> local
>> filesystem caching of data).
>>
>> Anyway, is this something you guys have considered for WebSlinger?
>
> I've got a commons-vfs filesystem implementation that uses git
> plumbing to store content. Every single mutation causes a new 'tree'
> hash to be created in git. It uses jgit to do this. However, we
> don't currently use it, it was more of a quick test. One major
> problem with jgit is that it reads the entire file into memory, which
> will not work with large files.
>
> I have not tested whether this interoperates with other git porcelain.
>
> However, all that is moot. GIT is not a shared-write system. Each
> instance is completely local. You have your own copy of the repo, per
> install. You mutate it however. Then either you push to another
> machine/repo, or the other machine pulls from you. This could be made
> to work, doing some kind of anonymous ssh pulse thing, but it'd be a
> heavy system integration, which ofbiz tends not to do.
>
>> For the OFBiz Content side of things you could pretty easily have a
>> DataResourceType for data in a source repo (ie instead of LOCAL_FILE
>> something like REPOSITORY_FILE). On the DataResource entity the
>> objectInfo field would have the URL/location of the resource (ie like
>> the SVN/HTTP URL), and we could add a field like "revisionNumber" to
>> specify which revision we want or null to get the head revision (I
>> was
>> thinking we could use the existing ContentRevision/Item entities for
>> this, but looking at them it seems they wouldn't work so well and are
>> really meant for a revision control built on top of the Content and
>> DataResource entities, and not one that would describe revision
>> information pointed to by them). The "revisionNumber" could also go
>> on
>> the Content entity so that we could have multiple Content records
>> with
>> different revision numbers pointing to the same DataResource
>> records and
>> reduce how many DataResource records we would require. That would
>> also
>> better fit how Content and DataResource are meant to work together,
>> but
>> on the other hand might be somewhat confusing.
>
> No, no, you can't use a revisionNumber. They don't exist.
> Distributed systems change that completely.
>
>> Thoughts anyone?
>>
>> Oh, one more thing... I know there are some Java libraries for SVN,
>> and
>> there probably are some for GIT... has anyone played with these?
>
> I've look at the documentation for svn/java; I've actually used
> jgit(however, it's been a few years).
Re: webslinger quick start guide?
Posted by Adam Heath <do...@brainfood.com>.
David E Jones wrote:
>
> This is an interesting overview and while I'm not sure why I hadn't
> thought along these lines before, at least it's through my thick skull
> now...
>
> I asked Adam about how this would deploy on multiple servers with the
> stuff in the filesystem versus the database, and I think what you've
> written Ean is the answer.
>
> Why not treat a source repo (either plain SVN or something more exotic
> like GIT) like the database? Each app server would read from and write
> to the source repo just like it would a database record. If SVN or GIT
> support 2-phase commits we could probably even do write operations in
> the a transaction that includes connections to both data stores.
>
> For performance reasons you'd want to cache content from the source repo
> just like you would content from a relational database. If it's really
> too terribly slow even doing that (ie reading directly from the repo and
> caching) you could cache it locally in the app server's file system,
> though it would probably be best to never write directly to the local
> filesystem and you'd want some sort of timeout or other logic to
> invalidate the file system cache just like you'd do with the in memory
> cache (actually UtilCache supports this sort of thing, though now with
> straight files in the filesystem, just a sort of mini-database for local
> filesystem caching of data).
>
> Anyway, is this something you guys have considered for WebSlinger?
I've got a commons-vfs filesystem implementation that uses git
plumbing to store content. Every single mutation causes a new 'tree'
hash to be created in git. It uses jgit to do this. However, we
don't currently use it, it was more of a quick test. One major
problem with jgit is that it reads the entire file into memory, which
will not work with large files.
I have not tested whether this interoperates with other git porcelain.
However, all that is moot. GIT is not a shared-write system. Each
instance is completely local. You have your own copy of the repo, per
install. You mutate it however. Then either you push to another
machine/repo, or the other machine pulls from you. This could be made
to work, doing some kind of anonymous ssh pulse thing, but it'd be a
heavy system integration, which ofbiz tends not to do.
> For the OFBiz Content side of things you could pretty easily have a
> DataResourceType for data in a source repo (ie instead of LOCAL_FILE
> something like REPOSITORY_FILE). On the DataResource entity the
> objectInfo field would have the URL/location of the resource (ie like
> the SVN/HTTP URL), and we could add a field like "revisionNumber" to
> specify which revision we want or null to get the head revision (I was
> thinking we could use the existing ContentRevision/Item entities for
> this, but looking at them it seems they wouldn't work so well and are
> really meant for a revision control built on top of the Content and
> DataResource entities, and not one that would describe revision
> information pointed to by them). The "revisionNumber" could also go on
> the Content entity so that we could have multiple Content records with
> different revision numbers pointing to the same DataResource records and
> reduce how many DataResource records we would require. That would also
> better fit how Content and DataResource are meant to work together, but
> on the other hand might be somewhat confusing.
No, no, you can't use a revisionNumber. They don't exist.
Distributed systems change that completely.
> Thoughts anyone?
>
> Oh, one more thing... I know there are some Java libraries for SVN, and
> there probably are some for GIT... has anyone played with these?
I've look at the documentation for svn/java; I've actually used
jgit(however, it's been a few years).
Re: webslinger quick start guide?
Posted by David E Jones <de...@me.com>.
This is an interesting overview and while I'm not sure why I hadn't
thought along these lines before, at least it's through my thick skull
now...
I asked Adam about how this would deploy on multiple servers with the
stuff in the filesystem versus the database, and I think what you've
written Ean is the answer.
Why not treat a source repo (either plain SVN or something more exotic
like GIT) like the database? Each app server would read from and write
to the source repo just like it would a database record. If SVN or GIT
support 2-phase commits we could probably even do write operations in
the a transaction that includes connections to both data stores.
For performance reasons you'd want to cache content from the source
repo just like you would content from a relational database. If it's
really too terribly slow even doing that (ie reading directly from the
repo and caching) you could cache it locally in the app server's file
system, though it would probably be best to never write directly to
the local filesystem and you'd want some sort of timeout or other
logic to invalidate the file system cache just like you'd do with the
in memory cache (actually UtilCache supports this sort of thing,
though now with straight files in the filesystem, just a sort of mini-
database for local filesystem caching of data).
Anyway, is this something you guys have considered for WebSlinger?
For the OFBiz Content side of things you could pretty easily have a
DataResourceType for data in a source repo (ie instead of LOCAL_FILE
something like REPOSITORY_FILE). On the DataResource entity the
objectInfo field would have the URL/location of the resource (ie like
the SVN/HTTP URL), and we could add a field like "revisionNumber" to
specify which revision we want or null to get the head revision (I was
thinking we could use the existing ContentRevision/Item entities for
this, but looking at them it seems they wouldn't work so well and are
really meant for a revision control built on top of the Content and
DataResource entities, and not one that would describe revision
information pointed to by them). The "revisionNumber" could also go on
the Content entity so that we could have multiple Content records with
different revision numbers pointing to the same DataResource records
and reduce how many DataResource records we would require. That would
also better fit how Content and DataResource are meant to work
together, but on the other hand might be somewhat confusing.
Thoughts anyone?
Oh, one more thing... I know there are some Java libraries for SVN,
and there probably are some for GIT... has anyone played with these?
-David
On Jul 1, 2009, at 9:36 AM, Ean Schuessler wrote:
> GIT excels at distributed workflows where parallel changes occur
> between multiple parties and then must be merged back together. We
> find
> that this can happen a lot in real-world content development
> situations.
> Let's take, as an example:
>
> - Us, with our development server
> - A customer production server
> - A development server in the customer's marketing department
> - A development server in the customer's IT department
>
> Then:
>
> - Marketing is making content changes and we merge with them,
> several times.
> - IT works on a separate effort that they have not informed us about
> but
> merges in marketing changes.
> - We build a tool for the marketing department that incorporates their
> changes and we push back to them.
> - IT pushes their merges to production.
> - Marketing asks IT to merge our changes into production.
>
> If you track content in the database I'm not sure how you can manage
> this kind of workflow. If you create a unique prefix for each database
> so that you can merge the entities then you will end up with
> duplicated
> party elements and other related roles. GIT, on the other hand,
> manages
> this (and much more complex) workflow with ease. We are looking at
> laminating some kind of content entity metadata integration on top of
> the revision control system based content but so far mostly store
> content in the filesystem.
>
> Al Byers wrote:
>> Could you please give a little more information about the type of
>> content
>> and merges that you see in practice, so I can see if the CMS could
>> be built
>> out to handle them?
>>
> --
> Ean Schuessler, CTO
> ean@brainfood.com
> 214-720-0700 x 315
> Brainfood, Inc.
> http://www.brainfood.com
>
Re: webslinger quick start guide?
Posted by Ean Schuessler <ea...@brainfood.com>.
Adrian Crum wrote:
> I haven't worked through it personally. In my research of WebDAV, I
> noticed that some shops were using SVN on WebDAV, and that authoring
> tools like Dreamweaver use WebDAV. So, it seemed logical to me to put
> a WebDAV interface on the content component.
I would imagine that these shops were using WebDAV on top of SVN rather
than the other way around. I believe that's what mod_dav_svn does.
> Let's say you wanted to use the content component as the backing store
> for a VCS. Then it would be possible to have a setting in the content
> component that would specify something like "Publish repository X,
> revision Y at URL Z." The content component could serve the website
> right from the repository.
Yes, a great idea. Even more useful is "publish repository X, *branch* Y
at URL Z" maybe with a from/thru_date while we're at it.
> I don't know if it's a good idea or not, but from my perspective it
> would be pretty cool.
I definitely think that its a cool idea.
--
Ean Schuessler, CTO
ean@brainfood.com
214-720-0700 x 315
Brainfood, Inc.
http://www.brainfood.com
Re: webslinger quick start guide?
Posted by Adrian Crum <ad...@hlmksw.com>.
I haven't worked through it personally. In my research of WebDAV, I
noticed that some shops were using SVN on WebDAV, and that authoring
tools like Dreamweaver use WebDAV. So, it seemed logical to me to put a
WebDAV interface on the content component.
Let's say you wanted to use the content component as the backing store
for a VCS. Then it would be possible to have a setting in the content
component that would specify something like "Publish repository X,
revision Y at URL Z." The content component could serve the website
right from the repository.
I don't know if it's a good idea or not, but from my perspective it
would be pretty cool.
-Adrian
Ean Schuessler wrote:
> Adrian Crum wrote:
>> Again, that can be handled with WebDAV. GIT can store its repository
>> on a WebDAV server. A WebDAV interface to the content component could
>> be designed.
> I'm not sure you are working through what would happen there. Would the
> .git directory also be stored in WebDAV? The number and complexity of
> Content objects that would create would be interesting. I would also be
> concerned about the performance.
>
Re: webslinger quick start guide?
Posted by Ean Schuessler <ea...@brainfood.com>.
Adrian Crum wrote:
> Again, that can be handled with WebDAV. GIT can store its repository
> on a WebDAV server. A WebDAV interface to the content component could
> be designed.
I'm not sure you are working through what would happen there. Would the
.git directory also be stored in WebDAV? The number and complexity of
Content objects that would create would be interesting. I would also be
concerned about the performance.
--
Ean Schuessler, CTO
ean@brainfood.com
214-720-0700 x 315
Brainfood, Inc.
http://www.brainfood.com
Re: webslinger quick start guide?
Posted by Adrian Crum <ad...@hlmksw.com>.
Again, that can be handled with WebDAV. GIT can store its repository on
a WebDAV server. A WebDAV interface to the content component could be
designed.
-Adrian
Ean Schuessler wrote:
> GIT excels at distributed workflows where parallel changes occur
> between multiple parties and then must be merged back together. We find
> that this can happen a lot in real-world content development situations.
> Let's take, as an example:
>
> - Us, with our development server
> - A customer production server
> - A development server in the customer's marketing department
> - A development server in the customer's IT department
>
> Then:
>
> - Marketing is making content changes and we merge with them, several times.
> - IT works on a separate effort that they have not informed us about but
> merges in marketing changes.
> - We build a tool for the marketing department that incorporates their
> changes and we push back to them.
> - IT pushes their merges to production.
> - Marketing asks IT to merge our changes into production.
>
> If you track content in the database I'm not sure how you can manage
> this kind of workflow. If you create a unique prefix for each database
> so that you can merge the entities then you will end up with duplicated
> party elements and other related roles. GIT, on the other hand, manages
> this (and much more complex) workflow with ease. We are looking at
> laminating some kind of content entity metadata integration on top of
> the revision control system based content but so far mostly store
> content in the filesystem.
>
> Al Byers wrote:
>> Could you please give a little more information about the type of content
>> and merges that you see in practice, so I can see if the CMS could be built
>> out to handle them?
>>
Re: webslinger quick start guide?
Posted by Ean Schuessler <ea...@brainfood.com>.
GIT excels at distributed workflows where parallel changes occur
between multiple parties and then must be merged back together. We find
that this can happen a lot in real-world content development situations.
Let's take, as an example:
- Us, with our development server
- A customer production server
- A development server in the customer's marketing department
- A development server in the customer's IT department
Then:
- Marketing is making content changes and we merge with them, several times.
- IT works on a separate effort that they have not informed us about but
merges in marketing changes.
- We build a tool for the marketing department that incorporates their
changes and we push back to them.
- IT pushes their merges to production.
- Marketing asks IT to merge our changes into production.
If you track content in the database I'm not sure how you can manage
this kind of workflow. If you create a unique prefix for each database
so that you can merge the entities then you will end up with duplicated
party elements and other related roles. GIT, on the other hand, manages
this (and much more complex) workflow with ease. We are looking at
laminating some kind of content entity metadata integration on top of
the revision control system based content but so far mostly store
content in the filesystem.
Al Byers wrote:
> Could you please give a little more information about the type of content
> and merges that you see in practice, so I can see if the CMS could be built
> out to handle them?
>
--
Ean Schuessler, CTO
ean@brainfood.com
214-720-0700 x 315
Brainfood, Inc.
http://www.brainfood.com
Re: webslinger quick start guide?
Posted by Adam Heath <do...@brainfood.com>.
David E Jones wrote:
>
> How do you handle live changes to systems with multiple applications
> servers?
You mean to the filesystem backing store? The OS deals with that.
But, in all honesty, we haven't needed to deploy on any multi-server
setups.
Re: webslinger quick start guide?
Posted by Ean Schuessler <ea...@brainfood.com>.
You are in very much the same position as with modifications to the
database. You have to have some kind of distributed cache clearing support.
David E Jones wrote:
> How do you handle live changes to systems with multiple applications
> servers?
--
Ean Schuessler, CTO
ean@brainfood.com
214-720-0700 x 315
Brainfood, Inc.
http://www.brainfood.com
Re: webslinger quick start guide?
Posted by David E Jones <de...@me.com>.
How do you handle live changes to systems with multiple applications
servers?
-David
On Jun 27, 2009, at 10:02 PM, Adam Heath wrote:
> Al Byers wrote:
>> Ean,
>>
>> Could you please give a little more information about the type of
>> content
>> and merges that you see in practice, so I can see if the CMS could
>> be built
>> out to handle them?
>
> Actually, storing these things in the database is not the best way to
> go. It keeps people from using dreamweaver, photoshop, illustrator,
> flashmx, vim, emacs.
>
>
> The simplest explanation for what we have, it that the very old region
> system in ofbiz has been converted to read everything from the
> filesystem.
>
> We also have a unified resource type system. When a browser requests
> some path, we do an extension->mime type mapping. The mime type
> mapping then provides a set of metadata attributes. Those are
> combined with any attributes set directly on the resource. These
> attributes specify the type of file, page title, wrapper, template,
> etc.
>
> Resource types we currently support(not inclusive) are bsf, code,
> template, servlet, php, binary, multi-binary.
>
> Once any code resources are done, and an eventual template resource is
> called, wrapper logic is run. Then, as each subsequent template
> resource is called, it can have other templates applied to it.
>
> There is also section support.
>
> In almost all cases, we have put all the data and configuration
> options as files on disk. web.xml init parameters, servlet/filter
> mappings, servlet/filter configuration, error pages, welcome file
> lists. This was done to make it easier for non-programmers to
> configure. It was also done so that our filesystem overlay(kinda like
> COW) would work.
>
> This was done by implementing a nested servlet container. We've
> implemented a servlet container, that itself can be deployed inside
> any other servlet container.
>
Re: webslinger quick start guide?
Posted by Adam Heath <do...@brainfood.com>.
Al Byers wrote:
> Ean,
>
> Could you please give a little more information about the type of content
> and merges that you see in practice, so I can see if the CMS could be built
> out to handle them?
Actually, storing these things in the database is not the best way to
go. It keeps people from using dreamweaver, photoshop, illustrator,
flashmx, vim, emacs.
The simplest explanation for what we have, it that the very old region
system in ofbiz has been converted to read everything from the filesystem.
We also have a unified resource type system. When a browser requests
some path, we do an extension->mime type mapping. The mime type
mapping then provides a set of metadata attributes. Those are
combined with any attributes set directly on the resource. These
attributes specify the type of file, page title, wrapper, template, etc.
Resource types we currently support(not inclusive) are bsf, code,
template, servlet, php, binary, multi-binary.
Once any code resources are done, and an eventual template resource is
called, wrapper logic is run. Then, as each subsequent template
resource is called, it can have other templates applied to it.
There is also section support.
In almost all cases, we have put all the data and configuration
options as files on disk. web.xml init parameters, servlet/filter
mappings, servlet/filter configuration, error pages, welcome file
lists. This was done to make it easier for non-programmers to
configure. It was also done so that our filesystem overlay(kinda like
COW) would work.
This was done by implementing a nested servlet container. We've
implemented a servlet container, that itself can be deployed inside
any other servlet container.
Re: webslinger quick start guide?
Posted by Al Byers <by...@automationgroups.com>.
Ean,
Could you please give a little more information about the type of content
and merges that you see in practice, so I can see if the CMS could be built
out to handle them?
Thanks,
-Al
Re: webslinger quick start guide?
Posted by Ean Schuessler <ea...@brainfood.com>.
We've done some work to build out a set of servlet facilities that all
use the same CommonsVFS based filesystem. We've discussed building out a
ContentEntity based filesystem but that is not something that we are
using in production.
The fundamental problem we've run into is that version control on
materials stored in a database is, at best, awkward. There is nothing
comparable to GIT or Mercurial for merging changes made in parallel
content pools and we've found that to happen a great deal in practice.
As a result, we've been focused on storing content in the filesystem and
adding support for metadata. We've considered linking the filesystem to
the content entities for reporting purposes but, again, that's not
something we have in production.
Adrian Crum wrote:
> While I was researching WebDAV for the iCalendar integration, I
> noticed DreamWeaver supports it. I was thinking it would be helpful to
> have a WebDAV interface for the content component so you could use
> DreamWeaver as a front end for the OFBiz CMS. Would that duplicate the
> Webslinger stuff?
--
Ean Schuessler, CTO
ean@brainfood.com
214-720-0700 x 315
Brainfood, Inc.
http://www.brainfood.com
Re: webslinger quick start guide?
Posted by Adrian Crum <ad...@hlmksw.com>.
Ean,
While I was researching WebDAV for the iCalendar integration, I noticed
DreamWeaver supports it. I was thinking it would be helpful to have a
WebDAV interface for the content component so you could use DreamWeaver
as a front end for the OFBiz CMS. Would that duplicate the Webslinger stuff?
-Adrian
Ean Schuessler wrote:
> Hans Bakker wrote:
>> I see all this webslinger stuff added and i really want to start using
>> it, is it possible to write a couple of sentences to get started?
>>
>> a single example test with direction how to run it would do it.
>>
> Adam and I have been talking about writing some introductory material
> and screencasts. Webslinger is mostly actually a different take on the
> control servlet more than a content management system. That said, we
> have some content management stuff that we have built on top of that
> infrastructure. The main thing it tries to achieve is a decoupling
> between the decorative part of your HTML (ie. "the template") and the
> actual content. In that sense I guess you could view it as a tool to
> further extract the "model" from the "view" in content management where
> those two things have a tendency to bleed into one and other.
>
> We'll work on some "get started" style material. We are currently well
> on the way to extracting some baseline functionality from our new
> website into a foundation that could be used in other things.
>
Re: webslinger quick start guide?
Posted by Ean Schuessler <ea...@brainfood.com>.
Hans Bakker wrote:
> I see all this webslinger stuff added and i really want to start using
> it, is it possible to write a couple of sentences to get started?
>
> a single example test with direction how to run it would do it.
>
Adam and I have been talking about writing some introductory material
and screencasts. Webslinger is mostly actually a different take on the
control servlet more than a content management system. That said, we
have some content management stuff that we have built on top of that
infrastructure. The main thing it tries to achieve is a decoupling
between the decorative part of your HTML (ie. "the template") and the
actual content. In that sense I guess you could view it as a tool to
further extract the "model" from the "view" in content management where
those two things have a tendency to bleed into one and other.
We'll work on some "get started" style material. We are currently well
on the way to extracting some baseline functionality from our new
website into a foundation that could be used in other things.
--
Ean Schuessler, CTO
ean@brainfood.com
214-720-0700 x 315
Brainfood, Inc.
http://www.brainfood.com