You are viewing a plain text version of this content. The canonical link for it is here.
Posted to derby-dev@db.apache.org by Rodrigo Madera <ro...@gmail.com> on 2006/05/05 16:30:34 UTC
New "segmented" StorageFactory Development
Hello to all,
I'm Rodrigo Madera, software developer. I need a certain feature in
the Derby database system, and I am volunteering to develop such a
feature.
My requirement is that a database can be spread into different
directories (please read the Derby Users mailing list thread:
"Spawning Data on Multiple Directories").
I am filing a JIRA feature request and I count on the community for
pointers and information on developing such a feature.
I am still seeking the knowledge and planning the necessary steps to
do such a thing. I don't have *lots* of time, but since this is a
requirement for me, I can spend some hours on it.
Thanks to all for any input/critics/suggestions,
Rodrigo Madera
Re: New "segmented" StorageFactory Development
Posted by Rodrigo Madera <ro...@gmail.com>.
Thanks for the welcome =o)
Looks like Derby has a nice structure already to get this done!
I'll follow the technical advice presented here, it sounds pretty
good. I'll be continuously posting my progress here for more advice
and feedback from the community.
Thanks again for the warm welcome,
Rodrigo Madera
On 5/5/06, Daniel John Debrunner <dj...@apache.org> wrote:
> Rodrigo Madera wrote:
>
> > Hello to all,
> >
> > I'm Rodrigo Madera, software developer. I need a certain feature in
> > the Derby database system, and I am volunteering to develop such a
> > feature.
>
> Great, please don't hesitate to ask questions on the list, people will
> help you, though remember sometimes it takes time for people to respond.
>
> >
> > My requirement is that a database can be spread into different
> > directories (please read the Derby Users mailing list thread:
> > "Spawning Data on Multiple Directories").
>
> I think Andrew's idea for the Storage factory is a good idea. I think
> the storage factory you would write would be pretty simple, the actual
> writing to disk would be by the existing file system storage factory.
> You factory would be making the decisions about which directory to
> select for any file. Something like:
>
> +------> FileSystemSF for dir1
> |
> derby --> SegmentSF ---+------> FileSystemSF for dir2
> |
> +------> FileSystemSF for dir3
>
> Thus the SegmentSF would be delegating to the FileSystemSF or maybe even
> returning the FileSystemSF objects directly.
>
> Something simple initially would get you going, say round-robin for file
> placement, then worry about total size allocation as a secondary item.
>
> It might make sense for the total size allocation to be a separate
> storage factory, that could then be used without the segmenting.
> E.g.
>
>
> +------> SizeSF(150M)->FileSystemSF for dir1
> |
> derby --> SegmentSF ---+------> SizeSF(100M)->FileSystemSF for dir2
> |
> +------> SizeSF(200M)->FileSystemSF for dir3
>
>
> Welcome to the community!
> Dan.
>
>
>
Re: New "segmented" StorageFactory Development
Posted by Daniel John Debrunner <dj...@apache.org>.
Rodrigo Madera wrote:
> Hello to all,
>
> I'm Rodrigo Madera, software developer. I need a certain feature in
> the Derby database system, and I am volunteering to develop such a
> feature.
Great, please don't hesitate to ask questions on the list, people will
help you, though remember sometimes it takes time for people to respond.
>
> My requirement is that a database can be spread into different
> directories (please read the Derby Users mailing list thread:
> "Spawning Data on Multiple Directories").
I think Andrew's idea for the Storage factory is a good idea. I think
the storage factory you would write would be pretty simple, the actual
writing to disk would be by the existing file system storage factory.
You factory would be making the decisions about which directory to
select for any file. Something like:
+------> FileSystemSF for dir1
|
derby --> SegmentSF ---+------> FileSystemSF for dir2
|
+------> FileSystemSF for dir3
Thus the SegmentSF would be delegating to the FileSystemSF or maybe even
returning the FileSystemSF objects directly.
Something simple initially would get you going, say round-robin for file
placement, then worry about total size allocation as a secondary item.
It might make sense for the total size allocation to be a separate
storage factory, that could then be used without the segmenting.
E.g.
+------> SizeSF(150M)->FileSystemSF for dir1
|
derby --> SegmentSF ---+------> SizeSF(100M)->FileSystemSF for dir2
|
+------> SizeSF(200M)->FileSystemSF for dir3
Welcome to the community!
Dan.
Re: New "segmented" StorageFactory Development
Posted by Kathey Marsden <km...@sbcglobal.net>.
Rodrigo Madera wrote:
> To all of the members here, I want to wish you good luck and success
> on the Derby project.
>
> For external reasons, I will not participate in the Derby development
> process.
>
Thanks for the update Rodrigo. Sometimes I find myself saying, "I
wonder what happened to ..."
I hope you will still be able to participate in the user community,
which means you can still impact what features are implemented in Derby.
- You can make sure issues are filed for the features you want with
clear requirements and pointers to the list discussions.
- You can vote for the features you want (if someone else filed the
issue)
- You can write tests or offer to tests the features as they come
available.
- You can try out snapshots as they become available and offer feedback.
- You can answer questions, and share your Derby experience on the
list and Wiki.
All of these things will make Derby stronger and might even make
developers more likely to jump on your features #:) So, I do hope you
will be able to stay with the user community and we see you from time to
time on derby-user.
Kathey
Re: New "segmented" StorageFactory Development
Posted by Rodrigo Madera <ro...@gmail.com>.
To all of the members here, I want to wish you good luck and success
on the Derby project.
For external reasons, I will not participate in the Derby development process.
My ideas and those of others here should be implemented since I
believe it's a major win for the project itself, however I cannot
participate, even though it's what I really want.
Thanks to all and Good Luck!
Rodrigo Madera
On 5/5/06, Mike Matrigali <mi...@sbcglobal.net> wrote:
>
>
> Rodrigo Madera wrote:
> > On 5/5/06, Mike Matrigali <mi...@sbcglobal.net> wrote:
> >
> >> Do you have any more details on your requirements, such as the following:
> >> 1) do you need for a single table and/or index to be spread across
> >> multiple disk?
> >
> >
> > It would be terrific and the absolute glory of the requirement,
> > however, it depends.
> >
> > Is Derby based on a table/index-is-a-single-file architecture? If so,
> > it's too much trouble to change this. Making the tables/indexes
> > segmented would only be viable (in my opinion) if Derby already
> > supports this.
>
> Yes the current Derby model is that a table/index is a single file
> architecture. This is known by the code implmenting the StorageFactory.
> Above that, for instance in the raw storage engine it just needs to
> open the abstract Storage file and be able to read/write pages - it is
> up to the StorageFactory to do the right mapping. Of course in the
> current implementation the mapping is very straight forward :-), ie.
> 1-1 mapping of the file.
>
> >
> > I vote to get the "divider" in place that routes the new tables/etc to
> > the different directories, and only then, when it's mature, begin a
> > table segmentation engine.
>
> I think this is a good first choice. Seems a natural 1st step.
>
> ... cut ...
> >
> >
> >
> > This is only my initial vision of the model, so please give your
> > opinions here to make it better.
>
> I continue to support the model where the storage engine provides the
> tools, and the caller uses them to. Along these lines I prefer the
> implementation that finishes the work of the storage engine allowing
> seg0, seg1, ... directories. It is up to the execution engine to
> decide at create table or create index time which of these to use.
> There would probably be the need for some new catalog that allows for
> defining segments, naming them, ...
>
> If you want to go that route, I would definitely be interested in fixing
> the current code to use the existing segment interface properly.
>
> On the other hand, implementing a separate StorageFactory may be easier
> for you to implement as it is a very small module of code located in
> one place. And if you can get your hands on the in memory
> implmentation, it should show what kind of changes are necessary to
> provide alternate implementations. You would have a plugable module
> which would make it easy to compare/contrast performance.
>
> Moving forward I believe doing the seg1 approach clears the way for
> other work in the execution engine in the future, as described above.
>
>
> >
> > Thanks,
> > Rodrigo Madera
> >
> >
>
>
Re: New "segmented" StorageFactory Development
Posted by Mike Matrigali <mi...@sbcglobal.net>.
Rodrigo Madera wrote:
> On 5/5/06, Mike Matrigali <mi...@sbcglobal.net> wrote:
>
>> Do you have any more details on your requirements, such as the following:
>> 1) do you need for a single table and/or index to be spread across
>> multiple disk?
>
>
> It would be terrific and the absolute glory of the requirement,
> however, it depends.
>
> Is Derby based on a table/index-is-a-single-file architecture? If so,
> it's too much trouble to change this. Making the tables/indexes
> segmented would only be viable (in my opinion) if Derby already
> supports this.
Yes the current Derby model is that a table/index is a single file
architecture. This is known by the code implmenting the StorageFactory.
Above that, for instance in the raw storage engine it just needs to
open the abstract Storage file and be able to read/write pages - it is
up to the StorageFactory to do the right mapping. Of course in the
current implementation the mapping is very straight forward :-), ie.
1-1 mapping of the file.
>
> I vote to get the "divider" in place that routes the new tables/etc to
> the different directories, and only then, when it's mature, begin a
> table segmentation engine.
I think this is a good first choice. Seems a natural 1st step.
... cut ...
>
>
>
> This is only my initial vision of the model, so please give your
> opinions here to make it better.
I continue to support the model where the storage engine provides the
tools, and the caller uses them to. Along these lines I prefer the
implementation that finishes the work of the storage engine allowing
seg0, seg1, ... directories. It is up to the execution engine to
decide at create table or create index time which of these to use.
There would probably be the need for some new catalog that allows for
defining segments, naming them, ...
If you want to go that route, I would definitely be interested in fixing
the current code to use the existing segment interface properly.
On the other hand, implementing a separate StorageFactory may be easier
for you to implement as it is a very small module of code located in
one place. And if you can get your hands on the in memory
implmentation, it should show what kind of changes are necessary to
provide alternate implementations. You would have a plugable module
which would make it easy to compare/contrast performance.
Moving forward I believe doing the seg1 approach clears the way for
other work in the execution engine in the future, as described above.
>
> Thanks,
> Rodrigo Madera
>
>
Re: New "segmented" StorageFactory Development
Posted by Mike Matrigali <mi...@sbcglobal.net>.
I agree. They are definitely two different areas in the
current implementation. I'll file a JIRA with the disk
quota, with some initial observations about the current system.
Note, anyone interested in working on it should the join
the discussion, at the present I don't plan on implementing.
Rodrigo Madera wrote:
> Good idea!
>
>
> On 5/5/06, Bryan Pendleton <bp...@amberpoint.com> wrote:
>
>> It seems to me that there may be two different features
>> being discussed here:
>>
>> 1) Allowing more than one segment for data storage.
>> 2) Establishing a disk "quota" on a segment, so that Derby
>> will limit its space usage to no more than its quota.
>>
>> It strikes me that it might be useful to implement these
>> features independently, as each seems like it could be
>> potentially useful without the other, and then both could
>> be profitably used together.
>>
>> thanks,
>>
>> bryan
>>
>>
>
>
Re: New "segmented" StorageFactory Development
Posted by Rodrigo Madera <ro...@gmail.com>.
Good idea!
On 5/5/06, Bryan Pendleton <bp...@amberpoint.com> wrote:
> It seems to me that there may be two different features
> being discussed here:
>
> 1) Allowing more than one segment for data storage.
> 2) Establishing a disk "quota" on a segment, so that Derby
> will limit its space usage to no more than its quota.
>
> It strikes me that it might be useful to implement these
> features independently, as each seems like it could be
> potentially useful without the other, and then both could
> be profitably used together.
>
> thanks,
>
> bryan
>
>
Re: New "segmented" StorageFactory Development
Posted by Bryan Pendleton <bp...@amberpoint.com>.
It seems to me that there may be two different features
being discussed here:
1) Allowing more than one segment for data storage.
2) Establishing a disk "quota" on a segment, so that Derby
will limit its space usage to no more than its quota.
It strikes me that it might be useful to implement these
features independently, as each seems like it could be
potentially useful without the other, and then both could
be profitably used together.
thanks,
bryan
Re: New "segmented" StorageFactory Development
Posted by Rodrigo Madera <ro...@gmail.com>.
Non taken =o)
I'm making sure everything is okay and then I'll proceed.
Thanks for you time,
Rodrigo
On 5/5/06, David Van Couvering <Da...@sun.com> wrote:
> Hi, Rodrigo. Hopefully you took no offense. It was a tease at the
> debate going on the derby-user alias. It's *great* to have you working
> on this, absolutely, I'm quite excited, this is something people
> regularly ask for.
>
> David
>
> Rodrigo Madera wrote:
> > Well, please, I'm not related to the Derby project at all.
> >
> > I work at IBM Brazil, on a client services project for Medco. Nothing
> > to do with Derby.
> >
> > Rodrigo
> >
> > On 5/5/06, David Van Couvering <Da...@sun.com> wrote:
> >> Oh, wait, how could you be working for IBM and adding a feature? I
> >> thought you guys were only doing bugfixes :)
> >>
> >> David
> >>
> >> Rodrigo Madera wrote:
> >> > Oh, just a technical detail... I work for IBM, but on a whole
> >> > different project...
> >> >
> >> > Is this a problem??
> >> >
> >> > Thanks
> >> >
> >> > On 5/5/06, Rodrigo Madera <ro...@gmail.com> wrote:
> >> >> On 5/5/06, Mike Matrigali <mi...@sbcglobal.net> wrote:
> >> >> > Do you have any more details on your requirements, such as the
> >> >> following:
> >> >> > 1) do you need for a single table and/or index to be spread across
> >> >> > multiple disk?
> >> >>
> >> >> It would be terrific and the absolute glory of the requirement,
> >> >> however, it depends.
> >> >>
> >> >> Is Derby based on a table/index-is-a-single-file architecture? If so,
> >> >> it's too much trouble to change this. Making the tables/indexes
> >> >> segmented would only be viable (in my opinion) if Derby already
> >> >> supports this.
> >> >>
> >> >> I vote to get the "divider" in place that routes the new tables/etc to
> >> >> the different directories, and only then, when it's mature, begin a
> >> >> table segmentation engine.
> >> >>
> >> >> > 2) do you want control when you create each table/index where it
> >> >> > goes and how?
> >> >>
> >> >> Yes. I'm planing on doing this automagicaly based on the specified
> >> >> directory/capacity pairs.
> >> >>
> >> >> > 3) Are you looking to limit the absolute size of tables/indexes
> >> >> > in each directory to a fixed size?
> >> >>
> >> >> Absolutely. This is very important for the approach I'm thinking of in
> >> >> #1.
> >> >>
> >> >> > The existing storage system had some support for spreading data
> >> >> > across disks built into the interfaces, but was never used. Data
> >> >> > is currently stored in the seg0 directory. The idea was that
> >> >> > support could be added to store data also in a seg1 directory
> >> >> > located on another device. If one were interested in this approach
> >> >> > they would first have to fix the code to pass around correctly
> >> >> > the seg argument (It has been observed that some code got lazy and
> >> >> > just used 0 rather than proprogating the argument).
> >> >>
> >> >> I'm in. I'll co the latest version and check it out. Is it still
> >> there?
> >> >>
> >> >> > The next decision is how the tables are to spread across the disks.
> >> >> > If putting whole tables or indexes fits your plan then I would use
> >> >> > the existing table metadata catalogs to track where a file is (these
> >> >> > may have to be upgraded to hold the new info - not sure).
> >> >>
> >> >> IMO: This is the way to go for now.
> >> >>
> >> >> > If one
> >> >> > wants to spread a single file across multiple segments then you need
> >> >> > to decide if you want to do it by key or by some mathematical block
> >> >> > number approach:
> >> >> >
> >> >> > partition by key
> >> >> > o would pave the road for future interesting parallel query
> >> >> > execution work.
> >> >> > o would recommend again top down implementation, having the
> >> >> > existing database metadata catalogs do the work.
> >> >> >
> >> >> > partition by block number
> >> >> > o If there is any per table/index control again use the existing
> >> >> > database metadata catalogs and pass the info down into
> >> >> > store. partitioning by block number probably would best be
> >> done
> >> >> > with some new module as Dan suggested with alternate storage
> >> >> > factory implementations.
> >> >>
> >> >> Too messy for now... Guess #1 is better for now...
> >> >>
> >> >> > If you want per table/index control I think the segX approach is the
> >> >> > best, since the obvious input would be from the create table
> >> command.
> >> >>
> >> >> Ok. But I preffer to have the array of {path, capacity} tuples (or
> >> >> table, or meta info, or ...).
> >> >>
> >> >> > If you rather do the bottom up approach, I would first start at
> >> looking
> >> >> > at the in memory patch that was done. If you don't need much per
> >> >> > file control it may be possible to only override the
> >> StorageFactory as
> >> >> > Dan described.
> >> >>
> >> >> I'll take a look at it immediately.
> >> >>
> >> >> > Whatever approach you pick a couple of issues come to mind:
> >> >> > o how do you config the new segements into the db (currently just
> >> >> > automatically done a db creation time).
> >> >>
> >> >> Via the configuration tuples.
> >> >>
> >> >> > o how do back up a multiple segment database
> >> >>
> >> >> Transversing the repositories.
> >> >>
> >> >> > o how do handle allocation of disk space to files, current model
> >> >> > is the db just uses all the disk space available on that disk and
> >> >> > fails if table allocation gets and out of disk space.
> >> >>
> >> >> DB uses all ${capacity} on ${path}.
> >> >>
> >> >>
> >> >>
> >> >> This is only my initial vision of the model, so please give your
> >> >> opinions here to make it better.
> >> >>
> >> >> Thanks,
> >> >> Rodrigo Madera
> >> >>
> >>
>
Re: New "segmented" StorageFactory Development
Posted by David Van Couvering <Da...@Sun.COM>.
Hi, Rodrigo. Hopefully you took no offense. It was a tease at the
debate going on the derby-user alias. It's *great* to have you working
on this, absolutely, I'm quite excited, this is something people
regularly ask for.
David
Rodrigo Madera wrote:
> Well, please, I'm not related to the Derby project at all.
>
> I work at IBM Brazil, on a client services project for Medco. Nothing
> to do with Derby.
>
> Rodrigo
>
> On 5/5/06, David Van Couvering <Da...@sun.com> wrote:
>> Oh, wait, how could you be working for IBM and adding a feature? I
>> thought you guys were only doing bugfixes :)
>>
>> David
>>
>> Rodrigo Madera wrote:
>> > Oh, just a technical detail... I work for IBM, but on a whole
>> > different project...
>> >
>> > Is this a problem??
>> >
>> > Thanks
>> >
>> > On 5/5/06, Rodrigo Madera <ro...@gmail.com> wrote:
>> >> On 5/5/06, Mike Matrigali <mi...@sbcglobal.net> wrote:
>> >> > Do you have any more details on your requirements, such as the
>> >> following:
>> >> > 1) do you need for a single table and/or index to be spread across
>> >> > multiple disk?
>> >>
>> >> It would be terrific and the absolute glory of the requirement,
>> >> however, it depends.
>> >>
>> >> Is Derby based on a table/index-is-a-single-file architecture? If so,
>> >> it's too much trouble to change this. Making the tables/indexes
>> >> segmented would only be viable (in my opinion) if Derby already
>> >> supports this.
>> >>
>> >> I vote to get the "divider" in place that routes the new tables/etc to
>> >> the different directories, and only then, when it's mature, begin a
>> >> table segmentation engine.
>> >>
>> >> > 2) do you want control when you create each table/index where it
>> >> > goes and how?
>> >>
>> >> Yes. I'm planing on doing this automagicaly based on the specified
>> >> directory/capacity pairs.
>> >>
>> >> > 3) Are you looking to limit the absolute size of tables/indexes
>> >> > in each directory to a fixed size?
>> >>
>> >> Absolutely. This is very important for the approach I'm thinking of in
>> >> #1.
>> >>
>> >> > The existing storage system had some support for spreading data
>> >> > across disks built into the interfaces, but was never used. Data
>> >> > is currently stored in the seg0 directory. The idea was that
>> >> > support could be added to store data also in a seg1 directory
>> >> > located on another device. If one were interested in this approach
>> >> > they would first have to fix the code to pass around correctly
>> >> > the seg argument (It has been observed that some code got lazy and
>> >> > just used 0 rather than proprogating the argument).
>> >>
>> >> I'm in. I'll co the latest version and check it out. Is it still
>> there?
>> >>
>> >> > The next decision is how the tables are to spread across the disks.
>> >> > If putting whole tables or indexes fits your plan then I would use
>> >> > the existing table metadata catalogs to track where a file is (these
>> >> > may have to be upgraded to hold the new info - not sure).
>> >>
>> >> IMO: This is the way to go for now.
>> >>
>> >> > If one
>> >> > wants to spread a single file across multiple segments then you need
>> >> > to decide if you want to do it by key or by some mathematical block
>> >> > number approach:
>> >> >
>> >> > partition by key
>> >> > o would pave the road for future interesting parallel query
>> >> > execution work.
>> >> > o would recommend again top down implementation, having the
>> >> > existing database metadata catalogs do the work.
>> >> >
>> >> > partition by block number
>> >> > o If there is any per table/index control again use the existing
>> >> > database metadata catalogs and pass the info down into
>> >> > store. partitioning by block number probably would best be
>> done
>> >> > with some new module as Dan suggested with alternate storage
>> >> > factory implementations.
>> >>
>> >> Too messy for now... Guess #1 is better for now...
>> >>
>> >> > If you want per table/index control I think the segX approach is the
>> >> > best, since the obvious input would be from the create table
>> command.
>> >>
>> >> Ok. But I preffer to have the array of {path, capacity} tuples (or
>> >> table, or meta info, or ...).
>> >>
>> >> > If you rather do the bottom up approach, I would first start at
>> looking
>> >> > at the in memory patch that was done. If you don't need much per
>> >> > file control it may be possible to only override the
>> StorageFactory as
>> >> > Dan described.
>> >>
>> >> I'll take a look at it immediately.
>> >>
>> >> > Whatever approach you pick a couple of issues come to mind:
>> >> > o how do you config the new segements into the db (currently just
>> >> > automatically done a db creation time).
>> >>
>> >> Via the configuration tuples.
>> >>
>> >> > o how do back up a multiple segment database
>> >>
>> >> Transversing the repositories.
>> >>
>> >> > o how do handle allocation of disk space to files, current model
>> >> > is the db just uses all the disk space available on that disk and
>> >> > fails if table allocation gets and out of disk space.
>> >>
>> >> DB uses all ${capacity} on ${path}.
>> >>
>> >>
>> >>
>> >> This is only my initial vision of the model, so please give your
>> >> opinions here to make it better.
>> >>
>> >> Thanks,
>> >> Rodrigo Madera
>> >>
>>
Re: New "segmented" StorageFactory Development
Posted by Rodrigo Madera <ro...@gmail.com>.
Well, please, I'm not related to the Derby project at all.
I work at IBM Brazil, on a client services project for Medco. Nothing
to do with Derby.
Rodrigo
On 5/5/06, David Van Couvering <Da...@sun.com> wrote:
> Oh, wait, how could you be working for IBM and adding a feature? I
> thought you guys were only doing bugfixes :)
>
> David
>
> Rodrigo Madera wrote:
> > Oh, just a technical detail... I work for IBM, but on a whole
> > different project...
> >
> > Is this a problem??
> >
> > Thanks
> >
> > On 5/5/06, Rodrigo Madera <ro...@gmail.com> wrote:
> >> On 5/5/06, Mike Matrigali <mi...@sbcglobal.net> wrote:
> >> > Do you have any more details on your requirements, such as the
> >> following:
> >> > 1) do you need for a single table and/or index to be spread across
> >> > multiple disk?
> >>
> >> It would be terrific and the absolute glory of the requirement,
> >> however, it depends.
> >>
> >> Is Derby based on a table/index-is-a-single-file architecture? If so,
> >> it's too much trouble to change this. Making the tables/indexes
> >> segmented would only be viable (in my opinion) if Derby already
> >> supports this.
> >>
> >> I vote to get the "divider" in place that routes the new tables/etc to
> >> the different directories, and only then, when it's mature, begin a
> >> table segmentation engine.
> >>
> >> > 2) do you want control when you create each table/index where it
> >> > goes and how?
> >>
> >> Yes. I'm planing on doing this automagicaly based on the specified
> >> directory/capacity pairs.
> >>
> >> > 3) Are you looking to limit the absolute size of tables/indexes
> >> > in each directory to a fixed size?
> >>
> >> Absolutely. This is very important for the approach I'm thinking of in
> >> #1.
> >>
> >> > The existing storage system had some support for spreading data
> >> > across disks built into the interfaces, but was never used. Data
> >> > is currently stored in the seg0 directory. The idea was that
> >> > support could be added to store data also in a seg1 directory
> >> > located on another device. If one were interested in this approach
> >> > they would first have to fix the code to pass around correctly
> >> > the seg argument (It has been observed that some code got lazy and
> >> > just used 0 rather than proprogating the argument).
> >>
> >> I'm in. I'll co the latest version and check it out. Is it still there?
> >>
> >> > The next decision is how the tables are to spread across the disks.
> >> > If putting whole tables or indexes fits your plan then I would use
> >> > the existing table metadata catalogs to track where a file is (these
> >> > may have to be upgraded to hold the new info - not sure).
> >>
> >> IMO: This is the way to go for now.
> >>
> >> > If one
> >> > wants to spread a single file across multiple segments then you need
> >> > to decide if you want to do it by key or by some mathematical block
> >> > number approach:
> >> >
> >> > partition by key
> >> > o would pave the road for future interesting parallel query
> >> > execution work.
> >> > o would recommend again top down implementation, having the
> >> > existing database metadata catalogs do the work.
> >> >
> >> > partition by block number
> >> > o If there is any per table/index control again use the existing
> >> > database metadata catalogs and pass the info down into
> >> > store. partitioning by block number probably would best be done
> >> > with some new module as Dan suggested with alternate storage
> >> > factory implementations.
> >>
> >> Too messy for now... Guess #1 is better for now...
> >>
> >> > If you want per table/index control I think the segX approach is the
> >> > best, since the obvious input would be from the create table command.
> >>
> >> Ok. But I preffer to have the array of {path, capacity} tuples (or
> >> table, or meta info, or ...).
> >>
> >> > If you rather do the bottom up approach, I would first start at looking
> >> > at the in memory patch that was done. If you don't need much per
> >> > file control it may be possible to only override the StorageFactory as
> >> > Dan described.
> >>
> >> I'll take a look at it immediately.
> >>
> >> > Whatever approach you pick a couple of issues come to mind:
> >> > o how do you config the new segements into the db (currently just
> >> > automatically done a db creation time).
> >>
> >> Via the configuration tuples.
> >>
> >> > o how do back up a multiple segment database
> >>
> >> Transversing the repositories.
> >>
> >> > o how do handle allocation of disk space to files, current model
> >> > is the db just uses all the disk space available on that disk and
> >> > fails if table allocation gets and out of disk space.
> >>
> >> DB uses all ${capacity} on ${path}.
> >>
> >>
> >>
> >> This is only my initial vision of the model, so please give your
> >> opinions here to make it better.
> >>
> >> Thanks,
> >> Rodrigo Madera
> >>
>
Re: New "segmented" StorageFactory Development
Posted by David Van Couvering <Da...@Sun.COM>.
Oh, wait, how could you be working for IBM and adding a feature? I
thought you guys were only doing bugfixes :)
David
Rodrigo Madera wrote:
> Oh, just a technical detail... I work for IBM, but on a whole
> different project...
>
> Is this a problem??
>
> Thanks
>
> On 5/5/06, Rodrigo Madera <ro...@gmail.com> wrote:
>> On 5/5/06, Mike Matrigali <mi...@sbcglobal.net> wrote:
>> > Do you have any more details on your requirements, such as the
>> following:
>> > 1) do you need for a single table and/or index to be spread across
>> > multiple disk?
>>
>> It would be terrific and the absolute glory of the requirement,
>> however, it depends.
>>
>> Is Derby based on a table/index-is-a-single-file architecture? If so,
>> it's too much trouble to change this. Making the tables/indexes
>> segmented would only be viable (in my opinion) if Derby already
>> supports this.
>>
>> I vote to get the "divider" in place that routes the new tables/etc to
>> the different directories, and only then, when it's mature, begin a
>> table segmentation engine.
>>
>> > 2) do you want control when you create each table/index where it
>> > goes and how?
>>
>> Yes. I'm planing on doing this automagicaly based on the specified
>> directory/capacity pairs.
>>
>> > 3) Are you looking to limit the absolute size of tables/indexes
>> > in each directory to a fixed size?
>>
>> Absolutely. This is very important for the approach I'm thinking of in
>> #1.
>>
>> > The existing storage system had some support for spreading data
>> > across disks built into the interfaces, but was never used. Data
>> > is currently stored in the seg0 directory. The idea was that
>> > support could be added to store data also in a seg1 directory
>> > located on another device. If one were interested in this approach
>> > they would first have to fix the code to pass around correctly
>> > the seg argument (It has been observed that some code got lazy and
>> > just used 0 rather than proprogating the argument).
>>
>> I'm in. I'll co the latest version and check it out. Is it still there?
>>
>> > The next decision is how the tables are to spread across the disks.
>> > If putting whole tables or indexes fits your plan then I would use
>> > the existing table metadata catalogs to track where a file is (these
>> > may have to be upgraded to hold the new info - not sure).
>>
>> IMO: This is the way to go for now.
>>
>> > If one
>> > wants to spread a single file across multiple segments then you need
>> > to decide if you want to do it by key or by some mathematical block
>> > number approach:
>> >
>> > partition by key
>> > o would pave the road for future interesting parallel query
>> > execution work.
>> > o would recommend again top down implementation, having the
>> > existing database metadata catalogs do the work.
>> >
>> > partition by block number
>> > o If there is any per table/index control again use the existing
>> > database metadata catalogs and pass the info down into
>> > store. partitioning by block number probably would best be done
>> > with some new module as Dan suggested with alternate storage
>> > factory implementations.
>>
>> Too messy for now... Guess #1 is better for now...
>>
>> > If you want per table/index control I think the segX approach is the
>> > best, since the obvious input would be from the create table command.
>>
>> Ok. But I preffer to have the array of {path, capacity} tuples (or
>> table, or meta info, or ...).
>>
>> > If you rather do the bottom up approach, I would first start at looking
>> > at the in memory patch that was done. If you don't need much per
>> > file control it may be possible to only override the StorageFactory as
>> > Dan described.
>>
>> I'll take a look at it immediately.
>>
>> > Whatever approach you pick a couple of issues come to mind:
>> > o how do you config the new segements into the db (currently just
>> > automatically done a db creation time).
>>
>> Via the configuration tuples.
>>
>> > o how do back up a multiple segment database
>>
>> Transversing the repositories.
>>
>> > o how do handle allocation of disk space to files, current model
>> > is the db just uses all the disk space available on that disk and
>> > fails if table allocation gets and out of disk space.
>>
>> DB uses all ${capacity} on ${path}.
>>
>>
>>
>> This is only my initial vision of the model, so please give your
>> opinions here to make it better.
>>
>> Thanks,
>> Rodrigo Madera
>>
Re: New "segmented" StorageFactory Development
Posted by Rodrigo Madera <ro...@gmail.com>.
Oh, just a technical detail... I work for IBM, but on a whole
different project...
Is this a problem??
Thanks
On 5/5/06, Rodrigo Madera <ro...@gmail.com> wrote:
> On 5/5/06, Mike Matrigali <mi...@sbcglobal.net> wrote:
> > Do you have any more details on your requirements, such as the following:
> > 1) do you need for a single table and/or index to be spread across
> > multiple disk?
>
> It would be terrific and the absolute glory of the requirement,
> however, it depends.
>
> Is Derby based on a table/index-is-a-single-file architecture? If so,
> it's too much trouble to change this. Making the tables/indexes
> segmented would only be viable (in my opinion) if Derby already
> supports this.
>
> I vote to get the "divider" in place that routes the new tables/etc to
> the different directories, and only then, when it's mature, begin a
> table segmentation engine.
>
> > 2) do you want control when you create each table/index where it
> > goes and how?
>
> Yes. I'm planing on doing this automagicaly based on the specified
> directory/capacity pairs.
>
> > 3) Are you looking to limit the absolute size of tables/indexes
> > in each directory to a fixed size?
>
> Absolutely. This is very important for the approach I'm thinking of in #1.
>
> > The existing storage system had some support for spreading data
> > across disks built into the interfaces, but was never used. Data
> > is currently stored in the seg0 directory. The idea was that
> > support could be added to store data also in a seg1 directory
> > located on another device. If one were interested in this approach
> > they would first have to fix the code to pass around correctly
> > the seg argument (It has been observed that some code got lazy and
> > just used 0 rather than proprogating the argument).
>
> I'm in. I'll co the latest version and check it out. Is it still there?
>
> > The next decision is how the tables are to spread across the disks.
> > If putting whole tables or indexes fits your plan then I would use
> > the existing table metadata catalogs to track where a file is (these
> > may have to be upgraded to hold the new info - not sure).
>
> IMO: This is the way to go for now.
>
> > If one
> > wants to spread a single file across multiple segments then you need
> > to decide if you want to do it by key or by some mathematical block
> > number approach:
> >
> > partition by key
> > o would pave the road for future interesting parallel query
> > execution work.
> > o would recommend again top down implementation, having the
> > existing database metadata catalogs do the work.
> >
> > partition by block number
> > o If there is any per table/index control again use the existing
> > database metadata catalogs and pass the info down into
> > store. partitioning by block number probably would best be done
> > with some new module as Dan suggested with alternate storage
> > factory implementations.
>
> Too messy for now... Guess #1 is better for now...
>
> > If you want per table/index control I think the segX approach is the
> > best, since the obvious input would be from the create table command.
>
> Ok. But I preffer to have the array of {path, capacity} tuples (or
> table, or meta info, or ...).
>
> > If you rather do the bottom up approach, I would first start at looking
> > at the in memory patch that was done. If you don't need much per
> > file control it may be possible to only override the StorageFactory as
> > Dan described.
>
> I'll take a look at it immediately.
>
> > Whatever approach you pick a couple of issues come to mind:
> > o how do you config the new segements into the db (currently just
> > automatically done a db creation time).
>
> Via the configuration tuples.
>
> > o how do back up a multiple segment database
>
> Transversing the repositories.
>
> > o how do handle allocation of disk space to files, current model
> > is the db just uses all the disk space available on that disk and
> > fails if table allocation gets and out of disk space.
>
> DB uses all ${capacity} on ${path}.
>
>
>
> This is only my initial vision of the model, so please give your
> opinions here to make it better.
>
> Thanks,
> Rodrigo Madera
>
Re: New "segmented" StorageFactory Development
Posted by Rodrigo Madera <ro...@gmail.com>.
On 5/5/06, Mike Matrigali <mi...@sbcglobal.net> wrote:
> Do you have any more details on your requirements, such as the following:
> 1) do you need for a single table and/or index to be spread across
> multiple disk?
It would be terrific and the absolute glory of the requirement,
however, it depends.
Is Derby based on a table/index-is-a-single-file architecture? If so,
it's too much trouble to change this. Making the tables/indexes
segmented would only be viable (in my opinion) if Derby already
supports this.
I vote to get the "divider" in place that routes the new tables/etc to
the different directories, and only then, when it's mature, begin a
table segmentation engine.
> 2) do you want control when you create each table/index where it
> goes and how?
Yes. I'm planing on doing this automagicaly based on the specified
directory/capacity pairs.
> 3) Are you looking to limit the absolute size of tables/indexes
> in each directory to a fixed size?
Absolutely. This is very important for the approach I'm thinking of in #1.
> The existing storage system had some support for spreading data
> across disks built into the interfaces, but was never used. Data
> is currently stored in the seg0 directory. The idea was that
> support could be added to store data also in a seg1 directory
> located on another device. If one were interested in this approach
> they would first have to fix the code to pass around correctly
> the seg argument (It has been observed that some code got lazy and
> just used 0 rather than proprogating the argument).
I'm in. I'll co the latest version and check it out. Is it still there?
> The next decision is how the tables are to spread across the disks.
> If putting whole tables or indexes fits your plan then I would use
> the existing table metadata catalogs to track where a file is (these
> may have to be upgraded to hold the new info - not sure).
IMO: This is the way to go for now.
> If one
> wants to spread a single file across multiple segments then you need
> to decide if you want to do it by key or by some mathematical block
> number approach:
>
> partition by key
> o would pave the road for future interesting parallel query
> execution work.
> o would recommend again top down implementation, having the
> existing database metadata catalogs do the work.
>
> partition by block number
> o If there is any per table/index control again use the existing
> database metadata catalogs and pass the info down into
> store. partitioning by block number probably would best be done
> with some new module as Dan suggested with alternate storage
> factory implementations.
Too messy for now... Guess #1 is better for now...
> If you want per table/index control I think the segX approach is the
> best, since the obvious input would be from the create table command.
Ok. But I preffer to have the array of {path, capacity} tuples (or
table, or meta info, or ...).
> If you rather do the bottom up approach, I would first start at looking
> at the in memory patch that was done. If you don't need much per
> file control it may be possible to only override the StorageFactory as
> Dan described.
I'll take a look at it immediately.
> Whatever approach you pick a couple of issues come to mind:
> o how do you config the new segements into the db (currently just
> automatically done a db creation time).
Via the configuration tuples.
> o how do back up a multiple segment database
Transversing the repositories.
> o how do handle allocation of disk space to files, current model
> is the db just uses all the disk space available on that disk and
> fails if table allocation gets and out of disk space.
DB uses all ${capacity} on ${path}.
This is only my initial vision of the model, so please give your
opinions here to make it better.
Thanks,
Rodrigo Madera
Re: New "segmented" StorageFactory Development
Posted by Mike Matrigali <mi...@sbcglobal.net>.
Do you have any more details on your requirements, such as the following:
1) do you need for a single table and/or index to be spread across
multiple disk?
2) do you want control when you create each table/index where it
goes and how?
3) Are you looking to limit the absolute size of tables/indexes
in each directory to a fixed size?
The existing storage system had some support for spreading data
across disks built into the interfaces, but was never used. Data
is currently stored in the seg0 directory. The idea was that
support could be added to store data also in a seg1 directory
located on another device. If one were interested in this approach
they would first have to fix the code to pass around correctly
the seg argument (It has been observed that some code got lazy and
just used 0 rather than proprogating the argument).
The next decision is how the tables are to spread across the disks.
If putting whole tables or indexes fits your plan then I would use
the existing table metadata catalogs to track where a file is (these
may have to be upgraded to hold the new info - not sure). If one
wants to spread a single file across multiple segments then you need
to decide if you want to do it by key or by some mathematical block
number approach:
partition by key
o would pave the road for future interesting parallel query
execution work.
o would recommend again top down implementation, having the
existing database metadata catalogs do the work.
partition by block number
o If there is any per table/index control again use the existing
database metadata catalogs and pass the info down into
store. partitioning by block number probably would best be done
with some new module as Dan suggested with alternate storage
factory implementations.
If you want per table/index control I think the segX approach is the
best, since the obvious input would be from the create table command.
If you rather do the bottom up approach, I would first start at looking
at the in memory patch that was done. If you don't need much per
file control it may be possible to only override the StorageFactory as
Dan described.
Whatever approach you pick a couple of issues come to mind:
o how do you config the new segements into the db (currently just
automatically done a db creation time).
o how do back up a multiple segment database
o how do handle allocation of disk space to files, current model
is the db just uses all the disk space available on that disk and
fails if table allocation gets and out of disk space.
Rodrigo Madera wrote:
> Hello to all,
>
> I'm Rodrigo Madera, software developer. I need a certain feature in
> the Derby database system, and I am volunteering to develop such a
> feature.
>
> My requirement is that a database can be spread into different
> directories (please read the Derby Users mailing list thread:
> "Spawning Data on Multiple Directories").
>
> I am filing a JIRA feature request and I count on the community for
> pointers and information on developing such a feature.
>
> I am still seeking the knowledge and planning the necessary steps to
> do such a thing. I don't have *lots* of time, but since this is a
> requirement for me, I can spend some hours on it.
>
> Thanks to all for any input/critics/suggestions,
>
> Rodrigo Madera
>
>