Adding support for MonetDB

[Tobias Willig <tw...@googlemail.com>]

Hi everyone,

I like to add support for MonetDB in SDB and I have two questions concerning
this project:

1. How much effort it takes to add support for a new database type?
2. Are there predefined extension points that allow adding a new database
type easily?
    If so could you give me the name of some classes and config files, which
are important to accomplish that task?

Thanks in advance!

Best Regards
Tobias Willig

Re: Adding support for MonetDB

[natlu 2809 <na...@gmail.com>]

Will do, yes. Once i get some time....day job getting in the way of course !

sent from my htc, forgive typos please !
On Sep 9, 2011 4:44 PM, "Paolo Castagna" <ca...@googlemail.com>
wrote:
> Hi,
> why don't you open a new JIRA issue (as a New Feature) for this?
> https://issues.apache.org/jira/browse/JENA
>
> You can then attach a patch to it. This way others can look at what you
have done so far (and maybe help you out).
>
> Thanks for your help,
> Paolo
>
> nat lu wrote:
>>
>> I made a start, and tried to use one of the existing flavours, but ended
>> up creating one for MonetDB - combination of derby and DB2. It doesnt
>> like longs or unbounded varchars.
>>
>> So, I got as far as getting SDBConfig to complete, but havent done an
>> sdbload yet
>>
>>
>> On 09/09/11 10:37, Andy Seaborne wrote:
>>>
>>>
>>> On 04/09/11 13:03, nat lu wrote:
>>>>
>>>> I'm going to give it a go sometime soon and report back on my
>>>> non-scientific findings. Your point about the small number of columns
is
>>>> well made, but the research paper cited earlier also mentions this and
>>>> reports that because of column store optimisations even when they
>>>> vertically partitioned their data rather than using a property-table
>>>> approach they still saw good improvement. However, again, I'm no column
>>>> store expert so perhaps I'm missing some point here :-). Anyway, time
to
>>>> "suck it and see@, all in the name of progress of course.
>>>>
>>>> On 03/09/11 16:29, David Jordan wrote:
>>>>> I have not used a column-oriented database, but I am somewhat familiar
>>>>> with them. My understanding of them is that the storage is partitioned
>>>>> on a column basis, such that there is no physical clustering together
>>>>> of all the columns for a given row. An advantage of this would be in
>>>>> the case where you have tables with many columns, but the particular
>>>>> application only needs a small subset of columns.
>>>>>
>>>>> With the SDB representation of triples (3 columns) and quads (4
>>>>> columns), and access typically based on having a specific value for
>>>>> one or two of the columns, I am not so sure that a column-based
>>>>> approach would offer any advantage.
>>>>>
>>>>> But again, I am no expert on these types of databases.
>>>>>
>>>>> These discussions about alternative datastore representations RDF/OWL
>>>>> data are very useful, to gain better understanding of which data
>>>>> architectures yield the best implementation approach for
>>>>> high-performance.
>>>>>
>>>>> p.s. I Monet provides support for JDBC, I would not think much effort
>>>>> is needed to support in with SDB.
>>>
>>> Shouldn't be too hard :-) SDB targets SQL-92 and there are a few
>>> extension points to cope with the vagaries of different SQL engines.
>>> It's one of the reasons there are ~10 small files to write, to capture
>>> the uniqueness of each SQL syntax.
>>>
>>> Andy
>>

Re: Adding support for MonetDB

[nat lu <na...@gmail.com>]

u sound like my dad :-)

I'm sure I will end up doing it myself, I agree. But it will be in time. 
Meanwhile, lets change the vocabulary a bit shall, and stop calling this 
a patch. Perhaps "kite" may be a better way to communicate its intent at 
this stage. And I don't have anything against unit tests (mostly), I use 
them and write them every day, its just that life and family take 
precedence sometimes, don't they ?



On 11/10/11 20:28, Paolo Castagna wrote:
> nat lu wrote:
>> Thanks, I hope the community can help progress this more, and that this
>> is just a very small start. I'm sure I'm not the only one interested in
>> the possibilities of column stores as RDF repositories.
> Some Italians used to say: "If you want something done, do it yourself".
>
> That's true in the open source world: you are part of the community and
> interested in MonetDB, so you are in a good position to get it done. ;-)
> Others might help you, but don't rely on it.
>
> If you want to help someone else help you, a good way is to list in a
> comment on the JIRA issue what are the things left to do and, if you have
> an idea on how to do those things, add a short paragraph describing how
> it should be done.
>
>> "I'll be back...." :-)
> I'll wait.
>
> Paolo
>
>> On 11/10/11 12:15, Paolo Castagna wrote:
>>> Hi nat lu
>>>
>>> nat lu wrote:
>>>> HI,
>>>>
>>>> Had a quick look - the patch seems to want to delete (in entirety) a
>>>> couple of existing SDB classes which I had made minimal modifications
>>>> to, so I've re-checked out SDB 1.3.4 and created a new patch which I can
>>>> upload later. So - the first attachment in Jira shouldnt be used.
>>> Great. Thanks for double checking and acting on this.
>>> This will make life easier to whoever review your patch.
>>>
>>>> Secondly, I haven't recreated all the unit test code that exists for
>>>> other RDBMS supported by SDB because, simply by the looks of it, there's
>>>> quite a bit of it, and I haven't had time so far.
>>> Ack.
>>>
>>> A patch without tests is better than no patch at all.
>>>
>>> However, tests are really important and should be considered part of the
>>> process of submitting a patch.
>>>
>>> They certainly help us a lot and can make a patch going into trunk
>>> faster.
>>>
>>>> I've also, as I said before, not used Monet with any data yet, just got
>>>> it to the point that SDBConfig doesn't fall over, and the Jira issue
>>>> raised is simply to record the desire to support it (and perhaps other
>>>> column stores if conceptually they prove to have merit when used as an
>>>> RDF repo).
>>> Ack.
>>>
>>> Others wanting to have MonetDB will see your issue and might come and
>>> help you with that.
>>>
>>> I personally don't use SDB much and I've never used MonetDB before.
>>>
>>>> So, personally, I'd prefer (I assume you-all as well) that nothing
>>>> happens too quickly with this, until I or someone else gets some time to
>>>> do some work and produce the unit tests.
>>> Sure.
>>>
>>> We tend not to commit untested code. :-)
>>>
>>> Having low quality or unfinished code committed to trunk is a bad
>>> practice
>>> and we avoid that.
>>>
>>> Rules have exceptions though, for example, I recently committed an
>>> RDF/JSON
>>> writer which isn't complete/fully tested, however it's not going to
>>> affect
>>> users in any way (see JENA-135). I did that to make Rob's life easier
>>> in case
>>> he want to have an RDF/JSON writer as well as a parser (which he
>>> contributed).
>>>
>>> Nat, don't take these last two my messages too much personally, I am just
>>> taking the opportunities to send across a few IMHO important messages:
>>>
>>>    - we welcome patches and new ideas/features (we really do!)
>>>    - proper patches, well tested and which apply cleanly make our life
>>>      so much easier (so we appreciate any effort in this direction).
>>>      It's very difficult to get it wrong with svn diff>
>>> JENA-XYZ.patch. :-)
>>>
>>> Thanks again and have fun with SDB + MonetDB!
>>>
>>> Paolo
>>>
>>>> On 11/10/11 08:07, Paolo Castagna wrote:
>>>>> Hi,
>>>>> first of all, thank you for your patch.
>>>>> I had a quick look, but I did not try to apply it (yet).
>>>>>
>>>>> May I ask how you created your patch?
>>>>>
>>>>> We added a section on the Getting Involved page on the Jena website:
>>>>>
>>>>>     "Patches should be attached to issues in Jira (click on
>>>>>      More Actions>    Attach Files). To create a patch you can simply
>>>>>      use the command:
>>>>>
>>>>>        svn diff>    JENA-XYZ.patch
>>>>>
>>>>>      Please, inspect your patch and make sure it includes all (and only)
>>>>>      the relevant changes for a single issue. Don't forget tests! If you
>>>>>      want to test if a patch applies cleanly you can use:
>>>>>
>>>>>        patch -p0<    JENA-XYZ.patch
>>>>>
>>>>>      If you use Eclipse: right click on the project name in Package
>>>>> Explorer,
>>>>>      select Team>    Create Patch or Team>    Apply Patch."
>>>>>
>>>>> -
>>>>> http://jena.staging.apache.org/jena/getting_involved/#submit_your_patches
>>>>>
>>>>>
>>>>>
>>>>> It really helps if a patch contains only the lines you added|removed
>>>>> and it applies cleanly. It saves a lot of time and speed-up reviewing
>>>>> it.
>>>>>
>>>>> Your patch may be perfectly fine, but I wanted to take the opportunity
>>>>> to send the message across.
>>>>>
>>>>> I am really curious to run a few benchmarks when it's done to compare
>>>>> MonetDB with a more traditional SQL system.
>>>>>
>>>>> By the way, about benchmarks Andy is (secretely) working on this:
>>>>> https://svn.apache.org/repos/asf/incubator/jena/Experimental/JenaPerf/trunk/
>>>>>
>>>>>
>>>>> I have not time to try it yet, but it seems very interesting. :-)
>>>>>
>>>>> Thank you again for the new interesting feature.
>>>>>
>>>>> Paolo
>>>>>
>>>>>
>>>>> nat lu wrote:
>>>>>> Added, with patch file, at
>>>>>>
>>>>>> https://issues.apache.org/jira/browse/JENA-134
>>>>>>
>>>>>>
>>>>>> I have made no more progress on testing it out other than sdbconfig so
>>>>>> far, hope too soon.
>>>>>>
>>>>>>
>>>>>> On 09/09/11 16:43, Paolo Castagna wrote:
>>>>>>> Hi,
>>>>>>> why don't you open a new JIRA issue (as a New Feature) for this?
>>>>>>> https://issues.apache.org/jira/browse/JENA
>>>>>>>
>>>>>>> You can then attach a patch to it. This way others can look at what
>>>>>>> you have done so far (and maybe help you out).
>>>>>>>
>>>>>>> Thanks for your help,
>>>>>>> Paolo
>>>>>>>
>>>>>>> nat lu wrote:
>>>>>>>> I made a start, and tried to use one of the existing flavours, but
>>>>>>>> ended
>>>>>>>> up creating one for MonetDB - combination of derby and DB2. It
>>>>>>>> doesnt
>>>>>>>> like longs or unbounded varchars.
>>>>>>>>
>>>>>>>> So, I got as far as getting SDBConfig to complete, but havent
>>>>>>>> done an
>>>>>>>> sdbload yet
>>>>>>>>
>>>>>>>>
>>>>>>>> On 09/09/11 10:37, Andy Seaborne wrote:
>>>>>>>>> On 04/09/11 13:03, nat lu wrote:
>>>>>>>>>> I'm going to give it a go sometime soon and report back on my
>>>>>>>>>> non-scientific findings. Your point about the small number of
>>>>>>>>>> columns is
>>>>>>>>>> well made, but the research paper cited earlier also mentions
>>>>>>>>>> this and
>>>>>>>>>> reports that because of column store optimisations even when they
>>>>>>>>>> vertically partitioned their data rather than using a
>>>>>>>>>> property-table
>>>>>>>>>> approach they still saw good improvement. However, again, I'm no
>>>>>>>>>> column
>>>>>>>>>> store expert so perhaps I'm missing some point here :-). Anyway,
>>>>>>>>>> time to
>>>>>>>>>> "suck it and see@, all in the name of progress of course.
>>>>>>>>>>
>>>>>>>>>> On 03/09/11 16:29, David Jordan wrote:
>>>>>>>>>>> I have not used a column-oriented database, but I am somewhat
>>>>>>>>>>> familiar
>>>>>>>>>>> with them. My understanding of them is that the storage is
>>>>>>>>>>> partitioned
>>>>>>>>>>> on a column basis, such that there is no physical clustering
>>>>>>>>>>> together
>>>>>>>>>>> of all the columns for a given row. An advantage of this would
>>>>>>>>>>> be in
>>>>>>>>>>> the case where you have tables with many columns, but the
>>>>>>>>>>> particular
>>>>>>>>>>> application only needs a small subset of columns.
>>>>>>>>>>>
>>>>>>>>>>> With the SDB representation of triples (3 columns) and quads (4
>>>>>>>>>>> columns), and access typically based on having a specific
>>>>>>>>>>> value for
>>>>>>>>>>> one or two of the columns, I am not so sure that a column-based
>>>>>>>>>>> approach would offer any advantage.
>>>>>>>>>>>
>>>>>>>>>>> But again, I am no expert on these types of databases.
>>>>>>>>>>>
>>>>>>>>>>> These discussions about alternative datastore representations
>>>>>>>>>>> RDF/OWL
>>>>>>>>>>> data are very useful, to gain better understanding of which data
>>>>>>>>>>> architectures yield the best implementation approach for
>>>>>>>>>>> high-performance.
>>>>>>>>>>>
>>>>>>>>>>> p.s. I Monet provides support for JDBC, I would not think much
>>>>>>>>>>> effort
>>>>>>>>>>> is needed to support in with SDB.
>>>>>>>>> Shouldn't be too hard :-)  SDB targets SQL-92 and there are a few
>>>>>>>>> extension points to cope with the vagaries of different SQL
>>>>>>>>> engines.
>>>>>>>>> It's one of the reasons there are ~10 small files to write, to
>>>>>>>>> capture
>>>>>>>>> the uniqueness of each SQL syntax.
>>>>>>>>>
>>>>>>>>>         Andy

Re: Adding support for MonetDB

[Paolo Castagna <ca...@googlemail.com>]

nat lu wrote:
> Thanks, I hope the community can help progress this more, and that this
> is just a very small start. I'm sure I'm not the only one interested in
> the possibilities of column stores as RDF repositories.

Some Italians used to say: "If you want something done, do it yourself".

That's true in the open source world: you are part of the community and
interested in MonetDB, so you are in a good position to get it done. ;-)
Others might help you, but don't rely on it.

If you want to help someone else help you, a good way is to list in a
comment on the JIRA issue what are the things left to do and, if you have
an idea on how to do those things, add a short paragraph describing how
it should be done.

> "I'll be back...." :-)

I'll wait.

Paolo

> 
> On 11/10/11 12:15, Paolo Castagna wrote:
>> Hi nat lu
>>
>> nat lu wrote:
>>> HI,
>>>
>>> Had a quick look - the patch seems to want to delete (in entirety) a
>>> couple of existing SDB classes which I had made minimal modifications
>>> to, so I've re-checked out SDB 1.3.4 and created a new patch which I can
>>> upload later. So - the first attachment in Jira shouldnt be used.
>> Great. Thanks for double checking and acting on this.
>> This will make life easier to whoever review your patch.
>>
>>> Secondly, I haven't recreated all the unit test code that exists for
>>> other RDBMS supported by SDB because, simply by the looks of it, there's
>>> quite a bit of it, and I haven't had time so far.
>> Ack.
>>
>> A patch without tests is better than no patch at all.
>>
>> However, tests are really important and should be considered part of the
>> process of submitting a patch.
>>
>> They certainly help us a lot and can make a patch going into trunk
>> faster.
>>
>>> I've also, as I said before, not used Monet with any data yet, just got
>>> it to the point that SDBConfig doesn't fall over, and the Jira issue
>>> raised is simply to record the desire to support it (and perhaps other
>>> column stores if conceptually they prove to have merit when used as an
>>> RDF repo).
>> Ack.
>>
>> Others wanting to have MonetDB will see your issue and might come and
>> help you with that.
>>
>> I personally don't use SDB much and I've never used MonetDB before.
>>
>>> So, personally, I'd prefer (I assume you-all as well) that nothing
>>> happens too quickly with this, until I or someone else gets some time to
>>> do some work and produce the unit tests.
>> Sure.
>>
>> We tend not to commit untested code. :-)
>>
>> Having low quality or unfinished code committed to trunk is a bad
>> practice
>> and we avoid that.
>>
>> Rules have exceptions though, for example, I recently committed an
>> RDF/JSON
>> writer which isn't complete/fully tested, however it's not going to
>> affect
>> users in any way (see JENA-135). I did that to make Rob's life easier
>> in case
>> he want to have an RDF/JSON writer as well as a parser (which he
>> contributed).
>>
>> Nat, don't take these last two my messages too much personally, I am just
>> taking the opportunities to send across a few IMHO important messages:
>>
>>   - we welcome patches and new ideas/features (we really do!)
>>   - proper patches, well tested and which apply cleanly make our life
>>     so much easier (so we appreciate any effort in this direction).
>>     It's very difficult to get it wrong with svn diff> 
>> JENA-XYZ.patch. :-)
>>
>> Thanks again and have fun with SDB + MonetDB!
>>
>> Paolo
>>
>>> On 11/10/11 08:07, Paolo Castagna wrote:
>>>> Hi,
>>>> first of all, thank you for your patch.
>>>> I had a quick look, but I did not try to apply it (yet).
>>>>
>>>> May I ask how you created your patch?
>>>>
>>>> We added a section on the Getting Involved page on the Jena website:
>>>>
>>>>    "Patches should be attached to issues in Jira (click on
>>>>     More Actions>   Attach Files). To create a patch you can simply
>>>>     use the command:
>>>>
>>>>       svn diff>   JENA-XYZ.patch
>>>>
>>>>     Please, inspect your patch and make sure it includes all (and only)
>>>>     the relevant changes for a single issue. Don't forget tests! If you
>>>>     want to test if a patch applies cleanly you can use:
>>>>
>>>>       patch -p0<   JENA-XYZ.patch
>>>>
>>>>     If you use Eclipse: right click on the project name in Package
>>>> Explorer,
>>>>     select Team>   Create Patch or Team>   Apply Patch."
>>>>
>>>> -
>>>> http://jena.staging.apache.org/jena/getting_involved/#submit_your_patches
>>>>
>>>>
>>>>
>>>> It really helps if a patch contains only the lines you added|removed
>>>> and it applies cleanly. It saves a lot of time and speed-up reviewing
>>>> it.
>>>>
>>>> Your patch may be perfectly fine, but I wanted to take the opportunity
>>>> to send the message across.
>>>>
>>>> I am really curious to run a few benchmarks when it's done to compare
>>>> MonetDB with a more traditional SQL system.
>>>>
>>>> By the way, about benchmarks Andy is (secretely) working on this:
>>>> https://svn.apache.org/repos/asf/incubator/jena/Experimental/JenaPerf/trunk/
>>>>
>>>>
>>>> I have not time to try it yet, but it seems very interesting. :-)
>>>>
>>>> Thank you again for the new interesting feature.
>>>>
>>>> Paolo
>>>>
>>>>
>>>> nat lu wrote:
>>>>> Added, with patch file, at
>>>>>
>>>>> https://issues.apache.org/jira/browse/JENA-134
>>>>>
>>>>>
>>>>> I have made no more progress on testing it out other than sdbconfig so
>>>>> far, hope too soon.
>>>>>
>>>>>
>>>>> On 09/09/11 16:43, Paolo Castagna wrote:
>>>>>> Hi,
>>>>>> why don't you open a new JIRA issue (as a New Feature) for this?
>>>>>> https://issues.apache.org/jira/browse/JENA
>>>>>>
>>>>>> You can then attach a patch to it. This way others can look at what
>>>>>> you have done so far (and maybe help you out).
>>>>>>
>>>>>> Thanks for your help,
>>>>>> Paolo
>>>>>>
>>>>>> nat lu wrote:
>>>>>>> I made a start, and tried to use one of the existing flavours, but
>>>>>>> ended
>>>>>>> up creating one for MonetDB - combination of derby and DB2. It
>>>>>>> doesnt
>>>>>>> like longs or unbounded varchars.
>>>>>>>
>>>>>>> So, I got as far as getting SDBConfig to complete, but havent
>>>>>>> done an
>>>>>>> sdbload yet
>>>>>>>
>>>>>>>
>>>>>>> On 09/09/11 10:37, Andy Seaborne wrote:
>>>>>>>> On 04/09/11 13:03, nat lu wrote:
>>>>>>>>> I'm going to give it a go sometime soon and report back on my
>>>>>>>>> non-scientific findings. Your point about the small number of
>>>>>>>>> columns is
>>>>>>>>> well made, but the research paper cited earlier also mentions
>>>>>>>>> this and
>>>>>>>>> reports that because of column store optimisations even when they
>>>>>>>>> vertically partitioned their data rather than using a
>>>>>>>>> property-table
>>>>>>>>> approach they still saw good improvement. However, again, I'm no
>>>>>>>>> column
>>>>>>>>> store expert so perhaps I'm missing some point here :-). Anyway,
>>>>>>>>> time to
>>>>>>>>> "suck it and see@, all in the name of progress of course.
>>>>>>>>>
>>>>>>>>> On 03/09/11 16:29, David Jordan wrote:
>>>>>>>>>> I have not used a column-oriented database, but I am somewhat
>>>>>>>>>> familiar
>>>>>>>>>> with them. My understanding of them is that the storage is
>>>>>>>>>> partitioned
>>>>>>>>>> on a column basis, such that there is no physical clustering
>>>>>>>>>> together
>>>>>>>>>> of all the columns for a given row. An advantage of this would
>>>>>>>>>> be in
>>>>>>>>>> the case where you have tables with many columns, but the
>>>>>>>>>> particular
>>>>>>>>>> application only needs a small subset of columns.
>>>>>>>>>>
>>>>>>>>>> With the SDB representation of triples (3 columns) and quads (4
>>>>>>>>>> columns), and access typically based on having a specific
>>>>>>>>>> value for
>>>>>>>>>> one or two of the columns, I am not so sure that a column-based
>>>>>>>>>> approach would offer any advantage.
>>>>>>>>>>
>>>>>>>>>> But again, I am no expert on these types of databases.
>>>>>>>>>>
>>>>>>>>>> These discussions about alternative datastore representations
>>>>>>>>>> RDF/OWL
>>>>>>>>>> data are very useful, to gain better understanding of which data
>>>>>>>>>> architectures yield the best implementation approach for
>>>>>>>>>> high-performance.
>>>>>>>>>>
>>>>>>>>>> p.s. I Monet provides support for JDBC, I would not think much
>>>>>>>>>> effort
>>>>>>>>>> is needed to support in with SDB.
>>>>>>>> Shouldn't be too hard :-)  SDB targets SQL-92 and there are a few
>>>>>>>> extension points to cope with the vagaries of different SQL
>>>>>>>> engines.
>>>>>>>> It's one of the reasons there are ~10 small files to write, to
>>>>>>>> capture
>>>>>>>> the uniqueness of each SQL syntax.
>>>>>>>>
>>>>>>>>        Andy
> 

Re: Adding support for MonetDB

[nat lu <na...@gmail.com>]

Thanks, I hope the community can help progress this more, and that this 
is just a very small start. I'm sure I'm not the only one interested in 
the possibilities of column stores as RDF repositories.

"I'll be back...." :-)

On 11/10/11 12:15, Paolo Castagna wrote:
> Hi nat lu
>
> nat lu wrote:
>> HI,
>>
>> Had a quick look - the patch seems to want to delete (in entirety) a
>> couple of existing SDB classes which I had made minimal modifications
>> to, so I've re-checked out SDB 1.3.4 and created a new patch which I can
>> upload later. So - the first attachment in Jira shouldnt be used.
> Great. Thanks for double checking and acting on this.
> This will make life easier to whoever review your patch.
>
>> Secondly, I haven't recreated all the unit test code that exists for
>> other RDBMS supported by SDB because, simply by the looks of it, there's
>> quite a bit of it, and I haven't had time so far.
> Ack.
>
> A patch without tests is better than no patch at all.
>
> However, tests are really important and should be considered part of the
> process of submitting a patch.
>
> They certainly help us a lot and can make a patch going into trunk faster.
>
>> I've also, as I said before, not used Monet with any data yet, just got
>> it to the point that SDBConfig doesn't fall over, and the Jira issue
>> raised is simply to record the desire to support it (and perhaps other
>> column stores if conceptually they prove to have merit when used as an
>> RDF repo).
> Ack.
>
> Others wanting to have MonetDB will see your issue and might come and
> help you with that.
>
> I personally don't use SDB much and I've never used MonetDB before.
>
>> So, personally, I'd prefer (I assume you-all as well) that nothing
>> happens too quickly with this, until I or someone else gets some time to
>> do some work and produce the unit tests.
> Sure.
>
> We tend not to commit untested code. :-)
>
> Having low quality or unfinished code committed to trunk is a bad practice
> and we avoid that.
>
> Rules have exceptions though, for example, I recently committed an RDF/JSON
> writer which isn't complete/fully tested, however it's not going to affect
> users in any way (see JENA-135). I did that to make Rob's life easier in case
> he want to have an RDF/JSON writer as well as a parser (which he contributed).
>
> Nat, don't take these last two my messages too much personally, I am just
> taking the opportunities to send across a few IMHO important messages:
>
>   - we welcome patches and new ideas/features (we really do!)
>   - proper patches, well tested and which apply cleanly make our life
>     so much easier (so we appreciate any effort in this direction).
>     It's very difficult to get it wrong with svn diff>  JENA-XYZ.patch. :-)
>
> Thanks again and have fun with SDB + MonetDB!
>
> Paolo
>
>> On 11/10/11 08:07, Paolo Castagna wrote:
>>> Hi,
>>> first of all, thank you for your patch.
>>> I had a quick look, but I did not try to apply it (yet).
>>>
>>> May I ask how you created your patch?
>>>
>>> We added a section on the Getting Involved page on the Jena website:
>>>
>>>    "Patches should be attached to issues in Jira (click on
>>>     More Actions>   Attach Files). To create a patch you can simply
>>>     use the command:
>>>
>>>       svn diff>   JENA-XYZ.patch
>>>
>>>     Please, inspect your patch and make sure it includes all (and only)
>>>     the relevant changes for a single issue. Don't forget tests! If you
>>>     want to test if a patch applies cleanly you can use:
>>>
>>>       patch -p0<   JENA-XYZ.patch
>>>
>>>     If you use Eclipse: right click on the project name in Package
>>> Explorer,
>>>     select Team>   Create Patch or Team>   Apply Patch."
>>>
>>> -
>>> http://jena.staging.apache.org/jena/getting_involved/#submit_your_patches
>>>
>>>
>>> It really helps if a patch contains only the lines you added|removed
>>> and it applies cleanly. It saves a lot of time and speed-up reviewing
>>> it.
>>>
>>> Your patch may be perfectly fine, but I wanted to take the opportunity
>>> to send the message across.
>>>
>>> I am really curious to run a few benchmarks when it's done to compare
>>> MonetDB with a more traditional SQL system.
>>>
>>> By the way, about benchmarks Andy is (secretely) working on this:
>>> https://svn.apache.org/repos/asf/incubator/jena/Experimental/JenaPerf/trunk/
>>>
>>> I have not time to try it yet, but it seems very interesting. :-)
>>>
>>> Thank you again for the new interesting feature.
>>>
>>> Paolo
>>>
>>>
>>> nat lu wrote:
>>>> Added, with patch file, at
>>>>
>>>> https://issues.apache.org/jira/browse/JENA-134
>>>>
>>>>
>>>> I have made no more progress on testing it out other than sdbconfig so
>>>> far, hope too soon.
>>>>
>>>>
>>>> On 09/09/11 16:43, Paolo Castagna wrote:
>>>>> Hi,
>>>>> why don't you open a new JIRA issue (as a New Feature) for this?
>>>>> https://issues.apache.org/jira/browse/JENA
>>>>>
>>>>> You can then attach a patch to it. This way others can look at what
>>>>> you have done so far (and maybe help you out).
>>>>>
>>>>> Thanks for your help,
>>>>> Paolo
>>>>>
>>>>> nat lu wrote:
>>>>>> I made a start, and tried to use one of the existing flavours, but
>>>>>> ended
>>>>>> up creating one for MonetDB - combination of derby and DB2. It doesnt
>>>>>> like longs or unbounded varchars.
>>>>>>
>>>>>> So, I got as far as getting SDBConfig to complete, but havent done an
>>>>>> sdbload yet
>>>>>>
>>>>>>
>>>>>> On 09/09/11 10:37, Andy Seaborne wrote:
>>>>>>> On 04/09/11 13:03, nat lu wrote:
>>>>>>>> I'm going to give it a go sometime soon and report back on my
>>>>>>>> non-scientific findings. Your point about the small number of
>>>>>>>> columns is
>>>>>>>> well made, but the research paper cited earlier also mentions
>>>>>>>> this and
>>>>>>>> reports that because of column store optimisations even when they
>>>>>>>> vertically partitioned their data rather than using a property-table
>>>>>>>> approach they still saw good improvement. However, again, I'm no
>>>>>>>> column
>>>>>>>> store expert so perhaps I'm missing some point here :-). Anyway,
>>>>>>>> time to
>>>>>>>> "suck it and see@, all in the name of progress of course.
>>>>>>>>
>>>>>>>> On 03/09/11 16:29, David Jordan wrote:
>>>>>>>>> I have not used a column-oriented database, but I am somewhat
>>>>>>>>> familiar
>>>>>>>>> with them. My understanding of them is that the storage is
>>>>>>>>> partitioned
>>>>>>>>> on a column basis, such that there is no physical clustering
>>>>>>>>> together
>>>>>>>>> of all the columns for a given row. An advantage of this would
>>>>>>>>> be in
>>>>>>>>> the case where you have tables with many columns, but the
>>>>>>>>> particular
>>>>>>>>> application only needs a small subset of columns.
>>>>>>>>>
>>>>>>>>> With the SDB representation of triples (3 columns) and quads (4
>>>>>>>>> columns), and access typically based on having a specific value for
>>>>>>>>> one or two of the columns, I am not so sure that a column-based
>>>>>>>>> approach would offer any advantage.
>>>>>>>>>
>>>>>>>>> But again, I am no expert on these types of databases.
>>>>>>>>>
>>>>>>>>> These discussions about alternative datastore representations
>>>>>>>>> RDF/OWL
>>>>>>>>> data are very useful, to gain better understanding of which data
>>>>>>>>> architectures yield the best implementation approach for
>>>>>>>>> high-performance.
>>>>>>>>>
>>>>>>>>> p.s. I Monet provides support for JDBC, I would not think much
>>>>>>>>> effort
>>>>>>>>> is needed to support in with SDB.
>>>>>>> Shouldn't be too hard :-)  SDB targets SQL-92 and there are a few
>>>>>>> extension points to cope with the vagaries of different SQL engines.
>>>>>>> It's one of the reasons there are ~10 small files to write, to
>>>>>>> capture
>>>>>>> the uniqueness of each SQL syntax.
>>>>>>>
>>>>>>>        Andy

Re: Adding support for MonetDB

[Paolo Castagna <ca...@googlemail.com>]

Hi nat lu

nat lu wrote:
> HI,
> 
> Had a quick look - the patch seems to want to delete (in entirety) a
> couple of existing SDB classes which I had made minimal modifications
> to, so I've re-checked out SDB 1.3.4 and created a new patch which I can
> upload later. So - the first attachment in Jira shouldnt be used.

Great. Thanks for double checking and acting on this.
This will make life easier to whoever review your patch.

> Secondly, I haven't recreated all the unit test code that exists for
> other RDBMS supported by SDB because, simply by the looks of it, there's
> quite a bit of it, and I haven't had time so far.

Ack.

A patch without tests is better than no patch at all.

However, tests are really important and should be considered part of the
process of submitting a patch.

They certainly help us a lot and can make a patch going into trunk faster.

> I've also, as I said before, not used Monet with any data yet, just got
> it to the point that SDBConfig doesn't fall over, and the Jira issue
> raised is simply to record the desire to support it (and perhaps other
> column stores if conceptually they prove to have merit when used as an
> RDF repo).

Ack.

Others wanting to have MonetDB will see your issue and might come and
help you with that.

I personally don't use SDB much and I've never used MonetDB before.

> So, personally, I'd prefer (I assume you-all as well) that nothing
> happens too quickly with this, until I or someone else gets some time to
> do some work and produce the unit tests.

Sure.

We tend not to commit untested code. :-)

Having low quality or unfinished code committed to trunk is a bad practice
and we avoid that.

Rules have exceptions though, for example, I recently committed an RDF/JSON
writer which isn't complete/fully tested, however it's not going to affect
users in any way (see JENA-135). I did that to make Rob's life easier in case
he want to have an RDF/JSON writer as well as a parser (which he contributed).

Nat, don't take these last two my messages too much personally, I am just
taking the opportunities to send across a few IMHO important messages:

 - we welcome patches and new ideas/features (we really do!)
 - proper patches, well tested and which apply cleanly make our life
   so much easier (so we appreciate any effort in this direction).
   It's very difficult to get it wrong with svn diff > JENA-XYZ.patch. :-)

Thanks again and have fun with SDB + MonetDB!

Paolo

> 
> On 11/10/11 08:07, Paolo Castagna wrote:
>> Hi,
>> first of all, thank you for your patch.
>> I had a quick look, but I did not try to apply it (yet).
>>
>> May I ask how you created your patch?
>>
>> We added a section on the Getting Involved page on the Jena website:
>>
>>   "Patches should be attached to issues in Jira (click on
>>    More Actions>  Attach Files). To create a patch you can simply
>>    use the command:
>>
>>      svn diff>  JENA-XYZ.patch
>>
>>    Please, inspect your patch and make sure it includes all (and only)
>>    the relevant changes for a single issue. Don't forget tests! If you
>>    want to test if a patch applies cleanly you can use:
>>
>>      patch -p0<  JENA-XYZ.patch
>>
>>    If you use Eclipse: right click on the project name in Package
>> Explorer,
>>    select Team>  Create Patch or Team>  Apply Patch."
>>
>> -
>> http://jena.staging.apache.org/jena/getting_involved/#submit_your_patches
>>
>>
>> It really helps if a patch contains only the lines you added|removed
>> and it applies cleanly. It saves a lot of time and speed-up reviewing
>> it.
>>
>> Your patch may be perfectly fine, but I wanted to take the opportunity
>> to send the message across.
>>
>> I am really curious to run a few benchmarks when it's done to compare
>> MonetDB with a more traditional SQL system.
>>
>> By the way, about benchmarks Andy is (secretely) working on this:
>> https://svn.apache.org/repos/asf/incubator/jena/Experimental/JenaPerf/trunk/
>>
>> I have not time to try it yet, but it seems very interesting. :-)
>>
>> Thank you again for the new interesting feature.
>>
>> Paolo
>>
>>
>> nat lu wrote:
>>> Added, with patch file, at
>>>
>>> https://issues.apache.org/jira/browse/JENA-134
>>>
>>>
>>> I have made no more progress on testing it out other than sdbconfig so
>>> far, hope too soon.
>>>
>>>
>>> On 09/09/11 16:43, Paolo Castagna wrote:
>>>> Hi,
>>>> why don't you open a new JIRA issue (as a New Feature) for this?
>>>> https://issues.apache.org/jira/browse/JENA
>>>>
>>>> You can then attach a patch to it. This way others can look at what
>>>> you have done so far (and maybe help you out).
>>>>
>>>> Thanks for your help,
>>>> Paolo
>>>>
>>>> nat lu wrote:
>>>>> I made a start, and tried to use one of the existing flavours, but
>>>>> ended
>>>>> up creating one for MonetDB - combination of derby and DB2. It doesnt
>>>>> like longs or unbounded varchars.
>>>>>
>>>>> So, I got as far as getting SDBConfig to complete, but havent done an
>>>>> sdbload yet
>>>>>
>>>>>
>>>>> On 09/09/11 10:37, Andy Seaborne wrote:
>>>>>> On 04/09/11 13:03, nat lu wrote:
>>>>>>> I'm going to give it a go sometime soon and report back on my
>>>>>>> non-scientific findings. Your point about the small number of
>>>>>>> columns is
>>>>>>> well made, but the research paper cited earlier also mentions
>>>>>>> this and
>>>>>>> reports that because of column store optimisations even when they
>>>>>>> vertically partitioned their data rather than using a property-table
>>>>>>> approach they still saw good improvement. However, again, I'm no
>>>>>>> column
>>>>>>> store expert so perhaps I'm missing some point here :-). Anyway,
>>>>>>> time to
>>>>>>> "suck it and see@, all in the name of progress of course.
>>>>>>>
>>>>>>> On 03/09/11 16:29, David Jordan wrote:
>>>>>>>> I have not used a column-oriented database, but I am somewhat
>>>>>>>> familiar
>>>>>>>> with them. My understanding of them is that the storage is
>>>>>>>> partitioned
>>>>>>>> on a column basis, such that there is no physical clustering
>>>>>>>> together
>>>>>>>> of all the columns for a given row. An advantage of this would
>>>>>>>> be in
>>>>>>>> the case where you have tables with many columns, but the
>>>>>>>> particular
>>>>>>>> application only needs a small subset of columns.
>>>>>>>>
>>>>>>>> With the SDB representation of triples (3 columns) and quads (4
>>>>>>>> columns), and access typically based on having a specific value for
>>>>>>>> one or two of the columns, I am not so sure that a column-based
>>>>>>>> approach would offer any advantage.
>>>>>>>>
>>>>>>>> But again, I am no expert on these types of databases.
>>>>>>>>
>>>>>>>> These discussions about alternative datastore representations
>>>>>>>> RDF/OWL
>>>>>>>> data are very useful, to gain better understanding of which data
>>>>>>>> architectures yield the best implementation approach for
>>>>>>>> high-performance.
>>>>>>>>
>>>>>>>> p.s. I Monet provides support for JDBC, I would not think much
>>>>>>>> effort
>>>>>>>> is needed to support in with SDB.
>>>>>> Shouldn't be too hard :-)  SDB targets SQL-92 and there are a few
>>>>>> extension points to cope with the vagaries of different SQL engines.
>>>>>> It's one of the reasons there are ~10 small files to write, to
>>>>>> capture
>>>>>> the uniqueness of each SQL syntax.
>>>>>>
>>>>>>       Andy
> 

RE: Patch Generation and svn:eol-style

["Dennis E. Hamilton" <or...@apache.org>]

The recommended http://www.apache.org/dev/svn-eol-style.txt settings in the SVN client are important. (Tortoise SVN just gives you another way to find and set them, this is all SVN-level behavior.)

I found it annoying to do that for XML and HTML, so I modified the change to leave those alone - those specifications do not care about whitespace the same way as it matters for a text file.

I also adjusted mine to always set the charset to UTF8 for text files and a few others.  The result is like this:

    *.html = svn:mime-type=text/html;;charset=UTF-8
    *.htm = svn:mime-type=text/html;;charset=UTF-8
    *.txt = svn:eol-style=native;;charset=UTF-8

Of course, the HMTL files with no extension in the Jena SVN serve up as text, which is somewhat annoying.

I also notice that there are a set of commented-out auto-props settings in the default SVN config file.  Most of these are defined by the Apache svn-eol-style.txt addition anyhow.

My understanding is that these take effect only as new files are added to the SVN, although that might not be true for the eol-native situation.  With existing files, you get what you get.

 - Dennis

-----Original Message-----
From: Robert Vesse [mailto:rvesse@cray.com] 
Sent: Tuesday, November 15, 2011 15:19
To: jena-dev@incubator.apache.org
Subject: RE: Patch Generation and svn:eol-style

Yes I did use TortioiseSVN but I also tried using svn diff from the command line and still had the same result either way

Perhaps if I'd had svn:eol-style set appropriately this would have fixed things

Rob

-----Original Message-----
From: Paolo Castagna [mailto:castagna.lists@googlemail.com] 
Sent: Tuesday, November 15, 2011 2:50 PM
To: jena-dev@incubator.apache.org
Subject: Re: Adding support for MonetDB

Hi Rob,
I don't remember, did you use TortoiseSVN on Windows to generate your patch?
I am still unsure if we should run svn propset svn:eol-style native
for all the text file (I usually do not do that).

Paolo

On 15 November 2011 23:45, Robert Vesse <rv...@cray.com> wrote:
> Thanks for figuring this out Paolo, sounds like the same issue we had with my RDF/JSON parsing patch
>
> Rob
>
> -----Original Message-----
> From: Paolo Castagna [mailto:castagna.lists@googlemail.com]
> Sent: Tuesday, November 15, 2011 2:42 PM
> To: jena-dev@incubator.apache.org
> Subject: Re: Adding support for MonetDB
>
> Hi,
> I think I might have identified what can go wrong and cause problems
> when users submit patches.
>
> On 11 October 2011 13:34, Paolo Castagna <ca...@googlemail.com> wrote:
>> Hi Damian
>>
>> Damian Steer wrote:
>>> On 11 Oct 2011, at 11:32, nat lu wrote:
>>>
>>>> HI,
>>>>
>>>> Had a quick look - the patch seems to want to delete (in entirety) a couple of existing SDB classes which I had made minimal modifications to, so I've re-checked out SDB 1.3.4 and created a new patch which I can upload later. So - the first attachment in Jira shouldnt be used.
>>>
>>> Thanks for looking at this. I just tried applying this and it went a bit wrong.
>>
>> It's strange, I think it's really important we understand what goes wrong when we receive a patch which do not apply cleanly.
>> If the patch is small, it's not a big deal to apply the changes manually. But it's time consuming.
>> If the patch is big, applying it manually is not an option.
>
> """
> Configuring the Subversion client
>
> Committers will need to properly configure their svn client. One
> particular issue is OS-specific line-endings for text files. [...] Add
> the contents of the file http://www.apache.org/dev/svn-eol-style.txt
> to the bottom of your ~/.subversion/config file.
> [...]
> Tip: If you use TortiseSVN, a popular Windows GUI client that
> integrates with Windows Explorer, you can simply right click in
> Explorer and select TortiseSVN - Settings, and then press the "Edit"
> button to update your "Subversion configuration file:". Simply copy
> the above svn-eol-style.txt file's contents into the end of the config
> editor (usually Notepad) that appears, and save the file.
> """
> -- http://www.apache.org/dev/version-control.html#https-svn-config
>
> Maybe is this what happened: all the lines were changed because of the
> line-endings.
> I am sending this email so that when this happens again we all know
> where to look for a fix.
>
> Cheers,
> Paolo
>

RE: Patch Generation and svn:eol-style

[Robert Vesse <rv...@cray.com>]

Yes I did use TortioiseSVN but I also tried using svn diff from the command line and still had the same result either way

Perhaps if I'd had svn:eol-style set appropriately this would have fixed things

Rob

-----Original Message-----
From: Paolo Castagna [mailto:castagna.lists@googlemail.com] 
Sent: Tuesday, November 15, 2011 2:50 PM
To: jena-dev@incubator.apache.org
Subject: Re: Adding support for MonetDB

Hi Rob,
I don't remember, did you use TortoiseSVN on Windows to generate your patch?
I am still unsure if we should run svn propset svn:eol-style native
for all the text file (I usually do not do that).

Paolo

On 15 November 2011 23:45, Robert Vesse <rv...@cray.com> wrote:
> Thanks for figuring this out Paolo, sounds like the same issue we had with my RDF/JSON parsing patch
>
> Rob
>
> -----Original Message-----
> From: Paolo Castagna [mailto:castagna.lists@googlemail.com]
> Sent: Tuesday, November 15, 2011 2:42 PM
> To: jena-dev@incubator.apache.org
> Subject: Re: Adding support for MonetDB
>
> Hi,
> I think I might have identified what can go wrong and cause problems
> when users submit patches.
>
> On 11 October 2011 13:34, Paolo Castagna <ca...@googlemail.com> wrote:
>> Hi Damian
>>
>> Damian Steer wrote:
>>> On 11 Oct 2011, at 11:32, nat lu wrote:
>>>
>>>> HI,
>>>>
>>>> Had a quick look - the patch seems to want to delete (in entirety) a couple of existing SDB classes which I had made minimal modifications to, so I've re-checked out SDB 1.3.4 and created a new patch which I can upload later. So - the first attachment in Jira shouldnt be used.
>>>
>>> Thanks for looking at this. I just tried applying this and it went a bit wrong.
>>
>> It's strange, I think it's really important we understand what goes wrong when we receive a patch which do not apply cleanly.
>> If the patch is small, it's not a big deal to apply the changes manually. But it's time consuming.
>> If the patch is big, applying it manually is not an option.
>
> """
> Configuring the Subversion client
>
> Committers will need to properly configure their svn client. One
> particular issue is OS-specific line-endings for text files. [...] Add
> the contents of the file http://www.apache.org/dev/svn-eol-style.txt
> to the bottom of your ~/.subversion/config file.
> [...]
> Tip: If you use TortiseSVN, a popular Windows GUI client that
> integrates with Windows Explorer, you can simply right click in
> Explorer and select TortiseSVN - Settings, and then press the "Edit"
> button to update your "Subversion configuration file:". Simply copy
> the above svn-eol-style.txt file's contents into the end of the config
> editor (usually Notepad) that appears, and save the file.
> """
> -- http://www.apache.org/dev/version-control.html#https-svn-config
>
> Maybe is this what happened: all the lines were changed because of the
> line-endings.
> I am sending this email so that when this happens again we all know
> where to look for a fix.
>
> Cheers,
> Paolo
>

Re: Adding support for MonetDB

[Paolo Castagna <ca...@googlemail.com>]

Hi Rob,
I don't remember, did you use TortoiseSVN on Windows to generate your patch?
I am still unsure if we should run svn propset svn:eol-style native
for all the text file (I usually do not do that).

Paolo

On 15 November 2011 23:45, Robert Vesse <rv...@cray.com> wrote:
> Thanks for figuring this out Paolo, sounds like the same issue we had with my RDF/JSON parsing patch
>
> Rob
>
> -----Original Message-----
> From: Paolo Castagna [mailto:castagna.lists@googlemail.com]
> Sent: Tuesday, November 15, 2011 2:42 PM
> To: jena-dev@incubator.apache.org
> Subject: Re: Adding support for MonetDB
>
> Hi,
> I think I might have identified what can go wrong and cause problems
> when users submit patches.
>
> On 11 October 2011 13:34, Paolo Castagna <ca...@googlemail.com> wrote:
>> Hi Damian
>>
>> Damian Steer wrote:
>>> On 11 Oct 2011, at 11:32, nat lu wrote:
>>>
>>>> HI,
>>>>
>>>> Had a quick look - the patch seems to want to delete (in entirety) a couple of existing SDB classes which I had made minimal modifications to, so I've re-checked out SDB 1.3.4 and created a new patch which I can upload later. So - the first attachment in Jira shouldnt be used.
>>>
>>> Thanks for looking at this. I just tried applying this and it went a bit wrong.
>>
>> It's strange, I think it's really important we understand what goes wrong when we receive a patch which do not apply cleanly.
>> If the patch is small, it's not a big deal to apply the changes manually. But it's time consuming.
>> If the patch is big, applying it manually is not an option.
>
> """
> Configuring the Subversion client
>
> Committers will need to properly configure their svn client. One
> particular issue is OS-specific line-endings for text files. [...] Add
> the contents of the file http://www.apache.org/dev/svn-eol-style.txt
> to the bottom of your ~/.subversion/config file.
> [...]
> Tip: If you use TortiseSVN, a popular Windows GUI client that
> integrates with Windows Explorer, you can simply right click in
> Explorer and select TortiseSVN - Settings, and then press the "Edit"
> button to update your "Subversion configuration file:". Simply copy
> the above svn-eol-style.txt file's contents into the end of the config
> editor (usually Notepad) that appears, and save the file.
> """
> -- http://www.apache.org/dev/version-control.html#https-svn-config
>
> Maybe is this what happened: all the lines were changed because of the
> line-endings.
> I am sending this email so that when this happens again we all know
> where to look for a fix.
>
> Cheers,
> Paolo
>

RE: Adding support for MonetDB

[Robert Vesse <rv...@cray.com>]

Thanks for figuring this out Paolo, sounds like the same issue we had with my RDF/JSON parsing patch

Rob

-----Original Message-----
From: Paolo Castagna [mailto:castagna.lists@googlemail.com] 
Sent: Tuesday, November 15, 2011 2:42 PM
To: jena-dev@incubator.apache.org
Subject: Re: Adding support for MonetDB

Hi,
I think I might have identified what can go wrong and cause problems
when users submit patches.

On 11 October 2011 13:34, Paolo Castagna <ca...@googlemail.com> wrote:
> Hi Damian
>
> Damian Steer wrote:
>> On 11 Oct 2011, at 11:32, nat lu wrote:
>>
>>> HI,
>>>
>>> Had a quick look - the patch seems to want to delete (in entirety) a couple of existing SDB classes which I had made minimal modifications to, so I've re-checked out SDB 1.3.4 and created a new patch which I can upload later. So - the first attachment in Jira shouldnt be used.
>>
>> Thanks for looking at this. I just tried applying this and it went a bit wrong.
>
> It's strange, I think it's really important we understand what goes wrong when we receive a patch which do not apply cleanly.
> If the patch is small, it's not a big deal to apply the changes manually. But it's time consuming.
> If the patch is big, applying it manually is not an option.

"""
Configuring the Subversion client

Committers will need to properly configure their svn client. One
particular issue is OS-specific line-endings for text files. [...] Add
the contents of the file http://www.apache.org/dev/svn-eol-style.txt
to the bottom of your ~/.subversion/config file.
[...]
Tip: If you use TortiseSVN, a popular Windows GUI client that
integrates with Windows Explorer, you can simply right click in
Explorer and select TortiseSVN - Settings, and then press the "Edit"
button to update your "Subversion configuration file:". Simply copy
the above svn-eol-style.txt file's contents into the end of the config
editor (usually Notepad) that appears, and save the file.
"""
-- http://www.apache.org/dev/version-control.html#https-svn-config

Maybe is this what happened: all the lines were changed because of the
line-endings.
I am sending this email so that when this happens again we all know
where to look for a fix.

Cheers,
Paolo

Re: Adding support for MonetDB

[Paolo Castagna <ca...@googlemail.com>]

Hi,
I think I might have identified what can go wrong and cause problems
when users submit patches.

On 11 October 2011 13:34, Paolo Castagna <ca...@googlemail.com> wrote:
> Hi Damian
>
> Damian Steer wrote:
>> On 11 Oct 2011, at 11:32, nat lu wrote:
>>
>>> HI,
>>>
>>> Had a quick look - the patch seems to want to delete (in entirety) a couple of existing SDB classes which I had made minimal modifications to, so I've re-checked out SDB 1.3.4 and created a new patch which I can upload later. So - the first attachment in Jira shouldnt be used.
>>
>> Thanks for looking at this. I just tried applying this and it went a bit wrong.
>
> It's strange, I think it's really important we understand what goes wrong when we receive a patch which do not apply cleanly.
> If the patch is small, it's not a big deal to apply the changes manually. But it's time consuming.
> If the patch is big, applying it manually is not an option.

"""
Configuring the Subversion client

Committers will need to properly configure their svn client. One
particular issue is OS-specific line-endings for text files. [...] Add
the contents of the file http://www.apache.org/dev/svn-eol-style.txt
to the bottom of your ~/.subversion/config file.
[...]
Tip: If you use TortiseSVN, a popular Windows GUI client that
integrates with Windows Explorer, you can simply right click in
Explorer and select TortiseSVN - Settings, and then press the "Edit"
button to update your "Subversion configuration file:". Simply copy
the above svn-eol-style.txt file's contents into the end of the config
editor (usually Notepad) that appears, and save the file.
"""
-- http://www.apache.org/dev/version-control.html#https-svn-config

Maybe is this what happened: all the lines were changed because of the
line-endings.
I am sending this email so that when this happens again we all know
where to look for a fix.

Cheers,
Paolo

Re: Adding support for MonetDB

[Paolo Castagna <ca...@googlemail.com>]

Hi Damian

Damian Steer wrote:
> On 11 Oct 2011, at 11:32, nat lu wrote:
> 
>> HI,
>>
>> Had a quick look - the patch seems to want to delete (in entirety) a couple of existing SDB classes which I had made minimal modifications to, so I've re-checked out SDB 1.3.4 and created a new patch which I can upload later. So - the first attachment in Jira shouldnt be used.
> 
> Thanks for looking at this. I just tried applying this and it went a bit wrong.

It's strange, I think it's really important we understand what goes wrong when we receive a patch which do not apply cleanly.
If the patch is small, it's not a big deal to apply the changes manually. But it's time consuming.
If the patch is big, applying it manually is not an option.

I use both the command line and Eclipse to create a patch:

command line:

    svn diff > JENA-XYZ.patch

Eclipse:

 1. right click on the project name in Package Explorer
 2. select Team > Create Patch

What can go wrong?

> 
>> Secondly, I haven't recreated all the unit test code that exists for other RDBMS supported by SDB because, simply by the looks of it, there's quite a bit of it, and I haven't had time so far.
> 
> Urgh, that is horrible.

:-)

Yeah, tests might come in a subsequent round...

> 
>> I've also, as I said before, not used Monet with any data yet, just got it to the point that SDBConfig doesn't fall over, and the Jira issue raised is simply to record the desire to support it (and perhaps other column stores if conceptually they prove to have merit when used as an RDF repo).
>>
>> So, personally, I'd prefer (I assume you-all as well) that nothing happens too quickly with this, until I or someone else gets some time to do some work and produce the unit tests.
> 
> 
> The current tests, in part, run the graph tests over the store. That ought to be easier to get going. Will investigate.
> 
> Damian

Re: Adding support for MonetDB

[nat lu <na...@gmail.com>]

Tomorrow, I'll check what version of Eclipse I did that on, and which 
SVN connector.
Can also produce a cmdline version as well.

BTW, what was "a bit wrong" ?


On 11/10/11 12:34, Paolo Castagna wrote:
> Hi Damian
>
> Damian Steer wrote:
>> On 11 Oct 2011, at 11:32, nat lu wrote:
>>
>>> HI,
>>>
>>> Had a quick look - the patch seems to want to delete (in entirety) a couple of existing SDB classes which I had made minimal modifications to, so I've re-checked out SDB 1.3.4 and created a new patch which I can upload later. So - the first attachment in Jira shouldnt be used.
>> Thanks for looking at this. I just tried applying this and it went a bit wrong.
> It's strange, I think it's really important we understand what goes wrong when we receive a patch which do not apply cleanly.
> If the patch is small, it's not a big deal to apply the changes manually. But it's time consuming.
> If the patch is big, applying it manually is not an option.
>
> I use both the command line and Eclipse to create a patch:
>
> command line:
>
>      svn diff>  JENA-XYZ.patch
>
> Eclipse:
>
>   1. right click on the project name in Package Explorer
>   2. select Team>  Create Patch
>
> What can go wrong?
>
>>> Secondly, I haven't recreated all the unit test code that exists for other RDBMS supported by SDB because, simply by the looks of it, there's quite a bit of it, and I haven't had time so far.
>> Urgh, that is horrible.
> :-)
>
> Yeah, tests might come in a subsequent round...
>
>>> I've also, as I said before, not used Monet with any data yet, just got it to the point that SDBConfig doesn't fall over, and the Jira issue raised is simply to record the desire to support it (and perhaps other column stores if conceptually they prove to have merit when used as an RDF repo).
>>>
>>> So, personally, I'd prefer (I assume you-all as well) that nothing happens too quickly with this, until I or someone else gets some time to do some work and produce the unit tests.
>>
>> The current tests, in part, run the graph tests over the store. That ought to be easier to get going. Will investigate.
>>
>> Damian

Re: Adding support for MonetDB

[Paolo Castagna <ca...@googlemail.com>]

Hi Damian

Damian Steer wrote:
> On 11 Oct 2011, at 11:32, nat lu wrote:
> 
>> HI,
>>
>> Had a quick look - the patch seems to want to delete (in entirety) a couple of existing SDB classes which I had made minimal modifications to, so I've re-checked out SDB 1.3.4 and created a new patch which I can upload later. So - the first attachment in Jira shouldnt be used.
> 
> Thanks for looking at this. I just tried applying this and it went a bit wrong.

It's strange, I think it's really important we understand what goes wrong when we receive a patch which do not apply cleanly.
If the patch is small, it's not a big deal to apply the changes manually. But it's time consuming.
If the patch is big, applying it manually is not an option.

I use both the command line and Eclipse to create a patch:

command line:

    svn diff > JENA-XYZ.patch

Eclipse:

 1. right click on the project name in Package Explorer
 2. select Team > Create Patch

What can go wrong?

> 
>> Secondly, I haven't recreated all the unit test code that exists for other RDBMS supported by SDB because, simply by the looks of it, there's quite a bit of it, and I haven't had time so far.
> 
> Urgh, that is horrible.

:-)

Yeah, tests might come in a subsequent round...

> 
>> I've also, as I said before, not used Monet with any data yet, just got it to the point that SDBConfig doesn't fall over, and the Jira issue raised is simply to record the desire to support it (and perhaps other column stores if conceptually they prove to have merit when used as an RDF repo).
>>
>> So, personally, I'd prefer (I assume you-all as well) that nothing happens too quickly with this, until I or someone else gets some time to do some work and produce the unit tests.
> 
> 
> The current tests, in part, run the graph tests over the store. That ought to be easier to get going. Will investigate.
> 
> Damian

Re: Adding support for MonetDB

[Damian Steer <d....@bristol.ac.uk>]

On 11 Oct 2011, at 11:32, nat lu wrote:

> HI,
> 
> Had a quick look - the patch seems to want to delete (in entirety) a couple of existing SDB classes which I had made minimal modifications to, so I've re-checked out SDB 1.3.4 and created a new patch which I can upload later. So - the first attachment in Jira shouldnt be used.

Thanks for looking at this. I just tried applying this and it went a bit wrong.

> Secondly, I haven't recreated all the unit test code that exists for other RDBMS supported by SDB because, simply by the looks of it, there's quite a bit of it, and I haven't had time so far.

Urgh, that is horrible.

> I've also, as I said before, not used Monet with any data yet, just got it to the point that SDBConfig doesn't fall over, and the Jira issue raised is simply to record the desire to support it (and perhaps other column stores if conceptually they prove to have merit when used as an RDF repo).
> 
> So, personally, I'd prefer (I assume you-all as well) that nothing happens too quickly with this, until I or someone else gets some time to do some work and produce the unit tests.


The current tests, in part, run the graph tests over the store. That ought to be easier to get going. Will investigate.

Damian

Re: Adding support for MonetDB

[nat lu <na...@gmail.com>]

HI,

Had a quick look - the patch seems to want to delete (in entirety) a 
couple of existing SDB classes which I had made minimal modifications 
to, so I've re-checked out SDB 1.3.4 and created a new patch which I can 
upload later. So - the first attachment in Jira shouldnt be used.

Secondly, I haven't recreated all the unit test code that exists for 
other RDBMS supported by SDB because, simply by the looks of it, there's 
quite a bit of it, and I haven't had time so far.

I've also, as I said before, not used Monet with any data yet, just got 
it to the point that SDBConfig doesn't fall over, and the Jira issue 
raised is simply to record the desire to support it (and perhaps other 
column stores if conceptually they prove to have merit when used as an 
RDF repo).

So, personally, I'd prefer (I assume you-all as well) that nothing 
happens too quickly with this, until I or someone else gets some time to 
do some work and produce the unit tests.

On 11/10/11 08:07, Paolo Castagna wrote:
> Hi,
> first of all, thank you for your patch.
> I had a quick look, but I did not try to apply it (yet).
>
> May I ask how you created your patch?
>
> We added a section on the Getting Involved page on the Jena website:
>
>   "Patches should be attached to issues in Jira (click on
>    More Actions>  Attach Files). To create a patch you can simply
>    use the command:
>
>      svn diff>  JENA-XYZ.patch
>
>    Please, inspect your patch and make sure it includes all (and only)
>    the relevant changes for a single issue. Don't forget tests! If you
>    want to test if a patch applies cleanly you can use:
>
>      patch -p0<  JENA-XYZ.patch
>
>    If you use Eclipse: right click on the project name in Package Explorer,
>    select Team>  Create Patch or Team>  Apply Patch."
>
> - http://jena.staging.apache.org/jena/getting_involved/#submit_your_patches
>
>
> It really helps if a patch contains only the lines you added|removed
> and it applies cleanly. It saves a lot of time and speed-up reviewing
> it.
>
> Your patch may be perfectly fine, but I wanted to take the opportunity
> to send the message across.
>
> I am really curious to run a few benchmarks when it's done to compare
> MonetDB with a more traditional SQL system.
>
> By the way, about benchmarks Andy is (secretely) working on this:
> https://svn.apache.org/repos/asf/incubator/jena/Experimental/JenaPerf/trunk/
> I have not time to try it yet, but it seems very interesting. :-)
>
> Thank you again for the new interesting feature.
>
> Paolo
>
>
> nat lu wrote:
>> Added, with patch file, at
>>
>> https://issues.apache.org/jira/browse/JENA-134
>>
>>
>> I have made no more progress on testing it out other than sdbconfig so
>> far, hope too soon.
>>
>>
>> On 09/09/11 16:43, Paolo Castagna wrote:
>>> Hi,
>>> why don't you open a new JIRA issue (as a New Feature) for this?
>>> https://issues.apache.org/jira/browse/JENA
>>>
>>> You can then attach a patch to it. This way others can look at what
>>> you have done so far (and maybe help you out).
>>>
>>> Thanks for your help,
>>> Paolo
>>>
>>> nat lu wrote:
>>>> I made a start, and tried to use one of the existing flavours, but ended
>>>> up creating one for MonetDB - combination of derby and DB2. It doesnt
>>>> like longs or unbounded varchars.
>>>>
>>>> So, I got as far as getting SDBConfig to complete, but havent done an
>>>> sdbload yet
>>>>
>>>>
>>>> On 09/09/11 10:37, Andy Seaborne wrote:
>>>>> On 04/09/11 13:03, nat lu wrote:
>>>>>> I'm going to give it a go sometime soon and report back on my
>>>>>> non-scientific findings. Your point about the small number of
>>>>>> columns is
>>>>>> well made, but the research paper cited earlier also mentions this and
>>>>>> reports that because of column store optimisations even when they
>>>>>> vertically partitioned their data rather than using a property-table
>>>>>> approach they still saw good improvement. However, again, I'm no
>>>>>> column
>>>>>> store expert so perhaps I'm missing some point here :-). Anyway,
>>>>>> time to
>>>>>> "suck it and see@, all in the name of progress of course.
>>>>>>
>>>>>> On 03/09/11 16:29, David Jordan wrote:
>>>>>>> I have not used a column-oriented database, but I am somewhat
>>>>>>> familiar
>>>>>>> with them. My understanding of them is that the storage is
>>>>>>> partitioned
>>>>>>> on a column basis, such that there is no physical clustering together
>>>>>>> of all the columns for a given row. An advantage of this would be in
>>>>>>> the case where you have tables with many columns, but the particular
>>>>>>> application only needs a small subset of columns.
>>>>>>>
>>>>>>> With the SDB representation of triples (3 columns) and quads (4
>>>>>>> columns), and access typically based on having a specific value for
>>>>>>> one or two of the columns, I am not so sure that a column-based
>>>>>>> approach would offer any advantage.
>>>>>>>
>>>>>>> But again, I am no expert on these types of databases.
>>>>>>>
>>>>>>> These discussions about alternative datastore representations RDF/OWL
>>>>>>> data are very useful, to gain better understanding of which data
>>>>>>> architectures yield the best implementation approach for
>>>>>>> high-performance.
>>>>>>>
>>>>>>> p.s. I Monet provides support for JDBC, I would not think much effort
>>>>>>> is needed to support in with SDB.
>>>>> Shouldn't be too hard :-)  SDB targets SQL-92 and there are a few
>>>>> extension points to cope with the vagaries of different SQL engines.
>>>>> It's one of the reasons there are ~10 small files to write, to capture
>>>>> the uniqueness of each SQL syntax.
>>>>>
>>>>>       Andy

Re: Adding support for MonetDB

[natlu 2809 <na...@gmail.com>]

I used eclipse to generate against svn. Will have to inspect in more detail
later...



sent from my htc, forgive typos please !
On Oct 11, 2011 8:08 AM, "Paolo Castagna" <ca...@googlemail.com>
wrote:

> Hi,
> first of all, thank you for your patch.
> I had a quick look, but I did not try to apply it (yet).
>
> May I ask how you created your patch?
>
> We added a section on the Getting Involved page on the Jena website:
>
>  "Patches should be attached to issues in Jira (click on
>  More Actions > Attach Files). To create a patch you can simply
>  use the command:
>
>    svn diff > JENA-XYZ.patch
>
>  Please, inspect your patch and make sure it includes all (and only)
>  the relevant changes for a single issue. Don't forget tests! If you
>  want to test if a patch applies cleanly you can use:
>
>    patch -p0 < JENA-XYZ.patch
>
>  If you use Eclipse: right click on the project name in Package Explorer,
>  select Team > Create Patch or Team > Apply Patch."
>
> -
> http://jena.staging.apache.org/jena/getting_involved/#submit_your_patches
>
>
> It really helps if a patch contains only the lines you added|removed
> and it applies cleanly. It saves a lot of time and speed-up reviewing
> it.
>
> Your patch may be perfectly fine, but I wanted to take the opportunity
> to send the message across.
>
> I am really curious to run a few benchmarks when it's done to compare
> MonetDB with a more traditional SQL system.
>
> By the way, about benchmarks Andy is (secretely) working on this:
>
> https://svn.apache.org/repos/asf/incubator/jena/Experimental/JenaPerf/trunk/
> I have not time to try it yet, but it seems very interesting. :-)
>
> Thank you again for the new interesting feature.
>
> Paolo
>
>
> nat lu wrote:
> > Added, with patch file, at
> >
> > https://issues.apache.org/jira/browse/JENA-134
> >
> >
> > I have made no more progress on testing it out other than sdbconfig so
> > far, hope too soon.
> >
> >
> > On 09/09/11 16:43, Paolo Castagna wrote:
> >> Hi,
> >> why don't you open a new JIRA issue (as a New Feature) for this?
> >> https://issues.apache.org/jira/browse/JENA
> >>
> >> You can then attach a patch to it. This way others can look at what
> >> you have done so far (and maybe help you out).
> >>
> >> Thanks for your help,
> >> Paolo
> >>
> >> nat lu wrote:
> >>> I made a start, and tried to use one of the existing flavours, but
> ended
> >>> up creating one for MonetDB - combination of derby and DB2. It doesnt
> >>> like longs or unbounded varchars.
> >>>
> >>> So, I got as far as getting SDBConfig to complete, but havent done an
> >>> sdbload yet
> >>>
> >>>
> >>> On 09/09/11 10:37, Andy Seaborne wrote:
> >>>>
> >>>> On 04/09/11 13:03, nat lu wrote:
> >>>>> I'm going to give it a go sometime soon and report back on my
> >>>>> non-scientific findings. Your point about the small number of
> >>>>> columns is
> >>>>> well made, but the research paper cited earlier also mentions this
> and
> >>>>> reports that because of column store optimisations even when they
> >>>>> vertically partitioned their data rather than using a property-table
> >>>>> approach they still saw good improvement. However, again, I'm no
> >>>>> column
> >>>>> store expert so perhaps I'm missing some point here :-). Anyway,
> >>>>> time to
> >>>>> "suck it and see@, all in the name of progress of course.
> >>>>>
> >>>>> On 03/09/11 16:29, David Jordan wrote:
> >>>>>> I have not used a column-oriented database, but I am somewhat
> >>>>>> familiar
> >>>>>> with them. My understanding of them is that the storage is
> >>>>>> partitioned
> >>>>>> on a column basis, such that there is no physical clustering
> together
> >>>>>> of all the columns for a given row. An advantage of this would be in
> >>>>>> the case where you have tables with many columns, but the particular
> >>>>>> application only needs a small subset of columns.
> >>>>>>
> >>>>>> With the SDB representation of triples (3 columns) and quads (4
> >>>>>> columns), and access typically based on having a specific value for
> >>>>>> one or two of the columns, I am not so sure that a column-based
> >>>>>> approach would offer any advantage.
> >>>>>>
> >>>>>> But again, I am no expert on these types of databases.
> >>>>>>
> >>>>>> These discussions about alternative datastore representations
> RDF/OWL
> >>>>>> data are very useful, to gain better understanding of which data
> >>>>>> architectures yield the best implementation approach for
> >>>>>> high-performance.
> >>>>>>
> >>>>>> p.s. I Monet provides support for JDBC, I would not think much
> effort
> >>>>>> is needed to support in with SDB.
> >>>> Shouldn't be too hard :-)  SDB targets SQL-92 and there are a few
> >>>> extension points to cope with the vagaries of different SQL engines.
> >>>> It's one of the reasons there are ~10 small files to write, to capture
> >>>> the uniqueness of each SQL syntax.
> >>>>
> >>>>      Andy
> >
>

Re: Adding support for MonetDB

[Andy Seaborne <an...@apache.org>]

> By the way, about benchmarks Andy is (secretely) working on this:
> https://svn.apache.org/repos/asf/incubator/jena/Experimental/JenaPerf/trunk/
> I have not time to try it yet, but it seems very interesting. :-)

Which is unfinished, unstable, and unreliable and does not work with SDB 
yet.

Do not expect it to work.

	Andy

Re: Adding support for MonetDB

[Paolo Castagna <ca...@googlemail.com>]

Hi,
first of all, thank you for your patch.
I had a quick look, but I did not try to apply it (yet).

May I ask how you created your patch?

We added a section on the Getting Involved page on the Jena website:

 "Patches should be attached to issues in Jira (click on
  More Actions > Attach Files). To create a patch you can simply
  use the command:

    svn diff > JENA-XYZ.patch

  Please, inspect your patch and make sure it includes all (and only)
  the relevant changes for a single issue. Don't forget tests! If you
  want to test if a patch applies cleanly you can use:

    patch -p0 < JENA-XYZ.patch

  If you use Eclipse: right click on the project name in Package Explorer,
  select Team > Create Patch or Team > Apply Patch."

- http://jena.staging.apache.org/jena/getting_involved/#submit_your_patches


It really helps if a patch contains only the lines you added|removed
and it applies cleanly. It saves a lot of time and speed-up reviewing
it.

Your patch may be perfectly fine, but I wanted to take the opportunity
to send the message across.

I am really curious to run a few benchmarks when it's done to compare
MonetDB with a more traditional SQL system.

By the way, about benchmarks Andy is (secretely) working on this:
https://svn.apache.org/repos/asf/incubator/jena/Experimental/JenaPerf/trunk/
I have not time to try it yet, but it seems very interesting. :-)

Thank you again for the new interesting feature.

Paolo


nat lu wrote:
> Added, with patch file, at
> 
> https://issues.apache.org/jira/browse/JENA-134
> 
> 
> I have made no more progress on testing it out other than sdbconfig so
> far, hope too soon.
> 
> 
> On 09/09/11 16:43, Paolo Castagna wrote:
>> Hi,
>> why don't you open a new JIRA issue (as a New Feature) for this?
>> https://issues.apache.org/jira/browse/JENA
>>
>> You can then attach a patch to it. This way others can look at what
>> you have done so far (and maybe help you out).
>>
>> Thanks for your help,
>> Paolo
>>
>> nat lu wrote:
>>> I made a start, and tried to use one of the existing flavours, but ended
>>> up creating one for MonetDB - combination of derby and DB2. It doesnt
>>> like longs or unbounded varchars.
>>>
>>> So, I got as far as getting SDBConfig to complete, but havent done an
>>> sdbload yet
>>>
>>>
>>> On 09/09/11 10:37, Andy Seaborne wrote:
>>>>
>>>> On 04/09/11 13:03, nat lu wrote:
>>>>> I'm going to give it a go sometime soon and report back on my
>>>>> non-scientific findings. Your point about the small number of
>>>>> columns is
>>>>> well made, but the research paper cited earlier also mentions this and
>>>>> reports that because of column store optimisations even when they
>>>>> vertically partitioned their data rather than using a property-table
>>>>> approach they still saw good improvement. However, again, I'm no
>>>>> column
>>>>> store expert so perhaps I'm missing some point here :-). Anyway,
>>>>> time to
>>>>> "suck it and see@, all in the name of progress of course.
>>>>>
>>>>> On 03/09/11 16:29, David Jordan wrote:
>>>>>> I have not used a column-oriented database, but I am somewhat
>>>>>> familiar
>>>>>> with them. My understanding of them is that the storage is
>>>>>> partitioned
>>>>>> on a column basis, such that there is no physical clustering together
>>>>>> of all the columns for a given row. An advantage of this would be in
>>>>>> the case where you have tables with many columns, but the particular
>>>>>> application only needs a small subset of columns.
>>>>>>
>>>>>> With the SDB representation of triples (3 columns) and quads (4
>>>>>> columns), and access typically based on having a specific value for
>>>>>> one or two of the columns, I am not so sure that a column-based
>>>>>> approach would offer any advantage.
>>>>>>
>>>>>> But again, I am no expert on these types of databases.
>>>>>>
>>>>>> These discussions about alternative datastore representations RDF/OWL
>>>>>> data are very useful, to gain better understanding of which data
>>>>>> architectures yield the best implementation approach for
>>>>>> high-performance.
>>>>>>
>>>>>> p.s. I Monet provides support for JDBC, I would not think much effort
>>>>>> is needed to support in with SDB.
>>>> Shouldn't be too hard :-)  SDB targets SQL-92 and there are a few
>>>> extension points to cope with the vagaries of different SQL engines.
>>>> It's one of the reasons there are ~10 small files to write, to capture
>>>> the uniqueness of each SQL syntax.
>>>>
>>>>      Andy
> 

Re: Adding support for MonetDB

[nat lu <na...@gmail.com>]

Added, with patch file, at

https://issues.apache.org/jira/browse/JENA-134


I have made no more progress on testing it out other than sdbconfig so 
far, hope too soon.


On 09/09/11 16:43, Paolo Castagna wrote:
> Hi,
> why don't you open a new JIRA issue (as a New Feature) for this?
> https://issues.apache.org/jira/browse/JENA
>
> You can then attach a patch to it. This way others can look at what you have done so far (and maybe help you out).
>
> Thanks for your help,
> Paolo
>
> nat lu wrote:
>> I made a start, and tried to use one of the existing flavours, but ended
>> up creating one for MonetDB - combination of derby and DB2. It doesnt
>> like longs or unbounded varchars.
>>
>> So, I got as far as getting SDBConfig to complete, but havent done an
>> sdbload yet
>>
>>
>> On 09/09/11 10:37, Andy Seaborne wrote:
>>>
>>> On 04/09/11 13:03, nat lu wrote:
>>>> I'm going to give it a go sometime soon and report back on my
>>>> non-scientific findings. Your point about the small number of columns is
>>>> well made, but the research paper cited earlier also mentions this and
>>>> reports that because of column store optimisations even when they
>>>> vertically partitioned their data rather than using a property-table
>>>> approach they still saw good improvement. However, again, I'm no column
>>>> store expert so perhaps I'm missing some point here :-). Anyway, time to
>>>> "suck it and see@, all in the name of progress of course.
>>>>
>>>> On 03/09/11 16:29, David Jordan wrote:
>>>>> I have not used a column-oriented database, but I am somewhat familiar
>>>>> with them. My understanding of them is that the storage is partitioned
>>>>> on a column basis, such that there is no physical clustering together
>>>>> of all the columns for a given row. An advantage of this would be in
>>>>> the case where you have tables with many columns, but the particular
>>>>> application only needs a small subset of columns.
>>>>>
>>>>> With the SDB representation of triples (3 columns) and quads (4
>>>>> columns), and access typically based on having a specific value for
>>>>> one or two of the columns, I am not so sure that a column-based
>>>>> approach would offer any advantage.
>>>>>
>>>>> But again, I am no expert on these types of databases.
>>>>>
>>>>> These discussions about alternative datastore representations RDF/OWL
>>>>> data are very useful, to gain better understanding of which data
>>>>> architectures yield the best implementation approach for
>>>>> high-performance.
>>>>>
>>>>> p.s. I Monet provides support for JDBC, I would not think much effort
>>>>> is needed to support in with SDB.
>>> Shouldn't be too hard :-)  SDB targets SQL-92 and there are a few
>>> extension points to cope with the vagaries of different SQL engines.
>>> It's one of the reasons there are ~10 small files to write, to capture
>>> the uniqueness of each SQL syntax.
>>>
>>>      Andy

Re: Adding support for MonetDB

[Paolo Castagna <ca...@googlemail.com>]

Hi,
why don't you open a new JIRA issue (as a New Feature) for this?
https://issues.apache.org/jira/browse/JENA

You can then attach a patch to it. This way others can look at what you have done so far (and maybe help you out).

Thanks for your help,
Paolo

nat lu wrote:
> 
> I made a start, and tried to use one of the existing flavours, but ended
> up creating one for MonetDB - combination of derby and DB2. It doesnt
> like longs or unbounded varchars.
> 
> So, I got as far as getting SDBConfig to complete, but havent done an
> sdbload yet
> 
> 
> On 09/09/11 10:37, Andy Seaborne wrote:
>>
>>
>> On 04/09/11 13:03, nat lu wrote:
>>>
>>> I'm going to give it a go sometime soon and report back on my
>>> non-scientific findings. Your point about the small number of columns is
>>> well made, but the research paper cited earlier also mentions this and
>>> reports that because of column store optimisations even when they
>>> vertically partitioned their data rather than using a property-table
>>> approach they still saw good improvement. However, again, I'm no column
>>> store expert so perhaps I'm missing some point here :-). Anyway, time to
>>> "suck it and see@, all in the name of progress of course.
>>>
>>> On 03/09/11 16:29, David Jordan wrote:
>>>> I have not used a column-oriented database, but I am somewhat familiar
>>>> with them. My understanding of them is that the storage is partitioned
>>>> on a column basis, such that there is no physical clustering together
>>>> of all the columns for a given row. An advantage of this would be in
>>>> the case where you have tables with many columns, but the particular
>>>> application only needs a small subset of columns.
>>>>
>>>> With the SDB representation of triples (3 columns) and quads (4
>>>> columns), and access typically based on having a specific value for
>>>> one or two of the columns, I am not so sure that a column-based
>>>> approach would offer any advantage.
>>>>
>>>> But again, I am no expert on these types of databases.
>>>>
>>>> These discussions about alternative datastore representations RDF/OWL
>>>> data are very useful, to gain better understanding of which data
>>>> architectures yield the best implementation approach for
>>>> high-performance.
>>>>
>>>> p.s. I Monet provides support for JDBC, I would not think much effort
>>>> is needed to support in with SDB.
>>
>> Shouldn't be too hard :-)  SDB targets SQL-92 and there are a few
>> extension points to cope with the vagaries of different SQL engines.
>> It's one of the reasons there are ~10 small files to write, to capture
>> the uniqueness of each SQL syntax.
>>
>>     Andy
> 

Re: Adding support for MonetDB

[nat lu <na...@gmail.com>]

I made a start, and tried to use one of the existing flavours, but ended 
up creating one for MonetDB - combination of derby and DB2. It doesnt 
like longs or unbounded varchars.

So, I got as far as getting SDBConfig to complete, but havent done an 
sdbload yet


On 09/09/11 10:37, Andy Seaborne wrote:
>
>
> On 04/09/11 13:03, nat lu wrote:
>>
>> I'm going to give it a go sometime soon and report back on my
>> non-scientific findings. Your point about the small number of columns is
>> well made, but the research paper cited earlier also mentions this and
>> reports that because of column store optimisations even when they
>> vertically partitioned their data rather than using a property-table
>> approach they still saw good improvement. However, again, I'm no column
>> store expert so perhaps I'm missing some point here :-). Anyway, time to
>> "suck it and see@, all in the name of progress of course.
>>
>> On 03/09/11 16:29, David Jordan wrote:
>>> I have not used a column-oriented database, but I am somewhat familiar
>>> with them. My understanding of them is that the storage is partitioned
>>> on a column basis, such that there is no physical clustering together
>>> of all the columns for a given row. An advantage of this would be in
>>> the case where you have tables with many columns, but the particular
>>> application only needs a small subset of columns.
>>>
>>> With the SDB representation of triples (3 columns) and quads (4
>>> columns), and access typically based on having a specific value for
>>> one or two of the columns, I am not so sure that a column-based
>>> approach would offer any advantage.
>>>
>>> But again, I am no expert on these types of databases.
>>>
>>> These discussions about alternative datastore representations RDF/OWL
>>> data are very useful, to gain better understanding of which data
>>> architectures yield the best implementation approach for
>>> high-performance.
>>>
>>> p.s. I Monet provides support for JDBC, I would not think much effort
>>> is needed to support in with SDB.
>
> Shouldn't be too hard :-)  SDB targets SQL-92 and there are a few 
> extension points to cope with the vagaries of different SQL engines. 
> It's one of the reasons there are ~10 small files to write, to capture 
> the uniqueness of each SQL syntax.
>
>     Andy

Re: Adding support for MonetDB

[Andy Seaborne <an...@epimorphics.com>]

On 04/09/11 13:03, nat lu wrote:
>
> I'm going to give it a go sometime soon and report back on my
> non-scientific findings. Your point about the small number of columns is
> well made, but the research paper cited earlier also mentions this and
> reports that because of column store optimisations even when they
> vertically partitioned their data rather than using a property-table
> approach they still saw good improvement. However, again, I'm no column
> store expert so perhaps I'm missing some point here :-). Anyway, time to
> "suck it and see@, all in the name of progress of course.
>
> On 03/09/11 16:29, David Jordan wrote:
>> I have not used a column-oriented database, but I am somewhat familiar
>> with them. My understanding of them is that the storage is partitioned
>> on a column basis, such that there is no physical clustering together
>> of all the columns for a given row. An advantage of this would be in
>> the case where you have tables with many columns, but the particular
>> application only needs a small subset of columns.
>>
>> With the SDB representation of triples (3 columns) and quads (4
>> columns), and access typically based on having a specific value for
>> one or two of the columns, I am not so sure that a column-based
>> approach would offer any advantage.
>>
>> But again, I am no expert on these types of databases.
>>
>> These discussions about alternative datastore representations RDF/OWL
>> data are very useful, to gain better understanding of which data
>> architectures yield the best implementation approach for
>> high-performance.
>>
>> p.s. I Monet provides support for JDBC, I would not think much effort
>> is needed to support in with SDB.

Shouldn't be too hard :-)  SDB targets SQL-92 and there are a few 
extension points to cope with the vagaries of different SQL engines. 
It's one of the reasons there are ~10 small files to write, to capture 
the uniqueness of each SQL syntax.

	Andy

Re: Adding support for MonetDB

[nat lu <na...@gmail.com>]

I'm going to give it a go sometime soon and report back on my 
non-scientific findings. Your point about the small number of columns is 
well made, but the research paper cited earlier also mentions this and 
reports that because of column store optimisations even when they 
vertically partitioned their data rather than using a property-table 
approach they still saw good improvement. However, again, I'm no column 
store expert so perhaps I'm missing some point here :-). Anyway, time to 
"suck it and see@, all in the name of progress of course.

On 03/09/11 16:29, David Jordan wrote:
> I have not used a column-oriented database, but I am somewhat familiar with them. My understanding of them is that the storage is partitioned on a column basis, such that there is no physical clustering together of all the columns for a given row. An advantage of this would be in the case where you have tables with many columns, but the particular application only needs a small subset of columns.
>
> With the SDB representation of triples (3 columns) and quads (4 columns), and access typically based on having a specific value for one or two of the columns, I am not so sure that a column-based approach would offer any advantage.
>
> But again, I am no expert on these types of databases.
>
> These discussions about alternative datastore representations RDF/OWL data are very useful, to gain better understanding of which data architectures yield the best implementation approach for high-performance.
>
> p.s. I Monet provides support for JDBC, I would not think much effort is needed to support in with SDB.
>
> -----Original Message-----
> From: nat lu [mailto:natlu2809@gmail.com]
> Sent: Saturday, September 03, 2011 6:59 AM
> To: jena-users@incubator.apache.org
> Subject: Re: Adding support for MonetDB
>
> 1) Has anyone tried using MonetDB just as a jdbc-SDB source ? I suppose the DDL jena uses to create the normalized schema may need adjusting to suit MonetDBs SQL flavour, but it should work, with some mileage, to try it out and do some gap analysis - right ?
>
> 2) It also seems reasonable that a SPARQL front end (top of its 3 layer
> stack) could be created in MAL to augment the existing SQL and Xquery modules. I've also seen some talk of an RDF module in newsgroups that is under development/experimental at this stage.
>
> I'm interested to see how an SDB column store will perform with my small dataset compared to and RDBMS backed SDB instance.
>
>
> On 02/09/11 10:11, Paolo Castagna wrote:
>> Andy Seaborne wrote:
>>> It seems to use a non-normalized table design (as did the CStore paper)
>>> and rely on indexing.  It would be interesting to see how that compares
>>> with a normalized design, which is what most RDF stores use.  When
>>> normalized, the joins for patterns are on fixed size numbers, not string
>>> comparisons, and don't use secondary indexes (and at least one uses
>>> bitmap indexes).
>>>
>>> SDB is built around a normalized design, and portable across various
>>> different SQL DBs.  The main triple table (or quad table) is 3 columns
>>> of 4 or 8 bytes (depending on hash vs index allocation of internal ids).
>>> That means the main table is a thin, long table.
>>>
>>> If you want a deep integration of SPARQL and MonetDB, the first thing to
>>> consider, if pure speed is the objective, is what schema design is best
>>> for MonetDB.
>>>
>>> The reported tests are on a dataset of 50 million triples which is
>>                                                                         ^
>>                                                                        not
>>
>>> particularly big - it means RAM caching is going to play a significant
>>> role nowadays (they only used a one-CPU, 4G machine).
>>>       Andy
>>>
>>>
>>> On 01/09/11 19:49, nat lu wrote:
>>>> Speed I think ! At least in some use cases. (But Im no MonetDB expert)
>>>> Heres an interesting article that tries a few of them out with MonetDB
>>>> and CStore.
>>>>
>>>> http://oai.cwi.nl/oai/asset/13806/13806B.pdf
>>>>
>>>>
>>>> Whats missing I believe is a the SPARQL endpoint integration.
>>>>
>>>> On 01/09/11 08:36, Andy Seaborne wrote:
>>>>> Tobias,
>>>>>
>>>>> To turn the question round - what unique features of MonetDB could be
>>>>> exposed through SDB to yield a better RDf store?
>>>>>
>>>>> The current design covers the current set of databases supported but
>>>>> it's not fixed - maybe there is something especially useful in MonetDB
>>>>> and maybe it needs an extension to the design. The design is based on
>>>>> templating via all those classes (the instance for each database is a
>>>>> small class). The SQL generator usually needs some per-DB work because
>>>>> SQL databases aren't exactly very "standard".
>>>>>
>>>>> Andy
>>>>>
>>>>> On 25/08/11 21:28, Paolo Castagna wrote:
>>>>>> Hi Tobias,
>>>>>> first of all, welcome on jena-users mailing list.
>>>>>>
>>>>>> Tobias Willig wrote:
>>>>>>> Hi everyone,
>>>>>>>
>>>>>>> I like to add support for MonetDB in SDB and I have two questions
>>>>>>> concerning
>>>>>>> this project:
>>>>>>>
>>>>>>> 1. How much effort it takes to add support for a new database type?
>>>>>> It requires some effort. You need to add new Java classes and the
>>>>>> necessary tests.
>>>>>>
>>>>>> Have you checked out the SDB sources yet?
>>>>>>
>>>>>> If not:
>>>>>>
>>>>>> svn co
>>>>>> https://svn.apache.org/repos/asf/incubator/jena/Jena2/SDB/trunk/ SDB
>>>>>> cd SDB
>>>>>> mvn test
>>>>>>
>>>>>> You can use Eclipse and search for "Derby" which is one of the DBMS
>>>>>> supported
>>>>>> by SDB. This way you'll find the list of Java classes in SDB to
>>>>>> support Derby.
>>>>>> Then, you can read and study those classes. While you do that, you'll
>>>>>> learn the
>>>>>> design of SDB and you will get an idea on what it is required to add
>>>>>> MonetDB.
>>>>>>
>>>>>>> 2. Are there predefined extension points that allow adding a new
>>>>>>> database
>>>>>>> type easily?
>>>>>> Yes, there are. Look at the super classes and interfaces from the
>>>>>> list of
>>>>>> classes above (i.e. searching for "Derby").
>>>>>>
>>>>>> There isn't an "how to add a new database to SDB" guide, however make
>>>>>> sure
>>>>>> you read the general SDB documentation (it does not hurt).
>>>>>>
>>>>>> Also, if you want to contribute an "how to add a new database to SDB"
>>>>>> you
>>>>>> are more than welcome to do so.
>>>>>>
>>>>>>> If so could you give me the name of some classes and config files,
>>>>>>> which
>>>>>>> are important to accomplish that task?
>>>>>> You can start from:
>>>>>>
>>>>>> GenerateSQLDerby -- extends -->   GenrateSQL
>>>>>> FormatterSimpleDerby -- extends -->   FormatterSimple
>>>>>> StoreSimpleDerby -- extends -->   StoreBase1
>>>>>> FmtLayout2HashDerby -- extends -->   FmtLayout2
>>>>>> StoreTriplesNodesHashDerby -- extends -->   StoreBaseHash
>>>>>> TupleLoaderHashDerby -- extends -->   TupleLoaderHashBase
>>>>>> FmtLayout2IndexDerby -- extends -->   FmtLayout2HashDerby
>>>>>> StoreTriplesNodesIndexDerby -- extends -->   StoreBaseIndex
>>>>>> TupleLoaderIndexDerby -- extends -->   TupleLoaderIndexBase
>>>>>>
>>>>>> Also look at:
>>>>>>
>>>>>> - JDBC.java
>>>>>> - DatabaseType.java
>>>>>> - StoreFactory.java
>>>>>> - SDB.java
>>>>>>
>>>>>> And the existing tests.
>>>>>>
>>>>>> May I ask you what motivates you in adding MonetDB?
>>>>>>
>>>>>> I've never used it myself and, indeed, I use TDB instead of SDB.
>>>>>> Transactions are coming, hopefully in the 0.9.0 release of TDB.
>>>>>>
>>>>>> Last but not least, it's not only about the code. You should be willing
>>>>>> to support the users of your code too. Once you add support for
>>>>>> MonetDB,
>>>>>> people will start using it and, as they use your code, they'll find
>>>>>> bugs
>>>>>> and they'll ask you for more features, eventually. You should be
>>>>>> willing
>>>>>> to put some effort in fixing the bugs, at least... and you can always
>>>>>> say
>>>>>> "no" politely to new features. Until, someone else, who really needs
>>>>>> the
>>>>>> new feature and he/she is willing to put some effort, will take over
>>>>>> and
>>>>>> push the software a step further.
>>>>>>
>>>>>> Once you start, you might have more specific questions on SDB design.
>>>>>> You can post your questions here or on jena-dev@incubator.apache.org
>>>>>> mailing list. The more you demonstrate you put some effort the more
>>>>>> likely you'll receive helpful answers back from the SDB developers.
>>>>>> If you don't put enough effort and expect others will do it for you,
>>>>>> I am afraid, you risk to have not much back.
>>>>>>
>>>>>> To conclude, it you are motivated and you think you'll have fun doing
>>>>>> it (and you need it for your work): go ahead, it's not a terribly huge
>>>>>> task and it could be a nice contribution (in particular for all the
>>>>>> MonetDB users out there).
>>>>>>
>>>>>> Paolo
>>>>>>
>>>>>>> Thanks in advance!
>>>>>>>
>>>>>>> Best Regards
>>>>>>> Tobias Willig
>>>>>>>
>
>

RE: Adding support for MonetDB

[David Jordan <Da...@sas.com>]

I have not used a column-oriented database, but I am somewhat familiar with them. My understanding of them is that the storage is partitioned on a column basis, such that there is no physical clustering together of all the columns for a given row. An advantage of this would be in the case where you have tables with many columns, but the particular application only needs a small subset of columns.

With the SDB representation of triples (3 columns) and quads (4 columns), and access typically based on having a specific value for one or two of the columns, I am not so sure that a column-based approach would offer any advantage.

But again, I am no expert on these types of databases.

These discussions about alternative datastore representations RDF/OWL data are very useful, to gain better understanding of which data architectures yield the best implementation approach for high-performance.

p.s. I Monet provides support for JDBC, I would not think much effort is needed to support in with SDB.

-----Original Message-----
From: nat lu [mailto:natlu2809@gmail.com] 
Sent: Saturday, September 03, 2011 6:59 AM
To: jena-users@incubator.apache.org
Subject: Re: Adding support for MonetDB

1) Has anyone tried using MonetDB just as a jdbc-SDB source ? I suppose the DDL jena uses to create the normalized schema may need adjusting to suit MonetDBs SQL flavour, but it should work, with some mileage, to try it out and do some gap analysis - right ?

2) It also seems reasonable that a SPARQL front end (top of its 3 layer
stack) could be created in MAL to augment the existing SQL and Xquery modules. I've also seen some talk of an RDF module in newsgroups that is under development/experimental at this stage.

I'm interested to see how an SDB column store will perform with my small dataset compared to and RDBMS backed SDB instance.


On 02/09/11 10:11, Paolo Castagna wrote:
>
> Andy Seaborne wrote:
>> It seems to use a non-normalized table design (as did the CStore paper)
>> and rely on indexing.  It would be interesting to see how that compares
>> with a normalized design, which is what most RDF stores use.  When
>> normalized, the joins for patterns are on fixed size numbers, not string
>> comparisons, and don't use secondary indexes (and at least one uses
>> bitmap indexes).
>>
>> SDB is built around a normalized design, and portable across various
>> different SQL DBs.  The main triple table (or quad table) is 3 columns
>> of 4 or 8 bytes (depending on hash vs index allocation of internal ids).
>> That means the main table is a thin, long table.
>>
>> If you want a deep integration of SPARQL and MonetDB, the first thing to
>> consider, if pure speed is the objective, is what schema design is best
>> for MonetDB.
>>
>> The reported tests are on a dataset of 50 million triples which is
>                                                                        ^
>                                                                       not
>
>> particularly big - it means RAM caching is going to play a significant
>> role nowadays (they only used a one-CPU, 4G machine).
>>      Andy
>>
>>
>> On 01/09/11 19:49, nat lu wrote:
>>> Speed I think ! At least in some use cases. (But Im no MonetDB expert)
>>> Heres an interesting article that tries a few of them out with MonetDB
>>> and CStore.
>>>
>>> http://oai.cwi.nl/oai/asset/13806/13806B.pdf
>>>
>>>
>>> Whats missing I believe is a the SPARQL endpoint integration.
>>>
>>> On 01/09/11 08:36, Andy Seaborne wrote:
>>>> Tobias,
>>>>
>>>> To turn the question round - what unique features of MonetDB could be
>>>> exposed through SDB to yield a better RDf store?
>>>>
>>>> The current design covers the current set of databases supported but
>>>> it's not fixed - maybe there is something especially useful in MonetDB
>>>> and maybe it needs an extension to the design. The design is based on
>>>> templating via all those classes (the instance for each database is a
>>>> small class). The SQL generator usually needs some per-DB work because
>>>> SQL databases aren't exactly very "standard".
>>>>
>>>> Andy
>>>>
>>>> On 25/08/11 21:28, Paolo Castagna wrote:
>>>>> Hi Tobias,
>>>>> first of all, welcome on jena-users mailing list.
>>>>>
>>>>> Tobias Willig wrote:
>>>>>> Hi everyone,
>>>>>>
>>>>>> I like to add support for MonetDB in SDB and I have two questions
>>>>>> concerning
>>>>>> this project:
>>>>>>
>>>>>> 1. How much effort it takes to add support for a new database type?
>>>>> It requires some effort. You need to add new Java classes and the
>>>>> necessary tests.
>>>>>
>>>>> Have you checked out the SDB sources yet?
>>>>>
>>>>> If not:
>>>>>
>>>>> svn co
>>>>> https://svn.apache.org/repos/asf/incubator/jena/Jena2/SDB/trunk/ SDB
>>>>> cd SDB
>>>>> mvn test
>>>>>
>>>>> You can use Eclipse and search for "Derby" which is one of the DBMS
>>>>> supported
>>>>> by SDB. This way you'll find the list of Java classes in SDB to
>>>>> support Derby.
>>>>> Then, you can read and study those classes. While you do that, you'll
>>>>> learn the
>>>>> design of SDB and you will get an idea on what it is required to add
>>>>> MonetDB.
>>>>>
>>>>>> 2. Are there predefined extension points that allow adding a new
>>>>>> database
>>>>>> type easily?
>>>>> Yes, there are. Look at the super classes and interfaces from the
>>>>> list of
>>>>> classes above (i.e. searching for "Derby").
>>>>>
>>>>> There isn't an "how to add a new database to SDB" guide, however make
>>>>> sure
>>>>> you read the general SDB documentation (it does not hurt).
>>>>>
>>>>> Also, if you want to contribute an "how to add a new database to SDB"
>>>>> you
>>>>> are more than welcome to do so.
>>>>>
>>>>>> If so could you give me the name of some classes and config files,
>>>>>> which
>>>>>> are important to accomplish that task?
>>>>> You can start from:
>>>>>
>>>>> GenerateSQLDerby -- extends -->  GenrateSQL
>>>>> FormatterSimpleDerby -- extends -->  FormatterSimple
>>>>> StoreSimpleDerby -- extends -->  StoreBase1
>>>>> FmtLayout2HashDerby -- extends -->  FmtLayout2
>>>>> StoreTriplesNodesHashDerby -- extends -->  StoreBaseHash
>>>>> TupleLoaderHashDerby -- extends -->  TupleLoaderHashBase
>>>>> FmtLayout2IndexDerby -- extends -->  FmtLayout2HashDerby
>>>>> StoreTriplesNodesIndexDerby -- extends -->  StoreBaseIndex
>>>>> TupleLoaderIndexDerby -- extends -->  TupleLoaderIndexBase
>>>>>
>>>>> Also look at:
>>>>>
>>>>> - JDBC.java
>>>>> - DatabaseType.java
>>>>> - StoreFactory.java
>>>>> - SDB.java
>>>>>
>>>>> And the existing tests.
>>>>>
>>>>> May I ask you what motivates you in adding MonetDB?
>>>>>
>>>>> I've never used it myself and, indeed, I use TDB instead of SDB.
>>>>> Transactions are coming, hopefully in the 0.9.0 release of TDB.
>>>>>
>>>>> Last but not least, it's not only about the code. You should be willing
>>>>> to support the users of your code too. Once you add support for
>>>>> MonetDB,
>>>>> people will start using it and, as they use your code, they'll find
>>>>> bugs
>>>>> and they'll ask you for more features, eventually. You should be
>>>>> willing
>>>>> to put some effort in fixing the bugs, at least... and you can always
>>>>> say
>>>>> "no" politely to new features. Until, someone else, who really needs
>>>>> the
>>>>> new feature and he/she is willing to put some effort, will take over
>>>>> and
>>>>> push the software a step further.
>>>>>
>>>>> Once you start, you might have more specific questions on SDB design.
>>>>> You can post your questions here or on jena-dev@incubator.apache.org
>>>>> mailing list. The more you demonstrate you put some effort the more
>>>>> likely you'll receive helpful answers back from the SDB developers.
>>>>> If you don't put enough effort and expect others will do it for you,
>>>>> I am afraid, you risk to have not much back.
>>>>>
>>>>> To conclude, it you are motivated and you think you'll have fun doing
>>>>> it (and you need it for your work): go ahead, it's not a terribly huge
>>>>> task and it could be a nice contribution (in particular for all the
>>>>> MonetDB users out there).
>>>>>
>>>>> Paolo
>>>>>
>>>>>> Thanks in advance!
>>>>>>
>>>>>> Best Regards
>>>>>> Tobias Willig
>>>>>>

Re: Adding support for MonetDB

[nat lu <na...@gmail.com>]

1) Has anyone tried using MonetDB just as a jdbc-SDB source ? I suppose 
the DDL jena uses to create the normalized schema may need adjusting to 
suit MonetDBs SQL flavour, but it should work, with some mileage, to try 
it out and do some gap analysis - right ?

2) It also seems reasonable that a SPARQL front end (top of its 3 layer 
stack) could be created in MAL to augment the existing SQL and Xquery 
modules. I've also seen some talk of an RDF module in newsgroups that is 
under development/experimental at this stage.

I'm interested to see how an SDB column store will perform with my small 
dataset compared to and RDBMS backed SDB instance.


On 02/09/11 10:11, Paolo Castagna wrote:
>
> Andy Seaborne wrote:
>> It seems to use a non-normalized table design (as did the CStore paper)
>> and rely on indexing.  It would be interesting to see how that compares
>> with a normalized design, which is what most RDF stores use.  When
>> normalized, the joins for patterns are on fixed size numbers, not string
>> comparisons, and don't use secondary indexes (and at least one uses
>> bitmap indexes).
>>
>> SDB is built around a normalized design, and portable across various
>> different SQL DBs.  The main triple table (or quad table) is 3 columns
>> of 4 or 8 bytes (depending on hash vs index allocation of internal ids).
>> That means the main table is a thin, long table.
>>
>> If you want a deep integration of SPARQL and MonetDB, the first thing to
>> consider, if pure speed is the objective, is what schema design is best
>> for MonetDB.
>>
>> The reported tests are on a dataset of 50 million triples which is
>                                                                        ^
>                                                                       not
>
>> particularly big - it means RAM caching is going to play a significant
>> role nowadays (they only used a one-CPU, 4G machine).
>>      Andy
>>
>>
>> On 01/09/11 19:49, nat lu wrote:
>>> Speed I think ! At least in some use cases. (But Im no MonetDB expert)
>>> Heres an interesting article that tries a few of them out with MonetDB
>>> and CStore.
>>>
>>> http://oai.cwi.nl/oai/asset/13806/13806B.pdf
>>>
>>>
>>> Whats missing I believe is a the SPARQL endpoint integration.
>>>
>>> On 01/09/11 08:36, Andy Seaborne wrote:
>>>> Tobias,
>>>>
>>>> To turn the question round - what unique features of MonetDB could be
>>>> exposed through SDB to yield a better RDf store?
>>>>
>>>> The current design covers the current set of databases supported but
>>>> it's not fixed - maybe there is something especially useful in MonetDB
>>>> and maybe it needs an extension to the design. The design is based on
>>>> templating via all those classes (the instance for each database is a
>>>> small class). The SQL generator usually needs some per-DB work because
>>>> SQL databases aren't exactly very "standard".
>>>>
>>>> Andy
>>>>
>>>> On 25/08/11 21:28, Paolo Castagna wrote:
>>>>> Hi Tobias,
>>>>> first of all, welcome on jena-users mailing list.
>>>>>
>>>>> Tobias Willig wrote:
>>>>>> Hi everyone,
>>>>>>
>>>>>> I like to add support for MonetDB in SDB and I have two questions
>>>>>> concerning
>>>>>> this project:
>>>>>>
>>>>>> 1. How much effort it takes to add support for a new database type?
>>>>> It requires some effort. You need to add new Java classes and the
>>>>> necessary tests.
>>>>>
>>>>> Have you checked out the SDB sources yet?
>>>>>
>>>>> If not:
>>>>>
>>>>> svn co
>>>>> https://svn.apache.org/repos/asf/incubator/jena/Jena2/SDB/trunk/ SDB
>>>>> cd SDB
>>>>> mvn test
>>>>>
>>>>> You can use Eclipse and search for "Derby" which is one of the DBMS
>>>>> supported
>>>>> by SDB. This way you'll find the list of Java classes in SDB to
>>>>> support Derby.
>>>>> Then, you can read and study those classes. While you do that, you'll
>>>>> learn the
>>>>> design of SDB and you will get an idea on what it is required to add
>>>>> MonetDB.
>>>>>
>>>>>> 2. Are there predefined extension points that allow adding a new
>>>>>> database
>>>>>> type easily?
>>>>> Yes, there are. Look at the super classes and interfaces from the
>>>>> list of
>>>>> classes above (i.e. searching for "Derby").
>>>>>
>>>>> There isn't an "how to add a new database to SDB" guide, however make
>>>>> sure
>>>>> you read the general SDB documentation (it does not hurt).
>>>>>
>>>>> Also, if you want to contribute an "how to add a new database to SDB"
>>>>> you
>>>>> are more than welcome to do so.
>>>>>
>>>>>> If so could you give me the name of some classes and config files,
>>>>>> which
>>>>>> are important to accomplish that task?
>>>>> You can start from:
>>>>>
>>>>> GenerateSQLDerby -- extends -->  GenrateSQL
>>>>> FormatterSimpleDerby -- extends -->  FormatterSimple
>>>>> StoreSimpleDerby -- extends -->  StoreBase1
>>>>> FmtLayout2HashDerby -- extends -->  FmtLayout2
>>>>> StoreTriplesNodesHashDerby -- extends -->  StoreBaseHash
>>>>> TupleLoaderHashDerby -- extends -->  TupleLoaderHashBase
>>>>> FmtLayout2IndexDerby -- extends -->  FmtLayout2HashDerby
>>>>> StoreTriplesNodesIndexDerby -- extends -->  StoreBaseIndex
>>>>> TupleLoaderIndexDerby -- extends -->  TupleLoaderIndexBase
>>>>>
>>>>> Also look at:
>>>>>
>>>>> - JDBC.java
>>>>> - DatabaseType.java
>>>>> - StoreFactory.java
>>>>> - SDB.java
>>>>>
>>>>> And the existing tests.
>>>>>
>>>>> May I ask you what motivates you in adding MonetDB?
>>>>>
>>>>> I've never used it myself and, indeed, I use TDB instead of SDB.
>>>>> Transactions are coming, hopefully in the 0.9.0 release of TDB.
>>>>>
>>>>> Last but not least, it's not only about the code. You should be willing
>>>>> to support the users of your code too. Once you add support for
>>>>> MonetDB,
>>>>> people will start using it and, as they use your code, they'll find
>>>>> bugs
>>>>> and they'll ask you for more features, eventually. You should be
>>>>> willing
>>>>> to put some effort in fixing the bugs, at least... and you can always
>>>>> say
>>>>> "no" politely to new features. Until, someone else, who really needs
>>>>> the
>>>>> new feature and he/she is willing to put some effort, will take over
>>>>> and
>>>>> push the software a step further.
>>>>>
>>>>> Once you start, you might have more specific questions on SDB design.
>>>>> You can post your questions here or on jena-dev@incubator.apache.org
>>>>> mailing list. The more you demonstrate you put some effort the more
>>>>> likely you'll receive helpful answers back from the SDB developers.
>>>>> If you don't put enough effort and expect others will do it for you,
>>>>> I am afraid, you risk to have not much back.
>>>>>
>>>>> To conclude, it you are motivated and you think you'll have fun doing
>>>>> it (and you need it for your work): go ahead, it's not a terribly huge
>>>>> task and it could be a nice contribution (in particular for all the
>>>>> MonetDB users out there).
>>>>>
>>>>> Paolo
>>>>>
>>>>>> Thanks in advance!
>>>>>>
>>>>>> Best Regards
>>>>>> Tobias Willig
>>>>>>

Re: Adding support for MonetDB

[natlu 2809 <na...@gmail.com>]

So it would be feasible to model the normalized rdf schema in MonetDB and
hook up as an sdb backend ?

sent from my htc, forgive typos please !
On Sep 2, 2011 10:12 AM, "Paolo Castagna" <ca...@googlemail.com>
wrote:
>
>
> Andy Seaborne wrote:
>>
>> It seems to use a non-normalized table design (as did the CStore paper)
>> and rely on indexing. It would be interesting to see how that compares
>> with a normalized design, which is what most RDF stores use. When
>> normalized, the joins for patterns are on fixed size numbers, not string
>> comparisons, and don't use secondary indexes (and at least one uses
>> bitmap indexes).
>>
>> SDB is built around a normalized design, and portable across various
>> different SQL DBs. The main triple table (or quad table) is 3 columns
>> of 4 or 8 bytes (depending on hash vs index allocation of internal ids).
>> That means the main table is a thin, long table.
>>
>> If you want a deep integration of SPARQL and MonetDB, the first thing to
>> consider, if pure speed is the objective, is what schema design is best
>> for MonetDB.
>>
>> The reported tests are on a dataset of 50 million triples which is
> ^
> not
>
>> particularly big - it means RAM caching is going to play a significant
>> role nowadays (they only used a one-CPU, 4G machine).
>> Andy
>>
>>
>> On 01/09/11 19:49, nat lu wrote:
>>> Speed I think ! At least in some use cases. (But Im no MonetDB expert)
>>> Heres an interesting article that tries a few of them out with MonetDB
>>> and CStore.
>>>
>>> http://oai.cwi.nl/oai/asset/13806/13806B.pdf
>>>
>>>
>>> Whats missing I believe is a the SPARQL endpoint integration.
>>>
>>> On 01/09/11 08:36, Andy Seaborne wrote:
>>>> Tobias,
>>>>
>>>> To turn the question round - what unique features of MonetDB could be
>>>> exposed through SDB to yield a better RDf store?
>>>>
>>>> The current design covers the current set of databases supported but
>>>> it's not fixed - maybe there is something especially useful in MonetDB
>>>> and maybe it needs an extension to the design. The design is based on
>>>> templating via all those classes (the instance for each database is a
>>>> small class). The SQL generator usually needs some per-DB work because
>>>> SQL databases aren't exactly very "standard".
>>>>
>>>> Andy
>>>>
>>>> On 25/08/11 21:28, Paolo Castagna wrote:
>>>>> Hi Tobias,
>>>>> first of all, welcome on jena-users mailing list.
>>>>>
>>>>> Tobias Willig wrote:
>>>>>> Hi everyone,
>>>>>>
>>>>>> I like to add support for MonetDB in SDB and I have two questions
>>>>>> concerning
>>>>>> this project:
>>>>>>
>>>>>> 1. How much effort it takes to add support for a new database type?
>>>>>
>>>>> It requires some effort. You need to add new Java classes and the
>>>>> necessary tests.
>>>>>
>>>>> Have you checked out the SDB sources yet?
>>>>>
>>>>> If not:
>>>>>
>>>>> svn co
>>>>> https://svn.apache.org/repos/asf/incubator/jena/Jena2/SDB/trunk/ SDB
>>>>> cd SDB
>>>>> mvn test
>>>>>
>>>>> You can use Eclipse and search for "Derby" which is one of the DBMS
>>>>> supported
>>>>> by SDB. This way you'll find the list of Java classes in SDB to
>>>>> support Derby.
>>>>> Then, you can read and study those classes. While you do that, you'll
>>>>> learn the
>>>>> design of SDB and you will get an idea on what it is required to add
>>>>> MonetDB.
>>>>>
>>>>>> 2. Are there predefined extension points that allow adding a new
>>>>>> database
>>>>>> type easily?
>>>>>
>>>>> Yes, there are. Look at the super classes and interfaces from the
>>>>> list of
>>>>> classes above (i.e. searching for "Derby").
>>>>>
>>>>> There isn't an "how to add a new database to SDB" guide, however make
>>>>> sure
>>>>> you read the general SDB documentation (it does not hurt).
>>>>>
>>>>> Also, if you want to contribute an "how to add a new database to SDB"
>>>>> you
>>>>> are more than welcome to do so.
>>>>>
>>>>>> If so could you give me the name of some classes and config files,
>>>>>> which
>>>>>> are important to accomplish that task?
>>>>>
>>>>> You can start from:
>>>>>
>>>>> GenerateSQLDerby -- extends --> GenrateSQL
>>>>> FormatterSimpleDerby -- extends --> FormatterSimple
>>>>> StoreSimpleDerby -- extends --> StoreBase1
>>>>> FmtLayout2HashDerby -- extends --> FmtLayout2
>>>>> StoreTriplesNodesHashDerby -- extends --> StoreBaseHash
>>>>> TupleLoaderHashDerby -- extends --> TupleLoaderHashBase
>>>>> FmtLayout2IndexDerby -- extends --> FmtLayout2HashDerby
>>>>> StoreTriplesNodesIndexDerby -- extends --> StoreBaseIndex
>>>>> TupleLoaderIndexDerby -- extends --> TupleLoaderIndexBase
>>>>>
>>>>> Also look at:
>>>>>
>>>>> - JDBC.java
>>>>> - DatabaseType.java
>>>>> - StoreFactory.java
>>>>> - SDB.java
>>>>>
>>>>> And the existing tests.
>>>>>
>>>>> May I ask you what motivates you in adding MonetDB?
>>>>>
>>>>> I've never used it myself and, indeed, I use TDB instead of SDB.
>>>>> Transactions are coming, hopefully in the 0.9.0 release of TDB.
>>>>>
>>>>> Last but not least, it's not only about the code. You should be
willing
>>>>> to support the users of your code too. Once you add support for
>>>>> MonetDB,
>>>>> people will start using it and, as they use your code, they'll find
>>>>> bugs
>>>>> and they'll ask you for more features, eventually. You should be
>>>>> willing
>>>>> to put some effort in fixing the bugs, at least... and you can always
>>>>> say
>>>>> "no" politely to new features. Until, someone else, who really needs
>>>>> the
>>>>> new feature and he/she is willing to put some effort, will take over
>>>>> and
>>>>> push the software a step further.
>>>>>
>>>>> Once you start, you might have more specific questions on SDB design.
>>>>> You can post your questions here or on jena-dev@incubator.apache.org
>>>>> mailing list. The more you demonstrate you put some effort the more
>>>>> likely you'll receive helpful answers back from the SDB developers.
>>>>> If you don't put enough effort and expect others will do it for you,
>>>>> I am afraid, you risk to have not much back.
>>>>>
>>>>> To conclude, it you are motivated and you think you'll have fun doing
>>>>> it (and you need it for your work): go ahead, it's not a terribly huge
>>>>> task and it could be a nice contribution (in particular for all the
>>>>> MonetDB users out there).
>>>>>
>>>>> Paolo
>>>>>
>>>>>>
>>>>>> Thanks in advance!
>>>>>>
>>>>>> Best Regards
>>>>>> Tobias Willig
>>>>>>
>>>

Re: Adding support for MonetDB

[Paolo Castagna <ca...@googlemail.com>]

Andy Seaborne wrote:
> 
> It seems to use a non-normalized table design (as did the CStore paper)
> and rely on indexing.  It would be interesting to see how that compares
> with a normalized design, which is what most RDF stores use.  When
> normalized, the joins for patterns are on fixed size numbers, not string
> comparisons, and don't use secondary indexes (and at least one uses
> bitmap indexes).
> 
> SDB is built around a normalized design, and portable across various
> different SQL DBs.  The main triple table (or quad table) is 3 columns
> of 4 or 8 bytes (depending on hash vs index allocation of internal ids).
> That means the main table is a thin, long table.
> 
> If you want a deep integration of SPARQL and MonetDB, the first thing to
> consider, if pure speed is the objective, is what schema design is best
> for MonetDB.
> 
> The reported tests are on a dataset of 50 million triples which is
                                                                      ^
                                                                     not

> particularly big - it means RAM caching is going to play a significant
> role nowadays (they only used a one-CPU, 4G machine).
>     Andy
> 
> 
> On 01/09/11 19:49, nat lu wrote:
>> Speed I think ! At least in some use cases. (But Im no MonetDB expert)
>> Heres an interesting article that tries a few of them out with MonetDB
>> and CStore.
>>
>> http://oai.cwi.nl/oai/asset/13806/13806B.pdf
>>
>>
>> Whats missing I believe is a the SPARQL endpoint integration.
>>
>> On 01/09/11 08:36, Andy Seaborne wrote:
>>> Tobias,
>>>
>>> To turn the question round - what unique features of MonetDB could be
>>> exposed through SDB to yield a better RDf store?
>>>
>>> The current design covers the current set of databases supported but
>>> it's not fixed - maybe there is something especially useful in MonetDB
>>> and maybe it needs an extension to the design. The design is based on
>>> templating via all those classes (the instance for each database is a
>>> small class). The SQL generator usually needs some per-DB work because
>>> SQL databases aren't exactly very "standard".
>>>
>>> Andy
>>>
>>> On 25/08/11 21:28, Paolo Castagna wrote:
>>>> Hi Tobias,
>>>> first of all, welcome on jena-users mailing list.
>>>>
>>>> Tobias Willig wrote:
>>>>> Hi everyone,
>>>>>
>>>>> I like to add support for MonetDB in SDB and I have two questions
>>>>> concerning
>>>>> this project:
>>>>>
>>>>> 1. How much effort it takes to add support for a new database type?
>>>>
>>>> It requires some effort. You need to add new Java classes and the
>>>> necessary tests.
>>>>
>>>> Have you checked out the SDB sources yet?
>>>>
>>>> If not:
>>>>
>>>> svn co
>>>> https://svn.apache.org/repos/asf/incubator/jena/Jena2/SDB/trunk/ SDB
>>>> cd SDB
>>>> mvn test
>>>>
>>>> You can use Eclipse and search for "Derby" which is one of the DBMS
>>>> supported
>>>> by SDB. This way you'll find the list of Java classes in SDB to
>>>> support Derby.
>>>> Then, you can read and study those classes. While you do that, you'll
>>>> learn the
>>>> design of SDB and you will get an idea on what it is required to add
>>>> MonetDB.
>>>>
>>>>> 2. Are there predefined extension points that allow adding a new
>>>>> database
>>>>> type easily?
>>>>
>>>> Yes, there are. Look at the super classes and interfaces from the
>>>> list of
>>>> classes above (i.e. searching for "Derby").
>>>>
>>>> There isn't an "how to add a new database to SDB" guide, however make
>>>> sure
>>>> you read the general SDB documentation (it does not hurt).
>>>>
>>>> Also, if you want to contribute an "how to add a new database to SDB"
>>>> you
>>>> are more than welcome to do so.
>>>>
>>>>> If so could you give me the name of some classes and config files,
>>>>> which
>>>>> are important to accomplish that task?
>>>>
>>>> You can start from:
>>>>
>>>> GenerateSQLDerby -- extends --> GenrateSQL
>>>> FormatterSimpleDerby -- extends --> FormatterSimple
>>>> StoreSimpleDerby -- extends --> StoreBase1
>>>> FmtLayout2HashDerby -- extends --> FmtLayout2
>>>> StoreTriplesNodesHashDerby -- extends --> StoreBaseHash
>>>> TupleLoaderHashDerby -- extends --> TupleLoaderHashBase
>>>> FmtLayout2IndexDerby -- extends --> FmtLayout2HashDerby
>>>> StoreTriplesNodesIndexDerby -- extends --> StoreBaseIndex
>>>> TupleLoaderIndexDerby -- extends --> TupleLoaderIndexBase
>>>>
>>>> Also look at:
>>>>
>>>> - JDBC.java
>>>> - DatabaseType.java
>>>> - StoreFactory.java
>>>> - SDB.java
>>>>
>>>> And the existing tests.
>>>>
>>>> May I ask you what motivates you in adding MonetDB?
>>>>
>>>> I've never used it myself and, indeed, I use TDB instead of SDB.
>>>> Transactions are coming, hopefully in the 0.9.0 release of TDB.
>>>>
>>>> Last but not least, it's not only about the code. You should be willing
>>>> to support the users of your code too. Once you add support for
>>>> MonetDB,
>>>> people will start using it and, as they use your code, they'll find
>>>> bugs
>>>> and they'll ask you for more features, eventually. You should be
>>>> willing
>>>> to put some effort in fixing the bugs, at least... and you can always
>>>> say
>>>> "no" politely to new features. Until, someone else, who really needs
>>>> the
>>>> new feature and he/she is willing to put some effort, will take over
>>>> and
>>>> push the software a step further.
>>>>
>>>> Once you start, you might have more specific questions on SDB design.
>>>> You can post your questions here or on jena-dev@incubator.apache.org
>>>> mailing list. The more you demonstrate you put some effort the more
>>>> likely you'll receive helpful answers back from the SDB developers.
>>>> If you don't put enough effort and expect others will do it for you,
>>>> I am afraid, you risk to have not much back.
>>>>
>>>> To conclude, it you are motivated and you think you'll have fun doing
>>>> it (and you need it for your work): go ahead, it's not a terribly huge
>>>> task and it could be a nice contribution (in particular for all the
>>>> MonetDB users out there).
>>>>
>>>> Paolo
>>>>
>>>>>
>>>>> Thanks in advance!
>>>>>
>>>>> Best Regards
>>>>> Tobias Willig
>>>>>
>>

Re: Adding support for MonetDB

[Andy Seaborne <an...@epimorphics.com>]

It seems to use a non-normalized table design (as did the CStore paper) 
and rely on indexing.  It would be interesting to see how that compares 
with a normalized design, which is what most RDF stores use.  When 
normalized, the joins for patterns are on fixed size numbers, not string 
comparisons, and don't use secondary indexes (and at least one uses 
bitmap indexes).

SDB is built around a normalized design, and portable across various 
different SQL DBs.  The main triple table (or quad table) is 3 columns 
of 4 or 8 bytes (depending on hash vs index allocation of internal ids). 
That means the main table is a thin, long table.

If you want a deep integration of SPARQL and MonetDB, the first thing to 
consider, if pure speed is the objective, is what schema design is best 
for MonetDB.

The reported tests are on a dataset of 50 million triples which is 
particularly big - it means RAM caching is going to play a significant 
role nowadays (they only used a one-CPU, 4G machine).
	Andy


On 01/09/11 19:49, nat lu wrote:
> Speed I think ! At least in some use cases. (But Im no MonetDB expert)
> Heres an interesting article that tries a few of them out with MonetDB
> and CStore.
>
> http://oai.cwi.nl/oai/asset/13806/13806B.pdf
 >
>
> Whats missing I believe is a the SPARQL endpoint integration.
>
> On 01/09/11 08:36, Andy Seaborne wrote:
>> Tobias,
>>
>> To turn the question round - what unique features of MonetDB could be
>> exposed through SDB to yield a better RDf store?
>>
>> The current design covers the current set of databases supported but
>> it's not fixed - maybe there is something especially useful in MonetDB
>> and maybe it needs an extension to the design. The design is based on
>> templating via all those classes (the instance for each database is a
>> small class). The SQL generator usually needs some per-DB work because
>> SQL databases aren't exactly very "standard".
>>
>> Andy
>>
>> On 25/08/11 21:28, Paolo Castagna wrote:
>>> Hi Tobias,
>>> first of all, welcome on jena-users mailing list.
>>>
>>> Tobias Willig wrote:
>>>> Hi everyone,
>>>>
>>>> I like to add support for MonetDB in SDB and I have two questions
>>>> concerning
>>>> this project:
>>>>
>>>> 1. How much effort it takes to add support for a new database type?
>>>
>>> It requires some effort. You need to add new Java classes and the
>>> necessary tests.
>>>
>>> Have you checked out the SDB sources yet?
>>>
>>> If not:
>>>
>>> svn co
>>> https://svn.apache.org/repos/asf/incubator/jena/Jena2/SDB/trunk/ SDB
>>> cd SDB
>>> mvn test
>>>
>>> You can use Eclipse and search for "Derby" which is one of the DBMS
>>> supported
>>> by SDB. This way you'll find the list of Java classes in SDB to
>>> support Derby.
>>> Then, you can read and study those classes. While you do that, you'll
>>> learn the
>>> design of SDB and you will get an idea on what it is required to add
>>> MonetDB.
>>>
>>>> 2. Are there predefined extension points that allow adding a new
>>>> database
>>>> type easily?
>>>
>>> Yes, there are. Look at the super classes and interfaces from the
>>> list of
>>> classes above (i.e. searching for "Derby").
>>>
>>> There isn't an "how to add a new database to SDB" guide, however make
>>> sure
>>> you read the general SDB documentation (it does not hurt).
>>>
>>> Also, if you want to contribute an "how to add a new database to SDB"
>>> you
>>> are more than welcome to do so.
>>>
>>>> If so could you give me the name of some classes and config files,
>>>> which
>>>> are important to accomplish that task?
>>>
>>> You can start from:
>>>
>>> GenerateSQLDerby -- extends --> GenrateSQL
>>> FormatterSimpleDerby -- extends --> FormatterSimple
>>> StoreSimpleDerby -- extends --> StoreBase1
>>> FmtLayout2HashDerby -- extends --> FmtLayout2
>>> StoreTriplesNodesHashDerby -- extends --> StoreBaseHash
>>> TupleLoaderHashDerby -- extends --> TupleLoaderHashBase
>>> FmtLayout2IndexDerby -- extends --> FmtLayout2HashDerby
>>> StoreTriplesNodesIndexDerby -- extends --> StoreBaseIndex
>>> TupleLoaderIndexDerby -- extends --> TupleLoaderIndexBase
>>>
>>> Also look at:
>>>
>>> - JDBC.java
>>> - DatabaseType.java
>>> - StoreFactory.java
>>> - SDB.java
>>>
>>> And the existing tests.
>>>
>>> May I ask you what motivates you in adding MonetDB?
>>>
>>> I've never used it myself and, indeed, I use TDB instead of SDB.
>>> Transactions are coming, hopefully in the 0.9.0 release of TDB.
>>>
>>> Last but not least, it's not only about the code. You should be willing
>>> to support the users of your code too. Once you add support for MonetDB,
>>> people will start using it and, as they use your code, they'll find bugs
>>> and they'll ask you for more features, eventually. You should be willing
>>> to put some effort in fixing the bugs, at least... and you can always
>>> say
>>> "no" politely to new features. Until, someone else, who really needs the
>>> new feature and he/she is willing to put some effort, will take over and
>>> push the software a step further.
>>>
>>> Once you start, you might have more specific questions on SDB design.
>>> You can post your questions here or on jena-dev@incubator.apache.org
>>> mailing list. The more you demonstrate you put some effort the more
>>> likely you'll receive helpful answers back from the SDB developers.
>>> If you don't put enough effort and expect others will do it for you,
>>> I am afraid, you risk to have not much back.
>>>
>>> To conclude, it you are motivated and you think you'll have fun doing
>>> it (and you need it for your work): go ahead, it's not a terribly huge
>>> task and it could be a nice contribution (in particular for all the
>>> MonetDB users out there).
>>>
>>> Paolo
>>>
>>>>
>>>> Thanks in advance!
>>>>
>>>> Best Regards
>>>> Tobias Willig
>>>>
>

Re: Adding support for MonetDB

[nat lu <na...@gmail.com>]

Speed I think ! At least in some use cases. (But Im no MonetDB expert) 
Heres an interesting article that tries a few of them out with MonetDB 
and CStore.

http://oai.cwi.nl/oai/asset/13806/13806B.pdf

Whats missing I believe is a the SPARQL endpoint integration.

On 01/09/11 08:36, Andy Seaborne wrote:
> Tobias,
>
> To turn the question round - what unique features of MonetDB could be 
> exposed through SDB to yield a better RDf store?
>
> The current design covers the current set of databases supported but 
> it's not fixed - maybe there is something especially useful in MonetDB 
> and maybe it needs an extension to the design.  The design is based on 
> templating via all those classes (the instance for each database is a 
> small class).  The SQL generator usually needs some per-DB work 
> because SQL databases aren't exactly very "standard".
>
>     Andy
>
> On 25/08/11 21:28, Paolo Castagna wrote:
>> Hi Tobias,
>> first of all, welcome on jena-users mailing list.
>>
>> Tobias Willig wrote:
>>> Hi everyone,
>>>
>>> I like to add support for MonetDB in SDB and I have two questions 
>>> concerning
>>> this project:
>>>
>>> 1. How much effort it takes to add support for a new database type?
>>
>> It requires some effort. You need to add new Java classes and the 
>> necessary tests.
>>
>> Have you checked out the SDB sources yet?
>>
>> If not:
>>
>>    svn co 
>> https://svn.apache.org/repos/asf/incubator/jena/Jena2/SDB/trunk/ SDB
>>    cd SDB
>>    mvn test
>>
>> You can use Eclipse and search for "Derby" which is one of the DBMS 
>> supported
>> by SDB. This way you'll find the list of Java classes in SDB to 
>> support Derby.
>> Then, you can read and study those classes. While you do that, you'll 
>> learn the
>> design of SDB and you will get an idea on what it is required to add 
>> MonetDB.
>>
>>> 2. Are there predefined extension points that allow adding a new 
>>> database
>>> type easily?
>>
>> Yes, there are. Look at the super classes and interfaces from the 
>> list of
>> classes above (i.e. searching for "Derby").
>>
>> There isn't an "how to add a new database to SDB" guide, however make 
>> sure
>> you read the general SDB documentation (it does not hurt).
>>
>> Also, if you want to contribute an "how to add a new database to SDB" 
>> you
>> are more than welcome to do so.
>>
>>>      If so could you give me the name of some classes and config 
>>> files, which
>>> are important to accomplish that task?
>>
>> You can start from:
>>
>>             GenerateSQLDerby -- extends -->  GenrateSQL
>>         FormatterSimpleDerby -- extends -->  FormatterSimple
>>             StoreSimpleDerby -- extends -->  StoreBase1
>>          FmtLayout2HashDerby -- extends -->  FmtLayout2
>>   StoreTriplesNodesHashDerby -- extends -->  StoreBaseHash
>>         TupleLoaderHashDerby -- extends -->  TupleLoaderHashBase
>>         FmtLayout2IndexDerby -- extends -->  FmtLayout2HashDerby
>> StoreTriplesNodesIndexDerby -- extends -->  StoreBaseIndex
>>        TupleLoaderIndexDerby -- extends -->  TupleLoaderIndexBase
>>
>> Also look at:
>>
>>   - JDBC.java
>>   - DatabaseType.java
>>   - StoreFactory.java
>>   - SDB.java
>>
>> And the existing tests.
>>
>> May I ask you what motivates you in adding MonetDB?
>>
>> I've never used it myself and, indeed, I use TDB instead of SDB.
>> Transactions are coming, hopefully in the 0.9.0 release of TDB.
>>
>> Last but not least, it's not only about the code. You should be willing
>> to support the users of your code too. Once you add support for MonetDB,
>> people will start using it and, as they use your code, they'll find bugs
>> and they'll ask you for more features, eventually. You should be willing
>> to put some effort in fixing the bugs, at least... and you can always 
>> say
>> "no" politely to new features. Until, someone else, who really needs the
>> new feature and he/she is willing to put some effort, will take over and
>> push the software a step further.
>>
>> Once you start, you might have more specific questions on SDB design.
>> You can post your questions here or on jena-dev@incubator.apache.org
>> mailing list. The more you demonstrate you put some effort the more
>> likely you'll receive helpful answers back from the SDB developers.
>> If you don't put enough effort and expect others will do it for you,
>> I am afraid, you risk to have not much back.
>>
>> To conclude, it you are motivated and you think you'll have fun doing
>> it (and you need it for your work): go ahead, it's not a terribly huge
>> task and it could be a nice contribution (in particular for all the
>> MonetDB users out there).
>>
>> Paolo
>>
>>>
>>> Thanks in advance!
>>>
>>> Best Regards
>>> Tobias Willig
>>>

Re: Adding support for MonetDB

[Andy Seaborne <an...@epimorphics.com>]

Tobias,

To turn the question round - what unique features of MonetDB could be 
exposed through SDB to yield a better RDf store?

The current design covers the current set of databases supported but 
it's not fixed - maybe there is something especially useful in MonetDB 
and maybe it needs an extension to the design.  The design is based on 
templating via all those classes (the instance for each database is a 
small class).  The SQL generator usually needs some per-DB work because 
SQL databases aren't exactly very "standard".

	Andy

On 25/08/11 21:28, Paolo Castagna wrote:
> Hi Tobias,
> first of all, welcome on jena-users mailing list.
>
> Tobias Willig wrote:
>> Hi everyone,
>>
>> I like to add support for MonetDB in SDB and I have two questions concerning
>> this project:
>>
>> 1. How much effort it takes to add support for a new database type?
>
> It requires some effort. You need to add new Java classes and the necessary tests.
>
> Have you checked out the SDB sources yet?
>
> If not:
>
>    svn co https://svn.apache.org/repos/asf/incubator/jena/Jena2/SDB/trunk/ SDB
>    cd SDB
>    mvn test
>
> You can use Eclipse and search for "Derby" which is one of the DBMS supported
> by SDB. This way you'll find the list of Java classes in SDB to support Derby.
> Then, you can read and study those classes. While you do that, you'll learn the
> design of SDB and you will get an idea on what it is required to add MonetDB.
>
>> 2. Are there predefined extension points that allow adding a new database
>> type easily?
>
> Yes, there are. Look at the super classes and interfaces from the list of
> classes above (i.e. searching for "Derby").
>
> There isn't an "how to add a new database to SDB" guide, however make sure
> you read the general SDB documentation (it does not hurt).
>
> Also, if you want to contribute an "how to add a new database to SDB" you
> are more than welcome to do so.
>
>>      If so could you give me the name of some classes and config files, which
>> are important to accomplish that task?
>
> You can start from:
>
>             GenerateSQLDerby -- extends -->  GenrateSQL
>         FormatterSimpleDerby -- extends -->  FormatterSimple
>             StoreSimpleDerby -- extends -->  StoreBase1
>          FmtLayout2HashDerby -- extends -->  FmtLayout2
>   StoreTriplesNodesHashDerby -- extends -->  StoreBaseHash
>         TupleLoaderHashDerby -- extends -->  TupleLoaderHashBase
>         FmtLayout2IndexDerby -- extends -->  FmtLayout2HashDerby
> StoreTriplesNodesIndexDerby -- extends -->  StoreBaseIndex
>        TupleLoaderIndexDerby -- extends -->  TupleLoaderIndexBase
>
> Also look at:
>
>   - JDBC.java
>   - DatabaseType.java
>   - StoreFactory.java
>   - SDB.java
>
> And the existing tests.
>
> May I ask you what motivates you in adding MonetDB?
>
> I've never used it myself and, indeed, I use TDB instead of SDB.
> Transactions are coming, hopefully in the 0.9.0 release of TDB.
>
> Last but not least, it's not only about the code. You should be willing
> to support the users of your code too. Once you add support for MonetDB,
> people will start using it and, as they use your code, they'll find bugs
> and they'll ask you for more features, eventually. You should be willing
> to put some effort in fixing the bugs, at least... and you can always say
> "no" politely to new features. Until, someone else, who really needs the
> new feature and he/she is willing to put some effort, will take over and
> push the software a step further.
>
> Once you start, you might have more specific questions on SDB design.
> You can post your questions here or on jena-dev@incubator.apache.org
> mailing list. The more you demonstrate you put some effort the more
> likely you'll receive helpful answers back from the SDB developers.
> If you don't put enough effort and expect others will do it for you,
> I am afraid, you risk to have not much back.
>
> To conclude, it you are motivated and you think you'll have fun doing
> it (and you need it for your work): go ahead, it's not a terribly huge
> task and it could be a nice contribution (in particular for all the
> MonetDB users out there).
>
> Paolo
>
>>
>> Thanks in advance!
>>
>> Best Regards
>> Tobias Willig
>>

Re: Adding support for MonetDB

[Paolo Castagna <ca...@googlemail.com>]

Hi Tobias,
first of all, welcome on jena-users mailing list.

Tobias Willig wrote:
> Hi everyone,
> 
> I like to add support for MonetDB in SDB and I have two questions concerning
> this project:
> 
> 1. How much effort it takes to add support for a new database type?

It requires some effort. You need to add new Java classes and the necessary tests.

Have you checked out the SDB sources yet?

If not:

  svn co https://svn.apache.org/repos/asf/incubator/jena/Jena2/SDB/trunk/ SDB
  cd SDB
  mvn test

You can use Eclipse and search for "Derby" which is one of the DBMS supported
by SDB. This way you'll find the list of Java classes in SDB to support Derby.
Then, you can read and study those classes. While you do that, you'll learn the
design of SDB and you will get an idea on what it is required to add MonetDB.

> 2. Are there predefined extension points that allow adding a new database
> type easily?

Yes, there are. Look at the super classes and interfaces from the list of
classes above (i.e. searching for "Derby").

There isn't an "how to add a new database to SDB" guide, however make sure
you read the general SDB documentation (it does not hurt).

Also, if you want to contribute an "how to add a new database to SDB" you
are more than welcome to do so.

>     If so could you give me the name of some classes and config files, which
> are important to accomplish that task?

You can start from:

           GenerateSQLDerby -- extends --> GenrateSQL
       FormatterSimpleDerby -- extends --> FormatterSimple
           StoreSimpleDerby -- extends --> StoreBase1
        FmtLayout2HashDerby -- extends --> FmtLayout2
 StoreTriplesNodesHashDerby -- extends --> StoreBaseHash
       TupleLoaderHashDerby -- extends --> TupleLoaderHashBase
       FmtLayout2IndexDerby -- extends --> FmtLayout2HashDerby
StoreTriplesNodesIndexDerby -- extends --> StoreBaseIndex
      TupleLoaderIndexDerby -- extends --> TupleLoaderIndexBase

Also look at:

 - JDBC.java
 - DatabaseType.java
 - StoreFactory.java
 - SDB.java

And the existing tests.

May I ask you what motivates you in adding MonetDB?

I've never used it myself and, indeed, I use TDB instead of SDB.
Transactions are coming, hopefully in the 0.9.0 release of TDB.

Last but not least, it's not only about the code. You should be willing
to support the users of your code too. Once you add support for MonetDB,
people will start using it and, as they use your code, they'll find bugs
and they'll ask you for more features, eventually. You should be willing
to put some effort in fixing the bugs, at least... and you can always say
"no" politely to new features. Until, someone else, who really needs the
new feature and he/she is willing to put some effort, will take over and
push the software a step further.

Once you start, you might have more specific questions on SDB design.
You can post your questions here or on jena-dev@incubator.apache.org
mailing list. The more you demonstrate you put some effort the more
likely you'll receive helpful answers back from the SDB developers.
If you don't put enough effort and expect others will do it for you,
I am afraid, you risk to have not much back.

To conclude, it you are motivated and you think you'll have fun doing
it (and you need it for your work): go ahead, it's not a terribly huge
task and it could be a nice contribution (in particular for all the
MonetDB users out there).

Paolo

> 
> Thanks in advance!
> 
> Best Regards
> Tobias Willig
>