You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@jena.apache.org by Rob Vesse <rv...@yarcdata.com> on 2012/10/18 23:21:18 UTC
Re: RIOT reader - new reader system for Jena.
Hey Andy
Sorry for taking forever to get back to you on this but comments inline:
On 8/17/12 5:54 AM, "Andy Seaborne" <an...@apache.org> wrote:
>I'm at the point of being ready to integrate RIOT and anew reader system
>into Jena properly. This means we can remove the old parsers in
>jena-core (not ARP).
>
>There is a "but" however.
>
>RIOT supports both triples and quads readers and model/graphs and
>datasets/datasetgraphs ... but classes for all things quad are in ARQ.
>
>I've created a JIRA but I thought I'd surface it here because it has the
>potential to be disruptive.
>
>https://issues.apache.org/jira/browse/JENA-300
>
>== Integration
>
>Possibilities:
>
>1/ Put the code in ARQ
>1a/ require a cal lto ARQ to initialize
>1b/ make jena-core do as reflection call to ARQ initialization.
>
>2/ Merge jena-arq and jena-core
>
>The obvious issue for (2) is that the result is a big project to work
>with. Whether a larger jena-core really makes a difference in the real
>world., I don't know. Long term, some redivision into separate modules
>would be good but it's quite hard to find any breakdown of core concepts
>if you want testing by module. It's hard to do anything much without a
>memory graph implementation!
>
>If (2), it would be good to time this with making an uber jar
>"jena-VERSION.jar" so that people switch to that and don't see any
>future reorg of the modules unless they take a detailed look.
How about this as a suggestion for the short term:
- Move Quad and the riot sub-system into jena-core
- Replace the jena-core reader machinery with the riot sub-system
This has the advantage of keeping everything query still in it's own
module and does not need to break down core. Ideally it would be nice to
split off the riot sub-system into it's own module but then you get into
problems of there being no reader/writer sub-system in core and requiring
users to pull in an extra dependency for one of the most common things
they are going to do. I assume you plan to integrate this after 2.7.4
perhaps with a minor version bump I.e. 2.8.0
Longer term I tried to think of some ways to nicely separate things out
but was kinda struggling, with the Model interface as it stands (wit it's
own read()/write() methods) there is no way to cleanly separate the riot
sub-system out from jena-core/jena-arq in the same way that Sesame
separate their IO subsystem into their RIO modules. They have a
sesame-rio-api module and then specific small modules implementing each
reader.
If we could remove the read()/write() methods from Model then we can start
to get a better separation of concerns:
- Interfaces for reading/writing form a jena-riot-api module
- Implementations form another module jena-riot-std module
In the place of a read()/write() method directly on a Model we can provide
a static ModelIO class with read() and write() methods. Wiring up of
readers and writers for use by this could perhaps be done automagically
through some package scanning and Java attributes combination?
Hope these thoughts help
Rob
>
>== Outline of the reader
>
>There is a single class "WebReader2" that captures the process of
>opening a connection to a resource/file/thing, deciding the syntax and
>then calling the right parser. This adds full http content negotiation
>over what Jena currently does.
>
>You can add new content-types and connect to the appropriate parser code.
>
>It includes going through FileManager and if/when that connected to
>model.read, all the conneg, redirection and location mapping is made
>fundamental. You can even could make all URLs of a pattern
> http://myhost/data/turtle/file{n}
>be Turtle files despite being served as text/plain.
>
>== Code
>
>In an "Experimental" project:
>
>https://svn.apache.org/repos/asf/jena/Experimental/riot-reader/
>
>Code browse;
>https://svn.apache.org/viewvc/jena/Experimental/riot-reader/src/main/java/
>riot_reader/
>
>The package layout isn't right for integration.
Re: RIOT reader - new reader system for Jena.
Posted by Andy Seaborne <an...@apache.org>.
I have now moved the RIOT reader functionality into jena-arq. Sorry
about the cascade of commits - there is something about deleting and
moving directory trees in svn that I don't "get" svn and I end end with
various fixups needed after a failed large modification to a code tree.
I went for the simplest option - the code is moved into ARQ and wired in
when RIOT is initalized. That happens when ARQ is touched so it does
not yet apply when jena-core only code is used. That can be short-term
fixed by a reflection call in Jena runtime initialization and long-term
but reorg of the code structure.
At present there is duplication/redundancy/replication of code across
org.openjena.riot and org.apache.jena.riot (e.g. you can find classes
with "2" in their name which are replacements). But with the code
merged, it's a matter of going through in small tasks to clean-up.
The external effect is a lot of code to make:
model.read("http://people.apache.org/~andy/RIOT-test/D.x")
and
model.read("http://people.apache.org/~andy/RIOT-test/D.ttl")
work by content negotiation.
(my .htaccess has, for testing:
AddType text/turtle .x
)
except at the moment .../D.ttl does not work when there is no MIME type
declaration. There's a switch to make text/plain to instead look at the
file name extension and, after some more testing, that will probably be
better.
Getting the fine details of what to do when remote configurations are
messier than what the standards describe will be needed but that's life
on the web.
It is also a lot easier to add new languages to the reader system
(JSON-LD? or converters like CSV->RDF) from code outside the Jena codebase.
Andy
Re: RIOT reader - new reader system for Jena.
Posted by Andy Seaborne <an...@apache.org>.
On 18/10/12 22:21, Rob Vesse wrote:
> Hey Andy
>
> Sorry for taking forever to get back to you on this but comments inline:
>
>
> On 8/17/12 5:54 AM, "Andy Seaborne" <an...@apache.org> wrote:
>
>> I'm at the point of being ready to integrate RIOT and anew reader system
>> into Jena properly. This means we can remove the old parsers in
>> jena-core (not ARP).
>>
>> There is a "but" however.
>>
>> RIOT supports both triples and quads readers and model/graphs and
>> datasets/datasetgraphs ... but classes for all things quad are in ARQ.
>>
>> I've created a JIRA but I thought I'd surface it here because it has the
>> potential to be disruptive.
>>
>> https://issues.apache.org/jira/browse/JENA-300
>>
>> == Integration
>>
>> Possibilities:
>>
>> 1/ Put the code in ARQ
>> 1a/ require a cal lto ARQ to initialize
>> 1b/ make jena-core do as reflection call to ARQ initialization.
>>
>> 2/ Merge jena-arq and jena-core
>>
>> The obvious issue for (2) is that the result is a big project to work
>> with. Whether a larger jena-core really makes a difference in the real
>> world., I don't know. Long term, some redivision into separate modules
>> would be good but it's quite hard to find any breakdown of core concepts
>> if you want testing by module. It's hard to do anything much without a
>> memory graph implementation!
>>
>> If (2), it would be good to time this with making an uber jar
>> "jena-VERSION.jar" so that people switch to that and don't see any
>> future reorg of the modules unless they take a detailed look.
>
>
> How about this as a suggestion for the short term:
>
> - Move Quad and the riot sub-system into jena-core
> - Replace the jena-core reader machinery with the riot sub-system
>
> This has the advantage of keeping everything query still in it's own
> module and does not need to break down core. Ideally it would be nice to
> split off the riot sub-system into it's own module but then you get into
> problems of there being no reader/writer sub-system in core and requiring
> users to pull in an extra dependency for one of the most common things
> they are going to do. I assume you plan to integrate this after 2.7.4
> perhaps with a minor version bump I.e. 2.8.0
>
> Longer term I tried to think of some ways to nicely separate things out
> but was kinda struggling, with the Model interface as it stands (wit it's
> own read()/write() methods) there is no way to cleanly separate the riot
> sub-system out from jena-core/jena-arq in the same way that Sesame
> separate their IO subsystem into their RIO modules. They have a
> sesame-rio-api module and then specific small modules implementing each
> reader.
>
> If we could remove the read()/write() methods from Model then we can start
> to get a better separation of concerns:
>
> - Interfaces for reading/writing form a jena-riot-api module
> - Implementations form another module jena-riot-std module
>
> In the place of a read()/write() method directly on a Model we can provide
> a static ModelIO class with read() and write() methods. Wiring up of
> readers and writers for use by this could perhaps be done automagically
> through some package scanning and Java attributes combination?
>
> Hope these thoughts help
>
> Rob
Moving just RIOT out of ARQ is the way to go. It's not just Quad - it's
Dataset as well which is public API. Quad informally is as well so it
needs to be coupled with a significant version change. While not in the
API, extensions and deep working with ARQ does tend to arrive at Quad.
After that, its the effect of pulling the thread that yanks more stuff.
The jena-*-api module idea would work although maybe some testing
might need to be put into a testing module to get ordering right. Hard
to test APIs without an implementation to hand.
RIOT would be its own subsystem - it does not need all of jena-core (it
should not need the client API for example, or OntAPI, it does need
datatypes).
I'm not convinced about one module per parser because they (this is "not
RDF/XML") share so much but one module for RIOT would be ideal. I
confess I don't like it when the internal need for a module structure
ends up dictaing the public API design - sometimes a public API with a
mix of things is easier to use but the mix may be across internal design.
All .read calls do become legacy ways of getting to a library - it is
inverting the structure. What it bites is WebReader2 - the class of
static functions that reads things. It has both Model and Dataset calls.
Having ModelIO for all Model calls and DatasetIO for all datasets calls
is a good thought. I'll give that a go and so if the dependencies work out.
But it is a nice example of where internal divisions force public API.
What if you read a web location, not knowing if its triples or quads?
Be nicer to get the right thing back not have to decide before the call
whether it's triples or whether if quads.
All .read calls do become legacy ways of getting to the new reader
structure. riot-reader rewires existing Jena to route the .read calls
to one piece of code. It was one single reader but two places require
the language to be known in advance so it is one very thin reader per
language to add in the default value. The existing code does
newInstance after deciding the language. The new code delays the
language decision until after conneg.
If we have a single "one jar", then trick of jena-core making a
reflection call on RIOT to initialize with RIOT reader will work. The
user will see new code without us beforehand having to undergo a deep
restructure but eventual the restructure should happen, on a timescale
that is relaxed, not forced by release cycles.
Andy
>
>>
>> == Outline of the reader
>>
>> There is a single class "WebReader2" that captures the process of
>> opening a connection to a resource/file/thing, deciding the syntax and
>> then calling the right parser. This adds full http content negotiation
>> over what Jena currently does.
>>
>> You can add new content-types and connect to the appropriate parser code.
>>
>> It includes going through FileManager and if/when that connected to
>> model.read, all the conneg, redirection and location mapping is made
>> fundamental. You can even could make all URLs of a pattern
>> http://myhost/data/turtle/file{n}
>> be Turtle files despite being served as text/plain.
>>
>> == Code
>>
>> In an "Experimental" project:
>>
>> https://svn.apache.org/repos/asf/jena/Experimental/riot-reader/
>>
>> Code browse;
>> https://svn.apache.org/viewvc/jena/Experimental/riot-reader/src/main/java/
>> riot_reader/
>>
>> The package layout isn't right for integration.
>
Re: RIOT reader - new reader system for Jena.
Posted by Andy Seaborne <an...@apache.org>.
On 18/10/12 22:21, Rob Vesse wrote:
> I assume you plan to integrate this after 2.7.4
> perhaps with a minor version bump I.e. 2.8.0
Full response another day but just to confirm this is not in 2.7.4.
Andy