You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@jena.apache.org by Ying Jiang <jp...@gmail.com> on 2014/07/09 04:05:26 UTC
Re: GSoC: CSV Property Tables.
Dear Andy,
Thanks a lot! I've been studying the source code of SPARQL algebra
following your instructions these days.
I've made a try-out of the optimization of querying a whole row in the
property table. The following query pattern can be transformed into a
row querying, without generating triples:
?x :prop1 ?v .
?x :prop2 ?w .
...
It's made by using the extension point of StageGenerator, because it's
now just concerned with BasicPattern. The detailed workflow goes in
this way:
1. Split the incoming BasicPattern by subjects, (i.e. it becomes
multiple sub BasicPatterns grouped by the same subjects. (see
QueryIterPropertyTable [1])
2. For each sub BasicPattern,
if the triple size within is greater than 1 (i.e. at least 2 triples),
it's turned into a row querying, and processed by
QueryIterPropertyTableRow [2]
else if it contains only 1 triple, it goes for the traditional triple
querying by graph.graphBaseFind()
The code have been committed. I'll think about other optimization ways
and test with real-life large CSV data you provided later on. Any
suggestion is very welcome!
Best regards,
Ying Jiang
[1] https://svn.apache.org/repos/asf/jena/Experimental/jena-csv/src/main/java/com/hp/hpl/jena/propertytable/impl/QueryIterPropertyTable.java
[2] https://svn.apache.org/repos/asf/jena/Experimental/jena-csv/src/main/java/com/hp/hpl/jena/propertytable/impl/QueryIterPropertyTableRow.java
On Mon, Jun 30, 2014 at 9:47 PM, Andy Seaborne <an...@seaborne.org> wrote:
> Hi Ying,
>
>
>
>> ---- Quote from the project proposal:
>> Accessing from SPARQL via Graph.find() will work, but it's not ideal.
>> Some optimizations can be done for processing a SPARQL basic graph
>> pattern. More explicitly, in the method of OpExecutor.execute(OpBGP,
>> ...), when the target for the query is a GraphPropertyTable, it can
>> get a whole row, or rows, of the table data and match the pattern with
>> the bindings.
>> ----
>> Can you show me a concrete example of how "getting a whole row, or
>> rows" can improve the query of (e.g. SUM/COUNT)? Suppose we have the
>> data:
>> ----
>> Town,Population
>> Southton,123000
>> Northville,654000
>> ----
>> If we want to query the total population of the (two or all) towns,
>> how does changing OpExecutor.execute(OpBGP, ...) help? Here's the
>> query string:
>> ----
>> SELECT (sum(?pop) AS ?totalPop) {
>> { GRAPH <file:///c:/town.csv> {
>> ?x :Popuation ?pop .
>> }
>> }
>> ----
>> If the above example is not appropriate for for this case (2.3), could
>> you please show me another one. I'm very grateful if you can help me
>> with this question before your holiday.
>
>
>
> OpExecutor is the way SPARQL is evaluated. The query is parsed, truned into
> SPARQL algebra (the Op* classes), optimized (yet more Op* classes and
> rearranging the ones in the query) then evaluated. There is one
> execute(Op...) for each Op operator in the SPARQL algebra.
>
> OpBGP is the operator for basic graph patterns, a basic graph pattern is a
> sequence of triple patterns. The default is iteratively use Graph.find and
> in the case of a single triple pattern
>
> The order of triple patterns matters for performance (you get the same
> answer regardless of order of execution, better orders of the pattern
> traverse less negative/rejected
>
> Consider, in some general data, not CSV related:
>
> ?x :predicate ?value . ## Triple pattern 1
> ?x :other 123 . ## Triple pattern 2
>
> and suppose there are 1000 subjects each with :predicate and :other as
> triples. And suppose there is one with predicate-value ":other 123".
>
> If you do the execution in the order above it's 1000 possibilities from the
> first triple pattern. If the iterative OpExecutor strategy is used, its then
> a lots of tests of "?x :other 123" for each possible ?x.
>
> But if the order is reversed, there is one match for "?x :other 123 ." then
> one access for "?x :predicate ?value ." (?x is now a fixed value).
>
> In the default setup (in-memory and any storage that does not override and
> add something better, (see StageGeneratorGeneric) the fixed reording is done
> based on finding the most grounded patterns. It should choose to do ":other
> 123" first.
>
> In the CSV case, there may be better things to do that are specific to CSV
> and the property table storage. This needs investigation and design before
> implementation; I don't have a definite answer.
>
> It will depend a lot on the datastructures for the property tables - this is
> where all the pieces fit together.
>
> One possibility is avoiding generating triples. If the pattern is
>
> ?x :prop1 ?v .
> ?x :prop2 ?w .
>
> what 's needed is bindings (a row in SPARQL results) for (?x,?v,?w). There
> is not need to create a lot of triples via find().
>
> Other patterns that might have optimizations are where the subject is not
> the same in each pattern:
>
> ?x :prop1 ?y .
> ?y :prop2 ?w .
>
> Better execution of aggregates like SUM is also possible (in your example,
> it could be a matter of traversing the "population" column adding the
> values, avoiding creating any triples) but I'm thinking of basic graph
> patterns for the moment.
>
> Real CSVs files are often denormalised tables - lots of repetition from row
> to row. This might be exploitable.
>
> There are some real worlds examples documented in:
>
> http://www.w3.org/TR/csvw-ucr/
>
> with links to the real CSV files.
>
> To investigate, its probably a good idea to understand how OpExecutor works
> and also to thin of some patterns and see what might be done. More
> complicated data (more columns, more rows)
>
>> Is there a tool in the Jena code base for parsing String to Jena
>> Literal for different data types? For example:
>> 1) "65000" -> "65000"^^<http://www.w3.org/2001/XMLSchema#integer>
>> 2) "65000.123" -> "65000.123"^^<http://www.w3.org/2001/XMLSchema#double>
>> 3) "true" -> "true"^^<http://www.w3.org/2001/XMLSchema#boolean>
>> There would be more advanced features like parsing date/time to
>> xsd:date or xsd:dateTime for CSV values.
>> The NodeFactory.createLiteral() method does not provide these features.
>
>
> There is the parse for Turtle and related langauges (although 65000.123 will
> be a xsd:decimal). See TokenizerText.
>
> You may be better off with your own code, using Long.decode pr
> Long.parseLong, Double.decode or Double.parseDouble, because real data is
> messy and it could be useful to clean up the input first, for example,
> remove "," [*]
>
> Andy
>
> [*] Caution - locale sensitive!
>
Re: GSoC: CSV Property Tables.
Posted by Andy Seaborne <an...@apache.org>.
Ying,
We're in the last week now - 18 August (19:00 UTC) is the firm 'pencils
down' date and time.
A message to users@ announcing the work would be good!
Andy
Re: GSoC: CSV Property Tables.
Posted by Andy Seaborne <an...@apache.org>.
On 11/08/14 03:32, Ying Jiang wrote:
> Dear Andy,
>
> I replaced the dependency of guava-osgi with
> com.googlecode.guava:guava (version 17). So the <repositories> section
> could be removed.
>
> I also updated the tests of jena-csv with some real world csv data
> [1]. However, many of the use cases in [1] are not work with jena-csv
> mainly because:
> 1) Not csv data: some of them are Excel, or user-defined custom
> tabular data structure)
> 2) Irregular csv data: some are multi-dimensional table
> 3) Not UTF-8 encoding: encoded with Western European (e.g. with some
> charterer like '£')
:-|
> 4) No table column header: jena-csv can not make the RDF predicates
> without the column names.
>
> From the tests, I fixed some bugs of LangCSV in jena-arq after
> jena-arq-2.12.0 release. However, the code were committed to Jena
> trunk, which is now 2.12.1-SNAPSHOT. If I want to release jena-csv
> (e.g. version 1.0), which version of jena-arq should it depends on? It
> seems to be a dilemma:
> - If jena-csv-1.0 depends on jena-arq-2.12.1-SNAPSHOT, new code in
> jena-arq are visible, but it's a SNAPSHOT.
> - If jena-csv-1.0 depneds on jena-arq-2.12.0 release, new code in
> jena-arq are not visible and the bugs are not eliminated.
You could unregister the 2.12.0 LangCSV and register a fixed one from
jena-csv. LangXXX doen't have to be in the org.apache.jena.riot.lang
package.
Andy
>
> Any suggestion?
>
> Best regards,
> Ying Jiang
>
> [1] http://www.w3.org/TR/2014/WD-csvw-ucr-20140327/
>
> On Wed, Aug 6, 2014 at 9:28 PM, Andy Seaborne <an...@apache.org> wrote:
>> On 03/08/14 11:41, Ying Jiang wrote:
>>>
>>> Hi Andy,
>>>
>>> Thanks for your instructions!
>>>
>>> Last week, I made the documentation of this GSoC project (jena-csv):
>>> - http://jena.staging.apache.org/documentation/csv/
>>> - http://jena.staging.apache.org/documentation/csv/get_started.html
>>> - http://jena.staging.apache.org/documentation/csv/design.html
>>> - http://jena.staging.apache.org/documentation/csv/implementation.html
>>>
>>> I'll make more tests and improve the documentation in the coming weeks.
>>
>>
>> Looks good - one thing missing is that needs to explain how to get the code.
>> That can be a maven build to the Apache snapshot repository.
>>
>> In POM.xml:
>>
>> guava-osgi -- is there anything in there that isn't in
>> com.googlecode.guava:guava?
>>
>>
>> Please remove the <repositories> section. The build needs to depend on
>> only maven central and on Apache repsoitories (but only for development use
>> of SNAPSHOTS).
>>
>> You should be able to do that as a committer and the configuration is set up
>> by the jena-parent POM.
>>
>> Give it a go!
>>
>> Then it's time to send email to users@ to announce it - if you can do that
>> soon, it'll give time for initial comments to come back before the end of
>> GSoC which woulc be perfect timing.
>>
>> Andy
>>
>>>
>>> Best regards,
>>> Ying Jiang
>>
>>
Re: GSoC: CSV Property Tables.
Posted by Ying Jiang <jp...@gmail.com>.
Dear Andy,
I replaced the dependency of guava-osgi with
com.googlecode.guava:guava (version 17). So the <repositories> section
could be removed.
I also updated the tests of jena-csv with some real world csv data
[1]. However, many of the use cases in [1] are not work with jena-csv
mainly because:
1) Not csv data: some of them are Excel, or user-defined custom
tabular data structure)
2) Irregular csv data: some are multi-dimensional table
3) Not UTF-8 encoding: encoded with Western European (e.g. with some
charterer like '£')
4) No table column header: jena-csv can not make the RDF predicates
without the column names.
>From the tests, I fixed some bugs of LangCSV in jena-arq after
jena-arq-2.12.0 release. However, the code were committed to Jena
trunk, which is now 2.12.1-SNAPSHOT. If I want to release jena-csv
(e.g. version 1.0), which version of jena-arq should it depends on? It
seems to be a dilemma:
- If jena-csv-1.0 depends on jena-arq-2.12.1-SNAPSHOT, new code in
jena-arq are visible, but it's a SNAPSHOT.
- If jena-csv-1.0 depneds on jena-arq-2.12.0 release, new code in
jena-arq are not visible and the bugs are not eliminated.
Any suggestion?
Best regards,
Ying Jiang
[1] http://www.w3.org/TR/2014/WD-csvw-ucr-20140327/
On Wed, Aug 6, 2014 at 9:28 PM, Andy Seaborne <an...@apache.org> wrote:
> On 03/08/14 11:41, Ying Jiang wrote:
>>
>> Hi Andy,
>>
>> Thanks for your instructions!
>>
>> Last week, I made the documentation of this GSoC project (jena-csv):
>> - http://jena.staging.apache.org/documentation/csv/
>> - http://jena.staging.apache.org/documentation/csv/get_started.html
>> - http://jena.staging.apache.org/documentation/csv/design.html
>> - http://jena.staging.apache.org/documentation/csv/implementation.html
>>
>> I'll make more tests and improve the documentation in the coming weeks.
>
>
> Looks good - one thing missing is that needs to explain how to get the code.
> That can be a maven build to the Apache snapshot repository.
>
> In POM.xml:
>
> guava-osgi -- is there anything in there that isn't in
> com.googlecode.guava:guava?
>
>
> Please remove the <repositories> section. The build needs to depend on
> only maven central and on Apache repsoitories (but only for development use
> of SNAPSHOTS).
>
> You should be able to do that as a committer and the configuration is set up
> by the jena-parent POM.
>
> Give it a go!
>
> Then it's time to send email to users@ to announce it - if you can do that
> soon, it'll give time for initial comments to come back before the end of
> GSoC which woulc be perfect timing.
>
> Andy
>
>>
>> Best regards,
>> Ying Jiang
>
>
Re: GSoC: CSV Property Tables.
Posted by Andy Seaborne <an...@apache.org>.
On 03/08/14 11:41, Ying Jiang wrote:
> Hi Andy,
>
> Thanks for your instructions!
>
> Last week, I made the documentation of this GSoC project (jena-csv):
> - http://jena.staging.apache.org/documentation/csv/
> - http://jena.staging.apache.org/documentation/csv/get_started.html
> - http://jena.staging.apache.org/documentation/csv/design.html
> - http://jena.staging.apache.org/documentation/csv/implementation.html
>
> I'll make more tests and improve the documentation in the coming weeks.
Looks good - one thing missing is that needs to explain how to get the
code. That can be a maven build to the Apache snapshot repository.
In POM.xml:
guava-osgi -- is there anything in there that isn't in
com.googlecode.guava:guava?
Please remove the <repositories> section. The build needs to depend on
only maven central and on Apache repsoitories (but only for development
use of SNAPSHOTS).
You should be able to do that as a committer and the configuration is
set up by the jena-parent POM.
Give it a go!
Then it's time to send email to users@ to announce it - if you can do
that soon, it'll give time for initial comments to come back before the
end of GSoC which woulc be perfect timing.
Andy
>
> Best regards,
> Ying Jiang
Re: GSoC: CSV Property Tables.
Posted by Ying Jiang <jp...@gmail.com>.
Hi Andy,
Thanks for your instructions!
Last week, I made the documentation of this GSoC project (jena-csv):
- http://jena.staging.apache.org/documentation/csv/
- http://jena.staging.apache.org/documentation/csv/get_started.html
- http://jena.staging.apache.org/documentation/csv/design.html
- http://jena.staging.apache.org/documentation/csv/implementation.html
I'll make more tests and improve the documentation in the coming weeks.
Best regards,
Ying Jiang
On Tue, Jul 29, 2014 at 6:59 PM, Andy Seaborne <an...@apache.org> wrote:
> On 27/07/14 16:20, Ying Jiang wrote:
>>
>> Hi Andy,
>>
>> Thanks for your comments!
>>
>> I just submitted the code of the csv2rdf tool. It's based on
>> CmdLangParse, because parsing is the first step for transforming.
>> csv2rdf inherits all of the command line arguments (for parsing) from
>> CmdLangParse, besides the new "-dest=file" argument for the
>> destination output file. You can try out:
>> java -cp ... riotcmd.csv2rdf -dest=test.ntriples
>> src/test/resources/test.csv
>>
>> The warnings you pointed out have been fixed already. It's now clean
>> for packaging jena-csv.
>>
>> As to the release, do you mean releasing jena-csv itself, or the whole
>> jena
>
>
> jena-csv on it's own. It means it can be released more frequently and
> out-of-step with the rest of Jena. As we are (trying) to release only 6
> monthly for the main distribution, (!!), coupling now does not work.
>
>
>> (i.e. recent [VOTE] Release Jena 2.12.0 and Fuseki 1.1.0 )?
>> Actually, I've made some code in jena-arq (e.g. LangCSV in RIOT),
>> while more other code reside in jena-csv (e.g. PropertyTable, csv2rdf
>> tool). jena-csv depends on jena-arq. Shall I integrate jena-csv into
>> jena-arq, or just leave jena-csv a separate module alone to release?
>
>
> For now, I'd leave it separate.
>
> The changes to jena-arq should have been picked up for 2.12.0.
>
> Hopefully, you can switch to using a released Jena - the POM is using
> 2.12.0-SNAPSHOT so it'll be good for the 2.12.0 release.
>
>
>> PropertyTable and its implementations now have good test coverage.
>> However, other tests are still under development, including some unit
>> tests and the tests for the real world csv data. I'll make it more
>> sufficiently complete. In the remaining weeks, I can also compose the
>> documentation you mentioned. After that, I think it's OK to release it
>> to the world. In short, I believe I can go with the plan. Thanks a lot
>> for your help during the project!
>
>
> Great!
>
> Andy
>
>
>>
>> Best,
>> Ying Jiang
>>
>> [1]
>> https://svn.apache.org/repos/asf/jena/Experimental/jena-csv/src/main/java/riotcmd/csv2rdf.java
>>
>> On Thu, Jul 24, 2014 at 8:47 PM, Andy Seaborne <an...@apache.org> wrote:
>>>
>>> Ying,
>>>
>>> jena-csv is looking good. Hows the csv2rdf tool coming along?
>>>
>>> Using the StageGenerator route is OK; if it were OpExecutor you could do
>>> some filtering as well but without a value based index (a general comment
>>> about Jena stores - not CSv specific) it wil no tmake a lot of
>>> performance
>>> difference. There is material in the join engine rewrite "quack" but
>>> it's
>>> not ready. I don't see any major thing that you've done is not
>>> applicable
>>> at some later (post-summer) date for someone interested to move it on.
>>>
>>> Looking at the time left, there is a couple weeks and then the room for
>>> manoeuvre.
>>>
>>> I think that the most important thing to get this out to people to use.
>>> do is to release it on the world!
>>>
>>> That means:
>>>
>>> 1/ Documentation for use
>>>
>>> "Getting started page"
>>> A page for full details.
>>> A page about the code (?)
>>>
>>> 2/ An Apache release.
>>>
>>> If everyone is OK with this, I suggest that this is a release in it's own
>>> right with it's own VOTE, etc etc. It's good expereince to
>>>
>>> Take a look at our release process documentation to know what's involved:
>>>
>>> https://cwiki.apache.org/confluence/display/JENA/Release+Process
>>>
>>> and reply quite soon about whether I'vemissed anything that needs doing
>>> before a release and whether this plan works for you. I'm very open to
>>> doing something different if you suggest something else. The goal is to
>>> get
>>> stuff to other people - there can be various different ways to do that.
>>>
>>> Andy
>>>
>>> Comments:
>>>
>>> I checked out a clean copy
>>>
>>> 1/ "mvn clean test" and I got a test failure.
>>>
>>> ------------
>>> Running org.apache.jena.propertytable.TS_PropertyTable
>>> log4j:ERROR Could not read configuration file from URL
>>> [file:src/test/resources/log4j.properties].
>>> java.io.FileNotFoundException: src/test/resources/log4j.properties (No
>>> such
>>> file or directory)
>>> ------------
>>> Missing file?
>>>
>>>
>>> 2/ fakign that file I hthen got some output from the tests:
>>>
>>>> 1(?x
>>>> <file:///home/afs/Projects/jena-csv/src/test/resources/test.csv#Town>
>>>> ?townName)
>>>
>>> (?x
>>>
>>> <file:///home/afs/Projects/jena-csv/src/test/resources/test.csv#Population>
>>> ?pop)
>>> <=1 (?x
>>>
>>> <file:///home/afs/Projects/jena-csv/src/test/resources/test.csv#Predicate%20With%20Space>
>>> "PredicateWithSpace2")
>>>
>>> Are there by any chance some stray System.out.println in the code? :-)
>>>
>>> 3/ I also got some javadoc warnings when packaging.
>>>
>>> e.g.
>>> [WARNING]
>>>
>>> /home/afs/Projects/jena-csv/src/main/java/org/apache/jena/propertytable/PropertyTable.java:45:
>>> warning - @param argument "column," is not a parameter name.
>>>
>>> 4/ Is the test coverage sufficiently complete?
>
>
Re: GSoC: CSV Property Tables.
Posted by Andy Seaborne <an...@apache.org>.
On 27/07/14 16:20, Ying Jiang wrote:
> Hi Andy,
>
> Thanks for your comments!
>
> I just submitted the code of the csv2rdf tool. It's based on
> CmdLangParse, because parsing is the first step for transforming.
> csv2rdf inherits all of the command line arguments (for parsing) from
> CmdLangParse, besides the new "-dest=file" argument for the
> destination output file. You can try out:
> java -cp ... riotcmd.csv2rdf -dest=test.ntriples src/test/resources/test.csv
>
> The warnings you pointed out have been fixed already. It's now clean
> for packaging jena-csv.
>
> As to the release, do you mean releasing jena-csv itself, or the whole
> jena
jena-csv on it's own. It means it can be released more frequently and
out-of-step with the rest of Jena. As we are (trying) to release only 6
monthly for the main distribution, (!!), coupling now does not work.
> (i.e. recent [VOTE] Release Jena 2.12.0 and Fuseki 1.1.0 )?
> Actually, I've made some code in jena-arq (e.g. LangCSV in RIOT),
> while more other code reside in jena-csv (e.g. PropertyTable, csv2rdf
> tool). jena-csv depends on jena-arq. Shall I integrate jena-csv into
> jena-arq, or just leave jena-csv a separate module alone to release?
For now, I'd leave it separate.
The changes to jena-arq should have been picked up for 2.12.0.
Hopefully, you can switch to using a released Jena - the POM is using
2.12.0-SNAPSHOT so it'll be good for the 2.12.0 release.
> PropertyTable and its implementations now have good test coverage.
> However, other tests are still under development, including some unit
> tests and the tests for the real world csv data. I'll make it more
> sufficiently complete. In the remaining weeks, I can also compose the
> documentation you mentioned. After that, I think it's OK to release it
> to the world. In short, I believe I can go with the plan. Thanks a lot
> for your help during the project!
Great!
Andy
>
> Best,
> Ying Jiang
>
> [1] https://svn.apache.org/repos/asf/jena/Experimental/jena-csv/src/main/java/riotcmd/csv2rdf.java
>
> On Thu, Jul 24, 2014 at 8:47 PM, Andy Seaborne <an...@apache.org> wrote:
>> Ying,
>>
>> jena-csv is looking good. Hows the csv2rdf tool coming along?
>>
>> Using the StageGenerator route is OK; if it were OpExecutor you could do
>> some filtering as well but without a value based index (a general comment
>> about Jena stores - not CSv specific) it wil no tmake a lot of performance
>> difference. There is material in the join engine rewrite "quack" but it's
>> not ready. I don't see any major thing that you've done is not applicable
>> at some later (post-summer) date for someone interested to move it on.
>>
>> Looking at the time left, there is a couple weeks and then the room for
>> manoeuvre.
>>
>> I think that the most important thing to get this out to people to use.
>> do is to release it on the world!
>>
>> That means:
>>
>> 1/ Documentation for use
>>
>> "Getting started page"
>> A page for full details.
>> A page about the code (?)
>>
>> 2/ An Apache release.
>>
>> If everyone is OK with this, I suggest that this is a release in it's own
>> right with it's own VOTE, etc etc. It's good expereince to
>>
>> Take a look at our release process documentation to know what's involved:
>>
>> https://cwiki.apache.org/confluence/display/JENA/Release+Process
>>
>> and reply quite soon about whether I'vemissed anything that needs doing
>> before a release and whether this plan works for you. I'm very open to
>> doing something different if you suggest something else. The goal is to get
>> stuff to other people - there can be various different ways to do that.
>>
>> Andy
>>
>> Comments:
>>
>> I checked out a clean copy
>>
>> 1/ "mvn clean test" and I got a test failure.
>>
>> ------------
>> Running org.apache.jena.propertytable.TS_PropertyTable
>> log4j:ERROR Could not read configuration file from URL
>> [file:src/test/resources/log4j.properties].
>> java.io.FileNotFoundException: src/test/resources/log4j.properties (No such
>> file or directory)
>> ------------
>> Missing file?
>>
>>
>> 2/ fakign that file I hthen got some output from the tests:
>>
>>> 1(?x <file:///home/afs/Projects/jena-csv/src/test/resources/test.csv#Town>
>>> ?townName)
>> (?x
>> <file:///home/afs/Projects/jena-csv/src/test/resources/test.csv#Population>
>> ?pop)
>> <=1 (?x
>> <file:///home/afs/Projects/jena-csv/src/test/resources/test.csv#Predicate%20With%20Space>
>> "PredicateWithSpace2")
>>
>> Are there by any chance some stray System.out.println in the code? :-)
>>
>> 3/ I also got some javadoc warnings when packaging.
>>
>> e.g.
>> [WARNING]
>> /home/afs/Projects/jena-csv/src/main/java/org/apache/jena/propertytable/PropertyTable.java:45:
>> warning - @param argument "column," is not a parameter name.
>>
>> 4/ Is the test coverage sufficiently complete?
Re: GSoC: CSV Property Tables.
Posted by Ying Jiang <jp...@gmail.com>.
Hi Andy,
Thanks for your comments!
I just submitted the code of the csv2rdf tool. It's based on
CmdLangParse, because parsing is the first step for transforming.
csv2rdf inherits all of the command line arguments (for parsing) from
CmdLangParse, besides the new "-dest=file" argument for the
destination output file. You can try out:
java -cp ... riotcmd.csv2rdf -dest=test.ntriples src/test/resources/test.csv
The warnings you pointed out have been fixed already. It's now clean
for packaging jena-csv.
As to the release, do you mean releasing jena-csv itself, or the whole
jena (i.e. recent [VOTE] Release Jena 2.12.0 and Fuseki 1.1.0 )?
Actually, I've made some code in jena-arq (e.g. LangCSV in RIOT),
while more other code reside in jena-csv (e.g. PropertyTable, csv2rdf
tool). jena-csv depends on jena-arq. Shall I integrate jena-csv into
jena-arq, or just leave jena-csv a separate module alone to release?
PropertyTable and its implementations now have good test coverage.
However, other tests are still under development, including some unit
tests and the tests for the real world csv data. I'll make it more
sufficiently complete. In the remaining weeks, I can also compose the
documentation you mentioned. After that, I think it's OK to release it
to the world. In short, I believe I can go with the plan. Thanks a lot
for your help during the project!
Best,
Ying Jiang
[1] https://svn.apache.org/repos/asf/jena/Experimental/jena-csv/src/main/java/riotcmd/csv2rdf.java
On Thu, Jul 24, 2014 at 8:47 PM, Andy Seaborne <an...@apache.org> wrote:
> Ying,
>
> jena-csv is looking good. Hows the csv2rdf tool coming along?
>
> Using the StageGenerator route is OK; if it were OpExecutor you could do
> some filtering as well but without a value based index (a general comment
> about Jena stores - not CSv specific) it wil no tmake a lot of performance
> difference. There is material in the join engine rewrite "quack" but it's
> not ready. I don't see any major thing that you've done is not applicable
> at some later (post-summer) date for someone interested to move it on.
>
> Looking at the time left, there is a couple weeks and then the room for
> manoeuvre.
>
> I think that the most important thing to get this out to people to use.
> do is to release it on the world!
>
> That means:
>
> 1/ Documentation for use
>
> "Getting started page"
> A page for full details.
> A page about the code (?)
>
> 2/ An Apache release.
>
> If everyone is OK with this, I suggest that this is a release in it's own
> right with it's own VOTE, etc etc. It's good expereince to
>
> Take a look at our release process documentation to know what's involved:
>
> https://cwiki.apache.org/confluence/display/JENA/Release+Process
>
> and reply quite soon about whether I'vemissed anything that needs doing
> before a release and whether this plan works for you. I'm very open to
> doing something different if you suggest something else. The goal is to get
> stuff to other people - there can be various different ways to do that.
>
> Andy
>
> Comments:
>
> I checked out a clean copy
>
> 1/ "mvn clean test" and I got a test failure.
>
> ------------
> Running org.apache.jena.propertytable.TS_PropertyTable
> log4j:ERROR Could not read configuration file from URL
> [file:src/test/resources/log4j.properties].
> java.io.FileNotFoundException: src/test/resources/log4j.properties (No such
> file or directory)
> ------------
> Missing file?
>
>
> 2/ fakign that file I hthen got some output from the tests:
>
>>1(?x <file:///home/afs/Projects/jena-csv/src/test/resources/test.csv#Town>
>> ?townName)
> (?x
> <file:///home/afs/Projects/jena-csv/src/test/resources/test.csv#Population>
> ?pop)
> <=1 (?x
> <file:///home/afs/Projects/jena-csv/src/test/resources/test.csv#Predicate%20With%20Space>
> "PredicateWithSpace2")
>
> Are there by any chance some stray System.out.println in the code? :-)
>
> 3/ I also got some javadoc warnings when packaging.
>
> e.g.
> [WARNING]
> /home/afs/Projects/jena-csv/src/main/java/org/apache/jena/propertytable/PropertyTable.java:45:
> warning - @param argument "column," is not a parameter name.
>
> 4/ Is the test coverage sufficiently complete?
Re: GSoC: CSV Property Tables.
Posted by Andy Seaborne <an...@apache.org>.
Ying,
jena-csv is looking good. Hows the csv2rdf tool coming along?
Using the StageGenerator route is OK; if it were OpExecutor you could do
some filtering as well but without a value based index (a general
comment about Jena stores - not CSv specific) it wil no tmake a lot of
performance difference. There is material in the join engine rewrite
"quack" but it's not ready. I don't see any major thing that you've
done is not applicable at some later (post-summer) date for someone
interested to move it on.
Looking at the time left, there is a couple weeks and then the room for
manoeuvre.
I think that the most important thing to get this out to people to use.
do is to release it on the world!
That means:
1/ Documentation for use
"Getting started page"
A page for full details.
A page about the code (?)
2/ An Apache release.
If everyone is OK with this, I suggest that this is a release in it's
own right with it's own VOTE, etc etc. It's good expereince to
Take a look at our release process documentation to know what's involved:
https://cwiki.apache.org/confluence/display/JENA/Release+Process
and reply quite soon about whether I'vemissed anything that needs doing
before a release and whether this plan works for you. I'm very open to
doing something different if you suggest something else. The goal is to
get stuff to other people - there can be various different ways to do that.
Andy
Comments:
I checked out a clean copy
1/ "mvn clean test" and I got a test failure.
------------
Running org.apache.jena.propertytable.TS_PropertyTable
log4j:ERROR Could not read configuration file from URL
[file:src/test/resources/log4j.properties].
java.io.FileNotFoundException: src/test/resources/log4j.properties (No
such file or directory)
------------
Missing file?
2/ fakign that file I hthen got some output from the tests:
>1(?x
<file:///home/afs/Projects/jena-csv/src/test/resources/test.csv#Town>
?townName)
(?x
<file:///home/afs/Projects/jena-csv/src/test/resources/test.csv#Population>
?pop)
<=1 (?x
<file:///home/afs/Projects/jena-csv/src/test/resources/test.csv#Predicate%20With%20Space>
"PredicateWithSpace2")
Are there by any chance some stray System.out.println in the code? :-)
3/ I also got some javadoc warnings when packaging.
e.g.
[WARNING]
/home/afs/Projects/jena-csv/src/main/java/org/apache/jena/propertytable/PropertyTable.java:45:
warning - @param argument "column," is not a parameter name.
4/ Is the test coverage sufficiently complete?
Re: GSoC: CSV Property Tables.
Posted by Ying Jiang <jp...@gmail.com>.
Dear Andy,
I made an implementation of PropertyTable using a two dimension array
[1]. The old HashMap implementation is not deleted, which I'll use for
the performance tests. I hope the array implementation can go well
with less memory consumption and querying response time, especially
for the query-a-whole case.
Best regards,
Ying Jiang
[1] https://svn.apache.org/repos/asf/jena/Experimental/jena-csv/src/main/java/com/hp/hpl/jena/propertytable/impl/PropertyTableArrayImpl.java
On Wed, Jul 9, 2014 at 10:05 AM, Ying Jiang <jp...@gmail.com> wrote:
> Dear Andy,
>
> Thanks a lot! I've been studying the source code of SPARQL algebra
> following your instructions these days.
>
> I've made a try-out of the optimization of querying a whole row in the
> property table. The following query pattern can be transformed into a
> row querying, without generating triples:
> ?x :prop1 ?v .
> ?x :prop2 ?w .
> ...
>
> It's made by using the extension point of StageGenerator, because it's
> now just concerned with BasicPattern. The detailed workflow goes in
> this way:
> 1. Split the incoming BasicPattern by subjects, (i.e. it becomes
> multiple sub BasicPatterns grouped by the same subjects. (see
> QueryIterPropertyTable [1])
> 2. For each sub BasicPattern,
> if the triple size within is greater than 1 (i.e. at least 2 triples),
> it's turned into a row querying, and processed by
> QueryIterPropertyTableRow [2]
> else if it contains only 1 triple, it goes for the traditional triple
> querying by graph.graphBaseFind()
>
> The code have been committed. I'll think about other optimization ways
> and test with real-life large CSV data you provided later on. Any
> suggestion is very welcome!
>
> Best regards,
> Ying Jiang
>
> [1] https://svn.apache.org/repos/asf/jena/Experimental/jena-csv/src/main/java/com/hp/hpl/jena/propertytable/impl/QueryIterPropertyTable.java
> [2] https://svn.apache.org/repos/asf/jena/Experimental/jena-csv/src/main/java/com/hp/hpl/jena/propertytable/impl/QueryIterPropertyTableRow.java
>
> On Mon, Jun 30, 2014 at 9:47 PM, Andy Seaborne <an...@seaborne.org> wrote:
>> Hi Ying,
>>
>>
>>
>>> ---- Quote from the project proposal:
>>> Accessing from SPARQL via Graph.find() will work, but it's not ideal.
>>> Some optimizations can be done for processing a SPARQL basic graph
>>> pattern. More explicitly, in the method of OpExecutor.execute(OpBGP,
>>> ...), when the target for the query is a GraphPropertyTable, it can
>>> get a whole row, or rows, of the table data and match the pattern with
>>> the bindings.
>>> ----
>>> Can you show me a concrete example of how "getting a whole row, or
>>> rows" can improve the query of (e.g. SUM/COUNT)? Suppose we have the
>>> data:
>>> ----
>>> Town,Population
>>> Southton,123000
>>> Northville,654000
>>> ----
>>> If we want to query the total population of the (two or all) towns,
>>> how does changing OpExecutor.execute(OpBGP, ...) help? Here's the
>>> query string:
>>> ----
>>> SELECT (sum(?pop) AS ?totalPop) {
>>> { GRAPH <file:///c:/town.csv> {
>>> ?x :Popuation ?pop .
>>> }
>>> }
>>> ----
>>> If the above example is not appropriate for for this case (2.3), could
>>> you please show me another one. I'm very grateful if you can help me
>>> with this question before your holiday.
>>
>>
>>
>> OpExecutor is the way SPARQL is evaluated. The query is parsed, truned into
>> SPARQL algebra (the Op* classes), optimized (yet more Op* classes and
>> rearranging the ones in the query) then evaluated. There is one
>> execute(Op...) for each Op operator in the SPARQL algebra.
>>
>> OpBGP is the operator for basic graph patterns, a basic graph pattern is a
>> sequence of triple patterns. The default is iteratively use Graph.find and
>> in the case of a single triple pattern
>>
>> The order of triple patterns matters for performance (you get the same
>> answer regardless of order of execution, better orders of the pattern
>> traverse less negative/rejected
>>
>> Consider, in some general data, not CSV related:
>>
>> ?x :predicate ?value . ## Triple pattern 1
>> ?x :other 123 . ## Triple pattern 2
>>
>> and suppose there are 1000 subjects each with :predicate and :other as
>> triples. And suppose there is one with predicate-value ":other 123".
>>
>> If you do the execution in the order above it's 1000 possibilities from the
>> first triple pattern. If the iterative OpExecutor strategy is used, its then
>> a lots of tests of "?x :other 123" for each possible ?x.
>>
>> But if the order is reversed, there is one match for "?x :other 123 ." then
>> one access for "?x :predicate ?value ." (?x is now a fixed value).
>>
>> In the default setup (in-memory and any storage that does not override and
>> add something better, (see StageGeneratorGeneric) the fixed reording is done
>> based on finding the most grounded patterns. It should choose to do ":other
>> 123" first.
>>
>> In the CSV case, there may be better things to do that are specific to CSV
>> and the property table storage. This needs investigation and design before
>> implementation; I don't have a definite answer.
>>
>> It will depend a lot on the datastructures for the property tables - this is
>> where all the pieces fit together.
>>
>> One possibility is avoiding generating triples. If the pattern is
>>
>> ?x :prop1 ?v .
>> ?x :prop2 ?w .
>>
>> what 's needed is bindings (a row in SPARQL results) for (?x,?v,?w). There
>> is not need to create a lot of triples via find().
>>
>> Other patterns that might have optimizations are where the subject is not
>> the same in each pattern:
>>
>> ?x :prop1 ?y .
>> ?y :prop2 ?w .
>>
>> Better execution of aggregates like SUM is also possible (in your example,
>> it could be a matter of traversing the "population" column adding the
>> values, avoiding creating any triples) but I'm thinking of basic graph
>> patterns for the moment.
>>
>> Real CSVs files are often denormalised tables - lots of repetition from row
>> to row. This might be exploitable.
>>
>> There are some real worlds examples documented in:
>>
>> http://www.w3.org/TR/csvw-ucr/
>>
>> with links to the real CSV files.
>>
>> To investigate, its probably a good idea to understand how OpExecutor works
>> and also to thin of some patterns and see what might be done. More
>> complicated data (more columns, more rows)
>>
>>> Is there a tool in the Jena code base for parsing String to Jena
>>> Literal for different data types? For example:
>>> 1) "65000" -> "65000"^^<http://www.w3.org/2001/XMLSchema#integer>
>>> 2) "65000.123" -> "65000.123"^^<http://www.w3.org/2001/XMLSchema#double>
>>> 3) "true" -> "true"^^<http://www.w3.org/2001/XMLSchema#boolean>
>>> There would be more advanced features like parsing date/time to
>>> xsd:date or xsd:dateTime for CSV values.
>>> The NodeFactory.createLiteral() method does not provide these features.
>>
>>
>> There is the parse for Turtle and related langauges (although 65000.123 will
>> be a xsd:decimal). See TokenizerText.
>>
>> You may be better off with your own code, using Long.decode pr
>> Long.parseLong, Double.decode or Double.parseDouble, because real data is
>> messy and it could be useful to clean up the input first, for example,
>> remove "," [*]
>>
>> Andy
>>
>> [*] Caution - locale sensitive!
>>