You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@jena.apache.org by Markus Neumann <mn...@meteomatics.com> on 2018/10/10 07:15:56 UTC
Re: Building a spatial index with jena/fuseki
Hi,
just wanted to confirm that version 3.9.0 solves the spatial indexing issue:
$ java -cp /home/process/fuseki-3.9.0/fuseki-server.jar jena.spatialindexer --desc=/home/process/fuseki-3.9.0/run/configuration/mm.ttl
...
INFO 539607479 (490106 per second) properties indexed
Spatial queries work like a charm now. Many thanks for the quick fix.
Kind regards
Markus
On 2018/09/19 14:25:41, Marco Neumann <m....@gmail.com> wrote:
> indeed Andy, I got confused with the naming here. same in the code>
> actually. it might be a good idea to come up with a new name for TDB2 to>
> reduce scope for confusion.>
>
>
> On Wed, Sep 19, 2018 at 1:47 PM Andy Seaborne <an...@apache.org> wrote:>
>
> >>
> >>
> > On 19/09/18 11:24, Marco Neumann wrote:>
> > > On Wed, Sep 19, 2018 at 11:04 AM Andy Seaborne <an...@apache.org> wrote:>
> > >>
> > >>>
> > >>>
> > >> On 17/09/18 15:26, ajs6f wrote:>
> > >>> Okay, I think we've got it. Nice debugging, Markus!>
> > >>>>
> > >>> If you look at the line of the spatial indexer code that is throwing an>
> > >> exception:>
> > >>>>
> > >>>>
> > >>>
> > https://github.com/apache/jena/blob/master/jena-spatial/src/main/java/jena/spatialindexer.java#L110>
> > >>>>
> > >>> indeed, it does not start a transaction against the dataset before>
> > >> starting to read quads to index them. Now, in TDB1 this isn't>
> > inherently a>
> > >> problem-- it's possible to work against a dataset without transactions.>
> > >> (Although it's generally not a good idea at all.) But IIRC, TDB2>
> > requires>
> > >> transactions. If that's true (and Andy can verify that for us) then>
> > this is>
> > >> a bug and we can fix it in a straightforward way.>
> > >>>>
> > >>> Andy, can you confirm that diagnosis?>
> > >>>
> > >> Yes - TDB2 only works transactionally.>
> > >>>
> > >>>
> > > Andy-->
> > >>
> > > I am just looking at the file system, I noticed that there is no binary>
> > > difference between tdb and tdb2 indexes. would it be fair to describe>
> > tdb2>
> > > as a tdb+txn with different access methods?>
> >>
> > Are you looking at tdblaod vs tdbloader2 (unfortunately named>
> >>
> > The TBD2 loader is tdb2.tdbloader.>
> >>
> > The disk format for TDB2 is different - it is pushed down one level plus>
> > the node table and the indexes are different byte structures. The>
> > indexes are nearly the same but not quite (there is no link field in the>
> > leaf blocks; tere is an extra control file ".bpt" per index).>
> >>
> > The journal in TDB1 can grow big, the journal in TDB2 is small - a few>
> > hundrad bytes maximum.>
> >>
> > And (by accident!) you can have both in the same directory. This is not>
> > a good idea.>
> >>
> > Andy>
> >>
> >>
> > TDB2:>
> > DB2:>
> > Data-0001/ tdb.lock>
> >>
> > DB2/Data-0001:>
> > GOSP.bpt GSPO.bpt OSP.bpt POSG.dat SPO.dat>
> > GOSP.dat GSPO.dat OSP.dat POSG.idn SPOG.bpt>
> > GOSP.idn GSPO.idn OSPG.bpt POS.idn SPOG.dat>
> > GPOS.bpt journal.jrnl OSPG.dat prefixes.bpt SPOG.idn>
> > GPOS.dat nodes.bpt OSPG.idn prefixes.dat SPO.idn>
> > GPOS.idn nodes.dat OSP.idn prefixes-data.bdf tdb.lock>
> > GPU.bpt nodes-data.bdf POS.bpt prefixes-data.obj>
> > GPU.dat nodes-data.obj POS.dat prefixes.idn>
> > GPU.idn nodes.idn POSG.bpt SPO.bpt>
> >>
> > TDB1:>
> > DB1:>
> > GOSP.dat journal.jrnl OSPG.idn prefix2id.dat SPOG.dat>
> > GOSP.idn node2id.dat OSP.idn prefix2id.idn SPOG.idn>
> > GPOS.dat node2id.idn POS.dat prefixes.dat SPO.idn>
> > GPOS.idn nodes.dat POSG.dat prefixIdx.dat stats.opt>
> > GSPO.dat OSP.dat POSG.idn prefixIdx.idn>
> > GSPO.idn OSPG.dat POS.idn SPO.dat>
> >>
> >>
> > >>
> > >>
> > >>>
> > >>> If so, I'll open a ticket and write a PR to use a Txn method to fix>
> > this.>
> > >>>
> > >> which ajs6f has merged into the development system.>
> > >>>
> > >>>>
> > >>> ajs6f>
> > >>>>
> > >>>> On Sep 17, 2018, at 10:14 AM, Markus Neumann <>
> > mneumann@meteomatics.com>>
> > >> wrote:>
> > >>>>>
> > >>>> It seems I made a mess with absolute and relative paths all together.>
> > >> So I created a minimal setup for testing:>
> > >>>> (I include the file-contents at the bottom)>
> > >>>>>
> > >>>> $ export JENA_HOME=/srv/linked_data_store/apache-jena-3.8.0>
> > >>>> $ export PATH=$PATH:$JENA_HOME/bin>
> > >>>> $ tdb2.tdbloader>
> > >> --desc=/srv/linked_data_store/fuseki-server/run/configuration/mm.ttl>
> > >> /srv/linked_data_store/rdf_python_connector/turtle_files/mm-loc-mini.ttl>
> > >> /srv/linked_data_store/rdf_python_connector/turtle_files/mm-rdf.ttl>
> > >>>> 13:50:54 INFO loader :: Loader = LoaderPhased>
> > >>>> 13:50:54 INFO loader :: Data:>
> > >>>
> > /srv/linked_data_store/rdf_python_connector/turtle_files/mm-loc-mini.ttl:>
> > >> 13 mm-loc-mini.ttl 0.06s (Avg: 232)>
> > >>>> 13:50:54 INFO loader :: Data:>
> > >> /srv/linked_data_store/rdf_python_connector/turtle_files/mm-rdf.ttl: 50>
> > >> mm-rdf.ttl 0.01s (Avg: 6,250)>
> > >>>> 13:50:55 INFO loader :: Finish - index SPO>
> > >>>> 13:50:55 INFO loader :: Start replay index SPO>
> > >>>> 13:50:55 INFO loader :: Index set: SPO => SPO->POS,>
> > >> SPO->OSP>
> > >>>> 13:50:55 INFO loader :: Index set: SPO => SPO->POS,>
> > >> SPO->OSP [63 items, 0.0 seconds]>
> > >>>> 13:50:55 INFO loader :: Finish - index POS>
> > >>>> 13:50:55 INFO loader :: Finish - index OSP>
> > >>>> $ java -cp>
> > >>>
> > /srv/linked_data_store/fuseki-server/fuseki-server.jar:/srv/linked_data_store/apache-jena-3.8.0/lib/*>
> > >> jena.spatialindexer>
> > >> --desc=/srv/linked_data_store/fuseki-server/run/configuration/mm.ttl>
> > >>>> SLF4J: Class path contains multiple SLF4J bindings.>
> > >>>> SLF4J: Found binding in>
> > >>>
> > [jar:file:/srv/linked_data_store/fuseki-server/fuseki-server.jar!/org/slf4j/impl/StaticLoggerBinder.class]>
> > >>>> SLF4J: Found binding in>
> > >>>
> > [jar:file:/srv/linked_data_store/apache-jena-3.8.0/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]>
> > >>>> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an>
> > >> explanation.>
> > >>>> SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]>
> > >>>> org.apache.jena.dboe.transaction.txn.TransactionException: Not in a>
> > >> transaction>
> > >>>> at>
> > >>>
> > org.apache.jena.dboe.transaction.txn.TransactionalComponentLifecycle.checkTxn(TransactionalComponentLifecycle.java:417)>
> > >>>> at>
> > >>>
> > org.apache.jena.dboe.trans.bplustree.BPlusTree.getRootRead(BPlusTree.java:159)>
> > >>>> at>
> > >>>
> > org.apache.jena.dboe.trans.bplustree.BPlusTree.iterator(BPlusTree.java:348)>
> > >>>> at>
> > >>>
> > org.apache.jena.tdb2.store.tupletable.TupleIndexRecord.all(TupleIndexRecord.java:251)>
> > >>>> at>
> > >>>
> > org.apache.jena.tdb2.store.tupletable.TupleTable.find(TupleTable.java:148)>
> > >>>> at>
> > >>>
> > org.apache.jena.tdb2.store.nodetupletable.NodeTupleTableConcrete.find(NodeTupleTableConcrete.java:161)>
> > >>>> at>
> > >>>
> > org.apache.jena.tdb2.store.nodetupletable.NodeTupleTableConcrete.find(NodeTupleTableConcrete.java:150)>
> > >>>> at>
> > >>>
> > org.apache.jena.tdb2.store.nodetupletable.NodeTupleTableConcrete.findAsNodeIds(NodeTupleTableConcrete.java:141)>
> > >>>> at>
> > org.apache.jena.tdb2.store.TripleTable.find(TripleTable.java:64)>
> > >>>> at>
> > >>>
> > org.apache.jena.tdb2.store.DatasetGraphTDB.findInDftGraph(DatasetGraphTDB.java:104)>
> > >>>> at>
> > >>>
> > org.apache.jena.sparql.core.DatasetGraphBaseFind.findAny(DatasetGraphBaseFind.java:72)>
> > >>>> at>
> > >>>
> > org.apache.jena.sparql.core.DatasetGraphBaseFind.find(DatasetGraphBaseFind.java:50)>
> > >>>> at>
> > >>>
> > org.apache.jena.sparql.core.DatasetGraphWrapper.find(DatasetGraphWrapper.java:152)>
> > >>>> at>
> > >>>
> > org.apache.jena.sparql.core.DatasetGraphWrapper.find(DatasetGraphWrapper.java:152)>
> > >>>> at jena.spatialindexer.exec(spatialindexer.java:110)>
> > >>>> at jena.cmd.CmdMain.mainMethod(CmdMain.java:93)>
> > >>>> at jena.cmd.CmdMain.mainRun(CmdMain.java:58)>
> > >>>> at jena.cmd.CmdMain.mainRun(CmdMain.java:45)>
> > >>>> at jena.spatialindexer.main(spatialindexer.java:53)>
> > >>>>>
> > >>>> I can get rid of the SLF4J warnings:>
> > >>>> $ java -cp /srv/linked_data_store/fuseki-server/fuseki-server.jar>
> > >> jena.spatialindexer>
> > >> --desc=/srv/linked_data_store/fuseki-server/run/configuration/mm.ttl>
> > >>>> org.apache.jena.dboe.transaction.txn.TransactionException: Not in a>
> > >> transaction>
> > >>>> at>
> > >>>
> > org.apache.jena.dboe.transaction.txn.TransactionalComponentLifecycle.checkTxn(TransactionalComponentLifecycle.java:417)>
> > >>>> at>
> > >>>
> > org.apache.jena.dboe.trans.bplustree.BPlusTree.getRootRead(BPlusTree.java:159)>
> > >>>> at>
> > >>>
> > org.apache.jena.dboe.trans.bplustree.BPlusTree.iterator(BPlusTree.java:348)>
> > >>>> at>
> > >>>
> > org.apache.jena.tdb2.store.tupletable.TupleIndexRecord.all(TupleIndexRecord.java:251)>
> > >>>> at>
> > >>>
> > org.apache.jena.tdb2.store.tupletable.TupleTable.find(TupleTable.java:148)>
> > >>>> at>
> > >>>
> > org.apache.jena.tdb2.store.nodetupletable.NodeTupleTableConcrete.find(NodeTupleTableConcrete.java:161)>
> > >>>> at>
> > >>>
> > org.apache.jena.tdb2.store.nodetupletable.NodeTupleTableConcrete.find(NodeTupleTableConcrete.java:150)>
> > >>>> at>
> > >>>
> > org.apache.jena.tdb2.store.nodetupletable.NodeTupleTableConcrete.findAsNodeIds(NodeTupleTableConcrete.java:141)>
> > >>>> at>
> > org.apache.jena.tdb2.store.TripleTable.find(TripleTable.java:64)>
> > >>>> at>
> > >>>
> > org.apache.jena.tdb2.store.DatasetGraphTDB.findInDftGraph(DatasetGraphTDB.java:104)>
> > >>>> at>
> > >>>
> > org.apache.jena.sparql.core.DatasetGraphBaseFind.findAny(DatasetGraphBaseFind.java:72)>
> > >>>> at>
> > >>>
> > org.apache.jena.sparql.core.DatasetGraphBaseFind.find(DatasetGraphBaseFind.java:50)>
> > >>>> at>
> > >>>
> > org.apache.jena.sparql.core.DatasetGraphWrapper.find(DatasetGraphWrapper.java:152)>
> > >>>> at>
> > >>>
> > org.apache.jena.sparql.core.DatasetGraphWrapper.find(DatasetGraphWrapper.java:152)>
> > >>>> at jena.spatialindexer.exec(spatialindexer.java:110)>
> > >>>> at jena.cmd.CmdMain.mainMethod(CmdMain.java:93)>
> > >>>> at jena.cmd.CmdMain.mainRun(CmdMain.java:58)>
> > >>>> at jena.cmd.CmdMain.mainRun(CmdMain.java:45)>
> > >>>> at jena.spatialindexer.main(spatialindexer.java:53)>
> > >>>>>
> > >>>> There is no other java instance running at the time when I run those>
> > >> commands.>
> > >>>> All paths are fully qualified and double checked. The destination>
> > >> directory and the Lucene directory were cleared just before.>
> > >>>>>
> > >>>> The server does start and serve the data without errors or warnings in>
> > >> the log, but spatial queries do not work.>
> > >>>>>
> > >>>> Does anyone see a reason for the TransactionException?>
> > >>>> Thanks and best regards>
> > >>>> Markus>
> > >>>>>
> > >>>> Attachments:>
> > >>>> /srv/linked_data_store/rdf_python_connector/turtle_files/mm-rdf.ttl:>
> > >>>> @prefix geo: <http://www.w3.org/2003/01/geo/wgs84_pos#> .>
> > >>>> @prefix rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#> .>
> > >>>> @prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#> .>
> > >>>> @prefix owl: <http://www.w3.org/2002/07/owl#> .>
> > >>>> @prefix dc: <http://purl.org/dc/elements/1.1/> .>
> > >>>> @prefix dcterms: <http://purl.org/dc/terms/> .>
> > >>>> @prefix xsd: <http://www.w3.org/2001/XMLSchema#>>
> > >>>> @prefix mm: <http://rdf.meteomatics.com/mm/> .>
> > >>>>>
> > >>>> <http://rdf.meteomatics.com/mm/>>
> > >>>> dcterms:publisher <http://api.meteomatics.com/Imprint.html> ;>
> > >>>> dcterms:title "Meteomatics RDF Vocabulary"@en .>
> > >>>>>
> > >>>> mm:location a rdf:Property ;>
> > >>>> rdfs:isDefinedBy <http://rdf.meteomatics.com/mm/> ;>
> > >>>> rdfs:label "location" ;>
> > >>>> dcterms:description "Location of a measurement"@en .>
> > >>>>>
> > >>>> mm:elevation a rdf:Property ;>
> > >>>> rdfs:isDefinedBy <http://rdf.meteomatics.com/mm/> ;>
> > >>>> rdfs:label "elevation" ;>
> > >>>> dcterms:description "Elevation of a location"@en .>
> > >>>>>
> > >>>> mm:validdate a rdf:Property ;>
> > >>>> rdfs:isDefinedBy <http://rdf.meteomatics.com/mm/> ;>
> > >>>> rdfs:label "validdate" ;>
> > >>>> dcterms:description "Temporal information describing when a>
> > >> Measurement is/was valid"@en .>
> > >>>>>
> > >>>> mm:value a rdf:Property ;>
> > >>>> rdfs:isDefinedBy <http://rdf.meteomatics.com/mm/> ;>
> > >>>> rdfs:label "value" ;>
> > >>>> dcterms:description "The (observed or predicted) value of a>
> > >> Measurement"@en .>
> > >>>>>
> > >>>> mm:parameter a rdf:Property ;>
> > >>>> rdfs:isDefinedBy <http://rdf.meteomatics.com/mm/> ;>
> > >>>> rdfs:label "parameter" ;>
> > >>>> dcterms:description "what was measured"@en .>
> > >>>>>
> > >>>> mm:station_name a rdf:Property ;>
> > >>>> rdfs:isDefinedBy <http://rdf.meteomatics.com/mm/> ;>
> > >>>> rdfs:label "station_name" ;>
> > >>>> dcterms:description "Name of a station"@en .>
> > >>>>>
> > >>>> mm:abbreviation a rdf:Property ;>
> > >>>> rdfs:isDefinedBy <http://rdf.meteomatics.com/mm/> ;>
> > >>>> rdfs:label "abbreviation" ;>
> > >>>> dcterms:description "Abbreviation of a station name"@en .>
> > >>>>>
> > >>>> mm:didok_id a rdf:Property ;>
> > >>>> rdfs:isDefinedBy <http://rdf.meteomatics.com/mm/> ;>
> > >>>> rdfs:label "didok_id" ;>
> > >>>> dcterms:description "didok id of a station"@en .>
> > >>>>>
> > >>>> mm:unit_symbol a rdf:Property ;>
> > >>>> rdfs:isDefinedBy <http://rdf.meteomatics.com/mm/> ;>
> > >>>> rdfs:label "unit symbol" ;>
> > >>>> dcterms:description "unit symbol of a parameter"@en .>
> > >>>>>
> > >>>> mm:Measurement a rdf:Class ;>
> > >>>> rdfs:isDefinedBy <http://rdf.meteomatics.com/mm/> ;>
> > >>>> rdfs:label "Measurement" ;>
> > >>>> dcterms:description "Measurement of a specific weather parameter>
> > at>
> > >> specific location at a specific date (both future or past)" .>
> > >>>>>
> > >>>> mm:Location a rdf:Class ;>
> > >>>> rdfs:isDefinedBy <http://rdf.meteomatics.com/mm/> ;>
> > >>>> rdfs:label "Location" ;>
> > >>>> dcterms:description "A Point on the globe where we observe>
> > >> weather"@en .>
> > >>>>>
> > >>>> mm:Parameter a rdf:Class ;>
> > >>>> rdfs:isDefinedBy <http://rdf.meteomatics.com/mm/> ;>
> > >>>> rdfs:label "Parameter" ;>
> > >>>> dcterms:description "A parameter describing an observable aspect>
> > of>
> > >> the weather"@en .>
> > >>>>>
> > >>>>>
> > /srv/linked_data_store/rdf_python_connector/turtle_files/mm-loc-mini.ttl>
> > >>>> @prefix geo: <http://www.w3.org/2003/01/geo/wgs84_pos#> .>
> > >>>> @prefix geosparql: <http://www.opengis.net/ont/geosparql#> .>
> > >>>> @prefix rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#> .>
> > >>>> @prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#> .>
> > >>>> @prefix owl: <http://www.w3.org/2002/07/owl#> .>
> > >>>> @prefix dc: <http://purl.org/dc/elements/1.1/> .>
> > >>>> @prefix dcterms: <http://purl.org/dc/terms/> .>
> > >>>> @prefix xsd: <http://www.w3.org/2001/XMLSchema#> .>
> > >>>> @prefix mm: <http://rdf.meteomatics.com/mm/> .>
> > >>>> @prefix mm-measurements: <http://rdf.meteomatics.com/mm/measurements/>
> > >>
> > >> .>
> > >>>> @prefix mm-locations: <http://rdf.meteomatics.com/mm/locations/> .>
> > >>>>>
> > >>>>>
> > >>>> mm-locations:8500015 a mm:Location ;>
> > >>>> a geosparql:Geometry ;>
> > >>>> mm:dido
[message truncated...]