You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@jena.apache.org by Claude Warren <cl...@xenei.com> on 2015/04/01 19:02:26 UTC
Re: Apache Jena for Android
Might it make sense to use a different persistence engine on Android since
it probably will not have the huge datasets that TDB is designed for. It
might make sense to have a small store available for small devices. Just a
thought.
Claude
On Tue, Mar 31, 2015 at 9:41 PM, Sören Brunk <so...@tu-dresden.de>
wrote:
> Hi Andy,
>
> thanks for your feedback and sorry for taking so long to reply. It would
> be great to get Android support into mainline Jena and I'm willing to help.
> It might be necessary to build an extra jar package for Android though,
> similar to what jena-osgi does.
>
>>
>> Is there are anything the Jena project can do that would make the
>> conversion? There may be things that Jena does, or the way is it packaged,
>> that are inconvenient for you but really make no differnce to the project -
>> sometimes things are just the way they are because it was done that way but
>> could easily be done another way. If you have any such points, do email
>> thsilist or raise a JIRA.
>>
>
> The httpclient issue might resolve itself as soon as Jena is able to
> switch to httpclient 4.3. I've looked into the javax.xml issues in more
> detail and the good news is that all the xsd datatype classes are actually
> there. What's missing are the StAX classes (everything in java.xml.stream)
> which seem to be used in ARQ and core (for SPARQL XML result sets, RDF/XML
> and TriX) and of course in Xerces. That means it's enough to repackage the
> StAX classes from xml-apis and to modify jena-arq, jena-core and
> xercesImpl. I've changed my build accordingly.
>
> Android uses XmlPullParser for stream parsing. So another solution could
> be to replace StAX with a XmlPull based parsing but I guess that would take
> some effort.
>
>>
>> On Android, how does TDB work well? TDB uses fairly traditional file
>> handling for 32 bit machines - "direct mode" - and uses memory mapped I/O
>> for 64 bit machines - "mapped mode" - if it can detect the mode correctly.
>> Detection is not subtle, it looks in system property "java.vm.info" and
>> default to 32 bit. In mapped mode, it does try to use as much memory as
>> possible which is not friendly to co-resident apps (you can force "direct"
>> mode programmatically).
>>
>
> Since most Android devices nowadays run on a 32 bit ARM architecture I
> guess it will run almost always in direct mode.
> But I realized that TDB isn't working at all at the moment, because it
> depends on JDK classes that aren't available on Android for things like
> getting the PID. I think it is possible to replace those calls by using
> similar classes provided by the Android SDK. I will try to get it working.
> A dependency on Android specific code would probably make mainline
> integration more difficult though.
>
> Sören
>
> --
> Dipl. Inf. Sören Brunk
> Research Associate
>
> Technische Universität Dresden
> Faculty of Computer Science
> Institute for Software- and Multimedia-Technology
> Junior Professorship in Software Engineering of Ubiquitous Systems
> 01062 Dresden
>
>
--
I like: Like Like - The likeliest place on the web
<http://like-like.xenei.com>
LinkedIn: http://www.linkedin.com/in/claudewarren
Re: Apache Jena for Android
Posted by Rob Vesse <rv...@dotnetrdf.org>.
Soren
It is also worth noting that ProcessUtils is only used to provide the
dataset locking functionality which helps prevents databases being used
from multiple JVMs at a time which may be a non-issue on Android since how
likely is is that more than one app will try and use the same TDB database?
If this isn't an issue a simpler alternative would just be to disable this
feature on Android via the setting:
SystemTDB.DiskLocationMultiJvmUsagePrevention = false;
Rob
On 20/04/2015 03:57, "Andy Seaborne" <an...@apache.org> wrote:
>On 15/04/15 14:49, Sören Brunk wrote:
>> Thanks for your suggestions. I got TDB running now. It only required a
>> small change in ProcessUtils to get the process id. This adds a
>> dependency on the Android sdk though, but it is possible to add a maven
>> dependency on the sdk with "provided" scope so I think it's feasible.
>>
>> According to the log TDB runs in direct mode on my test device as well
>> as x68_64 emulator so perhaps detection doesn't work on right on Android
>> yet. What would be a good way to test performance?
>>
>> I think a JIRA is more appropriate for further discussion so I created
>> one: https://issues.apache.org/jira/browse/JENA-914
>>
>> Cheers,
>> Sören
>
>
>Hi Sören,
>
>Detection is "unsubtle" - it looks for "64" in system property
>"java.vm.info" (see SystemTDB.determineIf64Bit()). Adding a "isAndroid"
>flag would be good - what's the best way to detect Android that does not
>rely on Android code?
>
>It might be better to run in direct mode on Android anyway as mapped
>mode can end up competing for memory resource from other processes while
>direct is more controllable. It's a server-style approach where any and
>all available free RAM is used for the database.
>
>In terms of performance, for smallish datasets, or at least
>small-RAM-footprint, the performance might be similar. Mapped mode (on
>Linux) comes into it's own when queries are repeatedly doing large index
>access (e.g. heavy FILTERing).
>
> Andy
>
>
Re: Apache Jena for Android
Posted by Andy Seaborne <an...@apache.org>.
On 15/04/15 14:49, Sören Brunk wrote:
> Thanks for your suggestions. I got TDB running now. It only required a
> small change in ProcessUtils to get the process id. This adds a
> dependency on the Android sdk though, but it is possible to add a maven
> dependency on the sdk with "provided" scope so I think it's feasible.
>
> According to the log TDB runs in direct mode on my test device as well
> as x68_64 emulator so perhaps detection doesn't work on right on Android
> yet. What would be a good way to test performance?
>
> I think a JIRA is more appropriate for further discussion so I created
> one: https://issues.apache.org/jira/browse/JENA-914
>
> Cheers,
> Sören
Hi Sören,
Detection is "unsubtle" - it looks for "64" in system property
"java.vm.info" (see SystemTDB.determineIf64Bit()). Adding a "isAndroid"
flag would be good - what's the best way to detect Android that does not
rely on Android code?
It might be better to run in direct mode on Android anyway as mapped
mode can end up competing for memory resource from other processes while
direct is more controllable. It's a server-style approach where any and
all available free RAM is used for the database.
In terms of performance, for smallish datasets, or at least
small-RAM-footprint, the performance might be similar. Mapped mode (on
Linux) comes into it's own when queries are repeatedly doing large index
access (e.g. heavy FILTERing).
Andy
Re: Apache Jena for Android
Posted by Sören Brunk <so...@tu-dresden.de>.
Thanks for your suggestions. I got TDB running now. It only required a
small change in ProcessUtils to get the process id. This adds a
dependency on the Android sdk though, but it is possible to add a maven
dependency on the sdk with "provided" scope so I think it's feasible.
According to the log TDB runs in direct mode on my test device as well
as x68_64 emulator so perhaps detection doesn't work on right on Android
yet. What would be a good way to test performance?
I think a JIRA is more appropriate for further discussion so I created
one: https://issues.apache.org/jira/browse/JENA-914
Cheers,
Sören
On 02.04.2015 09:47, Andy Seaborne wrote:
> You can configure the footprint of TDB in 32 bit "direct" mode for
> smaller datasets:
>
> http://jena.staging.apache.org/documentation/tdb/store-parameters.html
>
> Andy
>
> On 01/04/15 22:38, Stian Soiland-Reyes wrote:
>> +1 - and how well does the memory mapping of TDB work on Dalvik?
>>
>>
>> Would jena-sdb work with SQLite?
>>
>> https://developer.android.com/guide/topics/data/data-storage.html#db
>>
>> You would probably need https://github.com/SQLDroid/SQLDroid so you
>> get a JDBC driver.
>>
>>
>> On 1 April 2015 at 18:02, Claude Warren <cl...@xenei.com> wrote:
>>> Might it make sense to use a different persistence engine on Android
>>> since
>>> it probably will not have the huge datasets that TDB is designed
>>> for. It
>>> might make sense to have a small store available for small devices.
>>> Just a
>>> thought.
>>>
>>> Claude
>>>
>>> On Tue, Mar 31, 2015 at 9:41 PM, Sören Brunk
>>> <so...@tu-dresden.de>
>>> wrote:
>>>
>>>> Hi Andy,
>>>>
>>>> thanks for your feedback and sorry for taking so long to reply. It
>>>> would
>>>> be great to get Android support into mainline Jena and I'm willing
>>>> to help.
>>>> It might be necessary to build an extra jar package for Android
>>>> though,
>>>> similar to what jena-osgi does.
>>>>
>>>>>
>>>>> Is there are anything the Jena project can do that would make the
>>>>> conversion? There may be things that Jena does, or the way is it
>>>>> packaged,
>>>>> that are inconvenient for you but really make no differnce to the
>>>>> project -
>>>>> sometimes things are just the way they are because it was done
>>>>> that way but
>>>>> could easily be done another way. If you have any such points, do
>>>>> email
>>>>> thsilist or raise a JIRA.
>>>>>
>>>>
>>>> The httpclient issue might resolve itself as soon as Jena is able to
>>>> switch to httpclient 4.3. I've looked into the javax.xml issues in
>>>> more
>>>> detail and the good news is that all the xsd datatype classes are
>>>> actually
>>>> there. What's missing are the StAX classes (everything in
>>>> java.xml.stream)
>>>> which seem to be used in ARQ and core (for SPARQL XML result sets,
>>>> RDF/XML
>>>> and TriX) and of course in Xerces. That means it's enough to
>>>> repackage the
>>>> StAX classes from xml-apis and to modify jena-arq, jena-core and
>>>> xercesImpl. I've changed my build accordingly.
>>>>
>>>> Android uses XmlPullParser for stream parsing. So another solution
>>>> could
>>>> be to replace StAX with a XmlPull based parsing but I guess that
>>>> would take
>>>> some effort.
>>>>
>>>>>
>>>>> On Android, how does TDB work well? TDB uses fairly traditional file
>>>>> handling for 32 bit machines - "direct mode" - and uses memory
>>>>> mapped I/O
>>>>> for 64 bit machines - "mapped mode" - if it can detect the mode
>>>>> correctly.
>>>>> Detection is not subtle, it looks in system property
>>>>> "java.vm.info" and
>>>>> default to 32 bit. In mapped mode, it does try to use as much
>>>>> memory as
>>>>> possible which is not friendly to co-resident apps (you can force
>>>>> "direct"
>>>>> mode programmatically).
>>>>>
>>>>
>>>> Since most Android devices nowadays run on a 32 bit ARM architecture I
>>>> guess it will run almost always in direct mode.
>>>> But I realized that TDB isn't working at all at the moment, because it
>>>> depends on JDK classes that aren't available on Android for things
>>>> like
>>>> getting the PID. I think it is possible to replace those calls by
>>>> using
>>>> similar classes provided by the Android SDK. I will try to get it
>>>> working.
>>>> A dependency on Android specific code would probably make mainline
>>>> integration more difficult though.
>>>>
>>>> Sören
>>>>
>>>>
Re: Apache Jena for Android
Posted by Andy Seaborne <an...@apache.org>.
You can configure the footprint of TDB in 32 bit "direct" mode for
smaller datasets:
http://jena.staging.apache.org/documentation/tdb/store-parameters.html
Andy
On 01/04/15 22:38, Stian Soiland-Reyes wrote:
> +1 - and how well does the memory mapping of TDB work on Dalvik?
>
>
> Would jena-sdb work with SQLite?
>
> https://developer.android.com/guide/topics/data/data-storage.html#db
>
> You would probably need https://github.com/SQLDroid/SQLDroid so you
> get a JDBC driver.
>
>
> On 1 April 2015 at 18:02, Claude Warren <cl...@xenei.com> wrote:
>> Might it make sense to use a different persistence engine on Android since
>> it probably will not have the huge datasets that TDB is designed for. It
>> might make sense to have a small store available for small devices. Just a
>> thought.
>>
>> Claude
>>
>> On Tue, Mar 31, 2015 at 9:41 PM, Sören Brunk <so...@tu-dresden.de>
>> wrote:
>>
>>> Hi Andy,
>>>
>>> thanks for your feedback and sorry for taking so long to reply. It would
>>> be great to get Android support into mainline Jena and I'm willing to help.
>>> It might be necessary to build an extra jar package for Android though,
>>> similar to what jena-osgi does.
>>>
>>>>
>>>> Is there are anything the Jena project can do that would make the
>>>> conversion? There may be things that Jena does, or the way is it packaged,
>>>> that are inconvenient for you but really make no differnce to the project -
>>>> sometimes things are just the way they are because it was done that way but
>>>> could easily be done another way. If you have any such points, do email
>>>> thsilist or raise a JIRA.
>>>>
>>>
>>> The httpclient issue might resolve itself as soon as Jena is able to
>>> switch to httpclient 4.3. I've looked into the javax.xml issues in more
>>> detail and the good news is that all the xsd datatype classes are actually
>>> there. What's missing are the StAX classes (everything in java.xml.stream)
>>> which seem to be used in ARQ and core (for SPARQL XML result sets, RDF/XML
>>> and TriX) and of course in Xerces. That means it's enough to repackage the
>>> StAX classes from xml-apis and to modify jena-arq, jena-core and
>>> xercesImpl. I've changed my build accordingly.
>>>
>>> Android uses XmlPullParser for stream parsing. So another solution could
>>> be to replace StAX with a XmlPull based parsing but I guess that would take
>>> some effort.
>>>
>>>>
>>>> On Android, how does TDB work well? TDB uses fairly traditional file
>>>> handling for 32 bit machines - "direct mode" - and uses memory mapped I/O
>>>> for 64 bit machines - "mapped mode" - if it can detect the mode correctly.
>>>> Detection is not subtle, it looks in system property "java.vm.info" and
>>>> default to 32 bit. In mapped mode, it does try to use as much memory as
>>>> possible which is not friendly to co-resident apps (you can force "direct"
>>>> mode programmatically).
>>>>
>>>
>>> Since most Android devices nowadays run on a 32 bit ARM architecture I
>>> guess it will run almost always in direct mode.
>>> But I realized that TDB isn't working at all at the moment, because it
>>> depends on JDK classes that aren't available on Android for things like
>>> getting the PID. I think it is possible to replace those calls by using
>>> similar classes provided by the Android SDK. I will try to get it working.
>>> A dependency on Android specific code would probably make mainline
>>> integration more difficult though.
>>>
>>> Sören
>>>
>>> --
>>> Dipl. Inf. Sören Brunk
>>> Research Associate
>>>
>>> Technische Universität Dresden
>>> Faculty of Computer Science
>>> Institute for Software- and Multimedia-Technology
>>> Junior Professorship in Software Engineering of Ubiquitous Systems
>>> 01062 Dresden
>>>
>>>
>>
>>
>> --
>> I like: Like Like - The likeliest place on the web
>> <http://like-like.xenei.com>
>> LinkedIn: http://www.linkedin.com/in/claudewarren
>
>
>
Re: Apache Jena for Android
Posted by Stian Soiland-Reyes <st...@apache.org>.
+1 - and how well does the memory mapping of TDB work on Dalvik?
Would jena-sdb work with SQLite?
https://developer.android.com/guide/topics/data/data-storage.html#db
You would probably need https://github.com/SQLDroid/SQLDroid so you
get a JDBC driver.
On 1 April 2015 at 18:02, Claude Warren <cl...@xenei.com> wrote:
> Might it make sense to use a different persistence engine on Android since
> it probably will not have the huge datasets that TDB is designed for. It
> might make sense to have a small store available for small devices. Just a
> thought.
>
> Claude
>
> On Tue, Mar 31, 2015 at 9:41 PM, Sören Brunk <so...@tu-dresden.de>
> wrote:
>
>> Hi Andy,
>>
>> thanks for your feedback and sorry for taking so long to reply. It would
>> be great to get Android support into mainline Jena and I'm willing to help.
>> It might be necessary to build an extra jar package for Android though,
>> similar to what jena-osgi does.
>>
>>>
>>> Is there are anything the Jena project can do that would make the
>>> conversion? There may be things that Jena does, or the way is it packaged,
>>> that are inconvenient for you but really make no differnce to the project -
>>> sometimes things are just the way they are because it was done that way but
>>> could easily be done another way. If you have any such points, do email
>>> thsilist or raise a JIRA.
>>>
>>
>> The httpclient issue might resolve itself as soon as Jena is able to
>> switch to httpclient 4.3. I've looked into the javax.xml issues in more
>> detail and the good news is that all the xsd datatype classes are actually
>> there. What's missing are the StAX classes (everything in java.xml.stream)
>> which seem to be used in ARQ and core (for SPARQL XML result sets, RDF/XML
>> and TriX) and of course in Xerces. That means it's enough to repackage the
>> StAX classes from xml-apis and to modify jena-arq, jena-core and
>> xercesImpl. I've changed my build accordingly.
>>
>> Android uses XmlPullParser for stream parsing. So another solution could
>> be to replace StAX with a XmlPull based parsing but I guess that would take
>> some effort.
>>
>>>
>>> On Android, how does TDB work well? TDB uses fairly traditional file
>>> handling for 32 bit machines - "direct mode" - and uses memory mapped I/O
>>> for 64 bit machines - "mapped mode" - if it can detect the mode correctly.
>>> Detection is not subtle, it looks in system property "java.vm.info" and
>>> default to 32 bit. In mapped mode, it does try to use as much memory as
>>> possible which is not friendly to co-resident apps (you can force "direct"
>>> mode programmatically).
>>>
>>
>> Since most Android devices nowadays run on a 32 bit ARM architecture I
>> guess it will run almost always in direct mode.
>> But I realized that TDB isn't working at all at the moment, because it
>> depends on JDK classes that aren't available on Android for things like
>> getting the PID. I think it is possible to replace those calls by using
>> similar classes provided by the Android SDK. I will try to get it working.
>> A dependency on Android specific code would probably make mainline
>> integration more difficult though.
>>
>> Sören
>>
>> --
>> Dipl. Inf. Sören Brunk
>> Research Associate
>>
>> Technische Universität Dresden
>> Faculty of Computer Science
>> Institute for Software- and Multimedia-Technology
>> Junior Professorship in Software Engineering of Ubiquitous Systems
>> 01062 Dresden
>>
>>
>
>
> --
> I like: Like Like - The likeliest place on the web
> <http://like-like.xenei.com>
> LinkedIn: http://www.linkedin.com/in/claudewarren
--
Stian Soiland-Reyes
Apache Taverna (incubating), Apache Commons RDF (incubating)
http://orcid.org/0000-0001-9842-9718