You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@devicemap.apache.org by Reza <re...@yahoo.com> on 2013/06/23 07:10:35 UTC
device map java client
So I went ahead and finished the Java client. After much thought, I came to the conclusion that it would be easiest to start from scratch. I think it turned out good. Its less than 250 lines of code, has pretty good performance (~1ms classifications), and no dependencies on external libs.
I checked it in here:
http://svn.apache.org/viewvc/incubator/devicemap/trunk/devicemapjava/
It basically just indexes all the pattern variations into a hash and then searches for them. I took some liberties when it came to non alphanumeric matching and I stripped out all the regular expressions.
Its about 90% done, I just got to refactor a few minor things, add an option to load the DDR data from jar, comments, and add some proper tests.
Here is some code which uses the client:
http://svn.apache.org/viewvc/incubator/devicemap/trunk/devicemapjava/src/main/java/Main.java?view=markup
Let me know what you think.
Reza
Re: device map java client
Posted by Reza <re...@yahoo.com>.
Yes, all the loaders 'wrap' around this base loader:
https://svn.apache.org/repos/asf/incubator/devicemap/trunk/devicemapjava/src/main/java/org/apache/devicemap/client/loaders/Loader.java
The loader needs to return a set of Devices back to the client:
https://svn.apache.org/repos/asf/incubator/devicemap/trunk/devicemapjava/src/main/java/org/apache/devicemap/client/data/Device.java
The client then compiles a pattern index from the Device set and that's it, the client is ready to classify:
https://svn.apache.org/repos/asf/incubator/devicemap/trunk/devicemapjava/src/main/java/org/apache/devicemap/client/DeviceMapClient.java
So it should be pretty flexible... Just need to standardize the design around some interfaces so they can be properly extended.
________________________________
From: Bertrand Delacretaz <bd...@apache.org>
To: devicemap-dev@incubator.apache.org
Sent: Wednesday, June 26, 2013 10:59 AM
Subject: Re: device map java client
Hi,
On Wed, Jun 26, 2013 at 4:43 PM, Reza <re...@yahoo.com> wrote:
> ...add more methods for loading DDR data. Currently we load from
> filesystem (OpenDDR style), would like to:
> -load from user defined files
> -load from user included jar
> -load from default URL and user defined URL...
Would be good to accept InputStream or Reader (maybe in addition the
above), as this works for any data source. For my use cases I might
feed the thing from a JCR repository, so none of the above would work.
-Bertrand
Re: device map java client
Posted by Bertrand Delacretaz <bd...@apache.org>.
Hi,
On Wed, Jun 26, 2013 at 4:43 PM, Reza <re...@yahoo.com> wrote:
> ...add more methods for loading DDR data. Currently we load from
> filesystem (OpenDDR style), would like to:
> -load from user defined files
> -load from user included jar
> -load from default URL and user defined URL...
Would be good to accept InputStream or Reader (maybe in addition the
above), as this works for any data source. For my use cases I might
feed the thing from a JCR repository, so none of the above would work.
-Bertrand
Re: device map java client
Posted by Reza <re...@yahoo.com>.
Definitely, here is my todo:
-finalize any sort of design tweaks to get the pattern matching upto par with previous clients.
-add more methods for loading DDR data. Currently we load from filesystem (OpenDDR style), would like to:
-load from user defined files
-load from user included jar
-load from default URL and user defined URL
-clean up/polish code, use a standard logger, finalize API, etc
So yes, I think most of this should be ready soon, as in days/week. Radu had mentioned he would like to contribute, so that will be helpful. Eberhard has already ported the client to .NET.
Some of my longer term goals:
-The DDR data, need to get it updated regularly. Also would like to start cleaning it up and enhancing it. Updates need to happen regularly first...
-Supporting a new langauge? PHP? Python? Ruby? If the language supports Lists, Maps, string manipulation, and has decent performance it should be very straight forward.
________________________________
From: Bertrand Delacretaz <bd...@apache.org>
To: devicemap-dev@incubator.apache.org
Sent: Wednesday, June 26, 2013 10:15 AM
Subject: Re: device map java client
On Wed, Jun 26, 2013 at 4:04 PM, Reza <re...@yahoo.com> wrote:
> ...I think everything left is fine tuning now....
Very cool - I would suggest making a release of that module as soon as
you feel it's ready (in the "release early, release often" sense) -
that might help get feedback.
-Bertrand
Re: device map java client
Posted by Bertrand Delacretaz <bd...@apache.org>.
On Wed, Jun 26, 2013 at 4:04 PM, Reza <re...@yahoo.com> wrote:
> ...I think everything left is fine tuning now....
Very cool - I would suggest making a release of that module as soon as
you feel it's ready (in the "release early, release often" sense) -
that might help get feedback.
-Bertrand
Re: device map java client
Posted by Reza <re...@yahoo.com>.
ahh yes, so some trade offs were made, but I think they are mostly positive. We have been layering in some needed logic to bring the matching upto par, but those have been mainly centered out how best to index the patterns.
The heuristics are basically normalize the data as best as possible to allow for generic pattern matching. Right now this means stripping all whitespace, symbols, and regex and then doing whole word ngram matching. I would like to reintroduce whitespace and symbols, but there are some conflicts with regex. Next, choose the longest match and thats basically it. The SimpleBuilder is the lowest priority and all other builders are equal priority. I think everything left is fine tuning now.
https://svn.apache.org/repos/asf/incubator/devicemap/trunk/devicemapjava/src/main/java/org/apache/devicemap/client/DeviceMapClient.java
________________________________
From: Bertrand Delacretaz <bd...@apache.org>
To: devicemap-dev@incubator.apache.org
Sent: Wednesday, June 26, 2013 8:11 AM
Subject: Re: device map java client
On Wed, Jun 26, 2013 at 1:56 PM, Reza Naghibi <re...@yahoo.com> wrote:
> Do you know what heuristics drive the openddr module?...
Not precisely...what I mean is the numerous pattern-matching
expressions found in our openddr/java builders, so maybe heuristics is
not the right word.
The AndroidDeviceBuilder, for example, contains a number of hardcoded
regexps, IMO such things belong in a data file and not in code, and if
your new code doesn't need them at all it's even better.
-Bertrand
[1] https://svn.apache.org/repos/asf/incubator/devicemap/trunk/openddr/java/src/main/java/org/apache/devicemap/simpleddr/builder/device/AndroidDeviceBuilder.java
Re: device map java client
Posted by Bertrand Delacretaz <bd...@apache.org>.
On Wed, Jun 26, 2013 at 1:56 PM, Reza Naghibi <re...@yahoo.com> wrote:
> Do you know what heuristics drive the openddr module?...
Not precisely...what I mean is the numerous pattern-matching
expressions found in our openddr/java builders, so maybe heuristics is
not the right word.
The AndroidDeviceBuilder, for example, contains a number of hardcoded
regexps, IMO such things belong in a data file and not in code, and if
your new code doesn't need them at all it's even better.
-Bertrand
[1] https://svn.apache.org/repos/asf/incubator/devicemap/trunk/openddr/java/src/main/java/org/apache/devicemap/simpleddr/builder/device/AndroidDeviceBuilder.java
Re: device map java client
Posted by Reza Naghibi <re...@yahoo.com>.
Do you know what heuristics drive the openddr module?
---
Sent from a Blackberry 9900
-----Original Message-----
From: Bertrand Delacretaz <bd...@apache.org>
Date: Wed, 26 Jun 2013 12:23:26
To: <de...@incubator.apache.org>; Reza<re...@yahoo.com>
Reply-To: devicemap-dev@incubator.apache.org
Subject: Re: device map java client
Hi,
On Sun, Jun 23, 2013 at 7:10 AM, Reza <re...@yahoo.com> wrote:
> ...So I went ahead and finished the Java client. After much thought, I came
> to the conclusion that it would be easiest to start from scratch...
So IIUC you're ignoring the heuristics that are provided in our
openddr/java module, and still getting good results? If yes that's
very cool, as I'm not a fan of having those heuristics in code,
especially in view of implementing detection in several languages.
-Bertrand
Re: device map java client
Posted by Bertrand Delacretaz <bd...@apache.org>.
Hi,
On Sun, Jun 23, 2013 at 7:10 AM, Reza <re...@yahoo.com> wrote:
> ...So I went ahead and finished the Java client. After much thought, I came
> to the conclusion that it would be easiest to start from scratch...
So IIUC you're ignoring the heuristics that are provided in our
openddr/java module, and still getting good results? If yes that's
very cool, as I'm not a fan of having those heuristics in code,
especially in view of implementing detection in several languages.
-Bertrand