You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@nutch.apache.org by "lewismc (via GitHub)" <gi...@apache.org> on 2026/01/12 00:57:42 UTC
Re: [PR] NUTCH-3064 Upgrade index-geoip to GeoIP2 5.0.2 [nutch]
lewismc commented on PR #825:
URL: https://github.com/apache/nutch/pull/825#issuecomment-3736568735
This PR now upgrades the `index-geoip` plugin to use MaxMind GeoIP2 Java API 5.0.2, with significant architectural improvements including support for multiple database types and in-memory caching.
## Changes
### Dependency Updates
- `geoip2`: upgraded to **5.0.2**
- `maxmind-db`: upgraded to **4.0.2**
- `jackson-datatype-jsr310`: added **2.20.1** (new transitive dependency)
### Performance Improvement — CHMCache
Database readers now use `CHMCache` (ConcurrentHashMap Cache) from the maxmind-db library for improved lookup performance:
```java
DatabaseReader reader = new DatabaseReader.Builder(db)
.withCache(new CHMCache())
.build();
```
This caches parsed database nodes in memory, reducing disk I/O and improving throughput when the same IP prefixes are queried repeatedly during indexing.
### New Configuration Options in `conf/nutch-default.xml`
The plugin now supports multiple database types simultaneously. Configure each by setting its file path:
| Property | Description |
|----------|-------------|
| `index.geoip.db.anonymous` | Anonymous IP database — identifies VPNs, proxies, Tor exit nodes |
| `index.geoip.db.asn` | ASN database — autonomous system number and organization |
| `index.geoip.db.city` | City database — city, subdivision, country, continent, coordinates |
| `index.geoip.db.connection` | Connection Type database — Cable/DSL, Cellular, Corporate, Satellite |
| `index.geoip.db.domain` | Domain database — second-level domain for the IP |
| `index.geoip.db.isp` | ISP database — ISP name, organization, ASN |
### MaxMind Insights Web Service Support
| Property | Description |
|----------|-------------|
| `index.geoip.insights.userid` | User ID for MaxMind Precision Insights API |
| `index.geoip.insights.licensekey` | License key for the Insights API |
### Architecture Improvements
- Refactored to support multiple databases via `EnumMap<DatabaseType, DatabaseReader>`
- Each database type is loaded independently and queried in sequence
- Proper resource cleanup via `Closeable` implementation
- Graceful error handling per-database (one failure doesn't block others)
## Files Modified
- `src/plugin/index-geoip/` — plugin source, tests, dependencies, and config
- `build.xml` — root build configuration
- `conf/nutch-default.xml` — new GeoIP configuration properties
- `src/plugin/build.xml` — plugin build configuration
- `src/plugin/indexer-solr/schema.xml` — Solr schema field definitions
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@nutch.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org