Requests for ideas on getting into Solr dev.

[Mark Miller <ma...@gmail.com>]

A while back a couple outside devs reached out to me asking advice on how
to get started doing Solr development.

Basically, I generally just say to find some item to improve that has some
synergy with your Solr use needs or pet peeves. I got the feeling that was
not the best jump-starter for ideas though, and you usually have to come up
with a couple on the spot, which is when I have fewest ideas.

For those people and any others in a sim situation, here is a small list of
ideas I have:


   - Add support for Jetty Quickstart
   https://webtide.com/jetty-9-quick-start/
   - Upgrade Jetty to 10 or 11 - default offheap ByteBuffers for the
   client/server (this config went away in 9 and returns in 10/11) and
   improved HttpClient classes (old ones around but deprecated). Improved
   Quickstart support, the current dev momentum center.
   - Add the most basic Servlet Request async support, simply to run the
   request thread in our own threadpool, no async processing actually done.
   Then Solr’s pool can be the unlimited pool and Jetty can have a limited
   pool as it expects and works best with.
   - Add SolrQosFilter that extends Jetty’s QoSFilter. Allow allow internal
   requests to prevent deadlock, but use async Servlet Request feature via
   QoSFilter to suspend and prioritize un-susspend of external requests on
   overload.
   - Dig into removing @ThreadLeakLingering(linger = 10000) from
   SolrTestCase.
   - Add a custom Jetty LifeCycleListener that get’s notified of Shutdown
   before Jetty shutdown has already begun (the situation when we are
   currently notified by ServletFilter#destroy). Remove the live node entry
   there.
   - Look into current state of the static field checker for tests. Since
   the same JVM is used across many to potentially all of the tests, they
   should avoid leaving behind large static remnants in the test class. Is
   this check working? Are there offenders?
   - Look into the @BeforeClass @AfterClass shadowing method name test
   rule. This should prevent test class hierarchies from having the same name
   for these static methods because there can be hard to pin down bad issues
   when this happens. Look into it’s effectiveness and current violations.
   See NoClassHooksShadowingRule
   - Review for inefficient Collection / StringBuilder size init. Make
   sizing improvements where a lot of capacity grows are guaranteed and or
   likely or we already know the size.
   - Run the tests using IntelliJ’s allocate profiler, java flight
   recorder, YourKit with allocation recording on, etc and note the top
   allocated objects. Look at simple changes (say reuse or alternative
   implementations) for some of the largest outliers.
   - Change some ZooKeeper usages to use the much more efficient Async API.
   It’s performance is essentially that of a MultiOp, but without
   the requirement everything succeed/fail atomically. Bonus if you can change
   a path to run fully async, where step b only runs or consumes resources
   when zk async call A is finished. Most systems that provide async also
   provide reasonably good back pressure for free. Lots of wins in this area.
   - Look at making ZkClient calls that come in and hit ConnectionLoss
   simply wait until ZooKeeper has connected again rather than any retries or
   fails and repeated attempts. Verify that on ConnectionLoss, the system
   essentially goes quite to ZK until reconnection instead of ramping up
   trying to make something with ZK happen.
   - Investigate using the Lucene Segment Replicator replication strategy
   in PULL or TLOG replicas to take advantage of it’s NRT segment replication
   feature and awesome, isolated testing and Lucene integration.
   - Investigate combining or dropping calls to ZooKeeper. There are only
   so many types of calls that go to ZK. They are only usefully bringing back
   information at a pretty slow rate in computer time. Review calls that are
   hitting many times a second or just generally too much for the information
   trade happening. Updating some timely item once a second is perhaps
   reasonable. Trying to do something so fast the last call has not even
   completed, perhaps not. The system should not rely on cluster state being
   100% up to date the huge majority of cases, which means rapid updates are
   likely never very sensible.
   - Make it dead simple to setup Solr to log to a JSON output format.
   - Look at introducing fair locking into the TransactionLog. When locks
   on a path are constantly gotten and released like happens there, especially
   with updates that can depend on each other, you can end up with old
   requests lock attempts often getting beat by new requests along the
   lock/unlock chain, and this can cause traffic mayhem.

-- 
- Mark

http://about.me/markrmiller