You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@lucene.apache.org by ct...@apache.org on 2017/05/06 18:58:33 UTC
lucene-solr:jira/solr-10290: SOLR-10296: conversion,
letter G + some heading level cleanups
Repository: lucene-solr
Updated Branches:
refs/heads/jira/solr-10290 f060417a8 -> a967d5a39
SOLR-10296: conversion, letter G + some heading level cleanups
Project: http://git-wip-us.apache.org/repos/asf/lucene-solr/repo
Commit: http://git-wip-us.apache.org/repos/asf/lucene-solr/commit/a967d5a3
Tree: http://git-wip-us.apache.org/repos/asf/lucene-solr/tree/a967d5a3
Diff: http://git-wip-us.apache.org/repos/asf/lucene-solr/diff/a967d5a3
Branch: refs/heads/jira/solr-10290
Commit: a967d5a392e3cda08a6699dd751f597b8473f71e
Parents: f060417
Author: Cassandra Targett <ct...@apache.org>
Authored: Sat May 6 13:58:00 2017 -0500
Committer: Cassandra Targett <ct...@apache.org>
Committed: Sat May 6 13:58:00 2017 -0500
----------------------------------------------------------------------
solr/solr-ref-guide/src/about-this-guide.adoc | 10 +-
solr/solr-ref-guide/src/blob-store-api.adoc | 6 +-
solr/solr-ref-guide/src/config-api.adoc | 2 +-
solr/solr-ref-guide/src/defining-fields.adoc | 6 +-
solr/solr-ref-guide/src/documents-screen.adoc | 12 +-
solr/solr-ref-guide/src/getting-assistance.adoc | 9 +-
.../src/getting-started-with-solrcloud.adoc | 26 ++-
solr/solr-ref-guide/src/getting-started.adoc | 6 +-
solr/solr-ref-guide/src/graph-traversal.adoc | 214 ++++++++++---------
9 files changed, 151 insertions(+), 140 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/a967d5a3/solr/solr-ref-guide/src/about-this-guide.adoc
----------------------------------------------------------------------
diff --git a/solr/solr-ref-guide/src/about-this-guide.adoc b/solr/solr-ref-guide/src/about-this-guide.adoc
index 82137b0..a9d1564 100644
--- a/solr/solr-ref-guide/src/about-this-guide.adoc
+++ b/solr/solr-ref-guide/src/about-this-guide.adoc
@@ -2,14 +2,16 @@
:page-shortname: about-this-guide
:page-permalink: about-this-guide.html
-This guide describes all of the important features and functions of Apache Solr. It is free to download from http://lucene.apache.org/solr/.
+This guide describes all of the important features and functions of Apache Solr.
+
+Solr is free to download from http://lucene.apache.org/solr/.
Designed to provide high-level documentation, this guide is intended to be more encyclopedic and less of a cookbook. It is structured to address a broad spectrum of needs, ranging from new developers getting started to well-experienced developers extending their application or troubleshooting. It will be of use at any point in the application life cycle, for whenever you need authoritative information about Solr.
The material as presented assumes that you are familiar with some basic search concepts and that you can read XML. It does not assume that you are a Java programmer, although knowledge of Java is helpful when working directly with Lucene or when developing custom extensions to a Lucene/Solr installation.
[[AboutThisGuide-SpecialInlineNotes]]
-=== Special Inline Notes
+== Special Inline Notes
Special notes are included throughout these pages. There are several types of notes:
@@ -35,7 +37,7 @@ WARNING: These are meant to warn you from a possibly dangerous change or action.
[[AboutThisGuide-HostsandPortExamples]]
-=== Hosts and Port Examples
+== Hosts and Port Examples
The default port when running Solr is 8983. The samples, URLs and screenshots in this guide may show different ports, because the port number that Solr uses is configurable. If you have not customized your installation of Solr, please make sure that you use port 8983 when following the examples, or configure your own installation to use the port numbers shown in the examples. For information about configuring port numbers, see the section <<managing-solr.adoc#managing-solr,Managing Solr>>.
@@ -50,6 +52,6 @@ There are several items in this URL you might need to change locally. First, if
`\http://www.example.com/solr/mycollection/select?q=brown+cow`
[[AboutThisGuide-Paths]]
-=== Paths
+== Paths
Path information is given relative to `solr.home`, which is the location under the main Solr installation where Solr's collections and their `conf` and `data` directories are stored. When running the various examples mentioned through out this tutorial (i.e., `bin/solr -e techproducts`) the `solr.home` will be a sub-directory of `example/` created for you automatically.
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/a967d5a3/solr/solr-ref-guide/src/blob-store-api.adoc
----------------------------------------------------------------------
diff --git a/solr/solr-ref-guide/src/blob-store-api.adoc b/solr/solr-ref-guide/src/blob-store-api.adoc
index 1326e06..8e23ed9 100644
--- a/solr/solr-ref-guide/src/blob-store-api.adoc
+++ b/solr/solr-ref-guide/src/blob-store-api.adoc
@@ -13,7 +13,7 @@ When using the blob store, note that the API does not delete or overwrite a prev
The blob store API is implemented as a requestHandler. A special collection named ".system" is used to store the blobs. This collection can be created in advance, but if it does not exist it will be created automatically.
[[BlobStoreAPI-Aboutthe.systemCollection]]
-=== About the .system Collection
+== About the .system Collection
Before uploading blobs to the blob store, a special collection must be created and it must be named `.system`. Solr will automatically create this collection if it does not already exist, but you can also create it manually if you choose.
@@ -31,7 +31,7 @@ curl http://localhost:8983/solr/admin/collections?action=CREATE&name=.system&rep
IMPORTANT: The `bin/solr` script cannot be used to create the `.system` collection.
[[BlobStoreAPI-UploadFilestoBlobStore]]
-=== Upload Files to Blob Store
+== Upload Files to Blob Store
After the `.system` collection has been created, files can be uploaded to the blob store with a request similar to the following:
@@ -117,7 +117,7 @@ curl http://localhost:8983/solr/.system/blob/{blobname}?wt=filestream > {outputf
----
[[BlobStoreAPI-UseaBlobinaHandlerorComponent]]
-=== Use a Blob in a Handler or Component
+== Use a Blob in a Handler or Component
To use the blob as the class for a request handler or search component, you create a request handler in `solrconfig.xml` as usual. You will need to define the following parameters:
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/a967d5a3/solr/solr-ref-guide/src/config-api.adoc
----------------------------------------------------------------------
diff --git a/solr/solr-ref-guide/src/config-api.adoc b/solr/solr-ref-guide/src/config-api.adoc
index 3b4fc97..9c53c2b 100644
--- a/solr/solr-ref-guide/src/config-api.adoc
+++ b/solr/solr-ref-guide/src/config-api.adoc
@@ -348,7 +348,7 @@ Define the same properties with the Config API:
----
[[ConfigAPI-NameComponentsfortheConfigAPI]]
-==== Name Components for the Config API
+=== Name Components for the Config API
The Config API always allows changing the configuration of any component by name. However, some configurations such as `listener` or `initParams` do not require a name in `solrconfig.xml`. In order to be able to `update` and `delete` of the same item in `configoverlay.json`, the name attribute becomes mandatory.
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/a967d5a3/solr/solr-ref-guide/src/defining-fields.adoc
----------------------------------------------------------------------
diff --git a/solr/solr-ref-guide/src/defining-fields.adoc b/solr/solr-ref-guide/src/defining-fields.adoc
index 6de3aa5..1f00a89 100644
--- a/solr/solr-ref-guide/src/defining-fields.adoc
+++ b/solr/solr-ref-guide/src/defining-fields.adoc
@@ -5,7 +5,7 @@
Fields are defined in the fields element of `schema.xml`. Once you have the field types set up, defining the fields themselves is simple.
[[DefiningFields-Example]]
-=== Example
+== Example
The following example defines a field named `price` with a type named `float` and a default value of `0.0`; the `indexed` and `stored` properties are explicitly set to `true`, while any other properties specified on the `float` field type are inherited.
@@ -15,7 +15,7 @@ The following example defines a field named `price` with a type named `float` an
----
[[DefiningFields-FieldProperties]]
-=== Field Properties
+== Field Properties
[cols=",",options="header",]
|===
@@ -26,7 +26,7 @@ The following example defines a field named `price` with a type named `float` an
|===
[[DefiningFields-OptionalFieldTypeOverrideProperties]]
-=== Optional Field Type Override Properties
+== Optional Field Type Override Properties
Fields can have many of the same properties as field types. Properties from the table below which are specified on an individual field will override any explicit value for that property specified on the the `fieldType` of the field, or any implicit default property value provided by the underlying `fieldType` implementation. The table below is reproduced from <<field-type-definitions-and-properties.adoc#field-type-definitions-and-properties,Field Type Definitions and Properties>>, which has more details:
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/a967d5a3/solr/solr-ref-guide/src/documents-screen.adoc
----------------------------------------------------------------------
diff --git a/solr/solr-ref-guide/src/documents-screen.adoc b/solr/solr-ref-guide/src/documents-screen.adoc
index 0753746..a885e97 100644
--- a/solr/solr-ref-guide/src/documents-screen.adoc
+++ b/solr/solr-ref-guide/src/documents-screen.adoc
@@ -27,7 +27,7 @@ The first step is to define the RequestHandler to use (aka, 'qt'). By default `/
Then choose the Document Type to define the type of document to load. The remaining parameters will change depending on the document type selected.
[[DocumentsScreen-JSON]]
-=== JSON
+== JSON
When using the JSON document type, the functionality is similar to using a requestHandler on the command line. Instead of putting the documents in a curl command, they can instead be input into the Document entry box. The document structure should still be in proper JSON format.
@@ -36,19 +36,19 @@ Then you can choose when documents should be added to the index (Commit Within),
This option will only add or overwrite documents to the index; for other update tasks, see the <<DocumentsScreen-SolrCommand,Solr Command>> option.
[[DocumentsScreen-CSV]]
-=== CSV
+== CSV
When using the CSV document type, the functionality is similar to using a requestHandler on the command line. Instead of putting the documents in a curl command, they can instead be input into the Document entry box. The document structure should still be in proper CSV format, with columns delimited and one row per document.
Then you can choose when documents should be added to the index (Commit Within), and whether existing documents should be overwritten with incoming documents with the same id (if this is not *true*, then the incoming documents will be dropped).
[[DocumentsScreen-DocumentBuilder]]
-=== Document Builder
+== Document Builder
The Document Builder provides a wizard-like interface to enter fields of a document
[[DocumentsScreen-FileUpload]]
-=== File Upload
+== File Upload
The File Upload option allows choosing a prepared file and uploading it. If using only `/update` for the Request-Handler option, you will be limited to XML, CSV, and JSON.
@@ -57,14 +57,14 @@ However, to use the ExtractingRequestHandler (aka Solr Cell), you can modify the
Then you can choose when documents should be added to the index (Commit Within), and whether existing documents should be overwritten with incoming documents with the same id (if this is not *true*, then the incoming documents will be dropped).
[[DocumentsScreen-SolrCommand]]
-=== Solr Command
+== Solr Command
The Solr Command option allows you use XML or JSON to perform specific actions on documents, such as defining documents to be added or deleted, updating only certain fields of documents, or commit and optimize commands on the index.
The documents should be structured as they would be if using `/update` on the command line.
[[DocumentsScreen-XML]]
-=== XML
+== XML
When using the XML document type, the functionality is similar to using a requestHandler on the command line. Instead of putting the documents in a curl command, they can instead be input into the Document entry box. The document structure should still be in proper Solr XML format, with each document separated by `<doc>` tags and each field defined.
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/a967d5a3/solr/solr-ref-guide/src/getting-assistance.adoc
----------------------------------------------------------------------
diff --git a/solr/solr-ref-guide/src/getting-assistance.adoc b/solr/solr-ref-guide/src/getting-assistance.adoc
index 69b6fd1..78469ae 100644
--- a/solr/solr-ref-guide/src/getting-assistance.adoc
+++ b/solr/solr-ref-guide/src/getting-assistance.adoc
@@ -4,11 +4,10 @@
At the bottom of each screen of the Admin UI is a set of links that can be used to get more assistance with configuring and using Solr.
+.Assistance icons
image::images/getting-assistance/Assistance.png[image]
-_Assistance icons_
-
These icons include the following links.
[width="100%",cols="50%,50%",options="header",]
@@ -16,9 +15,9 @@ These icons include the following links.
|Link |Description
|Documentation |Navigates to the Apache Solr documentation hosted on https://lucene.apache.org/solr/.
|Issue Tracker |Navigates to the JIRA issue tracking server for the Apache Solr project. This server resides at https://issues.apache.org/jira/browse/SOLR.
-|IRC Channel |Navigates to an Apache Wiki page describing how to join Solr's http://en.wikipedia.org/wiki/Internet_Relay_Chat[IRC] live-chat room: https://wiki.apache.org/solr/IRCChannels.
+|IRC Channel |Navigates to Solr's http://en.wikipedia.org/wiki/Internet_Relay_Chat[IRC] live-chat room: http://webchat.freenode.net/?channels=#solr.
|Community forum |Navigates to the Apache Wiki page which has further information about ways to engage in the Solr User community mailing lists: https://wiki.apache.org/solr/UsingMailingLists.
-|Solr Query Syntax |Navigates to the section "<<query-syntax-and-parsing.adoc#query-syntax-and-parsing,Query Syntax and Parsing>>" in this reference guide.
+|Solr Query Syntax |Navigates to the section <<query-syntax-and-parsing.adoc#query-syntax-and-parsing,Query Syntax and Parsing>> in this Reference Guide.
|===
-These links cannot be modified without editing the `admin.html` in the `solr.war` that contains the Admin UI files.
+These links cannot be modified without editing the `index.html` in the `server/solr/solr-webapp` directory that contains the Admin UI files.
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/a967d5a3/solr/solr-ref-guide/src/getting-started-with-solrcloud.adoc
----------------------------------------------------------------------
diff --git a/solr/solr-ref-guide/src/getting-started-with-solrcloud.adoc b/solr/solr-ref-guide/src/getting-started-with-solrcloud.adoc
index 96319a0..526b6bf 100644
--- a/solr/solr-ref-guide/src/getting-started-with-solrcloud.adoc
+++ b/solr/solr-ref-guide/src/getting-started-with-solrcloud.adoc
@@ -2,17 +2,19 @@
:page-shortname: getting-started-with-solrcloud
:page-permalink: getting-started-with-solrcloud.html
-SolrCloud is designed to provide a highly available, fault tolerant environment for distributing your indexed content and query requests across multiple servers. It's a system in which data is organized into multiple pieces, or shards, that can be hosted on multiple machines, with replicas providing redundancy for both scalability and fault tolerance, and a ZooKeeper server that helps manage the overall structure so that both indexing and search requests can be routed properly.
+SolrCloud is designed to provide a highly available, fault tolerant environment for distributing your indexed content and query requests across multiple servers.
-This section explains SolrCloud and its inner workings in detail, but before you dive in, it's best to have an idea of what it is you're trying to accomplish. This page provides a simple tutorial to start Solr in SolrCloud mode, so you can begin to get a sense for how shards interact with each other during indexing and when serving queries. To that end, we'll use simple examples of configuring SolrCloud on a single machine, which is obviously not a real production environment, which would include several servers or virtual machines. In a real production environment, you'll also use the real machine names instead of "localhost" which we've used here.
+It's a system in which data is organized into multiple pieces, or shards, that can be hosted on multiple machines, with replicas providing redundancy for both scalability and fault tolerance, and a ZooKeeper server that helps manage the overall structure so that both indexing and search requests can be routed properly.
+
+This section explains SolrCloud and its inner workings in detail, but before you dive in, it's best to have an idea of what it is you're trying to accomplish.
+
+This page provides a simple tutorial to start Solr in SolrCloud mode, so you can begin to get a sense for how shards interact with each other during indexing and when serving queries. To that end, we'll use simple examples of configuring SolrCloud on a single machine, which is obviously not a real production environment, which would include several servers or virtual machines. In a real production environment, you'll also use the real machine names instead of "localhost" which we've used here.
In this section you will learn how to start a SolrCloud cluster using startup scripts and a specific configset.
[TIP]
====
-
This tutorial assumes that you're already familiar with the basics of using Solr. If you need a refresher, please see the <<getting-started.adoc#getting-started,Getting Started section>> to get a grounding in Solr concepts. If you load documents as part of that exercise, you should start over with a fresh Solr installation for these SolrCloud tutorials.
-
====
[[GettingStartedwithSolrCloud-SolrCloudExample]]
@@ -23,12 +25,14 @@ This tutorial assumes that you're already familiar with the basics of using Solr
The `bin/solr` script makes it easy to get started with SolrCloud as it walks you through the process of launching Solr nodes in cloud mode and adding a collection. To get started, simply do:
-[source,java]
+[source, bash]
----
$ bin/solr -e cloud
----
-This starts an interactive session to walk you through the steps of setting up a simple SolrCloud cluster with embedded ZooKeeper. The script starts by asking you how many Solr nodes you want to run in your local cluster, with the default being 2.
+This starts an interactive session to walk you through the steps of setting up a simple SolrCloud cluster with embedded ZooKeeper.
+
+The script starts by asking you how many Solr nodes you want to run in your local cluster, with the default being 2.
[source,plain]
----
@@ -44,7 +48,7 @@ Next, the script will prompt you for the port to bind each of the Solr nodes to,
[source,plain]
----
- Please enter the port for node1 [8983]
+ Please enter the port for node1 [8983]
----
Choose any available port for each node; the default for the first node is 8983 and 7574 for the second node. The script will start each node in order and show you the command it uses to start the server, such as:
@@ -67,9 +71,9 @@ The suggested default is "gettingstarted" but you might want to choose a name mo
Next, the script prompts you for the number of shards to distribute the collection across. <<shards-and-indexing-data-in-solrcloud.adoc#shards-and-indexing-data-in-solrcloud,Sharding>> is covered in more detail later on, so if you're unsure, we suggest using the default of 2 so that you can see how a collection is distributed across multiple nodes in a SolrCloud cluster.
-Next, the script will prompt you for the number of replicas to create for each shard. https://cwiki.apache.org/confluence/display/solr/NRT%2C+Replication%2C+and+Disaster+Recovery+with+SolrCloud[Replication] is covered in more detail later in the guide, so if you're unsure, then use the default of 2 so that you can see how replication is handled in SolrCloud.
+Next, the script will prompt you for the number of replicas to create for each shard. <<shards-and-indexing-data-in-solrcloud.adoc#shards-and-indexing-data-in-solrcloud,Replication>> is covered in more detail later in the guide, so if you're unsure, then use the default of 2 so that you can see how replication is handled in SolrCloud.
-Lastly, the script will prompt you for the name of a configuration directory for your collection. You can choose **basic_configs**, **data_driven_schema_configs**, or **sample_techproducts_configs**. The configuration directories are pulled from `server/solr/configsets/` so you can review them beforehand if you wish. The *data_driven_schema_configs* configuration (the default) is useful when you're still designing a schema for your documents and need some flexiblity as you experiment with Solr.
+Lastly, the script will prompt you for the name of a configuration directory for your collection. You can choose *basic_configs*, *data_driven_schema_configs*, or *sample_techproducts_configs*. The configuration directories are pulled from `server/solr/configsets/` so you can review them beforehand if you wish. The *data_driven_schema_configs* configuration (the default) is useful when you're still designing a schema for your documents and need some flexiblity as you experiment with Solr.
At this point, you should have a new collection created in your local SolrCloud cluster. To verify this, you can run the status command:
@@ -125,7 +129,7 @@ To restart node2 running on port 7574, you can do:
$ bin/solr restart -c -p 7574 -z localhost:9983 -s example/cloud/node2/solr
----
-Notice that you need to specify the ZooKeeper address (-z localhost:9983) when starting node2 so that it can join the cluster with node1.
+Notice that you need to specify the ZooKeeper address (`-z localhost:9983`) when starting node2 so that it can join the cluster with node1.
[[GettingStartedwithSolrCloud-Addinganodetoacluster]]
=== Adding a node to a cluster
@@ -135,7 +139,7 @@ Adding a node to an existing cluster is a bit advanced and involves a little mor
[source,plain]
----
$ mkdir <solr.home for new solr node>
- $ cp <existing solr.xml path> <new solr.home>
+ $ cp <existing solr.xml path> <new solr.home>
$ bin/solr start -cloud -s solr.home/solr -p <port num> -z <zk hosts string>
----
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/a967d5a3/solr/solr-ref-guide/src/getting-started.adoc
----------------------------------------------------------------------
diff --git a/solr/solr-ref-guide/src/getting-started.adoc b/solr/solr-ref-guide/src/getting-started.adoc
index 7cfcbd6..3fd7c76 100644
--- a/solr/solr-ref-guide/src/getting-started.adoc
+++ b/solr/solr-ref-guide/src/getting-started.adoc
@@ -3,7 +3,9 @@
:page-permalink: getting-started.html
:page-children: installing-solr, running-solr, a-quick-overview, a-step-closer, solr-control-script-reference
-Solr makes it easy for programmers to develop sophisticated, high-performance search applications with advanced features such as faceting (arranging search results in columns with numerical counts of key terms). Solr builds on another open source search technology: Lucene, a Java library that provides indexing and search technology, as well as spellchecking, hit highlighting and advanced analysis/tokenization capabilities. Both Solr and Lucene are managed by the Apache Software Foundation (http://www.apache.org/[www.apache.org)].
+Solr makes it easy for programmers to develop sophisticated, high-performance search applications with advanced features such as faceting (arranging search results in columns with numerical counts of key terms).
+
+Solr builds on another open source search technology: Lucene, a Java library that provides indexing and search technology, as well as spellchecking, hit highlighting and advanced analysis/tokenization capabilities. Both Solr and Lucene are managed by the Apache Software Foundation (http://www.apache.org/[www.apache.org)].
The Lucene search library currently ranks among the top 15 open source projects and is one of the top 5 Apache projects, with installations at over 4,000 companies. Lucene/Solr downloads have grown nearly ten times over the past three years, with a current run-rate of over 6,000 downloads a day. The Solr search server, which provides application builders a ready-to-use search platform on top of the Lucene search library, is the fastest growing Lucene sub-project. Apache Lucene/Solr offers an attractive alternative to the proprietary licensed search and discovery software vendors.
@@ -21,7 +23,5 @@ This section helps you get Solr up and running quickly, and introduces you to th
[TIP]
====
-
Solr includes a Quick Start tutorial which will be helpful if you are just starting out with Solr. You can find it online at http://lucene.apache.org/solr/quickstart.html, or in your Solr installation at `$SOLR_INSTALL_DIR/docs/quickstart.html`.
-
====
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/a967d5a3/solr/solr-ref-guide/src/graph-traversal.adoc
----------------------------------------------------------------------
diff --git a/solr/solr-ref-guide/src/graph-traversal.adoc b/solr/solr-ref-guide/src/graph-traversal.adoc
index 9708f0d..5a56949 100644
--- a/solr/solr-ref-guide/src/graph-traversal.adoc
+++ b/solr/solr-ref-guide/src/graph-traversal.adoc
@@ -2,7 +2,9 @@
:page-shortname: graph-traversal
:page-permalink: graph-traversal.html
-Graph traversal with streaming expressions uses the `gatherNodes` function to perform a breadth-first graph traversal. The `gatherNodes` function can be combined with the `scoreNodes` function to provide recommendations. `gatherNodes` can also be combined with the wider streaming expression library to perform complex operations on gathered node sets.
+Graph traversal with streaming expressions uses the `gatherNodes` function to perform a breadth-first graph traversal.
+
+The `gatherNodes` function can be combined with the `scoreNodes` function to provide recommendations. `gatherNodes` can also be combined with the wider streaming expression library to perform complex operations on gathered node sets.
`gatherNodes` traversals are distributed within a SolrCloud collection and can span collections.
@@ -10,9 +12,7 @@ Graph traversal with streaming expressions uses the `gatherNodes` function to pe
[IMPORTANT]
====
-
This document assumes a basic understanding of graph terminology and streaming expressions. You can begin exploring graph traversal concepts with this https://en.wikipedia.org/wiki/Graph_traversal[Wikipedia article]. More details about streaming expressions are available in this Guide, in the section <<streaming-expressions.adoc#streaming-expressions,Streaming Expressions>>.
-
====
[[GraphTraversal-BasicSyntax]]
@@ -20,16 +20,16 @@ This document assumes a basic understanding of graph terminology and streaming e
We'll start with the most basic syntax and slowly build up more complexity. The most basic syntax for `gatherNodes` is:
-[source,java]
+[source,plain]
----
-gatherNodes(emails,
- walk="johndoe@apache.org->from",
- gather="to")
+gatherNodes(emails,
+ walk="johndoe@apache.org->from",
+ gather="to")
----
Let's break down this simple expression.
-The first parameter, `emails`, is the collection being traversed. The second parameter, `walk`, maps a hard-coded node ID ("johndoe@apache.org") to a field in the index (`from`). This will return all the *edges* in the index that have `johndoe@apache.org` in the `from` field.
+The first parameter, `emails`, is the collection being traversed. The second parameter, `walk`, maps a hard-coded node ID ("\johndoe@apache.org") to a field in the index (`from`). This will return all the *edges* in the index that have `johndoe@apache.org` in the `from` field.
The `gather` parameter tells the function to gather the values in the `to `field. The values that are gathered are the node IDs emitted by the function.
@@ -37,11 +37,11 @@ In the example above the nodes emitted will be all of the people that "johndoe@a
The walk parameter also accepts a list of root node IDs:
-[source,java]
+[source,plain]
----
-gatherNodes(emails,
- walk="johndoe@apache.org, janesmith@apache.org->from",
- gather="to")
+gatherNodes(emails,
+ walk="johndoe@apache.org, janesmith@apache.org->from",
+ gather="to")
----
The `gatherNodes` function above finds all the edges with "johndoe@apache.org" or "janesmith@apache.org" in the `from` field and gathers the `to` field.
@@ -50,14 +50,14 @@ Like all <<streaming-expressions.adoc#streaming-expressions,Streaming Expression
[source,bash]
----
-curl --data-urlencode 'expr=gatherNodes(emails,
- walk="johndoe@apache.org, janesmith@apache.org->from",
+curl --data-urlencode 'expr=gatherNodes(emails,
+ walk="johndoe@apache.org, janesmith@apache.org->from",
gather="to")' http://localhost:8983/solr/emails/stream
----
The output of this expression would look like this:
-[source,java]
+[source,json]
----
{
"result-set": {
@@ -93,19 +93,19 @@ All of the tuples returned have the `node` field. The `node` field contains the
Notice that the level is "1" for each tuple in the example. The root nodes are level 0 (in the example above, the root nodes are "johndoe@apache.org, janesmith@apache.org") By default the `gatherNodes` function emits only the _*leaf nodes*_ of the traversal, which is the outer-most node set. To emit the root nodes you can specify the `scatter` parameter:
-[source,java]
+[source,plain]
----
-gatherNodes(emails,
- walk="johndoe@apache.org->from",
+gatherNodes(emails,
+ walk="johndoe@apache.org->from",
gather="to",
- scatter="branches, leaves")
+ scatter="branches, leaves")
----
-The `scatter` parameter controls whether to emit the _*branches*_ with the _*leaves*_ . The root nodes are considered "branches" because they are not the outer-most level of the traversal.
+The `scatter` parameter controls whether to emit the _branches_ with the _leaves_. The root nodes are considered "branches" because they are not the outer-most level of the traversal.
When scattering both branches and leaves the output would like this:
-[source,java]
+[source,json]
----
{
"result-set": {
@@ -150,19 +150,19 @@ Now the level 0 root node is included in the output.
`gatherNodes` also supports aggregations. For example:
-[source,java]
+[source,plain]
----
-gatherNodes(emails,
- walk="johndoe@apache.org, janesmith@apache.org->from",
+gatherNodes(emails,
+ walk="johndoe@apache.org, janesmith@apache.org->from",
gather="to",
- count(*))
+ count(*))
----
-The expression above finds the edges with "johndoe@apache.org" or "janesmith@apache.org" in the `from` field and gathers the values from the `to` field__.__ It also aggregates the count for each node ID gathered.
+The expression above finds the edges with "\johndoe@apache.org" or "\janesmith@apache.org" in the `from` field and gathers the values from the `to` field. It also aggregates the count for each node ID gathered.
-A gathered node could have a count of 2 if both "johndoe@apache.org" and "janesmith@apache.org" have emailed the same person. Node sets contain a unique set of nodes, so the same person won't appear twice in the node set, but the count will reflect that it appeared twice during the traversal.
+A gathered node could have a count of 2 if both "\johndoe@apache.org" and "\janesmith@apache.org" have emailed the same person. Node sets contain a unique set of nodes, so the same person won't appear twice in the node set, but the count will reflect that it appeared twice during the traversal.
-Edges are uniqued as part of the traversal so the count will *not* reflect the number of times "johndoe@apache.org" emailed the same person. For example, personA might have emailed personB 100 times. These edges would get uniqued and only be counted once. But if person personC also emailed personB this would increment the count for personB.
+Edges are uniqued as part of the traversal so the count will *not* reflect the number of times "\johndoe@apache.org" emailed the same person. For example, personA might have emailed personB 100 times. These edges would get uniqued and only be counted once. But if person personC also emailed personB this would increment the count for personB.
The aggregation functions supported are `count(*)`, `sum(field)`, `min(field)`, `max(field)`, and `avg(field)`. The fields being aggregated should be present in the edges collected during the traversal. Later examples (below) will show aggregations can be a powerful tool for providing recommendations and limiting the scope of traversals.
@@ -171,14 +171,14 @@ The aggregation functions supported are `count(*)`, `sum(field)`, `min(field)`,
The `gatherNodes` function can be nested to traverse deeper into the graph. For example:
-[source,java]
+[source,plain]
----
gatherNodes(emails,
- gatherNodes(emails,
- walk="johndoe@apache.org->from",
+ gatherNodes(emails,
+ walk="johndoe@apache.org->from",
gather="to"),
walk="node->from",
- gather="to")
+ gather="to")
----
In the example above the outer `gatherNodes` function operates on the node set collected from the inner `gatherNodes` function.
@@ -187,7 +187,7 @@ Notice that the inner `gatherNodes` function behaves exactly as the examples alr
In the outer `gatherNodes` function the `walk` parameter works with tuples coming from an internal streaming expression. In this scenario the `walk` parameter maps the `node` field to the `from` field. Remember that the node IDs collected from the inner `gatherNodes` expression are placed in the `node` field.
-Put more simply, the inner expression gathers all the people that "johndoe@apache.org" has emailed. We can call this group the "friends of johndoe@apache.org". The outer expression gathers all the people that the "friends of johndoe@apache.org" have emailed. This is a basic friends-of-friends traversal.
+Put more simply, the inner expression gathers all the people that "\johndoe@apache.org" has emailed. We can call this group the "friends of \johndoe@apache.org". The outer expression gathers all the people that the "friends of \johndoe@apache.org" have emailed. This is a basic friends-of-friends traversal.
This construct of nesting `gatherNodes` functions is the basic technique for doing a controlled traversal through the graph.
@@ -196,19 +196,19 @@ This construct of nesting `gatherNodes` functions is the basic technique for doi
The `gatherNodes` function performs cycle detection across the entire traversal. This ensures that nodes that have already been visited are not traversed again. Cycle detection is important for both limiting the size of traversals and gathering accurate aggregations. Without cycle detection the size of the traversal could grow exponentially with each hop in the traversal. With cycle detection only new nodes encountered are traversed.
-Cycle detection *does not* cross collection boundaries. This is because internally the collection name is part of the node ID. For example the node ID "johndoe@apache.org", is really `emails/johndoe@apache.org`. When traversing to another collection "johndoe@apache.org" will be traversed.
+Cycle detection *does not* cross collection boundaries. This is because internally the collection name is part of the node ID. For example the node ID "\johndoe@apache.org", is really `emails/johndoe@apache.org`. When traversing to another collection "\johndoe@apache.org" will be traversed.
[[GraphTraversal-FilteringtheTraversal]]
== Filtering the Traversal
Each level in the traversal can be filtered with a filter query. For example:
-[source,java]
+[source,plain]
----
-gatherNodes(emails,
- walk="johndoe@apache.org->from",
+gatherNodes(emails,
+ walk="johndoe@apache.org->from",
fq="body:(solr rocks)",
- gather="to")
+ gather="to")
----
In the example above only emails that match the filter query will be included in the traversal. Any Solr query can be included here. So you can do fun things like <<spatial-search.adoc#spatial-search,geospatial queries>>, apply any of the available <<query-syntax-and-parsing.adoc#query-syntax-and-parsing,query parsers>>, or even write custom query parsers to limit the traversal.
@@ -218,12 +218,12 @@ In the example above only emails that match the filter query will be included in
Any streaming expression can be used to provide the root nodes for a traversal. For example:
-[source,java]
+[source,plain]
----
-gatherNodes(emails,
+gatherNodes(emails,
search(emails, q="body:(solr rocks)", fl="to", sort="score desc", rows="20")
- walk="to->from",
- gather="to")
+ walk="to->from",
+ gather="to")
----
The example above provides the root nodes through a search expression. You can also provide arbitrarily complex, nested streaming expressions with joins, etc., to specify the root nodes.
@@ -237,20 +237,20 @@ It's often desirable to skip traversing high frequency nodes in the graph. This
Let's say that you want to recommend content for a user based on a collaborative filter. Below is one approach for a simple collaborative filter:
-1. Find all content userA has read.
-2. Find users whose reading list is closest to userA. These are users with similar tastes as userA.
-3. Recommend content based on what the users in step 2 have read, that userA has not yet read.
+. Find all content userA has read.
+. Find users whose reading list is closest to userA. These are users with similar tastes as userA.
+. Recommend content based on what the users in step 2 have read, that userA has not yet read.
Look closely at step 2. In large graphs, step 2 can lead to a very large traversal. This is because userA may have viewed content that has been viewed by millions of other people. We may want to skip these high frequency nodes for two reasons:
-1. A large traversal that visit millions of unique nodes is slow and takes a lot of memory because cycle detection is tracked in memory.
-2. High frequency nodes are also not useful in determining users with similar tastes. The content that fewer people have viewed provides a more precise recommendation.
+. A large traversal that visit millions of unique nodes is slow and takes a lot of memory because cycle detection is tracked in memory.
+. High frequency nodes are also not useful in determining users with similar tastes. The content that fewer people have viewed provides a more precise recommendation.
The `gatherNodes` function has the `maxDocFreq` param to allow for filtering out high frequency nodes. The sample code below shows steps 1 and 2 of the recommendation:
-[source,java]
+[source,plain]
----
- gatherNodes(logs,
+ gatherNodes(logs,
search(logs, q="userID:user1", fl="articleID", sort="articleID asc", fq="action:view", qt="/export"),
walk="articleID->articleID",
gather="userID",
@@ -270,16 +270,16 @@ For some use cases, such as graph visualization, we also need to output the edge
Below is a sample `gatherNodes` expression with `trackTraversal` set to true:
-[source,java]
+[source,plain]
----
gatherNodes(emails,
- gatherNodes(emails,
- walk="johndoe@apache.org->from",
+ gatherNodes(emails,
+ walk="johndoe@apache.org->from",
gather="to",
trackTraversal="true"),
walk="node->from",
trackTraversal="true",
- gather="to")
+ gather="to")
----
[[GraphTraversal-Cross-CollectionTraversals]]
@@ -289,17 +289,17 @@ Nested `gatherNodes` functions can operate on different SolrCloud collections. T
Below is a sample `gatherNodes` expression that traverses from the "emails" collection to the "logs" collection:
-[source,java]
+[source,plain]
----
gatherNodes(logs,
- gatherNodes(emails,
+ gatherNodes(emails,
search(emails, q="body:(solr rocks)", fl="from", sort="score desc", rows="20")
- walk="from->from",
+ walk="from->from",
gather="to",
scatter="leaves, branches"),
walk="node->user",
- fq="action:edit",
- gather="contentID")
+ fq="action:edit",
+ gather="contentID")
----
The example above finds all people who sent emails with a body that contains "solr rocks". It then finds all the people these people have emailed. Then it traverses to the logs collection and gathers all the content IDs that these people have edited.
@@ -309,28 +309,28 @@ The example above finds all people who sent emails with a body that contains "so
The `gatherNodes` function can act as both a stream source and a stream decorator. The connection with the wider stream expression library provides tremendous power and flexibility when performing graph traversals. Here is an example of using the streaming expression library to intersect two friend networks:
-[source,java]
+[source,plain]
----
intersect(on="node",
sort(by="node asc",
gatherNodes(emails,
- gatherNodes(emails,
- walk="johndoe@apache.org->from",
+ gatherNodes(emails,
+ walk="johndoe@apache.org->from",
gather="to"),
walk="node->from",
gather="to",
- scatter="branches,leaves")),
+ scatter="branches,leaves")),
sort(by="node asc",
gatherNodes(emails,
- gatherNodes(emails,
- walk="janedoe@apache.org->from",
+ gatherNodes(emails,
+ walk="janedoe@apache.org->from",
gather="to"),
walk="node->from",
gather="to",
- scatter="branches,leaves")))
+ scatter="branches,leaves")))
----
-The example above gathers two separate friend networks, one rooted with "johndoe@apache.org" and another rooted with "janedoe@apache.org". The friend networks are then sorted by the `node` field, and intersected. The resulting node set will be the intersection of the two friend networks.
+The example above gathers two separate friend networks, one rooted with "\johndoe@apache.org" and another rooted with "\janedoe@apache.org". The friend networks are then sorted by the `node` field, and intersected. The resulting node set will be the intersection of the two friend networks.
[[GraphTraversal-SampleUseCases]]
== Sample Use Cases
@@ -342,23 +342,23 @@ It is often useful to know which products are most frequently purchased with a p
Here is the sample syntax:
-[source,java]
+[source,plain]
----
-top(n="5",
+top(n="5",
sort="count(*) desc",
- gatherNodes(baskets,
+ gatherNodes(baskets,
random(baskets, q="productID:ABC", fl="basketID", rows="500"),
walk="basketID->basketID",
- fq="-productID:ABC",
+ fq="-productID:ABC",
gather="productID",
- count(*)))
+ count(*)))
----
Let's break down exactly what this traversal is doing.
-1. The first expression evaluated is the inner `random` expression, which returns 500 random basketIDs, from the `baskets` collection, that have the `productID` "ABC". The `random` expression is very useful for recommendations because it limits the traversal to a fixed set of baskets, and because it adds the element of surprise into the recommendation. Using the `random` function you can provide fast sample sets from very large graphs.
-2. The outer `gatherNodes` expression finds all the records in the `baskets` collection for the basketIDs generated in step 1. It also filters out `productID` "ABC" so it doesn't show up in the results. It then gathers and counts the productID's across these baskets.
-3. The outer `top` expression ranks the productIDs emitted in step 2 by the count and selects the top 5.
+. The first expression evaluated is the inner `random` expression, which returns 500 random basketIDs, from the `baskets` collection, that have the `productID` "ABC". The `random` expression is very useful for recommendations because it limits the traversal to a fixed set of baskets, and because it adds the element of surprise into the recommendation. Using the `random` function you can provide fast sample sets from very large graphs.
+. The outer `gatherNodes` expression finds all the records in the `baskets` collection for the basketIDs generated in step 1. It also filters out `productID` "ABC" so it doesn't show up in the results. It then gathers and counts the productID's across these baskets.
+. The outer `top` expression ranks the productIDs emitted in step 2 by the count and selects the top 5.
In a nutshell this expression finds the products that most frequently co-occur with product "ABC" in past shopping baskets.
@@ -381,25 +381,25 @@ The `scoreNodes` function adds the score to each node in the `nodeScore` field.
[[GraphTraversal-ExampleSyntax]]
==== *Example Syntax*
-[source,java]
+[source,plain]
----
top(n="1",
sort="nodeScore desc",
- scoreNodes(top(n="50",
+ scoreNodes(top(n="50",
sort="count(*) desc",
- gatherNodes(baskets,
+ gatherNodes(baskets,
random(baskets, q="productID:ABC", fl="basketID", rows="500"),
walk="basketID->basketID",
- fq="-productID:ABC",
+ fq="-productID:ABC",
gather="productID",
- count(*)))))
+ count(*)))))
----
This example builds on the earlier example "Calculate market basket co-occurrence".
-1. Notice that the inner-most `top` function is taking the top 50 products that co-occur most frequently with productID ABC. This provides 50 candidate recommendations.
-2. The `scoreNodes` function then assigns a score to the candidates based on the TF*IDF of each node.
-3. The outer `top` expression selects the highest scoring node. This is the recommendation.
+. Notice that the inner-most `top` function is taking the top 50 products that co-occur most frequently with productID ABC. This provides 50 candidate recommendations.
+. The `scoreNodes` function then assigns a score to the candidates based on the TF*IDF of each node.
+. The outer `top` expression selects the highest scoring node. This is the recommendation.
[[GraphTraversal-RecommendContentBasedonCollaborativeFilter]]
=== Recommend Content Based on Collaborative Filter
@@ -408,14 +408,14 @@ In this example we'll recommend content for a user based on a collaborative filt
Here is the sample syntax:
-[source,java]
+[source,plain]
----
top(n="5",
sort="count(*) desc",
gatherNodes(logs,
- top(n="30",
+ top(n="30",
sort="count(*) desc",
- gatherNodes(logs,
+ gatherNodes(logs,
search(logs, q="userID:user1", fl="articleID", sort="articleID asc", fq="action:read", qt="/export"),
walk="articleID->articleID",
gather="userID",
@@ -430,11 +430,17 @@ top(n="5",
Let's break down the expression above step-by-step.
-1. The first expression evaluated is the inner `search` expression. This expression searches the `logs` collection for all records matching "user1". This is the user we are making the recommendation for. There is a filter applied to pull back only records where the "action:read". It returns the `articleID` for each record found. In other words, this expression returns all the articles "user1" has read.
-2. The inner `gatherNodes` expression operates over the articleIDs returned from step 1. It takes each `articleID` found and searches them against the `articleID` field. Note that it skips high frequency nodes using the `maxDocFreq` param to filter out articles that appear over 10,000 times in the logs. It gathers userIDs and aggregates the counts for each user. This step finds the users that have read the same articles that "user1" has read and counts how many of the same articles they have read.
-3. The inner `top` expression ranks the users emitted from step 2. It will emit the top 30 users who have the most overlap with user1's reading list.
-4. The outer `gatherNodes` expression gathers the reading list for the users emitted from step 3. It counts the articleIDs that are gathered. Any article selected in step 1 (user1 reading list), will not appear in this step due to cycle detection. So this step returns the articles read by the users with the most similar readings habits to "user1" that "user1" has not read yet. It also counts the number of times each article has been read across this user group.
-5. The outer `top` expression takes the top articles emitted from step 4. This is the recommendation.
+. The first expression evaluated is the inner `search` expression. This expression searches the `logs` collection for all records matching "user1". This is the user we are making the recommendation for.
++
+There is a filter applied to pull back only records where the "action:read". It returns the `articleID` for each record found. In other words, this expression returns all the articles "user1" has read.
+. The inner `gatherNodes` expression operates over the articleIDs returned from step 1. It takes each `articleID` found and searches them against the `articleID` field.
++
+Note that it skips high frequency nodes using the `maxDocFreq` param to filter out articles that appear over 10,000 times in the logs. It gathers userIDs and aggregates the counts for each user. This step finds the users that have read the same articles that "user1" has read and counts how many of the same articles they have read.
+. The inner `top` expression ranks the users emitted from step 2. It will emit the top 30 users who have the most overlap with user1's reading list.
+. The outer `gatherNodes` expression gathers the reading list for the users emitted from step 3. It counts the articleIDs that are gathered.
++
+Any article selected in step 1 (user1 reading list), will not appear in this step due to cycle detection. So this step returns the articles read by the users with the most similar readings habits to "user1" that "user1" has not read yet. It also counts the number of times each article has been read across this user group.
+. The outer `top` expression takes the top articles emitted from step 4. This is the recommendation.
[[GraphTraversal-ProteinPathwayTraversal]]
=== Protein Pathway Traversal
@@ -443,7 +449,7 @@ In recent years, scientists have become increasingly able to rationally design d
The example below illustrates a protein pathway traversal:
-[source,java]
+[source,plain]
----
gatherNodes(proteins,
gatherNodes(proteins,
@@ -455,9 +461,9 @@ gatherNodes(proteins,
Let's break down exactly what this traversal is doing.
-1. The inner `gatherNodes` expression traverses in the `proteins` collection. It finds all the edges in the graph where the name of the protein is "NRAS". Then it gathers the proteins in the `interacts` field. This gathers all the proteins that "NRAS" interactions with.
-2. The outer `gatherNodes` expression also works with the `proteins` collection. It gathers all the drugs that correspond to proteins emitted from step 1.
-3. Using this stepwise approach you can gather the drugs along the pathway of interactions any number of steps away from the root protein.
+. The inner `gatherNodes` expression traverses in the `proteins` collection. It finds all the edges in the graph where the name of the protein is "NRAS". Then it gathers the proteins in the `interacts` field. This gathers all the proteins that "NRAS" interactions with.
+. The outer `gatherNodes` expression also works with the `proteins` collection. It gathers all the drugs that correspond to proteins emitted from step 1.
+. Using this stepwise approach you can gather the drugs along the pathway of interactions any number of steps away from the root protein.
[[GraphTraversal-ExportingGraphMLtoSupportGraphVisualization]]
== Exporting GraphML to Support Graph Visualization
@@ -468,20 +474,20 @@ There are other graph traversal use cases that involve graph visualization. Solr
http://graphml.graphdrawing.org/[GraphML] is an XML format supported by graph visualization tools such as https://gephi.org/[Gephi], which is a sophisticated open source tool for statistically analyzing and visualizing graphs. Using a `gatherNodes` expression, parts of a larger graph can be exported in GraphML and then imported into tools like Gephi.
-There are a few things to keep mind when exporting a graph in GraphML
+There are a few things to keep mind when exporting a graph in GraphML:
-1. The `/graph` handler can export both the nodes and edges in the graph. By default, it only exports the nodes. To export the edges you must set `trackTraversal="true"` in the `gatherNodes` expression.
-2. The `/graph` handler currently accepts an arbitrarily complex streaming expression which includes a `gatherNodes` expression. If the streaming expression doesn't include a `gatherNodes` expression, the `/graph` handler will not properly output GraphML.
-3. The `/graph` handler currently accepts a single arbitrarily complex, nested `gatherNodes` expression per request. This means you cannot send in a streaming expression that joins or intersects the node sets from multiple `gatherNodes` expressions. The `/graph` handler does support any level of nesting within a single `gatherNodes` expression. The `/stream` handler does support joining and intersecting node sets, but the `/graph` handler currently does not.
+. The `/graph` handler can export both the nodes and edges in the graph. By default, it only exports the nodes. To export the edges you must set `trackTraversal="true"` in the `gatherNodes` expression.
+. The `/graph` handler currently accepts an arbitrarily complex streaming expression which includes a `gatherNodes` expression. If the streaming expression doesn't include a `gatherNodes` expression, the `/graph` handler will not properly output GraphML.
+. The `/graph` handler currently accepts a single arbitrarily complex, nested `gatherNodes` expression per request. This means you cannot send in a streaming expression that joins or intersects the node sets from multiple `gatherNodes` expressions. The `/graph` handler does support any level of nesting within a single `gatherNodes` expression. The `/stream` handler does support joining and intersecting node sets, but the `/graph` handler currently does not.
[[GraphTraversal-SampleRequest]]
=== Sample Request
-[source,bash]
+[source,plain]
----
curl --data-urlencode 'expr=gatherNodes(enron_emails,
- gatherNodes(enron_emails,
- walk="kayne.coulter@enron.com->from",
+ gatherNodes(enron_emails,
+ walk="kayne.coulter@enron.com->from",
trackTraversal="true",
gather="to"),
walk="node->from",
@@ -493,10 +499,10 @@ curl --data-urlencode 'expr=gatherNodes(enron_emails,
[[GraphTraversal-SampleGraphMLOutput]]
=== Sample GraphML Output
-[source,java]
+[source,xml]
----
-<graphml xmlns="http://graphml.graphdrawing.org/xmlns"
-xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+<graphml xmlns="http://graphml.graphdrawing.org/xmlns"
+xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://graphml.graphdrawing.org/xmlns http://graphml.graphdrawing.org/xmlns/1.0/graphml.xsd">
<graph id="G" edgedefault="directed">
<node id="kayne.coulter@enron.com">