You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@lucene.apache.org by ct...@apache.org on 2021/01/28 16:33:32 UTC
svn commit: r1070647 [12/35] - in
/websites/production/lucene/content/solr/guide/8_8: ./
images/math-expressions/ meta-docs/
Added: websites/production/lucene/content/solr/guide/8_8/loading.html
==============================================================================
--- websites/production/lucene/content/solr/guide/8_8/loading.html (added)
+++ websites/production/lucene/content/solr/guide/8_8/loading.html Thu Jan 28 16:33:25 2021
@@ -0,0 +1,2114 @@
+<!DOCTYPE html>
+
+
+<head>
+ <meta charset="utf-8">
+<meta http-equiv="X-UA-Compatible" content="IE=edge">
+<meta name="viewport" content="width=device-width, initial-scale=1">
+<meta name="description" content="">
+<meta name="keywords" content=" ">
+
+<title>Loading Data | Apache Solr Reference Guide 8.8</title>
+
+<link rel="stylesheet" href="https://stackpath.bootstrapcdn.com/bootstrap/4.1.3/css/bootstrap.min.css" integrity="sha384-MCw98/SFnGE8fJT3GXwEOngsV7Zt27NXFoaoApmYm81iuXoPkFOJwJ8ERdknLPMO" crossorigin="anonymous">
+<link rel="stylesheet" type="text/css" href="https://maxcdn.bootstrapcdn.com/font-awesome/4.5.0/css/font-awesome.min.css">
+<link rel="stylesheet" href="css/navs.css ">
+<link rel="stylesheet" href="css/ref-guide.css">
+<link rel="stylesheet" href="css/decoration.css">
+<link rel="stylesheet" href="css/search.css">
+
+<!-- JQuery - we can't use slim because navgoco relies on ajax -->
+<script src="https://code.jquery.com/jquery-3.5.1.min.js" integrity="sha384-ZvpUoO/+PpLXR1lu4jmpXWu80pZlYUAfxl5NsBMWOEPSjUn/6Z/hRTt8+pR6L4N2" crossorigin="anonymous"></script>
+
+<!-- Scrollbar Custom CSS -->
+<link rel="stylesheet" href="https://cdnjs.cloudflare.com/ajax/libs/malihu-custom-scrollbar-plugin/3.1.5/jquery.mCustomScrollbar.min.css" crossorigin="anonymous">
+
+<script src="https://cdnjs.cloudflare.com/ajax/libs/jquery-cookie/1.4.1/jquery.cookie.min.js" crossorigin="anonymous"></script>
+
+<script src="js/jquery.navgoco.min.js"></script>
+
+<script src="https://cdnjs.cloudflare.com/ajax/libs/anchor-js/4.2.0/anchor.min.js" crossorigin="anonymous"></script>
+
+<link rel="shortcut icon" href="images/icons/favicon.ico">
+
+<!-- HTML5 Shim and Respond.js IE8 support of HTML5 elements and media queries -->
+<!-- WARNING: Respond.js doesn't work if you view the page via file:// -->
+<!--[if lt IE 9]>
+<script src="https://oss.maxcdn.com/libs/html5shiv/3.7.0/html5shiv.js"></script>
+<script src="https://oss.maxcdn.com/libs/respond.js/1.4.2/respond.min.js"></script>
+<![endif]-->
+
+ <script>
+ $(document).ready(function() {
+ // Initialize navgoco with default options
+ $("#mysidebar").navgoco({
+ caretHtml: '',
+ accordion: true,
+ openClass: 'active', // open
+ save: false, // we do *NOT* want cookies used to save the current stage of the sidebar
+ // instead the code in sidebar.html will ensure that the current page
+ // (and it's ancestors) are "active" on page load
+ slide: {
+ duration: 400,
+ easing: 'swing'
+ }
+ });
+
+ $("#collapseAll").click(function(e) {
+ e.preventDefault();
+ $("#mysidebar").navgoco('toggle', false);
+ });
+
+ $("#expandAll").click(function(e) {
+ e.preventDefault();
+ $("#mysidebar").navgoco('toggle', true);
+ });
+
+ });
+
+ </script>
+</head>
+<body class="" id="loading">
+<div class="container-fluid">
+ <div class="row">
+ <nav id="sidebar" class="col-2 d-none d-md-block">
+
+<div class="sidebar-header">
+ <div class="sidebarTitle text-center">Apache Solr Reference Guide</div>
+
+
+ <!--comment out this block if you want to hide search-->
+ <!--start search-->
+ <div id="search-demo-container">
+ <input type="text" id="search-input" class="form-control" placeholder="Page title lookup...">
+ <ul id="results-container"></ul>
+ </div>
+ <script src="js/jekyll-search.js" type="text/javascript"></script>
+ <script type="text/javascript">
+ SimpleJekyllSearch.init({
+ searchInput: document.getElementById('search-input'),
+ resultsContainer: document.getElementById('results-container'),
+ dataSource: 'search.json',
+ searchResultTemplate: '<li><a href="{url}" title="Loading Data">{title}</a></li>',
+ noResultsText: 'No results found.',
+ limit: 10,
+ fuzzy: true,
+ })
+ </script>
+ <!--end search-->
+
+ </div>
+
+
+<ul class="nav" id="mysidebar">
+
+ <li class="sb-level1">
+ <a href="about-this-guide.html">About This Guide</a>
+
+ </li>
+
+ <li class="sb-level1">
+ <a href="getting-started.html">Getting Started</a>
+
+ <ul>
+
+ <li class="sb-level2">
+ <a href="solr-tutorial.html">Solr Tutorial</a>
+
+ </li>
+
+ <li class="sb-level2">
+ <a href="a-quick-overview.html">A Quick Overview</a>
+
+ </li>
+
+ <li class="sb-level2">
+ <a href="solr-system-requirements.html">Solr System Requirements</a>
+
+ </li>
+
+ <li class="sb-level2">
+ <a href="installing-solr.html">Installing Solr</a>
+
+ </li>
+
+ </ul>
+
+ </li>
+
+ <li class="sb-level1">
+ <a href="deployment-and-operations.html">Deployment and Operations</a>
+
+ <ul>
+
+ <li class="sb-level2">
+ <a href="solr-control-script-reference.html">Solr Control Script Reference</a>
+
+ </li>
+
+ <li class="sb-level2">
+ <a href="solr-configuration-files.html">Solr Configuration Files</a>
+
+ </li>
+
+ <li class="sb-level2">
+ <a href="taking-solr-to-production.html">Taking Solr to Production</a>
+
+ </li>
+
+ <li class="sb-level2">
+ <a href="making-and-restoring-backups.html">Making and Restoring Backups</a>
+
+ </li>
+
+ <li class="sb-level2">
+ <a href="running-solr-on-hdfs.html">Running Solr on HDFS</a>
+
+ </li>
+
+ <li class="sb-level2">
+ <a href="aws-solrcloud-tutorial.html">SolrCloud on AWS EC2</a>
+
+ </li>
+
+ <li class="sb-level2">
+ <a href="upgrading-a-solr-cluster.html">Upgrading a Solr Cluster</a>
+
+ <ul>
+
+ <li class="sb-level3">
+ <a href="indexupgrader-tool.html">IndexUpgraderTool</a>
+
+ </li>
+
+ </ul>
+
+ </li>
+
+ <li class="sb-level2">
+ <a href="solr-upgrade-notes.html">Solr Upgrade Notes</a>
+
+ <ul>
+
+ <li class="sb-level3">
+ <a href="major-changes-in-solr-8.html">Major Changes in Solr 8</a>
+
+ </li>
+
+ <li class="sb-level3">
+ <a href="major-changes-in-solr-7.html">Major Changes in Solr 7</a>
+
+ </li>
+
+ <li class="sb-level3">
+ <a href="major-changes-from-solr-5-to-solr-6.html">Major Changes from Solr 5 to Solr 6</a>
+
+ </li>
+
+ </ul>
+
+ </li>
+
+ </ul>
+
+ </li>
+
+ <li class="sb-level1">
+ <a href="using-the-solr-administration-user-interface.html">Using the Solr Administration User Interface</a>
+
+ <ul>
+
+ <li class="sb-level2">
+ <a href="overview-of-the-solr-admin-ui.html">Overview of the Solr Admin UI</a>
+
+ </li>
+
+ <li class="sb-level2">
+ <a href="logging.html">Logging</a>
+
+ </li>
+
+ <li class="sb-level2">
+ <a href="cloud-screens.html">Cloud Screens</a>
+
+ </li>
+
+ <li class="sb-level2">
+ <a href="collections-core-admin.html">Collections / Core Admin</a>
+
+ </li>
+
+ <li class="sb-level2">
+ <a href="java-properties.html">Java Properties</a>
+
+ </li>
+
+ <li class="sb-level2">
+ <a href="thread-dump.html">Thread Dump</a>
+
+ </li>
+
+ <li class="sb-level2">
+ <a href="suggestions-screen.html">Suggestions Screen</a>
+
+ </li>
+
+ <li class="sb-level2">
+ <a href="collection-specific-tools.html">Collection-Specific Tools</a>
+
+ <ul>
+
+ <li class="sb-level3">
+ <a href="analysis-screen.html">Analysis Screen</a>
+
+ </li>
+
+ <li class="sb-level3">
+ <a href="dataimport-screen.html">Dataimport Screen</a>
+
+ </li>
+
+ <li class="sb-level3">
+ <a href="documents-screen.html">Documents Screen</a>
+
+ </li>
+
+ <li class="sb-level3">
+ <a href="files-screen.html">Files Screen</a>
+
+ </li>
+
+ <li class="sb-level3">
+ <a href="query-screen.html">Query Screen</a>
+
+ </li>
+
+ <li class="sb-level3">
+ <a href="stream-screen.html">Stream Screen</a>
+
+ </li>
+
+ <li class="sb-level3">
+ <a href="schema-browser-screen.html">Schema Browser Screen</a>
+
+ </li>
+
+ </ul>
+
+ </li>
+
+ <li class="sb-level2">
+ <a href="core-specific-tools.html">Core-Specific Tools</a>
+
+ <ul>
+
+ <li class="sb-level3">
+ <a href="ping.html">Ping</a>
+
+ </li>
+
+ <li class="sb-level3">
+ <a href="plugins-stats-screen.html">Plugins & Stats Screen</a>
+
+ </li>
+
+ <li class="sb-level3">
+ <a href="replication-screen.html">Replication Screen</a>
+
+ </li>
+
+ <li class="sb-level3">
+ <a href="segments-info.html">Segments Info</a>
+
+ </li>
+
+ </ul>
+
+ </li>
+
+ </ul>
+
+ </li>
+
+ <li class="sb-level1">
+ <a href="documents-fields-and-schema-design.html">Documents, Fields, and Schema Design</a>
+
+ <ul>
+
+ <li class="sb-level2">
+ <a href="overview-of-documents-fields-and-schema-design.html">Overview of Documents, Fields, and Schema Design</a>
+
+ </li>
+
+ <li class="sb-level2">
+ <a href="solr-field-types.html">Solr Field Types</a>
+
+ <ul>
+
+ <li class="sb-level3">
+ <a href="field-type-definitions-and-properties.html">Field Type Definitions and Properties</a>
+
+ </li>
+
+ <li class="sb-level3">
+ <a href="field-types-included-with-solr.html">Field Types Included with Solr</a>
+
+ </li>
+
+ <li class="sb-level3">
+ <a href="working-with-currencies-and-exchange-rates.html">Working with Currencies and Exchange Rates</a>
+
+ </li>
+
+ <li class="sb-level3">
+ <a href="working-with-dates.html">Working with Dates</a>
+
+ </li>
+
+ <li class="sb-level3">
+ <a href="working-with-enum-fields.html">Working with Enum Fields</a>
+
+ </li>
+
+ <li class="sb-level3">
+ <a href="working-with-external-files-and-processes.html">Working with External Files and Processes</a>
+
+ </li>
+
+ <li class="sb-level3">
+ <a href="field-properties-by-use-case.html">Field Properties by Use Case</a>
+
+ </li>
+
+ </ul>
+
+ </li>
+
+ <li class="sb-level2">
+ <a href="defining-fields.html">Defining Fields</a>
+
+ </li>
+
+ <li class="sb-level2">
+ <a href="copying-fields.html">Copying Fields</a>
+
+ </li>
+
+ <li class="sb-level2">
+ <a href="dynamic-fields.html">Dynamic Fields</a>
+
+ </li>
+
+ <li class="sb-level2">
+ <a href="other-schema-elements.html">Other Schema Elements</a>
+
+ </li>
+
+ <li class="sb-level2">
+ <a href="schema-api.html">Schema API</a>
+
+ </li>
+
+ <li class="sb-level2">
+ <a href="putting-the-pieces-together.html">Putting the Pieces Together</a>
+
+ </li>
+
+ <li class="sb-level2">
+ <a href="docvalues.html">DocValues</a>
+
+ </li>
+
+ <li class="sb-level2">
+ <a href="schemaless-mode.html">Schemaless Mode</a>
+
+ </li>
+
+ <li class="sb-level2">
+ <a href="luke-request-handler.html">Luke Request Handler</a>
+
+ </li>
+
+ </ul>
+
+ </li>
+
+ <li class="sb-level1">
+ <a href="understanding-analyzers-tokenizers-and-filters.html">Understanding Analyzers, Tokenizers, and Filters</a>
+
+ <ul>
+
+ <li class="sb-level2">
+ <a href="analyzers.html">Analyzers</a>
+
+ </li>
+
+ <li class="sb-level2">
+ <a href="about-tokenizers.html">About Tokenizers</a>
+
+ </li>
+
+ <li class="sb-level2">
+ <a href="about-filters.html">About Filters</a>
+
+ </li>
+
+ <li class="sb-level2">
+ <a href="tokenizers.html">Tokenizers</a>
+
+ </li>
+
+ <li class="sb-level2">
+ <a href="filter-descriptions.html">Filter Descriptions</a>
+
+ </li>
+
+ <li class="sb-level2">
+ <a href="charfilterfactories.html">CharFilterFactories</a>
+
+ </li>
+
+ <li class="sb-level2">
+ <a href="language-analysis.html">Language Analysis</a>
+
+ </li>
+
+ <li class="sb-level2">
+ <a href="phonetic-matching.html">Phonetic Matching</a>
+
+ </li>
+
+ <li class="sb-level2">
+ <a href="running-your-analyzer.html">Running Your Analyzer</a>
+
+ </li>
+
+ </ul>
+
+ </li>
+
+ <li class="sb-level1">
+ <a href="indexing-and-basic-data-operations.html">Indexing and Basic Data Operations</a>
+
+ <ul>
+
+ <li class="sb-level2">
+ <a href="introduction-to-solr-indexing.html">Introduction to Solr Indexing</a>
+
+ </li>
+
+ <li class="sb-level2">
+ <a href="post-tool.html">Post Tool</a>
+
+ </li>
+
+ <li class="sb-level2">
+ <a href="uploading-data-with-index-handlers.html">Uploading Data with Index Handlers</a>
+
+ <ul>
+
+ <li class="sb-level3">
+ <a href="transforming-and-indexing-custom-json.html">Transforming and Indexing Custom JSON</a>
+
+ </li>
+
+ </ul>
+
+ </li>
+
+ <li class="sb-level2">
+ <a href="indexing-nested-documents.html">Indexing Nested Child Documents</a>
+
+ </li>
+
+ <li class="sb-level2">
+ <a href="uploading-data-with-solr-cell-using-apache-tika.html">Uploading Data with Solr Cell using Apache Tika</a>
+
+ </li>
+
+ <li class="sb-level2">
+ <a href="uploading-structured-data-store-data-with-the-data-import-handler.html">Uploading Structured Data Store Data with the Data Import Handler</a>
+
+ </li>
+
+ <li class="sb-level2">
+ <a href="updating-parts-of-documents.html">Updating Parts of Documents</a>
+
+ </li>
+
+ <li class="sb-level2">
+ <a href="detecting-languages-during-indexing.html">Detecting Languages During Indexing</a>
+
+ </li>
+
+ <li class="sb-level2">
+ <a href="de-duplication.html">De-Duplication</a>
+
+ </li>
+
+ <li class="sb-level2">
+ <a href="content-streams.html">Content Streams</a>
+
+ </li>
+
+ <li class="sb-level2">
+ <a href="reindexing.html">Reindexing</a>
+
+ </li>
+
+ </ul>
+
+ </li>
+
+ <li class="sb-level1">
+ <a href="searching.html">Searching</a>
+
+ <ul>
+
+ <li class="sb-level2">
+ <a href="overview-of-searching-in-solr.html">Overview of Searching in Solr</a>
+
+ </li>
+
+ <li class="sb-level2">
+ <a href="velocity-search-ui.html">Velocity Search UI</a>
+
+ </li>
+
+ <li class="sb-level2">
+ <a href="relevance.html">Relevance</a>
+
+ </li>
+
+ <li class="sb-level2">
+ <a href="query-syntax-and-parsing.html">Query Syntax and Parsing</a>
+
+ <ul>
+
+ <li class="sb-level3">
+ <a href="common-query-parameters.html">Common Query Parameters</a>
+
+ </li>
+
+ <li class="sb-level3">
+ <a href="the-standard-query-parser.html">The Standard Query Parser</a>
+
+ </li>
+
+ <li class="sb-level3">
+ <a href="the-dismax-query-parser.html">The DisMax Query Parser</a>
+
+ </li>
+
+ <li class="sb-level3">
+ <a href="the-extended-dismax-query-parser.html">The Extended DisMax (eDismax) Query Parser</a>
+
+ </li>
+
+ <li class="sb-level3">
+ <a href="function-queries.html">Function Queries</a>
+
+ </li>
+
+ <li class="sb-level3">
+ <a href="local-parameters-in-queries.html">Local Parameters in Queries</a>
+
+ </li>
+
+ <li class="sb-level3">
+ <a href="other-parsers.html">Other Parsers</a>
+
+ </li>
+
+ </ul>
+
+ </li>
+
+ <li class="sb-level2">
+ <a href="json-request-api.html">JSON Request API</a>
+
+ <ul>
+
+ <li class="sb-level3">
+ <a href="json-query-dsl.html">JSON Query DSL</a>
+
+ </li>
+
+ </ul>
+
+ </li>
+
+ <li class="sb-level2">
+ <a href="json-facet-api.html">JSON Facet API</a>
+
+ <ul>
+
+ <li class="sb-level3">
+ <a href="json-faceting-domain-changes.html">JSON Faceting Domain Changes</a>
+
+ </li>
+
+ </ul>
+
+ </li>
+
+ <li class="sb-level2">
+ <a href="faceting.html">Faceting</a>
+
+ <ul>
+
+ <li class="sb-level3">
+ <a href="blockjoin-faceting.html">BlockJoin Faceting</a>
+
+ </li>
+
+ </ul>
+
+ </li>
+
+ <li class="sb-level2">
+ <a href="highlighting.html">Highlighting</a>
+
+ </li>
+
+ <li class="sb-level2">
+ <a href="spell-checking.html">Spell Checking</a>
+
+ </li>
+
+ <li class="sb-level2">
+ <a href="query-re-ranking.html">Query Re-Ranking</a>
+
+ <ul>
+
+ <li class="sb-level3">
+ <a href="learning-to-rank.html">Learning To Rank</a>
+
+ </li>
+
+ </ul>
+
+ </li>
+
+ <li class="sb-level2">
+ <a href="transforming-result-documents.html">Transforming Result Documents</a>
+
+ </li>
+
+ <li class="sb-level2">
+ <a href="searching-nested-documents.html">Searching Nested Child Documents</a>
+
+ </li>
+
+ <li class="sb-level2">
+ <a href="suggester.html">Suggester</a>
+
+ </li>
+
+ <li class="sb-level2">
+ <a href="morelikethis.html">MoreLikeThis</a>
+
+ </li>
+
+ <li class="sb-level2">
+ <a href="pagination-of-results.html">Pagination of Results</a>
+
+ </li>
+
+ <li class="sb-level2">
+ <a href="collapse-and-expand-results.html">Collapse and Expand Results</a>
+
+ </li>
+
+ <li class="sb-level2">
+ <a href="result-grouping.html">Result Grouping</a>
+
+ </li>
+
+ <li class="sb-level2">
+ <a href="spatial-search.html">Spatial Search</a>
+
+ </li>
+
+ <li class="sb-level2">
+ <a href="the-terms-component.html">The Terms Component</a>
+
+ </li>
+
+ <li class="sb-level2">
+ <a href="the-term-vector-component.html">The Term Vector Component</a>
+
+ </li>
+
+ <li class="sb-level2">
+ <a href="the-stats-component.html">The Stats Component</a>
+
+ </li>
+
+ <li class="sb-level2">
+ <a href="the-query-elevation-component.html">The Query Elevation Component</a>
+
+ </li>
+
+ <li class="sb-level2">
+ <a href="the-tagger-handler.html">The Tagger Handler</a>
+
+ </li>
+
+ <li class="sb-level2">
+ <a href="response-writers.html">Response Writers</a>
+
+ <ul>
+
+ <li class="sb-level3">
+ <a href="velocity-response-writer.html">Velocity Response Writer</a>
+
+ </li>
+
+ </ul>
+
+ </li>
+
+ <li class="sb-level2">
+ <a href="near-real-time-searching.html">Near Real Time Searching</a>
+
+ </li>
+
+ <li class="sb-level2">
+ <a href="realtime-get.html">RealTime Get</a>
+
+ </li>
+
+ <li class="sb-level2">
+ <a href="exporting-result-sets.html">Exporting Result Sets</a>
+
+ </li>
+
+ <li class="sb-level2">
+ <a href="parallel-sql-interface.html">Parallel SQL Interface</a>
+
+ <ul>
+
+ <li class="sb-level3">
+ <a href="solr-jdbc-dbvisualizer.html">Solr JDBC - DbVisualizer</a>
+
+ </li>
+
+ <li class="sb-level3">
+ <a href="solr-jdbc-squirrel-sql.html">Solr JDBC - SQuirreL SQL</a>
+
+ </li>
+
+ <li class="sb-level3">
+ <a href="solr-jdbc-apache-zeppelin.html">Solr JDBC - Apache Zeppelin</a>
+
+ </li>
+
+ <li class="sb-level3">
+ <a href="solr-jdbc-python-jython.html">Solr JDBC - Python/Jython</a>
+
+ </li>
+
+ <li class="sb-level3">
+ <a href="solr-jdbc-r.html">Solr JDBC - R</a>
+
+ </li>
+
+ </ul>
+
+ </li>
+
+ <li class="sb-level2">
+ <a href="analytics.html">Analytics Component</a>
+
+ <ul>
+
+ <li class="sb-level3">
+ <a href="analytics-expression-sources.html">Analytics Expression Sources</a>
+
+ </li>
+
+ <li class="sb-level3">
+ <a href="analytics-mapping-functions.html">Analytics Mapping Functions</a>
+
+ </li>
+
+ <li class="sb-level3">
+ <a href="analytics-reduction-functions.html">Analytics Reduction Functions</a>
+
+ </li>
+
+ </ul>
+
+ </li>
+
+ </ul>
+
+ </li>
+
+ <li class="sb-level1">
+ <a href="streaming-expressions.html">Streaming Expressions</a>
+
+ <ul>
+
+ <li class="sb-level2">
+ <a href="stream-source-reference.html">Stream Source Reference</a>
+
+ </li>
+
+ <li class="sb-level2">
+ <a href="stream-decorator-reference.html">Stream Decorator Reference</a>
+
+ </li>
+
+ <li class="sb-level2">
+ <a href="stream-evaluator-reference.html">Stream Evaluator Reference</a>
+
+ </li>
+
+ <li class="sb-level2">
+ <a href="math-expressions.html">Streaming Expressions and Math Expressions</a>
+
+ <ul>
+
+ <li class="sb-level3">
+ <a href="visualization.html">Visualization</a>
+
+ </li>
+
+ <li class="sb-level3">
+ <a href="math-start.html">Getting Started</a>
+
+ </li>
+
+ <li class="sb-level3">
+ <a href="loading.html">Loading Data</a>
+
+ </li>
+
+ <li class="sb-level3">
+ <a href="search-sample.html">Searching, Sampling and Aggregation</a>
+
+ </li>
+
+ <li class="sb-level3">
+ <a href="transform.html">Transforming Data</a>
+
+ </li>
+
+ <li class="sb-level3">
+ <a href="scalar-math.html">Scalar Math</a>
+
+ </li>
+
+ <li class="sb-level3">
+ <a href="vector-math.html">Vector Math</a>
+
+ </li>
+
+ <li class="sb-level3">
+ <a href="variables.html">Variables</a>
+
+ </li>
+
+ <li class="sb-level3">
+ <a href="matrix-math.html">Matrices and Matrix Math</a>
+
+ </li>
+
+ <li class="sb-level3">
+ <a href="term-vectors.html">Text Analysis and Term Vectors</a>
+
+ </li>
+
+ <li class="sb-level3">
+ <a href="probability-distributions.html">Probability Distributions</a>
+
+ </li>
+
+ <li class="sb-level3">
+ <a href="statistics.html">Statistics</a>
+
+ </li>
+
+ <li class="sb-level3">
+ <a href="regression.html">Linear Regression</a>
+
+ </li>
+
+ <li class="sb-level3">
+ <a href="curve-fitting.html">Curve Fitting</a>
+
+ </li>
+
+ <li class="sb-level3">
+ <a href="time-series.html">Time Series</a>
+
+ </li>
+
+ <li class="sb-level3">
+ <a href="numerical-analysis.html">Interpolation, Derivatives and Integrals</a>
+
+ </li>
+
+ <li class="sb-level3">
+ <a href="dsp.html">Digital Signal Processing</a>
+
+ </li>
+
+ <li class="sb-level3">
+ <a href="simulations.html">Monte Carlo Simulations</a>
+
+ </li>
+
+ <li class="sb-level3">
+ <a href="machine-learning.html">Machine Learning</a>
+
+ </li>
+
+ <li class="sb-level3">
+ <a href="computational-geometry.html">Computational Geometry</a>
+
+ </li>
+
+ <li class="sb-level3">
+ <a href="logs.html">Log Analytics</a>
+
+ </li>
+
+ </ul>
+
+ </li>
+
+ <li class="sb-level2">
+ <a href="graph-traversal.html">Graph Traversal</a>
+
+ </li>
+
+ <li class="sb-level2">
+ <a href="stream-api.html">Stream Request Handler API</a>
+
+ </li>
+
+ </ul>
+
+ </li>
+
+ <li class="sb-level1">
+ <a href="solrcloud.html">SolrCloud</a>
+
+ <ul>
+
+ <li class="sb-level2">
+ <a href="getting-started-with-solrcloud.html">Getting Started with SolrCloud</a>
+
+ </li>
+
+ <li class="sb-level2">
+ <a href="how-solrcloud-works.html">How SolrCloud Works</a>
+
+ <ul>
+
+ <li class="sb-level3">
+ <a href="shards-and-indexing-data-in-solrcloud.html">Shards and Indexing Data in SolrCloud</a>
+
+ </li>
+
+ <li class="sb-level3">
+ <a href="distributed-requests.html">Distributed Requests</a>
+
+ </li>
+
+ <li class="sb-level3">
+ <a href="aliases.html">Aliases</a>
+
+ </li>
+
+ </ul>
+
+ </li>
+
+ <li class="sb-level2">
+ <a href="solrcloud-resilience.html">SolrCloud Resilience</a>
+
+ <ul>
+
+ <li class="sb-level3">
+ <a href="solrcloud-recoveries-and-write-tolerance.html">SolrCloud Recoveries and Write Tolerance</a>
+
+ </li>
+
+ <li class="sb-level3">
+ <a href="solrcloud-query-routing-and-read-tolerance.html">SolrCloud Query Routing And Read Tolerance</a>
+
+ </li>
+
+ </ul>
+
+ </li>
+
+ <li class="sb-level2">
+ <a href="solrcloud-configuration-and-parameters.html">SolrCloud Configuration and Parameters</a>
+
+ <ul>
+
+ <li class="sb-level3">
+ <a href="setting-up-an-external-zookeeper-ensemble.html">Setting Up an External ZooKeeper Ensemble</a>
+
+ </li>
+
+ <li class="sb-level3">
+ <a href="using-zookeeper-to-manage-configuration-files.html">Using ZooKeeper to Manage Configuration Files</a>
+
+ </li>
+
+ <li class="sb-level3">
+ <a href="collections-api.html">Collections API</a>
+
+ <ul>
+
+ <li class="sb-level4">
+ <a href="cluster-node-management.html">Cluster and Node Managment Commands</a>
+
+ </li>
+
+ <li class="sb-level4">
+ <a href="collection-management.html">Collection Management Commands</a>
+
+ </li>
+
+ <li class="sb-level4">
+ <a href="collection-aliasing.html">Collection Aliasing</a>
+
+ </li>
+
+ <li class="sb-level4">
+ <a href="shard-management.html">Shard Management Commands</a>
+
+ </li>
+
+ <li class="sb-level4">
+ <a href="replica-management.html">Replica Management Commands</a>
+
+ </li>
+
+ </ul>
+
+ </li>
+
+ <li class="sb-level3">
+ <a href="parameter-reference.html">Parameter Reference</a>
+
+ </li>
+
+ <li class="sb-level3">
+ <a href="command-line-utilities.html">Command Line Utilities</a>
+
+ </li>
+
+ <li class="sb-level3">
+ <a href="solrcloud-with-legacy-configuration-files.html">SolrCloud with Legacy Configuration Files</a>
+
+ </li>
+
+ <li class="sb-level3">
+ <a href="configsets-api.html">Configsets API</a>
+
+ </li>
+
+ </ul>
+
+ </li>
+
+ <li class="sb-level2">
+ <a href="rule-based-replica-placement.html">Rule-based Replica Placement</a>
+
+ </li>
+
+ <li class="sb-level2">
+ <a href="cross-data-center-replication-cdcr.html">Cross Data Center Replication (CDCR)</a>
+
+ <ul>
+
+ <li class="sb-level3">
+ <a href="cdcr-architecture.html">CDCR Architecture</a>
+
+ </li>
+
+ <li class="sb-level3">
+ <a href="cdcr-config.html">CDCR Configuration</a>
+
+ </li>
+
+ <li class="sb-level3">
+ <a href="cdcr-operations.html">Cross Data Center Replication Operations</a>
+
+ </li>
+
+ <li class="sb-level3">
+ <a href="cdcr-api.html">CDCR API</a>
+
+ </li>
+
+ </ul>
+
+ </li>
+
+ <li class="sb-level2">
+ <a href="solrcloud-autoscaling.html">SolrCloud Autoscaling</a>
+
+ <ul>
+
+ <li class="sb-level3">
+ <a href="solrcloud-autoscaling-overview.html">Overview of SolrCloud Autoscaling</a>
+
+ </li>
+
+ <li class="sb-level3">
+ <a href="solrcloud-autoscaling-policy-preferences.html">Autoscaling Policy and Preferences</a>
+
+ </li>
+
+ <li class="sb-level3">
+ <a href="solrcloud-autoscaling-triggers.html">SolrCloud Autoscaling Triggers</a>
+
+ </li>
+
+ <li class="sb-level3">
+ <a href="solrcloud-autoscaling-trigger-actions.html">SolrCloud Autoscaling Trigger Actions</a>
+
+ </li>
+
+ <li class="sb-level3">
+ <a href="solrcloud-autoscaling-listeners.html">SolrCloud Autoscaling Listeners</a>
+
+ </li>
+
+ <li class="sb-level3">
+ <a href="solrcloud-autoscaling-auto-add-replicas.html">SolrCloud Autoscaling Automatically Adding Replicas</a>
+
+ </li>
+
+ <li class="sb-level3">
+ <a href="solrcloud-autoscaling-fault-tolerance.html">SolrCloud Autoscaling Fault Tolerance</a>
+
+ </li>
+
+ <li class="sb-level3">
+ <a href="solrcloud-autoscaling-api.html">Autoscaling API</a>
+
+ </li>
+
+ <li class="sb-level3">
+ <a href="migrate-to-policy-rule.html">Migrating Rule-Based Replica Rules to Autoscaling Policies</a>
+
+ </li>
+
+ </ul>
+
+ </li>
+
+ <li class="sb-level2">
+ <a href="colocating-collections.html">Colocating Collections</a>
+
+ </li>
+
+ </ul>
+
+ </li>
+
+ <li class="sb-level1">
+ <a href="legacy-scaling-and-distribution.html">Legacy Scaling and Distribution</a>
+
+ <ul>
+
+ <li class="sb-level2">
+ <a href="introduction-to-scaling-and-distribution.html">Introduction to Scaling and Distribution</a>
+
+ </li>
+
+ <li class="sb-level2">
+ <a href="distributed-search-with-index-sharding.html">Distributed Search with Index Sharding</a>
+
+ </li>
+
+ <li class="sb-level2">
+ <a href="index-replication.html">Index Replication</a>
+
+ </li>
+
+ <li class="sb-level2">
+ <a href="combining-distribution-and-replication.html">Combining Distribution and Replication</a>
+
+ </li>
+
+ <li class="sb-level2">
+ <a href="merging-indexes.html">Merging Indexes</a>
+
+ </li>
+
+ </ul>
+
+ </li>
+
+ <li class="sb-level1">
+ <a href="circuit-breakers.html">Circuit Breakers</a>
+
+ </li>
+
+ <li class="sb-level1">
+ <a href="solr-plugins.html">Solr Plugins</a>
+
+ <ul>
+
+ <li class="sb-level2">
+ <a href="libs.html">Lib Directories and Directives</a>
+
+ </li>
+
+ <li class="sb-level2">
+ <a href="package-manager.html">Package Management</a>
+
+ <ul>
+
+ <li class="sb-level3">
+ <a href="package-manager-internals.html">Package Manager Internals</a>
+
+ </li>
+
+ </ul>
+
+ </li>
+
+ <li class="sb-level2">
+ <a href="adding-custom-plugins-in-solrcloud-mode.html">Adding Custom Plugins in SolrCloud Mode</a>
+
+ </li>
+
+ </ul>
+
+ </li>
+
+ <li class="sb-level1">
+ <a href="the-well-configured-solr-instance.html">The Well-Configured Solr Instance</a>
+
+ <ul>
+
+ <li class="sb-level2">
+ <a href="configuring-solrconfig-xml.html">Configuring solrconfig.xml</a>
+
+ <ul>
+
+ <li class="sb-level3">
+ <a href="datadir-and-directoryfactory-in-solrconfig.html">DataDir and DirectoryFactory in SolrConfig</a>
+
+ </li>
+
+ <li class="sb-level3">
+ <a href="schema-factory-definition-in-solrconfig.html">Schema Factory Definition in SolrConfig</a>
+
+ </li>
+
+ <li class="sb-level3">
+ <a href="indexconfig-in-solrconfig.html">IndexConfig in SolrConfig</a>
+
+ </li>
+
+ <li class="sb-level3">
+ <a href="requesthandlers-and-searchcomponents-in-solrconfig.html">RequestHandlers and SearchComponents in SolrConfig</a>
+
+ </li>
+
+ <li class="sb-level3">
+ <a href="initparams-in-solrconfig.html">InitParams in SolrConfig</a>
+
+ </li>
+
+ <li class="sb-level3">
+ <a href="updatehandlers-in-solrconfig.html">UpdateHandlers in SolrConfig</a>
+
+ </li>
+
+ <li class="sb-level3">
+ <a href="query-settings-in-solrconfig.html">Query Settings in SolrConfig</a>
+
+ </li>
+
+ <li class="sb-level3">
+ <a href="requestdispatcher-in-solrconfig.html">RequestDispatcher in SolrConfig</a>
+
+ </li>
+
+ <li class="sb-level3">
+ <a href="update-request-processors.html">Update Request Processors</a>
+
+ </li>
+
+ <li class="sb-level3">
+ <a href="codec-factory.html">Codec Factory</a>
+
+ </li>
+
+ </ul>
+
+ </li>
+
+ <li class="sb-level2">
+ <a href="solr-cores-and-solr-xml.html">Solr Cores and solr.xml</a>
+
+ <ul>
+
+ <li class="sb-level3">
+ <a href="format-of-solr-xml.html">Format of solr.xml</a>
+
+ </li>
+
+ <li class="sb-level3">
+ <a href="defining-core-properties.html">Defining core.properties</a>
+
+ </li>
+
+ <li class="sb-level3">
+ <a href="coreadmin-api.html">CoreAdmin API</a>
+
+ </li>
+
+ <li class="sb-level3">
+ <a href="config-sets.html">Configsets</a>
+
+ </li>
+
+ </ul>
+
+ </li>
+
+ <li class="sb-level2">
+ <a href="resource-loading.html">Resource Loading</a>
+
+ </li>
+
+ <li class="sb-level2">
+ <a href="configuration-apis.html">Configuration APIs</a>
+
+ <ul>
+
+ <li class="sb-level3">
+ <a href="blob-store-api.html">Blob Store API</a>
+
+ </li>
+
+ <li class="sb-level3">
+ <a href="config-api.html">Config API</a>
+
+ </li>
+
+ <li class="sb-level3">
+ <a href="request-parameters-api.html">Request Parameters API</a>
+
+ </li>
+
+ <li class="sb-level3">
+ <a href="managed-resources.html">Managed Resources</a>
+
+ </li>
+
+ </ul>
+
+ </li>
+
+ <li class="sb-level2">
+ <a href="implicit-requesthandlers.html">Implicit RequestHandlers</a>
+
+ </li>
+
+ <li class="sb-level2">
+ <a href="jvm-settings.html">JVM Settings</a>
+
+ </li>
+
+ <li class="sb-level2">
+ <a href="v2-api.html">v2 API</a>
+
+ </li>
+
+ </ul>
+
+ </li>
+
+ <li class="sb-level1">
+ <a href="monitoring-solr.html">Monitoring Solr</a>
+
+ <ul>
+
+ <li class="sb-level2">
+ <a href="metrics-reporting.html">Metrics Reporting</a>
+
+ </li>
+
+ <li class="sb-level2">
+ <a href="metrics-history.html">Metrics History</a>
+
+ </li>
+
+ <li class="sb-level2">
+ <a href="mbean-request-handler.html">MBean Request Handler</a>
+
+ </li>
+
+ <li class="sb-level2">
+ <a href="configuring-logging.html">Configuring Logging</a>
+
+ </li>
+
+ <li class="sb-level2">
+ <a href="using-jmx-with-solr.html">Using JMX with Solr</a>
+
+ </li>
+
+ <li class="sb-level2">
+ <a href="monitoring-solr-with-prometheus-and-grafana.html">Monitoring Solr with Prometheus and Grafana</a>
+
+ </li>
+
+ <li class="sb-level2">
+ <a href="performance-statistics-reference.html">Performance Statistics Reference</a>
+
+ </li>
+
+ <li class="sb-level2">
+ <a href="solr-tracing.html">Distributed Solr Tracing</a>
+
+ </li>
+
+ </ul>
+
+ </li>
+
+ <li class="sb-level1">
+ <a href="securing-solr.html">Securing Solr</a>
+
+ <ul>
+
+ <li class="sb-level2">
+ <a href="authentication-and-authorization-plugins.html">Configuring Authentication, Authorization and Audit Logging</a>
+
+ <ul>
+
+ <li class="sb-level3">
+ <a href="basic-authentication-plugin.html">Basic Authentication Plugin</a>
+
+ </li>
+
+ <li class="sb-level3">
+ <a href="hadoop-authentication-plugin.html">Hadoop Authentication Plugin</a>
+
+ </li>
+
+ <li class="sb-level3">
+ <a href="kerberos-authentication-plugin.html">Kerberos Authentication Plugin</a>
+
+ </li>
+
+ <li class="sb-level3">
+ <a href="rule-based-authorization-plugin.html">Rule-Based Authorization Plugins</a>
+
+ </li>
+
+ <li class="sb-level3">
+ <a href="jwt-authentication-plugin.html">JWT Authentication Plugin</a>
+
+ </li>
+
+ </ul>
+
+ </li>
+
+ <li class="sb-level2">
+ <a href="enabling-ssl.html">Enabling SSL</a>
+
+ </li>
+
+ <li class="sb-level2">
+ <a href="audit-logging.html">Audit Logging</a>
+
+ </li>
+
+ <li class="sb-level2">
+ <a href="zookeeper-access-control.html">ZooKeeper Access Control</a>
+
+ </li>
+
+ </ul>
+
+ </li>
+
+ <li class="sb-level1">
+ <a href="client-apis.html">Client APIs</a>
+
+ <ul>
+
+ <li class="sb-level2">
+ <a href="introduction-to-client-apis.html">Introduction to Client APIs</a>
+
+ </li>
+
+ <li class="sb-level2">
+ <a href="choosing-an-output-format.html">Choosing an Output Format</a>
+
+ </li>
+
+ <li class="sb-level2">
+ <a href="using-solrj.html">Using SolrJ</a>
+
+ </li>
+
+ <li class="sb-level2">
+ <a href="using-javascript.html">Using JavaScript</a>
+
+ </li>
+
+ <li class="sb-level2">
+ <a href="using-python.html">Using Python</a>
+
+ </li>
+
+ <li class="sb-level2">
+ <a href="using-solr-from-ruby.html">Using Solr From Ruby</a>
+
+ </li>
+
+ <li class="sb-level2">
+ <a href="client-api-lineup.html">Other Clients</a>
+
+ </li>
+
+ </ul>
+
+ </li>
+
+ <li class="sb-level1">
+ <a href="further-assistance.html">Further Assistance</a>
+
+ </li>
+
+ <li class="sb-level1">
+ <a href="solr-glossary.html">Solr Glossary</a>
+
+ </li>
+
+ <li class="sb-level1">
+ <a href="errata.html">Errata</a>
+
+ </li>
+
+ <li class="sb-level1">
+ <a href="how-to-contribute.html">How to Contribute to Solr Documentation</a>
+
+ </li>
+
+</ul>
+
+
+
+<!-- set the 'active' class on the current page and ancestors -->
+<!-- this highlights the active parent class in the navgoco sidebar. this is critical so that the parent expands when you're viewing a page. This must appear below the sidebar code above. Otherwise, if placed inside customscripts.js, the script runs before the sidebar code runs and the class never gets inserted.-->
+<script>$("#mysidebar a[href='loading.html']").parents('li').toggleClass("active", true);</script>
+<!-- set the 'current' class on the current page and 'current-tree' on the current page + it's ancestors -->
+<!-- this can let us do css highlighting of the current page in the sidebar even if/when the user clicks around in the sidebar causing other sidebar elements to be 'active' -->
+<script>
+ $("#mysidebar a[href='loading.html']").parent('li').toggleClass("current", true);
+ $("#mysidebar a[href='loading.html']").parents('li').toggleClass("current-tree", true);
+</script>
+
+ </nav>
+ <div id="content" class="col-10">
+ <!-- Navigation -->
+<nav id="topnav" class="navbar navbar-expand-md fixed-top bg-light flex-nowrap col-10">
+ <div class="container-fluid">
+ <div class="navbar-header">
+
+ <img src="images/solr-sunOnly-small.png" width="30" height="30">
+ <a class="navbar-brand" href="index.html">
+ <span class="projectTitle">Solr 8.8</span>
+ </a>
+ </div>
+ <div class="collapse navbar-collapse justify-content-end" id="bs-example-navbar-collapse-1">
+ <ul class="nav navbar-nav navbar-right">
+ <!-- Link to Solr website -->
+ <li><a href="https://lucene.apache.org/solr/" target="_blank">Solr Website</a></li>
+ <!-- Other Guide Formats dropdown -->
+ <li class="dropdown">
+ <a href="#" class="dropdown-toggle" data-toggle="dropdown">Other Formats<b class="caret"></b></a>
+ <ul class="dropdown-menu">
+ <li><a href="https://archive.apache.org/dist/lucene/solr/ref-guide/" target="_blank">Archived PDFs</a></li>
+ <li><a href="https://lucene.apache.org/solr/guide/" target="_blank">Other Versions Online</a></li>
+ </ul>
+ </li>
+ <!-- Solr Resources dropdown -->
+ <li class="dropdown">
+ <a href="#" class="dropdown-toggle" data-toggle="dropdown">Solr Resources<b class="caret"></b></a>
+ <ul class="dropdown-menu">
+ <li><a href="https://lucene.apache.org/solr/8_8_0//solr-core/index.html" target="_blank">Solr Javadocs</a></li>
+ <li><a href="https://lucene.apache.org/solr/community.html#version-control" target="_blank">Lucene/Solr Source Code</a></li>
+ <li><a href="https://lucene.apache.org/solr/community.html" target="_blank">Solr Community Links</a></li>
+ </ul>
+ </li>
+
+ </ul>
+ </div>
+ </div>
+ <!-- /.container -->
+</nav>
+
+ <article class="post-content">
+
+
+<header class="header">
+ <h1 class="title-main">Loading Data</h1>
+</header>
+
+<div class="container-fluid">
+
+<!-- This makes the automatic TOC generation conditional.
+ Add ":page-show-toc: false" to front-matter to disable auto-TOC. -->
+
+ <nav class="toc float-right justify-content-end">
+ <ul class="sectlevel1">
+<li><a href="#reading-files">Reading Files</a></li>
+<li><a href="#parsing-csv-and-tsv-files">Parsing CSV and TSV Files</a></li>
+<li><a href="#visualizing">Visualizing</a></li>
+<li><a href="#selecting-fields-and-field-types">Selecting Fields and Field Types</a></li>
+<li><a href="#loading-data">Loading Data</a></li>
+<li><a href="#transforming-data">Transforming Data</a>
+<ul class="sectlevel2">
+<li><a href="#unique-ids">Unique IDs</a></li>
+<li><a href="#record-numbers">Record Numbers</a></li>
+<li><a href="#parsing-dates">Parsing Dates</a></li>
+<li><a href="#string-manipulation">String Manipulation</a></li>
+<li><a href="#filtering-results">Filtering Results</a></li>
+<li><a href="#handling-nulls">Handling Nulls</a></li>
+<li><a href="#text-analysis">Text Analysis</a></li>
+</ul>
+</li>
+</ul>
+ </nav>
+
+
+ <section class="content">
+ <section id="preamble" aria-label="Preamble"><p>Streaming expressions has support for reading, parsing, transforming, visualizing and loading CSV and TSV formatted data.
+These functions are designed to cut down the time spent on data preparation and allow users to begin data exploration before the data is loaded into Solr.</p></section>
+<section class="sect1"><h2 id="reading-files">Reading Files</h2><p>The <code>cat</code> function can be used to read files under the <strong>userfiles</strong> directory in
+<code>$SOLR_HOME</code>. The <code>cat</code> function takes two parameters.</p>
+<p>The first parameter is a comma-delimited list of paths.
+If the path list contains directories, <code>cat</code> will crawl all the files in the directory and sub-directories.
+If the path list contains only files <code>cat</code> will read just the specific files.</p>
+<p>The second parameter, <code>maxLines</code>, tells <code>cat</code> how many lines to read in total.
+If <code>maxLines</code> is not provided, <code>cat</code> will read all lines from each file it crawls.</p>
+<p>The <code>cat</code> function reads each line (up to <code>maxLines</code>) in the crawled files and for each line emits a tuple with two fields:</p>
+<div class="ulist"><ul><li><p><code>line</code>: The text in the line.</p></li><li><p><code>file</code>: The relative path of the file under $SOLR_HOME.</p></li></ul></div>
+<p>Below is an example of <code>cat</code> on the iris.csv file with a <code>maxLines</code> of <code>5</code>:</p>
+<div class="listingblock"><pre class="rouge highlight"><code class="language-text" data-lang="text"><pre class="highlight"><code>cat("iris.csv", maxLines="5")</code></pre></code></pre></div>
+<p>When this expression is sent to the <code>/stream</code> handler it responds with:</p>
+<div class="listingblock"><pre class="rouge highlight"><code class="language-json" data-lang="json"><pre class="highlight"><code><span style="background-color: #f8f8f8">{</span><span style="color: #bbbbbb">
+ </span><span style="color: #000080">"result-set"</span><span style="background-color: #f8f8f8">:</span><span style="color: #bbbbbb"> </span><span style="background-color: #f8f8f8">{</span><span style="color: #bbbbbb">
+ </span><span style="color: #000080">"docs"</span><span style="background-color: #f8f8f8">:</span><span style="color: #bbbbbb"> </span><span style="background-color: #f8f8f8">[</span><span style="color: #bbbbbb">
+ </span><span style="background-color: #f8f8f8">{</span><span style="color: #bbbbbb">
+ </span><span style="color: #000080">"line"</span><span style="background-color: #f8f8f8">:</span><span style="color: #bbbbbb"> </span><span style="color: #d14">"sepal_length,sepal_width,petal_length,petal_width,species"</span><span style="background-color: #f8f8f8">,</span><span style="color: #bbbbbb">
+ </span><span style="color: #000080">"file"</span><span style="background-color: #f8f8f8">:</span><span style="color: #bbbbbb"> </span><span style="color: #d14">"iris.csv"</span><span style="color: #bbbbbb">
+ </span><span style="background-color: #f8f8f8">},</span><span style="color: #bbbbbb">
+ </span><span style="background-color: #f8f8f8">{</span><span style="color: #bbbbbb">
+ </span><span style="color: #000080">"line"</span><span style="background-color: #f8f8f8">:</span><span style="color: #bbbbbb"> </span><span style="color: #d14">"5.1,3.5,1.4,0.2,setosa"</span><span style="background-color: #f8f8f8">,</span><span style="color: #bbbbbb">
+ </span><span style="color: #000080">"file"</span><span style="background-color: #f8f8f8">:</span><span style="color: #bbbbbb"> </span><span style="color: #d14">"iris.csv"</span><span style="color: #bbbbbb">
+ </span><span style="background-color: #f8f8f8">},</span><span style="color: #bbbbbb">
+ </span><span style="background-color: #f8f8f8">{</span><span style="color: #bbbbbb">
+ </span><span style="color: #000080">"line"</span><span style="background-color: #f8f8f8">:</span><span style="color: #bbbbbb"> </span><span style="color: #d14">"4.9,3,1.4,0.2,setosa"</span><span style="background-color: #f8f8f8">,</span><span style="color: #bbbbbb">
+ </span><span style="color: #000080">"file"</span><span style="background-color: #f8f8f8">:</span><span style="color: #bbbbbb"> </span><span style="color: #d14">"iris.csv"</span><span style="color: #bbbbbb">
+ </span><span style="background-color: #f8f8f8">},</span><span style="color: #bbbbbb">
+ </span><span style="background-color: #f8f8f8">{</span><span style="color: #bbbbbb">
+ </span><span style="color: #000080">"line"</span><span style="background-color: #f8f8f8">:</span><span style="color: #bbbbbb"> </span><span style="color: #d14">"4.7,3.2,1.3,0.2,setosa"</span><span style="background-color: #f8f8f8">,</span><span style="color: #bbbbbb">
+ </span><span style="color: #000080">"file"</span><span style="background-color: #f8f8f8">:</span><span style="color: #bbbbbb"> </span><span style="color: #d14">"iris.csv"</span><span style="color: #bbbbbb">
+ </span><span style="background-color: #f8f8f8">},</span><span style="color: #bbbbbb">
+ </span><span style="background-color: #f8f8f8">{</span><span style="color: #bbbbbb">
+ </span><span style="color: #000080">"line"</span><span style="background-color: #f8f8f8">:</span><span style="color: #bbbbbb"> </span><span style="color: #d14">"4.6,3.1,1.5,0.2,setosa"</span><span style="background-color: #f8f8f8">,</span><span style="color: #bbbbbb">
+ </span><span style="color: #000080">"file"</span><span style="background-color: #f8f8f8">:</span><span style="color: #bbbbbb"> </span><span style="color: #d14">"iris.csv"</span><span style="color: #bbbbbb">
+ </span><span style="background-color: #f8f8f8">},</span><span style="color: #bbbbbb">
+ </span><span style="background-color: #f8f8f8">{</span><span style="color: #bbbbbb">
+ </span><span style="color: #000080">"EOF"</span><span style="background-color: #f8f8f8">:</span><span style="color: #bbbbbb"> </span><span style="color: #000000;font-weight: bold">true</span><span style="background-color: #f8f8f8">,</span><span style="color: #bbbbbb">
+ </span><span style="color: #000080">"RESPONSE_TIME"</span><span style="background-color: #f8f8f8">:</span><span style="color: #bbbbbb"> </span><span style="color: #009999">0</span><span style="color: #bbbbbb">
+ </span><span style="background-color: #f8f8f8">}</span><span style="color: #bbbbbb">
+ </span><span style="background-color: #f8f8f8">]</span><span style="color: #bbbbbb">
+ </span><span style="background-color: #f8f8f8">}</span><span style="color: #bbbbbb">
+</span><span style="background-color: #f8f8f8">}</span></code></pre></code></pre></div></section>
+<section class="sect1"><h2 id="parsing-csv-and-tsv-files">Parsing CSV and TSV Files</h2><p>The <code>parseCSV</code> and <code>parseTSV</code> functions wrap the <code>cat</code> function and parse CSV
+(comma separated values) and TSV (tab separated values). Both of these functions
+expect a CSV or TSV header record at the beginning of each file.</p>
+<p>Both <code>parseCSV</code> and <code>parseTSV</code> emit tuples with the header values mapped to their
+corresponding values in each line.</p>
+<div class="listingblock"><pre class="rouge highlight"><code class="language-text" data-lang="text"><pre class="highlight"><code>parseCSV(cat("iris.csv", maxLines="5"))</code></pre></code></pre></div>
+<p>When this expression is sent to the <code>/stream</code> handler it responds with:</p>
+<div class="listingblock"><pre class="rouge highlight"><code class="language-json" data-lang="json"><pre class="highlight"><code><span style="background-color: #f8f8f8">{</span><span style="color: #bbbbbb">
+ </span><span style="color: #000080">"result-set"</span><span style="background-color: #f8f8f8">:</span><span style="color: #bbbbbb"> </span><span style="background-color: #f8f8f8">{</span><span style="color: #bbbbbb">
+ </span><span style="color: #000080">"docs"</span><span style="background-color: #f8f8f8">:</span><span style="color: #bbbbbb"> </span><span style="background-color: #f8f8f8">[</span><span style="color: #bbbbbb">
+ </span><span style="background-color: #f8f8f8">{</span><span style="color: #bbbbbb">
+ </span><span style="color: #000080">"sepal_width"</span><span style="background-color: #f8f8f8">:</span><span style="color: #bbbbbb"> </span><span style="color: #d14">"3.5"</span><span style="background-color: #f8f8f8">,</span><span style="color: #bbbbbb">
+ </span><span style="color: #000080">"species"</span><span style="background-color: #f8f8f8">:</span><span style="color: #bbbbbb"> </span><span style="color: #d14">"setosa"</span><span style="background-color: #f8f8f8">,</span><span style="color: #bbbbbb">
+ </span><span style="color: #000080">"petal_width"</span><span style="background-color: #f8f8f8">:</span><span style="color: #bbbbbb"> </span><span style="color: #d14">"0.2"</span><span style="background-color: #f8f8f8">,</span><span style="color: #bbbbbb">
+ </span><span style="color: #000080">"sepal_length"</span><span style="background-color: #f8f8f8">:</span><span style="color: #bbbbbb"> </span><span style="color: #d14">"5.1"</span><span style="background-color: #f8f8f8">,</span><span style="color: #bbbbbb">
+ </span><span style="color: #000080">"id"</span><span style="background-color: #f8f8f8">:</span><span style="color: #bbbbbb"> </span><span style="color: #d14">"iris.csv_2"</span><span style="background-color: #f8f8f8">,</span><span style="color: #bbbbbb">
+ </span><span style="color: #000080">"petal_length"</span><span style="background-color: #f8f8f8">:</span><span style="color: #bbbbbb"> </span><span style="color: #d14">"1.4"</span><span style="color: #bbbbbb">
+ </span><span style="background-color: #f8f8f8">},</span><span style="color: #bbbbbb">
+ </span><span style="background-color: #f8f8f8">{</span><span style="color: #bbbbbb">
+ </span><span style="color: #000080">"sepal_width"</span><span style="background-color: #f8f8f8">:</span><span style="color: #bbbbbb"> </span><span style="color: #d14">"3"</span><span style="background-color: #f8f8f8">,</span><span style="color: #bbbbbb">
+ </span><span style="color: #000080">"species"</span><span style="background-color: #f8f8f8">:</span><span style="color: #bbbbbb"> </span><span style="color: #d14">"setosa"</span><span style="background-color: #f8f8f8">,</span><span style="color: #bbbbbb">
+ </span><span style="color: #000080">"petal_width"</span><span style="background-color: #f8f8f8">:</span><span style="color: #bbbbbb"> </span><span style="color: #d14">"0.2"</span><span style="background-color: #f8f8f8">,</span><span style="color: #bbbbbb">
+ </span><span style="color: #000080">"sepal_length"</span><span style="background-color: #f8f8f8">:</span><span style="color: #bbbbbb"> </span><span style="color: #d14">"4.9"</span><span style="background-color: #f8f8f8">,</span><span style="color: #bbbbbb">
+ </span><span style="color: #000080">"id"</span><span style="background-color: #f8f8f8">:</span><span style="color: #bbbbbb"> </span><span style="color: #d14">"iris.csv_3"</span><span style="background-color: #f8f8f8">,</span><span style="color: #bbbbbb">
+ </span><span style="color: #000080">"petal_length"</span><span style="background-color: #f8f8f8">:</span><span style="color: #bbbbbb"> </span><span style="color: #d14">"1.4"</span><span style="color: #bbbbbb">
+ </span><span style="background-color: #f8f8f8">},</span><span style="color: #bbbbbb">
+ </span><span style="background-color: #f8f8f8">{</span><span style="color: #bbbbbb">
+ </span><span style="color: #000080">"sepal_width"</span><span style="background-color: #f8f8f8">:</span><span style="color: #bbbbbb"> </span><span style="color: #d14">"3.2"</span><span style="background-color: #f8f8f8">,</span><span style="color: #bbbbbb">
+ </span><span style="color: #000080">"species"</span><span style="background-color: #f8f8f8">:</span><span style="color: #bbbbbb"> </span><span style="color: #d14">"setosa"</span><span style="background-color: #f8f8f8">,</span><span style="color: #bbbbbb">
+ </span><span style="color: #000080">"petal_width"</span><span style="background-color: #f8f8f8">:</span><span style="color: #bbbbbb"> </span><span style="color: #d14">"0.2"</span><span style="background-color: #f8f8f8">,</span><span style="color: #bbbbbb">
+ </span><span style="color: #000080">"sepal_length"</span><span style="background-color: #f8f8f8">:</span><span style="color: #bbbbbb"> </span><span style="color: #d14">"4.7"</span><span style="background-color: #f8f8f8">,</span><span style="color: #bbbbbb">
+ </span><span style="color: #000080">"id"</span><span style="background-color: #f8f8f8">:</span><span style="color: #bbbbbb"> </span><span style="color: #d14">"iris.csv_4"</span><span style="background-color: #f8f8f8">,</span><span style="color: #bbbbbb">
+ </span><span style="color: #000080">"petal_length"</span><span style="background-color: #f8f8f8">:</span><span style="color: #bbbbbb"> </span><span style="color: #d14">"1.3"</span><span style="color: #bbbbbb">
+ </span><span style="background-color: #f8f8f8">},</span><span style="color: #bbbbbb">
+ </span><span style="background-color: #f8f8f8">{</span><span style="color: #bbbbbb">
+ </span><span style="color: #000080">"sepal_width"</span><span style="background-color: #f8f8f8">:</span><span style="color: #bbbbbb"> </span><span style="color: #d14">"3.1"</span><span style="background-color: #f8f8f8">,</span><span style="color: #bbbbbb">
+ </span><span style="color: #000080">"species"</span><span style="background-color: #f8f8f8">:</span><span style="color: #bbbbbb"> </span><span style="color: #d14">"setosa"</span><span style="background-color: #f8f8f8">,</span><span style="color: #bbbbbb">
+ </span><span style="color: #000080">"petal_width"</span><span style="background-color: #f8f8f8">:</span><span style="color: #bbbbbb"> </span><span style="color: #d14">"0.2"</span><span style="background-color: #f8f8f8">,</span><span style="color: #bbbbbb">
+ </span><span style="color: #000080">"sepal_length"</span><span style="background-color: #f8f8f8">:</span><span style="color: #bbbbbb"> </span><span style="color: #d14">"4.6"</span><span style="background-color: #f8f8f8">,</span><span style="color: #bbbbbb">
+ </span><span style="color: #000080">"id"</span><span style="background-color: #f8f8f8">:</span><span style="color: #bbbbbb"> </span><span style="color: #d14">"iris.csv_5"</span><span style="background-color: #f8f8f8">,</span><span style="color: #bbbbbb">
+ </span><span style="color: #000080">"petal_length"</span><span style="background-color: #f8f8f8">:</span><span style="color: #bbbbbb"> </span><span style="color: #d14">"1.5"</span><span style="color: #bbbbbb">
+ </span><span style="background-color: #f8f8f8">},</span><span style="color: #bbbbbb">
+ </span><span style="background-color: #f8f8f8">{</span><span style="color: #bbbbbb">
+ </span><span style="color: #000080">"EOF"</span><span style="background-color: #f8f8f8">:</span><span style="color: #bbbbbb"> </span><span style="color: #000000;font-weight: bold">true</span><span style="background-color: #f8f8f8">,</span><span style="color: #bbbbbb">
+ </span><span style="color: #000080">"RESPONSE_TIME"</span><span style="background-color: #f8f8f8">:</span><span style="color: #bbbbbb"> </span><span style="color: #009999">1</span><span style="color: #bbbbbb">
+ </span><span style="background-color: #f8f8f8">}</span><span style="color: #bbbbbb">
+ </span><span style="background-color: #f8f8f8">]</span><span style="color: #bbbbbb">
+ </span><span style="background-color: #f8f8f8">}</span><span style="color: #bbbbbb">
+</span><span style="background-color: #f8f8f8">}</span></code></pre></code></pre></div></section>
+<section class="sect1"><h2 id="visualizing">Visualizing</h2><p>Once that data has been parsed into tuples with <code>parseCSV</code> or <code>parseTSV</code> it can be
+visualized using Zeppelin-Solr.</p>
+<p>The example below shows the output of the <code>parseCSV</code> function visualized as a table.</p>
+<div class="imageblock"><img src="images/math-expressions/csvtable.png" alt="csvtable"></div>
+<p>Columns from the table can then be visualized using one of Apache Zeppelin’s
+visualizations. The example below shows a scatter plot of the <code>petal_length</code> and <code>petal_width</code>
+grouped by <code>species</code>.</p>
+<div class="imageblock"><img src="images/math-expressions/csv.png" alt="csv"></div></section>
+<section class="sect1"><h2 id="selecting-fields-and-field-types">Selecting Fields and Field Types</h2><p>The <code>select</code> function can be used to select specific fields from
+the CSV file and map them to new field names for indexing.</p>
+<p>Fields in the CSV file can be mapped to field names with
+dynamic field suffixes. This approach allows for fine grain
+control over schema field types without having to make any
+changes to schema files.</p>
+<p>Below is an example of selecting fields and mapping them
+to specific field types.</p>
+<div class="imageblock"><img src="images/math-expressions/csvselect.png" alt="csvselect"></div></section>
+<section class="sect1"><h2 id="loading-data">Loading Data</h2><p>When the data is ready to load, the <code>update</code> function can be used to send the
+data to a SolrCloud collection for indexing.
+The <code>update</code> function adds documents to Solr in batches and returns a tuple for each batch with summary information about the batch and load.</p>
+<p>In the example below the <code>update</code> expression is run using Zeppelin-Solr because the data set is small.
+For larger loads it’s best to run the load from a curl command where the output of the <code>update</code> function can be spooled to disk.</p>
+<div class="imageblock"><img src="images/math-expressions/update.png" alt="update"></div></section>
+<section class="sect1"><h2 id="transforming-data">Transforming Data</h2><p>Streaming expressions and math expressions provide a powerful set of functions
+for transforming data.
+The section below shows some useful transformations that can be applied while analyzing, visualizing, and loading CSV and TSV files.</p>
+<section class="sect2"><h3 id="unique-ids">Unique IDs</h3><p>Both <code>parseCSV</code> and <code>parseTSV</code> emit an <strong>id</strong> field if one is not present in the data already.
+The <strong>id</strong> field is a concatenation of the file path and the line number. This is a
+convenient way to ensure that records have consistent ids if an id
+is not present in the file.</p>
+<p>You can also map any fields in the file to the id field using the <code>select</code> function.
+The <code>concat</code> function can be used to concatenate two or more fields in the file
+to create an id. Or the <code>uuid</code> function can be used to create a random unique id. If
+the <code>uuid</code> function is used the data cannot be reloaded without first deleting
+the data, as the <code>uuid</code> function does not produce the same id for each document
+on subsequent loads.</p>
+<p>Below is an example using the <code>concat</code> function to create a new id.</p>
+<div class="imageblock"><img src="images/math-expressions/selectconcat.png" alt="selectconcat"></div>
+<p>Below is an example using the <code>uuid</code> function to create a new id.</p>
+<div class="imageblock"><img src="images/math-expressions/selectuuid.png" alt="selectuuid"></div></section>
+<section class="sect2"><h3 id="record-numbers">Record Numbers</h3><p>The <code>recNum</code> function can be used inside of a <code>select</code> function to add a record number
+to each tuple. The record number is useful for tracking location in the result set
+and can be used for filtering strategies such as skipping, paging and striding described in
+the <a href="#filtering-results">Filtering Results</a> section below.</p>
+<p>The example below shows the syntax of the <code>recNum</code> function:</p>
+<div class="imageblock"><img src="images/math-expressions/recNum.png" alt="recNum"></div></section>
+<section class="sect2"><h3 id="parsing-dates">Parsing Dates</h3><p>The <code>dateTime</code> function can be used to parse dates into the ISO-8601 format
+needed for loading into a Solr date field.</p>
+<p>We can first inspect the format of the data time field in the CSV file:</p>
+<div class="listingblock"><pre class="rouge highlight"><code class="language-text" data-lang="text"><pre class="highlight"><code>select(parseCSV(cat("yr2017.csv", maxLines="2")),
+ id,
+ Created.Date)</code></pre></code></pre></div>
+<p>When this expression is sent to the <code>/stream</code> handler it responds with:</p>
+<div class="listingblock"><pre class="rouge highlight"><code class="language-json" data-lang="json"><pre class="highlight"><code><span style="background-color: #f8f8f8">{</span><span style="color: #bbbbbb">
+ </span><span style="color: #000080">"result-set"</span><span style="background-color: #f8f8f8">:</span><span style="color: #bbbbbb"> </span><span style="background-color: #f8f8f8">{</span><span style="color: #bbbbbb">
+ </span><span style="color: #000080">"docs"</span><span style="background-color: #f8f8f8">:</span><span style="color: #bbbbbb"> </span><span style="background-color: #f8f8f8">[</span><span style="color: #bbbbbb">
+ </span><span style="background-color: #f8f8f8">{</span><span style="color: #bbbbbb">
+ </span><span style="color: #000080">"id"</span><span style="background-color: #f8f8f8">:</span><span style="color: #bbbbbb"> </span><span style="color: #d14">"yr2017.csv_2"</span><span style="background-color: #f8f8f8">,</span><span style="color: #bbbbbb">
+ </span><span style="color: #000080">"Created.Date"</span><span style="background-color: #f8f8f8">:</span><span style="color: #bbbbbb"> </span><span style="color: #d14">"01/01/2017 12:00:00 AM"</span><span style="color: #bbbbbb">
+ </span><span style="background-color: #f8f8f8">},</span><span style="color: #bbbbbb">
+ </span><span style="background-color: #f8f8f8">{</span><span style="color: #bbbbbb">
+ </span><span style="color: #000080">"EOF"</span><span style="background-color: #f8f8f8">:</span><span style="color: #bbbbbb"> </span><span style="color: #000000;font-weight: bold">true</span><span style="background-color: #f8f8f8">,</span><span style="color: #bbbbbb">
+ </span><span style="color: #000080">"RESPONSE_TIME"</span><span style="background-color: #f8f8f8">:</span><span style="color: #bbbbbb"> </span><span style="color: #009999">0</span><span style="color: #bbbbbb">
+ </span><span style="background-color: #f8f8f8">}</span><span style="color: #bbbbbb">
+ </span><span style="background-color: #f8f8f8">]</span><span style="color: #bbbbbb">
+ </span><span style="background-color: #f8f8f8">}</span><span style="color: #bbbbbb">
+</span><span style="background-color: #f8f8f8">}</span></code></pre></code></pre></div>
+<p>Then we can use the <code>dateTime</code> function to format the datetime and
+map it to a Solr date field.</p>
+<p>The <code>dateTime</code> function takes three parameters. The field in the data
+with the date string, a template to parse the date using a Java <a href="https://docs.oracle.com/javase/9/docs/api/java/text/SimpleDateFormat.html"><code>SimpleDateFormat</code> template</a>,
+and an optional time zone.</p>
+<p>If the time zone is not present the time zone defaults to GMT time unless
+it’s included in the date string itself.</p>
+<p>Below is an example of the <code>dateTime</code> function applied to the date format
+in the example above.</p>
+<div class="listingblock"><pre class="rouge highlight"><code class="language-text" data-lang="text"><pre class="highlight"><code>select(parseCSV(cat("yr2017.csv", maxLines="2")),
+ id,
+ dateTime(Created.Date, "MM/dd/yyyy hh:mm:ss a", "EST") as cdate_dt)</code></pre></code></pre></div>
+<p>When this expression is sent to the <code>/stream</code> handler it responds with:</p>
+<div class="listingblock"><pre class="rouge highlight"><code class="language-json" data-lang="json"><pre class="highlight"><code><span style="background-color: #f8f8f8">{</span><span style="color: #bbbbbb">
+ </span><span style="color: #000080">"result-set"</span><span style="background-color: #f8f8f8">:</span><span style="color: #bbbbbb"> </span><span style="background-color: #f8f8f8">{</span><span style="color: #bbbbbb">
+ </span><span style="color: #000080">"docs"</span><span style="background-color: #f8f8f8">:</span><span style="color: #bbbbbb"> </span><span style="background-color: #f8f8f8">[</span><span style="color: #bbbbbb">
+ </span><span style="background-color: #f8f8f8">{</span><span style="color: #bbbbbb">
+ </span><span style="color: #000080">"cdate_dt"</span><span style="background-color: #f8f8f8">:</span><span style="color: #bbbbbb"> </span><span style="color: #d14">"2017-01-01T05:00:00Z"</span><span style="background-color: #f8f8f8">,</span><span style="color: #bbbbbb">
+ </span><span style="color: #000080">"id"</span><span style="background-color: #f8f8f8">:</span><span style="color: #bbbbbb"> </span><span style="color: #d14">"yr2017.csv_2"</span><span style="color: #bbbbbb">
+ </span><span style="background-color: #f8f8f8">},</span><span style="color: #bbbbbb">
+ </span><span style="background-color: #f8f8f8">{</span><span style="color: #bbbbbb">
+ </span><span style="color: #000080">"EOF"</span><span style="background-color: #f8f8f8">:</span><span style="color: #bbbbbb"> </span><span style="color: #000000;font-weight: bold">true</span><span style="background-color: #f8f8f8">,</span><span style="color: #bbbbbb">
+ </span><span style="color: #000080">"RESPONSE_TIME"</span><span style="background-color: #f8f8f8">:</span><span style="color: #bbbbbb"> </span><span style="color: #009999">1</span><span style="color: #bbbbbb">
+ </span><span style="background-color: #f8f8f8">}</span><span style="color: #bbbbbb">
+ </span><span style="background-color: #f8f8f8">]</span><span style="color: #bbbbbb">
+ </span><span style="background-color: #f8f8f8">}</span><span style="color: #bbbbbb">
+</span><span style="background-color: #f8f8f8">}</span></code></pre></code></pre></div></section>
+<section class="sect2"><h3 id="string-manipulation">String Manipulation</h3><p>The <code>upper</code>, <code>lower</code>, <code>split</code>, <code>valueAt</code>, <code>trim</code>, and <code>concat</code> functions can be used to manipulate
+strings inside of the <code>select</code> function.</p>
+<p>The example below shows the <code>upper</code> function used to upper case the <strong>species</strong>
+field.</p>
+<div class="imageblock"><img src="images/math-expressions/selectupper.png" alt="selectupper"></div>
+<p>The example below shows the <code>split</code> function which splits a field on
+a delimiter. This can be used to create multi-value fields from fields
+with an internal delimiter.</p>
+<p>The example below demonstrates this with a direct call to
+the <code>/stream</code> handler:</p>
+<div class="listingblock"><pre class="rouge highlight"><code class="language-text" data-lang="text"><pre class="highlight"><code>select(parseCSV(cat("iris.csv")),
+ id,
+ split(id, "_") as parts_ss,
+ species as species_s,
+ sepal_length as sepal_length_d,
+ sepal_width as sepal_width_d,
+ petal_length as petal_length_d,
+ petal_width as petal_width_d)</code></pre></code></pre></div>
+<p>When this expression is sent to the <code>/stream</code> handler it responds with:</p>
+<div class="listingblock"><pre class="rouge highlight"><code class="language-json" data-lang="json"><pre class="highlight"><code><span style="background-color: #f8f8f8">{</span><span style="color: #bbbbbb">
+ </span><span style="color: #000080">"result-set"</span><span style="background-color: #f8f8f8">:</span><span style="color: #bbbbbb"> </span><span style="background-color: #f8f8f8">{</span><span style="color: #bbbbbb">
+ </span><span style="color: #000080">"docs"</span><span style="background-color: #f8f8f8">:</span><span style="color: #bbbbbb"> </span><span style="background-color: #f8f8f8">[</span><span style="color: #bbbbbb">
+ </span><span style="background-color: #f8f8f8">{</span><span style="color: #bbbbbb">
+ </span><span style="color: #000080">"petal_width_d"</span><span style="background-color: #f8f8f8">:</span><span style="color: #bbbbbb"> </span><span style="color: #d14">"0.2"</span><span style="background-color: #f8f8f8">,</span><span style="color: #bbbbbb">
+ </span><span style="color: #000080">"sepal_width_d"</span><span style="background-color: #f8f8f8">:</span><span style="color: #bbbbbb"> </span><span style="color: #d14">"3.5"</span><span style="background-color: #f8f8f8">,</span><span style="color: #bbbbbb">
+ </span><span style="color: #000080">"id"</span><span style="background-color: #f8f8f8">:</span><span style="color: #bbbbbb"> </span><span style="color: #d14">"iris.csv_2"</span><span style="background-color: #f8f8f8">,</span><span style="color: #bbbbbb">
+ </span><span style="color: #000080">"petal_length_d"</span><span style="background-color: #f8f8f8">:</span><span style="color: #bbbbbb"> </span><span style="color: #d14">"1.4"</span><span style="background-color: #f8f8f8">,</span><span style="color: #bbbbbb">
+ </span><span style="color: #000080">"species_s"</span><span style="background-color: #f8f8f8">:</span><span style="color: #bbbbbb"> </span><span style="color: #d14">"setosa"</span><span style="background-color: #f8f8f8">,</span><span style="color: #bbbbbb">
+ </span><span style="color: #000080">"sepal_length_d"</span><span style="background-color: #f8f8f8">:</span><span style="color: #bbbbbb"> </span><span style="color: #d14">"5.1"</span><span style="background-color: #f8f8f8">,</span><span style="color: #bbbbbb">
+ </span><span style="color: #000080">"parts_ss"</span><span style="background-color: #f8f8f8">:</span><span style="color: #bbbbbb"> </span><span style="background-color: #f8f8f8">[</span><span style="color: #bbbbbb">
+ </span><span style="color: #d14">"iris.csv"</span><span style="background-color: #f8f8f8">,</span><span style="color: #bbbbbb">
+ </span><span style="color: #d14">"2"</span><span style="color: #bbbbbb">
+ </span><span style="background-color: #f8f8f8">]</span><span style="color: #bbbbbb">
+ </span><span style="background-color: #f8f8f8">},</span><span style="color: #bbbbbb">
+ </span><span style="background-color: #f8f8f8">{</span><span style="color: #bbbbbb">
+ </span><span style="color: #000080">"petal_width_d"</span><span style="background-color: #f8f8f8">:</span><span style="color: #bbbbbb"> </span><span style="color: #d14">"0.2"</span><span style="background-color: #f8f8f8">,</span><span style="color: #bbbbbb">
+ </span><span style="color: #000080">"sepal_width_d"</span><span style="background-color: #f8f8f8">:</span><span style="color: #bbbbbb"> </span><span style="color: #d14">"3"</span><span style="background-color: #f8f8f8">,</span><span style="color: #bbbbbb">
+ </span><span style="color: #000080">"id"</span><span style="background-color: #f8f8f8">:</span><span style="color: #bbbbbb"> </span><span style="color: #d14">"iris.csv_3"</span><span style="background-color: #f8f8f8">,</span><span style="color: #bbbbbb">
+ </span><span style="color: #000080">"petal_length_d"</span><span style="background-color: #f8f8f8">:</span><span style="color: #bbbbbb"> </span><span style="color: #d14">"1.4"</span><span style="background-color: #f8f8f8">,</span><span style="color: #bbbbbb">
+ </span><span style="color: #000080">"species_s"</span><span style="background-color: #f8f8f8">:</span><span style="color: #bbbbbb"> </span><span style="color: #d14">"setosa"</span><span style="background-color: #f8f8f8">,</span><span style="color: #bbbbbb">
+ </span><span style="color: #000080">"sepal_length_d"</span><span style="background-color: #f8f8f8">:</span><span style="color: #bbbbbb"> </span><span style="color: #d14">"4.9"</span><span style="background-color: #f8f8f8">,</span><span style="color: #bbbbbb">
+ </span><span style="color: #000080">"parts_ss"</span><span style="background-color: #f8f8f8">:</span><span style="color: #bbbbbb"> </span><span style="background-color: #f8f8f8">[</span><span style="color: #bbbbbb">
+ </span><span style="color: #d14">"iris.csv"</span><span style="background-color: #f8f8f8">,</span><span style="color: #bbbbbb">
+ </span><span style="color: #d14">"3"</span><span style="color: #bbbbbb">
+ </span><span style="background-color: #f8f8f8">]</span><span style="color: #bbbbbb">
+ </span><span style="background-color: #f8f8f8">}]}}</span></code></pre></code></pre></div>
+<p>The <code>valueAt</code> function can be used to select a specific index from
+a split array.</p>
+<div class="imageblock"><img src="images/math-expressions/valueat.png" alt="valueat"></div></section>
+<section class="sect2"><h3 id="filtering-results">Filtering Results</h3><p>The <code>having</code> function can be used to filter records.
+Filtering can be used to systematically explore specific record sets before indexing or to filter records that are sent for indexing.
+The <code>having</code> function wraps another stream and applies a boolean function to each tuple.
+If the boolean logic function returns true the tuple is returned.</p>
+<p>The following boolean functions are supported: <code>eq</code>, <code>gt</code>, <code>gteq</code>, <code>lt</code>, <code>lteq</code>, <code>matches</code>, <code>and</code>, <code>or</code>,
+<code>not</code>, <code>notNull</code>, <code>isNull</code>.</p>
+<p>Below are some strategies for using the <code>having</code> function to filter records.</p>
+<section class="sect3"><h4 id="finding-a-specific-id-or-record-number">Finding a Specific Id or Record Number</h4><p>The <code>eq</code> (equals) function can be used with the <code>having</code> expression to filter the result set
+to a single record number:</p>
+<div class="imageblock"><img src="images/math-expressions/havingId.png" alt="havingId"></div></section>
+<section class="sect3"><h4 id="skipping">Skipping</h4><p>The <code>gt</code> (greater than) function can be used on the <code>recNum</code> field to filter the result set to
+records with a recNum greater then a specific value:</p>
+<div class="imageblock"><img src="images/math-expressions/skipping.png" alt="skipping"></div></section>
+<section class="sect3"><h4 id="paging">Paging</h4><p>The <code>and</code> function with nested <code>lt</code> and <code>gt</code> functions can be used to select records within a specific
+record number range:</p>
+<div class="imageblock"><img src="images/math-expressions/paging.png" alt="paging"></div></section>
+<section class="sect3"><h4 id="striding">Striding</h4><p>The <code>eq</code> and nested <code>mod</code> function can be used to stride through the data at specific
+record number intervals. This allows for a sample to be taken at different intervals in the data
+in a systematic way.</p>
+<div class="imageblock"><img src="images/math-expressions/striding.png" alt="striding"></div></section>
+<section class="sect3"><h4 id="regex-matching">Regex Matching</h4><p>The <code>matches</code> function can be used to test if a field in the record matches a specific
+regular expression. This provides a powerful <strong>grep</strong> like capability over the record set.</p>
+<div class="imageblock"><img src="images/math-expressions/matches.png" alt="matches"></div></section></section>
+<section class="sect2"><h3 id="handling-nulls">Handling Nulls</h3><p>In most cases nulls do not need to be handled directly unless there is specific logic needed
+to handle nulls during the load.</p>
+<p>The <code>select</code> function does not output fields that contain a null value.
+This means as nulls are encountered in the data the fields are not included in the tuples.</p>
+<p>The string manipulation functions all return null if they encounter a null.
+This means the null will be passed through to the <code>select</code> function and the fields with nulls will simply be left off the record.</p>
+<p>In certain scenarios it can be important to directly filter or replace nulls.
+The sections below cover these scenarios.</p>
+<section class="sect3"><h4 id="filtering-nulls">Filtering Nulls</h4><p>The <code>having</code> and <code>isNull</code>, <code>notNull</code> functions can be combined to filter records that can contain null
+values.</p>
+<p>In the example below the <code>having</code> function returns zero documents because the <code>notNull</code> function is applied to
+ <strong>field_a</strong> in each tuple.</p>
+<div class="imageblock"><img src="images/math-expressions/havingNotNull.png" alt="havingNotNull"></div>
+<p>In the example below the <code>having</code> function returns all documents because the <code>isNull</code> function is applied to
+<strong>field_a</strong> in each tuple.</p>
+<div class="imageblock"><img src="images/math-expressions/havingIsNull.png" alt="havingIsNull"></div></section>
+<section class="sect3"><h4 id="replacing-nulls">Replacing Nulls</h4><p>The <code>if</code> function and <code>isNull</code>, <code>notNull</code> functions can be combined to replace null values inside a <code>select</code> function.</p>
+<p>In the example below the <code>if</code> function applies the <code>isNull</code> boolean expression to two different fields.</p>
+<p>In the first example it replaces null <strong>petal_width</strong> values with 0, and returns the <strong>petal_width</strong> if present.
+In the second example it replaces null <strong>field1</strong> values with the string literal "NA" and returns <strong>field1</strong> if present.</p>
+<div class="imageblock"><img src="images/math-expressions/ifIsNull.png" alt="ifIsNull"></div></section></section>
+<section class="sect2"><h3 id="text-analysis">Text Analysis</h3><p>The <code>analyze</code> function can be used from inside a <code>select</code> function to analyze
+a text field with a Lucene/Solr analyzer.
+The output of <code>analyze</code> is a list of analyzed tokens which can be added to each tuple as a multi-valued field.</p>
+<p>The multi-valued field can then be sent to Solr for indexing or the <code>cartesianProduct</code>
+function can be used to expand the list of tokens to a stream of tuples.</p>
+<p>There are a number of interesting use cases for the <code>analyze</code> function:</p>
+<div class="ulist"><ul><li><p>Previewing the output of different analyzers before indexing.</p></li><li><p>Annotating documents with NLP generated tokens (entity extraction, noun phrases etc…​)
+before the documents reach the indexing pipeline.
+This removes heavy NLP processing from the servers that may also be handling queries. It also allows
+more compute resources to be applied to the NLP indexing then is available on the search cluster.</p></li><li><p>Using the <code>cartesianProduct</code> function the analyzed tokens can be indexed as individual documents which allows
+analyzed tokens to be searched and analyzed with Solr’s aggregation and graph expressions.</p></li><li><p>Also using <code>cartesianProduct</code> the analyzed tokens can be aggregated, analyzed and visualized using
+streaming expressions directly before indexing occurs.</p></li></ul></div>
+<p>Below is an example of the <code>analyze</code> function being applied to the <strong>Resolution.Description</strong>
+field in the tuples. The <strong>_text_</strong> fields analyzer is used to analyze the text and the
+analyzed tokens are added to the documents in the <strong>token_ss</strong> field.</p>
+<div class="listingblock"><pre class="rouge highlight"><code class="language-text" data-lang="text"><pre class="highlight"><code>select(parseCSV(cat("yr2017.csv", maxLines="2")),
+ Resolution.Description,
+ analyze(Resolution.Description, _text_) as tokens_ss)</code></pre></code></pre></div>
+<p>When this expression is sent to the <code>/stream</code> handler it responds with:</p>
[... 119 lines stripped ...]