You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@metamodel.apache.org by ashish-m-yh <gi...@git.apache.org> on 2015/04/17 21:04:28 UTC
[GitHub] metamodel pull request: Solr connector added to existing DataConte...
GitHub user ashish-m-yh opened a pull request:
https://github.com/apache/metamodel/pull/16
Solr connector added to existing DataContext classes
Have finally checked in the Solr DataContext code.
The following features will be worked on in future iterations -
1. Pivot search
2. More operations such as prefix search etc.
3. Implementation of aggregations through Solr Stat functions
4. Wider set of test cases
Any other suggestions welcome too.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/ashish-m-yh/metamodel master
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/metamodel/pull/16.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #16
----
commit 752aaea9f05148ab76f942f675319a5fe05bb5d7
Author: Ashish Mukherjee <as...@gmail.com>
Date: 2015-04-17T18:55:10Z
Solr connector added
commit 77cd4f2ab5eb1452bcc52d08f5768681c8b021f8
Author: Ashish Mukherjee <as...@gmail.com>
Date: 2015-04-17T18:55:50Z
Additions for Solr connector
commit a25b5fc7352fd0829dfd45814f3803bfbc418d22
Author: Ashish Mukherjee <as...@gmail.com>
Date: 2015-04-17T18:56:06Z
Dependency added for Solr connector
commit 85fe92faf6b95ac9a94566939f63afb4ddcd50fc
Author: Ashish Mukherjee <as...@gmail.com>
Date: 2015-04-17T18:56:46Z
Lines added for Solr connector
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] metamodel pull request: Solr connector added to existing DataConte...
Posted by ashish-m-yh <gi...@git.apache.org>.
Github user ashish-m-yh commented on a diff in the pull request:
https://github.com/apache/metamodel/pull/16#discussion_r29111608
--- Diff: solr/src/main/java/org/apache/metamodel/solr/SolrDataContext.java ---
@@ -0,0 +1,429 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.metamodel.solr;
+
+import java.io.InputStreamReader;
+import java.io.IOException;
+import java.lang.reflect.Method;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Comparator;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+
+import org.apache.http.HttpResponse;
+import org.apache.http.client.HttpClient;
+import org.apache.http.client.methods.HttpGet;
+import org.apache.http.impl.client.DefaultHttpClient;
+import org.apache.metamodel.DataContext;
+import org.apache.metamodel.MetaModelException;
+import org.apache.metamodel.query.Query;
+import org.apache.metamodel.QueryPostprocessDataContext;
+import org.apache.metamodel.data.DataSet;
+import org.apache.metamodel.data.DataSetHeader;
+import org.apache.metamodel.data.Row;
+import org.apache.metamodel.data.SimpleDataSetHeader;
+import org.apache.metamodel.query.FilterClause;
+import org.apache.metamodel.query.FilterItem;
+import org.apache.metamodel.query.GroupByItem;
+import org.apache.metamodel.query.OrderByItem;
+import org.apache.metamodel.query.SelectItem;
+import org.apache.metamodel.query.LogicalOperator;
+import org.apache.metamodel.query.OperatorType;
+import org.apache.metamodel.query.FunctionType;
+import org.apache.metamodel.schema.Column;
+import org.apache.metamodel.schema.ColumnType;
+import org.apache.metamodel.schema.MutableColumn;
+import org.apache.metamodel.schema.MutableSchema;
+import org.apache.metamodel.schema.MutableTable;
+import org.apache.metamodel.schema.Schema;
+import org.apache.metamodel.schema.Table;
+import org.apache.metamodel.util.SimpleTableDef;
+import org.apache.solr.client.solrj.SolrQuery;
+import org.apache.solr.client.solrj.SolrQuery.ORDER;
+import org.apache.solr.client.solrj.SolrServerException;
+import org.apache.solr.client.solrj.impl.HttpSolrServer;
+import org.apache.solr.client.solrj.response.QueryResponse;
+import org.apache.solr.client.solrj.response.FacetField;
+import org.apache.solr.common.SolrDocumentList;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.fasterxml.jackson.databind.ObjectReader;
+
+/**
+ * DataContext implementation for ElasticSearch analytics engine.
+ *
+ * ElasticSearch has a data storage structure hierarchy that briefly goes like
+ * this:
+ * <ul>
+ * <li>Index</li>
+ * <li>Document type (short: Type) (within an index)</li>
+ * <li>Documents (of a particular type)</li>
+ * </ul>
+ *
+ * When instantiating this DataContext, an index name is provided. Within this
+ * index, each document type is represented as a table.
+ *
+ * This implementation supports either automatic discovery of a schema or manual
+ * specification of a schema, through the {@link SimpleTableDef} class.
+ */
+
+public class SolrDataContext extends QueryPostprocessDataContext implements
+ DataContext {
+
+ private static final Logger logger = LoggerFactory
+ .getLogger(SolrDataContext.class);
+
+ public static final String FIELD_ID = "_id";
+
+ private final SimpleTableDef tableDef;
+ private final String indexName;
+ private final String url;
+
+ private SolrQuery query = new SolrQuery();
+
+ private static enum queryTypes {
+ ROW, FACET
+ };
+
+ private static final int MAX_RETRIES = 0;
+ private static final int SOCK_TIMEOUT = 1000;
+ private static final int CONN_TIMEOUT = 5000;
+ private static final int DEFAULT_LIMIT = 1000;
+
+ public SolrDataContext(String url, String indexName) {
+ this.url = url;
+ this.indexName = indexName;
+ this.tableDef = detectSchema(url, indexName);
+ }
+
+ private SimpleTableDef detectSchema(String url, String indexName) {
+ SimpleTableDef tableDef = null;
+
+ url += "/schema";
+
+ try {
+ HttpClient httpclient = new DefaultHttpClient();
+ HttpGet request = new HttpGet(url);
+ HttpResponse response = httpclient.execute(request);
+
+ InputStreamReader inputStream = new InputStreamReader(response
+ .getEntity().getContent());
+
+ ObjectReader reader = new ObjectMapper().reader(HashMap.class);
--- End diff --
There is no documented API for this. I have put in place necessary exception handling.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] metamodel pull request: Solr connector added to existing DataConte...
Posted by kaspersorensen <gi...@git.apache.org>.
Github user kaspersorensen commented on a diff in the pull request:
https://github.com/apache/metamodel/pull/16#discussion_r28686985
--- Diff: solr/src/test/java/org/apache/metamodel/solr/SolrDataContextTest1.java ---
@@ -0,0 +1,199 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.metamodel.solr;
+
+import static org.junit.Assert.*;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Date;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+
+import javax.swing.table.TableModel;
+
+import org.apache.metamodel.DataContext;
+import org.apache.metamodel.data.DataSet;
+import org.apache.metamodel.data.DataSetTableModel;
+import org.apache.metamodel.data.FilteredDataSet;
+import org.apache.metamodel.data.InMemoryDataSet;
+import org.apache.metamodel.data.Row;
+import org.apache.metamodel.query.FunctionType;
+import org.apache.metamodel.query.Query;
+import org.apache.metamodel.query.SelectItem;
+import org.apache.metamodel.schema.Column;
+import org.apache.metamodel.schema.ColumnType;
+import org.apache.metamodel.schema.Table;
+import org.junit.AfterClass;
+import org.junit.BeforeClass;
+import org.junit.Ignore;
+import org.junit.Test;
+
+public class SolrDataContextTest1 {
+ private static final String url = "http://localhost:8983/solr/collection1";
+ private static final String index = "collection1";
+
+ private static DataContext dataContext;
+
+ @BeforeClass
+ public static void beforeTests() throws Exception {
+ DataContext dataContext = new SolrDataContext(url, index);
+
+ }
+
+ @AfterClass
+ public static void afterTests() {
+ System.out.println("Solr server shut down!");
+ }
+
+ @Test
+ public void testWhereWithLimit() throws Exception {
+ DataSet dataSet = dataContext
+ .executeQuery("SELECT * FROM collection1 WHERE (manu='maxtor' or manu='samsung') LIMIT 1");
+ try {
+ assertTrue(dataSet.next());
+
+ Row row = dataSet.getRow();
+ Object val = row.getValue(1);
+
+ assertEquals("1491367446686203904", val.toString());
--- End diff --
Seems the data of the index is also something that only you have. Please make tests that either describe where to get data or make the tests themselves build up a test-database in the setUp() method or so.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] metamodel pull request: Solr connector added to existing DataConte...
Posted by kaspersorensen <gi...@git.apache.org>.
Github user kaspersorensen commented on a diff in the pull request:
https://github.com/apache/metamodel/pull/16#discussion_r28686228
--- Diff: solr/src/main/java/org/apache/metamodel/solr/SolrDataContext.java ---
@@ -0,0 +1,429 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.metamodel.solr;
+
+import java.io.InputStreamReader;
+import java.io.IOException;
+import java.lang.reflect.Method;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Comparator;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+
+import org.apache.http.HttpResponse;
+import org.apache.http.client.HttpClient;
+import org.apache.http.client.methods.HttpGet;
+import org.apache.http.impl.client.DefaultHttpClient;
+import org.apache.metamodel.DataContext;
+import org.apache.metamodel.MetaModelException;
+import org.apache.metamodel.query.Query;
+import org.apache.metamodel.QueryPostprocessDataContext;
+import org.apache.metamodel.data.DataSet;
+import org.apache.metamodel.data.DataSetHeader;
+import org.apache.metamodel.data.Row;
+import org.apache.metamodel.data.SimpleDataSetHeader;
+import org.apache.metamodel.query.FilterClause;
+import org.apache.metamodel.query.FilterItem;
+import org.apache.metamodel.query.GroupByItem;
+import org.apache.metamodel.query.OrderByItem;
+import org.apache.metamodel.query.SelectItem;
+import org.apache.metamodel.query.LogicalOperator;
+import org.apache.metamodel.query.OperatorType;
+import org.apache.metamodel.query.FunctionType;
+import org.apache.metamodel.schema.Column;
+import org.apache.metamodel.schema.ColumnType;
+import org.apache.metamodel.schema.MutableColumn;
+import org.apache.metamodel.schema.MutableSchema;
+import org.apache.metamodel.schema.MutableTable;
+import org.apache.metamodel.schema.Schema;
+import org.apache.metamodel.schema.Table;
+import org.apache.metamodel.util.SimpleTableDef;
+import org.apache.solr.client.solrj.SolrQuery;
+import org.apache.solr.client.solrj.SolrQuery.ORDER;
+import org.apache.solr.client.solrj.SolrServerException;
+import org.apache.solr.client.solrj.impl.HttpSolrServer;
+import org.apache.solr.client.solrj.response.QueryResponse;
+import org.apache.solr.client.solrj.response.FacetField;
+import org.apache.solr.common.SolrDocumentList;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.fasterxml.jackson.databind.ObjectReader;
+
+/**
+ * DataContext implementation for ElasticSearch analytics engine.
+ *
+ * ElasticSearch has a data storage structure hierarchy that briefly goes like
+ * this:
+ * <ul>
+ * <li>Index</li>
+ * <li>Document type (short: Type) (within an index)</li>
+ * <li>Documents (of a particular type)</li>
+ * </ul>
+ *
+ * When instantiating this DataContext, an index name is provided. Within this
+ * index, each document type is represented as a table.
+ *
+ * This implementation supports either automatic discovery of a schema or manual
+ * specification of a schema, through the {@link SimpleTableDef} class.
+ */
+
+public class SolrDataContext extends QueryPostprocessDataContext implements
+ DataContext {
+
+ private static final Logger logger = LoggerFactory
+ .getLogger(SolrDataContext.class);
+
+ public static final String FIELD_ID = "_id";
+
+ private final SimpleTableDef tableDef;
+ private final String indexName;
+ private final String url;
+
+ private SolrQuery query = new SolrQuery();
+
+ private static enum queryTypes {
+ ROW, FACET
+ };
+
+ private static final int MAX_RETRIES = 0;
+ private static final int SOCK_TIMEOUT = 1000;
+ private static final int CONN_TIMEOUT = 5000;
+ private static final int DEFAULT_LIMIT = 1000;
+
+ public SolrDataContext(String url, String indexName) {
+ this.url = url;
+ this.indexName = indexName;
+ this.tableDef = detectSchema(url, indexName);
+ }
+
+ private SimpleTableDef detectSchema(String url, String indexName) {
+ SimpleTableDef tableDef = null;
+
+ url += "/schema";
+
+ try {
+ HttpClient httpclient = new DefaultHttpClient();
+ HttpGet request = new HttpGet(url);
+ HttpResponse response = httpclient.execute(request);
+
+ InputStreamReader inputStream = new InputStreamReader(response
+ .getEntity().getContent());
+
+ ObjectReader reader = new ObjectMapper().reader(HashMap.class);
+ Map<String, Object> outerMap = reader.readValue(inputStream);
+
+ Map<String, Object> schemaMap = (Map) outerMap.get("schema");
+
+ if (outerMap != null) {
--- End diff --
I think this should be ```schemaMap != null```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] metamodel pull request: Solr connector added to existing DataConte...
Posted by kaspersorensen <gi...@git.apache.org>.
Github user kaspersorensen commented on a diff in the pull request:
https://github.com/apache/metamodel/pull/16#discussion_r28686010
--- Diff: solr/.project ---
@@ -0,0 +1,20 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<projectDescription>
--- End diff --
Ditto
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] metamodel pull request: Solr connector added to existing DataConte...
Posted by kaspersorensen <gi...@git.apache.org>.
Github user kaspersorensen commented on a diff in the pull request:
https://github.com/apache/metamodel/pull/16#discussion_r28686925
--- Diff: solr/src/test/java/org/apache/metamodel/solr/SolrDataContextTest1.java ---
@@ -0,0 +1,199 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.metamodel.solr;
+
+import static org.junit.Assert.*;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Date;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+
+import javax.swing.table.TableModel;
+
+import org.apache.metamodel.DataContext;
+import org.apache.metamodel.data.DataSet;
+import org.apache.metamodel.data.DataSetTableModel;
+import org.apache.metamodel.data.FilteredDataSet;
+import org.apache.metamodel.data.InMemoryDataSet;
+import org.apache.metamodel.data.Row;
+import org.apache.metamodel.query.FunctionType;
+import org.apache.metamodel.query.Query;
+import org.apache.metamodel.query.SelectItem;
+import org.apache.metamodel.schema.Column;
+import org.apache.metamodel.schema.ColumnType;
+import org.apache.metamodel.schema.Table;
+import org.junit.AfterClass;
+import org.junit.BeforeClass;
+import org.junit.Ignore;
+import org.junit.Test;
+
+public class SolrDataContextTest1 {
+ private static final String url = "http://localhost:8983/solr/collection1";
--- End diff --
Please follow the standard way in MM towards integration tests - provide some optional properties that can be supplied by the user. See all the other modules for inspiration. With a test like this the tests will break anywhere else but on your machine I guess.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] metamodel pull request: Solr connector added to existing DataConte...
Posted by kaspersorensen <gi...@git.apache.org>.
Github user kaspersorensen commented on a diff in the pull request:
https://github.com/apache/metamodel/pull/16#discussion_r28685955
--- Diff: full/src/main/java/org/apache/metamodel/DataContextFactory.java ---
@@ -652,6 +653,18 @@ public static UpdateableDataContext createElasticSearchDataContext(Client client
return new ElasticSearchDataContext(client, indexName);
}
+ /**
+ * Creates a new Solr datacontext
--- End diff --
Also indentation strangeness.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] metamodel pull request: Solr connector added to existing DataConte...
Posted by kaspersorensen <gi...@git.apache.org>.
Github user kaspersorensen commented on a diff in the pull request:
https://github.com/apache/metamodel/pull/16#discussion_r28686002
--- Diff: solr/.classpath ---
@@ -0,0 +1,24 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<classpath>
--- End diff --
We would like to keep the repo clean of project files from any particular IDE. Please create a .gitignore file instead of committing this file and the next ones.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] metamodel pull request: Solr connector added to existing DataConte...
Posted by ashish-m-yh <gi...@git.apache.org>.
Github user ashish-m-yh commented on a diff in the pull request:
https://github.com/apache/metamodel/pull/16#discussion_r29111641
--- Diff: solr/src/main/java/org/apache/metamodel/solr/SolrDataContext.java ---
@@ -0,0 +1,429 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.metamodel.solr;
+
+import java.io.InputStreamReader;
+import java.io.IOException;
+import java.lang.reflect.Method;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Comparator;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+
+import org.apache.http.HttpResponse;
+import org.apache.http.client.HttpClient;
+import org.apache.http.client.methods.HttpGet;
+import org.apache.http.impl.client.DefaultHttpClient;
+import org.apache.metamodel.DataContext;
+import org.apache.metamodel.MetaModelException;
+import org.apache.metamodel.query.Query;
+import org.apache.metamodel.QueryPostprocessDataContext;
+import org.apache.metamodel.data.DataSet;
+import org.apache.metamodel.data.DataSetHeader;
+import org.apache.metamodel.data.Row;
+import org.apache.metamodel.data.SimpleDataSetHeader;
+import org.apache.metamodel.query.FilterClause;
+import org.apache.metamodel.query.FilterItem;
+import org.apache.metamodel.query.GroupByItem;
+import org.apache.metamodel.query.OrderByItem;
+import org.apache.metamodel.query.SelectItem;
+import org.apache.metamodel.query.LogicalOperator;
+import org.apache.metamodel.query.OperatorType;
+import org.apache.metamodel.query.FunctionType;
+import org.apache.metamodel.schema.Column;
+import org.apache.metamodel.schema.ColumnType;
+import org.apache.metamodel.schema.MutableColumn;
+import org.apache.metamodel.schema.MutableSchema;
+import org.apache.metamodel.schema.MutableTable;
+import org.apache.metamodel.schema.Schema;
+import org.apache.metamodel.schema.Table;
+import org.apache.metamodel.util.SimpleTableDef;
+import org.apache.solr.client.solrj.SolrQuery;
+import org.apache.solr.client.solrj.SolrQuery.ORDER;
+import org.apache.solr.client.solrj.SolrServerException;
+import org.apache.solr.client.solrj.impl.HttpSolrServer;
+import org.apache.solr.client.solrj.response.QueryResponse;
+import org.apache.solr.client.solrj.response.FacetField;
+import org.apache.solr.common.SolrDocumentList;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.fasterxml.jackson.databind.ObjectReader;
+
+/**
+ * DataContext implementation for ElasticSearch analytics engine.
+ *
+ * ElasticSearch has a data storage structure hierarchy that briefly goes like
+ * this:
+ * <ul>
+ * <li>Index</li>
+ * <li>Document type (short: Type) (within an index)</li>
+ * <li>Documents (of a particular type)</li>
+ * </ul>
+ *
+ * When instantiating this DataContext, an index name is provided. Within this
+ * index, each document type is represented as a table.
+ *
+ * This implementation supports either automatic discovery of a schema or manual
+ * specification of a schema, through the {@link SimpleTableDef} class.
+ */
+
+public class SolrDataContext extends QueryPostprocessDataContext implements
+ DataContext {
+
+ private static final Logger logger = LoggerFactory
+ .getLogger(SolrDataContext.class);
+
+ public static final String FIELD_ID = "_id";
+
+ private final SimpleTableDef tableDef;
+ private final String indexName;
+ private final String url;
+
+ private SolrQuery query = new SolrQuery();
+
+ private static enum queryTypes {
+ ROW, FACET
+ };
+
+ private static final int MAX_RETRIES = 0;
+ private static final int SOCK_TIMEOUT = 1000;
+ private static final int CONN_TIMEOUT = 5000;
+ private static final int DEFAULT_LIMIT = 1000;
+
+ public SolrDataContext(String url, String indexName) {
+ this.url = url;
+ this.indexName = indexName;
+ this.tableDef = detectSchema(url, indexName);
+ }
+
+ private SimpleTableDef detectSchema(String url, String indexName) {
+ SimpleTableDef tableDef = null;
+
+ url += "/schema";
+
+ try {
+ HttpClient httpclient = new DefaultHttpClient();
+ HttpGet request = new HttpGet(url);
+ HttpResponse response = httpclient.execute(request);
+
+ InputStreamReader inputStream = new InputStreamReader(response
+ .getEntity().getContent());
+
+ ObjectReader reader = new ObjectMapper().reader(HashMap.class);
+ Map<String, Object> outerMap = reader.readValue(inputStream);
+
+ Map<String, Object> schemaMap = (Map) outerMap.get("schema");
+
+ if (outerMap != null) {
+ List<Map<String, String>> fieldMapList = (List) schemaMap
+ .get("fields");
+ List<String> columnNames = new ArrayList<String>();
+ List<ColumnType> columnTypes = new ArrayList<ColumnType>();
+
+ for (int i = 0; i < fieldMapList.size(); i++) {
+ Map<String, String> fieldMap = fieldMapList.get(i);
+
+ columnNames.add(fieldMap.get("name"));
+ columnTypes.add(ColumnType.VARCHAR);
+ }
+
+ String[] columnNamesArr = new String[columnNames.size()];
+ columnNamesArr = columnNames.toArray(columnNamesArr);
+
+ ColumnType[] columnTypesArr = new ColumnType[columnTypes.size()];
+ columnTypesArr = columnTypes.toArray(columnTypesArr);
+
+ tableDef = new SimpleTableDef(indexName, columnNamesArr,
+ columnTypesArr);
+ }
+ } catch (IOException e) {
+ logger.error("Failed to parse schema", "", e);
+ }
+
+ return tableDef;
+ }
+
+ @Override
+ protected Schema getMainSchema() throws MetaModelException {
+ try {
+ final MutableSchema theSchema = new MutableSchema(
+ getMainSchemaName());
+
+ final MutableTable table = tableDef.toTable().setSchema(theSchema);
+ final Column idColumn = table.getColumnByName(FIELD_ID);
--- End diff --
That can be a later enhancement.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] metamodel pull request: Solr connector added to existing DataConte...
Posted by kaspersorensen <gi...@git.apache.org>.
Github user kaspersorensen commented on a diff in the pull request:
https://github.com/apache/metamodel/pull/16#discussion_r28686018
--- Diff: solr/.settings/org.eclipse.jdt.core.prefs ---
@@ -0,0 +1,9 @@
+#Fri Jan 30 22:32:19 IST 2015
+org.eclipse.jdt.core.compiler.codegen.targetPlatform=1.6
+eclipse.preferences.version=1
+encoding/src/main/java=utf-8
--- End diff --
Ditto
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] metamodel pull request: Solr connector added to existing DataConte...
Posted by kaspersorensen <gi...@git.apache.org>.
Github user kaspersorensen commented on a diff in the pull request:
https://github.com/apache/metamodel/pull/16#discussion_r28686813
--- Diff: solr/src/main/java/org/apache/metamodel/solr/SolrDataContext.java ---
@@ -0,0 +1,429 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.metamodel.solr;
+
+import java.io.InputStreamReader;
+import java.io.IOException;
+import java.lang.reflect.Method;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Comparator;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+
+import org.apache.http.HttpResponse;
+import org.apache.http.client.HttpClient;
+import org.apache.http.client.methods.HttpGet;
+import org.apache.http.impl.client.DefaultHttpClient;
+import org.apache.metamodel.DataContext;
+import org.apache.metamodel.MetaModelException;
+import org.apache.metamodel.query.Query;
+import org.apache.metamodel.QueryPostprocessDataContext;
+import org.apache.metamodel.data.DataSet;
+import org.apache.metamodel.data.DataSetHeader;
+import org.apache.metamodel.data.Row;
+import org.apache.metamodel.data.SimpleDataSetHeader;
+import org.apache.metamodel.query.FilterClause;
+import org.apache.metamodel.query.FilterItem;
+import org.apache.metamodel.query.GroupByItem;
+import org.apache.metamodel.query.OrderByItem;
+import org.apache.metamodel.query.SelectItem;
+import org.apache.metamodel.query.LogicalOperator;
+import org.apache.metamodel.query.OperatorType;
+import org.apache.metamodel.query.FunctionType;
+import org.apache.metamodel.schema.Column;
+import org.apache.metamodel.schema.ColumnType;
+import org.apache.metamodel.schema.MutableColumn;
+import org.apache.metamodel.schema.MutableSchema;
+import org.apache.metamodel.schema.MutableTable;
+import org.apache.metamodel.schema.Schema;
+import org.apache.metamodel.schema.Table;
+import org.apache.metamodel.util.SimpleTableDef;
+import org.apache.solr.client.solrj.SolrQuery;
+import org.apache.solr.client.solrj.SolrQuery.ORDER;
+import org.apache.solr.client.solrj.SolrServerException;
+import org.apache.solr.client.solrj.impl.HttpSolrServer;
+import org.apache.solr.client.solrj.response.QueryResponse;
+import org.apache.solr.client.solrj.response.FacetField;
+import org.apache.solr.common.SolrDocumentList;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.fasterxml.jackson.databind.ObjectReader;
+
+/**
+ * DataContext implementation for ElasticSearch analytics engine.
+ *
+ * ElasticSearch has a data storage structure hierarchy that briefly goes like
+ * this:
+ * <ul>
+ * <li>Index</li>
+ * <li>Document type (short: Type) (within an index)</li>
+ * <li>Documents (of a particular type)</li>
+ * </ul>
+ *
+ * When instantiating this DataContext, an index name is provided. Within this
+ * index, each document type is represented as a table.
+ *
+ * This implementation supports either automatic discovery of a schema or manual
+ * specification of a schema, through the {@link SimpleTableDef} class.
+ */
+
+public class SolrDataContext extends QueryPostprocessDataContext implements
+ DataContext {
+
+ private static final Logger logger = LoggerFactory
+ .getLogger(SolrDataContext.class);
+
+ public static final String FIELD_ID = "_id";
+
+ private final SimpleTableDef tableDef;
+ private final String indexName;
+ private final String url;
+
+ private SolrQuery query = new SolrQuery();
+
+ private static enum queryTypes {
+ ROW, FACET
+ };
+
+ private static final int MAX_RETRIES = 0;
+ private static final int SOCK_TIMEOUT = 1000;
+ private static final int CONN_TIMEOUT = 5000;
+ private static final int DEFAULT_LIMIT = 1000;
+
+ public SolrDataContext(String url, String indexName) {
+ this.url = url;
+ this.indexName = indexName;
+ this.tableDef = detectSchema(url, indexName);
+ }
+
+ private SimpleTableDef detectSchema(String url, String indexName) {
+ SimpleTableDef tableDef = null;
+
+ url += "/schema";
+
+ try {
+ HttpClient httpclient = new DefaultHttpClient();
+ HttpGet request = new HttpGet(url);
+ HttpResponse response = httpclient.execute(request);
+
+ InputStreamReader inputStream = new InputStreamReader(response
+ .getEntity().getContent());
+
+ ObjectReader reader = new ObjectMapper().reader(HashMap.class);
+ Map<String, Object> outerMap = reader.readValue(inputStream);
+
+ Map<String, Object> schemaMap = (Map) outerMap.get("schema");
+
+ if (outerMap != null) {
+ List<Map<String, String>> fieldMapList = (List) schemaMap
+ .get("fields");
+ List<String> columnNames = new ArrayList<String>();
+ List<ColumnType> columnTypes = new ArrayList<ColumnType>();
+
+ for (int i = 0; i < fieldMapList.size(); i++) {
+ Map<String, String> fieldMap = fieldMapList.get(i);
+
+ columnNames.add(fieldMap.get("name"));
+ columnTypes.add(ColumnType.VARCHAR);
+ }
+
+ String[] columnNamesArr = new String[columnNames.size()];
+ columnNamesArr = columnNames.toArray(columnNamesArr);
+
+ ColumnType[] columnTypesArr = new ColumnType[columnTypes.size()];
+ columnTypesArr = columnTypes.toArray(columnTypesArr);
+
+ tableDef = new SimpleTableDef(indexName, columnNamesArr,
+ columnTypesArr);
+ }
+ } catch (IOException e) {
+ logger.error("Failed to parse schema", "", e);
+ }
+
+ return tableDef;
+ }
+
+ @Override
+ protected Schema getMainSchema() throws MetaModelException {
+ try {
+ final MutableSchema theSchema = new MutableSchema(
+ getMainSchemaName());
+
+ final MutableTable table = tableDef.toTable().setSchema(theSchema);
+ final Column idColumn = table.getColumnByName(FIELD_ID);
+ if (idColumn != null && idColumn instanceof MutableColumn) {
+ final MutableColumn mutableColumn = (MutableColumn) idColumn;
+ mutableColumn.setPrimaryKey(true);
+ }
+ theSchema.addTable(table);
+
+ return theSchema;
+ } catch (Exception e) {
+ throw new MetaModelException("Schema retrieval failed " + e);
+ }
+ }
+
+ @Override
+ protected Number executeCountQuery(Table table,
+ List<FilterItem> whereItems, boolean functionApproximationAllowed) {
+ String queryStr = "*:*";
+
+ if (whereItems != null) {
+ for (int i = 0; i < whereItems.size(); i++) {
+ FilterItem whereItem = whereItems.get(i);
+ queryStr = whereItem.toString();
+ }
+ }
+
+ QueryResponse response = selectRows(table, null, queryStr, 0, 0);
+ return response.getResults().getNumFound();
+ }
+
+ private String processSingleFilterItem(FilterItem filterItem) {
+ String clause = "";
+
+ if (filterItem != null) {
+ SelectItem item = filterItem.getSelectItem();
+ OperatorType operator = filterItem.getOperator();
+ Object operand = filterItem.getOperand();
+
+ if (item == null || operator == null || operand == null) {
+ clause = " " + filterItem.getExpression();
+ } else {
+ String columnName = item.getColumn().getName();
+ clause = " " + columnName + operator.toSql() + operand;
+ }
+ }
+
+ return clause;
+ }
+
+ private String createSearchClause(FilterItem filterItem, String clause) {
+ FilterItem[] filterItems = filterItem.getChildItems();
+
+ if (filterItems != null && filterItems.length > 0) {
+ for (FilterItem filterItemInner : filterItems) {
+ if (filterItemInner != null) {
+ clause += " " + processSingleFilterItem(filterItemInner);
+ }
+
+ return createSearchClause(filterItemInner, clause);
+ }
+ }
+
+ if (filterItem != null) {
+ clause += " " + processSingleFilterItem(filterItem);
+ }
+
+ return clause;
+ }
+
+ @Override
+ public DataSet executeQuery(Query q) throws MetaModelException {
--- End diff --
I would not override this method for our initial attempt at doing Solr ... Waaaay too much complexity involved in this method, so my advice is to start with just materializeMainTable and make that work well.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] metamodel pull request: Solr connector added to existing DataConte...
Posted by ashish-m-yh <gi...@git.apache.org>.
Github user ashish-m-yh commented on a diff in the pull request:
https://github.com/apache/metamodel/pull/16#discussion_r29111668
--- Diff: solr/src/main/java/org/apache/metamodel/solr/SolrDataContext.java ---
@@ -0,0 +1,429 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.metamodel.solr;
+
+import java.io.InputStreamReader;
+import java.io.IOException;
+import java.lang.reflect.Method;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Comparator;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+
+import org.apache.http.HttpResponse;
+import org.apache.http.client.HttpClient;
+import org.apache.http.client.methods.HttpGet;
+import org.apache.http.impl.client.DefaultHttpClient;
+import org.apache.metamodel.DataContext;
+import org.apache.metamodel.MetaModelException;
+import org.apache.metamodel.query.Query;
+import org.apache.metamodel.QueryPostprocessDataContext;
+import org.apache.metamodel.data.DataSet;
+import org.apache.metamodel.data.DataSetHeader;
+import org.apache.metamodel.data.Row;
+import org.apache.metamodel.data.SimpleDataSetHeader;
+import org.apache.metamodel.query.FilterClause;
+import org.apache.metamodel.query.FilterItem;
+import org.apache.metamodel.query.GroupByItem;
+import org.apache.metamodel.query.OrderByItem;
+import org.apache.metamodel.query.SelectItem;
+import org.apache.metamodel.query.LogicalOperator;
+import org.apache.metamodel.query.OperatorType;
+import org.apache.metamodel.query.FunctionType;
+import org.apache.metamodel.schema.Column;
+import org.apache.metamodel.schema.ColumnType;
+import org.apache.metamodel.schema.MutableColumn;
+import org.apache.metamodel.schema.MutableSchema;
+import org.apache.metamodel.schema.MutableTable;
+import org.apache.metamodel.schema.Schema;
+import org.apache.metamodel.schema.Table;
+import org.apache.metamodel.util.SimpleTableDef;
+import org.apache.solr.client.solrj.SolrQuery;
+import org.apache.solr.client.solrj.SolrQuery.ORDER;
+import org.apache.solr.client.solrj.SolrServerException;
+import org.apache.solr.client.solrj.impl.HttpSolrServer;
+import org.apache.solr.client.solrj.response.QueryResponse;
+import org.apache.solr.client.solrj.response.FacetField;
+import org.apache.solr.common.SolrDocumentList;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.fasterxml.jackson.databind.ObjectReader;
+
+/**
+ * DataContext implementation for ElasticSearch analytics engine.
+ *
+ * ElasticSearch has a data storage structure hierarchy that briefly goes like
+ * this:
+ * <ul>
+ * <li>Index</li>
+ * <li>Document type (short: Type) (within an index)</li>
+ * <li>Documents (of a particular type)</li>
+ * </ul>
+ *
+ * When instantiating this DataContext, an index name is provided. Within this
+ * index, each document type is represented as a table.
+ *
+ * This implementation supports either automatic discovery of a schema or manual
+ * specification of a schema, through the {@link SimpleTableDef} class.
+ */
+
+public class SolrDataContext extends QueryPostprocessDataContext implements
+ DataContext {
+
+ private static final Logger logger = LoggerFactory
+ .getLogger(SolrDataContext.class);
+
+ public static final String FIELD_ID = "_id";
+
+ private final SimpleTableDef tableDef;
+ private final String indexName;
+ private final String url;
+
+ private SolrQuery query = new SolrQuery();
+
+ private static enum queryTypes {
+ ROW, FACET
+ };
+
+ private static final int MAX_RETRIES = 0;
+ private static final int SOCK_TIMEOUT = 1000;
+ private static final int CONN_TIMEOUT = 5000;
+ private static final int DEFAULT_LIMIT = 1000;
+
+ public SolrDataContext(String url, String indexName) {
+ this.url = url;
+ this.indexName = indexName;
+ this.tableDef = detectSchema(url, indexName);
+ }
+
+ private SimpleTableDef detectSchema(String url, String indexName) {
+ SimpleTableDef tableDef = null;
+
+ url += "/schema";
+
+ try {
+ HttpClient httpclient = new DefaultHttpClient();
+ HttpGet request = new HttpGet(url);
+ HttpResponse response = httpclient.execute(request);
+
+ InputStreamReader inputStream = new InputStreamReader(response
+ .getEntity().getContent());
+
+ ObjectReader reader = new ObjectMapper().reader(HashMap.class);
+ Map<String, Object> outerMap = reader.readValue(inputStream);
+
+ Map<String, Object> schemaMap = (Map) outerMap.get("schema");
+
+ if (outerMap != null) {
+ List<Map<String, String>> fieldMapList = (List) schemaMap
+ .get("fields");
+ List<String> columnNames = new ArrayList<String>();
+ List<ColumnType> columnTypes = new ArrayList<ColumnType>();
+
+ for (int i = 0; i < fieldMapList.size(); i++) {
+ Map<String, String> fieldMap = fieldMapList.get(i);
+
+ columnNames.add(fieldMap.get("name"));
+ columnTypes.add(ColumnType.VARCHAR);
+ }
+
+ String[] columnNamesArr = new String[columnNames.size()];
+ columnNamesArr = columnNames.toArray(columnNamesArr);
+
+ ColumnType[] columnTypesArr = new ColumnType[columnTypes.size()];
+ columnTypesArr = columnTypes.toArray(columnTypesArr);
+
+ tableDef = new SimpleTableDef(indexName, columnNamesArr,
+ columnTypesArr);
+ }
+ } catch (IOException e) {
+ logger.error("Failed to parse schema", "", e);
+ }
+
+ return tableDef;
+ }
+
+ @Override
+ protected Schema getMainSchema() throws MetaModelException {
+ try {
+ final MutableSchema theSchema = new MutableSchema(
+ getMainSchemaName());
+
+ final MutableTable table = tableDef.toTable().setSchema(theSchema);
+ final Column idColumn = table.getColumnByName(FIELD_ID);
+ if (idColumn != null && idColumn instanceof MutableColumn) {
+ final MutableColumn mutableColumn = (MutableColumn) idColumn;
+ mutableColumn.setPrimaryKey(true);
+ }
+ theSchema.addTable(table);
+
+ return theSchema;
+ } catch (Exception e) {
+ throw new MetaModelException("Schema retrieval failed " + e);
+ }
+ }
+
+ @Override
+ protected Number executeCountQuery(Table table,
+ List<FilterItem> whereItems, boolean functionApproximationAllowed) {
+ String queryStr = "*:*";
+
+ if (whereItems != null) {
+ for (int i = 0; i < whereItems.size(); i++) {
+ FilterItem whereItem = whereItems.get(i);
+ queryStr = whereItem.toString();
+ }
+ }
+
+ QueryResponse response = selectRows(table, null, queryStr, 0, 0);
+ return response.getResults().getNumFound();
+ }
+
+ private String processSingleFilterItem(FilterItem filterItem) {
+ String clause = "";
+
+ if (filterItem != null) {
+ SelectItem item = filterItem.getSelectItem();
+ OperatorType operator = filterItem.getOperator();
+ Object operand = filterItem.getOperand();
+
+ if (item == null || operator == null || operand == null) {
+ clause = " " + filterItem.getExpression();
+ } else {
+ String columnName = item.getColumn().getName();
+ clause = " " + columnName + operator.toSql() + operand;
+ }
+ }
+
+ return clause;
+ }
+
+ private String createSearchClause(FilterItem filterItem, String clause) {
+ FilterItem[] filterItems = filterItem.getChildItems();
+
+ if (filterItems != null && filterItems.length > 0) {
+ for (FilterItem filterItemInner : filterItems) {
+ if (filterItemInner != null) {
+ clause += " " + processSingleFilterItem(filterItemInner);
+ }
+
+ return createSearchClause(filterItemInner, clause);
+ }
+ }
+
+ if (filterItem != null) {
+ clause += " " + processSingleFilterItem(filterItem);
+ }
+
+ return clause;
+ }
+
+ @Override
+ public DataSet executeQuery(Query q) throws MetaModelException {
--- End diff --
I have simplified this logic vastly but had to put in place a safe regex to handle one scenario which looks to be a MM bug. I will report this separately. However, this works reliably, as I have tested various query scenarios.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] metamodel pull request: Solr connector added to existing DataConte...
Posted by ashish-m-yh <gi...@git.apache.org>.
Github user ashish-m-yh commented on a diff in the pull request:
https://github.com/apache/metamodel/pull/16#discussion_r29111710
--- Diff: solr/src/test/java/org/apache/metamodel/solr/SolrDataContextTest1.java ---
@@ -0,0 +1,199 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.metamodel.solr;
+
+import static org.junit.Assert.*;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Date;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+
+import javax.swing.table.TableModel;
+
+import org.apache.metamodel.DataContext;
+import org.apache.metamodel.data.DataSet;
+import org.apache.metamodel.data.DataSetTableModel;
+import org.apache.metamodel.data.FilteredDataSet;
+import org.apache.metamodel.data.InMemoryDataSet;
+import org.apache.metamodel.data.Row;
+import org.apache.metamodel.query.FunctionType;
+import org.apache.metamodel.query.Query;
+import org.apache.metamodel.query.SelectItem;
+import org.apache.metamodel.schema.Column;
+import org.apache.metamodel.schema.ColumnType;
+import org.apache.metamodel.schema.Table;
+import org.junit.AfterClass;
+import org.junit.BeforeClass;
+import org.junit.Ignore;
+import org.junit.Test;
+
+public class SolrDataContextTest1 {
+ private static final String url = "http://localhost:8983/solr/collection1";
--- End diff --
This test would run fine if there is a local Solr deployment. Tested with Solr 4.x. Testing against Solr 5.x is next action item.
Embedded Solr does not work very reliably, so only way to run the test is if you have a real Solr deployment. The collection1 is a sample index which comes with Solr 4.x out of the box. I will try to find a better solution to this in near future when tackling Solr 5.x
There are other tests too in the MM suite which run in a similar way, such as SalesForce Data Context.
Same holds good for the below comment.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] metamodel pull request: Solr connector added to existing DataConte...
Posted by ashish-m-yh <gi...@git.apache.org>.
Github user ashish-m-yh commented on a diff in the pull request:
https://github.com/apache/metamodel/pull/16#discussion_r29111653
--- Diff: solr/src/main/java/org/apache/metamodel/solr/SolrDataContext.java ---
@@ -0,0 +1,429 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.metamodel.solr;
+
+import java.io.InputStreamReader;
+import java.io.IOException;
+import java.lang.reflect.Method;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Comparator;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+
+import org.apache.http.HttpResponse;
+import org.apache.http.client.HttpClient;
+import org.apache.http.client.methods.HttpGet;
+import org.apache.http.impl.client.DefaultHttpClient;
+import org.apache.metamodel.DataContext;
+import org.apache.metamodel.MetaModelException;
+import org.apache.metamodel.query.Query;
+import org.apache.metamodel.QueryPostprocessDataContext;
+import org.apache.metamodel.data.DataSet;
+import org.apache.metamodel.data.DataSetHeader;
+import org.apache.metamodel.data.Row;
+import org.apache.metamodel.data.SimpleDataSetHeader;
+import org.apache.metamodel.query.FilterClause;
+import org.apache.metamodel.query.FilterItem;
+import org.apache.metamodel.query.GroupByItem;
+import org.apache.metamodel.query.OrderByItem;
+import org.apache.metamodel.query.SelectItem;
+import org.apache.metamodel.query.LogicalOperator;
+import org.apache.metamodel.query.OperatorType;
+import org.apache.metamodel.query.FunctionType;
+import org.apache.metamodel.schema.Column;
+import org.apache.metamodel.schema.ColumnType;
+import org.apache.metamodel.schema.MutableColumn;
+import org.apache.metamodel.schema.MutableSchema;
+import org.apache.metamodel.schema.MutableTable;
+import org.apache.metamodel.schema.Schema;
+import org.apache.metamodel.schema.Table;
+import org.apache.metamodel.util.SimpleTableDef;
+import org.apache.solr.client.solrj.SolrQuery;
+import org.apache.solr.client.solrj.SolrQuery.ORDER;
+import org.apache.solr.client.solrj.SolrServerException;
+import org.apache.solr.client.solrj.impl.HttpSolrServer;
+import org.apache.solr.client.solrj.response.QueryResponse;
+import org.apache.solr.client.solrj.response.FacetField;
+import org.apache.solr.common.SolrDocumentList;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.fasterxml.jackson.databind.ObjectReader;
+
+/**
+ * DataContext implementation for ElasticSearch analytics engine.
+ *
+ * ElasticSearch has a data storage structure hierarchy that briefly goes like
+ * this:
+ * <ul>
+ * <li>Index</li>
+ * <li>Document type (short: Type) (within an index)</li>
+ * <li>Documents (of a particular type)</li>
+ * </ul>
+ *
+ * When instantiating this DataContext, an index name is provided. Within this
+ * index, each document type is represented as a table.
+ *
+ * This implementation supports either automatic discovery of a schema or manual
+ * specification of a schema, through the {@link SimpleTableDef} class.
+ */
+
+public class SolrDataContext extends QueryPostprocessDataContext implements
+ DataContext {
+
+ private static final Logger logger = LoggerFactory
+ .getLogger(SolrDataContext.class);
+
+ public static final String FIELD_ID = "_id";
+
+ private final SimpleTableDef tableDef;
+ private final String indexName;
+ private final String url;
+
+ private SolrQuery query = new SolrQuery();
+
+ private static enum queryTypes {
+ ROW, FACET
+ };
+
+ private static final int MAX_RETRIES = 0;
+ private static final int SOCK_TIMEOUT = 1000;
+ private static final int CONN_TIMEOUT = 5000;
+ private static final int DEFAULT_LIMIT = 1000;
+
+ public SolrDataContext(String url, String indexName) {
+ this.url = url;
+ this.indexName = indexName;
+ this.tableDef = detectSchema(url, indexName);
+ }
+
+ private SimpleTableDef detectSchema(String url, String indexName) {
+ SimpleTableDef tableDef = null;
+
+ url += "/schema";
+
+ try {
+ HttpClient httpclient = new DefaultHttpClient();
+ HttpGet request = new HttpGet(url);
+ HttpResponse response = httpclient.execute(request);
+
+ InputStreamReader inputStream = new InputStreamReader(response
+ .getEntity().getContent());
+
+ ObjectReader reader = new ObjectMapper().reader(HashMap.class);
+ Map<String, Object> outerMap = reader.readValue(inputStream);
+
+ Map<String, Object> schemaMap = (Map) outerMap.get("schema");
+
+ if (outerMap != null) {
+ List<Map<String, String>> fieldMapList = (List) schemaMap
+ .get("fields");
+ List<String> columnNames = new ArrayList<String>();
+ List<ColumnType> columnTypes = new ArrayList<ColumnType>();
+
+ for (int i = 0; i < fieldMapList.size(); i++) {
+ Map<String, String> fieldMap = fieldMapList.get(i);
+
+ columnNames.add(fieldMap.get("name"));
+ columnTypes.add(ColumnType.VARCHAR);
+ }
+
+ String[] columnNamesArr = new String[columnNames.size()];
+ columnNamesArr = columnNames.toArray(columnNamesArr);
+
+ ColumnType[] columnTypesArr = new ColumnType[columnTypes.size()];
+ columnTypesArr = columnTypes.toArray(columnTypesArr);
+
+ tableDef = new SimpleTableDef(indexName, columnNamesArr,
+ columnTypesArr);
+ }
+ } catch (IOException e) {
+ logger.error("Failed to parse schema", "", e);
+ }
+
+ return tableDef;
+ }
+
+ @Override
+ protected Schema getMainSchema() throws MetaModelException {
+ try {
+ final MutableSchema theSchema = new MutableSchema(
+ getMainSchemaName());
+
+ final MutableTable table = tableDef.toTable().setSchema(theSchema);
+ final Column idColumn = table.getColumnByName(FIELD_ID);
+ if (idColumn != null && idColumn instanceof MutableColumn) {
+ final MutableColumn mutableColumn = (MutableColumn) idColumn;
+ mutableColumn.setPrimaryKey(true);
+ }
+ theSchema.addTable(table);
+
+ return theSchema;
+ } catch (Exception e) {
+ throw new MetaModelException("Schema retrieval failed " + e);
+ }
+ }
+
+ @Override
+ protected Number executeCountQuery(Table table,
+ List<FilterItem> whereItems, boolean functionApproximationAllowed) {
+ String queryStr = "*:*";
+
+ if (whereItems != null) {
+ for (int i = 0; i < whereItems.size(); i++) {
+ FilterItem whereItem = whereItems.get(i);
+ queryStr = whereItem.toString();
--- End diff --
Question: For executeCountQuery() are all the whereItems usually AND-ed or OR-ed by MetaModel convention? I have not changed this right now, but will do so based on your response to this question.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] metamodel pull request: Solr connector added to existing DataConte...
Posted by kaspersorensen <gi...@git.apache.org>.
Github user kaspersorensen commented on a diff in the pull request:
https://github.com/apache/metamodel/pull/16#discussion_r28686195
--- Diff: solr/src/main/java/org/apache/metamodel/solr/SolrDataContext.java ---
@@ -0,0 +1,429 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.metamodel.solr;
+
+import java.io.InputStreamReader;
+import java.io.IOException;
+import java.lang.reflect.Method;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Comparator;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+
+import org.apache.http.HttpResponse;
+import org.apache.http.client.HttpClient;
+import org.apache.http.client.methods.HttpGet;
+import org.apache.http.impl.client.DefaultHttpClient;
+import org.apache.metamodel.DataContext;
+import org.apache.metamodel.MetaModelException;
+import org.apache.metamodel.query.Query;
+import org.apache.metamodel.QueryPostprocessDataContext;
+import org.apache.metamodel.data.DataSet;
+import org.apache.metamodel.data.DataSetHeader;
+import org.apache.metamodel.data.Row;
+import org.apache.metamodel.data.SimpleDataSetHeader;
+import org.apache.metamodel.query.FilterClause;
+import org.apache.metamodel.query.FilterItem;
+import org.apache.metamodel.query.GroupByItem;
+import org.apache.metamodel.query.OrderByItem;
+import org.apache.metamodel.query.SelectItem;
+import org.apache.metamodel.query.LogicalOperator;
+import org.apache.metamodel.query.OperatorType;
+import org.apache.metamodel.query.FunctionType;
+import org.apache.metamodel.schema.Column;
+import org.apache.metamodel.schema.ColumnType;
+import org.apache.metamodel.schema.MutableColumn;
+import org.apache.metamodel.schema.MutableSchema;
+import org.apache.metamodel.schema.MutableTable;
+import org.apache.metamodel.schema.Schema;
+import org.apache.metamodel.schema.Table;
+import org.apache.metamodel.util.SimpleTableDef;
+import org.apache.solr.client.solrj.SolrQuery;
+import org.apache.solr.client.solrj.SolrQuery.ORDER;
+import org.apache.solr.client.solrj.SolrServerException;
+import org.apache.solr.client.solrj.impl.HttpSolrServer;
+import org.apache.solr.client.solrj.response.QueryResponse;
+import org.apache.solr.client.solrj.response.FacetField;
+import org.apache.solr.common.SolrDocumentList;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.fasterxml.jackson.databind.ObjectReader;
+
+/**
+ * DataContext implementation for ElasticSearch analytics engine.
+ *
+ * ElasticSearch has a data storage structure hierarchy that briefly goes like
+ * this:
+ * <ul>
+ * <li>Index</li>
+ * <li>Document type (short: Type) (within an index)</li>
+ * <li>Documents (of a particular type)</li>
+ * </ul>
+ *
+ * When instantiating this DataContext, an index name is provided. Within this
+ * index, each document type is represented as a table.
+ *
+ * This implementation supports either automatic discovery of a schema or manual
+ * specification of a schema, through the {@link SimpleTableDef} class.
+ */
+
+public class SolrDataContext extends QueryPostprocessDataContext implements
+ DataContext {
+
+ private static final Logger logger = LoggerFactory
+ .getLogger(SolrDataContext.class);
+
+ public static final String FIELD_ID = "_id";
+
+ private final SimpleTableDef tableDef;
+ private final String indexName;
+ private final String url;
+
+ private SolrQuery query = new SolrQuery();
+
+ private static enum queryTypes {
+ ROW, FACET
+ };
+
+ private static final int MAX_RETRIES = 0;
+ private static final int SOCK_TIMEOUT = 1000;
+ private static final int CONN_TIMEOUT = 5000;
+ private static final int DEFAULT_LIMIT = 1000;
+
+ public SolrDataContext(String url, String indexName) {
+ this.url = url;
+ this.indexName = indexName;
+ this.tableDef = detectSchema(url, indexName);
+ }
+
+ private SimpleTableDef detectSchema(String url, String indexName) {
+ SimpleTableDef tableDef = null;
+
+ url += "/schema";
+
+ try {
+ HttpClient httpclient = new DefaultHttpClient();
+ HttpGet request = new HttpGet(url);
+ HttpResponse response = httpclient.execute(request);
+
+ InputStreamReader inputStream = new InputStreamReader(response
+ .getEntity().getContent());
+
+ ObjectReader reader = new ObjectMapper().reader(HashMap.class);
--- End diff --
Is there not a Solr client API for doing these things? If not, OK, but to me it seems like a lot of work to do the JSON parsing manually - I could imagine many things going wrong with this if the response is not quite like we expect it.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] metamodel pull request: Solr connector added to existing DataConte...
Posted by kaspersorensen <gi...@git.apache.org>.
Github user kaspersorensen commented on a diff in the pull request:
https://github.com/apache/metamodel/pull/16#discussion_r28685924
--- Diff: full/pom.xml ---
@@ -151,6 +151,11 @@ under the License.
<version>${project.version}</version>
</dependency>
<dependency>
+ <groupId>org.apache.metamodel</groupId>
--- End diff --
Indentation looks to be different here
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] metamodel pull request: Solr connector added to existing DataConte...
Posted by kaspersorensen <gi...@git.apache.org>.
Github user kaspersorensen commented on the pull request:
https://github.com/apache/metamodel/pull/16#issuecomment-96419179
For the future: You don't need to close the Pull Requests just because you have changes. The moment you commit new code to the branch then the existing PR is automatically updated. Closing it and opens a new one is a bit troublesome for the reviewers because we loose track of existing remarks.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] metamodel pull request: Solr connector added to existing DataConte...
Posted by ashish-m-yh <gi...@git.apache.org>.
Github user ashish-m-yh commented on a diff in the pull request:
https://github.com/apache/metamodel/pull/16#discussion_r29111634
--- Diff: solr/src/main/java/org/apache/metamodel/solr/SolrDataContext.java ---
@@ -0,0 +1,429 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.metamodel.solr;
+
+import java.io.InputStreamReader;
+import java.io.IOException;
+import java.lang.reflect.Method;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Comparator;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+
+import org.apache.http.HttpResponse;
+import org.apache.http.client.HttpClient;
+import org.apache.http.client.methods.HttpGet;
+import org.apache.http.impl.client.DefaultHttpClient;
+import org.apache.metamodel.DataContext;
+import org.apache.metamodel.MetaModelException;
+import org.apache.metamodel.query.Query;
+import org.apache.metamodel.QueryPostprocessDataContext;
+import org.apache.metamodel.data.DataSet;
+import org.apache.metamodel.data.DataSetHeader;
+import org.apache.metamodel.data.Row;
+import org.apache.metamodel.data.SimpleDataSetHeader;
+import org.apache.metamodel.query.FilterClause;
+import org.apache.metamodel.query.FilterItem;
+import org.apache.metamodel.query.GroupByItem;
+import org.apache.metamodel.query.OrderByItem;
+import org.apache.metamodel.query.SelectItem;
+import org.apache.metamodel.query.LogicalOperator;
+import org.apache.metamodel.query.OperatorType;
+import org.apache.metamodel.query.FunctionType;
+import org.apache.metamodel.schema.Column;
+import org.apache.metamodel.schema.ColumnType;
+import org.apache.metamodel.schema.MutableColumn;
+import org.apache.metamodel.schema.MutableSchema;
+import org.apache.metamodel.schema.MutableTable;
+import org.apache.metamodel.schema.Schema;
+import org.apache.metamodel.schema.Table;
+import org.apache.metamodel.util.SimpleTableDef;
+import org.apache.solr.client.solrj.SolrQuery;
+import org.apache.solr.client.solrj.SolrQuery.ORDER;
+import org.apache.solr.client.solrj.SolrServerException;
+import org.apache.solr.client.solrj.impl.HttpSolrServer;
+import org.apache.solr.client.solrj.response.QueryResponse;
+import org.apache.solr.client.solrj.response.FacetField;
+import org.apache.solr.common.SolrDocumentList;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.fasterxml.jackson.databind.ObjectReader;
+
+/**
+ * DataContext implementation for ElasticSearch analytics engine.
+ *
+ * ElasticSearch has a data storage structure hierarchy that briefly goes like
+ * this:
+ * <ul>
+ * <li>Index</li>
+ * <li>Document type (short: Type) (within an index)</li>
+ * <li>Documents (of a particular type)</li>
+ * </ul>
+ *
+ * When instantiating this DataContext, an index name is provided. Within this
+ * index, each document type is represented as a table.
+ *
+ * This implementation supports either automatic discovery of a schema or manual
+ * specification of a schema, through the {@link SimpleTableDef} class.
+ */
+
+public class SolrDataContext extends QueryPostprocessDataContext implements
+ DataContext {
+
+ private static final Logger logger = LoggerFactory
+ .getLogger(SolrDataContext.class);
+
+ public static final String FIELD_ID = "_id";
+
+ private final SimpleTableDef tableDef;
+ private final String indexName;
+ private final String url;
+
+ private SolrQuery query = new SolrQuery();
+
+ private static enum queryTypes {
+ ROW, FACET
+ };
+
+ private static final int MAX_RETRIES = 0;
+ private static final int SOCK_TIMEOUT = 1000;
+ private static final int CONN_TIMEOUT = 5000;
+ private static final int DEFAULT_LIMIT = 1000;
+
+ public SolrDataContext(String url, String indexName) {
+ this.url = url;
+ this.indexName = indexName;
+ this.tableDef = detectSchema(url, indexName);
+ }
+
+ private SimpleTableDef detectSchema(String url, String indexName) {
+ SimpleTableDef tableDef = null;
+
+ url += "/schema";
+
+ try {
+ HttpClient httpclient = new DefaultHttpClient();
+ HttpGet request = new HttpGet(url);
+ HttpResponse response = httpclient.execute(request);
+
+ InputStreamReader inputStream = new InputStreamReader(response
+ .getEntity().getContent());
+
+ ObjectReader reader = new ObjectMapper().reader(HashMap.class);
+ Map<String, Object> outerMap = reader.readValue(inputStream);
+
+ Map<String, Object> schemaMap = (Map) outerMap.get("schema");
+
+ if (outerMap != null) {
+ List<Map<String, String>> fieldMapList = (List) schemaMap
+ .get("fields");
+ List<String> columnNames = new ArrayList<String>();
+ List<ColumnType> columnTypes = new ArrayList<ColumnType>();
+
+ for (int i = 0; i < fieldMapList.size(); i++) {
+ Map<String, String> fieldMap = fieldMapList.get(i);
+
+ columnNames.add(fieldMap.get("name"));
+ columnTypes.add(ColumnType.VARCHAR);
--- End diff --
Changed to STRING to be most agnostic. There might be some minor differences in data-types across versions, so I am keeping it as STRING for now. This won't affect the querying.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] metamodel pull request: Solr connector added to existing DataConte...
Posted by ashish-m-yh <gi...@git.apache.org>.
Github user ashish-m-yh commented on a diff in the pull request:
https://github.com/apache/metamodel/pull/16#discussion_r29111614
--- Diff: solr/src/main/java/org/apache/metamodel/solr/SolrDataContext.java ---
@@ -0,0 +1,429 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.metamodel.solr;
+
+import java.io.InputStreamReader;
+import java.io.IOException;
+import java.lang.reflect.Method;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Comparator;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+
+import org.apache.http.HttpResponse;
+import org.apache.http.client.HttpClient;
+import org.apache.http.client.methods.HttpGet;
+import org.apache.http.impl.client.DefaultHttpClient;
+import org.apache.metamodel.DataContext;
+import org.apache.metamodel.MetaModelException;
+import org.apache.metamodel.query.Query;
+import org.apache.metamodel.QueryPostprocessDataContext;
+import org.apache.metamodel.data.DataSet;
+import org.apache.metamodel.data.DataSetHeader;
+import org.apache.metamodel.data.Row;
+import org.apache.metamodel.data.SimpleDataSetHeader;
+import org.apache.metamodel.query.FilterClause;
+import org.apache.metamodel.query.FilterItem;
+import org.apache.metamodel.query.GroupByItem;
+import org.apache.metamodel.query.OrderByItem;
+import org.apache.metamodel.query.SelectItem;
+import org.apache.metamodel.query.LogicalOperator;
+import org.apache.metamodel.query.OperatorType;
+import org.apache.metamodel.query.FunctionType;
+import org.apache.metamodel.schema.Column;
+import org.apache.metamodel.schema.ColumnType;
+import org.apache.metamodel.schema.MutableColumn;
+import org.apache.metamodel.schema.MutableSchema;
+import org.apache.metamodel.schema.MutableTable;
+import org.apache.metamodel.schema.Schema;
+import org.apache.metamodel.schema.Table;
+import org.apache.metamodel.util.SimpleTableDef;
+import org.apache.solr.client.solrj.SolrQuery;
+import org.apache.solr.client.solrj.SolrQuery.ORDER;
+import org.apache.solr.client.solrj.SolrServerException;
+import org.apache.solr.client.solrj.impl.HttpSolrServer;
+import org.apache.solr.client.solrj.response.QueryResponse;
+import org.apache.solr.client.solrj.response.FacetField;
+import org.apache.solr.common.SolrDocumentList;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.fasterxml.jackson.databind.ObjectReader;
+
+/**
+ * DataContext implementation for ElasticSearch analytics engine.
+ *
+ * ElasticSearch has a data storage structure hierarchy that briefly goes like
+ * this:
+ * <ul>
+ * <li>Index</li>
+ * <li>Document type (short: Type) (within an index)</li>
+ * <li>Documents (of a particular type)</li>
+ * </ul>
+ *
+ * When instantiating this DataContext, an index name is provided. Within this
+ * index, each document type is represented as a table.
+ *
+ * This implementation supports either automatic discovery of a schema or manual
+ * specification of a schema, through the {@link SimpleTableDef} class.
+ */
+
+public class SolrDataContext extends QueryPostprocessDataContext implements
+ DataContext {
+
+ private static final Logger logger = LoggerFactory
+ .getLogger(SolrDataContext.class);
+
+ public static final String FIELD_ID = "_id";
+
+ private final SimpleTableDef tableDef;
+ private final String indexName;
+ private final String url;
+
+ private SolrQuery query = new SolrQuery();
+
+ private static enum queryTypes {
+ ROW, FACET
+ };
+
+ private static final int MAX_RETRIES = 0;
+ private static final int SOCK_TIMEOUT = 1000;
+ private static final int CONN_TIMEOUT = 5000;
+ private static final int DEFAULT_LIMIT = 1000;
+
+ public SolrDataContext(String url, String indexName) {
+ this.url = url;
+ this.indexName = indexName;
+ this.tableDef = detectSchema(url, indexName);
+ }
+
+ private SimpleTableDef detectSchema(String url, String indexName) {
+ SimpleTableDef tableDef = null;
+
+ url += "/schema";
+
+ try {
+ HttpClient httpclient = new DefaultHttpClient();
+ HttpGet request = new HttpGet(url);
+ HttpResponse response = httpclient.execute(request);
+
+ InputStreamReader inputStream = new InputStreamReader(response
+ .getEntity().getContent());
+
+ ObjectReader reader = new ObjectMapper().reader(HashMap.class);
+ Map<String, Object> outerMap = reader.readValue(inputStream);
+
+ Map<String, Object> schemaMap = (Map) outerMap.get("schema");
+
+ if (outerMap != null) {
--- End diff --
Exception handling has been taken care of.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] metamodel pull request: Solr connector added to existing DataConte...
Posted by ashish-m-yh <gi...@git.apache.org>.
Github user ashish-m-yh closed the pull request at:
https://github.com/apache/metamodel/pull/16
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] metamodel pull request: Solr connector added to existing DataConte...
Posted by kaspersorensen <gi...@git.apache.org>.
Github user kaspersorensen commented on a diff in the pull request:
https://github.com/apache/metamodel/pull/16#discussion_r28686406
--- Diff: solr/src/main/java/org/apache/metamodel/solr/SolrDataContext.java ---
@@ -0,0 +1,429 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.metamodel.solr;
+
+import java.io.InputStreamReader;
+import java.io.IOException;
+import java.lang.reflect.Method;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Comparator;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+
+import org.apache.http.HttpResponse;
+import org.apache.http.client.HttpClient;
+import org.apache.http.client.methods.HttpGet;
+import org.apache.http.impl.client.DefaultHttpClient;
+import org.apache.metamodel.DataContext;
+import org.apache.metamodel.MetaModelException;
+import org.apache.metamodel.query.Query;
+import org.apache.metamodel.QueryPostprocessDataContext;
+import org.apache.metamodel.data.DataSet;
+import org.apache.metamodel.data.DataSetHeader;
+import org.apache.metamodel.data.Row;
+import org.apache.metamodel.data.SimpleDataSetHeader;
+import org.apache.metamodel.query.FilterClause;
+import org.apache.metamodel.query.FilterItem;
+import org.apache.metamodel.query.GroupByItem;
+import org.apache.metamodel.query.OrderByItem;
+import org.apache.metamodel.query.SelectItem;
+import org.apache.metamodel.query.LogicalOperator;
+import org.apache.metamodel.query.OperatorType;
+import org.apache.metamodel.query.FunctionType;
+import org.apache.metamodel.schema.Column;
+import org.apache.metamodel.schema.ColumnType;
+import org.apache.metamodel.schema.MutableColumn;
+import org.apache.metamodel.schema.MutableSchema;
+import org.apache.metamodel.schema.MutableTable;
+import org.apache.metamodel.schema.Schema;
+import org.apache.metamodel.schema.Table;
+import org.apache.metamodel.util.SimpleTableDef;
+import org.apache.solr.client.solrj.SolrQuery;
+import org.apache.solr.client.solrj.SolrQuery.ORDER;
+import org.apache.solr.client.solrj.SolrServerException;
+import org.apache.solr.client.solrj.impl.HttpSolrServer;
+import org.apache.solr.client.solrj.response.QueryResponse;
+import org.apache.solr.client.solrj.response.FacetField;
+import org.apache.solr.common.SolrDocumentList;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.fasterxml.jackson.databind.ObjectReader;
+
+/**
+ * DataContext implementation for ElasticSearch analytics engine.
+ *
+ * ElasticSearch has a data storage structure hierarchy that briefly goes like
+ * this:
+ * <ul>
+ * <li>Index</li>
+ * <li>Document type (short: Type) (within an index)</li>
+ * <li>Documents (of a particular type)</li>
+ * </ul>
+ *
+ * When instantiating this DataContext, an index name is provided. Within this
+ * index, each document type is represented as a table.
+ *
+ * This implementation supports either automatic discovery of a schema or manual
+ * specification of a schema, through the {@link SimpleTableDef} class.
+ */
+
+public class SolrDataContext extends QueryPostprocessDataContext implements
+ DataContext {
+
+ private static final Logger logger = LoggerFactory
+ .getLogger(SolrDataContext.class);
+
+ public static final String FIELD_ID = "_id";
+
+ private final SimpleTableDef tableDef;
+ private final String indexName;
+ private final String url;
+
+ private SolrQuery query = new SolrQuery();
+
+ private static enum queryTypes {
+ ROW, FACET
+ };
+
+ private static final int MAX_RETRIES = 0;
+ private static final int SOCK_TIMEOUT = 1000;
+ private static final int CONN_TIMEOUT = 5000;
+ private static final int DEFAULT_LIMIT = 1000;
+
+ public SolrDataContext(String url, String indexName) {
+ this.url = url;
+ this.indexName = indexName;
+ this.tableDef = detectSchema(url, indexName);
+ }
+
+ private SimpleTableDef detectSchema(String url, String indexName) {
+ SimpleTableDef tableDef = null;
+
+ url += "/schema";
+
+ try {
+ HttpClient httpclient = new DefaultHttpClient();
+ HttpGet request = new HttpGet(url);
+ HttpResponse response = httpclient.execute(request);
+
+ InputStreamReader inputStream = new InputStreamReader(response
+ .getEntity().getContent());
+
+ ObjectReader reader = new ObjectMapper().reader(HashMap.class);
+ Map<String, Object> outerMap = reader.readValue(inputStream);
+
+ Map<String, Object> schemaMap = (Map) outerMap.get("schema");
+
+ if (outerMap != null) {
+ List<Map<String, String>> fieldMapList = (List) schemaMap
+ .get("fields");
+ List<String> columnNames = new ArrayList<String>();
+ List<ColumnType> columnTypes = new ArrayList<ColumnType>();
+
+ for (int i = 0; i < fieldMapList.size(); i++) {
+ Map<String, String> fieldMap = fieldMapList.get(i);
+
+ columnNames.add(fieldMap.get("name"));
+ columnTypes.add(ColumnType.VARCHAR);
+ }
+
+ String[] columnNamesArr = new String[columnNames.size()];
+ columnNamesArr = columnNames.toArray(columnNamesArr);
+
+ ColumnType[] columnTypesArr = new ColumnType[columnTypes.size()];
+ columnTypesArr = columnTypes.toArray(columnTypesArr);
+
+ tableDef = new SimpleTableDef(indexName, columnNamesArr,
+ columnTypesArr);
+ }
+ } catch (IOException e) {
+ logger.error("Failed to parse schema", "", e);
+ }
+
+ return tableDef;
+ }
+
+ @Override
+ protected Schema getMainSchema() throws MetaModelException {
+ try {
+ final MutableSchema theSchema = new MutableSchema(
+ getMainSchemaName());
+
+ final MutableTable table = tableDef.toTable().setSchema(theSchema);
+ final Column idColumn = table.getColumnByName(FIELD_ID);
--- End diff --
Does a field named '_id' always exist in Solr? Or else I would expect some special treatment to introduce this as a virtual/surrogate field in the ```detectSchema``` method above.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] metamodel pull request: Solr connector added to existing DataConte...
Posted by ashish-m-yh <gi...@git.apache.org>.
Github user ashish-m-yh commented on a diff in the pull request:
https://github.com/apache/metamodel/pull/16#discussion_r29111638
--- Diff: solr/src/main/java/org/apache/metamodel/solr/SolrDataContext.java ---
@@ -0,0 +1,429 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.metamodel.solr;
+
+import java.io.InputStreamReader;
+import java.io.IOException;
+import java.lang.reflect.Method;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Comparator;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+
+import org.apache.http.HttpResponse;
+import org.apache.http.client.HttpClient;
+import org.apache.http.client.methods.HttpGet;
+import org.apache.http.impl.client.DefaultHttpClient;
+import org.apache.metamodel.DataContext;
+import org.apache.metamodel.MetaModelException;
+import org.apache.metamodel.query.Query;
+import org.apache.metamodel.QueryPostprocessDataContext;
+import org.apache.metamodel.data.DataSet;
+import org.apache.metamodel.data.DataSetHeader;
+import org.apache.metamodel.data.Row;
+import org.apache.metamodel.data.SimpleDataSetHeader;
+import org.apache.metamodel.query.FilterClause;
+import org.apache.metamodel.query.FilterItem;
+import org.apache.metamodel.query.GroupByItem;
+import org.apache.metamodel.query.OrderByItem;
+import org.apache.metamodel.query.SelectItem;
+import org.apache.metamodel.query.LogicalOperator;
+import org.apache.metamodel.query.OperatorType;
+import org.apache.metamodel.query.FunctionType;
+import org.apache.metamodel.schema.Column;
+import org.apache.metamodel.schema.ColumnType;
+import org.apache.metamodel.schema.MutableColumn;
+import org.apache.metamodel.schema.MutableSchema;
+import org.apache.metamodel.schema.MutableTable;
+import org.apache.metamodel.schema.Schema;
+import org.apache.metamodel.schema.Table;
+import org.apache.metamodel.util.SimpleTableDef;
+import org.apache.solr.client.solrj.SolrQuery;
+import org.apache.solr.client.solrj.SolrQuery.ORDER;
+import org.apache.solr.client.solrj.SolrServerException;
+import org.apache.solr.client.solrj.impl.HttpSolrServer;
+import org.apache.solr.client.solrj.response.QueryResponse;
+import org.apache.solr.client.solrj.response.FacetField;
+import org.apache.solr.common.SolrDocumentList;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.fasterxml.jackson.databind.ObjectReader;
+
+/**
+ * DataContext implementation for ElasticSearch analytics engine.
+ *
+ * ElasticSearch has a data storage structure hierarchy that briefly goes like
+ * this:
+ * <ul>
+ * <li>Index</li>
+ * <li>Document type (short: Type) (within an index)</li>
+ * <li>Documents (of a particular type)</li>
+ * </ul>
+ *
+ * When instantiating this DataContext, an index name is provided. Within this
+ * index, each document type is represented as a table.
+ *
+ * This implementation supports either automatic discovery of a schema or manual
+ * specification of a schema, through the {@link SimpleTableDef} class.
+ */
+
+public class SolrDataContext extends QueryPostprocessDataContext implements
+ DataContext {
+
+ private static final Logger logger = LoggerFactory
+ .getLogger(SolrDataContext.class);
+
+ public static final String FIELD_ID = "_id";
+
+ private final SimpleTableDef tableDef;
+ private final String indexName;
+ private final String url;
+
+ private SolrQuery query = new SolrQuery();
+
+ private static enum queryTypes {
+ ROW, FACET
+ };
+
+ private static final int MAX_RETRIES = 0;
+ private static final int SOCK_TIMEOUT = 1000;
+ private static final int CONN_TIMEOUT = 5000;
+ private static final int DEFAULT_LIMIT = 1000;
+
+ public SolrDataContext(String url, String indexName) {
+ this.url = url;
+ this.indexName = indexName;
+ this.tableDef = detectSchema(url, indexName);
+ }
+
+ private SimpleTableDef detectSchema(String url, String indexName) {
+ SimpleTableDef tableDef = null;
+
+ url += "/schema";
+
+ try {
+ HttpClient httpclient = new DefaultHttpClient();
+ HttpGet request = new HttpGet(url);
+ HttpResponse response = httpclient.execute(request);
+
+ InputStreamReader inputStream = new InputStreamReader(response
+ .getEntity().getContent());
+
+ ObjectReader reader = new ObjectMapper().reader(HashMap.class);
+ Map<String, Object> outerMap = reader.readValue(inputStream);
+
+ Map<String, Object> schemaMap = (Map) outerMap.get("schema");
+
+ if (outerMap != null) {
+ List<Map<String, String>> fieldMapList = (List) schemaMap
+ .get("fields");
+ List<String> columnNames = new ArrayList<String>();
+ List<ColumnType> columnTypes = new ArrayList<ColumnType>();
+
+ for (int i = 0; i < fieldMapList.size(); i++) {
+ Map<String, String> fieldMap = fieldMapList.get(i);
+
+ columnNames.add(fieldMap.get("name"));
+ columnTypes.add(ColumnType.VARCHAR);
+ }
+
+ String[] columnNamesArr = new String[columnNames.size()];
+ columnNamesArr = columnNames.toArray(columnNamesArr);
+
+ ColumnType[] columnTypesArr = new ColumnType[columnTypes.size()];
+ columnTypesArr = columnTypes.toArray(columnTypesArr);
+
+ tableDef = new SimpleTableDef(indexName, columnNamesArr,
+ columnTypesArr);
+ }
+ } catch (IOException e) {
+ logger.error("Failed to parse schema", "", e);
+ }
+
+ return tableDef;
+ }
+
+ @Override
+ protected Schema getMainSchema() throws MetaModelException {
+ try {
+ final MutableSchema theSchema = new MutableSchema(
+ getMainSchemaName());
+
+ final MutableTable table = tableDef.toTable().setSchema(theSchema);
+ final Column idColumn = table.getColumnByName(FIELD_ID);
--- End diff --
In Solr, id is usually part of the schema. Querying by Primary Key is anyway usually not the scenario in Solr.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] metamodel pull request: Solr connector added to existing DataConte...
Posted by kaspersorensen <gi...@git.apache.org>.
Github user kaspersorensen commented on a diff in the pull request:
https://github.com/apache/metamodel/pull/16#discussion_r28686291
--- Diff: solr/src/main/java/org/apache/metamodel/solr/SolrDataContext.java ---
@@ -0,0 +1,429 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.metamodel.solr;
+
+import java.io.InputStreamReader;
+import java.io.IOException;
+import java.lang.reflect.Method;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Comparator;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+
+import org.apache.http.HttpResponse;
+import org.apache.http.client.HttpClient;
+import org.apache.http.client.methods.HttpGet;
+import org.apache.http.impl.client.DefaultHttpClient;
+import org.apache.metamodel.DataContext;
+import org.apache.metamodel.MetaModelException;
+import org.apache.metamodel.query.Query;
+import org.apache.metamodel.QueryPostprocessDataContext;
+import org.apache.metamodel.data.DataSet;
+import org.apache.metamodel.data.DataSetHeader;
+import org.apache.metamodel.data.Row;
+import org.apache.metamodel.data.SimpleDataSetHeader;
+import org.apache.metamodel.query.FilterClause;
+import org.apache.metamodel.query.FilterItem;
+import org.apache.metamodel.query.GroupByItem;
+import org.apache.metamodel.query.OrderByItem;
+import org.apache.metamodel.query.SelectItem;
+import org.apache.metamodel.query.LogicalOperator;
+import org.apache.metamodel.query.OperatorType;
+import org.apache.metamodel.query.FunctionType;
+import org.apache.metamodel.schema.Column;
+import org.apache.metamodel.schema.ColumnType;
+import org.apache.metamodel.schema.MutableColumn;
+import org.apache.metamodel.schema.MutableSchema;
+import org.apache.metamodel.schema.MutableTable;
+import org.apache.metamodel.schema.Schema;
+import org.apache.metamodel.schema.Table;
+import org.apache.metamodel.util.SimpleTableDef;
+import org.apache.solr.client.solrj.SolrQuery;
+import org.apache.solr.client.solrj.SolrQuery.ORDER;
+import org.apache.solr.client.solrj.SolrServerException;
+import org.apache.solr.client.solrj.impl.HttpSolrServer;
+import org.apache.solr.client.solrj.response.QueryResponse;
+import org.apache.solr.client.solrj.response.FacetField;
+import org.apache.solr.common.SolrDocumentList;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.fasterxml.jackson.databind.ObjectReader;
+
+/**
+ * DataContext implementation for ElasticSearch analytics engine.
+ *
+ * ElasticSearch has a data storage structure hierarchy that briefly goes like
+ * this:
+ * <ul>
+ * <li>Index</li>
+ * <li>Document type (short: Type) (within an index)</li>
+ * <li>Documents (of a particular type)</li>
+ * </ul>
+ *
+ * When instantiating this DataContext, an index name is provided. Within this
+ * index, each document type is represented as a table.
+ *
+ * This implementation supports either automatic discovery of a schema or manual
+ * specification of a schema, through the {@link SimpleTableDef} class.
+ */
+
+public class SolrDataContext extends QueryPostprocessDataContext implements
+ DataContext {
+
+ private static final Logger logger = LoggerFactory
+ .getLogger(SolrDataContext.class);
+
+ public static final String FIELD_ID = "_id";
+
+ private final SimpleTableDef tableDef;
+ private final String indexName;
+ private final String url;
+
+ private SolrQuery query = new SolrQuery();
+
+ private static enum queryTypes {
+ ROW, FACET
+ };
+
+ private static final int MAX_RETRIES = 0;
+ private static final int SOCK_TIMEOUT = 1000;
+ private static final int CONN_TIMEOUT = 5000;
+ private static final int DEFAULT_LIMIT = 1000;
+
+ public SolrDataContext(String url, String indexName) {
+ this.url = url;
+ this.indexName = indexName;
+ this.tableDef = detectSchema(url, indexName);
+ }
+
+ private SimpleTableDef detectSchema(String url, String indexName) {
+ SimpleTableDef tableDef = null;
+
+ url += "/schema";
+
+ try {
+ HttpClient httpclient = new DefaultHttpClient();
+ HttpGet request = new HttpGet(url);
+ HttpResponse response = httpclient.execute(request);
+
+ InputStreamReader inputStream = new InputStreamReader(response
+ .getEntity().getContent());
+
+ ObjectReader reader = new ObjectMapper().reader(HashMap.class);
+ Map<String, Object> outerMap = reader.readValue(inputStream);
+
+ Map<String, Object> schemaMap = (Map) outerMap.get("schema");
+
+ if (outerMap != null) {
+ List<Map<String, String>> fieldMapList = (List) schemaMap
+ .get("fields");
+ List<String> columnNames = new ArrayList<String>();
+ List<ColumnType> columnTypes = new ArrayList<ColumnType>();
+
+ for (int i = 0; i < fieldMapList.size(); i++) {
+ Map<String, String> fieldMap = fieldMapList.get(i);
+
+ columnNames.add(fieldMap.get("name"));
+ columnTypes.add(ColumnType.VARCHAR);
--- End diff --
I would suggest using ColumnType.STRING since it's more agnostic (symbolically)... Question: Does Solr not support non-string types?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] metamodel pull request: Solr connector added to existing DataConte...
Posted by ashish-m-yh <gi...@git.apache.org>.
Github user ashish-m-yh commented on the pull request:
https://github.com/apache/metamodel/pull/16#issuecomment-96409034
I am closing this and creating a new pull request with responses to all the review comments.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] metamodel pull request: Solr connector added to existing DataConte...
Posted by kaspersorensen <gi...@git.apache.org>.
Github user kaspersorensen commented on a diff in the pull request:
https://github.com/apache/metamodel/pull/16#discussion_r28686560
--- Diff: solr/src/main/java/org/apache/metamodel/solr/SolrDataContext.java ---
@@ -0,0 +1,429 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.metamodel.solr;
+
+import java.io.InputStreamReader;
+import java.io.IOException;
+import java.lang.reflect.Method;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Comparator;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+
+import org.apache.http.HttpResponse;
+import org.apache.http.client.HttpClient;
+import org.apache.http.client.methods.HttpGet;
+import org.apache.http.impl.client.DefaultHttpClient;
+import org.apache.metamodel.DataContext;
+import org.apache.metamodel.MetaModelException;
+import org.apache.metamodel.query.Query;
+import org.apache.metamodel.QueryPostprocessDataContext;
+import org.apache.metamodel.data.DataSet;
+import org.apache.metamodel.data.DataSetHeader;
+import org.apache.metamodel.data.Row;
+import org.apache.metamodel.data.SimpleDataSetHeader;
+import org.apache.metamodel.query.FilterClause;
+import org.apache.metamodel.query.FilterItem;
+import org.apache.metamodel.query.GroupByItem;
+import org.apache.metamodel.query.OrderByItem;
+import org.apache.metamodel.query.SelectItem;
+import org.apache.metamodel.query.LogicalOperator;
+import org.apache.metamodel.query.OperatorType;
+import org.apache.metamodel.query.FunctionType;
+import org.apache.metamodel.schema.Column;
+import org.apache.metamodel.schema.ColumnType;
+import org.apache.metamodel.schema.MutableColumn;
+import org.apache.metamodel.schema.MutableSchema;
+import org.apache.metamodel.schema.MutableTable;
+import org.apache.metamodel.schema.Schema;
+import org.apache.metamodel.schema.Table;
+import org.apache.metamodel.util.SimpleTableDef;
+import org.apache.solr.client.solrj.SolrQuery;
+import org.apache.solr.client.solrj.SolrQuery.ORDER;
+import org.apache.solr.client.solrj.SolrServerException;
+import org.apache.solr.client.solrj.impl.HttpSolrServer;
+import org.apache.solr.client.solrj.response.QueryResponse;
+import org.apache.solr.client.solrj.response.FacetField;
+import org.apache.solr.common.SolrDocumentList;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.fasterxml.jackson.databind.ObjectReader;
+
+/**
+ * DataContext implementation for ElasticSearch analytics engine.
+ *
+ * ElasticSearch has a data storage structure hierarchy that briefly goes like
+ * this:
+ * <ul>
+ * <li>Index</li>
+ * <li>Document type (short: Type) (within an index)</li>
+ * <li>Documents (of a particular type)</li>
+ * </ul>
+ *
+ * When instantiating this DataContext, an index name is provided. Within this
+ * index, each document type is represented as a table.
+ *
+ * This implementation supports either automatic discovery of a schema or manual
+ * specification of a schema, through the {@link SimpleTableDef} class.
+ */
+
+public class SolrDataContext extends QueryPostprocessDataContext implements
+ DataContext {
+
+ private static final Logger logger = LoggerFactory
+ .getLogger(SolrDataContext.class);
+
+ public static final String FIELD_ID = "_id";
+
+ private final SimpleTableDef tableDef;
+ private final String indexName;
+ private final String url;
+
+ private SolrQuery query = new SolrQuery();
+
+ private static enum queryTypes {
+ ROW, FACET
+ };
+
+ private static final int MAX_RETRIES = 0;
+ private static final int SOCK_TIMEOUT = 1000;
+ private static final int CONN_TIMEOUT = 5000;
+ private static final int DEFAULT_LIMIT = 1000;
+
+ public SolrDataContext(String url, String indexName) {
+ this.url = url;
+ this.indexName = indexName;
+ this.tableDef = detectSchema(url, indexName);
+ }
+
+ private SimpleTableDef detectSchema(String url, String indexName) {
+ SimpleTableDef tableDef = null;
+
+ url += "/schema";
+
+ try {
+ HttpClient httpclient = new DefaultHttpClient();
+ HttpGet request = new HttpGet(url);
+ HttpResponse response = httpclient.execute(request);
+
+ InputStreamReader inputStream = new InputStreamReader(response
+ .getEntity().getContent());
+
+ ObjectReader reader = new ObjectMapper().reader(HashMap.class);
+ Map<String, Object> outerMap = reader.readValue(inputStream);
+
+ Map<String, Object> schemaMap = (Map) outerMap.get("schema");
+
+ if (outerMap != null) {
+ List<Map<String, String>> fieldMapList = (List) schemaMap
+ .get("fields");
+ List<String> columnNames = new ArrayList<String>();
+ List<ColumnType> columnTypes = new ArrayList<ColumnType>();
+
+ for (int i = 0; i < fieldMapList.size(); i++) {
+ Map<String, String> fieldMap = fieldMapList.get(i);
+
+ columnNames.add(fieldMap.get("name"));
+ columnTypes.add(ColumnType.VARCHAR);
+ }
+
+ String[] columnNamesArr = new String[columnNames.size()];
+ columnNamesArr = columnNames.toArray(columnNamesArr);
+
+ ColumnType[] columnTypesArr = new ColumnType[columnTypes.size()];
+ columnTypesArr = columnTypes.toArray(columnTypesArr);
+
+ tableDef = new SimpleTableDef(indexName, columnNamesArr,
+ columnTypesArr);
+ }
+ } catch (IOException e) {
+ logger.error("Failed to parse schema", "", e);
+ }
+
+ return tableDef;
+ }
+
+ @Override
+ protected Schema getMainSchema() throws MetaModelException {
+ try {
+ final MutableSchema theSchema = new MutableSchema(
+ getMainSchemaName());
+
+ final MutableTable table = tableDef.toTable().setSchema(theSchema);
+ final Column idColumn = table.getColumnByName(FIELD_ID);
+ if (idColumn != null && idColumn instanceof MutableColumn) {
+ final MutableColumn mutableColumn = (MutableColumn) idColumn;
+ mutableColumn.setPrimaryKey(true);
+ }
+ theSchema.addTable(table);
+
+ return theSchema;
+ } catch (Exception e) {
+ throw new MetaModelException("Schema retrieval failed " + e);
+ }
+ }
+
+ @Override
+ protected Number executeCountQuery(Table table,
+ List<FilterItem> whereItems, boolean functionApproximationAllowed) {
+ String queryStr = "*:*";
+
+ if (whereItems != null) {
+ for (int i = 0; i < whereItems.size(); i++) {
+ FilterItem whereItem = whereItems.get(i);
+ queryStr = whereItem.toString();
--- End diff --
the 'queryStr' variable is being overridden for each time in the loop. So if more WHERE items are there, only the last will be effective.
I also am doubtful if Solr's query language is similar to the toString() on FilterItem. Does this work?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---