You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pirk.apache.org by tellison <gi...@git.apache.org> on 2016/10/24 19:14:35 UTC
[GitHub] incubator-pirk pull request #112: [PIRK-78] Create a QuerySchema builder for...
GitHub user tellison opened a pull request:
https://github.com/apache/incubator-pirk/pull/112
[PIRK-78] Create a QuerySchema builder for assembling new schemas.
- Define the new QuerySchemaBuilder type.
- Remove QuerySchema building responsibilities from the loader.
- Tidy-up LoadQuerySchema tests.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/tellison/incubator-pirk PIRK-78
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-pirk/pull/112.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #112
----
commit 539259b57d8e7d60cc5110e609df45bad3112238
Author: Tim Ellison <t....@gmail.com>
Date: 2016-10-24T19:00:19Z
[PIRK-78] Create a QuerySchema builder for assembling new schemas.
- Define the new QuerySchemaBuilder type.
- Remove QuerySchema building responsibilities from the loader.
- Tidy-up LoadQuerySchema tests.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-pirk pull request #112: [PIRK-78] Create a QuerySchema builder for...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/incubator-pirk/pull/112
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-pirk pull request #112: [PIRK-78] Create a QuerySchema builder for...
Posted by tellison <gi...@git.apache.org>.
Github user tellison commented on a diff in the pull request:
https://github.com/apache/incubator-pirk/pull/112#discussion_r84772832
--- Diff: src/main/java/org/apache/pirk/schema/query/QuerySchemaLoader.java ---
@@ -226,50 +215,28 @@ public QuerySchema loadSchema(InputStream stream) throws IOException, PIRExcepti
Element elements = (Element) elementsList.item(0);
LinkedHashSet<String> elementNames = new LinkedHashSet<>();
- int dataElementSize = 0;
NodeList nList = elements.getElementsByTagName("name");
for (int i = 0; i < nList.getLength(); i++)
{
Node nNode = nList.item(i);
if (nNode.getNodeType() == Node.ELEMENT_NODE)
{
- // Pull the name
- String queryElementName = nNode.getFirstChild().getNodeValue().trim();
- if (!dataSchema.containsElement(queryElementName))
- {
- throw new PIRException("dataSchema = " + dataSchemaName + " does not contain requested element name = " + queryElementName);
- }
- elementNames.add(queryElementName);
- logger.info("name = " + queryElementName + " partitionerName = " + dataSchema.getPartitionerTypeName(queryElementName));
-
- // Compute the number of bits for this element.
- DataPartitioner partitioner = dataSchema.getPartitionerForElement(queryElementName);
- int bits = partitioner.getBits(dataSchema.getElementType(queryElementName));
-
- // Multiply by the number of array elements allowed, if applicable.
- if (dataSchema.isArrayElement(queryElementName))
- {
- bits *= Integer.parseInt(SystemConfiguration.getProperty("pir.numReturnArrayElements"));
- }
- dataElementSize += bits;
-
- logger.info("name = " + queryElementName + " bits = " + bits + " dataElementSize = " + dataElementSize);
+ elementNames.add(nNode.getFirstChild().getNodeValue().trim());
}
}
+ schemaBuilder.setQueryElementNames(elementNames);
// Extract the filter, if it exists
- String filterTypeName = NO_FILTER;
if (doc.getElementsByTagName("filter").item(0) != null)
{
- filterTypeName = doc.getElementsByTagName("filter").item(0).getTextContent().trim();
+ schemaBuilder.setFilterTypeName(doc.getElementsByTagName("filter").item(0).getTextContent().trim());
}
// Create a filter over the query elements.
- Set<String> filteredNamesSet = extractFilteredElementNames(doc);
- DataFilter filter = instantiateFilter(filterTypeName, filteredNamesSet);
+ schemaBuilder.setFilteredElementNames(extractFilteredElementNames(doc));
// Extract the additional fields, if they exists
- HashMap<String,String> additionalFields = new HashMap<String,String>();
+ Map<String,String> additionalFields = new HashMap<String,String>();
--- End diff --
Yep, I didn't spot that one - fixed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-pirk issue #112: [PIRK-78] Create a QuerySchema builder for assemb...
Posted by ellisonanne <gi...@git.apache.org>.
Github user ellisonanne commented on the issue:
https://github.com/apache/incubator-pirk/pull/112
+1 - looks good - will have to see why it's not passing the build with Travis-ci
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-pirk pull request #112: [PIRK-78] Create a QuerySchema builder for...
Posted by tellison <gi...@git.apache.org>.
Github user tellison commented on a diff in the pull request:
https://github.com/apache/incubator-pirk/pull/112#discussion_r84773126
--- Diff: src/test/java/org/apache/pirk/schema/query/LoadQuerySchemaTest.java ---
@@ -389,49 +320,42 @@ private void createDataSchema(String schemaFile) throws IOException
SystemConfiguration.setProperty("data.schemas", file.toString());
// Write to the file
- try
- {
- DocumentBuilderFactory dbFactory = DocumentBuilderFactory.newInstance();
- DocumentBuilder dBuilder = dbFactory.newDocumentBuilder();
- Document doc = dBuilder.newDocument();
-
- // root element
- Element rootElement = doc.createElement("schema");
- doc.appendChild(rootElement);
-
- // Add the schemaName
- Element schemaNameElement = doc.createElement("schemaName");
- schemaNameElement.appendChild(doc.createTextNode(dataSchemaName));
- rootElement.appendChild(schemaNameElement);
-
- // Add the elements
- // element1 -- single String
- TestUtils.addElement(doc, rootElement, element1, PrimitiveTypePartitioner.STRING, "false", PrimitiveTypePartitioner.class.getName());
-
- // element2 - -- array of Integers
- TestUtils.addElement(doc, rootElement, element2, PrimitiveTypePartitioner.INT, "true", PrimitiveTypePartitioner.class.getName());
-
- // element3 -- array of IP addresses
- TestUtils.addElement(doc, rootElement, element3, PrimitiveTypePartitioner.STRING, "true", IPDataPartitioner.class.getName());
-
- // element4 -- single byte type
- TestUtils.addElement(doc, rootElement, element4, PrimitiveTypePartitioner.BYTE, "false", PrimitiveTypePartitioner.class.getName());
-
- // Write to a xml file
- TransformerFactory transformerFactory = TransformerFactory.newInstance();
- Transformer transformer = transformerFactory.newTransformer();
- DOMSource source = new DOMSource(doc);
- StreamResult result = new StreamResult(file);
- transformer.transform(source, result);
-
- // Output for testing
- StreamResult consoleResult = new StreamResult(System.out);
- transformer.transform(source, consoleResult);
- System.out.println();
-
- } catch (Exception e)
- {
- e.printStackTrace();
- }
+ DocumentBuilderFactory dbFactory = DocumentBuilderFactory.newInstance();
+ DocumentBuilder dBuilder = dbFactory.newDocumentBuilder();
+ Document doc = dBuilder.newDocument();
+
+ // root element
+ Element rootElement = doc.createElement("schema");
+ doc.appendChild(rootElement);
+
+ // Add the schemaName
+ Element schemaNameElement = doc.createElement("schemaName");
+ schemaNameElement.appendChild(doc.createTextNode(dataSchemaName));
+ rootElement.appendChild(schemaNameElement);
+
+ // Add the elements
+ // element1 -- single String
+ TestUtils.addElement(doc, rootElement, element1, PrimitiveTypePartitioner.STRING, "false", PrimitiveTypePartitioner.class.getName());
+
+ // element2 - -- array of Integers
+ TestUtils.addElement(doc, rootElement, element2, PrimitiveTypePartitioner.INT, "true", PrimitiveTypePartitioner.class.getName());
+
+ // element3 -- array of IP addresses
+ TestUtils.addElement(doc, rootElement, element3, PrimitiveTypePartitioner.STRING, "true", IPDataPartitioner.class.getName());
+
+ // element4 -- single byte type
+ TestUtils.addElement(doc, rootElement, element4, PrimitiveTypePartitioner.BYTE, "false", PrimitiveTypePartitioner.class.getName());
+
+ // Write to a xml file
+ TransformerFactory transformerFactory = TransformerFactory.newInstance();
+ Transformer transformer = transformerFactory.newTransformer();
+ DOMSource source = new DOMSource(doc);
+ StreamResult result = new StreamResult(file);
+ transformer.transform(source, result);
+
+ // Output for testing
+ StreamResult consoleResult = new StreamResult(System.out);
+ transformer.transform(source, consoleResult);
+ System.out.println();
--- End diff --
Yeah, would be good. This is not changed from the original file, so I'm not going there right now as it's not immediately obvious to me how to transform the XML model onto the console stream.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-pirk issue #112: [PIRK-78] Create a QuerySchema builder for assemb...
Posted by tellison <gi...@git.apache.org>.
Github user tellison commented on the issue:
https://github.com/apache/incubator-pirk/pull/112
The local tests, and my travis-ci instance passes the tests ok. I see the failure is caused by a Maven PluginExecutionException.
I will push new changes to force a re-test.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-pirk pull request #112: [PIRK-78] Create a QuerySchema builder for...
Posted by smarthi <gi...@git.apache.org>.
Github user smarthi commented on a diff in the pull request:
https://github.com/apache/incubator-pirk/pull/112#discussion_r84764924
--- Diff: src/test/java/org/apache/pirk/schema/query/LoadQuerySchemaTest.java ---
@@ -389,49 +320,42 @@ private void createDataSchema(String schemaFile) throws IOException
SystemConfiguration.setProperty("data.schemas", file.toString());
// Write to the file
- try
- {
- DocumentBuilderFactory dbFactory = DocumentBuilderFactory.newInstance();
- DocumentBuilder dBuilder = dbFactory.newDocumentBuilder();
- Document doc = dBuilder.newDocument();
-
- // root element
- Element rootElement = doc.createElement("schema");
- doc.appendChild(rootElement);
-
- // Add the schemaName
- Element schemaNameElement = doc.createElement("schemaName");
- schemaNameElement.appendChild(doc.createTextNode(dataSchemaName));
- rootElement.appendChild(schemaNameElement);
-
- // Add the elements
- // element1 -- single String
- TestUtils.addElement(doc, rootElement, element1, PrimitiveTypePartitioner.STRING, "false", PrimitiveTypePartitioner.class.getName());
-
- // element2 - -- array of Integers
- TestUtils.addElement(doc, rootElement, element2, PrimitiveTypePartitioner.INT, "true", PrimitiveTypePartitioner.class.getName());
-
- // element3 -- array of IP addresses
- TestUtils.addElement(doc, rootElement, element3, PrimitiveTypePartitioner.STRING, "true", IPDataPartitioner.class.getName());
-
- // element4 -- single byte type
- TestUtils.addElement(doc, rootElement, element4, PrimitiveTypePartitioner.BYTE, "false", PrimitiveTypePartitioner.class.getName());
-
- // Write to a xml file
- TransformerFactory transformerFactory = TransformerFactory.newInstance();
- Transformer transformer = transformerFactory.newTransformer();
- DOMSource source = new DOMSource(doc);
- StreamResult result = new StreamResult(file);
- transformer.transform(source, result);
-
- // Output for testing
- StreamResult consoleResult = new StreamResult(System.out);
- transformer.transform(source, consoleResult);
- System.out.println();
-
- } catch (Exception e)
- {
- e.printStackTrace();
- }
+ DocumentBuilderFactory dbFactory = DocumentBuilderFactory.newInstance();
+ DocumentBuilder dBuilder = dbFactory.newDocumentBuilder();
+ Document doc = dBuilder.newDocument();
+
+ // root element
+ Element rootElement = doc.createElement("schema");
+ doc.appendChild(rootElement);
+
+ // Add the schemaName
+ Element schemaNameElement = doc.createElement("schemaName");
+ schemaNameElement.appendChild(doc.createTextNode(dataSchemaName));
+ rootElement.appendChild(schemaNameElement);
+
+ // Add the elements
+ // element1 -- single String
+ TestUtils.addElement(doc, rootElement, element1, PrimitiveTypePartitioner.STRING, "false", PrimitiveTypePartitioner.class.getName());
+
+ // element2 - -- array of Integers
+ TestUtils.addElement(doc, rootElement, element2, PrimitiveTypePartitioner.INT, "true", PrimitiveTypePartitioner.class.getName());
+
+ // element3 -- array of IP addresses
+ TestUtils.addElement(doc, rootElement, element3, PrimitiveTypePartitioner.STRING, "true", IPDataPartitioner.class.getName());
+
+ // element4 -- single byte type
+ TestUtils.addElement(doc, rootElement, element4, PrimitiveTypePartitioner.BYTE, "false", PrimitiveTypePartitioner.class.getName());
+
+ // Write to a xml file
+ TransformerFactory transformerFactory = TransformerFactory.newInstance();
+ Transformer transformer = transformerFactory.newTransformer();
+ DOMSource source = new DOMSource(doc);
+ StreamResult result = new StreamResult(file);
+ transformer.transform(source, result);
+
+ // Output for testing
+ StreamResult consoleResult = new StreamResult(System.out);
+ transformer.transform(source, consoleResult);
+ System.out.println();
--- End diff --
How about use a Log.Info here instead of System.out.println() ?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-pirk issue #112: [PIRK-78] Create a QuerySchema builder for assemb...
Posted by ellisonanne <gi...@git.apache.org>.
Github user ellisonanne commented on the issue:
https://github.com/apache/incubator-pirk/pull/112
merging this now...
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---