You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hawq.apache.org by Quikling <gi...@git.apache.org> on 2017/06/29 18:49:19 UTC
[GitHub] incubator-hawq pull request #1261: HAWQ-1490. Added new dummy plugin for off...
GitHub user Quikling opened a pull request:
https://github.com/apache/incubator-hawq/pull/1261
HAWQ-1490. Added new dummy plugin for offline pxf testing.
This plugin allows for pxf testing without any underlying file system or filestore.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/Quikling/incubator-hawq HAWQ-1490
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-hawq/pull/1261.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1261
----
commit 3bb594dafe035f15a44578d2dc614609f5232d65
Author: John Gaskin <jo...@gmail.com>
Date: 2017-06-26T23:36:37Z
added dummy profile, doesn't read from local file yet
commit 9132efc007942383fe1b88d060f22131a02a4fac
Author: shivzone <sh...@gmail.com>
Date: 2017-06-28T21:18:04Z
HAWQ-1490. Refactored Dummy profile into pxf-api project
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hawq pull request #1261: HAWQ-1490. Added new dummy plugin for off...
Posted by Quikling <gi...@git.apache.org>.
Github user Quikling commented on a diff in the pull request:
https://github.com/apache/incubator-hawq/pull/1261#discussion_r124941368
--- Diff: pxf/pxf-api/src/main/java/org/apache/hawq/pxf/api/examples/DummyResolver.java ---
@@ -0,0 +1,29 @@
+package org.apache.hawq.pxf.api.examples;
+
+import org.apache.hawq.pxf.api.OneField;
+import org.apache.hawq.pxf.api.OneRow;
+import org.apache.hawq.pxf.api.ReadResolver;
+import org.apache.hawq.pxf.api.utilities.InputData;
+import org.apache.hawq.pxf.api.utilities.Plugin;
+import java.util.LinkedList;
+import java.util.List;
+import static org.apache.hawq.pxf.api.io.DataType.INTEGER;
+import static org.apache.hawq.pxf.api.io.DataType.VARCHAR;
+
+public class DummyResolver extends Plugin implements ReadResolver {
+ private int rowNumber;
+ public DummyResolver(InputData metaData) {
+ super(metaData);
+ rowNumber = 0;
+ }
+ @Override
+ public List<OneField> getFields(OneRow row) throws Exception {
--- End diff --
At this stage, we're just trying to test simple demo functionality. We can introduce dynamic types later.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hawq pull request #1261: HAWQ-1490. Added new dummy plugin for off...
Posted by Quikling <gi...@git.apache.org>.
Github user Quikling commented on a diff in the pull request:
https://github.com/apache/incubator-hawq/pull/1261#discussion_r124934930
--- Diff: pxf/pxf-api/src/main/java/org/apache/hawq/pxf/api/examples/DummyFragmenter.java ---
@@ -0,0 +1,39 @@
+package org.apache.hawq.pxf.api.examples;
+
+import org.apache.hawq.pxf.api.Fragmenter;
+import org.apache.hawq.pxf.api.Fragment;
+import org.apache.hawq.pxf.api.utilities.InputData;
+import java.util.List;
+
+/**
+ * Fragmenter class for text data on local filesystem.
+ *
+ * Given an text file on local filesystem(a file, directory, or wild card pattern),divide
+ * the data into fragments and return a list of them along with a list of
+ * host:port locations for each.
+ */
+public class DummyFragmenter extends Fragmenter{
+ public DummyFragmenter(InputData metaData) {
+ super(metaData);
+ }
+ /*
+ * path is a data source URI that can appear as a file name, a directory name or a wildcard
+ * returns the data fragments - identifiers of data and a list of available hosts
+ */
+ @Override
+ public List<Fragment> getFragments() throws Exception {
+ String localhostname = java.net.InetAddress.getLocalHost().getHostName();
+ String[] localHosts = new String[]{localhostname, localhostname};
+ fragments.add(new Fragment(inputData.getDataSource() + ".1" /* source name */,
+ localHosts /* available hosts list */,
+ "fragment1".getBytes()));
--- End diff --
Thanks, removed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hawq pull request #1261: HAWQ-1490. Added new dummy plugin for off...
Posted by shivzone <gi...@git.apache.org>.
Github user shivzone commented on a diff in the pull request:
https://github.com/apache/incubator-hawq/pull/1261#discussion_r126821382
--- Diff: pxf/pxf-api/src/main/java/org/apache/hawq/pxf/api/examples/DemoResolver.java ---
@@ -0,0 +1,59 @@
+package org.apache.hawq.pxf.api.examples;
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import org.apache.hawq.pxf.api.OneField;
+import org.apache.hawq.pxf.api.OneRow;
+import org.apache.hawq.pxf.api.ReadResolver;
+import org.apache.hawq.pxf.api.utilities.InputData;
+import org.apache.hawq.pxf.api.utilities.Plugin;
+import java.util.LinkedList;
+import java.util.List;
+import static org.apache.hawq.pxf.api.io.DataType.INTEGER;
+import static org.apache.hawq.pxf.api.io.DataType.VARCHAR;
+
+/**
+ * Class that defines the deserializtion of one record brought from the external input data.
+ *
+ * Dummy implementation
+ */
+public class DemoResolver extends Plugin implements ReadResolver {
+ /**
+ * Constructs the DemoResolver
+ *
+ * @param metaData
+ */
+ public DemoResolver(InputData metaData) {
+ super(metaData);
+ }
+
+ @Override
+ public List<OneField> getFields(OneRow row) throws Exception {
+ List<OneField> output = new LinkedList<OneField>();
+ Object data = row.getData();
+
+ /* break up the row into fields */
+ String[] fields = ((String) data).split(",");
--- End diff --
Yes, it has now changed to make it flexible
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hawq pull request #1261: HAWQ-1490. Added new dummy plugin for off...
Posted by sansanichfb <gi...@git.apache.org>.
Github user sansanichfb commented on a diff in the pull request:
https://github.com/apache/incubator-hawq/pull/1261#discussion_r124893900
--- Diff: pxf/pxf-api/src/main/java/org/apache/hawq/pxf/api/examples/DummyFragmenter.java ---
@@ -0,0 +1,39 @@
+package org.apache.hawq.pxf.api.examples;
+
+import org.apache.hawq.pxf.api.Fragmenter;
+import org.apache.hawq.pxf.api.Fragment;
+import org.apache.hawq.pxf.api.utilities.InputData;
+import java.util.List;
+
+/**
+ * Fragmenter class for text data on local filesystem.
+ *
+ * Given an text file on local filesystem(a file, directory, or wild card pattern),divide
+ * the data into fragments and return a list of them along with a list of
+ * host:port locations for each.
+ */
+public class DummyFragmenter extends Fragmenter{
+ public DummyFragmenter(InputData metaData) {
+ super(metaData);
+ }
+ /*
+ * path is a data source URI that can appear as a file name, a directory name or a wildcard
+ * returns the data fragments - identifiers of data and a list of available hosts
+ */
+ @Override
+ public List<Fragment> getFragments() throws Exception {
+ String localhostname = java.net.InetAddress.getLocalHost().getHostName();
+ String[] localHosts = new String[]{localhostname, localhostname};
+ fragments.add(new Fragment(inputData.getDataSource() + ".1" /* source name */,
+ localHosts /* available hosts list */,
+ "fragment1".getBytes()));
--- End diff --
/* fragment's metadata */
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hawq pull request #1261: HAWQ-1490. Added new dummy plugin for off...
Posted by Quikling <gi...@git.apache.org>.
Github user Quikling commented on a diff in the pull request:
https://github.com/apache/incubator-hawq/pull/1261#discussion_r124934892
--- Diff: pxf/pxf-api/src/main/java/org/apache/hawq/pxf/api/examples/DummyAccessor.java ---
@@ -0,0 +1,46 @@
+package org.apache.hawq.pxf.api.examples;
+
+import org.apache.hawq.pxf.api.OneRow;
+import org.apache.hawq.pxf.api.ReadAccessor;
+import org.apache.hawq.pxf.api.utilities.InputData;
+import org.apache.hawq.pxf.api.utilities.Plugin;
+import org.apache.commons.logging.Log;
+import org.apache.commons.logging.LogFactory;
+
+public class DummyAccessor extends Plugin implements ReadAccessor {
--- End diff --
Thanks! changed to "Static"
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hawq issue #1261: HAWQ-1490. Added new dummy plugin for offline px...
Posted by shivzone <gi...@git.apache.org>.
Github user shivzone commented on the issue:
https://github.com/apache/incubator-hawq/pull/1261
@denalex i've added pxf-api jar to the pxf-private.classpath file. I haven't modified the hdp or the bigtop versions of the classpath file
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hawq pull request #1261: HAWQ-1490. Added new dummy plugin for off...
Posted by kdunn926 <gi...@git.apache.org>.
Github user kdunn926 commented on a diff in the pull request:
https://github.com/apache/incubator-hawq/pull/1261#discussion_r127317467
--- Diff: pxf/pxf-api/src/test/java/org/apache/hawq/pxf/api/DemoAccessorTest.java ---
@@ -0,0 +1,79 @@
+package org.apache.hawq.pxf.api;
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+
+
+
+import org.apache.hawq.pxf.api.examples.DemoAccessor;
+import org.apache.hawq.pxf.api.utilities.InputData;
+import org.junit.Before;
+import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.mockito.Mock;
+import org.powermock.core.classloader.annotations.PrepareForTest;
+import org.powermock.modules.junit4.PowerMockRunner;
+
+import static org.junit.Assert.*;
--- End diff --
IMO it would be better to explicitly identify each import, rather than a glob import.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hawq pull request #1261: HAWQ-1490. Added new dummy plugin for off...
Posted by sansanichfb <gi...@git.apache.org>.
Github user sansanichfb commented on a diff in the pull request:
https://github.com/apache/incubator-hawq/pull/1261#discussion_r124890744
--- Diff: pxf/pxf-api/src/main/java/org/apache/hawq/pxf/api/examples/DummyResolver.java ---
@@ -0,0 +1,29 @@
+package org.apache.hawq.pxf.api.examples;
+
+import org.apache.hawq.pxf.api.OneField;
+import org.apache.hawq.pxf.api.OneRow;
+import org.apache.hawq.pxf.api.ReadResolver;
+import org.apache.hawq.pxf.api.utilities.InputData;
+import org.apache.hawq.pxf.api.utilities.Plugin;
+import java.util.LinkedList;
+import java.util.List;
+import static org.apache.hawq.pxf.api.io.DataType.INTEGER;
+import static org.apache.hawq.pxf.api.io.DataType.VARCHAR;
+
+public class DummyResolver extends Plugin implements ReadResolver {
+ private int rowNumber;
--- End diff --
Why is this member needed?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hawq pull request #1261: HAWQ-1490. Added new dummy plugin for off...
Posted by Quikling <gi...@git.apache.org>.
Github user Quikling commented on a diff in the pull request:
https://github.com/apache/incubator-hawq/pull/1261#discussion_r124916656
--- Diff: pxf/pxf-api/src/main/java/org/apache/hawq/pxf/api/examples/DummyAccessor.java ---
@@ -0,0 +1,46 @@
+package org.apache.hawq.pxf.api.examples;
--- End diff --
Thanks, will add.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hawq pull request #1261: HAWQ-1490. Added new dummy plugin for off...
Posted by Quikling <gi...@git.apache.org>.
Github user Quikling commented on a diff in the pull request:
https://github.com/apache/incubator-hawq/pull/1261#discussion_r124920725
--- Diff: pxf/pxf-api/src/main/java/org/apache/hawq/pxf/api/examples/DummyResolver.java ---
@@ -0,0 +1,29 @@
+package org.apache.hawq.pxf.api.examples;
+
+import org.apache.hawq.pxf.api.OneField;
+import org.apache.hawq.pxf.api.OneRow;
+import org.apache.hawq.pxf.api.ReadResolver;
+import org.apache.hawq.pxf.api.utilities.InputData;
+import org.apache.hawq.pxf.api.utilities.Plugin;
+import java.util.LinkedList;
+import java.util.List;
+import static org.apache.hawq.pxf.api.io.DataType.INTEGER;
+import static org.apache.hawq.pxf.api.io.DataType.VARCHAR;
+
+public class DummyResolver extends Plugin implements ReadResolver {
+ private int rowNumber;
+ public DummyResolver(InputData metaData) {
+ super(metaData);
+ rowNumber = 0;
+ }
+ @Override
+ public List<OneField> getFields(OneRow row) throws Exception {
+ /* break up the row into fields */
+ List<OneField> output = new LinkedList<OneField>();
+ String[] fields = ((String) row.getData()).split(",");
--- End diff --
row can never be null here as the ReadBridge specifically check for null. None of the other resolvers have this check as well. Will add validation for other things you mentioned
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hawq pull request #1261: HAWQ-1490. Added new dummy plugin for off...
Posted by sansanichfb <gi...@git.apache.org>.
Github user sansanichfb commented on a diff in the pull request:
https://github.com/apache/incubator-hawq/pull/1261#discussion_r124894489
--- Diff: pxf/pxf-api/src/main/java/org/apache/hawq/pxf/api/examples/DummyResolver.java ---
@@ -0,0 +1,29 @@
+package org.apache.hawq.pxf.api.examples;
+
+import org.apache.hawq.pxf.api.OneField;
+import org.apache.hawq.pxf.api.OneRow;
+import org.apache.hawq.pxf.api.ReadResolver;
+import org.apache.hawq.pxf.api.utilities.InputData;
+import org.apache.hawq.pxf.api.utilities.Plugin;
+import java.util.LinkedList;
+import java.util.List;
+import static org.apache.hawq.pxf.api.io.DataType.INTEGER;
+import static org.apache.hawq.pxf.api.io.DataType.VARCHAR;
+
+public class DummyResolver extends Plugin implements ReadResolver {
+ private int rowNumber;
+ public DummyResolver(InputData metaData) {
+ super(metaData);
+ rowNumber = 0;
+ }
+ @Override
+ public List<OneField> getFields(OneRow row) throws Exception {
--- End diff --
This resolver returns static schema with two columns, let's return dynamic schema, which comes as a BridgeApi parameters. Please refer to InputData#getColumn(int index) and InputData#getColumns() method to get requested column's metadata.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hawq pull request #1261: HAWQ-1490. Added new dummy plugin for off...
Posted by sansanichfb <gi...@git.apache.org>.
Github user sansanichfb commented on a diff in the pull request:
https://github.com/apache/incubator-hawq/pull/1261#discussion_r125977351
--- Diff: pxf/pxf-api/src/main/java/org/apache/hawq/pxf/api/examples/DemoTextResolver.java ---
@@ -0,0 +1,55 @@
+package org.apache.hawq.pxf.api.examples;
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import org.apache.hawq.pxf.api.OneField;
+import org.apache.hawq.pxf.api.OneRow;
+import org.apache.hawq.pxf.api.ReadResolver;
+import org.apache.hawq.pxf.api.utilities.InputData;
+import org.apache.hawq.pxf.api.utilities.Plugin;
+
+import java.util.LinkedList;
+import java.util.List;
+
+import static org.apache.hawq.pxf.api.io.DataType.INTEGER;
+import static org.apache.hawq.pxf.api.io.DataType.VARCHAR;
+
+/**
+ * Class that defines the deserializtion of one record brought from the external input data.
+ *
+ * Dummy implementation
+ */
+public class DemoTextResolver extends Plugin implements ReadResolver {
+ /**
+ * Constructs the DemoResolver
+ *
+ * @param metaData
+ */
+ public DemoTextResolver(InputData metaData) {
+ super(metaData);
+ }
+ @Override
+ public List<OneField> getFields(OneRow row) throws Exception {
+ List<OneField> output = new LinkedList<OneField>();
+ Object data = row.getData();
+ output.add(new OneField(VARCHAR.getOID(), data));
--- End diff --
Instead of adding one field we can just call inputData.getColumns() and add that many fields to output how many was requested. In this way, it will be dynamic and work with any table definition.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hawq pull request #1261: HAWQ-1490. Added new dummy plugin for off...
Posted by Quikling <gi...@git.apache.org>.
Github user Quikling closed the pull request at:
https://github.com/apache/incubator-hawq/pull/1261
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hawq issue #1261: HAWQ-1490. Added new dummy plugin for offline px...
Posted by denalex <gi...@git.apache.org>.
Github user denalex commented on the issue:
https://github.com/apache/incubator-hawq/pull/1261
Do we also need to add pxf-api.jar to the private classpath so that the Demo* use case is available to users without modifications ?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hawq pull request #1261: HAWQ-1490. Added new dummy plugin for off...
Posted by Quikling <gi...@git.apache.org>.
Github user Quikling commented on a diff in the pull request:
https://github.com/apache/incubator-hawq/pull/1261#discussion_r124941281
--- Diff: pxf/pxf-api/src/main/java/org/apache/hawq/pxf/api/examples/DummyAccessor.java ---
@@ -0,0 +1,46 @@
+package org.apache.hawq.pxf.api.examples;
+
+import org.apache.hawq.pxf.api.OneRow;
+import org.apache.hawq.pxf.api.ReadAccessor;
+import org.apache.hawq.pxf.api.utilities.InputData;
+import org.apache.hawq.pxf.api.utilities.Plugin;
+import org.apache.commons.logging.Log;
+import org.apache.commons.logging.LogFactory;
+
+public class DummyAccessor extends Plugin implements ReadAccessor {
+ private static final Log LOG = LogFactory.getLog(DummyAccessor.class);
+ private int rowNumber;
+ private int fragmentNumber;
+ public DummyAccessor(InputData metaData) {
+ super(metaData);
+ }
+ @Override
+ public boolean openForRead() throws Exception {
+ /* fopen or similar */
+ return true;
+ }
+ @Override
+ public OneRow readNextObject() throws Exception {
+ /* return next row , <key=fragmentNo.rowNo, val=rowNo,text,fragmentNo>*/
--- End diff --
As long as you define the schema with text as the data type for each field, you will still see text data. So shouldn't matter
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hawq pull request #1261: HAWQ-1490. Added new dummy plugin for off...
Posted by denalex <gi...@git.apache.org>.
Github user denalex commented on a diff in the pull request:
https://github.com/apache/incubator-hawq/pull/1261#discussion_r124924223
--- Diff: pxf/pxf-api/src/main/java/org/apache/hawq/pxf/api/examples/DummyAccessor.java ---
@@ -0,0 +1,46 @@
+package org.apache.hawq.pxf.api.examples;
+
+import org.apache.hawq.pxf.api.OneRow;
+import org.apache.hawq.pxf.api.ReadAccessor;
+import org.apache.hawq.pxf.api.utilities.InputData;
+import org.apache.hawq.pxf.api.utilities.Plugin;
+import org.apache.commons.logging.Log;
+import org.apache.commons.logging.LogFactory;
+
+public class DummyAccessor extends Plugin implements ReadAccessor {
+ private static final Log LOG = LogFactory.getLog(DummyAccessor.class);
+ private int rowNumber;
+ private int fragmentNumber;
+ public DummyAccessor(InputData metaData) {
+ super(metaData);
+ }
+ @Override
+ public boolean openForRead() throws Exception {
+ /* fopen or similar */
+ return true;
+ }
+ @Override
+ public OneRow readNextObject() throws Exception {
+ /* return next row , <key=fragmentNo.rowNo, val=rowNo,text,fragmentNo>*/
--- End diff --
not sure whether this is relevant here, but can we produce "X,Hello World X" string where X is a segment id ?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hawq pull request #1261: HAWQ-1490. Added new dummy plugin for off...
Posted by Quikling <gi...@git.apache.org>.
Github user Quikling commented on a diff in the pull request:
https://github.com/apache/incubator-hawq/pull/1261#discussion_r124924601
--- Diff: pxf/pxf-api/src/main/java/org/apache/hawq/pxf/api/examples/DummyFragmenter.java ---
@@ -0,0 +1,39 @@
+package org.apache.hawq.pxf.api.examples;
+
+import org.apache.hawq.pxf.api.Fragmenter;
+import org.apache.hawq.pxf.api.Fragment;
+import org.apache.hawq.pxf.api.utilities.InputData;
+import java.util.List;
+
+/**
+ * Fragmenter class for text data on local filesystem.
--- End diff --
Thanks, will update.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hawq pull request #1261: HAWQ-1490. Added new dummy plugin for off...
Posted by sansanichfb <gi...@git.apache.org>.
Github user sansanichfb commented on a diff in the pull request:
https://github.com/apache/incubator-hawq/pull/1261#discussion_r124893371
--- Diff: pxf/pxf-api/src/main/java/org/apache/hawq/pxf/api/examples/DummyFragmenter.java ---
@@ -0,0 +1,39 @@
+package org.apache.hawq.pxf.api.examples;
+
+import org.apache.hawq.pxf.api.Fragmenter;
+import org.apache.hawq.pxf.api.Fragment;
+import org.apache.hawq.pxf.api.utilities.InputData;
+import java.util.List;
+
+/**
+ * Fragmenter class for text data on local filesystem.
--- End diff --
According to logic, it's not really a local filesystem fragmenter, so please update comments.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hawq pull request #1261: HAWQ-1490. Added new dummy plugin for off...
Posted by Quikling <gi...@git.apache.org>.
Github user Quikling commented on a diff in the pull request:
https://github.com/apache/incubator-hawq/pull/1261#discussion_r124941296
--- Diff: pxf/pxf-api/src/main/java/org/apache/hawq/pxf/api/examples/DummyFragmenter.java ---
@@ -0,0 +1,39 @@
+package org.apache.hawq.pxf.api.examples;
+
+import org.apache.hawq.pxf.api.Fragmenter;
+import org.apache.hawq.pxf.api.Fragment;
+import org.apache.hawq.pxf.api.utilities.InputData;
+import java.util.List;
+
+/**
+ * Fragmenter class for text data on local filesystem.
+ *
+ * Given an text file on local filesystem(a file, directory, or wild card pattern),divide
+ * the data into fragments and return a list of them along with a list of
+ * host:port locations for each.
+ */
+public class DummyFragmenter extends Fragmenter{
+ public DummyFragmenter(InputData metaData) {
+ super(metaData);
+ }
+ /*
+ * path is a data source URI that can appear as a file name, a directory name or a wildcard
--- End diff --
Will update.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hawq pull request #1261: HAWQ-1490. Added new dummy plugin for off...
Posted by shivzone <gi...@git.apache.org>.
Github user shivzone commented on a diff in the pull request:
https://github.com/apache/incubator-hawq/pull/1261#discussion_r126821163
--- Diff: pxf/pxf-api/src/main/java/org/apache/hawq/pxf/api/examples/DemoTextResolver.java ---
@@ -0,0 +1,55 @@
+package org.apache.hawq.pxf.api.examples;
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import org.apache.hawq.pxf.api.OneField;
+import org.apache.hawq.pxf.api.OneRow;
+import org.apache.hawq.pxf.api.ReadResolver;
+import org.apache.hawq.pxf.api.utilities.InputData;
+import org.apache.hawq.pxf.api.utilities.Plugin;
+
+import java.util.LinkedList;
+import java.util.List;
+
+import static org.apache.hawq.pxf.api.io.DataType.INTEGER;
+import static org.apache.hawq.pxf.api.io.DataType.VARCHAR;
+
+/**
+ * Class that defines the deserializtion of one record brought from the external input data.
+ *
+ * Dummy implementation
+ */
+public class DemoTextResolver extends Plugin implements ReadResolver {
+ /**
+ * Constructs the DemoResolver
+ *
+ * @param metaData
+ */
+ public DemoTextResolver(InputData metaData) {
+ super(metaData);
+ }
+ @Override
+ public List<OneField> getFields(OneRow row) throws Exception {
+ List<OneField> output = new LinkedList<OneField>();
+ Object data = row.getData();
+ output.add(new OneField(VARCHAR.getOID(), data));
--- End diff --
Text resolver only sends data as one field.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hawq pull request #1261: HAWQ-1490. Added new dummy plugin for off...
Posted by denalex <gi...@git.apache.org>.
Github user denalex commented on a diff in the pull request:
https://github.com/apache/incubator-hawq/pull/1261#discussion_r124922961
--- Diff: pxf/pxf-api/src/main/java/org/apache/hawq/pxf/api/examples/DummyAccessor.java ---
@@ -0,0 +1,46 @@
+package org.apache.hawq.pxf.api.examples;
+
+import org.apache.hawq.pxf.api.OneRow;
+import org.apache.hawq.pxf.api.ReadAccessor;
+import org.apache.hawq.pxf.api.utilities.InputData;
+import org.apache.hawq.pxf.api.utilities.Plugin;
+import org.apache.commons.logging.Log;
+import org.apache.commons.logging.LogFactory;
+
+public class DummyAccessor extends Plugin implements ReadAccessor {
--- End diff --
Not sure if "Dummy" is the best way to describe it, maybe Echo or smth less dummy :)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hawq pull request #1261: HAWQ-1490. Added new dummy plugin for off...
Posted by sansanichfb <gi...@git.apache.org>.
Github user sansanichfb commented on a diff in the pull request:
https://github.com/apache/incubator-hawq/pull/1261#discussion_r124890251
--- Diff: pxf/pxf-api/src/main/java/org/apache/hawq/pxf/api/examples/DummyAccessor.java ---
@@ -0,0 +1,46 @@
+package org.apache.hawq.pxf.api.examples;
+
+import org.apache.hawq.pxf.api.OneRow;
+import org.apache.hawq.pxf.api.ReadAccessor;
+import org.apache.hawq.pxf.api.utilities.InputData;
+import org.apache.hawq.pxf.api.utilities.Plugin;
+import org.apache.commons.logging.Log;
+import org.apache.commons.logging.LogFactory;
+
+public class DummyAccessor extends Plugin implements ReadAccessor {
+ private static final Log LOG = LogFactory.getLog(DummyAccessor.class);
+ private int rowNumber;
+ private int fragmentNumber;
+ public DummyAccessor(InputData metaData) {
+ super(metaData);
+ }
+ @Override
+ public boolean openForRead() throws Exception {
+ /* fopen or similar */
--- End diff --
What does this comment mean?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hawq pull request #1261: HAWQ-1490. Added new dummy plugin for off...
Posted by sansanichfb <gi...@git.apache.org>.
Github user sansanichfb commented on a diff in the pull request:
https://github.com/apache/incubator-hawq/pull/1261#discussion_r124891123
--- Diff: pxf/pxf-api/src/main/java/org/apache/hawq/pxf/api/examples/DummyResolver.java ---
@@ -0,0 +1,29 @@
+package org.apache.hawq.pxf.api.examples;
+
+import org.apache.hawq.pxf.api.OneField;
+import org.apache.hawq.pxf.api.OneRow;
+import org.apache.hawq.pxf.api.ReadResolver;
+import org.apache.hawq.pxf.api.utilities.InputData;
+import org.apache.hawq.pxf.api.utilities.Plugin;
+import java.util.LinkedList;
+import java.util.List;
+import static org.apache.hawq.pxf.api.io.DataType.INTEGER;
+import static org.apache.hawq.pxf.api.io.DataType.VARCHAR;
+
+public class DummyResolver extends Plugin implements ReadResolver {
+ private int rowNumber;
+ public DummyResolver(InputData metaData) {
+ super(metaData);
+ rowNumber = 0;
+ }
+ @Override
+ public List<OneField> getFields(OneRow row) throws Exception {
+ /* break up the row into fields */
+ List<OneField> output = new LinkedList<OneField>();
+ String[] fields = ((String) row.getData()).split(",");
--- End diff --
I would add validations here, whether a row is not null, whether it has data, and if it has two tokens separated by a comma.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hawq issue #1261: HAWQ-1490. Added new dummy plugin for offline px...
Posted by kdunn926 <gi...@git.apache.org>.
Github user kdunn926 commented on the issue:
https://github.com/apache/incubator-hawq/pull/1261
LGTM 👍
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hawq pull request #1261: HAWQ-1490. Added new dummy plugin for off...
Posted by sansanichfb <gi...@git.apache.org>.
Github user sansanichfb commented on a diff in the pull request:
https://github.com/apache/incubator-hawq/pull/1261#discussion_r125976886
--- Diff: pxf/pxf-api/src/main/java/org/apache/hawq/pxf/api/examples/DemoResolver.java ---
@@ -0,0 +1,59 @@
+package org.apache.hawq.pxf.api.examples;
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import org.apache.hawq.pxf.api.OneField;
+import org.apache.hawq.pxf.api.OneRow;
+import org.apache.hawq.pxf.api.ReadResolver;
+import org.apache.hawq.pxf.api.utilities.InputData;
+import org.apache.hawq.pxf.api.utilities.Plugin;
+import java.util.LinkedList;
+import java.util.List;
+import static org.apache.hawq.pxf.api.io.DataType.INTEGER;
+import static org.apache.hawq.pxf.api.io.DataType.VARCHAR;
+
+/**
+ * Class that defines the deserializtion of one record brought from the external input data.
+ *
+ * Dummy implementation
+ */
+public class DemoResolver extends Plugin implements ReadResolver {
+ /**
+ * Constructs the DemoResolver
+ *
+ * @param metaData
+ */
+ public DemoResolver(InputData metaData) {
+ super(metaData);
+ }
+
+ @Override
+ public List<OneField> getFields(OneRow row) throws Exception {
+ List<OneField> output = new LinkedList<OneField>();
+ Object data = row.getData();
+
+ /* break up the row into fields */
+ String[] fields = ((String) data).split(",");
--- End diff --
Why are we not handling a case when data doesn't have 3 tokens? It would be useful for a demo profile to show how to gracefully handle errors for real profile.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hawq pull request #1261: HAWQ-1490. Added new dummy plugin for off...
Posted by sansanichfb <gi...@git.apache.org>.
Github user sansanichfb commented on a diff in the pull request:
https://github.com/apache/incubator-hawq/pull/1261#discussion_r124889896
--- Diff: pxf/pxf-api/src/main/java/org/apache/hawq/pxf/api/examples/DummyAccessor.java ---
@@ -0,0 +1,46 @@
+package org.apache.hawq.pxf.api.examples;
--- End diff --
This will fail rat check, please add Apache license header.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hawq pull request #1261: HAWQ-1490. Added new dummy plugin for off...
Posted by sansanichfb <gi...@git.apache.org>.
Github user sansanichfb commented on a diff in the pull request:
https://github.com/apache/incubator-hawq/pull/1261#discussion_r124893752
--- Diff: pxf/pxf-api/src/main/java/org/apache/hawq/pxf/api/examples/DummyFragmenter.java ---
@@ -0,0 +1,39 @@
+package org.apache.hawq.pxf.api.examples;
+
+import org.apache.hawq.pxf.api.Fragmenter;
+import org.apache.hawq.pxf.api.Fragment;
+import org.apache.hawq.pxf.api.utilities.InputData;
+import java.util.List;
+
+/**
+ * Fragmenter class for text data on local filesystem.
+ *
+ * Given an text file on local filesystem(a file, directory, or wild card pattern),divide
+ * the data into fragments and return a list of them along with a list of
+ * host:port locations for each.
+ */
+public class DummyFragmenter extends Fragmenter{
+ public DummyFragmenter(InputData metaData) {
+ super(metaData);
+ }
+ /*
+ * path is a data source URI that can appear as a file name, a directory name or a wildcard
--- End diff --
Please follow Javadoc format of comments.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hawq pull request #1261: HAWQ-1490. Added new dummy plugin for off...
Posted by Quikling <gi...@git.apache.org>.
Github user Quikling commented on a diff in the pull request:
https://github.com/apache/incubator-hawq/pull/1261#discussion_r124917078
--- Diff: pxf/pxf-api/src/main/java/org/apache/hawq/pxf/api/examples/DummyAccessor.java ---
@@ -0,0 +1,46 @@
+package org.apache.hawq.pxf.api.examples;
+
+import org.apache.hawq.pxf.api.OneRow;
+import org.apache.hawq.pxf.api.ReadAccessor;
+import org.apache.hawq.pxf.api.utilities.InputData;
+import org.apache.hawq.pxf.api.utilities.Plugin;
+import org.apache.commons.logging.Log;
+import org.apache.commons.logging.LogFactory;
+
+public class DummyAccessor extends Plugin implements ReadAccessor {
+ private static final Log LOG = LogFactory.getLog(DummyAccessor.class);
+ private int rowNumber;
+ private int fragmentNumber;
+ public DummyAccessor(InputData metaData) {
+ super(metaData);
+ }
+ @Override
+ public boolean openForRead() throws Exception {
+ /* fopen or similar */
--- End diff --
It's an example of what function might go there. Will update the comment to reflect what the function currently does.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hawq pull request #1261: HAWQ-1490. Added new dummy plugin for off...
Posted by denalex <gi...@git.apache.org>.
Github user denalex commented on a diff in the pull request:
https://github.com/apache/incubator-hawq/pull/1261#discussion_r124923619
--- Diff: pxf/pxf-api/src/main/java/org/apache/hawq/pxf/api/examples/DummyFragmenter.java ---
@@ -0,0 +1,39 @@
+package org.apache.hawq.pxf.api.examples;
+
+import org.apache.hawq.pxf.api.Fragmenter;
+import org.apache.hawq.pxf.api.Fragment;
+import org.apache.hawq.pxf.api.utilities.InputData;
+import java.util.List;
+
+/**
+ * Fragmenter class for text data on local filesystem.
+ *
+ * Given an text file on local filesystem(a file, directory, or wild card pattern),divide
+ * the data into fragments and return a list of them along with a list of
+ * host:port locations for each.
+ */
+public class DummyFragmenter extends Fragmenter{
+ public DummyFragmenter(InputData metaData) {
+ super(metaData);
+ }
+ /*
+ * path is a data source URI that can appear as a file name, a directory name or a wildcard
+ * returns the data fragments - identifiers of data and a list of available hosts
+ */
+ @Override
+ public List<Fragment> getFragments() throws Exception {
+ String localhostname = java.net.InetAddress.getLocalHost().getHostName();
+ String[] localHosts = new String[]{localhostname, localhostname};
+ fragments.add(new Fragment(inputData.getDataSource() + ".1" /* source name */,
+ localHosts /* available hosts list */,
+ "fragment1".getBytes()));
--- End diff --
I personally don't find the style of including comments inside function call signature all that useful.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hawq pull request #1261: HAWQ-1490. Added new dummy plugin for off...
Posted by Quikling <gi...@git.apache.org>.
Github user Quikling commented on a diff in the pull request:
https://github.com/apache/incubator-hawq/pull/1261#discussion_r124919185
--- Diff: pxf/pxf-api/src/main/java/org/apache/hawq/pxf/api/examples/DummyResolver.java ---
@@ -0,0 +1,29 @@
+package org.apache.hawq.pxf.api.examples;
+
+import org.apache.hawq.pxf.api.OneField;
+import org.apache.hawq.pxf.api.OneRow;
+import org.apache.hawq.pxf.api.ReadResolver;
+import org.apache.hawq.pxf.api.utilities.InputData;
+import org.apache.hawq.pxf.api.utilities.Plugin;
+import java.util.LinkedList;
+import java.util.List;
+import static org.apache.hawq.pxf.api.io.DataType.INTEGER;
+import static org.apache.hawq.pxf.api.io.DataType.VARCHAR;
+
+public class DummyResolver extends Plugin implements ReadResolver {
+ private int rowNumber;
--- End diff --
will remove
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---