You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@streampipes.apache.org by "tenthe (via GitHub)" <gi...@apache.org> on 2023/02/15 09:23:23 UTC
[PR] Remove set adapters (streampipes)
tenthe opened a new pull request, #1290:
URL: https://github.com/apache/streampipes/pull/1290
<!--
~ Licensed to the Apache Software Foundation (ASF) under one or more
~ contributor license agreements. See the NOTICE file distributed with
~ this work for additional information regarding copyright ownership.
~ The ASF licenses this file to You under the Apache License, Version 2.0
~ (the "License"); you may not use this file except in compliance with
~ the License. You may obtain a copy of the License at
~
~ http://www.apache.org/licenses/LICENSE-2.0
~
~ Unless required by applicable law or agreed to in writing, software
~ distributed under the License is distributed on an "AS IS" BASIS,
~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
~ See the License for the specific language governing permissions and
~ limitations under the License.
~
-->
<!--
Thanks for contributing! Here are some tips you can follow to help us incorporate your contribution quickly and easily:
1. If this is your first time, please read our contributor guidelines:
- https://streampipes.apache.org/getinvolved.html
- https://cwiki.apache.org/confluence/display/STREAMPIPES/Getting+Started
2. Make sure the PR title is formatted like: `[#<GitHub issue id>] PR title ...`
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., `[WIP][#<GitHub issue id>] PR title ...`.
4. Please write your PR title to summarize what this PR proposes/fixes.
5. Link the PR to the corresponding GitHub issue (if present) in the `Development` section in the right menu bar.
6. Be sure to keep the PR description updated to reflect all changes.
7. If possible, provide a concise example to reproduce the issue for a faster review.
8. Make sure tests pass via `mvn clean install`.
9. (Optional) If the contribution is large, please file an Apache ICLA
- http://apache.org/licenses/icla.pdf
-->
### Purpose
<!--
Please clarify what changes you are proposing and describe how those changes will address the issue.
Furthermore, describe potential consequences the changes might have.
-->
This PR is a **draft**.
The PR removes the data set model as described in #1289.
### Remarks
<!--
Is there anything left we need to pay attention on?
Are there some references that might be important? E.g. links to Confluence, or discussions
on the mailing list or GitHub.
-->
PR introduces (a) breaking change(s): yes
PR introduces (a) deprecation(s): no
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@streampipes.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Remove set adapters (streampipes)
Posted by "tenthe (via GitHub)" <gi...@apache.org>.
tenthe commented on code in PR #1290:
URL: https://github.com/apache/streampipes/pull/1290#discussion_r1212200835
##########
streampipes-extensions-management/src/main/java/org/apache/streampipes/extensions/management/connect/adapter/parser/CsvParser.java:
##########
@@ -0,0 +1,203 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.streampipes.extensions.management.connect.adapter.parser;
+
+
+import org.apache.streampipes.commons.exceptions.connect.ParseException;
+import org.apache.streampipes.extensions.api.connect.IParser;
+import org.apache.streampipes.extensions.api.connect.IParserEventHandler;
+import org.apache.streampipes.extensions.management.connect.adapter.util.DatatypeUtils;
+import org.apache.streampipes.model.Tuple2;
+import org.apache.streampipes.model.connect.grounding.ParserDescription;
+import org.apache.streampipes.model.connect.guess.GuessSchema;
+import org.apache.streampipes.model.staticproperty.Option;
+import org.apache.streampipes.model.staticproperty.StaticProperty;
+import org.apache.streampipes.sdk.builder.adapter.ParserDescriptionBuilder;
+import org.apache.streampipes.sdk.extractor.StaticPropertyExtractor;
+import org.apache.streampipes.sdk.helpers.Labels;
+
+import com.opencsv.CSVParser;
+import com.opencsv.CSVParserBuilder;
+import com.opencsv.CSVReader;
+import com.opencsv.CSVReaderBuilder;
+import com.opencsv.exceptions.CsvValidationException;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.BufferedReader;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.InputStreamReader;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.stream.IntStream;
+
+public class CsvParser implements IParser {
+
+ private static final Logger LOG = LoggerFactory.getLogger(CsvParser.class);
+
+ public static final String ID = "org.apache.streampipes.extensions.management.connect.adapter.parser.csv";
+ public static final String LABEL = "CSV";
+
+ public static final String DELIMITER = "delimiter";
+ public static final String HEADER = "header";
+
+ public static final String DESCRIPTION = "Can be used to read CSV";
+
+ private final ParserUtils parserUtils;
+
+ private boolean header;
+ private char delimiter;
+
+ public CsvParser() {
+ parserUtils = new ParserUtils();
+ }
+
+ public CsvParser(boolean header, char delimiter) {
+ this();
+ this.header = header;
+ this.delimiter = delimiter;
+ }
+
+ @Override
+ public IParser fromDescription(List<StaticProperty> config) {
+ StaticPropertyExtractor extractor = StaticPropertyExtractor.from(config);
+
+ char delimiter = extractor.singleValueParameter(DELIMITER, String.class).charAt(0);
+
+ boolean header = extractor.selectedMultiValues(HEADER, String.class).stream()
+ .anyMatch("Header"::equals);
+
+ return new CsvParser(header, delimiter);
+ }
+
+
+ @Override
+ public ParserDescription declareDescription() {
+ return ParserDescriptionBuilder.create(ID, LABEL, DESCRIPTION)
+ .requiredTextParameter(Labels.from(DELIMITER, "Delimiter",
Review Comment:
Yes, I am not happy with the solution either. But we need to change the "labels" API in the future and I wasn't sure how else to provide the labels since the parsers don't have resources directly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@streampipes.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Remove set adapters (streampipes)
Posted by "bossenti (via GitHub)" <gi...@apache.org>.
bossenti commented on code in PR #1290:
URL: https://github.com/apache/streampipes/pull/1290#discussion_r1208528381
##########
streampipes-extensions/streampipes-extensions-all-jvm/src/main/java/org/apache/streampipes/extensions/all/jvm/AllExtensionsInit.java:
##########
@@ -43,7 +43,7 @@ public SpServiceDefinition provideServiceDefinition() {
return SpServiceDefinitionBuilder.create("org.apache.streampipes.extensions.all.jvm",
"StreamPipes Extensions (JVM)",
"", 8090)
- //.merge(new ConnectAdapterInit().provideServiceDefinition())
+// .merge(new ConnectAdapterInit().provideServiceDefinition())
Review Comment:
Can we remove this line of code?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@streampipes.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Remove set adapters (streampipes)
Posted by "tenthe (via GitHub)" <gi...@apache.org>.
tenthe commented on code in PR #1290:
URL: https://github.com/apache/streampipes/pull/1290#discussion_r1212198328
##########
streampipes-extensions-management/src/main/java/org/apache/streampipes/extensions/management/connect/adapter/parser/util/JsonEventProperty.java:
##########
@@ -77,13 +79,15 @@ public static EventProperty getEventProperty(String key, Object o) {
((EventPropertyList) resultProperty).setEventProperty(arrayContent);
resultProperty.setRuntimeName(key);
+
}
if (resultProperty == null) {
LOG.error("Property Type was not detected in JsonParser for the schema detection. "
- + "This should never happen!");
+ + "This should never happen!");
}
+ resultProperty.setDescription("");
Review Comment:
I am not quite sure. If I see it correctly the method `JsonEventProperty.getEventProperty` is mainly used in `GeoJsonParser`. There the description is not set.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@streampipes.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Remove set adapters (streampipes)
Posted by "bossenti (via GitHub)" <gi...@apache.org>.
bossenti commented on code in PR #1290:
URL: https://github.com/apache/streampipes/pull/1290#discussion_r1207896415
##########
streampipes-extensions-api/src/main/java/org/apache/streampipes/extensions/api/extractor/IParameterExtractor.java:
##########
@@ -0,0 +1,119 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.streampipes.extensions.api.extractor;
+
+import org.apache.streampipes.commons.exceptions.SpRuntimeException;
+import org.apache.streampipes.model.base.InvocableStreamPipesEntity;
+import org.apache.streampipes.model.schema.EventProperty;
+import org.apache.streampipes.model.schema.EventPropertyPrimitive;
+import org.apache.streampipes.model.schema.PropertyScope;
+import org.apache.streampipes.model.staticproperty.StaticProperty;
+import org.apache.streampipes.model.staticproperty.StaticPropertyGroup;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.util.List;
+
+public interface IParameterExtractor<T extends InvocableStreamPipesEntity> {
+ String measurementUnit(String runtimeName, Integer streamIndex);
+
+ String inputTopic(Integer streamIndex);
+
+ Object singleValueParameter(EventPropertyPrimitive targetType, String internalName);
+
+ <V> V singleValueParameter(String internalName, Class<V> targetClass);
+
+ String textParameter(String internalName);
+
+ String secretValue(String internalName);
+
+ boolean slideToggleValue(String internalName);
+
+ String codeblockValue(String internalName);
+
+ String selectedColor(String internalName);
+
+ @Deprecated(since = "0.90.0", forRemoval = true)
+ String fileContentsAsString(String internalName) throws IOException;
Review Comment:
Do we keep this for compatibility reasons? Or why do we introduce new code that is directly marked as deprecated?
##########
streampipes-extensions-management/src/main/java/org/apache/streampipes/extensions/management/connect/adapter/parser/util/JsonEventProperty.java:
##########
@@ -77,13 +79,15 @@ public static EventProperty getEventProperty(String key, Object o) {
((EventPropertyList) resultProperty).setEventProperty(arrayContent);
resultProperty.setRuntimeName(key);
+
}
if (resultProperty == null) {
LOG.error("Property Type was not detected in JsonParser for the schema detection. "
- + "This should never happen!");
+ + "This should never happen!");
}
+ resultProperty.setDescription("");
Review Comment:
Can't we retrieve the description from somewhere or could we pass it to the method?
##########
streampipes-extensions-api/src/main/java/org/apache/streampipes/extensions/api/extractor/IStaticPropertyExtractor.java:
##########
@@ -0,0 +1,24 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.streampipes.extensions.api.extractor;
+
+import org.apache.streampipes.model.graph.DataSinkInvocation;
+
+public interface IStaticPropertyExtractor extends IParameterExtractor<DataSinkInvocation> {
+}
Review Comment:
Do we need this level of abstraction? Does it introduce anything than the naming?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@streampipes.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Remove set adapters (streampipes)
Posted by "tenthe (via GitHub)" <gi...@apache.org>.
tenthe commented on code in PR #1290:
URL: https://github.com/apache/streampipes/pull/1290#discussion_r1212202826
##########
streampipes-extensions/streampipes-connect-adapters-iiot/src/main/java/org/apache/streampipes/connect/iiot/adapters/plc4x/passive/Plc4xPassiveAdapter.java:
##########
@@ -18,100 +18,84 @@
package org.apache.streampipes.connect.iiot.adapters.plc4x.passive;
-import org.apache.streampipes.extensions.api.connect.exception.AdapterException;
Review Comment:
+1 for removing it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@streampipes.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Remove set adapters (streampipes)
Posted by "bossenti (via GitHub)" <gi...@apache.org>.
bossenti commented on PR #1290:
URL: https://github.com/apache/streampipes/pull/1290#issuecomment-1568765726
It would be great if we could port changes from https://github.com/apache/streampipes/pull/1638 to here
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@streampipes.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Remove set adapters (streampipes)
Posted by "bossenti (via GitHub)" <gi...@apache.org>.
bossenti commented on PR #1290:
URL: https://github.com/apache/streampipes/pull/1290#issuecomment-1566147499
@dominikriemer @tenthe I removed some data set artifacts in https://github.com/apache/streampipes/pull/1290/commits/b00c01c1c9a109aa12c94428d9d8c28434e52581
Can you please verify that this is correct?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@streampipes.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Remove set adapters (streampipes)
Posted by "bossenti (via GitHub)" <gi...@apache.org>.
bossenti commented on code in PR #1290:
URL: https://github.com/apache/streampipes/pull/1290#discussion_r1212596944
##########
streampipes-extensions-api/src/main/java/org/apache/streampipes/extensions/api/extractor/IStaticPropertyExtractor.java:
##########
@@ -0,0 +1,24 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.streampipes.extensions.api.extractor;
+
+import org.apache.streampipes.model.graph.DataSinkInvocation;
+
+public interface IStaticPropertyExtractor extends IParameterExtractor<DataSinkInvocation> {
+}
Review Comment:
Another PR is fine for - lets create an issue immediately so we won't forget about it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@streampipes.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Remove set adapters (streampipes)
Posted by "bossenti (via GitHub)" <gi...@apache.org>.
bossenti commented on code in PR #1290:
URL: https://github.com/apache/streampipes/pull/1290#discussion_r1207938142
##########
streampipes-extensions-management/src/main/java/org/apache/streampipes/extensions/management/connect/adapter/parser/CsvParser.java:
##########
@@ -0,0 +1,203 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.streampipes.extensions.management.connect.adapter.parser;
+
+
+import org.apache.streampipes.commons.exceptions.connect.ParseException;
+import org.apache.streampipes.extensions.api.connect.IParser;
+import org.apache.streampipes.extensions.api.connect.IParserEventHandler;
+import org.apache.streampipes.extensions.management.connect.adapter.util.DatatypeUtils;
+import org.apache.streampipes.model.Tuple2;
+import org.apache.streampipes.model.connect.grounding.ParserDescription;
+import org.apache.streampipes.model.connect.guess.GuessSchema;
+import org.apache.streampipes.model.staticproperty.Option;
+import org.apache.streampipes.model.staticproperty.StaticProperty;
+import org.apache.streampipes.sdk.builder.adapter.ParserDescriptionBuilder;
+import org.apache.streampipes.sdk.extractor.StaticPropertyExtractor;
+import org.apache.streampipes.sdk.helpers.Labels;
+
+import com.opencsv.CSVParser;
+import com.opencsv.CSVParserBuilder;
+import com.opencsv.CSVReader;
+import com.opencsv.CSVReaderBuilder;
+import com.opencsv.exceptions.CsvValidationException;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.BufferedReader;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.InputStreamReader;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.stream.IntStream;
+
+public class CsvParser implements IParser {
+
+ private static final Logger LOG = LoggerFactory.getLogger(CsvParser.class);
+
+ public static final String ID = "org.apache.streampipes.extensions.management.connect.adapter.parser.csv";
+ public static final String LABEL = "CSV";
+
+ public static final String DELIMITER = "delimiter";
+ public static final String HEADER = "header";
+
+ public static final String DESCRIPTION = "Can be used to read CSV";
+
+ private final ParserUtils parserUtils;
+
+ private boolean header;
+ private char delimiter;
+
+ public CsvParser() {
+ parserUtils = new ParserUtils();
+ }
+
+ public CsvParser(boolean header, char delimiter) {
+ this();
+ this.header = header;
+ this.delimiter = delimiter;
+ }
+
+ @Override
+ public IParser fromDescription(List<StaticProperty> config) {
+ StaticPropertyExtractor extractor = StaticPropertyExtractor.from(config);
+
+ char delimiter = extractor.singleValueParameter(DELIMITER, String.class).charAt(0);
+
+ boolean header = extractor.selectedMultiValues(HEADER, String.class).stream()
+ .anyMatch("Header"::equals);
+
+ return new CsvParser(header, delimiter);
+ }
+
+
+ @Override
+ public ParserDescription declareDescription() {
+ return ParserDescriptionBuilder.create(ID, LABEL, DESCRIPTION)
+ .requiredTextParameter(Labels.from(DELIMITER, "Delimiter",
Review Comment:
This is a deprecated syntax. Is there any alternative or should we consider this a follow-up task?
I just want to avoid to increase the migration effort later
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@streampipes.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Remove set adapters (streampipes)
Posted by "bossenti (via GitHub)" <gi...@apache.org>.
bossenti commented on code in PR #1290:
URL: https://github.com/apache/streampipes/pull/1290#discussion_r1207971306
##########
streampipes-extensions-api/src/main/java/org/apache/streampipes/extensions/api/extractor/IParameterExtractor.java:
##########
@@ -0,0 +1,119 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.streampipes.extensions.api.extractor;
+
+import org.apache.streampipes.commons.exceptions.SpRuntimeException;
+import org.apache.streampipes.model.base.InvocableStreamPipesEntity;
+import org.apache.streampipes.model.schema.EventProperty;
+import org.apache.streampipes.model.schema.EventPropertyPrimitive;
+import org.apache.streampipes.model.schema.PropertyScope;
+import org.apache.streampipes.model.staticproperty.StaticProperty;
+import org.apache.streampipes.model.staticproperty.StaticPropertyGroup;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.util.List;
+
+public interface IParameterExtractor<T extends InvocableStreamPipesEntity> {
+ String measurementUnit(String runtimeName, Integer streamIndex);
+
+ String inputTopic(Integer streamIndex);
+
+ Object singleValueParameter(EventPropertyPrimitive targetType, String internalName);
+
+ <V> V singleValueParameter(String internalName, Class<V> targetClass);
+
+ String textParameter(String internalName);
+
+ String secretValue(String internalName);
+
+ boolean slideToggleValue(String internalName);
+
+ String codeblockValue(String internalName);
+
+ String selectedColor(String internalName);
+
+ @Deprecated(since = "0.90.0", forRemoval = true)
+ String fileContentsAsString(String internalName) throws IOException;
Review Comment:
Is it still somewhen required, e.g., for migration?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@streampipes.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Remove set adapters (streampipes)
Posted by "bossenti (via GitHub)" <gi...@apache.org>.
bossenti commented on code in PR #1290:
URL: https://github.com/apache/streampipes/pull/1290#discussion_r1212601680
##########
streampipes-extensions/streampipes-connect-adapters-iiot/src/main/java/org/apache/streampipes/connect/iiot/adapters/plc4x/passive/Plc4xPassiveAdapter.java:
##########
@@ -18,100 +18,84 @@
package org.apache.streampipes.connect.iiot.adapters.plc4x.passive;
-import org.apache.streampipes.extensions.api.connect.exception.AdapterException;
Review Comment:
Done :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@streampipes.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Remove set adapters (streampipes)
Posted by "tenthe (via GitHub)" <gi...@apache.org>.
tenthe commented on code in PR #1290:
URL: https://github.com/apache/streampipes/pull/1290#discussion_r1212199036
##########
streampipes-extensions-management/src/main/java/org/apache/streampipes/extensions/management/connect/adapter/parser/util/JsonEventProperty.java:
##########
@@ -77,13 +79,15 @@ public static EventProperty getEventProperty(String key, Object o) {
((EventPropertyList) resultProperty).setEventProperty(arrayContent);
resultProperty.setRuntimeName(key);
+
}
if (resultProperty == null) {
LOG.error("Property Type was not detected in JsonParser for the schema detection. "
- + "This should never happen!");
+ + "This should never happen!");
}
+ resultProperty.setDescription("");
Review Comment:
But we should not remove the `.setDescription` I guess it is there for a reason ;)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@streampipes.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Remove set adapters (streampipes)
Posted by "tenthe (via GitHub)" <gi...@apache.org>.
tenthe commented on code in PR #1290:
URL: https://github.com/apache/streampipes/pull/1290#discussion_r1212193432
##########
streampipes-extensions-api/src/main/java/org/apache/streampipes/extensions/api/extractor/IStaticPropertyExtractor.java:
##########
@@ -0,0 +1,24 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.streampipes.extensions.api.extractor;
+
+import org.apache.streampipes.model.graph.DataSinkInvocation;
+
+public interface IStaticPropertyExtractor extends IParameterExtractor<DataSinkInvocation> {
+}
Review Comment:
Good point, I think we can still do some cleanup here, but I would suggest doing that in another PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@streampipes.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Remove set adapters (streampipes)
Posted by "dominikriemer (via GitHub)" <gi...@apache.org>.
dominikriemer commented on code in PR #1290:
URL: https://github.com/apache/streampipes/pull/1290#discussion_r1212210187
##########
streampipes-extensions-api/src/main/java/org/apache/streampipes/extensions/api/extractor/IStaticPropertyExtractor.java:
##########
@@ -0,0 +1,24 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.streampipes.extensions.api.extractor;
+
+import org.apache.streampipes.model.graph.DataSinkInvocation;
+
+public interface IStaticPropertyExtractor extends IParameterExtractor<DataSinkInvocation> {
+}
Review Comment:
It's a bit clearer from an extension developer's point of view since no generics are used - but we should probably rename it to `IAdapterStaticPropertyExtractor` to be consistent with the other extractor in case the `IStaticPropertyExtractor` is only used for adapters...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@streampipes.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Remove set adapters (streampipes)
Posted by "tenthe (via GitHub)" <gi...@apache.org>.
tenthe merged PR #1290:
URL: https://github.com/apache/streampipes/pull/1290
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@streampipes.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Remove set adapters (streampipes)
Posted by "bossenti (via GitHub)" <gi...@apache.org>.
bossenti commented on PR #1290:
URL: https://github.com/apache/streampipes/pull/1290#issuecomment-1566068546
We should definitely think about writing documentation for this change as well.
From my point of view, we should at least cover:
- how to create an adapter with the new interfaces?
- how to implement additional parsers?
What do you think here?
I would then create a follow-up issue
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@streampipes.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Remove set adapters (streampipes)
Posted by "tenthe (via GitHub)" <gi...@apache.org>.
tenthe commented on code in PR #1290:
URL: https://github.com/apache/streampipes/pull/1290#discussion_r1212192179
##########
streampipes-extensions-api/src/main/java/org/apache/streampipes/extensions/api/extractor/IParameterExtractor.java:
##########
@@ -0,0 +1,119 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.streampipes.extensions.api.extractor;
+
+import org.apache.streampipes.commons.exceptions.SpRuntimeException;
+import org.apache.streampipes.model.base.InvocableStreamPipesEntity;
+import org.apache.streampipes.model.schema.EventProperty;
+import org.apache.streampipes.model.schema.EventPropertyPrimitive;
+import org.apache.streampipes.model.schema.PropertyScope;
+import org.apache.streampipes.model.staticproperty.StaticProperty;
+import org.apache.streampipes.model.staticproperty.StaticPropertyGroup;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.util.List;
+
+public interface IParameterExtractor<T extends InvocableStreamPipesEntity> {
+ String measurementUnit(String runtimeName, Integer streamIndex);
+
+ String inputTopic(Integer streamIndex);
+
+ Object singleValueParameter(EventPropertyPrimitive targetType, String internalName);
+
+ <V> V singleValueParameter(String internalName, Class<V> targetClass);
+
+ String textParameter(String internalName);
+
+ String secretValue(String internalName);
+
+ boolean slideToggleValue(String internalName);
+
+ String codeblockValue(String internalName);
+
+ String selectedColor(String internalName);
+
+ @Deprecated(since = "0.90.0", forRemoval = true)
+ String fileContentsAsString(String internalName) throws IOException;
Review Comment:
I don't think it is necessary for migration. I think we can remove it.
We definitely need to test the migration extensively so we can determine if there will be a problem.
Should we create an issue for that?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@streampipes.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Remove set adapters (streampipes)
Posted by "bossenti (via GitHub)" <gi...@apache.org>.
bossenti commented on code in PR #1290:
URL: https://github.com/apache/streampipes/pull/1290#discussion_r1212596148
##########
streampipes-extensions-api/src/main/java/org/apache/streampipes/extensions/api/extractor/IParameterExtractor.java:
##########
@@ -0,0 +1,119 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.streampipes.extensions.api.extractor;
+
+import org.apache.streampipes.commons.exceptions.SpRuntimeException;
+import org.apache.streampipes.model.base.InvocableStreamPipesEntity;
+import org.apache.streampipes.model.schema.EventProperty;
+import org.apache.streampipes.model.schema.EventPropertyPrimitive;
+import org.apache.streampipes.model.schema.PropertyScope;
+import org.apache.streampipes.model.staticproperty.StaticProperty;
+import org.apache.streampipes.model.staticproperty.StaticPropertyGroup;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.util.List;
+
+public interface IParameterExtractor<T extends InvocableStreamPipesEntity> {
+ String measurementUnit(String runtimeName, Integer streamIndex);
+
+ String inputTopic(Integer streamIndex);
+
+ Object singleValueParameter(EventPropertyPrimitive targetType, String internalName);
+
+ <V> V singleValueParameter(String internalName, Class<V> targetClass);
+
+ String textParameter(String internalName);
+
+ String secretValue(String internalName);
+
+ boolean slideToggleValue(String internalName);
+
+ String codeblockValue(String internalName);
+
+ String selectedColor(String internalName);
+
+ @Deprecated(since = "0.90.0", forRemoval = true)
+ String fileContentsAsString(String internalName) throws IOException;
Review Comment:
👍🏼
That would be great!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@streampipes.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Remove set adapters (streampipes)
Posted by "bossenti (via GitHub)" <gi...@apache.org>.
bossenti commented on PR #1290:
URL: https://github.com/apache/streampipes/pull/1290#issuecomment-1571388699
>
Here it is: https://github.com/apache/streampipes/issues/1645
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@streampipes.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Remove set adapters (streampipes)
Posted by "bossenti (via GitHub)" <gi...@apache.org>.
bossenti commented on PR #1290:
URL: https://github.com/apache/streampipes/pull/1290#issuecomment-1571388989
> We should definitely think about writing documentation for this change as well. From my point of view, we should at least cover:
>
> * how to create an adapter with the new interfaces?
> * how to implement additional parsers?
> * how to use the adapter migration?
>
> What do you think here? I would then create a follow-up issue that we, in my opinion, should tackle within the 0.93.0 version
Here it is https://github.com/apache/streampipes/issues/1645
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@streampipes.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Remove set adapters (streampipes)
Posted by "bossenti (via GitHub)" <gi...@apache.org>.
bossenti commented on code in PR #1290:
URL: https://github.com/apache/streampipes/pull/1290#discussion_r1208482981
##########
streampipes-extensions-management/src/main/java/org/apache/streampipes/extensions/management/connect/adapter/parser/CsvParser.java:
##########
@@ -0,0 +1,203 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.streampipes.extensions.management.connect.adapter.parser;
+
+
+import org.apache.streampipes.commons.exceptions.connect.ParseException;
+import org.apache.streampipes.extensions.api.connect.IParser;
+import org.apache.streampipes.extensions.api.connect.IParserEventHandler;
+import org.apache.streampipes.extensions.management.connect.adapter.util.DatatypeUtils;
+import org.apache.streampipes.model.Tuple2;
+import org.apache.streampipes.model.connect.grounding.ParserDescription;
+import org.apache.streampipes.model.connect.guess.GuessSchema;
+import org.apache.streampipes.model.staticproperty.Option;
+import org.apache.streampipes.model.staticproperty.StaticProperty;
+import org.apache.streampipes.sdk.builder.adapter.ParserDescriptionBuilder;
+import org.apache.streampipes.sdk.extractor.StaticPropertyExtractor;
+import org.apache.streampipes.sdk.helpers.Labels;
+
+import com.opencsv.CSVParser;
+import com.opencsv.CSVParserBuilder;
+import com.opencsv.CSVReader;
+import com.opencsv.CSVReaderBuilder;
+import com.opencsv.exceptions.CsvValidationException;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.BufferedReader;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.InputStreamReader;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.stream.IntStream;
+
+public class CsvParser implements IParser {
+
+ private static final Logger LOG = LoggerFactory.getLogger(CsvParser.class);
+
+ public static final String ID = "org.apache.streampipes.extensions.management.connect.adapter.parser.csv";
+ public static final String LABEL = "CSV";
+
+ public static final String DELIMITER = "delimiter";
+ public static final String HEADER = "header";
+
+ public static final String DESCRIPTION = "Can be used to read CSV";
+
+ private final ParserUtils parserUtils;
+
+ private boolean header;
+ private char delimiter;
+
+ public CsvParser() {
+ parserUtils = new ParserUtils();
+ }
+
+ public CsvParser(boolean header, char delimiter) {
+ this();
+ this.header = header;
+ this.delimiter = delimiter;
+ }
+
+ @Override
+ public IParser fromDescription(List<StaticProperty> config) {
+ StaticPropertyExtractor extractor = StaticPropertyExtractor.from(config);
+
+ char delimiter = extractor.singleValueParameter(DELIMITER, String.class).charAt(0);
+
+ boolean header = extractor.selectedMultiValues(HEADER, String.class).stream()
+ .anyMatch("Header"::equals);
+
+ return new CsvParser(header, delimiter);
+ }
+
+
+ @Override
+ public ParserDescription declareDescription() {
+ return ParserDescriptionBuilder.create(ID, LABEL, DESCRIPTION)
+ .requiredTextParameter(Labels.from(DELIMITER, "Delimiter",
Review Comment:
Alright, https://github.com/apache/streampipes/issues/251 needs to be done first here, right?
Then let's handle this within 251 and try to integrate 251 within the 0.93.0 release
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@streampipes.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Remove set adapters (streampipes)
Posted by "bossenti (via GitHub)" <gi...@apache.org>.
bossenti commented on code in PR #1290:
URL: https://github.com/apache/streampipes/pull/1290#discussion_r1208555420
##########
streampipes-extensions/streampipes-connect-adapters-iiot/src/main/java/org/apache/streampipes/connect/iiot/adapters/plc4x/passive/Plc4xPassiveAdapter.java:
##########
@@ -18,100 +18,84 @@
package org.apache.streampipes.connect.iiot.adapters.plc4x.passive;
-import org.apache.streampipes.extensions.api.connect.exception.AdapterException;
Review Comment:
What about this one?
Do we want to remove it entirely?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@streampipes.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Remove set adapters (streampipes)
Posted by "dominikriemer (via GitHub)" <gi...@apache.org>.
dominikriemer commented on code in PR #1290:
URL: https://github.com/apache/streampipes/pull/1290#discussion_r1207966294
##########
streampipes-extensions-api/src/main/java/org/apache/streampipes/extensions/api/extractor/IParameterExtractor.java:
##########
@@ -0,0 +1,119 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.streampipes.extensions.api.extractor;
+
+import org.apache.streampipes.commons.exceptions.SpRuntimeException;
+import org.apache.streampipes.model.base.InvocableStreamPipesEntity;
+import org.apache.streampipes.model.schema.EventProperty;
+import org.apache.streampipes.model.schema.EventPropertyPrimitive;
+import org.apache.streampipes.model.schema.PropertyScope;
+import org.apache.streampipes.model.staticproperty.StaticProperty;
+import org.apache.streampipes.model.staticproperty.StaticPropertyGroup;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.util.List;
+
+public interface IParameterExtractor<T extends InvocableStreamPipesEntity> {
+ String measurementUnit(String runtimeName, Integer streamIndex);
+
+ String inputTopic(Integer streamIndex);
+
+ Object singleValueParameter(EventPropertyPrimitive targetType, String internalName);
+
+ <V> V singleValueParameter(String internalName, Class<V> targetClass);
+
+ String textParameter(String internalName);
+
+ String secretValue(String internalName);
+
+ boolean slideToggleValue(String internalName);
+
+ String codeblockValue(String internalName);
+
+ String selectedColor(String internalName);
+
+ @Deprecated(since = "0.90.0", forRemoval = true)
+ String fileContentsAsString(String internalName) throws IOException;
Review Comment:
Yes, this is just the extracted interface which has all declarations from the existing implementation. Should we remove it now or in a later release?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@streampipes.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Remove set adapters (streampipes)
Posted by "bossenti (via GitHub)" <gi...@apache.org>.
bossenti commented on code in PR #1290:
URL: https://github.com/apache/streampipes/pull/1290#discussion_r1212598249
##########
streampipes-extensions-management/src/main/java/org/apache/streampipes/extensions/management/connect/adapter/parser/util/JsonEventProperty.java:
##########
@@ -77,13 +79,15 @@ public static EventProperty getEventProperty(String key, Object o) {
((EventPropertyList) resultProperty).setEventProperty(arrayContent);
resultProperty.setRuntimeName(key);
+
}
if (resultProperty == null) {
LOG.error("Property Type was not detected in JsonParser for the schema detection. "
- + "This should never happen!");
+ + "This should never happen!");
}
+ resultProperty.setDescription("");
Review Comment:
Alright, we can also try to investigate this in a/the follow-up issue
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@streampipes.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org