You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@iotdb.apache.org by GitBox <gi...@apache.org> on 2020/03/27 02:03:52 UTC
[GitHub] [incubator-iotdb] WeiZhong94 opened a new pull request #947:
[IOTDB-560] add TsFileInputFormat for reading TsFiles on flink.
WeiZhong94 opened a new pull request #947: [IOTDB-560] add TsFileInputFormat for reading TsFiles on flink.
URL: https://github.com/apache/incubator-iotdb/pull/947
TsFile is a columnar storage file format in Apache IoTDB. It is designed for time series data and supports efficient compression and query and is easy to be integrated into big data processing frameworks.
Apache Flink is a framework and distributed processing engine for stateful computations over unbounded and bounded data streams and becoming more and more popular in IOT scenes. So, it would be great to integrate IoTDB and Flink.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-iotdb] jixuan1989 commented on issue #947: [IOTDB-560]
add TsFileInputFormat for reading TsFiles on flink.
Posted by GitBox <gi...@apache.org>.
jixuan1989 commented on issue #947: [IOTDB-560] add TsFileInputFormat for reading TsFiles on flink.
URL: https://github.com/apache/incubator-iotdb/pull/947#issuecomment-610447147
3 days passed, @qiaojialin any suggestions?
If not, @WeiZhong94 , can you solve the travis conflict ? (The travis is simplified now, you just need to add the new module into the code-coverage test).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-iotdb] qiaojialin commented on a change in pull request
#947: [IOTDB-560] add TsFileInputFormat for reading TsFiles on flink.
Posted by GitBox <gi...@apache.org>.
qiaojialin commented on a change in pull request #947: [IOTDB-560] add TsFileInputFormat for reading TsFiles on flink.
URL: https://github.com/apache/incubator-iotdb/pull/947#discussion_r405483729
##########
File path: example/flink/src/main/java/org/apache/iotdb/flink/FlinkTsFileStreamSource.java
##########
@@ -0,0 +1,84 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.iotdb.flink;
+
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.api.java.typeutils.RowTypeInfo;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.streaming.api.datastream.DataStreamUtils;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.types.Row;
+import org.apache.iotdb.flink.tsfile.RowRowRecordParser;
+import org.apache.iotdb.flink.tsfile.TsFileInputFormat;
+import org.apache.iotdb.tsfile.common.constant.QueryConstant;
+import org.apache.iotdb.tsfile.read.common.Path;
+import org.apache.iotdb.tsfile.read.expression.QueryExpression;
+
+import java.io.File;
+import java.io.IOException;
+import java.util.Arrays;
+import java.util.Iterator;
+import java.util.List;
+import java.util.stream.Collectors;
+
+/**
+ * The example of reading TsFile via Flink DataStream API.
+ */
+public class FlinkTsFileStreamSource {
+
+ public static void main(String[] args) throws IOException {
+ String path = "test.tsfile";
+ TsFlieUtils.writeTsFile(path);
+ new File(path).deleteOnExit();
+ String[] filedNames = {
+ QueryConstant.RESERVED_TIME,
+ "device_1.sensor_1",
+ "device_1.sensor_2",
+ "device_1.sensor_3",
+ "device_2.sensor_1",
+ "device_2.sensor_2",
+ "device_2.sensor_3"
+ };
+ TypeInformation[] typeInformations = new TypeInformation[] {
+ Types.LONG,
+ Types.LONG,
+ Types.LONG,
+ Types.LONG,
+ Types.LONG,
+ Types.LONG,
+ Types.LONG
+ };
+ List<Path> paths = Arrays.stream(filedNames)
+ .filter(s -> !s.equals(QueryConstant.RESERVED_TIME))
+ .map(Path::new)
+ .collect(Collectors.toList());
+ RowTypeInfo rowTypeInfo = new RowTypeInfo(typeInformations, filedNames);
+ QueryExpression queryExpression = QueryExpression.create(paths, null);
Review comment:
I see, thanks!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-iotdb] sunjincheng121 commented on a change in pull
request #947: [IOTDB-560] add TsFileInputFormat for reading TsFiles on flink.
Posted by GitBox <gi...@apache.org>.
sunjincheng121 commented on a change in pull request #947: [IOTDB-560] add TsFileInputFormat for reading TsFiles on flink.
URL: https://github.com/apache/incubator-iotdb/pull/947#discussion_r401377443
##########
File path: flink-tsfile/README.md
##########
@@ -0,0 +1,93 @@
+<!--
+
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+-->
+# TsFile-Flink-Connector User Guide
+
+## 1. About TsFile-Flink-Connector
+
+TsFile-Flink-Connector implements the support of Flink for external data sources of Tsfile type.
+This enables users to read, write and query Tsfile by Flink via DataStream/DataSet API.
Review comment:
+1 add this in the follow up PR .
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-iotdb] sunjincheng121 commented on a change in pull
request #947: [IOTDB-560] add TsFileInputFormat for reading TsFiles on flink.
Posted by GitBox <gi...@apache.org>.
sunjincheng121 commented on a change in pull request #947: [IOTDB-560] add TsFileInputFormat for reading TsFiles on flink.
URL: https://github.com/apache/incubator-iotdb/pull/947#discussion_r399630044
##########
File path: tsfile/src/main/java/org/apache/iotdb/tsfile/read/expression/IExpression.java
##########
@@ -18,7 +18,9 @@
*/
package org.apache.iotdb.tsfile.read.expression;
-public interface IExpression{
+import java.io.Serializable;
+
+public interface IExpression extends Serializable {
Review comment:
This is a base interface +1 for this change. Just double check: Is there any concerns on this @jixuan1989 :)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-iotdb] sunjincheng121 commented on a change in pull
request #947: [IOTDB-560] add TsFileInputFormat for reading TsFiles on flink.
Posted by GitBox <gi...@apache.org>.
sunjincheng121 commented on a change in pull request #947: [IOTDB-560] add TsFileInputFormat for reading TsFiles on flink.
URL: https://github.com/apache/incubator-iotdb/pull/947#discussion_r399629363
##########
File path: example/flink/src/main/java/org/apache/iotdb/flink/FlinkTsFileBatchSource.java
##########
@@ -0,0 +1,77 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.iotdb.flink;
+
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.api.java.DataSet;
+import org.apache.flink.api.java.ExecutionEnvironment;
+import org.apache.flink.api.java.typeutils.RowTypeInfo;
+import org.apache.flink.types.Row;
+import org.apache.iotdb.flink.tsfile.RowRowRecordParser;
+import org.apache.iotdb.flink.tsfile.TsFileInputFormat;
+import org.apache.iotdb.tsfile.common.constant.QueryConstant;
+import org.apache.iotdb.tsfile.read.common.Path;
+import org.apache.iotdb.tsfile.read.expression.QueryExpression;
+
+import java.io.File;
+import java.util.Arrays;
+import java.util.List;
+import java.util.stream.Collectors;
+
+public class FlinkTsFileBatchSource {
Review comment:
Could you please add some Java DOC for the class?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-iotdb] qiaojialin commented on a change in pull request
#947: [IOTDB-560] add TsFileInputFormat for reading TsFiles on flink.
Posted by GitBox <gi...@apache.org>.
qiaojialin commented on a change in pull request #947: [IOTDB-560] add TsFileInputFormat for reading TsFiles on flink.
URL: https://github.com/apache/incubator-iotdb/pull/947#discussion_r405246086
##########
File path: example/flink/src/main/java/org/apache/iotdb/flink/FlinkTsFileStreamSource.java
##########
@@ -0,0 +1,84 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.iotdb.flink;
+
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.api.java.typeutils.RowTypeInfo;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.streaming.api.datastream.DataStreamUtils;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.types.Row;
+import org.apache.iotdb.flink.tsfile.RowRowRecordParser;
+import org.apache.iotdb.flink.tsfile.TsFileInputFormat;
+import org.apache.iotdb.tsfile.common.constant.QueryConstant;
+import org.apache.iotdb.tsfile.read.common.Path;
+import org.apache.iotdb.tsfile.read.expression.QueryExpression;
+
+import java.io.File;
+import java.io.IOException;
+import java.util.Arrays;
+import java.util.Iterator;
+import java.util.List;
+import java.util.stream.Collectors;
+
+/**
+ * The example of reading TsFile via Flink DataStream API.
+ */
+public class FlinkTsFileStreamSource {
+
+ public static void main(String[] args) throws IOException {
+ String path = "test.tsfile";
+ TsFlieUtils.writeTsFile(path);
+ new File(path).deleteOnExit();
+ String[] filedNames = {
+ QueryConstant.RESERVED_TIME,
+ "device_1.sensor_1",
+ "device_1.sensor_2",
+ "device_1.sensor_3",
+ "device_2.sensor_1",
+ "device_2.sensor_2",
+ "device_2.sensor_3"
+ };
+ TypeInformation[] typeInformations = new TypeInformation[] {
+ Types.LONG,
+ Types.LONG,
+ Types.LONG,
+ Types.LONG,
+ Types.LONG,
+ Types.LONG,
+ Types.LONG
+ };
+ List<Path> paths = Arrays.stream(filedNames)
+ .filter(s -> !s.equals(QueryConstant.RESERVED_TIME))
+ .map(Path::new)
+ .collect(Collectors.toList());
+ RowTypeInfo rowTypeInfo = new RowTypeInfo(typeInformations, filedNames);
+ QueryExpression queryExpression = QueryExpression.create(paths, null);
Review comment:
Hi, I wonder if the QueryExpression could be inferred automatically based on the RowTypeInfo?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-iotdb] WeiZhong94 commented on a change in pull request
#947: [IOTDB-560] add TsFileInputFormat for reading TsFiles on flink.
Posted by GitBox <gi...@apache.org>.
WeiZhong94 commented on a change in pull request #947: [IOTDB-560] add TsFileInputFormat for reading TsFiles on flink.
URL: https://github.com/apache/incubator-iotdb/pull/947#discussion_r399760098
##########
File path: flink-tsfile/src/test/java/org/apache/iotdb/flink/tool/TsFileWriteTool.java
##########
@@ -0,0 +1,234 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.flink.tool;
+
+import java.io.File;
+import org.apache.iotdb.tsfile.file.metadata.enums.TSDataType;
+import org.apache.iotdb.tsfile.file.metadata.enums.TSEncoding;
+import org.apache.iotdb.tsfile.write.TsFileWriter;
+import org.apache.iotdb.tsfile.write.record.TSRecord;
+import org.apache.iotdb.tsfile.write.record.datapoint.DataPoint;
+import org.apache.iotdb.tsfile.write.record.datapoint.FloatDataPoint;
+import org.apache.iotdb.tsfile.write.record.datapoint.IntDataPoint;
+import org.apache.iotdb.tsfile.write.schema.MeasurementSchema;
+
+/**
+ * An example of writing data to TsFile
+ */
+public class TsFileWriteTool {
Review comment:
This class is a util used to prepare the test TsFiles. I'll make its name and doc more clearly.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-iotdb] qiaojialin commented on a change in pull request
#947: [IOTDB-560] add TsFileInputFormat for reading TsFiles on flink.
Posted by GitBox <gi...@apache.org>.
qiaojialin commented on a change in pull request #947: [IOTDB-560] add TsFileInputFormat for reading TsFiles on flink.
URL: https://github.com/apache/incubator-iotdb/pull/947#discussion_r405247103
##########
File path: flink-tsfile/src/main/java/org/apache/iotdb/flink/tsfile/util/TSFileConfigUtil.java
##########
@@ -0,0 +1,65 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.flink.tsfile.util;
+
+import org.apache.iotdb.tsfile.common.conf.TSFileConfig;
+import org.apache.iotdb.tsfile.common.conf.TSFileDescriptor;
+
+/**
+ * Utils of setting global TSFileConfig.
+ */
+public class TSFileConfigUtil {
+
+ public static void setGlobalTSFileConfig(TSFileConfig config) {
+ TSFileConfig globalConfig = TSFileDescriptor.getInstance().getConfig();
Review comment:
I know, thank you!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-iotdb] WeiZhong94 commented on issue #947: [IOTDB-560]
add TsFileInputFormat for reading TsFiles on flink.
Posted by GitBox <gi...@apache.org>.
WeiZhong94 commented on issue #947: [IOTDB-560] add TsFileInputFormat for reading TsFiles on flink.
URL: https://github.com/apache/incubator-iotdb/pull/947#issuecomment-607235107
@qiaojialin @jixuan1989 Thanks for your review! I have updated this PR according to your comments. Please take a look.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-iotdb] WeiZhong94 commented on issue #947: [IOTDB-560]
add TsFileInputFormat for reading TsFiles on flink.
Posted by GitBox <gi...@apache.org>.
WeiZhong94 commented on issue #947: [IOTDB-560] add TsFileInputFormat for reading TsFiles on flink.
URL: https://github.com/apache/incubator-iotdb/pull/947#issuecomment-605599047
@sunjincheng121 Thanks for your review! I have updated this PR.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-iotdb] qiaojialin merged pull request #947: [IOTDB-560]
add TsFileInputFormat for reading TsFiles on flink.
Posted by GitBox <gi...@apache.org>.
qiaojialin merged pull request #947: [IOTDB-560] add TsFileInputFormat for reading TsFiles on flink.
URL: https://github.com/apache/incubator-iotdb/pull/947
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-iotdb] qiaojialin commented on a change in pull request
#947: [IOTDB-560] add TsFileInputFormat for reading TsFiles on flink.
Posted by GitBox <gi...@apache.org>.
qiaojialin commented on a change in pull request #947: [IOTDB-560] add TsFileInputFormat for reading TsFiles on flink.
URL: https://github.com/apache/incubator-iotdb/pull/947#discussion_r405246086
##########
File path: example/flink/src/main/java/org/apache/iotdb/flink/FlinkTsFileStreamSource.java
##########
@@ -0,0 +1,84 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.iotdb.flink;
+
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.api.java.typeutils.RowTypeInfo;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.streaming.api.datastream.DataStreamUtils;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.types.Row;
+import org.apache.iotdb.flink.tsfile.RowRowRecordParser;
+import org.apache.iotdb.flink.tsfile.TsFileInputFormat;
+import org.apache.iotdb.tsfile.common.constant.QueryConstant;
+import org.apache.iotdb.tsfile.read.common.Path;
+import org.apache.iotdb.tsfile.read.expression.QueryExpression;
+
+import java.io.File;
+import java.io.IOException;
+import java.util.Arrays;
+import java.util.Iterator;
+import java.util.List;
+import java.util.stream.Collectors;
+
+/**
+ * The example of reading TsFile via Flink DataStream API.
+ */
+public class FlinkTsFileStreamSource {
+
+ public static void main(String[] args) throws IOException {
+ String path = "test.tsfile";
+ TsFlieUtils.writeTsFile(path);
+ new File(path).deleteOnExit();
+ String[] filedNames = {
+ QueryConstant.RESERVED_TIME,
+ "device_1.sensor_1",
+ "device_1.sensor_2",
+ "device_1.sensor_3",
+ "device_2.sensor_1",
+ "device_2.sensor_2",
+ "device_2.sensor_3"
+ };
+ TypeInformation[] typeInformations = new TypeInformation[] {
+ Types.LONG,
+ Types.LONG,
+ Types.LONG,
+ Types.LONG,
+ Types.LONG,
+ Types.LONG,
+ Types.LONG
+ };
+ List<Path> paths = Arrays.stream(filedNames)
+ .filter(s -> !s.equals(QueryConstant.RESERVED_TIME))
+ .map(Path::new)
+ .collect(Collectors.toList());
+ RowTypeInfo rowTypeInfo = new RowTypeInfo(typeInformations, filedNames);
+ QueryExpression queryExpression = QueryExpression.create(paths, null);
Review comment:
Hi, I wonder if the QueryExpression could be inferred automatically based on the RowTypeInfo in the InputFormat? If so, users do not have to maintain this and ensure the consistency of queried paths and RowTypeInfo. We also don't have to check whether they match.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-iotdb] WeiZhong94 commented on issue #947: [IOTDB-560]
add TsFileInputFormat for reading TsFiles on flink.
Posted by GitBox <gi...@apache.org>.
WeiZhong94 commented on issue #947: [IOTDB-560] add TsFileInputFormat for reading TsFiles on flink.
URL: https://github.com/apache/incubator-iotdb/pull/947#issuecomment-610707477
@jixuan1989 Thanks for your notification, I have rebased this PR on top of the master branch.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-iotdb] WeiZhong94 commented on a change in pull request
#947: [IOTDB-560] add TsFileInputFormat for reading TsFiles on flink.
Posted by GitBox <gi...@apache.org>.
WeiZhong94 commented on a change in pull request #947: [IOTDB-560] add TsFileInputFormat for reading TsFiles on flink.
URL: https://github.com/apache/incubator-iotdb/pull/947#discussion_r399759767
##########
File path: flink-tsfile/src/main/java/org/apache/iotdb/flink/tsfile/util/TSFileConfigUtil.java
##########
@@ -0,0 +1,62 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.flink.tsfile.util;
+
+import org.apache.iotdb.tsfile.common.conf.TSFileConfig;
+import org.apache.iotdb.tsfile.common.conf.TSFileDescriptor;
+
+public class TSFileConfigUtil {
Review comment:
The `Utils` is a helper class for the examples of the TsFile Flink connectors, and the `TSFileConfigUtil` is used to set the user config as global TSFileConfig while opening a FileInputSplit in Flink Runtime.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-iotdb] jixuan1989 commented on a change in pull request
#947: [IOTDB-560] add TsFileInputFormat for reading TsFiles on flink.
Posted by GitBox <gi...@apache.org>.
jixuan1989 commented on a change in pull request #947: [IOTDB-560] add TsFileInputFormat for reading TsFiles on flink.
URL: https://github.com/apache/incubator-iotdb/pull/947#discussion_r400184321
##########
File path: flink-tsfile/README.md
##########
@@ -0,0 +1,93 @@
+<!--
+
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+-->
+# TsFile-Flink-Connector User Guide
+
+## 1. About TsFile-Flink-Connector
+
+TsFile-Flink-Connector implements the support of Flink for external data sources of Tsfile type.
+This enables users to read, write and query Tsfile by Flink via DataStream/DataSet API.
Review comment:
Does this module give the capacity to write TsFile?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-iotdb] qiaojialin commented on a change in pull request
#947: [IOTDB-560] add TsFileInputFormat for reading TsFiles on flink.
Posted by GitBox <gi...@apache.org>.
qiaojialin commented on a change in pull request #947: [IOTDB-560] add TsFileInputFormat for reading TsFiles on flink.
URL: https://github.com/apache/incubator-iotdb/pull/947#discussion_r399787181
##########
File path: flink-tsfile/src/main/java/org/apache/iotdb/flink/tsfile/util/TSFileConfigUtil.java
##########
@@ -0,0 +1,65 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.flink.tsfile.util;
+
+import org.apache.iotdb.tsfile.common.conf.TSFileConfig;
+import org.apache.iotdb.tsfile.common.conf.TSFileDescriptor;
+
+/**
+ * Utils of setting global TSFileConfig.
+ */
+public class TSFileConfigUtil {
+
+ public static void setGlobalTSFileConfig(TSFileConfig config) {
+ TSFileConfig globalConfig = TSFileDescriptor.getInstance().getConfig();
Review comment:
Hi, this may be hard to maintain... how about let the user directly use TSFileConfig as follows if they want to change some configuration:
TSFileConfig config = TsFileDescriptor.getInstance().getConfig();
config.setXXX();
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-iotdb] jixuan1989 commented on a change in pull request
#947: [IOTDB-560] add TsFileInputFormat for reading TsFiles on flink.
Posted by GitBox <gi...@apache.org>.
jixuan1989 commented on a change in pull request #947: [IOTDB-560] add TsFileInputFormat for reading TsFiles on flink.
URL: https://github.com/apache/incubator-iotdb/pull/947#discussion_r403468661
##########
File path: tsfile/src/main/java/org/apache/iotdb/tsfile/read/expression/IExpression.java
##########
@@ -18,7 +18,9 @@
*/
package org.apache.iotdb.tsfile.read.expression;
-public interface IExpression{
+import java.io.Serializable;
+
+public interface IExpression extends Serializable {
Review comment:
I have no more opinions.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-iotdb] sunjincheng121 commented on a change in pull
request #947: [IOTDB-560] add TsFileInputFormat for reading TsFiles on flink.
Posted by GitBox <gi...@apache.org>.
sunjincheng121 commented on a change in pull request #947: [IOTDB-560] add TsFileInputFormat for reading TsFiles on flink.
URL: https://github.com/apache/incubator-iotdb/pull/947#discussion_r399629378
##########
File path: example/flink/src/main/java/org/apache/iotdb/flink/FlinkTsFileStreamSource.java
##########
@@ -0,0 +1,81 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.iotdb.flink;
+
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.api.java.typeutils.RowTypeInfo;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.streaming.api.datastream.DataStreamUtils;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.types.Row;
+import org.apache.iotdb.flink.tsfile.RowRowRecordParser;
+import org.apache.iotdb.flink.tsfile.TsFileInputFormat;
+import org.apache.iotdb.tsfile.common.constant.QueryConstant;
+import org.apache.iotdb.tsfile.read.common.Path;
+import org.apache.iotdb.tsfile.read.expression.QueryExpression;
+
+import java.io.File;
+import java.io.IOException;
+import java.util.Arrays;
+import java.util.Iterator;
+import java.util.List;
+import java.util.stream.Collectors;
+
+public class FlinkTsFileStreamSource {
Review comment:
Could you please add some Java DOC for the class?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-iotdb] WeiZhong94 commented on a change in pull request
#947: [IOTDB-560] add TsFileInputFormat for reading TsFiles on flink.
Posted by GitBox <gi...@apache.org>.
WeiZhong94 commented on a change in pull request #947: [IOTDB-560] add TsFileInputFormat for reading TsFiles on flink.
URL: https://github.com/apache/incubator-iotdb/pull/947#discussion_r405289563
##########
File path: example/flink/src/main/java/org/apache/iotdb/flink/FlinkTsFileStreamSource.java
##########
@@ -0,0 +1,84 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.iotdb.flink;
+
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.api.java.typeutils.RowTypeInfo;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.streaming.api.datastream.DataStreamUtils;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.types.Row;
+import org.apache.iotdb.flink.tsfile.RowRowRecordParser;
+import org.apache.iotdb.flink.tsfile.TsFileInputFormat;
+import org.apache.iotdb.tsfile.common.constant.QueryConstant;
+import org.apache.iotdb.tsfile.read.common.Path;
+import org.apache.iotdb.tsfile.read.expression.QueryExpression;
+
+import java.io.File;
+import java.io.IOException;
+import java.util.Arrays;
+import java.util.Iterator;
+import java.util.List;
+import java.util.stream.Collectors;
+
+/**
+ * The example of reading TsFile via Flink DataStream API.
+ */
+public class FlinkTsFileStreamSource {
+
+ public static void main(String[] args) throws IOException {
+ String path = "test.tsfile";
+ TsFlieUtils.writeTsFile(path);
+ new File(path).deleteOnExit();
+ String[] filedNames = {
+ QueryConstant.RESERVED_TIME,
+ "device_1.sensor_1",
+ "device_1.sensor_2",
+ "device_1.sensor_3",
+ "device_2.sensor_1",
+ "device_2.sensor_2",
+ "device_2.sensor_3"
+ };
+ TypeInformation[] typeInformations = new TypeInformation[] {
+ Types.LONG,
+ Types.LONG,
+ Types.LONG,
+ Types.LONG,
+ Types.LONG,
+ Types.LONG,
+ Types.LONG
+ };
+ List<Path> paths = Arrays.stream(filedNames)
+ .filter(s -> !s.equals(QueryConstant.RESERVED_TIME))
+ .map(Path::new)
+ .collect(Collectors.toList());
+ RowTypeInfo rowTypeInfo = new RowTypeInfo(typeInformations, filedNames);
+ QueryExpression queryExpression = QueryExpression.create(paths, null);
Review comment:
If `TsFileInputFormat` only accepts the `RowRowRecordParser`(default implementation of `RowRecordParser` which parses the `RowRecord` to `Row`) then the answer is "yes". But the `TsFileInputFormat` is designed to accept any `RowRecordParser`, which may be implemented by users or 3rd parities. If the output type of `RowRecordParser` is not a Row, this approach may not work.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-iotdb] sunjincheng121 commented on a change in pull
request #947: [IOTDB-560] add TsFileInputFormat for reading TsFiles on flink.
Posted by GitBox <gi...@apache.org>.
sunjincheng121 commented on a change in pull request #947: [IOTDB-560] add TsFileInputFormat for reading TsFiles on flink.
URL: https://github.com/apache/incubator-iotdb/pull/947#discussion_r399629934
##########
File path: flink-tsfile/src/test/java/org/apache/iotdb/flink/tool/TsFileWriteTool.java
##########
@@ -0,0 +1,234 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.flink.tool;
+
+import java.io.File;
+import org.apache.iotdb.tsfile.file.metadata.enums.TSDataType;
+import org.apache.iotdb.tsfile.file.metadata.enums.TSEncoding;
+import org.apache.iotdb.tsfile.write.TsFileWriter;
+import org.apache.iotdb.tsfile.write.record.TSRecord;
+import org.apache.iotdb.tsfile.write.record.datapoint.DataPoint;
+import org.apache.iotdb.tsfile.write.record.datapoint.FloatDataPoint;
+import org.apache.iotdb.tsfile.write.record.datapoint.IntDataPoint;
+import org.apache.iotdb.tsfile.write.schema.MeasurementSchema;
+
+/**
+ * An example of writing data to TsFile
+ */
+public class TsFileWriteTool {
Review comment:
I'm confused about your naming of utils and tools. How do you define a class as utils or tool?
If this class is a example, I think we can add a method in some utils class? What do you think?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-iotdb] sunjincheng121 commented on a change in pull
request #947: [IOTDB-560] add TsFileInputFormat for reading TsFiles on flink.
Posted by GitBox <gi...@apache.org>.
sunjincheng121 commented on a change in pull request #947: [IOTDB-560] add TsFileInputFormat for reading TsFiles on flink.
URL: https://github.com/apache/incubator-iotdb/pull/947#discussion_r399629778
##########
File path: flink-tsfile/src/main/java/org/apache/iotdb/flink/tsfile/util/TSFileConfigUtil.java
##########
@@ -0,0 +1,62 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.flink.tsfile.util;
+
+import org.apache.iotdb.tsfile.common.conf.TSFileConfig;
+import org.apache.iotdb.tsfile.common.conf.TSFileDescriptor;
+
+public class TSFileConfigUtil {
Review comment:
How are the responsibilities of `Utils` and `TSFileConfigUtil` different? Merge it into a utils or specify the name of `Utils`? And could you please add some Java DOC for the class?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-iotdb] WeiZhong94 commented on a change in pull request
#947: [IOTDB-560] add TsFileInputFormat for reading TsFiles on flink.
Posted by GitBox <gi...@apache.org>.
WeiZhong94 commented on a change in pull request #947: [IOTDB-560] add TsFileInputFormat for reading TsFiles on flink.
URL: https://github.com/apache/incubator-iotdb/pull/947#discussion_r401464448
##########
File path: flink-tsfile/README.md
##########
@@ -0,0 +1,93 @@
+<!--
+
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+-->
+# TsFile-Flink-Connector User Guide
+
+## 1. About TsFile-Flink-Connector
+
+TsFile-Flink-Connector implements the support of Flink for external data sources of Tsfile type.
+This enables users to read, write and query Tsfile by Flink via DataStream/DataSet API.
Review comment:
Currently not but it is in the implementation plan. I'll remove it and add back after we support the sink connector.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-iotdb] WeiZhong94 commented on a change in pull request
#947: [IOTDB-560] add TsFileInputFormat for reading TsFiles on flink.
Posted by GitBox <gi...@apache.org>.
WeiZhong94 commented on a change in pull request #947: [IOTDB-560] add TsFileInputFormat for reading TsFiles on flink.
URL: https://github.com/apache/incubator-iotdb/pull/947#discussion_r401335445
##########
File path: flink-tsfile/src/main/java/org/apache/iotdb/flink/tsfile/util/TSFileConfigUtil.java
##########
@@ -0,0 +1,65 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.flink.tsfile.util;
+
+import org.apache.iotdb.tsfile.common.conf.TSFileConfig;
+import org.apache.iotdb.tsfile.common.conf.TSFileDescriptor;
+
+/**
+ * Utils of setting global TSFileConfig.
+ */
+public class TSFileConfigUtil {
+
+ public static void setGlobalTSFileConfig(TSFileConfig config) {
+ TSFileConfig globalConfig = TSFileDescriptor.getInstance().getConfig();
Review comment:
Yes this seems not a good solution.. But users could not set the TSFileConfig of TsFile flink connector directly because the connector is running on flink cluster, which may be on other machines. So in our design the connector object (i.e. `TsFileInputFormat`) could store a TSFileConfig object, which will be serialized and upload to remote flink cluster and set as the global TSFileConfig when the connector is initializing on the remote flink cluster.
We certainly have other solutions, e.g. expose a setter on `TsFileDescriptor` or just using reflection. I think this needs to be discussed so keep the original implementation when creating this PR.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-iotdb] jixuan1989 commented on issue #947: [IOTDB-560]
add TsFileInputFormat for reading TsFiles on flink.
Posted by GitBox <gi...@apache.org>.
jixuan1989 commented on issue #947: [IOTDB-560] add TsFileInputFormat for reading TsFiles on flink.
URL: https://github.com/apache/incubator-iotdb/pull/947#issuecomment-605993026
1. For homology (we have a module called flink-iotdb-connector), I think we can rename the folder `flink-tsfile` to `flink-tsfile-connector` or `flink-tsfile-source`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-iotdb] sunjincheng121 commented on a change in pull
request #947: [IOTDB-560] add TsFileInputFormat for reading TsFiles on flink.
Posted by GitBox <gi...@apache.org>.
sunjincheng121 commented on a change in pull request #947: [IOTDB-560] add TsFileInputFormat for reading TsFiles on flink.
URL: https://github.com/apache/incubator-iotdb/pull/947#discussion_r399629425
##########
File path: example/flink/src/main/java/org/apache/iotdb/flink/Utils.java
##########
@@ -0,0 +1,69 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.iotdb.flink;
+
+import org.apache.iotdb.tsfile.file.metadata.enums.TSDataType;
+import org.apache.iotdb.tsfile.file.metadata.enums.TSEncoding;
+import org.apache.iotdb.tsfile.fileSystem.FSFactoryProducer;
+import org.apache.iotdb.tsfile.write.TsFileWriter;
+import org.apache.iotdb.tsfile.write.record.TSRecord;
+import org.apache.iotdb.tsfile.write.record.datapoint.DataPoint;
+import org.apache.iotdb.tsfile.write.record.datapoint.LongDataPoint;
+import org.apache.iotdb.tsfile.write.schema.MeasurementSchema;
+
+import java.io.File;
+
+public class Utils {
Review comment:
TsFlieUtils?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services