You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@seatunnel.apache.org by GitBox <gi...@apache.org> on 2022/10/28 10:36:46 UTC
[GitHub] [incubator-seatunnel] Bingz2 opened a new pull request, #2585: [Feature][ConnectorV2]add file excel sink
Bingz2 opened a new pull request, #2585:
URL: https://github.com/apache/incubator-seatunnel/pull/2585
<!--
Thank you for contributing to SeaTunnel! Please make sure that your code changes
are covered with tests. And in case of new features or big changes
remember to adjust the documentation.
Feel free to ping committers for the review!
## Contribution Checklist
- Make sure that the pull request corresponds to a [GITHUB issue](https://github.com/apache/incubator-seatunnel/issues).
- Name the pull request in the form "[Feature] [component] Title of the pull request", where *Feature* can be replaced by `Hotfix`, `Bug`, etc.
- Minor fixes should be named following this pattern: `[hotfix] [docs] Fix typo in README.md doc`.
-->
## Purpose of this pull request
add [File]excel sink
https://github.com/apache/incubator-seatunnel/issues/1946
<!-- Describe the purpose of this pull request. For example: This pull request adds checkstyle plugin.-->
## Check list
* [x] Code changed are covered with tests, or it does not need tests for reason:
* [ ] If any new Jar binary package adding in your PR, please add License Notice according
[New License Guide](https://github.com/apache/incubator-seatunnel/blob/dev/docs/en/contribution/new-license.md)
* [ ] If necessary, please update the documentation to describe the new feature. https://github.com/apache/incubator-seatunnel/tree/dev/docs
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] Bingz2 commented on pull request #2585: [Feature][ConnectorV2]add file excel sink
Posted by GitBox <gi...@apache.org>.
Bingz2 commented on PR #2585:
URL: https://github.com/apache/incubator-seatunnel/pull/2585#issuecomment-1237013761
> > > > > 
> > > > > You e2e test cases also have some problems, please check.
> > > >
> > > >
> > > >  If I add this code, Excel sink Write's `finishAndCloseWriteFile` method will be called twice when executed with Flink, resulting in an error!
> > > > However, if you remove this code and run it with Spark, Text sink Write is executed instead of Excel sink Write
> > >
> > >
> > > Can you provide the detailed call stack information?
> >
> >
> >  
> > `SparkDataWriter's` `commit()` method clears `commitInfo` after execution, However, the `prepareCommit` of `FlinkSinkWrite` does not clear commitInfo, so it writes `ExcelWorkBook` when it calls the close() method of `FlinkSlinkWrite`, but it does the WorkBook when it executes `prepareCommit` It has been closed. I don't understand why SparkDataWriter and FlinkSinkWrite have different commit logic. Do I need to determine whether the workbook is close in `finishAndCloseWriteFile`?
>
> The reason already in comment: `combine the prepareCommit and commit in this method.`. Flink Support Committer and GlobalCommitter, but spark only support GlobalCommitter(same logic different name). So spark use commit() to run prepareCommit() and Committer.commit().
ok,My local E2E and compile passed, but CI/CD failed and I didn't use ES.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] hailin0 commented on a diff in pull request #2585: [Feature][ConnectorV2]add file excel sink
Posted by GitBox <gi...@apache.org>.
hailin0 commented on code in PR #2585:
URL: https://github.com/apache/incubator-seatunnel/pull/2585#discussion_r964779699
##########
seatunnel-connectors-v2/connector-file/connector-file-base/src/main/java/org/apache/seatunnel/connectors/seatunnel/file/sink/util/ExcelGenerator.java:
##########
@@ -0,0 +1,137 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.seatunnel.connectors.seatunnel.file.sink.util;
+
+import org.apache.seatunnel.api.table.type.BasicType;
+import org.apache.seatunnel.api.table.type.SeaTunnelDataType;
+import org.apache.seatunnel.api.table.type.SeaTunnelRow;
+import org.apache.seatunnel.api.table.type.SeaTunnelRowType;
+import org.apache.seatunnel.api.table.type.SqlType;
+import org.apache.seatunnel.common.utils.JsonUtils;
+
+import org.apache.poi.ss.usermodel.Cell;
+import org.apache.poi.ss.usermodel.CellStyle;
+import org.apache.poi.ss.usermodel.CreationHelper;
+import org.apache.poi.ss.usermodel.Row;
+import org.apache.poi.ss.usermodel.Sheet;
+import org.apache.poi.ss.usermodel.Workbook;
+import org.apache.poi.xssf.usermodel.XSSFWorkbook;
+
+import java.io.IOException;
+import java.io.OutputStream;
+import java.lang.reflect.Array;
+import java.util.ArrayList;
+import java.util.List;
+
+public class ExcelGenerator {
+ private Workbook wb;
+ private CellStyle wholeNumberCellStyle;
+ private CellStyle stringCellStyle;
+ private int row = 0;
+ private List<Integer> sinkColumnsIndexInRow;
+ private SeaTunnelRowType seaTunnelRowType;
+ private Sheet st;
+
+ public ExcelGenerator(List<Integer> sinkColumnsIndexInRow, SeaTunnelRowType seaTunnelRowType) {
+ this.sinkColumnsIndexInRow = sinkColumnsIndexInRow;
+ this.seaTunnelRowType = seaTunnelRowType;
+ wb = new XSSFWorkbook();
+ this.st = wb.createSheet("Sheet1");
+ Row row = st.createRow(this.row);
+ for (Integer i : sinkColumnsIndexInRow) {
+ String fieldName = seaTunnelRowType.getFieldName(i);
+ row.createCell(i).setCellValue(fieldName);
+ }
+
+ wholeNumberCellStyle = createStyle(wb, "General");
+ stringCellStyle = createStyle(wb, "@");
+ this.row += 1;
+ }
+
+ public void writeData(SeaTunnelRow seaTunnelRow) {
+ Row excelRow = this.st.createRow(this.row);
+ SeaTunnelDataType<?>[] fieldTypes = seaTunnelRowType.getFieldTypes();
+ for (Integer i : sinkColumnsIndexInRow) {
+ Cell cell = excelRow.createCell(i);
+ Object value = seaTunnelRow.getField(i);
+ makeConverter(fieldTypes[i], value, cell);
+ }
+ this.row += 1;
+ }
+
+ public void flushAndCloseExcel(OutputStream output) throws IOException {
+ if (wb != null) {
+ wb.write(output);
+ wb.close();
+ wb = null;
+ }
+ }
+
+ private void makeConverter(SeaTunnelDataType<?> type, Object value, Cell cell) {
+ if (value == null) {
+ cell.setBlank();
+ } else if (BasicType.STRING_TYPE.equals(type)) {
+ cell.setCellValue((String) value);
+ cell.setCellStyle(stringCellStyle);
+ } else if (BasicType.BOOLEAN_TYPE.equals(type)) {
+ cell.setCellValue((Boolean) value);
+ cell.setCellStyle(wholeNumberCellStyle);
+ } else if (BasicType.BYTE_TYPE.equals(type)) {
+ cell.setCellValue((byte) value);
+ cell.setCellStyle(wholeNumberCellStyle);
+ } else if (BasicType.SHORT_TYPE.equals(type)) {
+ cell.setCellValue((short) value);
+ cell.setCellStyle(wholeNumberCellStyle);
+ } else if (BasicType.INT_TYPE.equals(type)) {
+ cell.setCellValue((int) value);
+ cell.setCellStyle(wholeNumberCellStyle);
+ } else if (BasicType.LONG_TYPE.equals(type) || type.getSqlType().equals(SqlType.TIMESTAMP)) {
+ cell.setCellValue((long) value);
Review Comment:
`value` type is LocalDateTime?
##########
seatunnel-connectors-v2/connector-file/connector-file-base/src/main/java/org/apache/seatunnel/connectors/seatunnel/file/sink/util/ExcelGenerator.java:
##########
@@ -0,0 +1,137 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.seatunnel.connectors.seatunnel.file.sink.util;
+
+import org.apache.seatunnel.api.table.type.BasicType;
+import org.apache.seatunnel.api.table.type.SeaTunnelDataType;
+import org.apache.seatunnel.api.table.type.SeaTunnelRow;
+import org.apache.seatunnel.api.table.type.SeaTunnelRowType;
+import org.apache.seatunnel.api.table.type.SqlType;
+import org.apache.seatunnel.common.utils.JsonUtils;
+
+import org.apache.poi.ss.usermodel.Cell;
+import org.apache.poi.ss.usermodel.CellStyle;
+import org.apache.poi.ss.usermodel.CreationHelper;
+import org.apache.poi.ss.usermodel.Row;
+import org.apache.poi.ss.usermodel.Sheet;
+import org.apache.poi.ss.usermodel.Workbook;
+import org.apache.poi.xssf.usermodel.XSSFWorkbook;
+
+import java.io.IOException;
+import java.io.OutputStream;
+import java.lang.reflect.Array;
+import java.util.ArrayList;
+import java.util.List;
+
+public class ExcelGenerator {
+ private Workbook wb;
+ private CellStyle wholeNumberCellStyle;
+ private CellStyle stringCellStyle;
+ private int row = 0;
+ private List<Integer> sinkColumnsIndexInRow;
+ private SeaTunnelRowType seaTunnelRowType;
+ private Sheet st;
+
+ public ExcelGenerator(List<Integer> sinkColumnsIndexInRow, SeaTunnelRowType seaTunnelRowType) {
+ this.sinkColumnsIndexInRow = sinkColumnsIndexInRow;
+ this.seaTunnelRowType = seaTunnelRowType;
+ wb = new XSSFWorkbook();
+ this.st = wb.createSheet("Sheet1");
+ Row row = st.createRow(this.row);
+ for (Integer i : sinkColumnsIndexInRow) {
+ String fieldName = seaTunnelRowType.getFieldName(i);
+ row.createCell(i).setCellValue(fieldName);
+ }
+
+ wholeNumberCellStyle = createStyle(wb, "General");
+ stringCellStyle = createStyle(wb, "@");
+ this.row += 1;
+ }
+
+ public void writeData(SeaTunnelRow seaTunnelRow) {
+ Row excelRow = this.st.createRow(this.row);
+ SeaTunnelDataType<?>[] fieldTypes = seaTunnelRowType.getFieldTypes();
+ for (Integer i : sinkColumnsIndexInRow) {
+ Cell cell = excelRow.createCell(i);
+ Object value = seaTunnelRow.getField(i);
+ makeConverter(fieldTypes[i], value, cell);
+ }
+ this.row += 1;
+ }
+
+ public void flushAndCloseExcel(OutputStream output) throws IOException {
+ if (wb != null) {
+ wb.write(output);
+ wb.close();
+ wb = null;
+ }
+ }
+
+ private void makeConverter(SeaTunnelDataType<?> type, Object value, Cell cell) {
+ if (value == null) {
+ cell.setBlank();
+ } else if (BasicType.STRING_TYPE.equals(type)) {
+ cell.setCellValue((String) value);
+ cell.setCellStyle(stringCellStyle);
+ } else if (BasicType.BOOLEAN_TYPE.equals(type)) {
+ cell.setCellValue((Boolean) value);
+ cell.setCellStyle(wholeNumberCellStyle);
+ } else if (BasicType.BYTE_TYPE.equals(type)) {
+ cell.setCellValue((byte) value);
+ cell.setCellStyle(wholeNumberCellStyle);
+ } else if (BasicType.SHORT_TYPE.equals(type)) {
+ cell.setCellValue((short) value);
+ cell.setCellStyle(wholeNumberCellStyle);
+ } else if (BasicType.INT_TYPE.equals(type)) {
+ cell.setCellValue((int) value);
+ cell.setCellStyle(wholeNumberCellStyle);
+ } else if (BasicType.LONG_TYPE.equals(type) || type.getSqlType().equals(SqlType.TIMESTAMP)) {
+ cell.setCellValue((long) value);
+ cell.setCellStyle(wholeNumberCellStyle);
+ } else if (BasicType.FLOAT_TYPE.equals(type)) {
+ cell.setCellValue((float) value);
+ cell.setCellStyle(wholeNumberCellStyle);
+ } else if (BasicType.DOUBLE_TYPE.equals(type)) {
+ cell.setCellValue((double) value);
+ cell.setCellStyle(wholeNumberCellStyle);
+ } else if (type.getSqlType().equals(SqlType.BYTES) || type.getSqlType().equals(SqlType.ARRAY)) {
+ List<String> arrayData = new ArrayList<>();
+ for (int i = 0; i < Array.getLength(value); i++) {
+ arrayData.add(String.valueOf(Array.get(value, i)));
+ }
+ cell.setCellValue(arrayData.toString());
+ cell.setCellStyle(stringCellStyle);
+ } else if (type.getSqlType().equals(SqlType.MAP)) {
+ cell.setCellValue(JsonUtils.toJsonString(value));
+ cell.setCellStyle(stringCellStyle);
+ } else if (type.getSqlType().equals(SqlType.DATE) || type.getSqlType().equals(SqlType.TIME)) {
+ cell.setCellValue((String) value);
Review Comment:
`value` type is LocalTime or LocalDate?
##########
seatunnel-e2e/seatunnel-flink-connector-v2-e2e/src/test/resources/file/fakesource_to_hdfs_excel.conf:
##########
@@ -0,0 +1,70 @@
+#
Review Comment:
add testcase
https://github.com/apache/incubator-seatunnel/blob/dev/seatunnel-e2e/seatunnel-flink-connector-v2-e2e/connector-file-flink-e2e/src/test/java/org/apache/seatunnel/e2e/flink/v2/file/FakeSourceToFileIT.java
##########
seatunnel-connectors-v2/connector-file/connector-file-base/src/main/java/org/apache/seatunnel/connectors/seatunnel/file/sink/util/ExcelGenerator.java:
##########
@@ -0,0 +1,137 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.seatunnel.connectors.seatunnel.file.sink.util;
+
+import org.apache.seatunnel.api.table.type.BasicType;
+import org.apache.seatunnel.api.table.type.SeaTunnelDataType;
+import org.apache.seatunnel.api.table.type.SeaTunnelRow;
+import org.apache.seatunnel.api.table.type.SeaTunnelRowType;
+import org.apache.seatunnel.api.table.type.SqlType;
+import org.apache.seatunnel.common.utils.JsonUtils;
+
+import org.apache.poi.ss.usermodel.Cell;
+import org.apache.poi.ss.usermodel.CellStyle;
+import org.apache.poi.ss.usermodel.CreationHelper;
+import org.apache.poi.ss.usermodel.Row;
+import org.apache.poi.ss.usermodel.Sheet;
+import org.apache.poi.ss.usermodel.Workbook;
+import org.apache.poi.xssf.usermodel.XSSFWorkbook;
+
+import java.io.IOException;
+import java.io.OutputStream;
+import java.lang.reflect.Array;
+import java.util.ArrayList;
+import java.util.List;
+
+public class ExcelGenerator {
+ private Workbook wb;
+ private CellStyle wholeNumberCellStyle;
+ private CellStyle stringCellStyle;
+ private int row = 0;
+ private List<Integer> sinkColumnsIndexInRow;
+ private SeaTunnelRowType seaTunnelRowType;
+ private Sheet st;
+
+ public ExcelGenerator(List<Integer> sinkColumnsIndexInRow, SeaTunnelRowType seaTunnelRowType) {
+ this.sinkColumnsIndexInRow = sinkColumnsIndexInRow;
+ this.seaTunnelRowType = seaTunnelRowType;
+ wb = new XSSFWorkbook();
+ this.st = wb.createSheet("Sheet1");
+ Row row = st.createRow(this.row);
+ for (Integer i : sinkColumnsIndexInRow) {
+ String fieldName = seaTunnelRowType.getFieldName(i);
+ row.createCell(i).setCellValue(fieldName);
+ }
+
+ wholeNumberCellStyle = createStyle(wb, "General");
+ stringCellStyle = createStyle(wb, "@");
+ this.row += 1;
+ }
+
+ public void writeData(SeaTunnelRow seaTunnelRow) {
+ Row excelRow = this.st.createRow(this.row);
+ SeaTunnelDataType<?>[] fieldTypes = seaTunnelRowType.getFieldTypes();
+ for (Integer i : sinkColumnsIndexInRow) {
+ Cell cell = excelRow.createCell(i);
+ Object value = seaTunnelRow.getField(i);
+ makeConverter(fieldTypes[i], value, cell);
+ }
+ this.row += 1;
+ }
+
+ public void flushAndCloseExcel(OutputStream output) throws IOException {
+ if (wb != null) {
+ wb.write(output);
+ wb.close();
+ wb = null;
+ }
+ }
+
+ private void makeConverter(SeaTunnelDataType<?> type, Object value, Cell cell) {
+ if (value == null) {
+ cell.setBlank();
+ } else if (BasicType.STRING_TYPE.equals(type)) {
+ cell.setCellValue((String) value);
+ cell.setCellStyle(stringCellStyle);
+ } else if (BasicType.BOOLEAN_TYPE.equals(type)) {
+ cell.setCellValue((Boolean) value);
+ cell.setCellStyle(wholeNumberCellStyle);
+ } else if (BasicType.BYTE_TYPE.equals(type)) {
+ cell.setCellValue((byte) value);
+ cell.setCellStyle(wholeNumberCellStyle);
+ } else if (BasicType.SHORT_TYPE.equals(type)) {
+ cell.setCellValue((short) value);
+ cell.setCellStyle(wholeNumberCellStyle);
+ } else if (BasicType.INT_TYPE.equals(type)) {
+ cell.setCellValue((int) value);
+ cell.setCellStyle(wholeNumberCellStyle);
+ } else if (BasicType.LONG_TYPE.equals(type) || type.getSqlType().equals(SqlType.TIMESTAMP)) {
+ cell.setCellValue((long) value);
+ cell.setCellStyle(wholeNumberCellStyle);
+ } else if (BasicType.FLOAT_TYPE.equals(type)) {
+ cell.setCellValue((float) value);
+ cell.setCellStyle(wholeNumberCellStyle);
+ } else if (BasicType.DOUBLE_TYPE.equals(type)) {
+ cell.setCellValue((double) value);
+ cell.setCellStyle(wholeNumberCellStyle);
+ } else if (type.getSqlType().equals(SqlType.BYTES) || type.getSqlType().equals(SqlType.ARRAY)) {
Review Comment:
ARRAY should use json?
##########
seatunnel-e2e/seatunnel-flink-connector-v2-e2e/src/test/resources/file/fakesource_to_local_excel.conf:
##########
@@ -0,0 +1,69 @@
+#
Review Comment:
add testcase
https://github.com/apache/incubator-seatunnel/blob/dev/seatunnel-e2e/seatunnel-flink-connector-v2-e2e/connector-file-flink-e2e/src/test/java/org/apache/seatunnel/e2e/flink/v2/file/FakeSourceToFileIT.java
##########
seatunnel-e2e/seatunnel-spark-connector-v2-e2e/src/test/resources/file/fakesource_to_local_excel.conf:
##########
@@ -0,0 +1,82 @@
+#
Review Comment:
add testcase
https://github.com/apache/incubator-seatunnel/blob/dev/seatunnel-e2e/seatunnel-spark-connector-v2-e2e/connector-file-spark-e2e/src/test/java/org/apache/seatunnel/e2e/spark/v2/file/FakeSourceToFileIT.java
##########
seatunnel-e2e/seatunnel-flink-connector-v2-e2e/src/test/resources/file/fakesource_to_hdfs_excel.conf:
##########
@@ -0,0 +1,70 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+######
+###### This config file is a demonstration of streaming processing in seatunnel config
+######
+
+env {
+ # You can set flink configuration here
+ execution.parallelism = 1
+ job.mode = "BATCH"
+ #execution.checkpoint.interval = 10000
+ #execution.checkpoint.data-uri = "hdfs://localhost:9000/checkpoint"
+}
+
+source {
+ FakeSource {
+ result_table_name = "fake"
+ schema = {
+ fields {
+ name = "string"
+ age = "int"
Review Comment:
Test all datatypes?
##########
seatunnel-e2e/seatunnel-spark-connector-v2-e2e/src/test/resources/file/fakesource_to_hdfs_excel.conf:
##########
@@ -0,0 +1,82 @@
+#
Review Comment:
add testcase
https://github.com/apache/incubator-seatunnel/blob/dev/seatunnel-e2e/seatunnel-spark-connector-v2-e2e/connector-file-spark-e2e/src/test/java/org/apache/seatunnel/e2e/spark/v2/file/FakeSourceToFileIT.java
##########
seatunnel-connectors-v2/connector-file/connector-file-base/src/main/java/org/apache/seatunnel/connectors/seatunnel/file/sink/util/ExcelGenerator.java:
##########
@@ -0,0 +1,137 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.seatunnel.connectors.seatunnel.file.sink.util;
+
+import org.apache.seatunnel.api.table.type.BasicType;
+import org.apache.seatunnel.api.table.type.SeaTunnelDataType;
+import org.apache.seatunnel.api.table.type.SeaTunnelRow;
+import org.apache.seatunnel.api.table.type.SeaTunnelRowType;
+import org.apache.seatunnel.api.table.type.SqlType;
+import org.apache.seatunnel.common.utils.JsonUtils;
+
+import org.apache.poi.ss.usermodel.Cell;
+import org.apache.poi.ss.usermodel.CellStyle;
+import org.apache.poi.ss.usermodel.CreationHelper;
+import org.apache.poi.ss.usermodel.Row;
+import org.apache.poi.ss.usermodel.Sheet;
+import org.apache.poi.ss.usermodel.Workbook;
+import org.apache.poi.xssf.usermodel.XSSFWorkbook;
+
+import java.io.IOException;
+import java.io.OutputStream;
+import java.lang.reflect.Array;
+import java.util.ArrayList;
+import java.util.List;
+
+public class ExcelGenerator {
+ private Workbook wb;
+ private CellStyle wholeNumberCellStyle;
+ private CellStyle stringCellStyle;
+ private int row = 0;
+ private List<Integer> sinkColumnsIndexInRow;
+ private SeaTunnelRowType seaTunnelRowType;
+ private Sheet st;
+
+ public ExcelGenerator(List<Integer> sinkColumnsIndexInRow, SeaTunnelRowType seaTunnelRowType) {
+ this.sinkColumnsIndexInRow = sinkColumnsIndexInRow;
+ this.seaTunnelRowType = seaTunnelRowType;
+ wb = new XSSFWorkbook();
Review Comment:
`XSSFWorkbook` store data to memory, is this acceptable? you consider using SXSSFWorkbook? SXSSFWorkbook supports streaming data writing
@TyrantLucifer @Hisoka-X @CalvinKirs @EricJoy2048
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] EricJoy2048 closed pull request #2585: [Feature][ConnectorV2]add file excel sink
Posted by GitBox <gi...@apache.org>.
EricJoy2048 closed pull request #2585: [Feature][ConnectorV2]add file excel sink
URL: https://github.com/apache/incubator-seatunnel/pull/2585
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] EricJoy2048 closed pull request #2585: [Feature][ConnectorV2]add file excel sink
Posted by GitBox <gi...@apache.org>.
EricJoy2048 closed pull request #2585: [Feature][ConnectorV2]add file excel sink
URL: https://github.com/apache/incubator-seatunnel/pull/2585
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] Bingz2 commented on pull request #2585: [Feature][ConnectorV2]add file excel sink
Posted by GitBox <gi...@apache.org>.
Bingz2 commented on PR #2585:
URL: https://github.com/apache/incubator-seatunnel/pull/2585#issuecomment-1236080807
> > > 
> > > You e2e test cases also have some problems, please check.
> >
> >
> >  If I add this code, Excel sink Write's `finishAndCloseWriteFile` method will be called twice when executed with Flink, resulting in an error!
> > However, if you remove this code and run it with Spark, Text sink Write is executed instead of Excel sink Write
>
> Can you provide the detailed call stack information?
`SparkDataWriter's` `commit()` method clears `commitInfo` after execution, However, the `prepareCommit` of `FlinkSinkWrite` does not clear commitInfo, so it writes `ExcelWorkBook` when it calls the close() method of `FlinkSlinkWrite`, but it does the WorkBook when it executes `prepareCommit` It has been closed.
I don't understand why SparkDataWriter and FlinkSinkWrite have different commit logic. Do I need to determine whether the workbook is close in `finishAndCloseWriteFile`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] Bingz2 commented on a diff in pull request #2585: [Feature][ConnectorV2]add file excel sink
Posted by GitBox <gi...@apache.org>.
Bingz2 commented on code in PR #2585:
URL: https://github.com/apache/incubator-seatunnel/pull/2585#discussion_r961240466
##########
seatunnel-connectors-v2/connector-file/connector-file-base/src/main/java/org/apache/seatunnel/connectors/seatunnel/file/sink/util/ExcelGenerator.java:
##########
@@ -0,0 +1,134 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.seatunnel.connectors.seatunnel.file.sink.util;
+
+import org.apache.seatunnel.api.table.type.BasicType;
+import org.apache.seatunnel.api.table.type.SeaTunnelDataType;
+import org.apache.seatunnel.api.table.type.SeaTunnelRow;
+import org.apache.seatunnel.api.table.type.SeaTunnelRowType;
+import org.apache.seatunnel.api.table.type.SqlType;
+import org.apache.seatunnel.common.utils.JsonUtils;
+
+import org.apache.poi.ss.usermodel.Cell;
+import org.apache.poi.ss.usermodel.CellStyle;
+import org.apache.poi.ss.usermodel.CreationHelper;
+import org.apache.poi.ss.usermodel.Row;
+import org.apache.poi.ss.usermodel.Sheet;
+import org.apache.poi.ss.usermodel.Workbook;
+import org.apache.poi.xssf.usermodel.XSSFWorkbook;
+
+import java.io.IOException;
+import java.io.OutputStream;
+import java.lang.reflect.Array;
+import java.util.ArrayList;
+import java.util.List;
+
+public class ExcelGenerator {
+ private Workbook wb;
+ private CellStyle wholeNumberCellStyle;
+ private CellStyle stringCellStyle;
+ private int row = 0;
+ private List<Integer> sinkColumnsIndexInRow;
+ private SeaTunnelRowType seaTunnelRowType;
+
+ public ExcelGenerator(List<Integer> sinkColumnsIndexInRow, SeaTunnelRowType seaTunnelRowType) {
+ this.sinkColumnsIndexInRow = sinkColumnsIndexInRow;
+ this.seaTunnelRowType = seaTunnelRowType;
+ wb = new XSSFWorkbook();
+ Sheet st = wb.createSheet("Sheet1");
+ Row row = st.createRow(this.row);
+ for (Integer i : sinkColumnsIndexInRow) {
+ String fieldName = seaTunnelRowType.getFieldName(i);
+ row.createCell(i).setCellValue(fieldName);
+ }
+
+ wholeNumberCellStyle = createStyle(wb, "General");
+ stringCellStyle = createStyle(wb, "@");
+ this.row += 1;
+ }
+
+ public void writeData(SeaTunnelRow seaTunnelRow) {
+ Sheet st = wb.getSheet("Sheet1");
Review Comment:
oh ,Let me modify that
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] hailin0 commented on a diff in pull request #2585: [Feature][ConnectorV2]add file excel sink
Posted by GitBox <gi...@apache.org>.
hailin0 commented on code in PR #2585:
URL: https://github.com/apache/incubator-seatunnel/pull/2585#discussion_r960165564
##########
seatunnel-e2e/seatunnel-spark-connector-v2-e2e/src/test/resources/file/fakesource_to_hdfs_excel.conf:
##########
@@ -0,0 +1,68 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+env {
+ # You can set spark configuration here
+ spark.app.name = "SeaTunnel"
+ spark.executor.instances = 2
+ spark.executor.cores = 1
+ spark.executor.memory = "1g"
+ spark.master = local
+ job.mode = "BATCH"
+}
+
+source {
+ FakeSource {
+ result_table_name = "fake"
+ schema = {
+ fields {
+ name = "string"
+ age = "int"
Review Comment:
Test all datatypes？
##########
seatunnel-e2e/seatunnel-spark-connector-v2-e2e/src/test/java/org/apache/seatunnel/e2e/spark/v2/file/FakeSourceToFileIT.java:
##########
@@ -85,4 +94,13 @@ public void testFakeSourceToHdfsFileJson() throws IOException, InterruptedExcept
Container.ExecResult execResult = executeSeaTunnelSparkJob("/file/fakesource_to_hdfs_json.conf");
Assertions.assertEquals(0, execResult.getExitCode());
}
+
+ /**
+ * fake source -> hdfs excel file sink
+ */
+ @Test
+ public void testFakeSourceToHdfsFileExcel() throws IOException, InterruptedException {
+ Container.ExecResult execResult = executeSeaTunnelSparkJob("/file/fakesource_to_hdfs_excel.conf");
+ Assertions.assertEquals(0, execResult.getExitCode());
Review Comment:
validate sink data row & datatypes？
##########
seatunnel-e2e/seatunnel-flink-connector-v2-e2e/src/test/java/org/apache/seatunnel/e2e/flink/v2/file/FakeSourceToFileIT.java:
##########
@@ -80,4 +89,13 @@ public void testFakeSourceToHdfsFileJson() throws IOException, InterruptedExcept
Container.ExecResult execResult = executeSeaTunnelFlinkJob("/file/fakesource_to_hdfs_json.conf");
Assertions.assertEquals(0, execResult.getExitCode());
}
+
+ /**
+ * fake source -> hdfs excel file sink
+ */
+ @Test
+ public void testFakeSourceToHdfsFileExcel() throws IOException, InterruptedException {
+ Container.ExecResult execResult = executeSeaTunnelFlinkJob("/file/fakesource_to_hdfs_excel.conf");
+ Assertions.assertEquals(0, execResult.getExitCode());
Review Comment:
uncommited `fakesource_to_local_excel.conf`、`fakesource_to_hdfs_excel.conf`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] Bingz2 commented on a diff in pull request #2585: [Feature][ConnectorV2]add file excel sink
Posted by GitBox <gi...@apache.org>.
Bingz2 commented on code in PR #2585:
URL: https://github.com/apache/incubator-seatunnel/pull/2585#discussion_r960186950
##########
seatunnel-e2e/seatunnel-spark-connector-v2-e2e/src/test/resources/file/fakesource_to_hdfs_excel.conf:
##########
@@ -0,0 +1,68 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+env {
+ # You can set spark configuration here
+ spark.app.name = "SeaTunnel"
+ spark.executor.instances = 2
+ spark.executor.cores = 1
+ spark.executor.memory = "1g"
+ spark.master = local
+ job.mode = "BATCH"
+}
+
+source {
+ FakeSource {
+ result_table_name = "fake"
+ schema = {
+ fields {
+ name = "string"
+ age = "int"
Review Comment:
test `string,boolean,tinyint,smallint,int,bigint,float,double,null` is ok
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] Bingz2 commented on a diff in pull request #2585: [Feature][ConnectorV2]add file excel sink
Posted by GitBox <gi...@apache.org>.
Bingz2 commented on code in PR #2585:
URL: https://github.com/apache/incubator-seatunnel/pull/2585#discussion_r979350754
##########
seatunnel-connectors-v2/connector-file/connector-file-base/src/main/java/org/apache/seatunnel/connectors/seatunnel/file/sink/util/ExcelGenerator.java:
##########
@@ -0,0 +1,137 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.seatunnel.connectors.seatunnel.file.sink.util;
+
+import org.apache.seatunnel.api.table.type.BasicType;
+import org.apache.seatunnel.api.table.type.SeaTunnelDataType;
+import org.apache.seatunnel.api.table.type.SeaTunnelRow;
+import org.apache.seatunnel.api.table.type.SeaTunnelRowType;
+import org.apache.seatunnel.api.table.type.SqlType;
+import org.apache.seatunnel.common.utils.JsonUtils;
+
+import org.apache.poi.ss.usermodel.Cell;
+import org.apache.poi.ss.usermodel.CellStyle;
+import org.apache.poi.ss.usermodel.CreationHelper;
+import org.apache.poi.ss.usermodel.Row;
+import org.apache.poi.ss.usermodel.Sheet;
+import org.apache.poi.ss.usermodel.Workbook;
+import org.apache.poi.xssf.usermodel.XSSFWorkbook;
+
+import java.io.IOException;
+import java.io.OutputStream;
+import java.lang.reflect.Array;
+import java.util.ArrayList;
+import java.util.List;
+
+public class ExcelGenerator {
+ private Workbook wb;
+ private CellStyle wholeNumberCellStyle;
+ private CellStyle stringCellStyle;
+ private int row = 0;
+ private List<Integer> sinkColumnsIndexInRow;
+ private SeaTunnelRowType seaTunnelRowType;
+ private Sheet st;
+
+ public ExcelGenerator(List<Integer> sinkColumnsIndexInRow, SeaTunnelRowType seaTunnelRowType) {
+ this.sinkColumnsIndexInRow = sinkColumnsIndexInRow;
+ this.seaTunnelRowType = seaTunnelRowType;
+ wb = new XSSFWorkbook();
Review Comment:
When using `SXSSFWorkbook` to create a sheet,local operation is normal, but E2E will report an error.
using `openjdk` will report a null pointer exception because there is no font.
reference:
https://blog.csdn.net/progammer10086/article/details/107154814?spm=1001.2101.3001.6650.3&utm_medium=distribute.pc_relevant.none-task-blog-2%7Edefault%7ECTRLIST%7ERate-3-107154814-blog-116048731.t0_edu_mlt&depth_1-utm_source=distribute.pc_relevant.none-task-blog-2%7Edefault%7ECTRLIST%7ERate-3-107154814-blog-116048731.t0_edu_mlt&utm_relevant_index=6
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] EricJoy2048 closed pull request #2585: [Feature][ConnectorV2]add file excel sink
Posted by GitBox <gi...@apache.org>.
EricJoy2048 closed pull request #2585: [Feature][ConnectorV2]add file excel sink
URL: https://github.com/apache/incubator-seatunnel/pull/2585
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] TyrantLucifer commented on pull request #2585: [Feature][ConnectorV2]add file excel sink
Posted by GitBox <gi...@apache.org>.
TyrantLucifer commented on PR #2585:
URL: https://github.com/apache/incubator-seatunnel/pull/2585#issuecomment-1235098463
You e2e test cases also have some problems, please check.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] Hisoka-X commented on pull request #2585: [Feature][ConnectorV2]add file excel sink
Posted by GitBox <gi...@apache.org>.
Hisoka-X commented on PR #2585:
URL: https://github.com/apache/incubator-seatunnel/pull/2585#issuecomment-1257413783
Your e2e not passed, please solve the problem. Thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] hailin0 commented on a diff in pull request #2585: [Feature][ConnectorV2]add file excel sink
Posted by GitBox <gi...@apache.org>.
hailin0 commented on code in PR #2585:
URL: https://github.com/apache/incubator-seatunnel/pull/2585#discussion_r964832346
##########
seatunnel-connectors-v2/connector-file/connector-file-base/src/main/java/org/apache/seatunnel/connectors/seatunnel/file/sink/util/ExcelGenerator.java:
##########
@@ -0,0 +1,137 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.seatunnel.connectors.seatunnel.file.sink.util;
+
+import org.apache.seatunnel.api.table.type.BasicType;
+import org.apache.seatunnel.api.table.type.SeaTunnelDataType;
+import org.apache.seatunnel.api.table.type.SeaTunnelRow;
+import org.apache.seatunnel.api.table.type.SeaTunnelRowType;
+import org.apache.seatunnel.api.table.type.SqlType;
+import org.apache.seatunnel.common.utils.JsonUtils;
+
+import org.apache.poi.ss.usermodel.Cell;
+import org.apache.poi.ss.usermodel.CellStyle;
+import org.apache.poi.ss.usermodel.CreationHelper;
+import org.apache.poi.ss.usermodel.Row;
+import org.apache.poi.ss.usermodel.Sheet;
+import org.apache.poi.ss.usermodel.Workbook;
+import org.apache.poi.xssf.usermodel.XSSFWorkbook;
+
+import java.io.IOException;
+import java.io.OutputStream;
+import java.lang.reflect.Array;
+import java.util.ArrayList;
+import java.util.List;
+
+public class ExcelGenerator {
+ private Workbook wb;
+ private CellStyle wholeNumberCellStyle;
+ private CellStyle stringCellStyle;
+ private int row = 0;
+ private List<Integer> sinkColumnsIndexInRow;
+ private SeaTunnelRowType seaTunnelRowType;
+ private Sheet st;
+
+ public ExcelGenerator(List<Integer> sinkColumnsIndexInRow, SeaTunnelRowType seaTunnelRowType) {
+ this.sinkColumnsIndexInRow = sinkColumnsIndexInRow;
+ this.seaTunnelRowType = seaTunnelRowType;
+ wb = new XSSFWorkbook();
Review Comment:
`XSSFWorkbook` store data to memory, is this acceptable? you consider using `SXSSFWorkbook`? `SXSSFWorkbook` supports streaming data writing
@TyrantLucifer @Hisoka-X @CalvinKirs @EricJoy2048
reference
https://github.com/apache/poi/blob/trunk/poi-ooxml/src/main/java/org/apache/poi/xssf/streaming/SXSSFWorkbook.java
https://github.com/apache/poi/blob/trunk/poi-ooxml/src/main/java/org/apache/poi/xssf/usermodel/XSSFWorkbook.java
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] EricJoy2048 commented on pull request #2585: [Feature][ConnectorV2]add file excel sink
Posted by GitBox <gi...@apache.org>.
EricJoy2048 commented on PR #2585:
URL: https://github.com/apache/incubator-seatunnel/pull/2585#issuecomment-1279981594
@TyrantLucifer PTAL
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] hailin0 commented on a diff in pull request #2585: [Feature][ConnectorV2]add file excel sink
Posted by GitBox <gi...@apache.org>.
hailin0 commented on code in PR #2585:
URL: https://github.com/apache/incubator-seatunnel/pull/2585#discussion_r964784889
##########
seatunnel-e2e/seatunnel-flink-connector-v2-e2e/src/test/resources/file/fakesource_to_hdfs_excel.conf:
##########
@@ -0,0 +1,70 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+######
+###### This config file is a demonstration of streaming processing in seatunnel config
+######
+
+env {
+ # You can set flink configuration here
+ execution.parallelism = 1
+ job.mode = "BATCH"
+ #execution.checkpoint.interval = 10000
+ #execution.checkpoint.data-uri = "hdfs://localhost:9000/checkpoint"
+}
+
+source {
+ FakeSource {
+ result_table_name = "fake"
+ schema = {
+ fields {
+ name = "string"
+ age = "int"
Review Comment:
Test all datatypes?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] Bingz2 commented on a diff in pull request #2585: [Feature][ConnectorV2]add file excel sink
Posted by GitBox <gi...@apache.org>.
Bingz2 commented on code in PR #2585:
URL: https://github.com/apache/incubator-seatunnel/pull/2585#discussion_r1024822062
##########
seatunnel-connectors-v2/connector-file/connector-file-base/src/main/java/org/apache/seatunnel/connectors/seatunnel/file/sink/util/ExcelGenerator.java:
##########
@@ -0,0 +1,250 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.seatunnel.connectors.seatunnel.file.sink.util;
+
+import org.apache.seatunnel.api.table.type.SeaTunnelDataType;
+import org.apache.seatunnel.api.table.type.SeaTunnelRow;
+import org.apache.seatunnel.api.table.type.SeaTunnelRowType;
+import org.apache.seatunnel.common.utils.DateTimeUtils;
+import org.apache.seatunnel.common.utils.DateUtils;
+import org.apache.seatunnel.common.utils.JsonUtils;
+import org.apache.seatunnel.common.utils.TimeUtils;
+import org.apache.seatunnel.connectors.seatunnel.file.sink.config.TextFileSinkConfig;
+
+import org.apache.poi.ss.usermodel.Cell;
+import org.apache.poi.ss.usermodel.CellStyle;
+import org.apache.poi.ss.usermodel.CreationHelper;
+import org.apache.poi.ss.usermodel.Row;
+import org.apache.poi.ss.usermodel.Sheet;
+import org.apache.poi.ss.usermodel.Workbook;
+import org.apache.poi.xssf.streaming.SXSSFWorkbook;
+import org.apache.poi.xssf.usermodel.XSSFWorkbook;
+
+import java.io.IOException;
+import java.io.OutputStream;
+import java.lang.reflect.Array;
+import java.sql.Timestamp;
+import java.time.LocalDate;
+import java.time.LocalDateTime;
+import java.time.LocalTime;
+import java.util.ArrayList;
+import java.util.List;
+
+public class ExcelGenerator {
+ private List<Integer> sinkColumnsIndexInRow;
+ private SeaTunnelRowType seaTunnelRowType;
+ private final DateUtils.Formatter dateFormat;
+ private final DateTimeUtils.Formatter dateTimeFormat;
+ private final TimeUtils.Formatter timeFormat;
+ private final String fieldDelimiter;
+ private Workbook wb;
+ private CellStyle wholeNumberCellStyle;
+ private CellStyle stringCellStyle;
+ private CellStyle dateCellStyle;
+ private CellStyle dateTimeCellStyle;
+ private CellStyle timeCellStyle;
+ private Sheet st;
+ private int row = 0;
+ public ExcelGenerator(List<Integer> sinkColumnsIndexInRow, SeaTunnelRowType seaTunnelRowType, TextFileSinkConfig textFileSinkConfig) {
+ this.sinkColumnsIndexInRow = sinkColumnsIndexInRow;
+ this.seaTunnelRowType = seaTunnelRowType;
+ if (textFileSinkConfig.getMaxRowsInMemory() > 0) {
+ wb = new SXSSFWorkbook(textFileSinkConfig.getMaxRowsInMemory());
+ } else {
+ wb = new XSSFWorkbook();
Review Comment:
ok, I'm going to set a default
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] Hisoka-X commented on pull request #2585: [Feature][ConnectorV2]add file excel sink
Posted by GitBox <gi...@apache.org>.
Hisoka-X commented on PR #2585:
URL: https://github.com/apache/incubator-seatunnel/pull/2585#issuecomment-1236679797
> > > > 
> > > > You e2e test cases also have some problems, please check.
> > >
> > >
> > >  If I add this code, Excel sink Write's `finishAndCloseWriteFile` method will be called twice when executed with Flink, resulting in an error!
> > > However, if you remove this code and run it with Spark, Text sink Write is executed instead of Excel sink Write
> >
> >
> > Can you provide the detailed call stack information?
>
>  
>
> `SparkDataWriter's` `commit()` method clears `commitInfo` after execution, However, the `prepareCommit` of `FlinkSinkWrite` does not clear commitInfo, so it writes `ExcelWorkBook` when it calls the close() method of `FlinkSlinkWrite`, but it does the WorkBook when it executes `prepareCommit` It has been closed. I don't understand why SparkDataWriter and FlinkSinkWrite have different commit logic. Do I need to determine whether the workbook is close in `finishAndCloseWriteFile`?
The reason already in comment: `combine the prepareCommit and commit in this method.`. Flink Support Committer and GlobalCommitter, but spark only support GlobalCommitter(same logic different name). So spark use commit() to run prepareCommit() and Committer.commit().
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] Bingz2 commented on a diff in pull request #2585: [Feature][ConnectorV2]add file excel sink
Posted by GitBox <gi...@apache.org>.
Bingz2 commented on code in PR #2585:
URL: https://github.com/apache/incubator-seatunnel/pull/2585#discussion_r963327124
##########
pom.xml:
##########
@@ -758,6 +760,19 @@
<artifactId>aliyun-sdk-datahub</artifactId>
<version>${datahub.version}</version>
</dependency>
+
+ <dependency>
Review Comment:
Ok, they have been moved to connector's pom.xml
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] TyrantLucifer commented on a diff in pull request #2585: [Feature][ConnectorV2]add file excel sink
Posted by GitBox <gi...@apache.org>.
TyrantLucifer commented on code in PR #2585:
URL: https://github.com/apache/incubator-seatunnel/pull/2585#discussion_r973685212
##########
seatunnel-connectors-v2/connector-file/connector-file-base/src/main/java/org/apache/seatunnel/connectors/seatunnel/file/sink/util/ExcelGenerator.java:
##########
@@ -0,0 +1,137 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.seatunnel.connectors.seatunnel.file.sink.util;
+
+import org.apache.seatunnel.api.table.type.BasicType;
+import org.apache.seatunnel.api.table.type.SeaTunnelDataType;
+import org.apache.seatunnel.api.table.type.SeaTunnelRow;
+import org.apache.seatunnel.api.table.type.SeaTunnelRowType;
+import org.apache.seatunnel.api.table.type.SqlType;
+import org.apache.seatunnel.common.utils.JsonUtils;
+
+import org.apache.poi.ss.usermodel.Cell;
+import org.apache.poi.ss.usermodel.CellStyle;
+import org.apache.poi.ss.usermodel.CreationHelper;
+import org.apache.poi.ss.usermodel.Row;
+import org.apache.poi.ss.usermodel.Sheet;
+import org.apache.poi.ss.usermodel.Workbook;
+import org.apache.poi.xssf.usermodel.XSSFWorkbook;
+
+import java.io.IOException;
+import java.io.OutputStream;
+import java.lang.reflect.Array;
+import java.util.ArrayList;
+import java.util.List;
+
+public class ExcelGenerator {
+ private Workbook wb;
+ private CellStyle wholeNumberCellStyle;
+ private CellStyle stringCellStyle;
+ private int row = 0;
+ private List<Integer> sinkColumnsIndexInRow;
+ private SeaTunnelRowType seaTunnelRowType;
+ private Sheet st;
+
+ public ExcelGenerator(List<Integer> sinkColumnsIndexInRow, SeaTunnelRowType seaTunnelRowType) {
+ this.sinkColumnsIndexInRow = sinkColumnsIndexInRow;
+ this.seaTunnelRowType = seaTunnelRowType;
+ wb = new XSSFWorkbook();
Review Comment:
Agree with you.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] Bingz2 closed pull request #2585: [Feature][ConnectorV2]add file excel sink
Posted by GitBox <gi...@apache.org>.
Bingz2 closed pull request #2585: [Feature][ConnectorV2]add file excel sink
URL: https://github.com/apache/incubator-seatunnel/pull/2585
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] EricJoy2048 commented on pull request #2585: [Feature][ConnectorV2]add file excel sink
Posted by GitBox <gi...@apache.org>.
EricJoy2048 commented on PR #2585:
URL: https://github.com/apache/incubator-seatunnel/pull/2585#issuecomment-1279719745
> > Hi, @Bingz2 thanks for your contribution, what's the news about this pr? Do you need some help?
> > Sorry, I'm a little busy recently. I'll finish it this weekend
Thank you very much.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] EricJoy2048 commented on pull request #2585: [Feature][ConnectorV2]add file excel sink
Posted by GitBox <gi...@apache.org>.
EricJoy2048 commented on PR #2585:
URL: https://github.com/apache/incubator-seatunnel/pull/2585#issuecomment-1296846923
> > > > Hi, @Bingz2 thanks for your contribution, what's the news about this pr? Do you need some help?
> > > > Sorry, I'm a little busy recently. I'll finish it this weekend
> >
> >
> > Thank you very much.
>
>  An error in the CI,but I executed E2E locally with no errors,I don't know why.
Don't worry, just resolve conflicts and retry the CI, Thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] TyrantLucifer commented on a diff in pull request #2585: [Feature][ConnectorV2]add file excel sink
Posted by GitBox <gi...@apache.org>.
TyrantLucifer commented on code in PR #2585:
URL: https://github.com/apache/incubator-seatunnel/pull/2585#discussion_r961216601
##########
seatunnel-connectors-v2/connector-file/connector-file-base/src/main/java/org/apache/seatunnel/connectors/seatunnel/file/sink/util/ExcelGenerator.java:
##########
@@ -0,0 +1,134 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.seatunnel.connectors.seatunnel.file.sink.util;
+
+import org.apache.seatunnel.api.table.type.BasicType;
+import org.apache.seatunnel.api.table.type.SeaTunnelDataType;
+import org.apache.seatunnel.api.table.type.SeaTunnelRow;
+import org.apache.seatunnel.api.table.type.SeaTunnelRowType;
+import org.apache.seatunnel.api.table.type.SqlType;
+import org.apache.seatunnel.common.utils.JsonUtils;
+
+import org.apache.poi.ss.usermodel.Cell;
+import org.apache.poi.ss.usermodel.CellStyle;
+import org.apache.poi.ss.usermodel.CreationHelper;
+import org.apache.poi.ss.usermodel.Row;
+import org.apache.poi.ss.usermodel.Sheet;
+import org.apache.poi.ss.usermodel.Workbook;
+import org.apache.poi.xssf.usermodel.XSSFWorkbook;
+
+import java.io.IOException;
+import java.io.OutputStream;
+import java.lang.reflect.Array;
+import java.util.ArrayList;
+import java.util.List;
+
+public class ExcelGenerator {
+ private Workbook wb;
+ private CellStyle wholeNumberCellStyle;
+ private CellStyle stringCellStyle;
+ private int row = 0;
+ private List<Integer> sinkColumnsIndexInRow;
+ private SeaTunnelRowType seaTunnelRowType;
+
+ public ExcelGenerator(List<Integer> sinkColumnsIndexInRow, SeaTunnelRowType seaTunnelRowType) {
+ this.sinkColumnsIndexInRow = sinkColumnsIndexInRow;
+ this.seaTunnelRowType = seaTunnelRowType;
+ wb = new XSSFWorkbook();
+ Sheet st = wb.createSheet("Sheet1");
+ Row row = st.createRow(this.row);
+ for (Integer i : sinkColumnsIndexInRow) {
+ String fieldName = seaTunnelRowType.getFieldName(i);
+ row.createCell(i).setCellValue(fieldName);
+ }
+
+ wholeNumberCellStyle = createStyle(wb, "General");
+ stringCellStyle = createStyle(wb, "@");
+ this.row += 1;
+ }
+
+ public void writeData(SeaTunnelRow seaTunnelRow) {
+ Sheet st = wb.getSheet("Sheet1");
Review Comment:
How about treated `st` as a class attribute, when a SeaTunnelRow needs be consumed `createSheet` will be invoked. Of course, I have no detailed source code understanding, in each write whether to obtain a sheet object needs a double check.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] Bingz2 commented on pull request #2585: [Feature][ConnectorV2]add file excel sink
Posted by GitBox <gi...@apache.org>.
Bingz2 commented on PR #2585:
URL: https://github.com/apache/incubator-seatunnel/pull/2585#issuecomment-1235482098
> 
>
> You e2e test cases also have some problems, please check.
If I add this code, Excel sink Write's `finishAndCloseWriteFile` method will be called twice when executed with Flink, resulting in an error!
However, if you remove this code and run it with Spark, Text sink Write is executed instead of Excel sink Write
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] CalvinKirs commented on pull request #2585: [Feature][ConnectorV2]add file excel sink
Posted by GitBox <gi...@apache.org>.
CalvinKirs commented on PR #2585:
URL: https://github.com/apache/incubator-seatunnel/pull/2585#issuecomment-1233600923
@TyrantLucifer Hi, PTAL
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] EricJoy2048 closed pull request #2585: [Feature][ConnectorV2]add file excel sink
Posted by GitBox <gi...@apache.org>.
EricJoy2048 closed pull request #2585: [Feature][ConnectorV2]add file excel sink
URL: https://github.com/apache/incubator-seatunnel/pull/2585
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] EricJoy2048 commented on pull request #2585: [Feature][ConnectorV2]add file excel sink
Posted by GitBox <gi...@apache.org>.
EricJoy2048 commented on PR #2585:
URL: https://github.com/apache/incubator-seatunnel/pull/2585#issuecomment-1277057014
Hi, @Bingz2 thanks for your contribution, what's the news about this pr? Do you need some help?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] Hisoka-X commented on a diff in pull request #2585: [Feature][ConnectorV2]add file excel sink
Posted by GitBox <gi...@apache.org>.
Hisoka-X commented on code in PR #2585:
URL: https://github.com/apache/incubator-seatunnel/pull/2585#discussion_r963216648
##########
pom.xml:
##########
@@ -758,6 +760,19 @@
<artifactId>aliyun-sdk-datahub</artifactId>
<version>${datahub.version}</version>
</dependency>
+
+ <dependency>
Review Comment:
Hi, please move connector dependency to connector's pom.xml. Reference: https://github.com/apache/incubator-seatunnel/pull/2630
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] Hisoka-X commented on pull request #2585: [Feature][ConnectorV2]add file excel sink
Posted by GitBox <gi...@apache.org>.
Hisoka-X commented on PR #2585:
URL: https://github.com/apache/incubator-seatunnel/pull/2585#issuecomment-1257563600
> > Your e2e not passed, please solve the problem. Thanks
>
> When using SXSSFWorkbook to create a sheet,local operation is normal, but E2E will report an error. using openjdk will report a null pointer exception because there is no font. reference: https://blog.csdn.net/progammer10086/article/details/107154814?spm=1001.2101.3001.6650.3&utm_medium=distribute.pc_relevant.none-task-blog-2%7Edefault%7ECTRLIST%7ERate-3-107154814-blog-116048731.t0_edu_mlt&depth_1-utm_source=distribute.pc_relevant.none-task-blog-2%7Edefault%7ECTRLIST%7ERate-3-107154814-blog-116048731.t0_edu_mlt&utm_relevant_index=6
Can you install font in docker before you submit job use `executeExtraCommands`? And add doc to tell users if they use openjdk, what should them do.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] TaoZex commented on a diff in pull request #2585: [Feature][ConnectorV2]add file excel sink
Posted by GitBox <gi...@apache.org>.
TaoZex commented on code in PR #2585:
URL: https://github.com/apache/incubator-seatunnel/pull/2585#discussion_r1046757313
##########
seatunnel-e2e/seatunnel-flink-connector-v2-e2e/src/test/resources/file/fakesource_to_hdfs_excel.conf:
##########
@@ -0,0 +1,70 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+######
+###### This config file is a demonstration of streaming processing in seatunnel config
+######
+
+env {
+ # You can set flink configuration here
+ execution.parallelism = 1
+ job.mode = "BATCH"
+ #execution.checkpoint.interval = 10000
+ #execution.checkpoint.data-uri = "hdfs://localhost:9000/checkpoint"
+}
+
+source {
+ FakeSource {
+ result_table_name = "fake"
+ schema = {
+ fields {
+ name = "string"
+ age = "int"
+ }
+ }
+ }
+
+ # If you would like to get more information about how to configure seatunnel and see full list of source plugins,
+ # please go to https://seatunnel.apache.org/docs/connector-v2/source/FakeSource
+}
+
+transform {
+ sql {
+ sql = "select name,age from fake"
+ }
+
+ # If you would like to get more information about how to configure seatunnel and see full list of transform plugins,
+ # please go to https://seatunnel.apache.org/docs/transform/sql
+}
Review Comment:
Maybe you need to write dockerfile to do that.Add the required content based on this image, then generate your own image and push it to docker hub.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] Bingz2 commented on pull request #2585: [Feature][ConnectorV2]add file excel sink
Posted by GitBox <gi...@apache.org>.
Bingz2 commented on PR #2585:
URL: https://github.com/apache/incubator-seatunnel/pull/2585#issuecomment-1386543925
> @Bingz2 Will this pr go ahead? If not, I would like to take over and finish it
ok tks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] MonsterChenzhuo commented on pull request #2585: [Feature][ConnectorV2]add file excel sink
Posted by GitBox <gi...@apache.org>.
MonsterChenzhuo commented on PR #2585:
URL: https://github.com/apache/incubator-seatunnel/pull/2585#issuecomment-1385408953
@Bingz2 Will this pr go ahead? If not, I would like to take over and finish it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] TyrantLucifer commented on pull request #2585: [Feature][ConnectorV2]add file excel sink
Posted by GitBox <gi...@apache.org>.
TyrantLucifer commented on PR #2585:
URL: https://github.com/apache/incubator-seatunnel/pull/2585#issuecomment-1236074694
> > 
> > You e2e test cases also have some problems, please check.
>
>  If I add this code, Excel sink Write's `finishAndCloseWriteFile` method will be called twice when executed with Flink, resulting in an error!
>
> However, if you remove this code and run it with Spark, Text sink Write is executed instead of Excel sink Write
Can you provide the detailed call stack information?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] Bingz2 commented on pull request #2585: [Feature][ConnectorV2]add file excel sink
Posted by GitBox <gi...@apache.org>.
Bingz2 commented on PR #2585:
URL: https://github.com/apache/incubator-seatunnel/pull/2585#issuecomment-1257547462
> Your e2e not passed, please solve the problem. Thanks
When using SXSSFWorkbook to create a sheet,local operation is normal, but E2E will report an error.
using openjdk will report a null pointer exception because there is no font.
reference:
https://blog.csdn.net/progammer10086/article/details/107154814?spm=1001.2101.3001.6650.3&utm_medium=distribute.pc_relevant.none-task-blog-2%7Edefault%7ECTRLIST%7ERate-3-107154814-blog-116048731.t0_edu_mlt&depth_1-utm_source=distribute.pc_relevant.none-task-blog-2%7Edefault%7ECTRLIST%7ERate-3-107154814-blog-116048731.t0_edu_mlt&utm_relevant_index=6
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] hailin0 commented on a diff in pull request #2585: [Feature][ConnectorV2]add file excel sink
Posted by GitBox <gi...@apache.org>.
hailin0 commented on code in PR #2585:
URL: https://github.com/apache/incubator-seatunnel/pull/2585#discussion_r1024699014
##########
seatunnel-connectors-v2/connector-file/connector-file-base/src/main/java/org/apache/seatunnel/connectors/seatunnel/file/sink/util/ExcelGenerator.java:
##########
@@ -0,0 +1,250 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.seatunnel.connectors.seatunnel.file.sink.util;
+
+import org.apache.seatunnel.api.table.type.SeaTunnelDataType;
+import org.apache.seatunnel.api.table.type.SeaTunnelRow;
+import org.apache.seatunnel.api.table.type.SeaTunnelRowType;
+import org.apache.seatunnel.common.utils.DateTimeUtils;
+import org.apache.seatunnel.common.utils.DateUtils;
+import org.apache.seatunnel.common.utils.JsonUtils;
+import org.apache.seatunnel.common.utils.TimeUtils;
+import org.apache.seatunnel.connectors.seatunnel.file.sink.config.TextFileSinkConfig;
+
+import org.apache.poi.ss.usermodel.Cell;
+import org.apache.poi.ss.usermodel.CellStyle;
+import org.apache.poi.ss.usermodel.CreationHelper;
+import org.apache.poi.ss.usermodel.Row;
+import org.apache.poi.ss.usermodel.Sheet;
+import org.apache.poi.ss.usermodel.Workbook;
+import org.apache.poi.xssf.streaming.SXSSFWorkbook;
+import org.apache.poi.xssf.usermodel.XSSFWorkbook;
+
+import java.io.IOException;
+import java.io.OutputStream;
+import java.lang.reflect.Array;
+import java.sql.Timestamp;
+import java.time.LocalDate;
+import java.time.LocalDateTime;
+import java.time.LocalTime;
+import java.util.ArrayList;
+import java.util.List;
+
+public class ExcelGenerator {
+ private List<Integer> sinkColumnsIndexInRow;
+ private SeaTunnelRowType seaTunnelRowType;
+ private final DateUtils.Formatter dateFormat;
+ private final DateTimeUtils.Formatter dateTimeFormat;
+ private final TimeUtils.Formatter timeFormat;
+ private final String fieldDelimiter;
+ private Workbook wb;
+ private CellStyle wholeNumberCellStyle;
+ private CellStyle stringCellStyle;
+ private CellStyle dateCellStyle;
+ private CellStyle dateTimeCellStyle;
+ private CellStyle timeCellStyle;
+ private Sheet st;
+ private int row = 0;
+ public ExcelGenerator(List<Integer> sinkColumnsIndexInRow, SeaTunnelRowType seaTunnelRowType, TextFileSinkConfig textFileSinkConfig) {
+ this.sinkColumnsIndexInRow = sinkColumnsIndexInRow;
+ this.seaTunnelRowType = seaTunnelRowType;
+ if (textFileSinkConfig.getMaxRowsInMemory() > 0) {
+ wb = new SXSSFWorkbook(textFileSinkConfig.getMaxRowsInMemory());
+ } else {
+ wb = new XSSFWorkbook();
Review Comment:
`MaxRowsInMemory` default value is Long.MAX_VALUE ?
You can choose from the following:
1. It is stated in the document that if `MaxRowsInMemory` is not set, all data will be buffered to memory
2. Set default value xxx for `MaxRowsInMemory`
##########
docs/en/connector-v2/sink/HdfsFile.md:
##########
@@ -86,6 +87,10 @@ The separator between columns in a row of data. Only needed by `text` and `csv`
The separator between rows in a file. Only needed by `text` and `csv` file format.
+### max_rows_in_memory [int]
Review Comment:
add `max_rows_in_memory` to options
<img width="1371" alt="image" src="https://user-images.githubusercontent.com/14371345/202341095-4660868c-e97f-4694-99f5-f4c464ef81b1.png">
##########
seatunnel-e2e/seatunnel-flink-connector-v2-e2e/src/test/resources/file/fakesource_to_hdfs_excel.conf:
##########
@@ -0,0 +1,70 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+######
+###### This config file is a demonstration of streaming processing in seatunnel config
+######
+
+env {
+ # You can set flink configuration here
+ execution.parallelism = 1
+ job.mode = "BATCH"
+ #execution.checkpoint.interval = 10000
+ #execution.checkpoint.data-uri = "hdfs://localhost:9000/checkpoint"
+}
+
+source {
+ FakeSource {
+ result_table_name = "fake"
+ schema = {
+ fields {
+ name = "string"
+ age = "int"
Review Comment:
test all data type write to excel file
```suggestion
c_map = "map<string, string>"
c_array = "array<int>"
c_string = string
c_boolean = boolean
c_tinyint = tinyint
c_smallint = smallint
c_int = int
c_bigint = bigint
c_float = float
c_double = double
c_bytes = bytes
c_date = date
c_decimal = "decimal(38, 18)"
c_timestamp = timestamp
c_row = {
c_map = "map<string, string>"
c_array = "array<int>"
c_string = string
c_boolean = boolean
c_tinyint = tinyint
c_smallint = smallint
c_int = int
c_bigint = bigint
c_float = float
c_double = double
c_bytes = bytes
c_date = date
c_decimal = "decimal(38, 18)"
c_timestamp = timestamp
```
##########
seatunnel-e2e/seatunnel-flink-connector-v2-e2e/src/test/resources/file/fakesource_to_local_excel.conf:
##########
@@ -0,0 +1,69 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+######
+###### This config file is a demonstration of streaming processing in seatunnel config
+######
+
+env {
+ # You can set flink configuration here
+ execution.parallelism = 1
+ job.mode = "BATCH"
+ #execution.checkpoint.interval = 10000
+ #execution.checkpoint.data-uri = "hdfs://localhost:9000/checkpoint"
+}
+
+source {
+ FakeSource {
+ result_table_name = "fake"
+ schema = {
+ fields {
+ name = "string"
+ age = "int"
+ }
+ }
+ }
+
+ # If you would like to get more information about how to configure seatunnel and see full list of source plugins,
+ # please go to https://seatunnel.apache.org/docs/connector-v2/source/FakeSource
+}
+
+transform {
+ sql {
+ sql = "select name,age from fake"
+ }
+
+ # If you would like to get more information about how to configure seatunnel and see full list of transform plugins,
+ # please go to https://seatunnel.apache.org/docs/flink/configuration/transform-plugins/Sql
+}
Review Comment:
remove
```suggestion
```
##########
seatunnel-e2e/seatunnel-flink-connector-v2-e2e/src/test/resources/file/fakesource_to_hdfs_excel.conf:
##########
@@ -0,0 +1,70 @@
+#
Review Comment:
2. testing run this file
Add this file path to here https://github.com/apache/incubator-seatunnel/blob/dev/seatunnel-e2e/seatunnel-connector-v2-e2e/connector-file-local-e2e/src/test/java/org/apache/seatunnel/e2e/connector/file/local/LocalFileIT.java#L52
##########
seatunnel-e2e/seatunnel-flink-connector-v2-e2e/src/test/resources/file/fakesource_to_local_excel.conf:
##########
@@ -0,0 +1,69 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+######
+###### This config file is a demonstration of streaming processing in seatunnel config
+######
+
+env {
+ # You can set flink configuration here
+ execution.parallelism = 1
+ job.mode = "BATCH"
+ #execution.checkpoint.interval = 10000
+ #execution.checkpoint.data-uri = "hdfs://localhost:9000/checkpoint"
+}
+
+source {
+ FakeSource {
+ result_table_name = "fake"
+ schema = {
+ fields {
+ name = "string"
+ age = "int"
+ }
+ }
+ }
+
+ # If you would like to get more information about how to configure seatunnel and see full list of source plugins,
+ # please go to https://seatunnel.apache.org/docs/connector-v2/source/FakeSource
+}
+
+transform {
+ sql {
+ sql = "select name,age from fake"
+ }
+
+ # If you would like to get more information about how to configure seatunnel and see full list of transform plugins,
+ # please go to https://seatunnel.apache.org/docs/flink/configuration/transform-plugins/Sql
+}
+
+sink {
+ LocalFile {
+ path="/tmp/hive/warehouse/test2"
+ partition_by=["age"]
+ partition_dir_expression="${k0}=${v0}"
+ is_partition_field_write_in_file=true
+ file_name_expression="${transactionId}_${now}"
+ file_format="excel"
+ sink_columns=["name","age"]
Review Comment:
remove
```suggestion
```
##########
seatunnel-e2e/seatunnel-flink-connector-v2-e2e/src/test/resources/file/fakesource_to_local_excel.conf:
##########
@@ -0,0 +1,69 @@
+#
Review Comment:
1. move this file to https://github.com/apache/incubator-seatunnel/tree/dev/seatunnel-e2e/seatunnel-connector-v2-e2e/connector-file-local-e2e/src/test/resources
##########
seatunnel-e2e/seatunnel-flink-connector-v2-e2e/src/test/resources/file/fakesource_to_hdfs_excel.conf:
##########
@@ -0,0 +1,70 @@
+#
Review Comment:
1. move this file to https://github.com/apache/incubator-seatunnel/tree/dev/seatunnel-e2e/seatunnel-connector-v2-e2e/connector-file-local-e2e/src/test/resources
##########
seatunnel-e2e/seatunnel-flink-connector-v2-e2e/src/test/resources/file/fakesource_to_local_excel.conf:
##########
@@ -0,0 +1,69 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+######
+###### This config file is a demonstration of streaming processing in seatunnel config
+######
+
+env {
+ # You can set flink configuration here
+ execution.parallelism = 1
+ job.mode = "BATCH"
+ #execution.checkpoint.interval = 10000
+ #execution.checkpoint.data-uri = "hdfs://localhost:9000/checkpoint"
+}
+
+source {
+ FakeSource {
+ result_table_name = "fake"
+ schema = {
+ fields {
+ name = "string"
+ age = "int"
Review Comment:
test all data type write to excel file
```suggestion
c_map = "map<string, string>"
c_array = "array<int>"
c_string = string
c_boolean = boolean
c_tinyint = tinyint
c_smallint = smallint
c_int = int
c_bigint = bigint
c_float = float
c_double = double
c_bytes = bytes
c_date = date
c_decimal = "decimal(38, 18)"
c_timestamp = timestamp
c_row = {
c_map = "map<string, string>"
c_array = "array<int>"
c_string = string
c_boolean = boolean
c_tinyint = tinyint
c_smallint = smallint
c_int = int
c_bigint = bigint
c_float = float
c_double = double
c_bytes = bytes
c_date = date
c_decimal = "decimal(38, 18)"
c_timestamp = timestamp
```
##########
seatunnel-e2e/seatunnel-flink-connector-v2-e2e/src/test/resources/file/fakesource_to_hdfs_excel.conf:
##########
@@ -0,0 +1,70 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+######
+###### This config file is a demonstration of streaming processing in seatunnel config
+######
+
+env {
+ # You can set flink configuration here
+ execution.parallelism = 1
+ job.mode = "BATCH"
+ #execution.checkpoint.interval = 10000
+ #execution.checkpoint.data-uri = "hdfs://localhost:9000/checkpoint"
+}
+
+source {
+ FakeSource {
+ result_table_name = "fake"
+ schema = {
+ fields {
+ name = "string"
+ age = "int"
+ }
+ }
+ }
+
+ # If you would like to get more information about how to configure seatunnel and see full list of source plugins,
+ # please go to https://seatunnel.apache.org/docs/connector-v2/source/FakeSource
+}
+
+transform {
+ sql {
+ sql = "select name,age from fake"
+ }
+
+ # If you would like to get more information about how to configure seatunnel and see full list of transform plugins,
+ # please go to https://seatunnel.apache.org/docs/transform/sql
+}
+
+sink {
+ HdfsFile {
+ path="/tmp/hive/warehouse/test2"
+ row_delimiter="\n"
+ partition_by=["age"]
+ partition_dir_expression="${k0}=${v0}"
+ is_partition_field_write_in_file=true
+ file_name_expression="${transactionId}_${now}"
+ file_format="excel"
+ sink_columns=["name","age"]
Review Comment:
remove
```suggestion
```
##########
seatunnel-e2e/seatunnel-flink-connector-v2-e2e/src/test/resources/file/fakesource_to_hdfs_excel.conf:
##########
@@ -0,0 +1,70 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+######
+###### This config file is a demonstration of streaming processing in seatunnel config
+######
+
+env {
+ # You can set flink configuration here
+ execution.parallelism = 1
+ job.mode = "BATCH"
+ #execution.checkpoint.interval = 10000
+ #execution.checkpoint.data-uri = "hdfs://localhost:9000/checkpoint"
+}
+
+source {
+ FakeSource {
+ result_table_name = "fake"
+ schema = {
+ fields {
+ name = "string"
+ age = "int"
+ }
+ }
+ }
+
+ # If you would like to get more information about how to configure seatunnel and see full list of source plugins,
+ # please go to https://seatunnel.apache.org/docs/connector-v2/source/FakeSource
+}
+
+transform {
+ sql {
+ sql = "select name,age from fake"
+ }
+
+ # If you would like to get more information about how to configure seatunnel and see full list of transform plugins,
+ # please go to https://seatunnel.apache.org/docs/transform/sql
+}
Review Comment:
remove
```suggestion
```
##########
seatunnel-e2e/seatunnel-flink-connector-v2-e2e/src/test/resources/file/fakesource_to_local_excel.conf:
##########
@@ -0,0 +1,69 @@
+#
Review Comment:
2. testing run this file
Add this file path to here https://github.com/apache/incubator-seatunnel/blob/dev/seatunnel-e2e/seatunnel-connector-v2-e2e/connector-file-local-e2e/src/test/java/org/apache/seatunnel/e2e/connector/file/local/LocalFileIT.java#L52
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] Bingz2 commented on a diff in pull request #2585: [Feature][ConnectorV2]add file excel sink
Posted by GitBox <gi...@apache.org>.
Bingz2 commented on code in PR #2585:
URL: https://github.com/apache/incubator-seatunnel/pull/2585#discussion_r1029202752
##########
seatunnel-e2e/seatunnel-flink-connector-v2-e2e/src/test/resources/file/fakesource_to_hdfs_excel.conf:
##########
@@ -0,0 +1,70 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+######
+###### This config file is a demonstration of streaming processing in seatunnel config
+######
+
+env {
+ # You can set flink configuration here
+ execution.parallelism = 1
+ job.mode = "BATCH"
+ #execution.checkpoint.interval = 10000
+ #execution.checkpoint.data-uri = "hdfs://localhost:9000/checkpoint"
+}
+
+source {
+ FakeSource {
+ result_table_name = "fake"
+ schema = {
+ fields {
+ name = "string"
+ age = "int"
+ }
+ }
+ }
+
+ # If you would like to get more information about how to configure seatunnel and see full list of source plugins,
+ # please go to https://seatunnel.apache.org/docs/connector-v2/source/FakeSource
+}
+
+transform {
+ sql {
+ sql = "select name,age from fake"
+ }
+
+ # If you would like to get more information about how to configure seatunnel and see full list of transform plugins,
+ # please go to https://seatunnel.apache.org/docs/transform/sql
+}
Review Comment:
hi, I need to install fonts to use SXSSFWorkbook, how do I install them in the test container? For example, the following command`RUN apk add --update font-adobe-100dpi ttf-dejavu fontconfig`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] EricJoy2048 closed pull request #2585: [Feature][ConnectorV2]add file excel sink
Posted by GitBox <gi...@apache.org>.
EricJoy2048 closed pull request #2585: [Feature][ConnectorV2]add file excel sink
URL: https://github.com/apache/incubator-seatunnel/pull/2585
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] Bingz2 commented on pull request #2585: [Feature][ConnectorV2]add file excel sink
Posted by GitBox <gi...@apache.org>.
Bingz2 commented on PR #2585:
URL: https://github.com/apache/incubator-seatunnel/pull/2585#issuecomment-1278356401
> Hi, @Bingz2 thanks for your contribution, what's the news about this pr? Do you need some help?
Sorry, I'm a little busy recently. I'll finish it this weekend
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] Bingz2 commented on pull request #2585: [Feature][ConnectorV2]add file excel sink
Posted by GitBox <gi...@apache.org>.
Bingz2 commented on PR #2585:
URL: https://github.com/apache/incubator-seatunnel/pull/2585#issuecomment-1296185264
> > > Hi, @Bingz2 thanks for your contribution, what's the news about this pr? Do you need some help?
> > > Sorry, I'm a little busy recently. I'll finish it this weekend
>
> Thank you very much.
An error in the CI,but I executed E2E locally with no errors,I don't know why.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org