You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by GitBox <gi...@apache.org> on 2022/03/18 10:51:10 UTC
[GitHub] [flink] zhoulii opened a new pull request #19152: [FLINK-26692][e2e] migrate TpcdsTestProgram.java to new source
zhoulii opened a new pull request #19152:
URL: https://github.com/apache/flink/pull/19152
## What is the purpose of the change
As described in [FLINK-26692](https://issues.apache.org/jira/browse/FLINK-26692), we should migrate TpcdsTestProgram.java to new source.
## Verifying this change
- e2e test
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): (**no**)
- The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (**no**)
- The serializers: (**no**)
- The runtime per-record code paths (performance sensitive): (**no**)
- Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (**no**)
- The S3 file system connector: (**no**)
## Documentation
- Does this pull request introduce a new feature? (**no**)
- If yes, how is the feature documented? (**not applicable**)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] wanglijie95 commented on pull request #19152: [FLINK-26692][e2e] migrate TpcdsTestProgram.java to new source
Posted by GitBox <gi...@apache.org>.
wanglijie95 commented on pull request #19152:
URL: https://github.com/apache/flink/pull/19152#issuecomment-1074923533
I found that after changing to new source, tpcds runs slower than before. This is probably mainly because the new csv source is slower than the legacy `CsvTableSource`. It only took 20 min before, and now it takes 30 min. I think we need to wait for FLINK-26760 to have a conclusion before merging this pr.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #19152: [FLINK-26692][e2e] migrate TpcdsTestProgram.java to new source
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #19152:
URL: https://github.com/apache/flink/pull/19152#issuecomment-1072298017
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "3342d35de40f5ef19c29dd4ba8d26be76e5319eb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=33343",
"triggerID" : "3342d35de40f5ef19c29dd4ba8d26be76e5319eb",
"triggerType" : "PUSH"
}, {
"hash" : "14454d0b4f4047b1bfb5f71d131460e7bc6987e8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=33377",
"triggerID" : "14454d0b4f4047b1bfb5f71d131460e7bc6987e8",
"triggerType" : "PUSH"
}, {
"hash" : "a3674835e935ea4ed2e413b1793832ebc4fe7349",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=33390",
"triggerID" : "a3674835e935ea4ed2e413b1793832ebc4fe7349",
"triggerType" : "PUSH"
}, {
"hash" : "1c77abc2997c2682e540f8a544c13a7e0dd9675a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=33396",
"triggerID" : "1c77abc2997c2682e540f8a544c13a7e0dd9675a",
"triggerType" : "PUSH"
}, {
"hash" : "a1e060857ac5d00775554b1c300b1f6eebf8b30c",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=33403",
"triggerID" : "a1e060857ac5d00775554b1c300b1f6eebf8b30c",
"triggerType" : "PUSH"
}, {
"hash" : "aebb4eeba15978ae17c1c8194287548b781ee0c1",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=33475",
"triggerID" : "aebb4eeba15978ae17c1c8194287548b781ee0c1",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* a1e060857ac5d00775554b1c300b1f6eebf8b30c Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=33403)
* aebb4eeba15978ae17c1c8194287548b781ee0c1 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=33475)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] zhoulii commented on a change in pull request #19152: [FLINK-26692][e2e] migrate TpcdsTestProgram.java to new source
Posted by GitBox <gi...@apache.org>.
zhoulii commented on a change in pull request #19152:
URL: https://github.com/apache/flink/pull/19152#discussion_r830852576
##########
File path: flink-end-to-end-tests/flink-tpcds-test/tpcds-tool/data_generator.sh
##########
@@ -130,6 +130,10 @@ elif [[ "$OS_TYPE" == "linux" ]]; then
chmod +x $generator_dir/dsdgen_linux
cd $generator_dir
./dsdgen_linux -SCALE $scale_factor -FORCE Y -DIR $data_dir
+ echo "[INFO] `date +%H:%M:%S` Convert file encoding of customer.dat to UTF-8 start."
Review comment:
I will add a "TODO" comments here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] wanglijie95 commented on a change in pull request #19152: [FLINK-26692][e2e] migrate TpcdsTestProgram.java to new source
Posted by GitBox <gi...@apache.org>.
wanglijie95 commented on a change in pull request #19152:
URL: https://github.com/apache/flink/pull/19152#discussion_r830881454
##########
File path: flink-end-to-end-tests/flink-tpcds-test/src/main/java/org/apache/flink/table/tpcds/TpcdsTestProgram.java
##########
@@ -109,16 +113,21 @@ public static void main(String[] args) throws Exception {
// register sink table
String sinkTableName = QUERY_PREFIX + queryId + "_sinkTable";
- ((TableEnvironmentInternal) tableEnvironment)
- .registerTableSinkInternal(
- sinkTableName,
- new CsvTableSink(
- sinkTablePath + FILE_SEPARATOR + queryId + RESULT_SUFFIX,
- COL_DELIMITER,
- 1,
- FileSystem.WriteMode.OVERWRITE,
- resultTable.getSchema().getFieldNames(),
- resultTable.getSchema().getFieldDataTypes()));
+ tableEnvironment.createTable(
+ sinkTableName,
+ TableDescriptor.forConnector(FILE_CONNECTOR_NAME)
+ .schema(resultTable.getSchema().toSchema())
+ .format(
+ FormatDescriptor.forFormat(CSV_FORMAT)
+ .option(CsvFormatOptions.FIELD_DELIMITER, COL_DELIMITER)
+ .option(CsvFormatOptions.DISABLE_QUOTE_CHARACTER, true)
+ .build())
+ .option(
+ FILE_PATH,
+ sinkTablePath + FILE_SEPARATOR + queryId + RESULT_SUFFIX)
+ .option(SINK_PARALLELISM, CSV_SINK_PARALLELISM)
+ .build());
+
TableResult tableResult = resultTable.executeInsert(sinkTableName);
Review comment:
In this place, maybe `executeInsert(sinkTableName, true)` should be used to align the behavior of the original `FileSystem.WriteMode.OVERWRITE`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] zhoulii commented on a change in pull request #19152: [FLINK-26692][e2e] migrate TpcdsTestProgram.java to new source
Posted by GitBox <gi...@apache.org>.
zhoulii commented on a change in pull request #19152:
URL: https://github.com/apache/flink/pull/19152#discussion_r830889065
##########
File path: flink-end-to-end-tests/flink-tpcds-test/src/main/java/org/apache/flink/table/tpcds/TpcdsTestProgram.java
##########
@@ -109,16 +113,21 @@ public static void main(String[] args) throws Exception {
// register sink table
String sinkTableName = QUERY_PREFIX + queryId + "_sinkTable";
- ((TableEnvironmentInternal) tableEnvironment)
- .registerTableSinkInternal(
- sinkTableName,
- new CsvTableSink(
- sinkTablePath + FILE_SEPARATOR + queryId + RESULT_SUFFIX,
- COL_DELIMITER,
- 1,
- FileSystem.WriteMode.OVERWRITE,
- resultTable.getSchema().getFieldNames(),
- resultTable.getSchema().getFieldDataTypes()));
+ tableEnvironment.createTable(
+ sinkTableName,
+ TableDescriptor.forConnector(FILE_CONNECTOR_NAME)
+ .schema(resultTable.getSchema().toSchema())
+ .format(
+ FormatDescriptor.forFormat(CSV_FORMAT)
+ .option(CsvFormatOptions.FIELD_DELIMITER, COL_DELIMITER)
+ .option(CsvFormatOptions.DISABLE_QUOTE_CHARACTER, true)
+ .build())
+ .option(
+ FILE_PATH,
+ sinkTablePath + FILE_SEPARATOR + queryId + RESULT_SUFFIX)
+ .option(SINK_PARALLELISM, CSV_SINK_PARALLELISM)
+ .build());
+
TableResult tableResult = resultTable.executeInsert(sinkTableName);
Review comment:
Agree, we should use executeInsert(sinkTableName, true) here, I will fix it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #19152: [FLINK-26692][e2e] migrate TpcdsTestProgram.java to new source
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #19152:
URL: https://github.com/apache/flink/pull/19152#issuecomment-1072298017
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "3342d35de40f5ef19c29dd4ba8d26be76e5319eb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=33343",
"triggerID" : "3342d35de40f5ef19c29dd4ba8d26be76e5319eb",
"triggerType" : "PUSH"
}, {
"hash" : "14454d0b4f4047b1bfb5f71d131460e7bc6987e8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=33377",
"triggerID" : "14454d0b4f4047b1bfb5f71d131460e7bc6987e8",
"triggerType" : "PUSH"
}, {
"hash" : "a3674835e935ea4ed2e413b1793832ebc4fe7349",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=33390",
"triggerID" : "a3674835e935ea4ed2e413b1793832ebc4fe7349",
"triggerType" : "PUSH"
}, {
"hash" : "1c77abc2997c2682e540f8a544c13a7e0dd9675a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=33396",
"triggerID" : "1c77abc2997c2682e540f8a544c13a7e0dd9675a",
"triggerType" : "PUSH"
}, {
"hash" : "a1e060857ac5d00775554b1c300b1f6eebf8b30c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=33403",
"triggerID" : "a1e060857ac5d00775554b1c300b1f6eebf8b30c",
"triggerType" : "PUSH"
}, {
"hash" : "aebb4eeba15978ae17c1c8194287548b781ee0c1",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=33475",
"triggerID" : "aebb4eeba15978ae17c1c8194287548b781ee0c1",
"triggerType" : "PUSH"
}, {
"hash" : "2f19a1110b9821980e37eff0a69c77aeecd65438",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "2f19a1110b9821980e37eff0a69c77aeecd65438",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* aebb4eeba15978ae17c1c8194287548b781ee0c1 Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=33475)
* 2f19a1110b9821980e37eff0a69c77aeecd65438 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #19152: [FLINK-26692][e2e] migrate TpcdsTestProgram.java to new source
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #19152:
URL: https://github.com/apache/flink/pull/19152#issuecomment-1072298017
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "3342d35de40f5ef19c29dd4ba8d26be76e5319eb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=33343",
"triggerID" : "3342d35de40f5ef19c29dd4ba8d26be76e5319eb",
"triggerType" : "PUSH"
}, {
"hash" : "14454d0b4f4047b1bfb5f71d131460e7bc6987e8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=33377",
"triggerID" : "14454d0b4f4047b1bfb5f71d131460e7bc6987e8",
"triggerType" : "PUSH"
}, {
"hash" : "a3674835e935ea4ed2e413b1793832ebc4fe7349",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=33390",
"triggerID" : "a3674835e935ea4ed2e413b1793832ebc4fe7349",
"triggerType" : "PUSH"
}, {
"hash" : "1c77abc2997c2682e540f8a544c13a7e0dd9675a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=33396",
"triggerID" : "1c77abc2997c2682e540f8a544c13a7e0dd9675a",
"triggerType" : "PUSH"
}, {
"hash" : "a1e060857ac5d00775554b1c300b1f6eebf8b30c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=33403",
"triggerID" : "a1e060857ac5d00775554b1c300b1f6eebf8b30c",
"triggerType" : "PUSH"
}, {
"hash" : "aebb4eeba15978ae17c1c8194287548b781ee0c1",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=33475",
"triggerID" : "aebb4eeba15978ae17c1c8194287548b781ee0c1",
"triggerType" : "PUSH"
}, {
"hash" : "2f19a1110b9821980e37eff0a69c77aeecd65438",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=33493",
"triggerID" : "2f19a1110b9821980e37eff0a69c77aeecd65438",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* aebb4eeba15978ae17c1c8194287548b781ee0c1 Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=33475)
* 2f19a1110b9821980e37eff0a69c77aeecd65438 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=33493)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #19152: [FLINK-26692][e2e] migrate TpcdsTestProgram.java to new source
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #19152:
URL: https://github.com/apache/flink/pull/19152#issuecomment-1072298017
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "3342d35de40f5ef19c29dd4ba8d26be76e5319eb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=33343",
"triggerID" : "3342d35de40f5ef19c29dd4ba8d26be76e5319eb",
"triggerType" : "PUSH"
}, {
"hash" : "14454d0b4f4047b1bfb5f71d131460e7bc6987e8",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=33377",
"triggerID" : "14454d0b4f4047b1bfb5f71d131460e7bc6987e8",
"triggerType" : "PUSH"
}, {
"hash" : "a3674835e935ea4ed2e413b1793832ebc4fe7349",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=33390",
"triggerID" : "a3674835e935ea4ed2e413b1793832ebc4fe7349",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 14454d0b4f4047b1bfb5f71d131460e7bc6987e8 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=33377)
* a3674835e935ea4ed2e413b1793832ebc4fe7349 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=33390)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #19152: [FLINK-26692][e2e] migrate TpcdsTestProgram.java to new source
Posted by GitBox <gi...@apache.org>.
flinkbot commented on pull request #19152:
URL: https://github.com/apache/flink/pull/19152#issuecomment-1072298017
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "3342d35de40f5ef19c29dd4ba8d26be76e5319eb",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "3342d35de40f5ef19c29dd4ba8d26be76e5319eb",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 3342d35de40f5ef19c29dd4ba8d26be76e5319eb UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #19152: [FLINK-26692][e2e] migrate TpcdsTestProgram.java to new source
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #19152:
URL: https://github.com/apache/flink/pull/19152#issuecomment-1072298017
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "3342d35de40f5ef19c29dd4ba8d26be76e5319eb",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=33343",
"triggerID" : "3342d35de40f5ef19c29dd4ba8d26be76e5319eb",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 3342d35de40f5ef19c29dd4ba8d26be76e5319eb Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=33343)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] wanglijie95 commented on a change in pull request #19152: [FLINK-26692][e2e] migrate TpcdsTestProgram.java to new source
Posted by GitBox <gi...@apache.org>.
wanglijie95 commented on a change in pull request #19152:
URL: https://github.com/apache/flink/pull/19152#discussion_r830782057
##########
File path: flink-end-to-end-tests/flink-tpcds-test/tpcds-tool/data_generator.sh
##########
@@ -130,6 +130,10 @@ elif [[ "$OS_TYPE" == "linux" ]]; then
chmod +x $generator_dir/dsdgen_linux
cd $generator_dir
./dsdgen_linux -SCALE $scale_factor -FORCE Y -DIR $data_dir
+ echo "[INFO] `date +%H:%M:%S` Convert file encoding of customer.dat to UTF-8 start."
Review comment:
+1, maybe add a "TODO" comments. I 've created an issue for the bug: FLINK-26760
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] zhoulii commented on a change in pull request #19152: [FLINK-26692][e2e] migrate TpcdsTestProgram.java to new source
Posted by GitBox <gi...@apache.org>.
zhoulii commented on a change in pull request #19152:
URL: https://github.com/apache/flink/pull/19152#discussion_r830847334
##########
File path: flink-end-to-end-tests/flink-tpcds-test/src/main/java/org/apache/flink/table/tpcds/TpcdsTestProgram.java
##########
@@ -157,34 +166,33 @@ private static TableEnvironment prepareTableEnv(String sourceTablePath, Boolean
TPCDS_TABLES.forEach(
table -> {
TpcdsSchema schema = TpcdsSchemaProvider.getTableSchema(table);
- CsvTableSource.Builder builder = CsvTableSource.builder();
- builder.path(sourceTablePath + FILE_SEPARATOR + table + DATA_SUFFIX);
- for (int i = 0; i < schema.getFieldNames().size(); i++) {
- builder.field(
- schema.getFieldNames().get(i),
- TypeConversions.fromDataTypeToLegacyInfo(
- schema.getFieldTypes().get(i)));
- }
- builder.fieldDelimiter(COL_DELIMITER);
- builder.emptyColumnAsNull();
- builder.lineDelimiter("\n");
- CsvTableSource tableSource = builder.build();
- ConnectorCatalogTable catalogTable =
- ConnectorCatalogTable.source(tableSource, true);
- tEnv.getCatalog(tEnv.getCurrentCatalog())
- .ifPresent(
- catalog -> {
- try {
- catalog.createTable(
- new ObjectPath(
- tEnv.getCurrentDatabase(), table),
- catalogTable,
- false);
- } catch (Exception e) {
- throw new RuntimeException(e);
- }
- });
+ String filePath = sourceTablePath + FILE_SEPARATOR + table + DATA_SUFFIX;
+
+ tEnv.createTable(
+ table,
+ TableDescriptor.forConnector(FILE_CONNECTOR_NAME)
+ .schema(
+ Schema.newBuilder()
+ .fromFields(
+ schema.getFieldNames(),
+ schema.getFieldTypes())
+ .build())
+ .format(
+ FormatDescriptor.forFormat(CSV_FORMAT)
+ .option(
+ CsvFormatOptions.FIELD_DELIMITER,
+ COL_DELIMITER)
+ .option(
+ CsvFormatOptions.IGNORE_PARSE_ERRORS,
Review comment:
you are right, **_CsvFormatOptions.IGNORE_PARSE_ERRORS_** is not needed. I run tpcds without this config, the job still works well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] zhoulii commented on pull request #19152: [FLINK-26692][e2e] migrate TpcdsTestProgram.java to new source
Posted by GitBox <gi...@apache.org>.
zhoulii commented on pull request #19152:
URL: https://github.com/apache/flink/pull/19152#issuecomment-1073413419
Hi @zhuzhurk @wanglijie95 , This PR is ready, can you take a look when you are convenient ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] zhoulii commented on pull request #19152: [FLINK-26692][e2e] migrate TpcdsTestProgram.java to new source
Posted by GitBox <gi...@apache.org>.
zhoulii commented on pull request #19152:
URL: https://github.com/apache/flink/pull/19152#issuecomment-1074936494
> I found that after changing to new source, tpcds runs slower than before. This is probably mainly because the new csv source is slower than the legacy `CsvTableSource`. It only took 20 min before, and now it takes 30 min. I think we need to wait for [FLINK-26760](https://issues.apache.org/jira/browse/FLINK-26760) to have a conclusion before merging this pr.
I agree. The way that parsing csv data between [CsvInputFormat.java#L87 which legacy csv source used](https://github.com/apache/flink/blob/master/flink-java/src/main/java/org/apache/flink/api/java/io/CsvInputFormat.java#L87) and [CsvReaderFormat.java#L193 which new csv source used](https://github.com/apache/flink/blob/master/flink-formats/flink-csv/src/main/java/org/apache/flink/formats/csv/CsvReaderFormat.java#L193) is quite different, may be we can reuse the parse method of CsvInputFormat in CsvReaderFormat.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #19152: [FLINK-26692][e2e] migrate TpcdsTestProgram.java to new source
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #19152:
URL: https://github.com/apache/flink/pull/19152#issuecomment-1072298017
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "3342d35de40f5ef19c29dd4ba8d26be76e5319eb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=33343",
"triggerID" : "3342d35de40f5ef19c29dd4ba8d26be76e5319eb",
"triggerType" : "PUSH"
}, {
"hash" : "14454d0b4f4047b1bfb5f71d131460e7bc6987e8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=33377",
"triggerID" : "14454d0b4f4047b1bfb5f71d131460e7bc6987e8",
"triggerType" : "PUSH"
}, {
"hash" : "a3674835e935ea4ed2e413b1793832ebc4fe7349",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=33390",
"triggerID" : "a3674835e935ea4ed2e413b1793832ebc4fe7349",
"triggerType" : "PUSH"
}, {
"hash" : "1c77abc2997c2682e540f8a544c13a7e0dd9675a",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=33396",
"triggerID" : "1c77abc2997c2682e540f8a544c13a7e0dd9675a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* a3674835e935ea4ed2e413b1793832ebc4fe7349 Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=33390)
* 1c77abc2997c2682e540f8a544c13a7e0dd9675a Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=33396)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #19152: [FLINK-26692][e2e] migrate TpcdsTestProgram.java to new source
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #19152:
URL: https://github.com/apache/flink/pull/19152#issuecomment-1072298017
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "3342d35de40f5ef19c29dd4ba8d26be76e5319eb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=33343",
"triggerID" : "3342d35de40f5ef19c29dd4ba8d26be76e5319eb",
"triggerType" : "PUSH"
}, {
"hash" : "14454d0b4f4047b1bfb5f71d131460e7bc6987e8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=33377",
"triggerID" : "14454d0b4f4047b1bfb5f71d131460e7bc6987e8",
"triggerType" : "PUSH"
}, {
"hash" : "a3674835e935ea4ed2e413b1793832ebc4fe7349",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=33390",
"triggerID" : "a3674835e935ea4ed2e413b1793832ebc4fe7349",
"triggerType" : "PUSH"
}, {
"hash" : "1c77abc2997c2682e540f8a544c13a7e0dd9675a",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=33396",
"triggerID" : "1c77abc2997c2682e540f8a544c13a7e0dd9675a",
"triggerType" : "PUSH"
}, {
"hash" : "a1e060857ac5d00775554b1c300b1f6eebf8b30c",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=33403",
"triggerID" : "a1e060857ac5d00775554b1c300b1f6eebf8b30c",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 1c77abc2997c2682e540f8a544c13a7e0dd9675a Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=33396)
* a1e060857ac5d00775554b1c300b1f6eebf8b30c Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=33403)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #19152: [FLINK-26692][e2e] migrate TpcdsTestProgram.java to new source
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #19152:
URL: https://github.com/apache/flink/pull/19152#issuecomment-1072298017
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "3342d35de40f5ef19c29dd4ba8d26be76e5319eb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=33343",
"triggerID" : "3342d35de40f5ef19c29dd4ba8d26be76e5319eb",
"triggerType" : "PUSH"
}, {
"hash" : "14454d0b4f4047b1bfb5f71d131460e7bc6987e8",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=33377",
"triggerID" : "14454d0b4f4047b1bfb5f71d131460e7bc6987e8",
"triggerType" : "PUSH"
}, {
"hash" : "a3674835e935ea4ed2e413b1793832ebc4fe7349",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "a3674835e935ea4ed2e413b1793832ebc4fe7349",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 14454d0b4f4047b1bfb5f71d131460e7bc6987e8 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=33377)
* a3674835e935ea4ed2e413b1793832ebc4fe7349 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #19152: [FLINK-26692][e2e] migrate TpcdsTestProgram.java to new source
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #19152:
URL: https://github.com/apache/flink/pull/19152#issuecomment-1072298017
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "3342d35de40f5ef19c29dd4ba8d26be76e5319eb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=33343",
"triggerID" : "3342d35de40f5ef19c29dd4ba8d26be76e5319eb",
"triggerType" : "PUSH"
}, {
"hash" : "14454d0b4f4047b1bfb5f71d131460e7bc6987e8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=33377",
"triggerID" : "14454d0b4f4047b1bfb5f71d131460e7bc6987e8",
"triggerType" : "PUSH"
}, {
"hash" : "a3674835e935ea4ed2e413b1793832ebc4fe7349",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=33390",
"triggerID" : "a3674835e935ea4ed2e413b1793832ebc4fe7349",
"triggerType" : "PUSH"
}, {
"hash" : "1c77abc2997c2682e540f8a544c13a7e0dd9675a",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=33396",
"triggerID" : "1c77abc2997c2682e540f8a544c13a7e0dd9675a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 1c77abc2997c2682e540f8a544c13a7e0dd9675a Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=33396)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #19152: [FLINK-26692][e2e] migrate TpcdsTestProgram.java to new source
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #19152:
URL: https://github.com/apache/flink/pull/19152#issuecomment-1072298017
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "3342d35de40f5ef19c29dd4ba8d26be76e5319eb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=33343",
"triggerID" : "3342d35de40f5ef19c29dd4ba8d26be76e5319eb",
"triggerType" : "PUSH"
}, {
"hash" : "14454d0b4f4047b1bfb5f71d131460e7bc6987e8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=33377",
"triggerID" : "14454d0b4f4047b1bfb5f71d131460e7bc6987e8",
"triggerType" : "PUSH"
}, {
"hash" : "a3674835e935ea4ed2e413b1793832ebc4fe7349",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=33390",
"triggerID" : "a3674835e935ea4ed2e413b1793832ebc4fe7349",
"triggerType" : "PUSH"
}, {
"hash" : "1c77abc2997c2682e540f8a544c13a7e0dd9675a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=33396",
"triggerID" : "1c77abc2997c2682e540f8a544c13a7e0dd9675a",
"triggerType" : "PUSH"
}, {
"hash" : "a1e060857ac5d00775554b1c300b1f6eebf8b30c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=33403",
"triggerID" : "a1e060857ac5d00775554b1c300b1f6eebf8b30c",
"triggerType" : "PUSH"
}, {
"hash" : "aebb4eeba15978ae17c1c8194287548b781ee0c1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=33475",
"triggerID" : "aebb4eeba15978ae17c1c8194287548b781ee0c1",
"triggerType" : "PUSH"
}, {
"hash" : "2f19a1110b9821980e37eff0a69c77aeecd65438",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=33493",
"triggerID" : "2f19a1110b9821980e37eff0a69c77aeecd65438",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 2f19a1110b9821980e37eff0a69c77aeecd65438 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=33493)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #19152: [FLINK-26692][e2e] migrate TpcdsTestProgram.java to new source
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #19152:
URL: https://github.com/apache/flink/pull/19152#issuecomment-1072298017
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "3342d35de40f5ef19c29dd4ba8d26be76e5319eb",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=33343",
"triggerID" : "3342d35de40f5ef19c29dd4ba8d26be76e5319eb",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 3342d35de40f5ef19c29dd4ba8d26be76e5319eb Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=33343)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #19152: [FLINK-26692][e2e] migrate TpcdsTestProgram.java to new source
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #19152:
URL: https://github.com/apache/flink/pull/19152#issuecomment-1072298017
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "3342d35de40f5ef19c29dd4ba8d26be76e5319eb",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=33343",
"triggerID" : "3342d35de40f5ef19c29dd4ba8d26be76e5319eb",
"triggerType" : "PUSH"
}, {
"hash" : "14454d0b4f4047b1bfb5f71d131460e7bc6987e8",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "14454d0b4f4047b1bfb5f71d131460e7bc6987e8",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 3342d35de40f5ef19c29dd4ba8d26be76e5319eb Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=33343)
* 14454d0b4f4047b1bfb5f71d131460e7bc6987e8 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #19152: [FLINK-26692][e2e] migrate TpcdsTestProgram.java to new source
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #19152:
URL: https://github.com/apache/flink/pull/19152#issuecomment-1072298017
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "3342d35de40f5ef19c29dd4ba8d26be76e5319eb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=33343",
"triggerID" : "3342d35de40f5ef19c29dd4ba8d26be76e5319eb",
"triggerType" : "PUSH"
}, {
"hash" : "14454d0b4f4047b1bfb5f71d131460e7bc6987e8",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=33377",
"triggerID" : "14454d0b4f4047b1bfb5f71d131460e7bc6987e8",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 14454d0b4f4047b1bfb5f71d131460e7bc6987e8 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=33377)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #19152: [FLINK-26692][e2e] migrate TpcdsTestProgram.java to new source
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #19152:
URL: https://github.com/apache/flink/pull/19152#issuecomment-1072298017
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "3342d35de40f5ef19c29dd4ba8d26be76e5319eb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=33343",
"triggerID" : "3342d35de40f5ef19c29dd4ba8d26be76e5319eb",
"triggerType" : "PUSH"
}, {
"hash" : "14454d0b4f4047b1bfb5f71d131460e7bc6987e8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=33377",
"triggerID" : "14454d0b4f4047b1bfb5f71d131460e7bc6987e8",
"triggerType" : "PUSH"
}, {
"hash" : "a3674835e935ea4ed2e413b1793832ebc4fe7349",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=33390",
"triggerID" : "a3674835e935ea4ed2e413b1793832ebc4fe7349",
"triggerType" : "PUSH"
}, {
"hash" : "1c77abc2997c2682e540f8a544c13a7e0dd9675a",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=33396",
"triggerID" : "1c77abc2997c2682e540f8a544c13a7e0dd9675a",
"triggerType" : "PUSH"
}, {
"hash" : "a1e060857ac5d00775554b1c300b1f6eebf8b30c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "a1e060857ac5d00775554b1c300b1f6eebf8b30c",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 1c77abc2997c2682e540f8a544c13a7e0dd9675a Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=33396)
* a1e060857ac5d00775554b1c300b1f6eebf8b30c UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #19152: [FLINK-26692][e2e] migrate TpcdsTestProgram.java to new source
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #19152:
URL: https://github.com/apache/flink/pull/19152#issuecomment-1072298017
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "3342d35de40f5ef19c29dd4ba8d26be76e5319eb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=33343",
"triggerID" : "3342d35de40f5ef19c29dd4ba8d26be76e5319eb",
"triggerType" : "PUSH"
}, {
"hash" : "14454d0b4f4047b1bfb5f71d131460e7bc6987e8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=33377",
"triggerID" : "14454d0b4f4047b1bfb5f71d131460e7bc6987e8",
"triggerType" : "PUSH"
}, {
"hash" : "a3674835e935ea4ed2e413b1793832ebc4fe7349",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=33390",
"triggerID" : "a3674835e935ea4ed2e413b1793832ebc4fe7349",
"triggerType" : "PUSH"
}, {
"hash" : "1c77abc2997c2682e540f8a544c13a7e0dd9675a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=33396",
"triggerID" : "1c77abc2997c2682e540f8a544c13a7e0dd9675a",
"triggerType" : "PUSH"
}, {
"hash" : "a1e060857ac5d00775554b1c300b1f6eebf8b30c",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=33403",
"triggerID" : "a1e060857ac5d00775554b1c300b1f6eebf8b30c",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* a1e060857ac5d00775554b1c300b1f6eebf8b30c Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=33403)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #19152: [FLINK-26692][e2e] migrate TpcdsTestProgram.java to new source
Posted by GitBox <gi...@apache.org>.
flinkbot commented on pull request #19152:
URL: https://github.com/apache/flink/pull/19152#issuecomment-1072298017
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "3342d35de40f5ef19c29dd4ba8d26be76e5319eb",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "3342d35de40f5ef19c29dd4ba8d26be76e5319eb",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 3342d35de40f5ef19c29dd4ba8d26be76e5319eb UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #19152: [FLINK-26692][e2e] migrate TpcdsTestProgram.java to new source
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #19152:
URL: https://github.com/apache/flink/pull/19152#issuecomment-1072298017
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "3342d35de40f5ef19c29dd4ba8d26be76e5319eb",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=33343",
"triggerID" : "3342d35de40f5ef19c29dd4ba8d26be76e5319eb",
"triggerType" : "PUSH"
}, {
"hash" : "14454d0b4f4047b1bfb5f71d131460e7bc6987e8",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=33377",
"triggerID" : "14454d0b4f4047b1bfb5f71d131460e7bc6987e8",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 3342d35de40f5ef19c29dd4ba8d26be76e5319eb Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=33343)
* 14454d0b4f4047b1bfb5f71d131460e7bc6987e8 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=33377)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] zhoulii commented on a change in pull request #19152: [FLINK-26692][e2e] migrate TpcdsTestProgram.java to new source
Posted by GitBox <gi...@apache.org>.
zhoulii commented on a change in pull request #19152:
URL: https://github.com/apache/flink/pull/19152#discussion_r830847334
##########
File path: flink-end-to-end-tests/flink-tpcds-test/src/main/java/org/apache/flink/table/tpcds/TpcdsTestProgram.java
##########
@@ -157,34 +166,33 @@ private static TableEnvironment prepareTableEnv(String sourceTablePath, Boolean
TPCDS_TABLES.forEach(
table -> {
TpcdsSchema schema = TpcdsSchemaProvider.getTableSchema(table);
- CsvTableSource.Builder builder = CsvTableSource.builder();
- builder.path(sourceTablePath + FILE_SEPARATOR + table + DATA_SUFFIX);
- for (int i = 0; i < schema.getFieldNames().size(); i++) {
- builder.field(
- schema.getFieldNames().get(i),
- TypeConversions.fromDataTypeToLegacyInfo(
- schema.getFieldTypes().get(i)));
- }
- builder.fieldDelimiter(COL_DELIMITER);
- builder.emptyColumnAsNull();
- builder.lineDelimiter("\n");
- CsvTableSource tableSource = builder.build();
- ConnectorCatalogTable catalogTable =
- ConnectorCatalogTable.source(tableSource, true);
- tEnv.getCatalog(tEnv.getCurrentCatalog())
- .ifPresent(
- catalog -> {
- try {
- catalog.createTable(
- new ObjectPath(
- tEnv.getCurrentDatabase(), table),
- catalogTable,
- false);
- } catch (Exception e) {
- throw new RuntimeException(e);
- }
- });
+ String filePath = sourceTablePath + FILE_SEPARATOR + table + DATA_SUFFIX;
+
+ tEnv.createTable(
+ table,
+ TableDescriptor.forConnector(FILE_CONNECTOR_NAME)
+ .schema(
+ Schema.newBuilder()
+ .fromFields(
+ schema.getFieldNames(),
+ schema.getFieldTypes())
+ .build())
+ .format(
+ FormatDescriptor.forFormat(CSV_FORMAT)
+ .option(
+ CsvFormatOptions.FIELD_DELIMITER,
+ COL_DELIMITER)
+ .option(
+ CsvFormatOptions.IGNORE_PARSE_ERRORS,
Review comment:
you are right, **_CsvFormatOptions.IGNORE_PARSE_ERRORS_** is not needed. I run tpcds without this config, the job still works well. I will remove this config.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] zhoulii commented on pull request #19152: [FLINK-26692][e2e] migrate TpcdsTestProgram.java to new source
Posted by GitBox <gi...@apache.org>.
zhoulii commented on pull request #19152:
URL: https://github.com/apache/flink/pull/19152#issuecomment-1074632632
Thanks for review @zhuzhurk @wanglijie95 . I've addressed all your comments. Please take a look..
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #19152: [FLINK-26692][e2e] migrate TpcdsTestProgram.java to new source
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #19152:
URL: https://github.com/apache/flink/pull/19152#issuecomment-1072298017
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "3342d35de40f5ef19c29dd4ba8d26be76e5319eb",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=33343",
"triggerID" : "3342d35de40f5ef19c29dd4ba8d26be76e5319eb",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 3342d35de40f5ef19c29dd4ba8d26be76e5319eb Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=33343)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #19152: [FLINK-26692][e2e] migrate TpcdsTestProgram.java to new source
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #19152:
URL: https://github.com/apache/flink/pull/19152#issuecomment-1072298017
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "3342d35de40f5ef19c29dd4ba8d26be76e5319eb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=33343",
"triggerID" : "3342d35de40f5ef19c29dd4ba8d26be76e5319eb",
"triggerType" : "PUSH"
}, {
"hash" : "14454d0b4f4047b1bfb5f71d131460e7bc6987e8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=33377",
"triggerID" : "14454d0b4f4047b1bfb5f71d131460e7bc6987e8",
"triggerType" : "PUSH"
}, {
"hash" : "a3674835e935ea4ed2e413b1793832ebc4fe7349",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=33390",
"triggerID" : "a3674835e935ea4ed2e413b1793832ebc4fe7349",
"triggerType" : "PUSH"
}, {
"hash" : "1c77abc2997c2682e540f8a544c13a7e0dd9675a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=33396",
"triggerID" : "1c77abc2997c2682e540f8a544c13a7e0dd9675a",
"triggerType" : "PUSH"
}, {
"hash" : "a1e060857ac5d00775554b1c300b1f6eebf8b30c",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=33403",
"triggerID" : "a1e060857ac5d00775554b1c300b1f6eebf8b30c",
"triggerType" : "PUSH"
}, {
"hash" : "aebb4eeba15978ae17c1c8194287548b781ee0c1",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=33475",
"triggerID" : "aebb4eeba15978ae17c1c8194287548b781ee0c1",
"triggerType" : "PUSH"
}, {
"hash" : "2f19a1110b9821980e37eff0a69c77aeecd65438",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "2f19a1110b9821980e37eff0a69c77aeecd65438",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* a1e060857ac5d00775554b1c300b1f6eebf8b30c Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=33403)
* aebb4eeba15978ae17c1c8194287548b781ee0c1 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=33475)
* 2f19a1110b9821980e37eff0a69c77aeecd65438 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] zhuzhurk commented on a change in pull request #19152: [FLINK-26692][e2e] migrate TpcdsTestProgram.java to new source
Posted by GitBox <gi...@apache.org>.
zhuzhurk commented on a change in pull request #19152:
URL: https://github.com/apache/flink/pull/19152#discussion_r830729038
##########
File path: flink-end-to-end-tests/flink-tpcds-test/src/main/java/org/apache/flink/table/tpcds/TpcdsTestProgram.java
##########
@@ -157,34 +166,33 @@ private static TableEnvironment prepareTableEnv(String sourceTablePath, Boolean
TPCDS_TABLES.forEach(
table -> {
TpcdsSchema schema = TpcdsSchemaProvider.getTableSchema(table);
- CsvTableSource.Builder builder = CsvTableSource.builder();
- builder.path(sourceTablePath + FILE_SEPARATOR + table + DATA_SUFFIX);
- for (int i = 0; i < schema.getFieldNames().size(); i++) {
- builder.field(
- schema.getFieldNames().get(i),
- TypeConversions.fromDataTypeToLegacyInfo(
- schema.getFieldTypes().get(i)));
- }
- builder.fieldDelimiter(COL_DELIMITER);
- builder.emptyColumnAsNull();
- builder.lineDelimiter("\n");
- CsvTableSource tableSource = builder.build();
- ConnectorCatalogTable catalogTable =
- ConnectorCatalogTable.source(tableSource, true);
- tEnv.getCatalog(tEnv.getCurrentCatalog())
- .ifPresent(
- catalog -> {
- try {
- catalog.createTable(
- new ObjectPath(
- tEnv.getCurrentDatabase(), table),
- catalogTable,
- false);
- } catch (Exception e) {
- throw new RuntimeException(e);
- }
- });
+ String filePath = sourceTablePath + FILE_SEPARATOR + table + DATA_SUFFIX;
+
+ tEnv.createTable(
+ table,
+ TableDescriptor.forConnector(FILE_CONNECTOR_NAME)
+ .schema(
+ Schema.newBuilder()
+ .fromFields(
+ schema.getFieldNames(),
+ schema.getFieldTypes())
+ .build())
+ .format(
+ FormatDescriptor.forFormat(CSV_FORMAT)
+ .option(
+ CsvFormatOptions.FIELD_DELIMITER,
+ COL_DELIMITER)
+ .option(
+ CsvFormatOptions.IGNORE_PARSE_ERRORS,
Review comment:
Looks to me this config it is not needed to be `true` because `ignoreParseErrors()` was not invoked on the old `CsvTableSource`.
Correct me if parse errors do happen in this case.
##########
File path: flink-end-to-end-tests/flink-tpcds-test/src/main/java/org/apache/flink/table/tpcds/utils/TpcdsResultComparator.java
##########
@@ -61,7 +62,8 @@ public static void main(String[] args) throws Exception {
for (String queryId : VALIDATE_QUERIES) {
File expectedFile = new File(expectedDir, queryId + RESULT_SUFFIX);
- File actualFile = new File(actualDir, queryId + RESULT_SUFFIX);
+ File actualFileDirectory = new File(actualDir, queryId + RESULT_SUFFIX);
+ File actualFile = Objects.requireNonNull(actualFileDirectory.listFiles())[0];
Review comment:
Would you explain a bit that why the actual file is changed in this case?
Does this means that there is behavior change when switching from an old csv sink to new a new csv sink?
##########
File path: flink-end-to-end-tests/flink-tpcds-test/tpcds-tool/data_generator.sh
##########
@@ -130,6 +130,10 @@ elif [[ "$OS_TYPE" == "linux" ]]; then
chmod +x $generator_dir/dsdgen_linux
cd $generator_dir
./dsdgen_linux -SCALE $scale_factor -FORCE Y -DIR $data_dir
+ echo "[INFO] `date +%H:%M:%S` Convert file encoding of customer.dat to UTF-8 start."
Review comment:
Would you add a comment to explain that "the data generator may generate files encoded in latin1, which the new csv source cannot read"?
And looks to me this is a bug of the new csv source and needs to be fixed. After it is fixed, we can remove this workaround.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #19152: [FLINK-26692][e2e] migrate TpcdsTestProgram.java to new source
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #19152:
URL: https://github.com/apache/flink/pull/19152#issuecomment-1072298017
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "3342d35de40f5ef19c29dd4ba8d26be76e5319eb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=33343",
"triggerID" : "3342d35de40f5ef19c29dd4ba8d26be76e5319eb",
"triggerType" : "PUSH"
}, {
"hash" : "14454d0b4f4047b1bfb5f71d131460e7bc6987e8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=33377",
"triggerID" : "14454d0b4f4047b1bfb5f71d131460e7bc6987e8",
"triggerType" : "PUSH"
}, {
"hash" : "a3674835e935ea4ed2e413b1793832ebc4fe7349",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=33390",
"triggerID" : "a3674835e935ea4ed2e413b1793832ebc4fe7349",
"triggerType" : "PUSH"
}, {
"hash" : "1c77abc2997c2682e540f8a544c13a7e0dd9675a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=33396",
"triggerID" : "1c77abc2997c2682e540f8a544c13a7e0dd9675a",
"triggerType" : "PUSH"
}, {
"hash" : "a1e060857ac5d00775554b1c300b1f6eebf8b30c",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=33403",
"triggerID" : "a1e060857ac5d00775554b1c300b1f6eebf8b30c",
"triggerType" : "PUSH"
}, {
"hash" : "aebb4eeba15978ae17c1c8194287548b781ee0c1",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "aebb4eeba15978ae17c1c8194287548b781ee0c1",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* a1e060857ac5d00775554b1c300b1f6eebf8b30c Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=33403)
* aebb4eeba15978ae17c1c8194287548b781ee0c1 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] zhoulii commented on a change in pull request #19152: [FLINK-26692][e2e] migrate TpcdsTestProgram.java to new source
Posted by GitBox <gi...@apache.org>.
zhoulii commented on a change in pull request #19152:
URL: https://github.com/apache/flink/pull/19152#discussion_r830851402
##########
File path: flink-end-to-end-tests/flink-tpcds-test/src/main/java/org/apache/flink/table/tpcds/utils/TpcdsResultComparator.java
##########
@@ -61,7 +62,8 @@ public static void main(String[] args) throws Exception {
for (String queryId : VALIDATE_QUERIES) {
File expectedFile = new File(expectedDir, queryId + RESULT_SUFFIX);
- File actualFile = new File(actualDir, queryId + RESULT_SUFFIX);
+ File actualFileDirectory = new File(actualDir, queryId + RESULT_SUFFIX);
+ File actualFile = Objects.requireNonNull(actualFileDirectory.listFiles())[0];
Review comment:
the output of old csv sink is a file, but for new csv sink, output is a directory. so we need add a step to get the actualFile.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #19152: [FLINK-26692][e2e] migrate TpcdsTestProgram.java to new source
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #19152:
URL: https://github.com/apache/flink/pull/19152#issuecomment-1072298017
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "3342d35de40f5ef19c29dd4ba8d26be76e5319eb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=33343",
"triggerID" : "3342d35de40f5ef19c29dd4ba8d26be76e5319eb",
"triggerType" : "PUSH"
}, {
"hash" : "14454d0b4f4047b1bfb5f71d131460e7bc6987e8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=33377",
"triggerID" : "14454d0b4f4047b1bfb5f71d131460e7bc6987e8",
"triggerType" : "PUSH"
}, {
"hash" : "a3674835e935ea4ed2e413b1793832ebc4fe7349",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=33390",
"triggerID" : "a3674835e935ea4ed2e413b1793832ebc4fe7349",
"triggerType" : "PUSH"
}, {
"hash" : "1c77abc2997c2682e540f8a544c13a7e0dd9675a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=33396",
"triggerID" : "1c77abc2997c2682e540f8a544c13a7e0dd9675a",
"triggerType" : "PUSH"
}, {
"hash" : "a1e060857ac5d00775554b1c300b1f6eebf8b30c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=33403",
"triggerID" : "a1e060857ac5d00775554b1c300b1f6eebf8b30c",
"triggerType" : "PUSH"
}, {
"hash" : "aebb4eeba15978ae17c1c8194287548b781ee0c1",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=33475",
"triggerID" : "aebb4eeba15978ae17c1c8194287548b781ee0c1",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* aebb4eeba15978ae17c1c8194287548b781ee0c1 Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=33475)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #19152: [FLINK-26692][e2e] migrate TpcdsTestProgram.java to new source
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #19152:
URL: https://github.com/apache/flink/pull/19152#issuecomment-1072298017
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "3342d35de40f5ef19c29dd4ba8d26be76e5319eb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=33343",
"triggerID" : "3342d35de40f5ef19c29dd4ba8d26be76e5319eb",
"triggerType" : "PUSH"
}, {
"hash" : "14454d0b4f4047b1bfb5f71d131460e7bc6987e8",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=33377",
"triggerID" : "14454d0b4f4047b1bfb5f71d131460e7bc6987e8",
"triggerType" : "PUSH"
}, {
"hash" : "a3674835e935ea4ed2e413b1793832ebc4fe7349",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=33390",
"triggerID" : "a3674835e935ea4ed2e413b1793832ebc4fe7349",
"triggerType" : "PUSH"
}, {
"hash" : "1c77abc2997c2682e540f8a544c13a7e0dd9675a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "1c77abc2997c2682e540f8a544c13a7e0dd9675a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 14454d0b4f4047b1bfb5f71d131460e7bc6987e8 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=33377)
* a3674835e935ea4ed2e413b1793832ebc4fe7349 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=33390)
* 1c77abc2997c2682e540f8a544c13a7e0dd9675a UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org