You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by GitBox <gi...@apache.org> on 2022/07/21 10:47:34 UTC
[GitHub] [doris] carlvinhust2012 opened a new pull request, #11082: [regression] (array-type) add the cases for csv/orc/parquet file format
carlvinhust2012 opened a new pull request, #11082:
URL: https://github.com/apache/doris/pull/11082
# Proposed changes
1. this pr is used to add the cases for csv/orc/parquet file format.
Issue Number: close #xxx
## Problem Summary:
Describe the overview of changes.
## Checklist(Required)
1. Does it affect the original behavior: (No)
2. Has unit tests been added: (No)
3. Has document been added or modified: (No)
4. Does it need to update dependencies: (No)
5. Are there any changes that cannot be rolled back: (No)
## Further comments
If this is a relatively large or complex change, kick off the discussion at [dev@doris.apache.org](mailto:dev@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] carlvinhust2012 commented on a diff in pull request #11082: [regression] (array-type) add the cases for csv/orc/parquet file format
Posted by GitBox <gi...@apache.org>.
carlvinhust2012 commented on code in PR #11082:
URL: https://github.com/apache/doris/pull/11082#discussion_r927317709
##########
regression-test/suites/load/broker_load/test_array_load.groovy:
##########
@@ -116,13 +137,30 @@ suite("test_array_load", "load") {
assertTrue(result1[0][0] == 0, "Query OK, 0 rows affected")
}
+ def check_load_result = {checklabel, testTablex ->
+ max_try_micro_secs = 10000
Review Comment:
this is a good suggestion. I will update it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] carlvinhust2012 commented on a diff in pull request #11082: [regression] (array-type) add the cases for csv/orc/parquet file format
Posted by GitBox <gi...@apache.org>.
carlvinhust2012 commented on code in PR #11082:
URL: https://github.com/apache/doris/pull/11082#discussion_r927320538
##########
regression-test/suites/load/broker_load/test_array_load.groovy:
##########
@@ -116,13 +137,30 @@ suite("test_array_load", "load") {
assertTrue(result1[0][0] == 0, "Query OK, 0 rows affected")
}
+ def check_load_result = {checklabel, testTablex ->
+ max_try_micro_secs = 10000
+ while(max_try_micro_secs--) {
Review Comment:
yes, you are right.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] yiguolei merged pull request #11082: [regression] (array-type) add the cases for csv/orc/parquet file format
Posted by GitBox <gi...@apache.org>.
yiguolei merged PR #11082:
URL: https://github.com/apache/doris/pull/11082
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #11082: [regression] (array-type) add the cases for csv/orc/parquet file format
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on PR #11082:
URL: https://github.com/apache/doris/pull/11082#issuecomment-1192318551
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] cambyzju commented on pull request #11082: [regression] (array-type) add the cases for csv/orc/parquet file format
Posted by GitBox <gi...@apache.org>.
cambyzju commented on PR #11082:
URL: https://github.com/apache/doris/pull/11082#issuecomment-1192122877
LGTM
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] carlvinhust2012 commented on a diff in pull request #11082: [regression] (array-type) add the cases for csv/orc/parquet file format
Posted by GitBox <gi...@apache.org>.
carlvinhust2012 commented on code in PR #11082:
URL: https://github.com/apache/doris/pull/11082#discussion_r927321325
##########
regression-test/suites/load/broker_load/test_array_load.groovy:
##########
@@ -116,13 +137,30 @@ suite("test_array_load", "load") {
assertTrue(result1[0][0] == 0, "Query OK, 0 rows affected")
}
+ def check_load_result = {checklabel, testTablex ->
+ max_try_micro_secs = 10000
+ while(max_try_micro_secs--) {
+ result = sql "show load where label = '${checklabel}'"
+ if(result[0][2] == "FINISHED") {
+ qt_select "select * from ${testTablex} order by k1"
+ break
+ } else {
+ sleep(1000) // wait 1 second every time
+ max_try_micro_secs = max_try_micro_secs - 1000
+ if(max_try_micro_secs < 0) {
Review Comment:
if no need "max_try_micro_secs--", I think it should be 'max_try_micro_secs <= 0'.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #11082: [regression] (array-type) add the cases for csv/orc/parquet file format
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on PR #11082:
URL: https://github.com/apache/doris/pull/11082#issuecomment-1192318571
PR approved by anyone and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] xy720 commented on a diff in pull request #11082: [regression] (array-type) add the cases for csv/orc/parquet file format
Posted by GitBox <gi...@apache.org>.
xy720 commented on code in PR #11082:
URL: https://github.com/apache/doris/pull/11082#discussion_r927303526
##########
regression-test/suites/load/broker_load/test_array_load.groovy:
##########
@@ -116,13 +137,30 @@ suite("test_array_load", "load") {
assertTrue(result1[0][0] == 0, "Query OK, 0 rows affected")
}
+ def check_load_result = {checklabel, testTablex ->
+ max_try_micro_secs = 10000
Review Comment:
```suggestion
max_try_milli_secs = 10000
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] xy720 commented on a diff in pull request #11082: [regression] (array-type) add the cases for csv/orc/parquet file format
Posted by GitBox <gi...@apache.org>.
xy720 commented on code in PR #11082:
URL: https://github.com/apache/doris/pull/11082#discussion_r927300894
##########
regression-test/suites/load/broker_load/test_array_load.groovy:
##########
@@ -116,13 +137,30 @@ suite("test_array_load", "load") {
assertTrue(result1[0][0] == 0, "Query OK, 0 rows affected")
}
+ def check_load_result = {checklabel, testTablex ->
+ max_try_micro_secs = 10000
Review Comment:
```suggestion
max_try_micro_milli_secs = 10000
```
##########
regression-test/suites/load/broker_load/test_array_load.groovy:
##########
@@ -101,7 +102,27 @@ suite("test_array_load", "load") {
def result1= sql """
LOAD LABEL ${label} (
DATA INFILE("${hdfsFilePath}")
- INTO TABLE ${testTablex}
+ INTO TABLE ${testTablex}
+ FORMAT as "${format}")
+ with BROKER "${brokerName}" (
+ "username"="${hdfsUser}",
+ "password"="${hdfsPasswd}")
+ PROPERTIES (
+ "timeout"="1200",
+ "max_filter_ratio"="0.1");
+ """
+
+ assertTrue(result1.size() == 1)
+ assertTrue(result1[0].size() == 1)
+ assertTrue(result1[0][0] == 0, "Query OK, 0 rows affected")
+ }
+
+ def load_from_hdfs1 = {testTablex, label, hdfsFilePath, format, brokerName, hdfsUser, hdfsPasswd ->
+ def result1= sql """
+ LOAD LABEL ${label} (
+ DATA INFILE("${hdfsFilePath}")
+ INTO TABLE ${testTablex}
+ COLUMNS TERMINATED BY "/"
Review Comment:
Pass column sep as a parameter.
##########
regression-test/suites/load/broker_load/test_array_load.groovy:
##########
@@ -116,13 +137,30 @@ suite("test_array_load", "load") {
assertTrue(result1[0][0] == 0, "Query OK, 0 rows affected")
}
+ def check_load_result = {checklabel, testTablex ->
+ max_try_micro_secs = 10000
+ while(max_try_micro_secs--) {
Review Comment:
"--" is no need.
##########
regression-test/suites/load/broker_load/test_array_load.groovy:
##########
@@ -116,13 +137,30 @@ suite("test_array_load", "load") {
assertTrue(result1[0][0] == 0, "Query OK, 0 rows affected")
}
+ def check_load_result = {checklabel, testTablex ->
+ max_try_micro_secs = 10000
+ while(max_try_micro_secs--) {
+ result = sql "show load where label = '${checklabel}'"
+ if(result[0][2] == "FINISHED") {
+ qt_select "select * from ${testTablex} order by k1"
+ break
+ } else {
+ sleep(1000) // wait 1 second every time
+ max_try_micro_secs = max_try_micro_secs - 1000
+ if(max_try_micro_secs < 0) {
Review Comment:
why not max_try_micro_secs <= 0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] carlvinhust2012 commented on a diff in pull request #11082: [regression] (array-type) add the cases for csv/orc/parquet file format
Posted by GitBox <gi...@apache.org>.
carlvinhust2012 commented on code in PR #11082:
URL: https://github.com/apache/doris/pull/11082#discussion_r927317267
##########
regression-test/suites/load/broker_load/test_array_load.groovy:
##########
@@ -101,7 +102,27 @@ suite("test_array_load", "load") {
def result1= sql """
LOAD LABEL ${label} (
DATA INFILE("${hdfsFilePath}")
- INTO TABLE ${testTablex}
+ INTO TABLE ${testTablex}
+ FORMAT as "${format}")
+ with BROKER "${brokerName}" (
+ "username"="${hdfsUser}",
+ "password"="${hdfsPasswd}")
+ PROPERTIES (
+ "timeout"="1200",
+ "max_filter_ratio"="0.1");
+ """
+
+ assertTrue(result1.size() == 1)
+ assertTrue(result1[0].size() == 1)
+ assertTrue(result1[0][0] == 0, "Query OK, 0 rows affected")
+ }
+
+ def load_from_hdfs1 = {testTablex, label, hdfsFilePath, format, brokerName, hdfsUser, hdfsPasswd ->
+ def result1= sql """
+ LOAD LABEL ${label} (
+ DATA INFILE("${hdfsFilePath}")
+ INTO TABLE ${testTablex}
+ COLUMNS TERMINATED BY "/"
Review Comment:
I have test this but it not work.
Because 'COLUMNS TERMINATED BY' can't recognize the incoming parameter.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] hqx871 commented on pull request #11082: [regression] (array-type) add the cases for csv/orc/parquet file format
Posted by GitBox <gi...@apache.org>.
hqx871 commented on PR #11082:
URL: https://github.com/apache/doris/pull/11082#issuecomment-1252140197
hi @whutpencil, does broker load support array type?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org