You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@seatunnel.apache.org by "FlechazoW (via GitHub)" <gi...@apache.org> on 2023/07/25 12:18:59 UTC

[GitHub] [seatunnel] FlechazoW opened a new pull request, #5153: [Feature][Connector V2][File] Add config of 'file_filter_pattern', which used for filtering files.

FlechazoW opened a new pull request, #5153:
URL: https://github.com/apache/seatunnel/pull/5153

   Add config of 'file_filter_pattern', which used for filtering files.
   
   WIP:
   miss doc.
   miss e2e.
   
   <!--
   
   Thank you for contributing to SeaTunnel! Please make sure that your code changes
   are covered with tests. And in case of new features or big changes
   remember to adjust the documentation.
   
   Feel free to ping committers for the review!
   
   ## Contribution Checklist
   
     - Make sure that the pull request corresponds to a [GITHUB issue](https://github.com/apache/seatunnel/issues).
   
     - Name the pull request in the form "[Feature] [component] Title of the pull request", where *Feature* can be replaced by `Hotfix`, `Bug`, etc.
   
     - Minor fixes should be named following this pattern: `[hotfix] [docs] Fix typo in README.md doc`.
   
   -->
   
   ## Purpose of this pull request
   
   <!-- Describe the purpose of this pull request. For example: This pull request adds checkstyle plugin.-->
   
   ## Check list
   
   * [ ] Code changed are covered with tests, or it does not need tests for reason:
   * [ ] If any new Jar binary package adding in your PR, please add License Notice according
     [New License Guide](https://github.com/apache/seatunnel/blob/dev/docs/en/contribution/new-license.md)
   * [ ] If necessary, please update the documentation to describe the new feature. https://github.com/apache/seatunnel/tree/dev/docs
   * [ ] If you are contributing the connector code, please check that the following files are updated:
     1. Update change log that in connector document. For more details you can refer to [connector-v2](https://github.com/apache/seatunnel/tree/dev/docs/en/connector-v2)
     2. Update [plugin-mapping.properties](https://github.com/apache/seatunnel/blob/dev/plugin-mapping.properties) and add new connector information in it
     3. Update the pom file of [seatunnel-dist](https://github.com/apache/seatunnel/blob/dev/seatunnel-dist/pom.xml)
   * [ ] Update the [`release-note`](https://github.com/apache/seatunnel/blob/dev/release-note.md).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [seatunnel] TyrantLucifer commented on a diff in pull request #5153: [Feature][Connector V2][File] Add config of 'file_filter_pattern', which used for filtering files.

Posted by "TyrantLucifer (via GitHub)" <gi...@apache.org>.
TyrantLucifer commented on code in PR #5153:
URL: https://github.com/apache/seatunnel/pull/5153#discussion_r1278665313


##########
seatunnel-connectors-v2/connector-file/connector-file-base/src/main/java/org/apache/seatunnel/connectors/seatunnel/file/source/reader/AbstractReadStrategy.java:
##########
@@ -146,6 +151,12 @@ public List<String> getFileNamesByPath(HadoopConf hadoopConf, String path) throw
                 }
             }
         }
+
+        if (fileNames.isEmpty()) {
+            throw new IllegalArgumentException(

Review Comment:
   throw unified exception



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [seatunnel] TyrantLucifer commented on a diff in pull request #5153: [Feature][Connector V2][File] Add config of 'file_filter_pattern', which used for filtering files.

Posted by "TyrantLucifer (via GitHub)" <gi...@apache.org>.
TyrantLucifer commented on code in PR #5153:
URL: https://github.com/apache/seatunnel/pull/5153#discussion_r1277418600


##########
seatunnel-connectors-v2/connector-file/connector-file-base/src/main/java/org/apache/seatunnel/connectors/seatunnel/file/config/BaseSourceConfig.java:
##########
@@ -112,4 +112,11 @@ public class BaseSourceConfig {
                     .stringType()
                     .noDefaultValue()
                     .withDescription("To be read sheet name,only valid for excel files");
+
+    public static final Option<String> FILE_FILTER_PATTERN =
+            Options.key("file_filter_pattern")
+                    .stringType()
+                    .defaultValue("*")

Review Comment:
   +1, agree with @hailin0 , this is an optional parameter, should do nothing when this parameter is empty.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [seatunnel] FlechazoW commented on pull request #5153: [Feature][Connector V2][File] Add config of 'file_filter_pattern', which used for filtering files.

Posted by "FlechazoW (via GitHub)" <gi...@apache.org>.
FlechazoW commented on PR #5153:
URL: https://github.com/apache/seatunnel/pull/5153#issuecomment-1649732338

   Issue ping #5098


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [seatunnel] hailin0 commented on a diff in pull request #5153: [Feature][Connector V2][File] Add config of 'file_filter_pattern', which used for filtering files.

Posted by "hailin0 (via GitHub)" <gi...@apache.org>.
hailin0 commented on code in PR #5153:
URL: https://github.com/apache/seatunnel/pull/5153#discussion_r1277367375


##########
seatunnel-connectors-v2/connector-file/connector-file-base/src/main/java/org/apache/seatunnel/connectors/seatunnel/file/config/BaseSourceConfig.java:
##########
@@ -112,4 +112,11 @@ public class BaseSourceConfig {
                     .stringType()
                     .noDefaultValue()
                     .withDescription("To be read sheet name,only valid for excel files");
+
+    public static final Option<String> FILE_FILTER_PATTERN =
+            Options.key("file_filter_pattern")
+                    .stringType()
+                    .defaultValue("*")

Review Comment:
   unused default value?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [seatunnel] FlechazoW commented on pull request #5153: [Feature][Connector V2][File] Add config of 'file_filter_pattern', which used for filtering files.

Posted by "FlechazoW (via GitHub)" <gi...@apache.org>.
FlechazoW commented on PR #5153:
URL: https://github.com/apache/seatunnel/pull/5153#issuecomment-1655052332

   PTAL. @TaoZex @TyrantLucifer 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [seatunnel] TyrantLucifer commented on pull request #5153: [Feature][Connector V2][File] Add config of 'file_filter_pattern', which used for filtering files.

Posted by "TyrantLucifer (via GitHub)" <gi...@apache.org>.
TyrantLucifer commented on PR #5153:
URL: https://github.com/apache/seatunnel/pull/5153#issuecomment-1652868822

   > Do we need to add e2e test for special cases? For example, none of the files meet the filtering requirements.
   
   Recently, if file list is empty, connector will throw exception. So I think it does not make sense.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [seatunnel] hailin0 merged pull request #5153: [Feature][Connector V2][File] Add config of 'file_filter_pattern', which used for filtering files.

Posted by "hailin0 (via GitHub)" <gi...@apache.org>.
hailin0 merged PR #5153:
URL: https://github.com/apache/seatunnel/pull/5153


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [seatunnel] TaoZex commented on pull request #5153: [Feature][Connector V2][File] Add config of 'file_filter_pattern', which used for filtering files.

Posted by "TaoZex (via GitHub)" <gi...@apache.org>.
TaoZex commented on PR #5153:
URL: https://github.com/apache/seatunnel/pull/5153#issuecomment-1652810818

   good pr!CC @TyrantLucifer 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [seatunnel] FlechazoW commented on a diff in pull request #5153: [Feature][Connector V2][File] Add config of 'file_filter_pattern', which used for filtering files.

Posted by "FlechazoW (via GitHub)" <gi...@apache.org>.
FlechazoW commented on code in PR #5153:
URL: https://github.com/apache/seatunnel/pull/5153#discussion_r1278227146


##########
seatunnel-connectors-v2/connector-file/connector-file-base/src/main/java/org/apache/seatunnel/connectors/seatunnel/file/config/BaseSourceConfig.java:
##########
@@ -112,4 +112,11 @@ public class BaseSourceConfig {
                     .stringType()
                     .noDefaultValue()
                     .withDescription("To be read sheet name,only valid for excel files");
+
+    public static final Option<String> FILE_FILTER_PATTERN =
+            Options.key("file_filter_pattern")
+                    .stringType()
+                    .defaultValue("*")

Review Comment:
   done.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [seatunnel] FlechazoW commented on a diff in pull request #5153: [Feature][Connector V2][File] Add config of 'file_filter_pattern', which used for filtering files.

Posted by "FlechazoW (via GitHub)" <gi...@apache.org>.
FlechazoW commented on code in PR #5153:
URL: https://github.com/apache/seatunnel/pull/5153#discussion_r1278674946


##########
seatunnel-connectors-v2/connector-file/connector-file-base/src/main/java/org/apache/seatunnel/connectors/seatunnel/file/source/reader/AbstractReadStrategy.java:
##########
@@ -146,6 +151,12 @@ public List<String> getFileNamesByPath(HadoopConf hadoopConf, String path) throw
                 }
             }
         }
+
+        if (fileNames.isEmpty()) {
+            throw new IllegalArgumentException(

Review Comment:
   done.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org