You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@seatunnel.apache.org by GitBox <gi...@apache.org> on 2023/01/03 09:31:51 UTC
[GitHub] [incubator-seatunnel] TyrantLucifer opened a new issue, #3271: [Umbrella][Feature][Connector-V2] Support create table automaticly
TyrantLucifer opened a new issue, #3271:
URL: https://github.com/apache/incubator-seatunnel/issues/3271
### Search before asking
- [X] I had searched in the [feature](https://github.com/apache/incubator-seatunnel/issues?q=is%3Aissue+label%3A%22Feature%22) and found no similar feature requirement.
### Description
Now, many sink connectors can not create target table if table not exists. This feature is very important for many users.
### Usage Scenario
_No response_
### Related issues
_No response_
### Are you willing to submit a PR?
- [X] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] ashulin commented on issue #3271: [Umbrella][Feature][Connector-V2] Support create table automaticly
Posted by GitBox <gi...@apache.org>.
ashulin commented on issue #3271:
URL: https://github.com/apache/incubator-seatunnel/issues/3271#issuecomment-1307143864
You can add a method using catalog.
```java
// org.apache.seatunnel.api.table.catalog.Catalog
public interface Catalog {
void createTable(TablePath tablePath, CatalogTable table, boolean ignoreIfExists);
}
```
CatalogTable can represent partitioning keys, primary keys, comments, and more.
But the factory class is not enabled for the entire process: #2490
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] TyrantLucifer commented on issue #3271: [Umbrella][Feature][Connector-V2] Support create table automaticly
Posted by GitBox <gi...@apache.org>.
TyrantLucifer commented on issue #3271:
URL: https://github.com/apache/incubator-seatunnel/issues/3271#issuecomment-1369569602
## Overview
As we know, the ability to automate table before writing out data is important to many users. Mapping of types is often required for automatic table creation, fortunately, this is one of our strengths, so I proposal this feature in community.
## Design
In sink connector, The data type is obtained after the `SeaTunnelSink#setTypeInfo`, so we can add a new life cycle in `SeaTunnelSink` to do this thing.
```java
default void createTable(SeaTunnelRowType seaTunnelRowType) {
// do nothing
}
```
Override and implement this method yourself for different connectors and execute it in starter module:
```java
public List<DataStream<Row>> execute(List<DataStream<Row>> upstreamDataStreams) throws TaskExecuteException {
DataStream<Row> input = upstreamDataStreams.get(0);
for (int i = 0; i < plugins.size(); i++) {
Config sinkConfig = pluginConfigs.get(i);
SeaTunnelSink<SeaTunnelRow, Serializable, Serializable, Serializable> seaTunnelSink = plugins.get(i);
DataStream<Row> stream = fromSourceTable(sinkConfig).orElse(input);
seaTunnelSink.setTypeInfo((SeaTunnelRowType) TypeConverterUtils.convert(stream.getType()));
seaTunnelSink.createTable((SeaTunnelRowType) TypeConverterUtils.convert(stream.getType()));
DataStreamSink<Row> dataStreamSink = stream.sinkTo(new FlinkSink<>(seaTunnelSink)).name(seaTunnelSink.getPluginName());
if (sinkConfig.hasPath(SourceCommonOptions.PARALLELISM.key())) {
int parallelism = sinkConfig.getInt(SourceCommonOptions.PARALLELISM.key());
dataStreamSink.setParallelism(parallelism);
}
}
// the sink is the last stream
return null;
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] github-actions[bot] commented on issue #3271: [Umbrella][Feature][Connector-V2] Support create table automaticly
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #3271:
URL: https://github.com/apache/incubator-seatunnel/issues/3271#issuecomment-1419984996
This issue has been automatically marked as stale because it has not had recent activity for 30 days. It will be closed in next 7 days if no further activity occurs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] github-actions[bot] commented on issue #3271: [Umbrella][Feature][Connector-V2] Support create table automaticly
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on issue #3271:
URL: https://github.com/apache/incubator-seatunnel/issues/3271#issuecomment-1363464736
This issue has been closed because it has not received response for too long time. You could reopen it if you encountered similar problems in the future.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] github-actions[bot] commented on issue #3271: [Umbrella][Feature][Connector-V2] Support create table automaticly
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #3271:
URL: https://github.com/apache/incubator-seatunnel/issues/3271#issuecomment-1430573596
This issue has been closed because it has not received response for too long time. You could reopen it if you encountered similar problems in the future.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] github-actions[bot] commented on issue #3271: [Umbrella][Feature][Connector-V2] Support create table automaticly
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on issue #3271:
URL: https://github.com/apache/incubator-seatunnel/issues/3271#issuecomment-1352397185
This issue has been automatically marked as stale because it has not had recent activity for 30 days. It will be closed in next 7 days if no further activity occurs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] github-actions[bot] closed issue #3271: [Umbrella][Feature][Connector-V2] Support create table automaticly
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #3271: [Umbrella][Feature][Connector-V2] Support create table automaticly
URL: https://github.com/apache/incubator-seatunnel/issues/3271
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] cason0126 commented on issue #3271: [Umbrella][Feature][Connector-V2] Support create table automaticly
Posted by GitBox <gi...@apache.org>.
cason0126 commented on issue #3271:
URL: https://github.com/apache/incubator-seatunnel/issues/3271#issuecomment-1314125088
To solve this problem, we should first list a default policy that How SeaTunnel handle with data type mapping between Source and Sink ， SeaTunnel should allow users to customize them.
Because sometimes there will be some datatype in Source Side But Sink side does not support 。
For example, the GEOM I met recently ，but not support IN HIVE
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] github-actions[bot] closed issue #3271: [Umbrella][Feature][Connector-V2] Support create table automaticly
Posted by GitBox <gi...@apache.org>.
github-actions[bot] closed issue #3271: [Umbrella][Feature][Connector-V2] Support create table automaticly
URL: https://github.com/apache/incubator-seatunnel/issues/3271
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org