You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by GitBox <gi...@apache.org> on 2021/06/14 07:17:42 UTC

[GitHub] [iceberg] WenfengJ opened a new issue #2693: flink sink to iceberg with different data 

WenfengJ opened a new issue #2693:
URL: https://github.com/apache/iceberg/issues/2693


   Hi,
   I use  Flinksink to iceberg table,if data are offline data,the table is ok, but if data is realtime ,for example .
   kafka,mysql cdc.....,the table created ,but data not commit . 
   
   
   ![微信截图_20210611135925](https://user-images.githubusercontent.com/34698232/121638628-dc626e00-cabd-11eb-821f-ec7f5f5786d3.png)
   
   ![2](https://user-images.githubusercontent.com/34698232/121638623-da001400-cabd-11eb-9fff-5e56e4f878f0.png)
   ![3](https://user-images.githubusercontent.com/34698232/121638624-db314100-cabd-11eb-8090-9dbe33799e67.png)
   ![4](https://user-images.githubusercontent.com/34698232/121638627-dbc9d780-cabd-11eb-9d0b-96cca1756dd9.png)
   
   
   ```
      env.setStreamTimeCharacteristic(TimeCharacteristic.EventTime);
           env.enableCheckpointing(6000);
           env.setMaxParallelism(1);
   
   ......   (source is realtime data or offline data )
   
        FlinkSink.
                   forRow(source, SimpleDataUtil.FLINK_SCHEMA)
                   .table(table)
                   .tableLoader(tableLoader)
                   .overwrite(false)
                   .tableSchema(SimpleDataUtil.FLINK_SCHEMA)
                   .writeParallelism(1)
                   .build();  
   ```
   As shown in the figure:


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org

[GitHub] [iceberg] openinx closed issue #2693: flink sink to iceberg with different data

Posted by GitBox <gi...@apache.org>.
openinx closed issue #2693:
URL: https://github.com/apache/iceberg/issues/2693


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org

[GitHub] [iceberg] openinx commented on issue #2693: flink sink to iceberg with different data

Posted by GitBox <gi...@apache.org>.
openinx commented on issue #2693:
URL: https://github.com/apache/iceberg/issues/2693#issuecomment-869658838


   > Have you turned on the checkpoint?
   
   Yeah, I think you need to enable the flink checkpoint. if you are submitting the flink by datastream job, then you will need to follow this [document](https://ci.apache.org/projects/flink/flink-docs-master/docs/dev/datastream/fault-tolerance/checkpointing/) to enable your checkpoint.   If you want to use flink sql to write data to iceberg, then you will need to add the following configurations in your `flink-conf.yaml`: 
   
   ```yaml
   # milliseconds.
   execution.checkpointing.interval: 120000                                          
   state.backend: filesystem                                                         
   state.checkpoints.dir: file:///path/to/your/checkpoints                     
   state.savepoints.dir: file:///path/to/your/savepoints
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org

[GitHub] [iceberg] openinx closed issue #2693: flink sink to iceberg with different data

Posted by GitBox <gi...@apache.org>.
openinx closed issue #2693:
URL: https://github.com/apache/iceberg/issues/2693


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org

[GitHub] [iceberg] lltcatl commented on issue #2693: flink sink to iceberg with different data

Posted by GitBox <gi...@apache.org>.
lltcatl commented on issue #2693:
URL: https://github.com/apache/iceberg/issues/2693#issuecomment-869302178


   Have you turned on the checkpoint?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org

[GitHub] [iceberg] openinx commented on issue #2693: flink sink to iceberg with different data

Posted by GitBox <gi...@apache.org>.
openinx commented on issue #2693:
URL: https://github.com/apache/iceberg/issues/2693#issuecomment-869658838


   > Have you turned on the checkpoint?
   
   Yeah, I think you need to enable the flink checkpoint. if you are submitting the flink by datastream job, then you will need to follow this [document](https://ci.apache.org/projects/flink/flink-docs-master/docs/dev/datastream/fault-tolerance/checkpointing/) to enable your checkpoint.   If you want to use flink sql to write data to iceberg, then you will need to add the following configurations in your `flink-conf.yaml`: 
   
   ```yaml
   # milliseconds.
   execution.checkpointing.interval: 120000                                          
   state.backend: filesystem                                                         
   state.checkpoints.dir: file:///path/to/your/checkpoints                     
   state.savepoints.dir: file:///path/to/your/savepoints
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org