You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@flink.apache.org by Aviral Srivastava <av...@gmail.com> on 2020/07/26 22:02:41 UTC

Unable to deduce RocksDB api calls in streaming.

Hi all!

I want to profile the time taken to make snapshot calls to RocksDB when
using Flink in streaming mode.

I have forked the flink core repo, added the example of fraud detection,
configured the state backend and checkpointing. The program is running
successfully.

I have also been able to import the code base into IntelliJ IDE and debug
it. However, I could not trace the function calls to the RocksDB. Is there
any documentation that would help me out in this regard? Or can someone
help me here?

Corresponding question on SO:
https://stackoverflow.com/questions/63103295/what-are-the-api-calls-made-to-create-snapshot-while-using-rocksdb-as-the-state

Best,
Aviral Srivastava
LinkedIn <https://www.linkedin.com/in/sriavi/> | Website
<https://aviralsrivastava.com/>

Re: Unable to deduce RocksDB api calls in streaming.

Posted by Timo Walther <tw...@apache.org>.

Hi Aviral,

as far as I know we are not calling RocksDB API to perform snapshots. As 
the Stackoverflow answer also indicates most of the snapshotting is done 
outside of RocksDB by just dealing with the SST files. Have you checked 
the available metrics in the web UI?

https://ci.apache.org/projects/flink/flink-docs-master/monitoring/checkpoint_monitoring.html

But maybe a RocksDB expert (in CC) knows more about this topic.

Regards,
Timo

On 27.07.20 00:02, Aviral Srivastava wrote:
> Hi all!
> 
> I want to profile the time taken to make snapshot calls to RocksDB when 
> using Flink in streaming mode.
> 
> I have forked the flink core repo, added the example of fraud detection, 
> configured the state backend and checkpointing. The program is running 
> successfully.
> 
> I have also been able to import the code base into IntelliJ IDE and 
> debug it. However, I could not trace the function calls to the RocksDB. 
> Is there any documentation that would help me out in this regard? Or can 
> someone help me here?
> 
> Corresponding question on SO: 
> https://stackoverflow.com/questions/63103295/what-are-the-api-calls-made-to-create-snapshot-while-using-rocksdb-as-the-state
> 
> Best,
> Aviral Srivastava
> LinkedIn <https://www.linkedin.com/in/sriavi/> | Website 
> <https://aviralsrivastava.com/>