You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pulsar.apache.org by Apache Pulsar Slack <ap...@gmail.com> on 2020/09/17 09:11:03 UTC
Slack digest for #dev - 2020-09-17

2020-09-16 13:17:05 UTC - sindhushree: Hi ,
we are using pulsar flink connector and getting the millions of offset in the trace logs .and leading to state restoration error .
2020-09-16 11:44:39.764 [Source: opr_event -&gt;
c9ee23b has remote state SubtaskState{operatorStateFromBackend=StateObjectCollection{[OperatorStateHandle{stateNameToPartitionOffsets={topic-partition-offset-states_subName=StateMetaInfo{offsets=[1106, 1127 
, 1148, 1169, 1190, 1211, 1232, 1253, 1274, 1295, 1316, 1337, 1358, 1379, 1400, 1421, 1442, 1463, 1484, 1505, 1526, 1547, 1568, 1589, 1610, 1631, 1652, 1673, 1694, 1715, 1736, 1757, 1778, 1799, 1820, 
 
   it has like millions of offsets . and we are not able to restore.
----
2020-09-16 15:11:11 UTC - hangc: you may turn the flink log level from trace to info, and check the state storage.
----
2020-09-16 17:53:43 UTC - sindhushree: Am unable to store the state back because of large frame size is the error am getting after this .
----
2020-09-16 19:40:56 UTC - sindhushree: Does at all flink purges its offset ?
----
2020-09-16 20:48:30 UTC - Rich Adams: @Rich Adams has joined the channel
----
2020-09-16 22:04:13 UTC - Muljadi: When will Zookeeper removed from Pulsar?
----
2020-09-16 22:14:45 UTC - Rich Adams: Does anyone know if the decision to use prestosql was intentional? Seems like most documentation I read points at prestodb, but looking at what is going into master pulsar is going with prestosql. I don’t know a ton about the presto community, but based on what I have seen the SPIs are not compatible:
- <https://github.com/prestodb/presto/blob/master/presto-spi/src/main/java/com/facebook/presto/spi/Plugin.java>
- <https://github.com/prestosql/presto/blob/master/presto-spi/src/main/java/io/prestosql/spi/Plugin.java>
----
2020-09-16 22:48:23 UTC - Addison Higham: prestosql is a fork from prestodb, there was some drama around the original stewardship of prestodb and prestosql was formed with a more clear non profit foundation, and hence is why pulsar initially went with prestosql as that seemed to be where the community was headed.

In the meantime, prestodb has now been itself is now in a better position with a new foundation being created to act as stewards.

(see <https://blog.openbridge.com/prestodb-vs-prestosql-the-new-presto-foundation-341405fb7ba3> for some good background)

At this point, AFAICT, it isn't entirely clear what the best step forward is or if there will be some unification between the two projects.

I think moving to prestodb is certainly something that would be considered if that is where all the community ends up, but for now it is somewhat of a sticky situation :confused:
----
2020-09-16 23:08:44 UTC - Rich Adams: Yeah, looks fairly dramatic.  So Pulsar switched to prestosql in June, there isn’t actually a released version of Pulsar with prestosql in it right? I think v2.6.0 and v2.6.1 both use presto v0.206 which was still in the com.facebook package.
----
2020-09-17 07:57:37 UTC - Linton: @Linton has joined the channel
----