Apache Pinot Daily Email Digest (2021-10-19)

[Pinot Slack Email Digest <ap...@gmail.com>]

### _#general_

  
 **@piyush.chauhan:** @piyush.chauhan has joined the channel  
 **@chad:** @chad has joined the channel  
 **@r.sachdeva9355:** @r.sachdeva9355 has joined the channel  

###  _#random_

  
 **@piyush.chauhan:** @piyush.chauhan has joined the channel  
 **@chad:** @chad has joined the channel  
 **@r.sachdeva9355:** @r.sachdeva9355 has joined the channel  

###  _#troubleshooting_

  
 **@deemish2:** Hi Team , there is a workaround to fix data gap issue
temporarily -  . Update watermark manually from where the data is available in
realtime  will move data from realtime table to offline table when the task is
scheduled. Is it a feasible way to do this approach  
**@mayanks:** Thanks @deemish2. You may want to add the details on how to
update the watermark manually in the GH issue, for community members who might
not know how to do so.  
**@jackie.jxt:** I'd prefer handling it in the task scheduler. It will be hard
to manage if we expect user to find the gap and manually bump up the watermark  
**@jackie.jxt:** Will take a closer look at the code and see how we can handle
it in the task scheduler. Shouldn't be hard since the data gap is handled
within the merge-rollup task  
 **@piyush.chauhan:** @piyush.chauhan has joined the channel  
 **@chad:** @chad has joined the channel  
 **@r.sachdeva9355:** @r.sachdeva9355 has joined the channel  

###  _#pinot-dev_

  
 **@xiangfu0:** I’m seeing the kinesis integration test failed, seems the
localstack doesn’t start ```Error:
org.apache.pinot.integration.tests.RealtimeKinesisIntegrationTest.setUp Time
elapsed: 175.147 s <<< FAILURE!
cloud.localstack.docker.exception.LocalstackDockerException: Could not start
the localstack docker container. at
cloud.localstack.Localstack.startup(Localstack.java:97) at
org.apache.pinot.integration.tests.RealtimeKinesisIntegrationTest.startKinesis(RealtimeKinesisIntegrationTest.java:224)
at
org.apache.pinot.integration.tests.RealtimeKinesisIntegrationTest.setUp(RealtimeKinesisIntegrationTest.java:135)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native
Method) at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566) at
org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:108)
at org.testng.internal.Invoker.invokeConfigurationMethod(Invoker.java:523) at
org.testng.internal.Invoker.invokeConfigurations(Invoker.java:224) at
org.testng.internal.Invoker.invokeConfigurations(Invoker.java:146) at
org.testng.internal.TestMethodWorker.invokeBeforeClassMethods(TestMethodWorker.java:166)
at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:105) at
org.testng.TestRunner.privateRun(TestRunner.java:744) at
org.testng.TestRunner.run(TestRunner.java:602) at
org.testng.SuiteRunner.runTest(SuiteRunner.java:380) at
org.testng.SuiteRunner.runSequentially(SuiteRunner.java:375) at
org.testng.SuiteRunner.privateRun(SuiteRunner.java:340) at
org.testng.SuiteRunner.run(SuiteRunner.java:289) at
org.testng.SuiteRunnerWorker.runSuite(SuiteRunnerWorker.java:52) at
org.testng.SuiteRunnerWorker.run(SuiteRunnerWorker.java:86) at
org.testng.TestNG.runSuitesSequentially(TestNG.java:1301) at
org.testng.TestNG.runSuitesLocally(TestNG.java:1226) at
org.testng.TestNG.runSuites(TestNG.java:1144) at
org.testng.TestNG.run(TestNG.java:1115) at
org.apache.maven.surefire.testng.TestNGExecutor.run(TestNGExecutor.java:136)
at
org.apache.maven.surefire.testng.TestNGDirectoryTestSuite.executeSingleClass(TestNGDirectoryTestSuite.java:112)
at
org.apache.maven.surefire.testng.TestNGDirectoryTestSuite.execute(TestNGDirectoryTestSuite.java:99)
at
org.apache.maven.surefire.testng.TestNGProvider.invoke(TestNGProvider.java:145)
at
org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:428)
at
org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:162)
at org.apache.maven.surefire.booter.ForkedBooter.run(ForkedBooter.java:562) at
org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:548)
Caused by: java.lang.IllegalStateException: Could not find token: Ready\\. in
Docker logs: 2021-10-19T08:14:44:DEBUG:plugin.manager: instantiating plugin
PluginSpec(localstack.aws.provider.sts:default = <function sts at
0x7fa537678e60>)```  
**@kharekartik:** Hi, yes I also noticed that today. Looking into it.  
 **@kharekartik:** It was working till yesterday so something changed in that
time.  
 **@xiangfu0:** is there any change from github ci ? AlsoI cannot run it from
my local laptop as well  
 **@kharekartik:** Testing in local. Will update you in few minutes.  
 **@kharekartik:** working fine in local. updating docker to latest version to
see if that causes any issue.  
 **@xiangfu0:** I also updated the lib to 0.2.15, seems solved the issue  

###  _#announcements_

  
 **@chad:** @chad has joined the channel  

###  _#roadmap_

  
 **@hardik.chheda:** @hardik.chheda has joined the channel  

###  _#releases_

  
 **@hardik.chheda:** @hardik.chheda has joined the channel  

###  _#product-launch_

  
 **@hardik.chheda:** @hardik.chheda has joined the channel  
\--------------------------------------------------------------------- To
unsubscribe, e-mail: dev-unsubscribe@pinot.apache.org For additional commands,
e-mail: dev-help@pinot.apache.org