You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Hemalatha A <he...@googlemail.com> on 2016/04/16 18:39:26 UTC
Spark streaming batch time displayed is not current system time but
it is processing current messages
Can anyone help me in debugging this issue please.
On Thu, Apr 14, 2016 at 12:24 PM, Hemalatha A <
hemalatha.amrutha@googlemail.com> wrote:
> Hi,
>
> I am facing a problem in Spark streaming.
>
Time: 1460823006000 ms
-------------------------------------------
-------------------------------------------
Time: 1460823008000 ms
-------------------------------------------
> The time displayed in Spark streaming console as above is 4 days prior
> i.e., April 10th, which is not current system time of the cluster but the
> job is processing current messages that is pushed right now April 14th.
>
> Can anyone please advice what time does Spark streaming display? Also,
> when there is scheduling delay of say 8 hours, what time does Spark
> display- current rime or hours behind?
>
> --
>
>
> Regards
> Hemalatha
>
--
Regards
Hemalatha
Re: Spark streaming batch time displayed is not current system time
but it is processing current messages
Posted by Ted Yu <yu...@gmail.com>.
Using
http://www.ruddwire.com/handy-code/date-to-millisecond-calculators/#.VxZh3iMrKuo
, 1460823008000 is shown to be 'Sat Apr 16 2016 09:10:08 GMT-0700'
Can you clarify the 4 day difference ?
bq. 'right now April 14th'
The date of your email was Apr 16th.
On Sat, Apr 16, 2016 at 9:39 AM, Hemalatha A <
hemalatha.amrutha@googlemail.com> wrote:
> Can anyone help me in debugging this issue please.
>
>
> On Thu, Apr 14, 2016 at 12:24 PM, Hemalatha A <
> hemalatha.amrutha@googlemail.com> wrote:
>
>> Hi,
>>
>> I am facing a problem in Spark streaming.
>>
>
>
> Time: 1460823006000 ms
> -------------------------------------------
>
> -------------------------------------------
> Time: 1460823008000 ms
> -------------------------------------------
>
>
>
>
>> The time displayed in Spark streaming console as above is 4 days prior
>> i.e., April 10th, which is not current system time of the cluster but the
>> job is processing current messages that is pushed right now April 14th.
>>
>> Can anyone please advice what time does Spark streaming display? Also,
>> when there is scheduling delay of say 8 hours, what time does Spark
>> display- current rime or hours behind?
>>
>> --
>>
>>
>> Regards
>> Hemalatha
>>
>
>
>
> --
>
>
> Regards
> Hemalatha
>
Re: Spark streaming batch time displayed is not current system time
but it is processing current messages
Posted by Prashant Sharma <sc...@gmail.com>.
This can happen if system time is not in sync. By default, streaming uses
SystemClock(it also supports ManualClock) and that relies
on System.currentTimeMillis() for determining start time.
Prashant Sharma
On Sat, Apr 16, 2016 at 10:09 PM, Hemalatha A <
hemalatha.amrutha@googlemail.com> wrote:
> Can anyone help me in debugging this issue please.
>
>
> On Thu, Apr 14, 2016 at 12:24 PM, Hemalatha A <
> hemalatha.amrutha@googlemail.com> wrote:
>
>> Hi,
>>
>> I am facing a problem in Spark streaming.
>>
>
>
> Time: 1460823006000 ms
> -------------------------------------------
>
> -------------------------------------------
> Time: 1460823008000 ms
> -------------------------------------------
>
>
>
>
>> The time displayed in Spark streaming console as above is 4 days prior
>> i.e., April 10th, which is not current system time of the cluster but the
>> job is processing current messages that is pushed right now April 14th.
>>
>> Can anyone please advice what time does Spark streaming display? Also,
>> when there is scheduling delay of say 8 hours, what time does Spark
>> display- current rime or hours behind?
>>
>> --
>>
>>
>> Regards
>> Hemalatha
>>
>
>
>
> --
>
>
> Regards
> Hemalatha
>