You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by KhajaAsmath Mohammed <md...@gmail.com> on 2020/08/15 16:49:58 UTC
Time stamp in Kafka
Hi,
We have a producer application that has written data to Kafka topic.
We are reading the data from Kafka topic using spark streaming but the time stamp on Kafka is 1969-12-31 format for all the data.
Is there a way to fix this while reading ?
Thanks,
Asmath
Sent from my iPhone
Re: Time stamp in Kafka
Posted by KhajaAsmath Mohammed <md...@gmail.com>.
Thanks Liam. This is helpful. I will search how to use that.
Sent from my iPhone
> On Aug 15, 2020, at 4:32 PM, Liam Clarke-Hutchinson <li...@adscale.co.nz> wrote:
>
> One option could be to configure the topic to use log append time as the
> timestamp, not producer timestamp.
>
>> On Sun, 16 Aug. 2020, 7:23 am Matthias J. Sax, <mj...@apache.org> wrote:
>>
>> What is you configs? There are many configs that interact with each other.
>>
>> I did a Kafka Summit talk about this topic that may help:
>>
>> https://www.confluent.io/kafka-summit-san-francisco-2019/whats-the-time-and-why/
>>
>>
>> -Matthias
>>
>>> On 8/15/20 11:48 AM, KhajaAsmath Mohammed wrote:
>>> Hi,
>>>
>>> I don’t have code related to it. It is pushed by other team who uses
>> .net framework to push data into Kafka topic. I am suspecting it is issue
>> with library they use.
>>>
>>> I used java library to push data and it works as expected.
>>>
>>> Thanks,
>>> Asmath
>>>
>>> Sent from my iPhone
>>>
>>>> On Aug 15, 2020, at 1:44 PM, pierreneter <pi...@gmail.com> wrote:
>>>>
>>>> can you show a little of your code related to that?
>>>>
>>>>> On Sat, Aug 15, 2020 at 11:50 PM KhajaAsmath Mohammed <
>>>>> mdkhajaasmath@gmail.com> wrote:
>>>>>
>>>>> Hi,
>>>>>
>>>>> We have a producer application that has written data to Kafka topic.
>>>>>
>>>>> We are reading the data from Kafka topic using spark streaming but the
>>>>> time stamp on Kafka is 1969-12-31 format for all the data.
>>>>>
>>>>> Is there a way to fix this while reading ?
>>>>>
>>>>> Thanks,
>>>>> Asmath
>>>>>
>>>>> Sent from my iPhone
>>
>>
Re: Time stamp in Kafka
Posted by Liam Clarke-Hutchinson <li...@adscale.co.nz>.
One option could be to configure the topic to use log append time as the
timestamp, not producer timestamp.
On Sun, 16 Aug. 2020, 7:23 am Matthias J. Sax, <mj...@apache.org> wrote:
> What is you configs? There are many configs that interact with each other.
>
> I did a Kafka Summit talk about this topic that may help:
>
> https://www.confluent.io/kafka-summit-san-francisco-2019/whats-the-time-and-why/
>
>
> -Matthias
>
> On 8/15/20 11:48 AM, KhajaAsmath Mohammed wrote:
> > Hi,
> >
> > I don’t have code related to it. It is pushed by other team who uses
> .net framework to push data into Kafka topic. I am suspecting it is issue
> with library they use.
> >
> > I used java library to push data and it works as expected.
> >
> > Thanks,
> > Asmath
> >
> > Sent from my iPhone
> >
> >> On Aug 15, 2020, at 1:44 PM, pierreneter <pi...@gmail.com> wrote:
> >>
> >> can you show a little of your code related to that?
> >>
> >>> On Sat, Aug 15, 2020 at 11:50 PM KhajaAsmath Mohammed <
> >>> mdkhajaasmath@gmail.com> wrote:
> >>>
> >>> Hi,
> >>>
> >>> We have a producer application that has written data to Kafka topic.
> >>>
> >>> We are reading the data from Kafka topic using spark streaming but the
> >>> time stamp on Kafka is 1969-12-31 format for all the data.
> >>>
> >>> Is there a way to fix this while reading ?
> >>>
> >>> Thanks,
> >>> Asmath
> >>>
> >>> Sent from my iPhone
>
>
Re: Time stamp in Kafka
Posted by "Matthias J. Sax" <mj...@apache.org>.
What is you configs? There are many configs that interact with each other.
I did a Kafka Summit talk about this topic that may help:
https://www.confluent.io/kafka-summit-san-francisco-2019/whats-the-time-and-why/
-Matthias
On 8/15/20 11:48 AM, KhajaAsmath Mohammed wrote:
> Hi,
>
> I don’t have code related to it. It is pushed by other team who uses .net framework to push data into Kafka topic. I am suspecting it is issue with library they use.
>
> I used java library to push data and it works as expected.
>
> Thanks,
> Asmath
>
> Sent from my iPhone
>
>> On Aug 15, 2020, at 1:44 PM, pierreneter <pi...@gmail.com> wrote:
>>
>> can you show a little of your code related to that?
>>
>>> On Sat, Aug 15, 2020 at 11:50 PM KhajaAsmath Mohammed <
>>> mdkhajaasmath@gmail.com> wrote:
>>>
>>> Hi,
>>>
>>> We have a producer application that has written data to Kafka topic.
>>>
>>> We are reading the data from Kafka topic using spark streaming but the
>>> time stamp on Kafka is 1969-12-31 format for all the data.
>>>
>>> Is there a way to fix this while reading ?
>>>
>>> Thanks,
>>> Asmath
>>>
>>> Sent from my iPhone
Re: Time stamp in Kafka
Posted by KhajaAsmath Mohammed <md...@gmail.com>.
Hi,
I don’t have code related to it. It is pushed by other team who uses .net framework to push data into Kafka topic. I am suspecting it is issue with library they use.
I used java library to push data and it works as expected.
Thanks,
Asmath
Sent from my iPhone
> On Aug 15, 2020, at 1:44 PM, pierreneter <pi...@gmail.com> wrote:
>
> can you show a little of your code related to that?
>
>> On Sat, Aug 15, 2020 at 11:50 PM KhajaAsmath Mohammed <
>> mdkhajaasmath@gmail.com> wrote:
>>
>> Hi,
>>
>> We have a producer application that has written data to Kafka topic.
>>
>> We are reading the data from Kafka topic using spark streaming but the
>> time stamp on Kafka is 1969-12-31 format for all the data.
>>
>> Is there a way to fix this while reading ?
>>
>> Thanks,
>> Asmath
>>
>> Sent from my iPhone
Re: Time stamp in Kafka
Posted by pierreneter <pi...@gmail.com>.
can you show a little of your code related to that?
On Sat, Aug 15, 2020 at 11:50 PM KhajaAsmath Mohammed <
mdkhajaasmath@gmail.com> wrote:
> Hi,
>
> We have a producer application that has written data to Kafka topic.
>
> We are reading the data from Kafka topic using spark streaming but the
> time stamp on Kafka is 1969-12-31 format for all the data.
>
> Is there a way to fix this while reading ?
>
> Thanks,
> Asmath
>
> Sent from my iPhone
Fwd: Time stamp in Kafka
Posted by KhajaAsmath Mohammed <md...@gmail.com>.
Hi,
>
> We have a producer application that has written data to Kafka topic.
>
> We are reading the data from Kafka topic using spark streaming but the time stamp on Kafka is 1969-12-31 format for all the data.
>
> Is there a way to fix this while reading ?
>
> Thanks,
> Asmath
>
---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscribe@spark.apache.org