You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Faisal Durrani <te...@gmail.com> on 2018/10/11 08:07:45 UTC
Kafka UnknownProducerIdException
Does anyone know the reason for the below error in Kafka? I have a Golden
gate server publishing change transaction of tables to its corresponding
topics in Kafka (which is running with the default configuration).It is a
multi node cluster but the topic mentioned in the error has a single
partition. Golden gate failed today and kafka shows the below error at the
time of Golden gate failure.
[ReplicaManager broker=1002] Error processing append operation on
partition TABLE-0 (kafka.server.ReplicaManager)
org.apache.kafka.common.errors.UnknownProducerIdException: Found no
record of producerId=96007 on the broker. It is possible that the last
message with the producerId=96007 has been removed due to hitting the
retention limit.
Re: Kafka on Eclipse Setup
Posted by James Kwan <jw...@gmail.com>.
Matthias,
Thanks for your suggestion but it did not work. I finally have to remove all reference of test libraries and just use main. I was able to build all libraries without any errors.
James
> On Oct 15, 2018, at 12:45 PM, Matthias J. Sax <ma...@confluent.io> wrote:
>
> You can try to run "mvn eclipse:eclipse" can restart Eclipse afterwards.
>
> -Matthias
>
> On 10/15/18 8:41 AM, James Kwan wrote:
>> Vahid
>>
>> I have followed the link, but it does not resolve the stream compilation issues. I will try Manna’s suggestion. The issue is not to start a zookeeper or Kafka. I have a Kafka running on my linux vm. I want to create a build free environment so that I can start writing some client and stream codes. I can manually copy all Kafka related jar files but is this the right way? I thought after I download the codes from Github, everything should work.
>>
>> James
>>
>>> On Oct 13, 2018, at 10:08 AM, Vahid Hashemian <va...@gmail.com> wrote:
>>>
>>> This article was written a while back but still could help:
>>> https://developer.ibm.com/opentech/2016/06/06/how-to-develop-for-apache-kafka-using-eclipse/
>>>
>>> Regards!
>>> --Vahid
>>>
>>> On Sat, Oct 13, 2018 at 2:15 AM M. Manna <ma...@gmail.com> wrote:
>>>
>>>> I have done the following and it worked for me.
>>>>
>>>> 1) Manually removed and added the jar locations in eclipse run
>>>> configuration e.g. client, core, etc.
>>>> 2) Added program argument for log4j file and server properties, and
>>>> zookeeper properties.
>>>> 3) Started zookeeper first
>>>> 4) Then started Kafka.
>>>>
>>>> I never had any problem with Zookeeper, so I actually ran
>>>> zookeeper-server-start script to start zookeeper manually. In this way, I
>>>> only need to start Kafka in debug mode from Eclipse.
>>>>
>>>> Let me know if that helps.
>>>>
>>>>
>>>>
>>>> On Sat, 13 Oct 2018 at 02:49, James Kwan <jw...@gmail.com> wrote:
>>>>
>>>>> Has anyone successfully set up an eclipse environment for Kafka on
>>>>> Windows? I have tried many times with different options but still have
>>>>> build issues with stream project. There are always some libraries
>>>>> missing.
>>>>>
>>>>> The steps I used
>>>>> 1) Run git command in command prompt to download the Kafka. I used
>>>>> windows command as well as using eclipse GIT options
>>>>> 2) Run gradle command to build the base.
>>>>> 3) Run gradlew jar to build the jar file
>>>>> 4) Run gradlew eclipse command to build projects
>>>>> 5) In eclipse, use Import/General/Existing Projects into Workspace option
>>>>> to import all projects
>>>>> 6) Remove duplicate source entries in build path for core project
>>>>> 7) Remove test projects that caused not found error message in the build
>>>>> path for core project
>>>>> 8) Remove the dependence of test-utils lib in stream project build path
>>>> to
>>>>> remove the build cycle error.
>>>>> 9) after that I got whole bunch of class or path not defined messages. I
>>>>> don’t know if I need to fix them one by one or if there is any jar files
>>>>> that I need to include.
>>>>>
>>>>>
>>>>> James
>>>>
>>
>
Re: Kafka on Eclipse Setup
Posted by "Matthias J. Sax" <ma...@confluent.io>.
You can try to run "mvn eclipse:eclipse" can restart Eclipse afterwards.
-Matthias
On 10/15/18 8:41 AM, James Kwan wrote:
> Vahid
>
> I have followed the link, but it does not resolve the stream compilation issues. I will try Manna’s suggestion. The issue is not to start a zookeeper or Kafka. I have a Kafka running on my linux vm. I want to create a build free environment so that I can start writing some client and stream codes. I can manually copy all Kafka related jar files but is this the right way? I thought after I download the codes from Github, everything should work.
>
> James
>
>> On Oct 13, 2018, at 10:08 AM, Vahid Hashemian <va...@gmail.com> wrote:
>>
>> This article was written a while back but still could help:
>> https://developer.ibm.com/opentech/2016/06/06/how-to-develop-for-apache-kafka-using-eclipse/
>>
>> Regards!
>> --Vahid
>>
>> On Sat, Oct 13, 2018 at 2:15 AM M. Manna <ma...@gmail.com> wrote:
>>
>>> I have done the following and it worked for me.
>>>
>>> 1) Manually removed and added the jar locations in eclipse run
>>> configuration e.g. client, core, etc.
>>> 2) Added program argument for log4j file and server properties, and
>>> zookeeper properties.
>>> 3) Started zookeeper first
>>> 4) Then started Kafka.
>>>
>>> I never had any problem with Zookeeper, so I actually ran
>>> zookeeper-server-start script to start zookeeper manually. In this way, I
>>> only need to start Kafka in debug mode from Eclipse.
>>>
>>> Let me know if that helps.
>>>
>>>
>>>
>>> On Sat, 13 Oct 2018 at 02:49, James Kwan <jw...@gmail.com> wrote:
>>>
>>>> Has anyone successfully set up an eclipse environment for Kafka on
>>>> Windows? I have tried many times with different options but still have
>>>> build issues with stream project. There are always some libraries
>>>> missing.
>>>>
>>>> The steps I used
>>>> 1) Run git command in command prompt to download the Kafka. I used
>>>> windows command as well as using eclipse GIT options
>>>> 2) Run gradle command to build the base.
>>>> 3) Run gradlew jar to build the jar file
>>>> 4) Run gradlew eclipse command to build projects
>>>> 5) In eclipse, use Import/General/Existing Projects into Workspace option
>>>> to import all projects
>>>> 6) Remove duplicate source entries in build path for core project
>>>> 7) Remove test projects that caused not found error message in the build
>>>> path for core project
>>>> 8) Remove the dependence of test-utils lib in stream project build path
>>> to
>>>> remove the build cycle error.
>>>> 9) after that I got whole bunch of class or path not defined messages. I
>>>> don’t know if I need to fix them one by one or if there is any jar files
>>>> that I need to include.
>>>>
>>>>
>>>> James
>>>
>
Re: Kafka on Eclipse Setup
Posted by James Kwan <jw...@gmail.com>.
Vahid
I have followed the link, but it does not resolve the stream compilation issues. I will try Manna’s suggestion. The issue is not to start a zookeeper or Kafka. I have a Kafka running on my linux vm. I want to create a build free environment so that I can start writing some client and stream codes. I can manually copy all Kafka related jar files but is this the right way? I thought after I download the codes from Github, everything should work.
James
> On Oct 13, 2018, at 10:08 AM, Vahid Hashemian <va...@gmail.com> wrote:
>
> This article was written a while back but still could help:
> https://developer.ibm.com/opentech/2016/06/06/how-to-develop-for-apache-kafka-using-eclipse/
>
> Regards!
> --Vahid
>
> On Sat, Oct 13, 2018 at 2:15 AM M. Manna <ma...@gmail.com> wrote:
>
>> I have done the following and it worked for me.
>>
>> 1) Manually removed and added the jar locations in eclipse run
>> configuration e.g. client, core, etc.
>> 2) Added program argument for log4j file and server properties, and
>> zookeeper properties.
>> 3) Started zookeeper first
>> 4) Then started Kafka.
>>
>> I never had any problem with Zookeeper, so I actually ran
>> zookeeper-server-start script to start zookeeper manually. In this way, I
>> only need to start Kafka in debug mode from Eclipse.
>>
>> Let me know if that helps.
>>
>>
>>
>> On Sat, 13 Oct 2018 at 02:49, James Kwan <jw...@gmail.com> wrote:
>>
>>> Has anyone successfully set up an eclipse environment for Kafka on
>>> Windows? I have tried many times with different options but still have
>>> build issues with stream project. There are always some libraries
>>> missing.
>>>
>>> The steps I used
>>> 1) Run git command in command prompt to download the Kafka. I used
>>> windows command as well as using eclipse GIT options
>>> 2) Run gradle command to build the base.
>>> 3) Run gradlew jar to build the jar file
>>> 4) Run gradlew eclipse command to build projects
>>> 5) In eclipse, use Import/General/Existing Projects into Workspace option
>>> to import all projects
>>> 6) Remove duplicate source entries in build path for core project
>>> 7) Remove test projects that caused not found error message in the build
>>> path for core project
>>> 8) Remove the dependence of test-utils lib in stream project build path
>> to
>>> remove the build cycle error.
>>> 9) after that I got whole bunch of class or path not defined messages. I
>>> don’t know if I need to fix them one by one or if there is any jar files
>>> that I need to include.
>>>
>>>
>>> James
>>
Re: Kafka on Eclipse Setup
Posted by Vahid Hashemian <va...@gmail.com>.
This article was written a while back but still could help:
https://developer.ibm.com/opentech/2016/06/06/how-to-develop-for-apache-kafka-using-eclipse/
Regards!
--Vahid
On Sat, Oct 13, 2018 at 2:15 AM M. Manna <ma...@gmail.com> wrote:
> I have done the following and it worked for me.
>
> 1) Manually removed and added the jar locations in eclipse run
> configuration e.g. client, core, etc.
> 2) Added program argument for log4j file and server properties, and
> zookeeper properties.
> 3) Started zookeeper first
> 4) Then started Kafka.
>
> I never had any problem with Zookeeper, so I actually ran
> zookeeper-server-start script to start zookeeper manually. In this way, I
> only need to start Kafka in debug mode from Eclipse.
>
> Let me know if that helps.
>
>
>
> On Sat, 13 Oct 2018 at 02:49, James Kwan <jw...@gmail.com> wrote:
>
> > Has anyone successfully set up an eclipse environment for Kafka on
> > Windows? I have tried many times with different options but still have
> > build issues with stream project. There are always some libraries
> > missing.
> >
> > The steps I used
> > 1) Run git command in command prompt to download the Kafka. I used
> > windows command as well as using eclipse GIT options
> > 2) Run gradle command to build the base.
> > 3) Run gradlew jar to build the jar file
> > 4) Run gradlew eclipse command to build projects
> > 5) In eclipse, use Import/General/Existing Projects into Workspace option
> > to import all projects
> > 6) Remove duplicate source entries in build path for core project
> > 7) Remove test projects that caused not found error message in the build
> > path for core project
> > 8) Remove the dependence of test-utils lib in stream project build path
> to
> > remove the build cycle error.
> > 9) after that I got whole bunch of class or path not defined messages. I
> > don’t know if I need to fix them one by one or if there is any jar files
> > that I need to include.
> >
> >
> > James
>
Re: Kafka on Eclipse Setup
Posted by "M. Manna" <ma...@gmail.com>.
I have done the following and it worked for me.
1) Manually removed and added the jar locations in eclipse run
configuration e.g. client, core, etc.
2) Added program argument for log4j file and server properties, and
zookeeper properties.
3) Started zookeeper first
4) Then started Kafka.
I never had any problem with Zookeeper, so I actually ran
zookeeper-server-start script to start zookeeper manually. In this way, I
only need to start Kafka in debug mode from Eclipse.
Let me know if that helps.
On Sat, 13 Oct 2018 at 02:49, James Kwan <jw...@gmail.com> wrote:
> Has anyone successfully set up an eclipse environment for Kafka on
> Windows? I have tried many times with different options but still have
> build issues with stream project. There are always some libraries
> missing.
>
> The steps I used
> 1) Run git command in command prompt to download the Kafka. I used
> windows command as well as using eclipse GIT options
> 2) Run gradle command to build the base.
> 3) Run gradlew jar to build the jar file
> 4) Run gradlew eclipse command to build projects
> 5) In eclipse, use Import/General/Existing Projects into Workspace option
> to import all projects
> 6) Remove duplicate source entries in build path for core project
> 7) Remove test projects that caused not found error message in the build
> path for core project
> 8) Remove the dependence of test-utils lib in stream project build path to
> remove the build cycle error.
> 9) after that I got whole bunch of class or path not defined messages. I
> don’t know if I need to fix them one by one or if there is any jar files
> that I need to include.
>
>
> James
Kafka on Eclipse Setup
Posted by James Kwan <jw...@gmail.com>.
Has anyone successfully set up an eclipse environment for Kafka on Windows? I have tried many times with different options but still have build issues with stream project. There are always some libraries missing.
The steps I used
1) Run git command in command prompt to download the Kafka. I used windows command as well as using eclipse GIT options
2) Run gradle command to build the base.
3) Run gradlew jar to build the jar file
4) Run gradlew eclipse command to build projects
5) In eclipse, use Import/General/Existing Projects into Workspace option to import all projects
6) Remove duplicate source entries in build path for core project
7) Remove test projects that caused not found error message in the build path for core project
8) Remove the dependence of test-utils lib in stream project build path to remove the build cycle error.
9) after that I got whole bunch of class or path not defined messages. I don’t know if I need to fix them one by one or if there is any jar files that I need to include.
James
Re: Kafka UnknownProducerIdException
Posted by "Matthias J. Sax" <ma...@confluent.io>.
This error is related to Kafka EOS feature. I am not familiar with
Golden Gate though.
The error can happen, if the state of a transactional producer is lost.
The state is stored in the topic itself, thus, it could be lost due to
long truncation (for example, if a producer does not send data for a
very long time, ie, longer than retention time).
-Matthias
On 10/11/18 1:07 AM, Faisal Durrani wrote:
> Does anyone know the reason for the below error in Kafka? I have a Golden
> gate server publishing change transaction of tables to its corresponding
> topics in Kafka (which is running with the default configuration).It is a
> multi node cluster but the topic mentioned in the error has a single
> partition. Golden gate failed today and kafka shows the below error at the
> time of Golden gate failure.
>
> [ReplicaManager broker=1002] Error processing append operation on
> partition TABLE-0 (kafka.server.ReplicaManager)
> org.apache.kafka.common.errors.UnknownProducerIdException: Found no
> record of producerId=96007 on the broker. It is possible that the last
> message with the producerId=96007 has been removed due to hitting the
> retention limit.
>