You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@nifi.apache.org by Mike Thomsen <mi...@gmail.com> on 2018/06/07 11:05:28 UTC
Re: Issue when inserting array with PutMongoRecord processor
You can open a Jira ticket, a PR on GitHub or just drop a patch file on the
dev list if you're not comfortable with either. A sanitized example record
set would also be helpful if that's possible.
Thanks,
Mike
On Thu, Jun 7, 2018 at 4:16 AM Yves HAMEL <yh...@macif.fr> wrote:
>
> Hi,
> When I try to insert json document containing array in mongoDB using
> PutMongoRecord processor I have the exception :
> org.bson.codecs.configuration.CodecConfigurationException: Can't find a
> codec for class [Ljava.lang.Object;.
> at org.bson.codecs.configuration.CodecCache.getOrThrow
> (CodecCache.java:46)
> at org.bson.codecs.configuration.ProvidersCodecRegistry.get
> (ProvidersCodecRegistry.java:63)
> at org.bson.codecs.configuration.ChildCodecRegistry.get
> (ChildCodecRegistry.java:51)
> at org.bson.codecs.DocumentCodec.writeValue(DocumentCodec.java:174)
> at org.bson.codecs.DocumentCodec.writeMap(DocumentCodec.java:189)
> at org.bson.codecs.DocumentCodec.encode(DocumentCodec.java:131)
> at org.bson.codecs.DocumentCodec.encode(DocumentCodec.java:45)
> at org.bson.codecs.BsonDocumentWrapperCodec.encode
> (BsonDocumentWrapperCodec.java:63)
> at org.bson.codecs.BsonDocumentWrapperCodec.encode
> (BsonDocumentWrapperCodec.java:29)
> at com.mongodb.connection.InsertCommandMessage.writeTheWrites
> (InsertCommandMessage.java:101)
> at com.mongodb.connection.InsertCommandMessage.writeTheWrites
> (InsertCommandMessage.java:43)
> at
>
> com.mongodb.connection.BaseWriteCommandMessage.encodeMessageBodyWithMetadata
> (BaseWriteCommandMessage.java:129)
> at com.mongodb.connection.RequestMessage.encodeWithMetadata
> (RequestMessage.java:160)
> at com.mongodb.connection.WriteCommandProtocol.sendMessage
> (WriteCommandProtocol.java:212)
> at com.mongodb.connection.WriteCommandProtocol.execute
> (WriteCommandProtocol.java:101)
> at com.mongodb.connection.InsertCommandProtocol.execute
> (InsertCommandProtocol.java:67)
> at com.mongodb.connection.InsertCommandProtocol.execute
> (InsertCommandProtocol.java:37)
> at com.mongodb.connection.DefaultServer
> $DefaultServerProtocolExecutor.execute(DefaultServer.java:159)
> at com.mongodb.connection.DefaultServerConnection.executeProtocol
> (DefaultServerConnection.java:286)
> at com.mongodb.connection.DefaultServerConnection.insertCommand
> (DefaultServerConnection.java:115)
> at com.mongodb.operation.MixedBulkWriteOperation$Run
> $2.executeWriteCommandProtocol(MixedBulkWriteOperation.java:455)
> at com.mongodb.operation.MixedBulkWriteOperation$Run
> $RunExecutor.execute(MixedBulkWriteOperation.java:646)
> at com.mongodb.operation.MixedBulkWriteOperation$Run.execute
> (MixedBulkWriteOperation.java:401)
> at com.mongodb.operation.MixedBulkWriteOperation$1.call
> (MixedBulkWriteOperation.java:179)
> at com.mongodb.operation.MixedBulkWriteOperation$1.call
> (MixedBulkWriteOperation.java:168)
> at com.mongodb.operation.OperationHelper.withConnectionSource
> (OperationHelper.java:230)
> at com.mongodb.operation.OperationHelper.withConnection
> (OperationHelper.java:221)
> at com.mongodb.operation.MixedBulkWriteOperation.execute
> (MixedBulkWriteOperation.java:168)
> at com.mongodb.operation.MixedBulkWriteOperation.execute
> (MixedBulkWriteOperation.java:74)
> at com.mongodb.Mongo.execute(Mongo.java:781)
> at com.mongodb.Mongo$2.execute(Mongo.java:764)
> at com.mongodb.MongoCollectionImpl.insertMany
> (MongoCollectionImpl.java:323)
> at com.mongodb.MongoCollectionImpl.insertMany
> (MongoCollectionImpl.java:311)
> at org.apache.nifi.processors.mongodb.PutMongoRecord.onTrigger
> (PutMongoRecord.java:142)
> at org.apache.nifi.processor.AbstractProcessor.onTrigger
> (AbstractProcessor.java:27)
> at org.apache.nifi.controller.StandardProcessorNode.onTrigger
> (StandardProcessorNode.java:1141)
> at
> org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call
> (ContinuallyRunProcessorTask.java:147)
> at
> org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call
> (ContinuallyRunProcessorTask.java:47)
> at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent
> $1.run(TimerDrivenSchedulingAgent.java:128)
> at java.util.concurrent.Executors$RunnableAdapter.call
> (Executors.java:511)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
> at java.util.concurrent.ScheduledThreadPoolExecutor
> $ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> at java.util.concurrent.ScheduledThreadPoolExecutor
> $ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
> at java.util.concurrent.ThreadPoolExecutor.runWorker
> (ThreadPoolExecutor.java:1149)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run
> (ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
>
> My json document is {"nom":"HAMEL","prenom":"YVES","tab":["aa","bb"]}
> My mecord reader use the schema (generated by InferAvroSchema):
> {
> "type" : "record",
> "name" : "Test",
> "fields" : [ {
> "name" : "nom",
> "type" : "string",
> "doc" : "Type inferred from '\"HAMEL\"'"
> }, {
> "name" : "prenom",
> "type" : "string",
> "doc" : "Type inferred from '\"YVES\"'"
> }, {
> "name" : "tab",
> "type" : {
> "type" : "array",
> "items" : "string"
> },
> "doc" : "Type inferred from '[\"aa\",\"bb\"]'"
> } ]
> }
>
> I did a little debug and I think I get this exeception because
> PuMongoRecord maps json array to java array. But the mongodb java drivers
> doesn't support java array but only support List.
>
> Because this issue blocks our project I made a custom processor correcting
> this. How can I propose this correction to the community ?
>
> Thanks
>
> Yves HAMEL
> (Embedded image moved to file: pic14989.jpg)
>
> Yves HAMEL
> Direction Digital & Systèmes d’Information Groupe
> LM_DATA
>
> MACIF - 2 et 4, rue Pied de Fond
> <https://maps.google.com/?q=4,+rue+Pied+de+Fond&entry=gmail&source=g> -
> 79037 Niort cedex 9
> Tél. : +33 (0)5 49 09 36 06 <+33%205%2049%2009%2036%2006>
> Email : mon.email@macif.fr / Pré Doyen 2 – bureau 999
>
> www.macif.fr - Appli présente sur Google Play Store & Apple Store
> (Embedded image moved to file: pic32702.jpg)
Re: Issue when inserting array with PutMongoRecord processor
Posted by Matt Burgess <ma...@apache.org>.
I left some comments on the PR, will do my best to get it merged if
the changes go in by tomorrow night.
Regards,
Matt
On Sat, Jun 9, 2018 at 3:41 PM Mike Thomsen <mi...@gmail.com> wrote:
>
> Yves,
>
> Please take a look:
>
> https://github.com/apache/nifi/pull/2778
>
> https://issues.apache.org/jira/browse/NIFI-5288
>
> Thanks,
>
> Mike
>
> On Thu, Jun 7, 2018 at 7:05 AM Mike Thomsen <mi...@gmail.com> wrote:
>>
>> You can open a Jira ticket, a PR on GitHub or just drop a patch file on the dev list if you're not comfortable with either. A sanitized example record set would also be helpful if that's possible.
>>
>> Thanks,
>>
>> Mike
>>
>> On Thu, Jun 7, 2018 at 4:16 AM Yves HAMEL <yh...@macif.fr> wrote:
>>>
>>>
>>> Hi,
>>> When I try to insert json document containing array in mongoDB using
>>> PutMongoRecord processor I have the exception :
>>> org.bson.codecs.configuration.CodecConfigurationException: Can't find a
>>> codec for class [Ljava.lang.Object;.
>>> at org.bson.codecs.configuration.CodecCache.getOrThrow
>>> (CodecCache.java:46)
>>> at org.bson.codecs.configuration.ProvidersCodecRegistry.get
>>> (ProvidersCodecRegistry.java:63)
>>> at org.bson.codecs.configuration.ChildCodecRegistry.get
>>> (ChildCodecRegistry.java:51)
>>> at org.bson.codecs.DocumentCodec.writeValue(DocumentCodec.java:174)
>>> at org.bson.codecs.DocumentCodec.writeMap(DocumentCodec.java:189)
>>> at org.bson.codecs.DocumentCodec.encode(DocumentCodec.java:131)
>>> at org.bson.codecs.DocumentCodec.encode(DocumentCodec.java:45)
>>> at org.bson.codecs.BsonDocumentWrapperCodec.encode
>>> (BsonDocumentWrapperCodec.java:63)
>>> at org.bson.codecs.BsonDocumentWrapperCodec.encode
>>> (BsonDocumentWrapperCodec.java:29)
>>> at com.mongodb.connection.InsertCommandMessage.writeTheWrites
>>> (InsertCommandMessage.java:101)
>>> at com.mongodb.connection.InsertCommandMessage.writeTheWrites
>>> (InsertCommandMessage.java:43)
>>> at
>>> com.mongodb.connection.BaseWriteCommandMessage.encodeMessageBodyWithMetadata
>>> (BaseWriteCommandMessage.java:129)
>>> at com.mongodb.connection.RequestMessage.encodeWithMetadata
>>> (RequestMessage.java:160)
>>> at com.mongodb.connection.WriteCommandProtocol.sendMessage
>>> (WriteCommandProtocol.java:212)
>>> at com.mongodb.connection.WriteCommandProtocol.execute
>>> (WriteCommandProtocol.java:101)
>>> at com.mongodb.connection.InsertCommandProtocol.execute
>>> (InsertCommandProtocol.java:67)
>>> at com.mongodb.connection.InsertCommandProtocol.execute
>>> (InsertCommandProtocol.java:37)
>>> at com.mongodb.connection.DefaultServer
>>> $DefaultServerProtocolExecutor.execute(DefaultServer.java:159)
>>> at com.mongodb.connection.DefaultServerConnection.executeProtocol
>>> (DefaultServerConnection.java:286)
>>> at com.mongodb.connection.DefaultServerConnection.insertCommand
>>> (DefaultServerConnection.java:115)
>>> at com.mongodb.operation.MixedBulkWriteOperation$Run
>>> $2.executeWriteCommandProtocol(MixedBulkWriteOperation.java:455)
>>> at com.mongodb.operation.MixedBulkWriteOperation$Run
>>> $RunExecutor.execute(MixedBulkWriteOperation.java:646)
>>> at com.mongodb.operation.MixedBulkWriteOperation$Run.execute
>>> (MixedBulkWriteOperation.java:401)
>>> at com.mongodb.operation.MixedBulkWriteOperation$1.call
>>> (MixedBulkWriteOperation.java:179)
>>> at com.mongodb.operation.MixedBulkWriteOperation$1.call
>>> (MixedBulkWriteOperation.java:168)
>>> at com.mongodb.operation.OperationHelper.withConnectionSource
>>> (OperationHelper.java:230)
>>> at com.mongodb.operation.OperationHelper.withConnection
>>> (OperationHelper.java:221)
>>> at com.mongodb.operation.MixedBulkWriteOperation.execute
>>> (MixedBulkWriteOperation.java:168)
>>> at com.mongodb.operation.MixedBulkWriteOperation.execute
>>> (MixedBulkWriteOperation.java:74)
>>> at com.mongodb.Mongo.execute(Mongo.java:781)
>>> at com.mongodb.Mongo$2.execute(Mongo.java:764)
>>> at com.mongodb.MongoCollectionImpl.insertMany
>>> (MongoCollectionImpl.java:323)
>>> at com.mongodb.MongoCollectionImpl.insertMany
>>> (MongoCollectionImpl.java:311)
>>> at org.apache.nifi.processors.mongodb.PutMongoRecord.onTrigger
>>> (PutMongoRecord.java:142)
>>> at org.apache.nifi.processor.AbstractProcessor.onTrigger
>>> (AbstractProcessor.java:27)
>>> at org.apache.nifi.controller.StandardProcessorNode.onTrigger
>>> (StandardProcessorNode.java:1141)
>>> at
>>> org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call
>>> (ContinuallyRunProcessorTask.java:147)
>>> at
>>> org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call
>>> (ContinuallyRunProcessorTask.java:47)
>>> at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent
>>> $1.run(TimerDrivenSchedulingAgent.java:128)
>>> at java.util.concurrent.Executors$RunnableAdapter.call
>>> (Executors.java:511)
>>> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
>>> at java.util.concurrent.ScheduledThreadPoolExecutor
>>> $ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
>>> at java.util.concurrent.ScheduledThreadPoolExecutor
>>> $ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
>>> at java.util.concurrent.ThreadPoolExecutor.runWorker
>>> (ThreadPoolExecutor.java:1149)
>>> at java.util.concurrent.ThreadPoolExecutor$Worker.run
>>> (ThreadPoolExecutor.java:624)
>>> at java.lang.Thread.run(Thread.java:748)
>>>
>>> My json document is {"nom":"HAMEL","prenom":"YVES","tab":["aa","bb"]}
>>> My mecord reader use the schema (generated by InferAvroSchema):
>>> {
>>> "type" : "record",
>>> "name" : "Test",
>>> "fields" : [ {
>>> "name" : "nom",
>>> "type" : "string",
>>> "doc" : "Type inferred from '\"HAMEL\"'"
>>> }, {
>>> "name" : "prenom",
>>> "type" : "string",
>>> "doc" : "Type inferred from '\"YVES\"'"
>>> }, {
>>> "name" : "tab",
>>> "type" : {
>>> "type" : "array",
>>> "items" : "string"
>>> },
>>> "doc" : "Type inferred from '[\"aa\",\"bb\"]'"
>>> } ]
>>> }
>>>
>>> I did a little debug and I think I get this exeception because
>>> PuMongoRecord maps json array to java array. But the mongodb java drivers
>>> doesn't support java array but only support List.
>>>
>>> Because this issue blocks our project I made a custom processor correcting
>>> this. How can I propose this correction to the community ?
>>>
>>> Thanks
>>>
>>> Yves HAMEL
>>> (Embedded image moved to file: pic14989.jpg)
>>>
>>> Yves HAMEL
>>> Direction Digital & Systèmes d’Information Groupe
>>> LM_DATA
>>>
>>> MACIF - 2 et 4, rue Pied de Fond - 79037 Niort cedex 9
>>> Tél. : +33 (0)5 49 09 36 06
>>> Email : mon.email@macif.fr / Pré Doyen 2 – bureau 999
>>>
>>> www.macif.fr - Appli présente sur Google Play Store & Apple Store
>>> (Embedded image moved to file: pic32702.jpg)
Re: Issue when inserting array with PutMongoRecord processor
Posted by Mike Thomsen <mi...@gmail.com>.
Yves,
Please take a look:
https://github.com/apache/nifi/pull/2778
https://issues.apache.org/jira/browse/NIFI-5288
Thanks,
Mike
On Thu, Jun 7, 2018 at 7:05 AM Mike Thomsen <mi...@gmail.com> wrote:
> You can open a Jira ticket, a PR on GitHub or just drop a patch file on
> the dev list if you're not comfortable with either. A sanitized example
> record set would also be helpful if that's possible.
>
> Thanks,
>
> Mike
>
> On Thu, Jun 7, 2018 at 4:16 AM Yves HAMEL <yh...@macif.fr> wrote:
>
>>
>> Hi,
>> When I try to insert json document containing array in mongoDB using
>> PutMongoRecord processor I have the exception :
>> org.bson.codecs.configuration.CodecConfigurationException: Can't find a
>> codec for class [Ljava.lang.Object;.
>> at org.bson.codecs.configuration.CodecCache.getOrThrow
>> (CodecCache.java:46)
>> at org.bson.codecs.configuration.ProvidersCodecRegistry.get
>> (ProvidersCodecRegistry.java:63)
>> at org.bson.codecs.configuration.ChildCodecRegistry.get
>> (ChildCodecRegistry.java:51)
>> at
>> org.bson.codecs.DocumentCodec.writeValue(DocumentCodec.java:174)
>> at org.bson.codecs.DocumentCodec.writeMap(DocumentCodec.java:189)
>> at org.bson.codecs.DocumentCodec.encode(DocumentCodec.java:131)
>> at org.bson.codecs.DocumentCodec.encode(DocumentCodec.java:45)
>> at org.bson.codecs.BsonDocumentWrapperCodec.encode
>> (BsonDocumentWrapperCodec.java:63)
>> at org.bson.codecs.BsonDocumentWrapperCodec.encode
>> (BsonDocumentWrapperCodec.java:29)
>> at com.mongodb.connection.InsertCommandMessage.writeTheWrites
>> (InsertCommandMessage.java:101)
>> at com.mongodb.connection.InsertCommandMessage.writeTheWrites
>> (InsertCommandMessage.java:43)
>> at
>>
>> com.mongodb.connection.BaseWriteCommandMessage.encodeMessageBodyWithMetadata
>> (BaseWriteCommandMessage.java:129)
>> at com.mongodb.connection.RequestMessage.encodeWithMetadata
>> (RequestMessage.java:160)
>> at com.mongodb.connection.WriteCommandProtocol.sendMessage
>> (WriteCommandProtocol.java:212)
>> at com.mongodb.connection.WriteCommandProtocol.execute
>> (WriteCommandProtocol.java:101)
>> at com.mongodb.connection.InsertCommandProtocol.execute
>> (InsertCommandProtocol.java:67)
>> at com.mongodb.connection.InsertCommandProtocol.execute
>> (InsertCommandProtocol.java:37)
>> at com.mongodb.connection.DefaultServer
>> $DefaultServerProtocolExecutor.execute(DefaultServer.java:159)
>> at com.mongodb.connection.DefaultServerConnection.executeProtocol
>> (DefaultServerConnection.java:286)
>> at com.mongodb.connection.DefaultServerConnection.insertCommand
>> (DefaultServerConnection.java:115)
>> at com.mongodb.operation.MixedBulkWriteOperation$Run
>> $2.executeWriteCommandProtocol(MixedBulkWriteOperation.java:455)
>> at com.mongodb.operation.MixedBulkWriteOperation$Run
>> $RunExecutor.execute(MixedBulkWriteOperation.java:646)
>> at com.mongodb.operation.MixedBulkWriteOperation$Run.execute
>> (MixedBulkWriteOperation.java:401)
>> at com.mongodb.operation.MixedBulkWriteOperation$1.call
>> (MixedBulkWriteOperation.java:179)
>> at com.mongodb.operation.MixedBulkWriteOperation$1.call
>> (MixedBulkWriteOperation.java:168)
>> at com.mongodb.operation.OperationHelper.withConnectionSource
>> (OperationHelper.java:230)
>> at com.mongodb.operation.OperationHelper.withConnection
>> (OperationHelper.java:221)
>> at com.mongodb.operation.MixedBulkWriteOperation.execute
>> (MixedBulkWriteOperation.java:168)
>> at com.mongodb.operation.MixedBulkWriteOperation.execute
>> (MixedBulkWriteOperation.java:74)
>> at com.mongodb.Mongo.execute(Mongo.java:781)
>> at com.mongodb.Mongo$2.execute(Mongo.java:764)
>> at com.mongodb.MongoCollectionImpl.insertMany
>> (MongoCollectionImpl.java:323)
>> at com.mongodb.MongoCollectionImpl.insertMany
>> (MongoCollectionImpl.java:311)
>> at org.apache.nifi.processors.mongodb.PutMongoRecord.onTrigger
>> (PutMongoRecord.java:142)
>> at org.apache.nifi.processor.AbstractProcessor.onTrigger
>> (AbstractProcessor.java:27)
>> at org.apache.nifi.controller.StandardProcessorNode.onTrigger
>> (StandardProcessorNode.java:1141)
>> at
>> org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call
>> (ContinuallyRunProcessorTask.java:147)
>> at
>> org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call
>> (ContinuallyRunProcessorTask.java:47)
>> at
>> org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent
>> $1.run(TimerDrivenSchedulingAgent.java:128)
>> at java.util.concurrent.Executors$RunnableAdapter.call
>> (Executors.java:511)
>> at
>> java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
>> at java.util.concurrent.ScheduledThreadPoolExecutor
>> $ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
>> at java.util.concurrent.ScheduledThreadPoolExecutor
>> $ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
>> at java.util.concurrent.ThreadPoolExecutor.runWorker
>> (ThreadPoolExecutor.java:1149)
>> at java.util.concurrent.ThreadPoolExecutor$Worker.run
>> (ThreadPoolExecutor.java:624)
>> at java.lang.Thread.run(Thread.java:748)
>>
>> My json document is {"nom":"HAMEL","prenom":"YVES","tab":["aa","bb"]}
>> My mecord reader use the schema (generated by InferAvroSchema):
>> {
>> "type" : "record",
>> "name" : "Test",
>> "fields" : [ {
>> "name" : "nom",
>> "type" : "string",
>> "doc" : "Type inferred from '\"HAMEL\"'"
>> }, {
>> "name" : "prenom",
>> "type" : "string",
>> "doc" : "Type inferred from '\"YVES\"'"
>> }, {
>> "name" : "tab",
>> "type" : {
>> "type" : "array",
>> "items" : "string"
>> },
>> "doc" : "Type inferred from '[\"aa\",\"bb\"]'"
>> } ]
>> }
>>
>> I did a little debug and I think I get this exeception because
>> PuMongoRecord maps json array to java array. But the mongodb java drivers
>> doesn't support java array but only support List.
>>
>> Because this issue blocks our project I made a custom processor correcting
>> this. How can I propose this correction to the community ?
>>
>> Thanks
>>
>> Yves HAMEL
>> (Embedded image moved to file: pic14989.jpg)
>>
>> Yves HAMEL
>> Direction Digital & Systèmes d’Information Groupe
>> LM_DATA
>>
>> MACIF - 2 et 4, rue Pied de Fond
>> <https://maps.google.com/?q=4,+rue+Pied+de+Fond&entry=gmail&source=g> -
>> 79037 Niort cedex 9
>> Tél. : +33 (0)5 49 09 36 06 <+33%205%2049%2009%2036%2006>
>> Email : mon.email@macif.fr / Pré Doyen 2 – bureau 999
>>
>> www.macif.fr - Appli présente sur Google Play Store & Apple Store
>> (Embedded image moved to file: pic32702.jpg)
>
>