You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@nifi.apache.org by Yves HAMEL <yh...@macif.fr> on 2018/07/09 13:42:39 UTC
Issue with PutMongoProcessor using upsert mode.
Hi,
I try to "upsert" document in a mongodb collection.
When the document exists, PutMongo processor update the document but when
the document doesn't exist the processor do not create it.
Here is the setting of my PutMongo processor :
Mode : updateg
Upsert : true
Update Query Key : identifiant
Update Query : null
Update Mode : With Operator Enabled
I have read the PutMongodb.java and notice a strange thing when preparing
mongodb update request :
if (!StringUtils.isBlank(updateKey)) {
query = parseUpdateKey(updateKey, (Map)doc);
removeUpdateKeys(updateKey, (Map)doc);
} else {
query = Document.parse(filterQuery);
}
When using updateKey, it builds the query with the updateKey and removes
this key from the document so it creates the document without the key.
The document is created but there is missing part of the document.
When using update_query mode, it does remove anything form the document.
I simply remove the removeUpdateKeys() method call in a test custum
processor. And it works fine.
Is that path correct ?
Thank
Yves HAMEL
(Embedded image moved to file: pic07711.jpg)
Yves HAMEL
Direction Digital & Systèmes d’Information Groupe
LM_DATA
MACIF - 2 et 4, rue Pied de Fond - 79037 Niort cedex 9
Tél. : +33 (0)5 49 09 36 06
Email : mon.email@macif.fr / Pré Doyen 2 – bureau 999
www.macif.fr - Appli présente sur Google Play Store & Apple Store
(Embedded image moved to file: pic28253.jpg)
Re: Issue with PutMongoProcessor using upsert mode.
Posted by Yves HAMEL <yh...@macif.fr>.
Hi,
Ok I think I have post to quickly.
So I run more test.
UPSERT document with Operator Enable
test1 : upsert the document {"id1":1,$set{"value":"abc"}}, PutMongo
updateQueryKey=id1 => ok
test2 : upsert the document {"id1":1,"id2":1,$set:{"value":"abc"}},
PutMongo updateQueryKey=id1,id2
line 220 : removeUpdateKeys() doen't remove anything because this
method deals only with property with daot (p1.p2)
line 229 : update.remove() doesn't remove anything because trying to
remove many keys from the map
So the update parameter send to mongodb is :
{"id1":1,"id2":1,$set:{"value":"abc"}} and mongo throws Invalid BSON field
name id1 exception
test3 : upsert the document {"ids.id1":1,"value":"abc"}, PutMongo
updateQueryKey=ids.id1 => ok
test4 : upsert the document {"ids.id1":1,"ids.id2":1,"value":"abc"},
PutMongo updateQueryKey=ids.id1,ids.id2 => ok
test5 : upsert the document {"id1":1,"ids.id2":1,"value":"abc"}, PutMongo
updateQueryKey=id1,ids.id2
line 220 : removeUpdateKeys() remove "ids.id2" from the document
ine 229 : update.remove() doesn't remove anything because trying to
remove many keys from the map
So the update parameter send to mongodb is :
{"id1":1,$set:{"value":"abc"}} and mongo throws Invalid BSON field name id1
exception
UPSERT document With whole document
test 6 : upsert the document {"id1":1,"value":"abc"}, PutMongo
updateQuertKey=id1 => ok
test 7 : upsert the document {"id1":1,"id2":1,"value":"abc"}, PutMongo
updateQuertKey=id1;id2 => ok
test 8 : upsert the document {"ids.id1":1,"value":"abc"}, PutMongo
updateQueryKey=ids.id1
query : {ids.id1=1} doc : {value=abc}
inserted : { "_id" : ObjectId("5b44c5aeab8179ca72466a2f"),
"value" : "abc" }
test 9 : upsert the document {"ids.id1":1,"ids.id2":1,"value":"abc"},
PutMongo updateQueryKey=ids.id1,ids.id2
query : {ids.id1=1, ids.id2=1} doc : {value=abc}
inserted : { "_id" : ObjectId("5b44c51cab8179ca7246648c"),
"value" : "abc" }
test 10 : upsert the document {"id1":1,"ids.id2":1,"value":"abc"},
PutMongo updateQueryKey=id1,ids.id2
query : {id1=1, value=abc}, doc : {id1=1, value=abc}
inserted :{ "_id" : ObjectId("5b44c44bab8179ca72465c96"),
"id1" : 1, "value" : "abc" }
I think we should distinguish 2 update mode :
- first use case : using updateQuery (not empty). In this case the
query and the doc (flowfile) should be transmit to mongoDB without any
change.
- second use case : using udateQueryKey : the processor have to build
the query based on the document and updateQueryKey paramter: then we have
2 mode :
- WithOperatorEnable : the doc (flowfile) have the following form
{"key":"a","keys.k1":"b",....,$set:{"v1":"abc","v.v1":"abcd"},$unset:{....}....}
the updateQuertKey parameter :
key,keys.k1
so the processor have to
build the query parameter : {"key":"a","keys.k1":"b"} and remove the keys
from de document.
- WithWholeDocument : the json doc is
{"key":"a","keys":{"k1":"b","k2":"c"},"v1":"abc","v":{"v1":"abcd"}...}
the updateQuertKey
parameter : key,keys.k1
so the processor have to
extract the valurs of the different keys and build the
query :{"key":"a","keys.k1":"b"}, the doc remains unchanged.
I update the code of my custom processor, it works.
Here is the java code part I have modified for putMongoProcessor. (See
attached file: PutMongo-part.java)
What do you think about that ?
Thanks
(Embedded image moved to file: pic04966.jpg)
Yves HAMEL
Direction Digital & Systèmes d’Information Groupe
LM_DATA
MACIF - 2 et 4, rue Pied de Fond - 79037 Niort cedex 9
Tél. : +33 (0)5 49 09 36 06
Email : mon.email@macif.fr / Pré Doyen 2 – bureau 999
www.macif.fr - Appli présente sur Google Play Store & Apple Store
(Embedded image moved to file: pic07376.jpg)
De : Mike Thomsen <mi...@gmail.com>
A : users@nifi.apache.org,
Date : 09/07/2018 17:10
Objet : Re: Issue with PutMongoProcessor using upsert mode.
removeUpdateKeys should be skipping your key because it's a simple key.
private void removeUpdateKeys(String updateKeyParam, Map doc) {
String[] parts = updateKeyParam.split(",[\\s]*");
for (String part : parts) {
if (part.contains(".")) {
doc.remove(part);
}
}
}
Are you sure the value is just identifiant and not something like something
.identifiant?
On Mon, Jul 9, 2018 at 9:42 AM Yves HAMEL <yh...@macif.fr> wrote:
Hi,
I try to "upsert" document in a mongodb collection.
When the document exists, PutMongo processor update the document but
when
the document doesn't exist the processor do not create it.
Here is the setting of my PutMongo processor :
Mode : updateg
Upsert : true
Update Query Key : identifiant
Update Query : null
Update Mode : With Operator Enabled
I have read the PutMongodb.java and notice a strange thing when
preparing
mongodb update request :
if (!StringUtils.isBlank(updateKey)) {
query = parseUpdateKey(updateKey, (Map)doc);
removeUpdateKeys(updateKey, (Map)doc);
} else {
query = Document.parse(filterQuery);
}
When using updateKey, it builds the query with the updateKey and
removes
this key from the document so it creates the document without the
key.
The document is created but there is missing part of the document.
When using update_query mode, it does remove anything form the
document.
I simply remove the removeUpdateKeys() method call in a test custum
processor. And it works fine.
Is that path correct ?
Thank
Yves HAMEL
(Embedded image moved to file: pic07711.jpg)
Yves HAMEL
Direction Digital & Systèmes d’Information Groupe
LM_DATA
MACIF - 2 et 4, rue Pied de Fond - 79037 Niort cedex 9
Tél. : +33 (0)5 49 09 36 06
Email : mon.email@macif.fr / Pré Doyen 2 – bureau 999
www.macif.fr - Appli présente sur Google Play Store & Apple Store
(Embedded image moved to file: pic28253.jpg)
Re: Issue with PutMongoProcessor using upsert mode.
Posted by Mike Thomsen <mi...@gmail.com>.
removeUpdateKeys should be skipping your key because it's a simple key.
private void removeUpdateKeys(String updateKeyParam, Map doc) {
String[] parts = updateKeyParam.split(",[\\s]*");
for (String part : parts) {
if (part.contains(".")) {
doc.remove(part);
}
}
}
Are you sure the value is just *identifiant* and not something like
*something.identifiant*?
On Mon, Jul 9, 2018 at 9:42 AM Yves HAMEL <yh...@macif.fr> wrote:
>
> Hi,
> I try to "upsert" document in a mongodb collection.
> When the document exists, PutMongo processor update the document but when
> the document doesn't exist the processor do not create it.
> Here is the setting of my PutMongo processor :
> Mode : updateg
> Upsert : true
> Update Query Key : identifiant
> Update Query : null
> Update Mode : With Operator Enabled
>
> I have read the PutMongodb.java and notice a strange thing when preparing
> mongodb update request :
> if (!StringUtils.isBlank(updateKey)) {
> query = parseUpdateKey(updateKey, (Map)doc);
> removeUpdateKeys(updateKey, (Map)doc);
> } else {
> query = Document.parse(filterQuery);
> }
> When using updateKey, it builds the query with the updateKey and removes
> this key from the document so it creates the document without the key.
> The document is created but there is missing part of the document.
> When using update_query mode, it does remove anything form the document.
>
> I simply remove the removeUpdateKeys() method call in a test custum
> processor. And it works fine.
>
> Is that path correct ?
>
> Thank
>
> Yves HAMEL
> (Embedded image moved to file: pic07711.jpg)
>
> Yves HAMEL
> Direction Digital & Systèmes d’Information Groupe
> LM_DATA
>
> MACIF - 2 et 4, rue Pied de Fond
> <https://maps.google.com/?q=4,+rue+Pied+de+Fond&entry=gmail&source=g> -
> 79037 Niort cedex 9
> Tél. : +33 (0)5 49 09 36 06 <+33%205%2049%2009%2036%2006>
> Email : mon.email@macif.fr / Pré Doyen 2 – bureau 999
>
> www.macif.fr - Appli présente sur Google Play Store & Apple Store
> (Embedded image moved to file: pic28253.jpg)