You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@manifoldcf.apache.org by ritika jain <ri...@gmail.com> on 2021/03/01 12:18:13 UTC
Manifoldcf working Process
Hi All,
I am using Manifoldcf 2.14 version and Web connector as Repository and
Elastic search connector for output connector.
Web connector is crawling a dynamic website. My query is to understand the
nature of manifoldcf that is how manifoldcf works. Till now What I assume
was that once the record is ingested in the index and the reference is
stored in postgres tables. Manifold will not crawl that particular record
until unless there is any kind of content modification and updation in any
case. Which is the case with Window Shares connectors too (when tested).
But why is Manifoldcf again indexing the same URL again, when there is not
any modification in the website. Here is some example I have tested.
*When Job runs for First time*
"_source": {
"og:image": "
https://fca7603378a4e3ebeab2-4e03b1ac88f27f7b20b4cf232f717383.ssl.cf1.rackcdn.com/photos/social/landing_page_share2.jpg
",
"og:image:width": "1600",
"twitter:card": "summary_large_image",
"og:site_name": "World Wildlife Fund",
"Server": "cloudflare",
"twitter:url": "https://www.worldwildlife.org/",
"X-Runtime": "0.014362",
"X-Permitted-Cross-Domain-Policies": "none",
"og:description": "World Wildlife Fund - The leading organization
in wildlife conservation and endangered species. Learn how you can help WWF
make a difference. ",
"Referrer-Policy": "strict-origin-when-cross-origin",
"stream_name": "index.html",
"dc:title": "WWF - Endangered Species Conservation | World
Wildlife Fund",
"csrf-token":
"ZK/ZLm3Su0RnCi67L235sRzMt4KzhnvVyVYsAg56WgnezNH7jJc4VVD7hnXsYuasROQopMMZKVjUHB+JPyye5Q==",
"Content-Encoding": "UTF-8",
"robots": "all",
"X-Powered-By": "Phusion Passenger Enterprise",
"cf-request-id": "088ec6d96600004be27a252000000001",
"Status": "200 OK",
"twitter:site:id": "16029780",
"google-site-verification":
"5Xsbfjlv4FIZ6v_GWak8W3vx-haqUaMYureDmZY8QUU",
"X-Request-Id": "ceba9eec-386b-4e08-a7a9-960276e49f44",
"X-Content-Type-Options": "nosniff",
"Cache-Control": "max-age=0, private, must-revalidate",
"alt-svc": """h3-27=":443"; ma=86400, h3-28=":443"; ma=86400,
h3-29=":443"; ma=86400""",
"Content-Language": "en",
"description": "World Wildlife Fund - The leading organization in
wildlife conservation and endangered species. Learn how you can help WWF
make a difference. ",
"title": "WWF - Endangered Species Conservation | World Wildlife
Fund",
"twitter:image": "
https://fca7603378a4e3ebeab2-4e03b1ac88f27f7b20b4cf232f717383.ssl.cf1.rackcdn.com/photos/social/landing_page_share2.jpg
",
"X-Frame-Options": "SAMEORIGIN, SAMEORIGIN",
"Strict-Transport-Security": "max-age=63072000",
"twitter:site": "@world_wildlife",
"X-XSS-Protection": "1; mode=block",
"Content-Type": "text/html; charset=UTF-8",
"Transfer-Encoding": "chunked",
"stream_size": "102376",
"X-Protected-By": "Sqreen",
"X-Parsed-By": "org.apache.tika.parser.DefaultParser",
"og:type": "website",
"twitter:title": "WWF - Endangered Species Conservation | World
Wildlife Fund",
"CF-RAY": "62917408af754be2-AMS",
"og:title": "WWF - Endangered Species Conservation | World
Wildlife Fund",
"Connection": "keep-alive",
"og:image:height": "800",
"X-Download-Options": "noopen",
"csrf-param": "authenticity_token",
"CF-Cache-Status": "DYNAMIC",
"fb:app_id": "643916752395762",
"viewport": "width=device-width, initial-scale=1.0",
"twitter:description": "World Wildlife Fund - The leading
organization in wildlife conservation and endangered species. Learn how you
can help WWF make a difference. ",
"Vary": "Accept-Encoding",
"og:url": "https://www.worldwildlife.org/",
"Expect-CT": """max-age=604800, report-uri="
https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"""",
"indexed": "2021-03-01T09:45:34.444+0000",
"mime-type": "text/html; charset=utf-8",
"allow_token_document": "__nosecurity__",
"deny_token_document": "__nosecurity__",
"allow_token_share": "__nosecurity__",
"deny_token_share": "__nosecurity__",
"allow_token_parent": "__nosecurity__",
"deny_token_parent": "__nosecurity__",
"content": " "
}
}
]
}
[image: image.png]
*When Job runs for second Time*
"_source": {
"og:image": "
https://fca7603378a4e3ebeab2-4e03b1ac88f27f7b20b4cf232f717383.ssl.cf1.rackcdn.com/photos/social/landing_page_share2.jpg
",
"og:image:width": "1600",
"twitter:card": "summary_large_image",
"og:site_name": "World Wildlife Fund",
"Server": "cloudflare",
"twitter:url": "https://www.worldwildlife.org/",
"X-Runtime": "0.013794",
"X-Permitted-Cross-Domain-Policies": "none",
"og:description": "World Wildlife Fund - The leading organization
in wildlife conservation and endangered species. Learn how you can help WWF
make a difference. ",
"Referrer-Policy": "strict-origin-when-cross-origin",
"stream_name": "index.html",
"dc:title": "WWF - Endangered Species Conservation | World
Wildlife Fund",
"csrf-token":
"ZK/ZLm3Su0RnCi67L235sRzMt4KzhnvVyVYsAg56WgnezNH7jJc4VVD7hnXsYuasROQopMMZKVjUHB+JPyye5Q==",
"Content-Encoding": "UTF-8",
"robots": "all",
"X-Powered-By": "Phusion Passenger Enterprise",
"cf-request-id": "088ec8c9f600004be258828000000001",
"Status": "200 OK",
"twitter:site:id": "16029780",
"google-site-verification":
"5Xsbfjlv4FIZ6v_GWak8W3vx-haqUaMYureDmZY8QUU",
"X-Request-Id": "b98f39f0-e191-4c38-a219-56911d924790",
"X-Content-Type-Options": "nosniff",
"resourceName": "index.html",
"Cache-Control": "max-age=0, private, must-revalidate",
"alt-svc": """h3-27=":443"; ma=86400, h3-28=":443"; ma=86400,
h3-29=":443"; ma=86400""",
"Content-Language": "en",
"description": "World Wildlife Fund - The leading organization in
wildlife conservation and endangered species. Learn how you can help WWF
make a difference. ",
"title": "WWF - Endangered Species Conservation | World Wildlife
Fund",
"twitter:image": "
https://fca7603378a4e3ebeab2-4e03b1ac88f27f7b20b4cf232f717383.ssl.cf1.rackcdn.com/photos/social/landing_page_share2.jpg
",
"X-Frame-Options": "SAMEORIGIN, SAMEORIGIN",
"Strict-Transport-Security": "max-age=63072000",
"twitter:site": "@world_wildlife",
"X-XSS-Protection": "1; mode=block",
"Content-Type": "text/html; charset=UTF-8",
"Transfer-Encoding": "chunked",
"stream_size": "102376",
"X-Protected-By": "Sqreen",
"X-Parsed-By": "org.apache.tika.parser.DefaultParser",
"og:type": "website",
"twitter:title": "WWF - Endangered Species Conservation | World
Wildlife Fund",
"CF-RAY": "629177232a514be2-AMS",
"og:title": "WWF - Endangered Species Conservation | World
Wildlife Fund",
"Connection": "keep-alive",
"og:image:height": "800",
"X-Download-Options": "noopen",
"csrf-param": "authenticity_token",
"CF-Cache-Status": "DYNAMIC",
"fb:app_id": "643916752395762",
"viewport": "width=device-width, initial-scale=1.0",
"twitter:description": "World Wildlife Fund - The leading
organization in wildlife conservation and endangered species. Learn how you
can help WWF make a difference. ",
"Vary": "Accept-Encoding",
"og:url": "https://www.worldwildlife.org/",
"Expect-CT": """max-age=604800, report-uri="
https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"""",
"indexed": "2021-03-01T09:47:41.537+0000",
"mime-type": "text/html; charset=utf-8",
"allow_token_document": "__nosecurity__",
"deny_token_document": "__nosecurity__",
"allow_token_share": "__nosecurity__",
"deny_token_share": "__nosecurity__",
"allow_token_parent": "__nosecurity__",
"deny_token_parent": "__nosecurity__",
"content": """
[image: image.png]
Why is manifoldcf again ingesting the same records when reference is
already present in DB and records have not been modified. Is there anything
else behind this to understand.
Thanks Ritika
Re: Manifoldcf working Process
Posted by Karl Wright <da...@gmail.com>.
Hi,
The web connector constructs a version string for every document it crawls,
and will recrawl if that version string changes.
This gives me pause: 'Web connector is crawling a dynamic website.'
The reason it gives me pause is because the web connector will honor any
content time stamps that come along with the content. It does not do a
string diff but rather relies on the website to tell it when the content
was last changed. I suspect the dynamic pages you are serving up give a
fresh timestamp every time they are generated.
You can look at the code in WebCrawlerConnector.java to see exactly what
goes into this version string for this connector, but I suspect this is
indeed what is happening.
Karl
On Mon, Mar 1, 2021 at 7:19 AM ritika jain <ri...@gmail.com> wrote:
> Hi All,
>
> I am using Manifoldcf 2.14 version and Web connector as Repository and
> Elastic search connector for output connector.
> Web connector is crawling a dynamic website. My query is to understand the
> nature of manifoldcf that is how manifoldcf works. Till now What I assume
> was that once the record is ingested in the index and the reference is
> stored in postgres tables. Manifold will not crawl that particular record
> until unless there is any kind of content modification and updation in any
> case. Which is the case with Window Shares connectors too (when tested).
>
> But why is Manifoldcf again indexing the same URL again, when there is not
> any modification in the website. Here is some example I have tested.
> *When Job runs for First time*
> "_source": {
> "og:image": "
> https://fca7603378a4e3ebeab2-4e03b1ac88f27f7b20b4cf232f717383.ssl.cf1.rackcdn.com/photos/social/landing_page_share2.jpg
> ",
> "og:image:width": "1600",
> "twitter:card": "summary_large_image",
> "og:site_name": "World Wildlife Fund",
> "Server": "cloudflare",
> "twitter:url": "https://www.worldwildlife.org/",
> "X-Runtime": "0.014362",
> "X-Permitted-Cross-Domain-Policies": "none",
> "og:description": "World Wildlife Fund - The leading
> organization in wildlife conservation and endangered species. Learn how you
> can help WWF make a difference. ",
> "Referrer-Policy": "strict-origin-when-cross-origin",
> "stream_name": "index.html",
> "dc:title": "WWF - Endangered Species Conservation | World
> Wildlife Fund",
> "csrf-token":
> "ZK/ZLm3Su0RnCi67L235sRzMt4KzhnvVyVYsAg56WgnezNH7jJc4VVD7hnXsYuasROQopMMZKVjUHB+JPyye5Q==",
> "Content-Encoding": "UTF-8",
> "robots": "all",
> "X-Powered-By": "Phusion Passenger Enterprise",
> "cf-request-id": "088ec6d96600004be27a252000000001",
> "Status": "200 OK",
> "twitter:site:id": "16029780",
> "google-site-verification":
> "5Xsbfjlv4FIZ6v_GWak8W3vx-haqUaMYureDmZY8QUU",
> "X-Request-Id": "ceba9eec-386b-4e08-a7a9-960276e49f44",
> "X-Content-Type-Options": "nosniff",
>
> "Cache-Control": "max-age=0, private, must-revalidate",
> "alt-svc": """h3-27=":443"; ma=86400, h3-28=":443"; ma=86400,
> h3-29=":443"; ma=86400""",
> "Content-Language": "en",
> "description": "World Wildlife Fund - The leading organization
> in wildlife conservation and endangered species. Learn how you can help WWF
> make a difference. ",
> "title": "WWF - Endangered Species Conservation | World Wildlife
> Fund",
> "twitter:image": "
> https://fca7603378a4e3ebeab2-4e03b1ac88f27f7b20b4cf232f717383.ssl.cf1.rackcdn.com/photos/social/landing_page_share2.jpg
> ",
> "X-Frame-Options": "SAMEORIGIN, SAMEORIGIN",
> "Strict-Transport-Security": "max-age=63072000",
> "twitter:site": "@world_wildlife",
> "X-XSS-Protection": "1; mode=block",
> "Content-Type": "text/html; charset=UTF-8",
> "Transfer-Encoding": "chunked",
> "stream_size": "102376",
> "X-Protected-By": "Sqreen",
> "X-Parsed-By": "org.apache.tika.parser.DefaultParser",
> "og:type": "website",
> "twitter:title": "WWF - Endangered Species Conservation | World
> Wildlife Fund",
> "CF-RAY": "62917408af754be2-AMS",
> "og:title": "WWF - Endangered Species Conservation | World
> Wildlife Fund",
> "Connection": "keep-alive",
> "og:image:height": "800",
> "X-Download-Options": "noopen",
> "csrf-param": "authenticity_token",
> "CF-Cache-Status": "DYNAMIC",
> "fb:app_id": "643916752395762",
> "viewport": "width=device-width, initial-scale=1.0",
> "twitter:description": "World Wildlife Fund - The leading
> organization in wildlife conservation and endangered species. Learn how you
> can help WWF make a difference. ",
> "Vary": "Accept-Encoding",
> "og:url": "https://www.worldwildlife.org/",
> "Expect-CT": """max-age=604800, report-uri="
> https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"""",
> "indexed": "2021-03-01T09:45:34.444+0000",
> "mime-type": "text/html; charset=utf-8",
> "allow_token_document": "__nosecurity__",
> "deny_token_document": "__nosecurity__",
> "allow_token_share": "__nosecurity__",
> "deny_token_share": "__nosecurity__",
> "allow_token_parent": "__nosecurity__",
> "deny_token_parent": "__nosecurity__",
> "content": " "
> }
> }
> ]
> }
> [image: image.png]
>
> *When Job runs for second Time*
> "_source": {
> "og:image": "
> https://fca7603378a4e3ebeab2-4e03b1ac88f27f7b20b4cf232f717383.ssl.cf1.rackcdn.com/photos/social/landing_page_share2.jpg
> ",
> "og:image:width": "1600",
> "twitter:card": "summary_large_image",
> "og:site_name": "World Wildlife Fund",
> "Server": "cloudflare",
> "twitter:url": "https://www.worldwildlife.org/",
> "X-Runtime": "0.013794",
> "X-Permitted-Cross-Domain-Policies": "none",
> "og:description": "World Wildlife Fund - The leading
> organization in wildlife conservation and endangered species. Learn how you
> can help WWF make a difference. ",
> "Referrer-Policy": "strict-origin-when-cross-origin",
> "stream_name": "index.html",
> "dc:title": "WWF - Endangered Species Conservation | World
> Wildlife Fund",
> "csrf-token":
> "ZK/ZLm3Su0RnCi67L235sRzMt4KzhnvVyVYsAg56WgnezNH7jJc4VVD7hnXsYuasROQopMMZKVjUHB+JPyye5Q==",
> "Content-Encoding": "UTF-8",
> "robots": "all",
> "X-Powered-By": "Phusion Passenger Enterprise",
> "cf-request-id": "088ec8c9f600004be258828000000001",
> "Status": "200 OK",
> "twitter:site:id": "16029780",
> "google-site-verification":
> "5Xsbfjlv4FIZ6v_GWak8W3vx-haqUaMYureDmZY8QUU",
> "X-Request-Id": "b98f39f0-e191-4c38-a219-56911d924790",
> "X-Content-Type-Options": "nosniff",
> "resourceName": "index.html",
> "Cache-Control": "max-age=0, private, must-revalidate",
> "alt-svc": """h3-27=":443"; ma=86400, h3-28=":443"; ma=86400,
> h3-29=":443"; ma=86400""",
> "Content-Language": "en",
> "description": "World Wildlife Fund - The leading organization
> in wildlife conservation and endangered species. Learn how you can help WWF
> make a difference. ",
> "title": "WWF - Endangered Species Conservation | World Wildlife
> Fund",
> "twitter:image": "
> https://fca7603378a4e3ebeab2-4e03b1ac88f27f7b20b4cf232f717383.ssl.cf1.rackcdn.com/photos/social/landing_page_share2.jpg
> ",
> "X-Frame-Options": "SAMEORIGIN, SAMEORIGIN",
> "Strict-Transport-Security": "max-age=63072000",
> "twitter:site": "@world_wildlife",
> "X-XSS-Protection": "1; mode=block",
> "Content-Type": "text/html; charset=UTF-8",
> "Transfer-Encoding": "chunked",
> "stream_size": "102376",
> "X-Protected-By": "Sqreen",
> "X-Parsed-By": "org.apache.tika.parser.DefaultParser",
> "og:type": "website",
> "twitter:title": "WWF - Endangered Species Conservation | World
> Wildlife Fund",
> "CF-RAY": "629177232a514be2-AMS",
> "og:title": "WWF - Endangered Species Conservation | World
> Wildlife Fund",
> "Connection": "keep-alive",
> "og:image:height": "800",
> "X-Download-Options": "noopen",
> "csrf-param": "authenticity_token",
> "CF-Cache-Status": "DYNAMIC",
> "fb:app_id": "643916752395762",
> "viewport": "width=device-width, initial-scale=1.0",
> "twitter:description": "World Wildlife Fund - The leading
> organization in wildlife conservation and endangered species. Learn how you
> can help WWF make a difference. ",
> "Vary": "Accept-Encoding",
> "og:url": "https://www.worldwildlife.org/",
> "Expect-CT": """max-age=604800, report-uri="
> https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"""",
> "indexed": "2021-03-01T09:47:41.537+0000",
> "mime-type": "text/html; charset=utf-8",
> "allow_token_document": "__nosecurity__",
> "deny_token_document": "__nosecurity__",
> "allow_token_share": "__nosecurity__",
> "deny_token_share": "__nosecurity__",
> "allow_token_parent": "__nosecurity__",
> "deny_token_parent": "__nosecurity__",
> "content": """
> [image: image.png]
>
> Why is manifoldcf again ingesting the same records when reference is
> already present in DB and records have not been modified. Is there anything
> else behind this to understand.
>
> Thanks Ritika
>