[jira] [Updated] (CONNECTORS-1748) Add "User-Agent platform" option for crawling mobile websites

["Piergiorgio Lucidi (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/CONNECTORS-1748?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Piergiorgio Lucidi updated CONNECTORS-1748:
-------------------------------------------
    Fix Version/s: ManifoldCF next

> Add "User-Agent platform" option for crawling mobile websites
> -------------------------------------------------------------
>
>                 Key: CONNECTORS-1748
>                 URL: https://issues.apache.org/jira/browse/CONNECTORS-1748
>             Project: ManifoldCF
>          Issue Type: New Feature
>            Reporter: Mingchun Zhao
>            Priority: Major
>             Fix For: ManifoldCF next
>
>
> Hi [~daddywri] ,
> Allow me propose a new feature for crawling mobile sites which restrict access to content based on User-Agent header.
> Actually, Our customer's mobile website was failing to crawl because access was restricted based on whether the User-Agent request header includes the mobile info. For this reason, we added the "User-Agent platform" option to the new "Request Headers" tab on the web repository connector page so that this mobile website doesn't fail to crawl.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)