Skip to content

GiosgCrawler

GiosgCrawler is automated part of the giosg platform designed to browse Customer's website and collect information for different usecases like automatically building AI Chatbot Knowledge Bases or collecting product data for product recommendations.

User-agent

GiosgCrawler has it's own unique user-agent: 

1
    Mozilla/5.0 (compatible; GiosgCrawler/1.0; +https://docs.giosg.com/common_information/giosgcrawler/)

Note that organisations can specify secret part for the User Agent string. In below example it is the {$organization secret} placeholder, this is unique for each organization. You can configure that on Organisation settings page in giosg platform.

By default the value is empty but if you wish to identify the crawler in your logs you can set it to a value that is meaningful to you. This can also be used to block other crawlers. If you give value for the secret, the user-agent will look like this:

1
    Mozilla/5.0 (compatible; GiosgCrawler-{$organization secret}/1.0; +https://docs.giosg.com/common_information/giosgcrawler/)

Note

Your website might have crawler prevention in use. This is the most common issue if GiosgCrawler can not fetch content from your website. In that situation you should contact your website admin and request them to allow our user-agent to access your website.