This protocol is used by the website owners to inform the search engine bots about the areas of the website that the owner wants to get scanned or indexed and also the areas which remain exclusive to the owner only and are not be scanned.