A Robots.txt file serves the purpose of declaring the off-limit crawling parts of the website. This is not a mandatory file; however it does serve as an opt-out process. Without which, a website can be subject for web robots, much like those of search engine crawlers, to access and index the directory of the website.
Other exclusion commands, like an HTML meta tag, can perform the same function, but the benefit of using a single text file to control the omission of pages is for the ease of maintenance.
The favicon.ico can be defined as a small image that can represent a website, much like a desktop application shortcut. Favicon, or favourite icon, can be placed in the browser’s address bar. This unique opportunity can add both style and additional identity to a website with the browser’s favourites and bookmarks functions. Plus, all major browsers already have a built-in support for Favicon.ico, making it a solid extra file.
This file aids search engines to index the website correctly. The Robots.txt file does control the exclusion of files, but it is the Sitemap.xml file that lists the structure of a website and all its pages so that the search engine crawlers can get a grasp of where specific content are located on the site.
The Dublin.rdf file serves the purpose of storing officially recognized meta elements HTML, metadata, microformats and well-written content so that they appear in the appropriate search results. It is these elements, after all, that augment the semantic value of the media you provide.
It is this file that lets a website perform its own search engine listing by means of a search feature which appears in major browsers. It also serves the purpose of enhancing an in-browser search, thus complementing the search mechanism of the website.