What is a search engine index?
Please use the “Print” function at the bottom of the page to create a PDF.
The index (also known as the web index or crawler index) is a search engine's database. It contains information on all websites accessible through the search engine. This article provides a brief explanation of the term.
What is a search engine index?
A search engine index is a huge database created by search engines such as Google, Bing, or Yandex. This index stores information about websites, pages, and documents that have been found and analyzed by the web crawler (software that searches the internet).
The search engine index typically contains the following information:
URLs: The addresses of the websites and pages included in the index.
Titles and descriptions: The titles and descriptions of the websites and pages.
Content: The text content of the websites and pages.
Metadata: Information such as keywords, descriptions, and author information.
Links: Information about the links between websites and pages.
You can find more information on search engine indexing in the IONOS Digital Guide Search indexing content online: Functionality and optimization.
Control indexing with robots.txt
To a certain extent, you can control indexing by creating a robots.txt at the top level of your web space. The robots.txt is a text file that contains instructions for web crawlers (also known as robots or spiders). This file informs the crawlers which pages and documents of a website they may or may not crawl.
Explanations and examples of a robots.txt can be found in Google Search Central and the IONOS Digital Guide Index management with robots.txt files.