Googlebot & changed tasks

Triss Gray's picture

UPDATE:

Google once again struck down on our site, and found a backdoor to clicking the arrows on our tasklist :). This again causes many tasks to be seen as updated by the site. I adapted the robots.txt again, it should now no loinger happen. I can only wait untill the next crawl to verify this...
On the good side: we now have most of our pages in google: Click here to see our pages on google.


It looks like google has found our little corner on the web, and started indexing us. Google however does not seem to play nice with our tasklist, which causes all tasks to be listed as updated after a visit from google. In order to prevent this, I created an appropriate robots.txt and a sitemap. These are pages that tell search engines what pages on your site they can and cannot visit. It also helps them "understand" how a webpage is built up. It does take some time before google (and other search engines) will fully take these files into account, so the problems may continue for another day or so.

This also means we are know known to google, so the babylonia web page will soon be coming to a search engine near you ;)