User-agent: * Disallow: /logs/ Disallow: /css/ Disallow: /scripts/ User-agent: TurnitinBot Disallow: / #http://www.turnitin.com/robot/crawlerinfo.html