# Allow all crawlers to browse the whole site, except for dynamically generated ps and pdf. # Wildcards forbidden in Disallow, so I had to put "todo" parameter first in search.ml # win_estb added lots of entries in log files # added former users # Archie 2005-06-17 User-agent: * #Disallow: /cgi-bin/search?todo=pdf #Disallow: /cgi-bin/search?todo=ps Disallow: /ltst/download/win_estb Disallow: /ext/ltst/download/win_estb Disallow: /win_estb Disallow: /ltst/data Disallow: /ext/ltst/data Disallow: /~yann Disallow: /paparazzi/doc Disallow: /~granger