Make parsing of robots.txt more resilient

[alex-ball]

The code for parsing robots.txt introduced in commit 336f693 can run into problems if the content is not in the right form:

• If the file is empty, so too will be @lines, so lc will throw an uninitialized value warning at line 84.
• If the file does not contain a "User-agent: *" line, the code will get stuck in an infinite loop.
• If the file does not contain blank line after "User-agent: *", the pattern match at line 89 will throw an uninitialized value warning when the loop terminates.

These problems are resolved in this commit.