There you go, it is called items now, personally I don't see much difference this is a tiny demo class and the var is private, you better understand before copying and pasting. Anyway, lets not argue about semi columns. The class is correct for multi threaded access and the API is pretty much perfect for my needs.
Glad you liked the rest of the article, I hope this helps others
Starting to ignore robots.txt? Unfortunately, there's no way to prove that anyone respects it. A well-identified bot can back off, then return from a different IP address with a different User-Agent and attempt to mimic a human user. Webmasters really have no defense against policy violations. If you run a bot of any kind, including a link checker or SSL tester, please respect robots.txt. If not, be prepared to be identified as malicious and blocked by an IDS.
Let me add: Since the purpose of your bot is to verify links and protect/serve your users, consider removing the links from your site if robots.txt prohibits you from checking them. That's what I would prefer as a webmaster who explicitly set that policy on a site, since I have no control over who posts the links.
The point of blocking a link with robots.txt is to say "Hey, web crawlers, please don't load and index this page". it does not mean "Hey, users, please don't come and load and read this page".
So the script written, for all intents and purposes, is just the same as a regular old user clicking the link and reading the page then keeping a list of the links that work and those that don't. It's not a crawler, it's an automated user.
If you are a webmaster than wants to block people from posting links to your page all around the web allowing others to come and read it, make the page 403.
Why? I'm not familiar with C# idioms, but it looks like a sane way to ensure that "items" and "expireOverride" are only modified by a single thread at a time.
Shame as the rest of the article is quite good, but that really flags me that this is a little bit cowboy code.
Also interesting to read some sites are taking a 'white-list' approach to robots.txt, as he says this is resulting in people starting to ignore it.