Sure, but we've decided that for some crimes (like murder) there is no statute of limitations, and we have a greater interest in prosecuting those crimes than in ensuring it's reasonable for a defendant to defend themself many years later.

I think the same should be true of our civil servants who wield so much power over the populace.