"Only white people" doesn't mean "no racism". As an Englishman, let me assure you my ancestors managed racism against white Irish people extremely well, and - somewhat ironically - plenty of the less educated people around me today are extremely racist to eastern Europeans (who are supposedly "takin' er jerbs").
Your point is "I don't do racist things, therefore there isn't a problem", which is just not true - there are still huge problems for tons of people with racism, sexism and other bigotry, and just because you don't see it or notice it doesn't make it not a problem. You got annoyed? I'm sure some non-racists were annoyed when all anyone could talk about was how bad slavery was. If you don't care about the issue, feel free to ignore it - but saying that people are imposing on you by caring is just unreasonable.
I mean racism. It's commonly accepted that race is a fuzzy term when it comes to this kind of thing and encompasses discrimination against people based on background, nationality and ethnic groups. However you want to define racism, the discrimination against Irish people in England fits the same pattern and has largely the same problems as racism in other forms.
In case you weren't aware, yes, Irish is an ethnicity, dating back over 9,000 years, making it older than most existing ethnicities. Wikipedia has more information: https://en.wikipedia.org/wiki/Irish_people
Being an island, Ireland was quite cut off genetically from the world for quite a long time, allowing a distinct ethnicity to develop. And as you may be aware, "race" isn't a real scientifically valid concept, but the word is used colloquially to refer to ethnicity, which is a scientifically valid concept that can be measured using genetics.
Your point is "I don't do racist things, therefore there isn't a problem", which is just not true - there are still huge problems for tons of people with racism, sexism and other bigotry, and just because you don't see it or notice it doesn't make it not a problem. You got annoyed? I'm sure some non-racists were annoyed when all anyone could talk about was how bad slavery was. If you don't care about the issue, feel free to ignore it - but saying that people are imposing on you by caring is just unreasonable.