It seems like a lot of the logic behind "tests shouldn't break due to a refactor" presumes that tests only test the publicly facing endpoints into the code. The tests that do the best job of reassuring me are tests against code in utility functions and the like. It's hardly a refactor if none of that changes.
However if you test too much of your "internal" code, you just end up cementing the currently implemented logic.
I often see tests that are basically checking if the code didn't change. I.e. checking if function calls (on injected (mock) objects) have been done in the exact specific order, instead of checking for some correct end result.
If you have a user facing component that uses utility functions, let's say a component that shows a table with a sum (that uses a utility function). Then refactoring your sum function should not break your tests of the table component.
I'm not saying you shouldn't test your sum function using unit tests, but ultimately users don't care so much about the sum function, they care that they have a table, and it should show a sum, and so that functionality should be tested.
