* Who will be liable when a self-driving car hits someone.
* Will people, on mass, accept being driven around by a machines, on public roads.
* How will they deal with "trolley problem" type ethical scenarios? How will these decisions be made.
* How will self-driving cars cope with rural roads (where they are arguably MORE useful than in Urban settings better served by public transport)?
* How will they cope with signs/marking intentionally designed to fool machine vision systems. Who will be liable for accidents caused in these cases?
There are other issues too. However, overall, I think with some infrastructure investment we could have had self-driving cars since the late 90s early 2000s and there have been many prototypes. It's the ethical/social issues that are the bigger barrier in my mind.
Even if a sign is modified, a human will sanity check/ignore it. Currently many machine vision systems would be fooled. Basically in this scenario I think it's likely humans would fail safe.