The one that few people think about is that use of LIDAR interferes with other LIDAR. LIDAR works mostly because almost no one is using LIDAR. By contrast, passive optical scales infinitely even if it is more difficult and there is an existence proof that it is possible. Automotive companies view LIDAR as a dead-end even though it works well (in some environments) now.
LIDAR has some issues in rain and snow, and can't read road markings or signs (or make out colours). And it's pretty expensive currently. A combination of LIDAR and cameras is likely what'll end up working best, they complement each other nicely.
Just to nitpick/paint a less absolute picture: Having worked at a couple companies using lidar, you can make out stuff like road markings based on relative brightness (especially if retroreflective paint is used), in ideal conditions, but that's of course not everywhere.
