Booting into a mainline Linux kernel on your average junk-level SBC with all the hardware working (without simply sticking to an Android-like downstream/proprietary BSP) is quite hard, and that's what you need in order to make a phone usable as a daily driver. That's really the root issue; mobile phones are built as embedded devices, with no consideration for running a generic OS kernel. This isn't even an Android issue, OpenMoko was the same deal. If anything, Android was the first mobile platform to even loosely approach any kind of PC-like openness.