Make: Electronics by Charles Platt will take you through all the whys and wheres, starting at the very basics. I highly recommend it.
Also, I'll recommend the "Follow the example" approach of just looking at the data sheets for various components and modules. Especially stuff like the Arduino Uno - It's got a microcontroller, and then everything around it is just support. You start to see patterns where there (for example) a 10uF and 0.1uF Capacitor around power inputs. And they're everywhere so even if you don't know why (yet) you do know that you need capacitors around power inputs.
Also, I'll recommend the "Follow the example" approach of just looking at the data sheets for various components and modules. Especially stuff like the Arduino Uno - It's got a microcontroller, and then everything around it is just support. You start to see patterns where there (for example) a 10uF and 0.1uF Capacitor around power inputs. And they're everywhere so even if you don't know why (yet) you do know that you need capacitors around power inputs.