Does that mean Signal has access to my contacts? So much for privacy!

It does not, actually. Signal only gets the contact number if you click on the button.

In my case I allowed Signal to access my contacts and these options only appear for people who use it, don't know what happens if you deny the access.