Especially when they already have access to your entire email history. If they wanted your phone number for nefarious means it will probably be somewhere in that history already. Your email already requires complete trust in the email provider service, there’s so much more sensitive stuff they already have access to.
Have you people never heard of a phone book? Phone numbers aren’t sensitive information. If they want to scrape your phone number they can legally and trivially do so through public data sources. Google does plenty of sketchy things around privacy, but this isn’t one of them, it’s just about security.