It’s a very valid question in my opinion and as is often the case with security, it really depends on your individual threat model and threat tolerance. As you said it seems pretty unlikely that a maintainer would install malicious code as they have a reputation to protect. And as mentioned by another commenter, even if you compiled the code yourself, unless you can audit code yourself you still have to just trust the developers. Personally for my threat tolerance, I do not see the risk as big enough to warrant the extra effort.
It’s a very valid question in my opinion and as is often the case with security, it really depends on your individual threat model and threat tolerance. As you said it seems pretty unlikely that a maintainer would install malicious code as they have a reputation to protect. And as mentioned by another commenter, even if you compiled the code yourself, unless you can audit code yourself you still have to just trust the developers. Personally for my threat tolerance, I do not see the risk as big enough to warrant the extra effort.