Have you considered running Wireguard or Headscale instead of keeping SSH open? I don’t know how big an issue it is since you’ve changed the SSH port and use keys, but opening SSH in any respect freaks me out.