I suggest having a threat model about what attack(s) your security is protecting against.

I’d suggest this probably isn’t giving much extra security over a long unique password for your password manager:

• A remote attacker who doesn’t control your machine, but is trying to phish you will succeed the same - dependent on your practices and password manager to prevent copying text.
• A remote attacker who does control your machine will also not be affected. Once your password database in the password manager is decrypted, they can take the whole thing, whether or not you used a password or hardware key to decrypt it. The only difference is maybe they need slightly more technical skill than copying the file + using a keylogger - but the biggest threats probably automate this anyway and there is no material difference.
• A local attacker who makes a single entry to steal your hardware, and then tries to extract data from it, is either advantaged by having a hardware key (if they can steal it, and you don’t also use a password), or is in a neutral position (can’t crack the locked password safe protected by password, don’t have the hardware key / can’t bypass its physical security). It might be an advantage if you can physically protect your hardware key (e.g. take it with you, and your threat model is people who take the database while you are away from it), if you can’t remember a sufficiently unique passphrase.
• A local attacker who can make a surreptitious entry, and then come back later for the results is in basically the same position as a remote attacker who does control your machine after the first visit.

That said, it might be able to give you more convenience at the expense of slightly less security - particularly if your threat model is entirely around remote attackers - on the convenience/security trade-off. You would touch a button to decrypt instead of entering a long passphrase.