First of all, you can assume the server can infer this in a number of ways - there is actually no way to fully block it, but we can try.

The main issue for privacy is that it makes your browser behave in ways that are a bit too specific (i.e. less private by comparison with the rest of the browsers in the known universe).

As for techniques the site can use

• javascript can test the geometry of something that was rendered to draw conclusions - was this font actually used? test several options and check for variations
• measure font work between network events i.e. generate a site that makes the browser use unique links for 1) fetches a font 2) renders text and 3) only then another fetch - measure the time between 1) and 3) and draw conclusions. Repeat for test cases and draw conclusions - e.g. is the browser really fast using monospace vs custom huge font? not a great method, but not completely worthless
• some techniques can actually do some of this without Javascript, provided you can generate some weird CSS/HTML that conditionally triggers a fetch

By the away not downloading the fonts also makes you “less private”. Some of this is a stretch but not impossible.

Now for a more practical problem. Lots of sites use custom fonts for icons. Which means some sites will be very hard to use, because they only display buttons with an icon (actually a letter with a custom font).

FWIW these two lines are in my Firefox profile to disable downloads and skip document provided fonts:

user_pref("gfx.downloadable_fonts.enabled", false);
user_pref("browser.display.use_document_fonts", 0);

If someone has better/different settings please share.

Finally the Tor browser folks did good work on privacy protections over FF. Maybe their issue tracker is a good source of inspiration https://gitlab.torproject.org/tpo/applications/tor-browser/-/issues/18097

Perhaps more importantly, websites can determine which fonts are installed on your system (regardless of which you’re enforcing), making fingerprinting much easier.

The websites will just think that it’s your default font.