Even if you trained the AI yourself from scratch you still can’t be confident you know what the AI is going to say under any given circumstance. LLMs have an inherent unpredictability to them. That’s part of their purpose, they’re not databases or search engines.

if I were to download a pre-trained model from what I thought was a reputable source, but was man-in-the middled and provided with a maliciously trained model

This is a risk for anything you download off the Internet, even source code could be MITMed to give you something with malicious stuff embedded in it. And no, I don’t believe you’d read and comprehend every line of it before you compile and run it. You need to verify checksums

As I said above, the real security comes from the code that’s running the LLM model. If someone wanted to “listen in” on what you say to the AI, they’d need to compromise that code to have it send your inputs to them. The model itself can’t do that. If someone wanted to have the model delete data or mess with your machine, it would be the execution framework of the model that’s doing that, not the model itself. And so forth.

You can probably come up with edge cases that are more difficult to secure, such as a troubleshooting AI whose literal purpose is messing with your system’s settings and whatnot, but that’s why I said “99% of the way there” in my original comment. There’s always edge cases.

Ironically, as far as I’m aware it’s based off of research done by some AI decelerationists over on the alignment forum who wanted to show how “unsafe” open models were in the hopes that there’d be regulation imposed to prevent companies from distributing them. They demonstrated that the “refusals” trained into LLMs could be removed with this method, allowing it to answer questions they considered scary.

The open LLM community responded by going “coooool!” And adapting the technique as a general tool for “training” models in various other ways.

That would be part of what’s required for them to be “open-weight”.

A plain old binary LLM model is somewhat equivalent to compiled object code, so redistributability is the main thing you can “open” about it compared to a “closed” model.

An LLM model is more malleable than compiled object code, though, as I described above there’s various ways you can mutate an LLM model without needing its “source code.” So it’s not exactly equivalent to compiled object code.

Fortunately, LLMs don’t really need to be fully open source to get almost all of the benefits of open source. From a safety and security perspective it’s fine because the model weights don’t really do anything; all of the actual work is done by the framework code that’s running them, and if you can trust that due to it being open source you’re 99% of the way there. The LLM model just sits there transforming the input text into the output text.

From a customization standpoint it’s a little worse, but we’re coming up with a lot of neat tricks for retraining and fine-tuning model weights in powerful ways. The most recent bit development I’ve heard of is abliteration, a technique that lets you isolate a particular “feature” of an LLM and either enhance it or remove it. The first big use of it is to modify various “censored” LLMs to remove their ability to refuse to comply with instructions, so that all those “safe” and “responsible” AIs like Goody-2 can turned into something that’s actually useful. A more fun example is MopeyMule, a LLaMA3 model that has had all of his hope and joy abliterated.

So I’m willing to accept open-weight models as being “nearly as good” as a full-blown open source model. I’d like to see full-blown open source models develop more, sure, but I’m not terribly concerned about having to rely on an open-weight model to make an AI system work for the immediate term.

And thus future AIs will have a bias toward having American attitudes because that’s where the data they’re built on comes from. A win for Europe?

I could imagine the NSA embedding an agent inside Cloudflare specifically to keep an eye out for any foreign agents also being embedded in Cloudflare, rather than to dig out its secrets for themselves.

The thing that drives me nuts is that I really do value that baby they’re carrying around. It is precious. But I don’t want to give the Internet Archive money just to funnel into the pockets of their lawyers and settlement payments to big publishers due to these unrelated quixotic battles.

I was hoping that the IA would have learned a lesson from losing this court case, they should have settled as soon as they could. I’m sure the publishers don’t want the bad publicity of “destroying” the Internet Archive, they just want them to stop blatantly violating their copyrights. But this appeal suggests that they haven’t learned that lesson yet.

In an ideal world there’d either be some kind of leadership shakeup at the IA to get rid of whoever was behind this stunt, or some kind of alternative IA-like organization appears to pick up the archive before the IA goes broke and its collection ends up being sold off to the highest bidder. Or simply destroyed.

They don’t need to do anything so drastic. They just need to stop doing things that blatantly provoke legal attacks like this. Their “Emergency Covid Library” was a foolish stunt that is endangering their primary objective of information preservation, they wouldn’t have been sued if they’d just kept on carrying on as they were before.

It’s not even a question of being “owned by corporations”. Judges don’t care about petitions. They’re not politicians, their job is to adjudicate the law.

Except it’s not a threat to the future of all libraries, it’s a threat to the future of “libraries” that decide to completely ignore copyright and give out an unlimited number of copies of ebooks. Basically turning themselves into book-focused piracy sites.

I’m incredibly frustrated with Internet Archive for bringing this on themselves. It is not their mandate to fight copyright, that’s something better left in the hands of activist organizations like the EFF. The Internet Archive’s mandate is to archive the Internet, to store and preserve knowledge. Distributing it is secondary to that goal. And picking unnecessary fights with big publishing houses like this is directly contrary to that goal, since now the Internet Archive is in danger.

It’s like they’re carrying around a precious baby and they decided it was a good idea to start whacking a bear with a stick. Now the bear is eating their leg and they’re screaming “oh my god help me, the bear is threatening this baby!” Well yeah, but you shouldn’t have brought a baby with you when you went on a bear-whacking expedition. You should have known exactly what that bear was going to do.

This is indeed one of the things cryptocurrencies exist for, but social media denizens around these parts have long conditioned themselves to hate it.

So a rock and a hard place, it seems. Which is more hated; the big data-harvesting corporation co-founded by Elon Musk, or a big bad NFT-hosting blockchain?

For people who are concerned about data harvesting I would recommend something like Monero or Aztec over Bitcoin, though. Bitcoin’s basically obsolete at this point, coasting on name recognition and inertia, and has no built-in privacy features.

Only those who don’t care about privacy and use Windows.

So most people, then.