Recommendation systems don’t need to be that complicated. In its base form it’s just a list of videos you’ve watched (or content creators or topics). It can then be compared with the watching lists of other people to get an idea of what else you might be interested in. No need for any advanced video recognition.
Maybe this list is isolated within a single instance. Maybe it can be shared between instances. Different instances might use different recommendation systems.
Again, it might not work as well as YouTube’s, but I don’t think it needs to.
Recommendation systems are well studied. I don’t think it’s unreasonable to add some form of recommendation layer separate from (or on top of) the content delivery. It doesn’t need to be up to par with YouTube’s, at least before there’s any major content.
Most YouTubers rely on sponsors or Patreon. Podcasters are doing the same - many of which are self hosting. So I don’t think an ad delivery system is that needed.
I don’t see how it would have to work much differently compared to how Pocketcast or Overcast already works.
The first problem is getting content to the platform.
Easy solution: host an FTP with all the videos. It has existed long before YouTube was a thing.
More advanced solution: Torrent ala Pirate Bay. High quality videos have been distributed this way long before YouTube even supported 1080p. Peertube is based on similar solution as this.
The main problem is to attract content creators to the platform. The problem isn’t technical.
People use YouTube because that’s where you get biggest outreach. YouTube pay a little, but YouTubers mostly rely on secondary incomes like sponsors and Patreon. Both of these are viable on any other platform.
Podcasts have mainly been using this model for a long time.