Most security experts who actually know what they are talking about do recommend Signal for most users, including [https://twitter.com/Snowden/status/661313394906161152](Edward Snowden), [https://www.schneier.com/blog/archives/2018/06/russian_censors.html] (Bruce Schneier) and [https://linktr.ee/glenngreenwald](Glenn Greenwald). Eveyone should consider whether they would rather follow the advise of people who have literally fought the NSA and read the entire Snowden documents or belive in the FUD spread by some people here.
The entire protocol is build under the assumption that you do not need to trust the servers. Let the NSA have then, it doesnt matter. On the other hand 95% of Matrix users are hosted on Matrix.org which was not only hacked several times, but would be an ideal target for any agency to compromise. Its naiive to belive the big Matrix hosts arent compromised. The only effective defense is to build your system aroudn the assumption that the server is compromised, which si what Signal did.
All decentralized protocols have this issue. The servers need to handle metadata for chat groups, like who is part of which group. If the servers are under individual control, nobody can force them to delete this data. The question is, do you trust a non profit organisation like signal to minimize and delete metadata (which court orders have proven they do) or do you trust all individuals of a group chat to do the same when you manually ask them to.
There is no reason to assume that based on the fact that they do not posess any information on their servers that would not be available to authorities anyway. You are making up nonesense based on your own delusions.