The Problems with Balancing 40K
First, let’s talk about balancing techniques. Speaking broadly, there’s two ways to balance a game: incrementally and in bulk. Incremental balancing is releasing relatively frequent patches or FAQs to change the game, which you see in online games like League of Legends or in the FAQs for the X Wing Miniatures game. Bulk balancing is releasing large, infrequent updates at once like in Warhammer or Warmachine. From a pure balance perspective, incremental balancing is superior. It allows for smaller tweaks that (hopefully) won’t massively upset game balance. Unfortunately, frequent updates like this require a large amount of data to work with, and put more strain on the players to adapt to their strategies. Online games can store data from every single game being played all the time, while X Wing has an official competitive circuit to receive data from, Warhammer lacks both of these. Also, imagine if Warhammer updated every week, with small changes to statlines for units that are too predominant or weak. As a player it would be both frustrating and expensive to need to change out your models or buy new ones every few weeks based on changes. X Wing already has a problem with this, where significant changes may be made to cards with the FAQs, but players with the older printed cards won't necessarily know about these updates. 40k games are too long, the hobby is too expensive, and there isn’t enough reliable data for incremental buffs to be a good way to balance the game.
The problems with bulk balancing are the ones you see already, even if you hadn’t noticed. In 4th edition 40k melee was very powerful. In 5th edition it was nerfed a bit, but alpha strikes became incredibly powerful. 6th and 7th over compensated, making both alpha strikes and melee very weak, and we’ve been in that state of the game for a few years (as we were with 4th and 5th as well). For people who are willing to look deeper, GW actually has a pretty good track record for trying to compensate for overpowered/underpowered strategies, they are mostly hamstrung by the necessity of bulk balancing for their games. In addition to the exampls above, Space Marines lacked a way to deal with monstrous creatures while Tau couldn’t handle death stars. The solutions were grav weapons and the Hunter Contingent, which both accomplished their goal, although grav did so too well.
The last problem with balancing 40k is that leaving it up to the players is almost always a terrible idea. Games like League of Legends and X Wing balance around top-tier play while 40k, lacking an official competitive circuit, doesn’t really have the data to see what the top players are using except through independent tournament results. If you’re asking why balancing around what all players want rather than the top-end is bad, the answer is that it punishes low-skill cap, armies much harder, while leaving incredibly powerful but hard to master armies untouched. For example, in reddit discussions you very often see complaints about Tau being wildly overpowered, but Daemons are almost never even mentioned as a top army. This is because Tau are fantastic at sweeping low-to-average skill players, while daemons are difficult to master, but Daemons almost always out-perform Tau in competitive events. Nerfing Tau based on the wishes of lower-skill players would only hurt the diversity of competitive events, while probably not changing much on the bottom-end anyways.
To elucidate with a real-life example: I went to an event awhile ago where the TO arbitrarily banned all formations, superheavies, and gargantuan creatures. He was then very confused about the number of daemon and tau players at the event. When I told him he had buffed both of those armies, he said he had only removed things, and I had to explain to him that removing things that countered both of those armies made them much stronger. Stomps meant there were few ways to deal with daemons’ rerollable 2+ invulnerable, while tau don’t lose much from lack of formations.
I hope this article gives you some thoughts on why balancing the game isn’t as easy as it sounds, and why most players aren’t qualified to be doing so. As always, you know I advocate casual play as centering around fun for both players, so don’t let the competitive meta dictate what you play with, but consider the ramifications before you arbitrarily change the rules of the game.