I think the biggest problem is that one ump controls too much of the game.
In football, any ref can call a penalty and they all have their jobs. Same for basketball. Control of the game is spread out fairly evenly so that one ref having a bad game isn’t as noticeable unless they are really bad like happens in Big Ten basketball.
In baseball, only one guy calls balls and strikes every pitch. He has too much control and it is easily noticeable if he sucks all game long. The other umps in the game aren’t all that important in the grand scheme of things. Their jobs are the easiest officiating jobs in all of sport, with the first base ump being a bit harder than the other two spots.
What is amazing is that we don’t hear much about hockey officiating. Those players are moving with insane speed and the puck can travel faster than a pitched ball. Yet those guys keep up. I wonder if it is because they have to be in great shape to do the job since they’re out there skating around on the ice and having to avoid getting destroyed by a slap shot or player skating backwards,