There's a few factors, but I have to think the price is #1.
I haven't bought tickets through the cardinals in 3 years. It's always about double what I can spend for the same tickets on Stubhub or Vivid Seats. For just the 3 of us to go to a Sunday afternoon game, it's typically 40-50 bucks just to sit in the nosebleeds. Then factor in parking, food, beer. It adds up.
Now take in a Friday or Sat night game, at least double or triple that number. If you want to actually get good seats, it's typically going to run you at least 50 each to sit in the lower bowl.
That's just not feasible for most families. It's a long season. I can't imagine why teams don't drop the price to get more fans in the seats for more than 1 or 2 games per year.