The lawyer's general comment was something to the tune of "I've never seen an HOA win a case brought by a disabled resident" regarding an access issue. So, you can take that for what it's worth.
Perhaps in the smaller, cheap complexes, you're right. The ones I've been on the boards of have $600k-$1M+ homes. Rich people with disabilities or disabled children are more than willing to pay 5-6 figures in legal fees to get what they want/need for their kids. Partly for their families, partly on principle.
The other comment made was that any HOA providing disabled access opens up the housing markets to folks with disabled kids, and the affluent ones are more than willing to outbid non-disabled folks for a home in an HOA that meets their needs. In other words, the lawyer was pointing out economic, long-term benefits for doing it, aside from mitigating risks.
Again, you can listen to that advice, or you can ignore it. The HOA lawyers get paid either way.