I used to lean that way too, that Brady was a great quarterback but not necessarily the GOAT just because his team won four, then five, then six super bowls. He was a great quarterback with the greatest coach of all time, Bill B was the wizard who was responsible for most of it. The defenses, the roster management, seemingly always ahead of the game for where the NFL is headed.
And there's probably still a case to be made, but I think it got a lot harder to make the case when Brady heads to Tampa and they win a Super Bowl and the Patriots crumble and Bill B is essentially fired.