So, if they are registered then there is a way of identifying the owner of said aircraft. The determination then could be made if it was indeed an accident or an irresponsible owner or something worse (e.g., purposeful action).
I completely agree with you that the majority of owners are responsible. However, why is it such an affront to require registration of the types of UAS that could pose legitimate risk if not properly used?
The problem is that 30-40 years ago you had a few hobbyists who built RC models and used small, little used airfields (or built their own) and never really bothered anyone (such as what you describe). Unfortunately, over the years they have become cheaper, smaller, and "easier" to fly, and you end up with a whole lot of people who care not one iota about anybody else's safety, and in fact have purposely put others at risk for the 'thrill of it.'
It's a common problem: you have something that should be self-regulating and safe; however, that turns out not to be the case and ends up requiring regulations that on the surface may seem overarching. But, in my opinion (and those in the airline industry especially) some form of regulation has to be in place. They can be tweaked at some point, but it's certainly better than no regulations at all.