I was also waiting for someone else to answer this one! I think the suggestion to try mediation is excellent. It sounds like this is not a hostile situation yet, but could become one if not handled carefully. Professional mediators are trained in how to get both parties to listen to each other's concerns and understand each other's viewpoints so they can reach an agreement. You could try FCM (Family and Community Mediation.) Or, if there is agricultural or forest land involved, you may be eligible for mediation through the Cooperative Extension's Agricultural Mediation program, which is offered free or at nominal cost.
As for who has rights under the law to do what, that's more complicated. I would want to look carefully at the deeds to see exactly what is granted. Part of the beauty of the road association statutes is that they do allow quite a bit of flexibility when it comes to deciding how to assess each property for its fair share. I think it's generous for this landowner to maintain a significant stretch of road at his own expense (other than asking for gravel) AND to pay his dues. In the old days, public roads were often maintained by the abutting landowners, who were then given credit towards their property tax, based on their time, materials, and equipment.
I think my biggest concern would be liability, and you might want to point that out to this landowner. If the road association has liability insurance, it probably would not cover the landowner if anyone were to be injured or to suffer property damage as a result of the way he maintained (or failed to maintain) the road. But I hate being scared by attorneys and insurance salesmen! If he's doing a reasonable job of maintenance, and apparently most of the members agree that he is, then what is the chance that anyone would actually make a claim? He could somewhat reduce his liability by posting "pass at your own risk" signs. Again, a good mediator might be the best bet for sorting this out. I would request a mediator who is also a real estate attorney - I know there are some.