It sounds like your road association is NOT a "Statutory" road association formed under 23 MRS 3101-3104. That's unfortunate because if it were, the statutes could answer a lot of your questions. (Your members might want to consider re-forming as a statutory association, and MARA can help you with that.)
If it were a Statutory association, every owner of land that "benefits from" the road would be required to be a member, and would have voting rights and the responsibility for paying dues. That should avoid some of the friction you mention. Am I correct in understanding that you have members who are required to pay but who don't have voting rights? If so, I find that very strange.
I don't see how the association could legally require you to pay without giving you a right to vote. We have seen sort of the flip side of this, where there used to be some statutory associations that only allowed members to vote if they were up to date on their dues payments. As it turned out, that raised questions of Constitutionality, since members should not be required to pay if they have no say in how the funds will be spent. A member might withhold payment because they had a legitimate concern about misappropriation of funds, and they should have the opportunity to make their concerns known by voting against inappropriate expenditures. In your case, it sounds like some members have no right to vote even if they have paid their dues.
Do you know if your road association is a registered Non-Profit? Your association apparently has rules that should allow billing the owners whose contractors damaged the road. In a Statutory Association, there is enough flexibility to allow you to write bylaws in such a way as to require a member whose use of the road damages it beyond normal wear and tear to bear the cost of repairing it. But if no one will speak up to invoke that rule, it won't matter if it's under your current rules or under a Statutory Association's bylaws. The only way to get your rule enforced is to stand up and bring that rule to the attention of the whole membership, and point out to them that it isn't right for them to have to pay to repair damage they did not cause.
Removing yourself from membership isn't the answer. That will simply force your neighbors to bear more of the cost, which isn't going to win you any friends or reduce any of the friction. And if you are not there to speak up about the rule, who else will? You may find that if you stick your neck out and say something, others will find the courage to stand with you. The hard part is being the first one to speak up for what's right. You might want to get together with the others you mentioned who think the members who did the building should pay the cost, and agree to stand firm together.
If you are not in a Statutory Association, there is not much the Association can do to force you to pay your dues, short of Small Claims Court. If the Association were to take it that far, I would definitely point out to the judge the fundamental unfairness of making you pay while denying you the right to vote on how your dues will be spent, plus the issue of not abiding by the rule that says the members who caused the damage should have been billed for the repairs. (Bear in mind that I am not an attorney, so please don't take that as legal advice - that's simply what I would do.)
One other option occurs to me, and that is that it sounds like you may have erosion from the road going onto a water body. You might see if there is a local Lake Association that would help you get a grant to repair the road so it doesn't damage the lake, or if your county's Soil and Water Conservation District can help. Other possible sources of help include Heather Dumais, who is the State Hazard Mitigation Officer. Her number is 441-9510, and her email is heather.dumais@maine.gov One of our speakers at the MARA conference Sat Oct 1, 2022 is from Cumberland County Soil and Water Conservation District and is going to be speaking on road maintenance and how it relates to lake water quality.