I would say yes they should pay a fee for having the ability to access the property at anytime of the year.  Whether the owner takes advantage of the access is their own discretion.

I currently live in a Non Profit road association.  We have lots that do not have buildings, but every lot owner pays the same fees.  The fees cover the cost of maintenance, plowing, insurance, and miscellaneous expenses for running the association. This is clearly defined in our bylaws so there is no surprise.
Previously I was in a statutory road association and we did have a reduced fee for lots without any improvements.  It was defined in the bylaws that this would be 10% of the fee paid by owners with lots with improvements.