I think your policy of charging only actual shipping charges is more than fair. I realize, however, that you wouldn't have been able to stay in business for half a century if you weren't pricing your items in such a way as to assure yourself of a profit at the end of each and every transaction.
The bottom line here is that the buyer must take it upon themselves to evaluate the transaction as a whole. From the buyers viewpoint, purchasing an item for $5 and paying $2.50 shipping is EXACTLY the same as paying $7.50 for the item and getting free shipping. Often the buyer who got the "free" shipping gets the warm/fuzzies because of the "free" shipping. The buyer should simply look at the total cost of getting the item delivered to them and then decide if they are willing to acquire the item at that price. If the purchase results in resentment over the total cost, then, in my opinion, they buyer has only one person to place the blame on.....and it isn't the seller.