Give me a ration if you will, but it is my opinion that you and others who go "roulette harvesting", as it is called, are, in reality, "roulette stealing". It is clearly against most, if not all casinos rules to leave the casino property with their roulette chips. I think everyone who does this knows it is wrong and I have no sympathy for anyone who gets caught.
The casino has every right to take whatever measures they need to to protect their property. Roulette chips are casino property.

To answer your question....I think the answer is clear,i'd take them back.(I'd never take them,but if I had, i'd get them back there in a jiffy.) Mayby even apologize. If 20 or 40 chips is worth being barred from a casino you enjoy going to, then keep them. Otherwise,"knock it off".

I've been meaning to express my opinion on this subject for quite a while. Thanks Scott, for the opportunity.

Later.....Rick