There is one case where it is possible for them to know your email address - if you click through on an spam email you have received some spammers will pass a identifing number back to the server that will be able to identify the response as coming from you. Of course in that case that already had your address. This is most often used by spammers that generate a bunch of random email address and send them out and use the responses to validate which address are good. Other simply track those that they do not receive a reject from and assume all the rest are good.