I've recently had the same problem - nothing will stick to silicon sealant.
I think I've had some sucess with spraying zinser primer over bits 2 or 3 mil wide - time will tell
One supposed solution I found on the web was to clean it with 100% isopropyl alcohol - you can buy it a chemist and it has 101 other uses.If you drink it you will go blind and then the silicon problem will have less significance. :-D
If the IP or zinser does'nt work, you're going to have to remove it completlely back to the original surface - including the very thin film left after you think you've removed it all.
It is a real PITA.