technically you can get oils from you skin on the gold contacts and deteriorate it over time as certain sweat like chemicals can corrode the contacts causing failures or problems in the future. You should be fine though. Haven't had a problem with that, ever! I even had some incidents where I accidentally got thermal paste on the CPU contacts or in the pins in the socket. Pretty scary stuff, but all the hardware I've had since like 2005 are still working fine even though I've been through so many disasters.

