Is it ok to go to lunch with guy work colleagues?

Im friends with a guy in work and weve just started going to lunch together. He has a gf so its purely platonic but i worry bout going to lunch with him as people are always making remarks like oh what u 2 get up to and stuff like that. It makes me embarrased but dont want to lose a friend or have to hide him