Do what makes you happy. If you'll are just friends, than, personally, I don't see the problem. If your friends don't like him, ok, but just go. Stay with it until he gives YOU a reason not to like him. I'm sure your friends are great but the opinion that really matters, is yours of him and vice verse. Go to lunch. Hang out and have fun. Don't do anything you're not comfortable with. But have fun. It's a holiday. :)
Follow your heart. In my opinion, I think if you honestly do want to go to lunch with him, then go and enjoy yourself! If you don't, then just tell him you're sorry but possibly another time. Who cares what your friend's think on him? Most of my friend's don't like my boyfriend but I don't care because I do. It all mainly depends on your heart and mind set.