Just for the peace of mind of those having problem with wives and concubines, ONLY read the spoiler if it is an important factor to continue watching the drama:
He never touched any of them. They think he did. But he didn't. The only one he ever was intimate with in novel was FL and he will leave all the wives and concubine behind in the end of novel. I doubt the drama will change this aspect of novel
He is in this for revenge and to act as a corrupt official, he let's whoever wants to curry favors with him to send him spy/concubine. He doesn't touch them though... they think he does, but who knows who under the dark of night ...