Just for the peace of mind of those having problem with wives and concubines, ONLY read the spoiler if it is an important factor to continue watching the drama:

He never touched any of them. They think he did. But he didn't. The only one he ever was intimate with in novel was FL and he will leave all the wives and concubine behind in the end of novel. I doubt the drama will change this aspect of novel

He is in this for revenge and to act as a corrupt official, he let's whoever wants to curry favors with him to send him spy/concubine. He doesn't touch them though... they think he does, but who knows who under the dark of night ...

I was surprised at first but seeing how the wife and concubines act, they appeared more as pawn or political marriages instead of real emotions.

Thank you for this discussion Jasmine,i totally dislike ML sharing and hope as u state the other harem members are just political pawns and 

Hope ML is devoted to our lovely FL and them be a power couple