Just for the peace of mind of those having problem with wives and concubines, ONLY read the spoiler if it is an important factor to continue watching the drama:

He never touched any of them. They think he did. But he didn't. The only one he ever was intimate with in novel was FL and he will leave all the wives and concubine behind in the end of novel. I doubt the drama will change this aspect of novel

He is in this for revenge and to act as a corrupt official, he let's whoever wants to curry favors with him to send him spy/concubine. He doesn't touch them though... they think he does, but who knows who under the dark of night ...

It's normal if you think about it. He is one of the powerful man in the kingdom. So it's kind of expected that the royal and noble families will try to stuff his backyard with spies especially the royal family in the name of bestowing a marriage. 

I was surprised at first but seeing how the wife and concubines act, they appeared more as pawn or political marriages instead of real emotions.

Yup, it will be more unrealistic if he doesn't have any in his backyard. 

Thank you for this discussion Jasmine,i totally dislike ML sharing and hope as u state the other harem members are just political pawns and 

Hope ML is devoted to our lovely FL and them be a power couple

These are arranged marriages and lots of Cdramas are in this premise hoping that once they are already inside the household they can gain the trust and love of the ML. But since this are all political marriages we can see that the ML spend most of his times inside the room doing a mural. He also said that he is not interested in women or he has not found the one.