Sometimes I wonder whether the villainy is beside the point. For both Iago and Petyr Baelish, the art of manipulation is the true source of pleasure.
Not very many writers have attempted to reproduce Iago. In the last case of Hercule Poirot ("Curtain"), Agatha Christie made an effort, but the Iago she created, a man whose name I had forgotten and had to look up on wikipedia (it's Norton), left very little impression. His techniques might be described artfully, but the character itself was faceless next to the vicious and mesmerizing Iago. Although not a complete success, Christie ably distilled the genius of Iago --- he extracts his revenge without lifting a finger. He pulls the strings, and others gladly obey without feeling the tug.
As far as I have seen, no character has come as close to Iago as Petyr Baelish, a smallish but rather charming middle-aged man from a barren and dreadful piece of land no one, not even himself, wants. He inspires a sense of awe in us, mixed with horror, revulsion, and a bit of pity, just like Iago. (Iago only wanted a job promotion, damn it.)
If we were to believe one of the later interpretations of Iago's motive, ie, he is in love with Othello and could not bear to watch Othello love Desdemona, then Petyr Baelish follows a perfect symmetry in A Game of Thrones, Book 1 of the series. Perhaps his revenge is not as perfect as Iago's, because he has not been able to make his love, Catelyn Tully, hate her husband Ned Stark. Nevertheless he is able to smoothly get rid of the pesky romantic rival. I have no doubt that if only the events occurred in a different way --- after all, Baelish couldn't control the progress of the war or Lord Tywin's Campaign of Letters (a term invented by me) --- he would have tried to reunite with his Cat some day soon.
The awe inspired by both Iago and Baelish is their invisibility, how they move events forward through other people's VOLUNTARY behavior. Their nudges are so subtle as to be undetectable. They plant ideas into someone's head and make him believe the idea to be his own. Iago is no Richard III. He does not need to dirty his own little hands.
A key piece in this art is, as GRRM correctly noted, the lack of apparent motive. Shakespeare stumbled slightly on this one. Iago's motive is indeed pretty weak: My boss has blocked my career success and hurt my feelings, so I have him murder his wife in revenge. Hmm, that's a bit ... mad, especially because career and romance are so not in the same line of thoughts. (OK Iago does try to implicate his professional rival in the process, but the disproportion is still pretty big. What's Desdemona ever done to him?) No wonder Othello never suspected his lies about Desdemona. No wonder a theory of romantic rivalry was concocted to make more sense of Iago's motive.
Norton's motive is nonexistence, although it turns out that motiveless manipulation to inflict pain on others is in fact a real phenomenon among psychopaths.
Petyr Baelish is more successful in this aspect. In some creepy uncomfortable way we can empathize with the rage of the 15-year-old boy, lying there bleeding and watching the girl of his dreams walk away arm in arm with her handsome, tall, and strong fiance from a prominent family. The rage that has never been soothed or released, and has been festering into a gasping black hole in the place where his heart used to be. "I will never let this happen to me again."