In Washington, engagement is a word very much in vogue. While campaigning for president in 2008, Barack Obama insisted that the U.S. must "talk to its enemies" and blasted his predecessor's "ridiculous" policy of shunning America's worst adversaries. Since coming to office, Mr. Obama has followed through on his campaign vows, reaching out to Iran, Russia and China, to the chagrin of traditional allies such as Israel, Saudi Arabia and Japan.
In "Dancing With the Devil," Michael Rubin makes the opposite case: that engaging with rogue regimes often exacts heavier costs than not and, worse, can make war with them more likely. "Diplomats diving into negotiations with rogues," Mr. Rubin writes, "are like compulsive gamblers who, no matter how much they lose, believe that one more round might reverse their fortunes."
Engaging with rogues, the author argues in this exhaustive assessment of U.S. diplomatic policy, squanders precious time, momentum and leverage. It rewards bad behavior—states that play by the rules never get the same attention—and confers legitimacy on illegitimate actors. And once begun, engagement is seldom dialed back: Unlike rogues, Western negotiators are generally loath to walk away, lest they be seen as having failed.
Mr. Rubin is a Persian speaker who spent several months in the 1990s living in Iran. Western missteps with the Islamic Republic—such as granting overgenerous concessions that Tehran then pockets as starting points for the next round of nuclear talks—are a recurring motif of this book. Mr. Obama's apparent indication that he would accept some level of Iranian nuclear enrichment, in violation of multiple U.N. Security Council resolutions, is a case in point. "Rather than enable diplomacy, he poisoned it," the author writes. "Iranian strategists concluded that defiance pays."
Mr. Rubin, a fellow at the American Enterprise Institute and a former Pentagon official, draws a distinction in his book between governments that are unsavory—even repellent—and genuine rogues. Saudi Arabia is an illustrative example: an absolute monarchy that bans churches and synagogues, prohibits women from driving and publicly beheads people for crimes from "sorcery" to "sodomy." Saudi Arabia is, by the author's reckoning, a medieval place. But it is also one whose leaders adhere to the norms of international conduct and therefore is no rogue.
Iran's leadership, by contrast, has been the world's archetypal rogue ever since it trampled the norms of diplomacy by taking U.S. Embassy workers hostage after the country's 1979 revolution. Since then, Tehran's reckless international behavior—its support for terror groups including Hezbollah and Hamas, its assassination campaigns against dissidents and world leaders beyond Iranian borders, and its decades-long record of nuclear deception—has kept it firmly in the rogues gallery.
Pakistan also earns a place on the list, for the inordinate power of its rogue intelligence agency, which time and again has been caught in bed with the Taliban, al Qaeda and sundry other terrorists. North Korea completes the roster of contemporary governments that have unequivocally gone rogue.
The concept of rogue states is of recent origin. The U.S. State Department first started labeling governments as state sponsors of terrorism in the late 1970s. Inaugural members of the club included Baathist Iraq and Syria, Moammar Gadhafi's Libya and Communist South Yemen. It was during Bill Clinton's presidency, however, that "rogue" came into common political usage. Rogues, the Clinton administration concluded, were global actors that operated outside international norms, and an entirely different diplomatic tool kit was required for dealing with them.
But since then, the author writes, officials from both U.S. parties have fallen into the engagement trap. The very structure of America's foreign-policy apparatus has itself contributed to the problem: The State Department, Mr. Rubin laments, tends to view engagement as its very raison d'être. After all, if the metric for success is merely sitting down and talking, it's easy for diplomats to claim progress. When was the last time the State Department described negotiations as anything other than "productive," "constructive" or "useful"? Surely more than a few have been unproductive, unconstructive and emphatically useless.
Mr. Rubin's argument is not that military force and sanctions are the only reliable means of diplomacy; both come with heavy costs. But the costliness of those two options, he maintains, does not therefore mean that dialogue is always the solution. Rather, timing is everything. The fall of the Soviet Union, for example, opened world-wide diplomatic opportunities for the West that had previously been unthinkable. Egypt's Anwar Sadat only made peace with Israel after concluding that it couldn't be destroyed militarily; in Mr. Rubin's words, Sadat "sought engagement only after trying war."
Equally important is leverage. Victory in the first Gulf War, for instance, gave the West leverage in dealing with the Palestine Liberation Organization, which during the war had backed Saddam Hussein. Or take the 2003 U.S. invasion of Iraq, which prompted Gadhafi to relinquish his nuclear-weapons program and Iran to temporarily suspend its own. In each of these cases, Western shows of strength convinced rogue actors that their interests lay in international cooperation rather than confrontation.
The lessons of such diplomatic achievements are clear: Consider well the costs before dining with the world's worst leaders. Choose the time wisely and set the table with at least as much vinegar as honey. Most important: If your guest begins stealing silverware or smashing plates, be prepared to get up and leave.
—Mr. Kessler is a Middle East research fellow at the Henry Jackson Society.