I wish I could find my copy of Carolyn Heilbrun's Hamlet's Mother and Other Essays, because the essay on "The Detective Novel of Manners" would be useful.
Keller's thesis
Keller defines the predominant influences on fantasy of manners as TV (in general), the Beatles, children's literature, Georgette Heyer, Jane Austen, Dorothy Dunnett, Fritz Leiber, Michael Moorcock, and M. John Harrison and the characteristic themes as:
negotiability of social structure (he calls this "a peculiarly US view of European institutions" in the Encyclopedia of Fantasy)
the importance of disguise, especially gender disguise, as a means of that negotiation
the importance of childhood in the formation of the adult
the necessity of finding one's place in the world by being true to one's own nature
the importance of manners not only in fashion and behavior but also in language -- because control of words and of tone is power. "What these characters say is even more important than what they do."
The NYRSF article reviews works by Delia Sherman, Sherwood Smith, Patricia Wrede, and Caroline Stevermer; the Encyclopedia entry defines the subgenre's quintessential writers as Steven Brust, Emma Bull, Ellen Kushner, Delia Sherman, Caroline Stevermer, and Terri Windling.
My thesis
Any work of fantasy of manners may be reasonably described in terms of Jane Austen.
I'm not entirely joking.
(And no, formulations such as "It's exactly the novel Jane Austen would have written, if she didn't have a sense of humor" don't count. "It reads like the product of the bastard lovechild of Jane Austen and Raymond Chandler," however, is permissible.)
Having looked over the definitions above, I'm not sure that Don and I are speaking of the same thing when we speak of "fantasy of manners." Since he coined the term, you'd think he'd get final say on the definition; but I'm going to make my case anyway.
Characteristics of fantasy of manners
Tone - More than anything else, I think of fantasy of manners in terms of tone. I'm finding this hard to translate in a way that makes sense to anyone else, kind of like a synaesthete trying to explain what color a number is.
FoM books strike me as "cool" in tone--there's a distance between the narration and the event, even if the narration isn't omniscient, or else there is a focus on the wit of the telling more than the emotion. There *is* emotion, but it's more likely to be conveyed by action than introspection. In a book with multiple POVs, when the protagonist is feeling most intensely, the scene is *more* likely to be told from the perspective of someone observing him than from the pov of the protagonist himself.
Cities and modernity - These fantasies engage with what I will loosely and pretentiously call "the condition of modernity." They tend to take place in cities or suburbs or in communities characteristic of the modern world, i.e. part of extensive communication and transportation networks. The actual technology level may be contemporary (War for the Oaks), early Industrial (Swordspoint), or even medieval (the Secret Country trilogy)--but there will be a kind of fluidity of location, status, and identity that are generally defined as "modern". (In my schools. Anyway.)
Fantasy - This may seem to go without saying, but in fact I think it's important to specify, especially in relation to the use of cities and modernity. Traditionally, fantasy's been defined as pastoral and backward-looking; the definition of the genre has been Tolkien, rather than (say) E. Nesbit. These writers weren't the first to set fantasy in the cities, but prior to the eighties or so, urban fantasy had tended to be comic (cf. Unknown fantasy and magazine slick fantasy, wherein the "archaic" traditions of fantasy were juxtaposed with contemporary settings for the humorous disjunction), horrific, or limited to YA/children's books; the general tradition of fantasy was for rural/epic imaginary worlds.
"Fantasy of manners" is not synonymous with contemporary or urban fantasy, but the two subgenres did grow around the same time and there is considerable overlap.
It's in the context of fantasy that the use of contemporary and/or urban settings is so striking -- or at least was in the 80s and early 90s; it's very common now. There is science fiction that's recognizably akin to fantasy of manners, but by the nature of the genre, the urban settings do not stand out so much.
Domesticity/scope - This isn't universal -- I think the "fantasy" part of "fantasy of manners" allows "epic fantasy of manners" not to be a contradiction in terms, something that wouldn't be possible with other variants of the novel of manners -- but frequently these novels are unusually small in scale, focused either on personal destinies rather than the destinies of cities and nations, or showing those large-scale destinies from the perspective of individuals negotiating the power structures rather than controlling them. The traditional novel of manners (cf. Austen, James, Wharton) focuses on some of the most powerless of the acknowledged members of "good society", and something of that approach remains even in books which aren't predicated on extreme gender inequality.
Note also that the "novel of manners" by definition does involve "acknowledged members of good society". Protagonists attempt to negotiate a balance between social dictates and their own desires; they seldom attempt to overthrow or entirely disregard social dictates.
The term "negotiation" keeps coming up, again and again, in these descriptions. Negotiation as opposed to struggle is key here; it's the usual approach to obstacles and the presumed way to manage power.
Though this won't get in the way of a good swordfight or a magical duel with fireworks every now and then.
Themes
I suppose I do agree with Don here; I just consider this less significant to the definition than the other characteristics. And I would say that the negotiation of complex customs (whether familiar or foreign, and whether for political, romantic, or educational purposes) is the most significant of the themes, with all the others being secondary.
Literary influences
I'm a lot more reluctant to suggest these than Don is, but I will say, definitely, Austen and Heyer; and I'd be surprised if most of the people on the list hadn't read Sayers.
I think Dunnett is much less of an influence than Don does, and even where I see her as influencing the writers, it's in different works than Don cites. (E.g., Frances in Bull's Bone Dance is an obvious response to Francis Crawford, but I don't see much influence in Bull's other books; I can read Kushner's Unicorn stories as responses to Dunnett, but the novels don't strike me as fitting in the same way.)
I think you could make just as much of a case for Zelazny as an influence (certainly he's a much more obvious influence than Dunnett on Brust and Willey), although again, I don't think he's essential.
Core books
My fantasy of manners 101:
Steven Brust, the Khaavren novels
Emma Bull, War for the Oaks
Susanna Clarke, all her short fiction to date
Pamela Dean, Tam Lin
Teresa Edgerton, Goblin Moon/The Gnome's Engine, The Queen's Necklace
Doris Egan/Jane Emerson, the Stephen Price stories
John M. Ford, The Illusionist
P.C. Hodgell, Godstalk
Ellen Kushner, Swordspoint
Delia Sherman, The Porcelain Dove
Caroline Stevermer, A College of Magics, The Serpent's Egg
Martha Wells, The Element of Fire
Elizabeth Willey, all three novels
Patricia Wrede & Caroline Stevermer, Sorcery & Cecelia
Patricia Wrede, Talking to Dragons
Anthology series: Liavek, Bordertown
Some peculiarities: Everyone else in the world categorizes Thomas the Rhymer, Ellen Kushner's second novel, as fantasy of manners; I don't. Godstalk is fantasy of manners but arguably its sequels (not focused on Jame negotiating the labyrinth of the city-edifice, and the large scope of the affairs by then fully revealed) aren't. The prequels to Talking to Dragons strike me as less concerned with manners, although my memory may be faulty there.
I don't necessarily like all the books above, or consider them equally influential; but they all fit well within my personal definition of "fantasy of manners".
Precursors, outliers, descendents
Books which aren't part of the group because they predate them or aren't fantasy, but which strike me as being appreciably similar:
Lois McMaster Bujold, A Civil Campaign
Alexei Panshin, the Thurb books
Elizabeth Marie Pope, The Perilous Gard, The Sherwood Ring
Sylvia Townsend Warner, Kingdoms of Elfin
Elinor Wylie, The Venetian Glass Nephew
Books which I haven't read but which seem likely to fit:
Madeleine Robins, Point of Honour
Wendy Walker, The Secret Service
Jo Walton, Tooth and Claw
Francine Woodbury, Shade and Shadow
Books other people have suggested which I don't think work at all:
Holly Black, Tithe - reworks War for the Oaks in very interesting ways, some of which make it very much not fantasy of manners; details would be spoilers
Jacqueline Carey, the Kushiel books - exotic decadence, epic travelogues, BDSM bildungsromans, yes; novels of manners, no.
Charles de Lint - urban fantasies, not mannered enough
Llynn Flewelling, the Nightrunner books - haven't read, but I think these are straightforward adventures
Liz Williams, The Poison Master - I've only read the first three chapters, but I really don't think this belongs on the list
Regency fantasies which aren't fantasies of manners:
Karen Harbaugh, all
Alis A. Rasmussen, The Labyrinth Gate