I've been thinking about families for a lot of reasons. One is the ongoing U.S. debate over gay marriage; another is that I've been reading The Tale of Genji which features a very different family structure from the Imperial court of Japan in the Heian period; another is that my latest alien species, the Cochee-coco, have very different concepts of family. So I'll come back to each of these below.
What do we think of when we hear the word "family"? Well, I think of mother-father-sister-brother. This is most likely because that was the shape of the family that I grew up in. Of course, that's not the only shape a family can take. At this point I start thinking of all the permutations I know of, and really it would be both tedious and exhausting just to list them all. I'll refer you though to a wonderful Sesame Street song called "Doing the Family Thing," which shows lots and lots of different kinds of families.
What I'd like to do is start by taking a look at some of the underlying principles behind family structure, and then looking at their consequences.
The prototypical family is based on the concept of matehood - physical matehood, by which I mean two individuals who can mate with one another and produce children (not that they necessarily do). Although gay marriages don't fall into this category, the criterion is not as restrictive as you might think. Monogamy is only one example of a family type based on the matehood concept; step-families also fall into this group with the simple addition of divorce or some other form of detachability. Families born of arranged marriages fall into this group, and so do polygynous and polyandrous family types (collectively known as polygamous).
The Imperial family of Heian Japan was quite complex because it was polygamous. Here's a quote from the introduction to Royall Tyler's translation of The Tale of Genji: "...the Emperor normally had a range of recognized relationships with women, less because of sexual acquisitiveness on his part than because he was required to make his prestige relatively widely accessible to the members of the upper aristocracy."(Penguin Classics 2001, p. xiii) So in addition to the single Empress there were Consorts and Intimates who had different status, and following from this, their many children had different importance within the society.
Other families might not have followed this same pattern. While the character Genji, who is officially a commoner, ends up having four or five wives, that was probably not so much the case outside the aristocracy. Which is to say that a single society doesn't always follow the identical family pattern throughout - something to think about if you're designing a society of your own.
Another way of organizing a family is based on the concept of soul-matehood. The change between the criterion of physical matehood and the criterion of soul-matehood is both inclusive and exclusive: it excludes families born of arranged marriage, and includes same-sex couples and their families. It also easily accommodates a happy family I know with two lesbian moms and two gay dads and their children (this one could very well be seen as lying on the border between the physical and soul matehood types).
A variant of the soul-matehood criterion appears among the otterlike aliens I created for my most recent alien linguistics story, "At Cross Purposes." The Cochee-coco are born as twins, and the twin relationship is the major organizing relationship of their society - but it is flexible, so twins who don't feel compatible can separate and go look for someone else to serve as their "twin." In fact it is a type of soul-matehood that is far closer than that of human societies, but it is primarily non-sexual. Cochee-coco can choose physical mates to be their de facto twins, but they don't always.
So, I thought to myself as I designed them, what would that mean for their family structure? Well, you could have a physically and soul-mated pair who lived with their children, and that would look like a typical family of humans. Then you could have a pair of unrelated male or female soul-mates who would live with their children, which might in some ways resemble a gay family. You could also have a pair of twins - male and female siblings, brothers, or sisters, who would live with their children. These might look to humans at first glance like incestuous families, but in fact the children would not be conceived with the siblings, but with physical mates outside the pair. You could also have very large family groups where twin pairs would want to live with their physical mates and their twins...
At a certain point I realized that this was far too interesting to include in my first story about the Cochee-coco, so I kept things simple and I'm thinking about how to include it in a story later on. I have no doubt that there are ripe opportunities for humans to misunderstand the nature of Cochee-coco families and for strife to arise from it.
One useful thing to think of when working with families of different types in your writing is to consider kinship terminology. What do people call each other? In some societies, the maternal uncle and paternal uncle are called by different terms because they are seen to have different social functions. Keep in mind also that some societies allow kin terms to be applied outside the family, and some don't. In America typically a person outside the family must be very close to merit a kinship term like Aunt or Uncle, while in Japan it's pretty standard to call people you don't know by kinship terms. In Japan the term you apply generally depends on the gender and the perceived age of the person you're talking to - a young boy would be "big brother" and an older man "uncle" and an even older man "grandfather." Mike Flynn does a great job of creating a society which applies kinship terms to everyone in his latest story in Analog, entitled "Cargo" (June 2010).
Another thing to consider when working with families is child-rearing. Who does it? Mom or dad? Or older siblings? Or the entire village? Do people keep track of paternity or not, and why? Do adults talk with children at all, or do they leave that to the older children?
If you're creating a society which uses different criteria for the creation of families, think through the possible permutations that the new criteria entail. Ask yourself how the people talk about one another, and how they think about one another. What does it mean to be a brother or sister? To half-brothers and half-sisters in the Japanese Imperial family, it would mean something very different from what I think of when I think of my own brother, and something very different again from the way that my Cochee-coco would think of their siblings. What does it mean to be a husband or wife (if they use those terms at all)?
Don't necessarily content yourself with maximally restrictive assumptions. Think about all your options, and make an informed choice. Your world, and your story, will be all the stronger for it.