I read a novel once which had been handed to me with the highest recommendation as a book which created a "truly alien world." And guess what? It was really well done, really alien, very deeply thought through.
I had one problem: there wasn't enough language.
Now, I don't mean to say that there was no alien language used. There was. But for example I noticed that there was a lack of biodiversity: extensive areas of the world growing with the same two or three plant species, and we kept seeing the same animal species. The same pattern was found in a few other places, too (like artifacts), where I sensed the available vocabulary was just too small.
Surely we don't want this. On the other hand, we don't want our readers floundering in piles and piles of alien vocabulary that they don't understand - they will stop reading. I've remarked before on this blog that if you want to create a sense of intimacy and internal perspective, you should avoid using alien vocabulary as much as possible, and make sure to back every instance of it up with a lot of contextual support - conversely, if you want a sense of alienness, more alien vocabulary is all right. It still has to be comprehensible.
One way to avoid overwhelming readers is to use a translation approach. Whatever concept it is that you want readers to understand, find a way of rendering it in English. In fact, I use this approach all the time, because I want my readers to feel like they are insiders (insiders=minimize alien vocabulary).
In "At Cross Purposes" (Analog Jan/Feb 2011) I had two alien concepts that I needed readers to understand. The first one was "Purpose." Our English word "purpose" was a subset, or a partial meaning, for what the aliens meant when they used it. Because of this, I could start having them use it in context in a place where the two meanings overlapped, and then slowly have them extend their use of the world with a lot of contextual support. Notice here: I was teaching my readers deliberately. I start readers in a place they can easily grasp, the area where English "purpose" overlaps with Khachee-translated "purpose." Then I make sure to extend that usage so that the reader will question the places where the meanings don't overlap, and start creating a new definition for it. I did the same thing with the word "cold" in "Cold Words" (Analog Oct. 2009). The second concept I needed readers to understand was much harder. I tried and tried to encapsulate it in a single word, but everything I found was either too corny, too awkward-sounding or too context-specific to our world. Only when I'd exhausted all the possibilities for English translation did I decide to use an alien word, "apfaa." That single term was able to cover all the multiple meanings of the social relationship of twinhood in their society - matching, symmetry, mutual social support, but also conflict, criticism and mutually beneficial argument. I had to make sure the aliens had a good reason to consider the concept consciously (they were wondering whether humans had it). Then I gave a basic definition for it: "the duality that holds agreement in one hand and conflict in the other." After that I tried to make sure all of its uses were supported by context suggesting the different ways it might be used.
I guess what I'm saying is that when you have alien language and concepts (whether those are sf "alien" aliens or just fantasy concepts that are unfamiliar), you need to decide how much language teaching you actually want to do. I'm less interested, in my stories, in teaching vocabulary than I am in teaching different ways to think - but I am very interested in teaching that.
The other thing I'm trying to say is that if you have invented a language with lots of words and all those words have history, and connotations, etc. etc. it doesn't mean that any of that information and insight is making it onto the page. If you want to show that words are related to one another, and make it meaningful to readers, you have to put in a lot of contextual support to make sure those implications are clear. If you have a really cool compound word that you love and you want to teach to readers, make sure that it's highly relevant to the events of the story, and put it in in such a way as to maximize easy understanding of it. If it's pivotal to the story's success, you might even want to break it into parts and show the meaning of the parts before you put it together. Just don't assume that because you have the grand language concept that everyone else shares it too. Remember that there is teaching going on, whether you are teaching actual new words, or just concepts and local metaphors.
In fact, there is teaching in all kinds of stories, even ones without aliens. Real world stories with local dialects? Yes, there's teaching. How about real world stories with hugely extended metaphors like "Snow Falling on Cedars," where the author spends lots of time comparing the snow and its effect on the countryside with the mental states of his main character? There's teaching going on there too. Every place you need to teach, you'll be looking for reader engagement to reward the effort of learning. So don't make people work for things that are trivial.
It's something to think about.