This post was requested by CWJ, my friend from the forum over at Analog - thanks so much for the question, CWJ! It also strikes me that this may be a timely topic for people who are considering the Na'vi language that was used in Avatar.
First, the Chomsky question. Chomsky proposed the idea that there was some basic sense of grammar universal to all humans, that was passed on as an instinct.
Now, human languages are very diverse. The most thorough article I've seen on this topic was recently published in the Economist, and you can check it out here.
In fact, it's hard to say how much of human language is innate and how much is learned. Humans are oriented towards language from birth or even earlier; this is well known, as newborn infants prefer to listen to language sounds over non-language sounds, and their mother's native language over other languages (studies measured strength of sucking response!). They also go through a number of language development stages, like early babbling, even if they don't have any auditory language input (say, with non-hearing babies). Non-hearing babies are also known to babble with their fingers. People have also looked at pidgin languages, which tend to take on grammatical structure - and very similar grammar structure - when they're passed on to the second generation, and used this as evidence for a more extensive innate language faculty.
On the other hand...
I've been quite impressed by research which looks at language acquisition from a neural-network point of view. Neural net computers have been shown to learn language patterns like English past tense -ed in much the same progress trajectory as human kids. I've also heard about research that says re-occurrence of phrases may play a larger role than we thought in language acquisition. Certainly human brains are very good at tracking the frequency of occurrence of things (sounds, words, etc.) - a critical skill for language learning. So in the end I'm not personally convinced that grammar is really what's innate and at the root of the drive for language acquisition.
I suppose if there were some kind of universal grammar beneath all human language, then it might be restrictive for the learning of alien tongues. Since I'm not really in the innate grammar camp, I don't think that the primary restriction on learning alien tongues would come from that department.
I think it would come from perception problems. More on this below.
The evolution of language is not simply a matter of brain evolution, but of the co-evolution of the brain, the ear, and the vocal tract as language developed. As a result, they are all well-suited to one another, and babies can hear any language sound from any language in the world, and learn it natively, given the opportunity and a normal course of development. We listen for audible building blocks from the vocal tract that are put together sequentially. Our brains are excellently tuned to process them, and we associate them with physical, temporal and social context in lots of complex ways. But alter these very basic prerequisites, and the problem becomes much harder (even if we assume maximum language-learning ability like that of a child). Sign language shows that the language stream need not be auditory. But what if the language stream is not sequential but simultaneous? Or what if the language producing organs of the alien create a language stream that our eyes and ears are unable, or only partially able, to perceive?
Sheila Finch's stories about the Guild of Xenolinguists speak more directly to the kind of language problems that CWJ mentions - basic problems of auditory vs. visual, processing in the brain and such - than my own. In fact, CWJ, if you haven't read them, you might find them very interesting, because she takes a very Chomskyan approach to her concept of language. She has humans being able to understand all sorts of things, but requiring special drugs to make them forget their existing categorizations of perception, for example. Speak with her in person, though, and she'll tell you - as I will - that any communication with aliens would be next to impossible.
In the realm of animal communication on Earth, we're still discovering things, like the super-low sounds produced by giraffes. We're working hard on the communication of dolphins, too (see a very interesting article on dolphin intelligence, here). We've taught some creatures how to interpret basic signals on a behavioral basis (I remember Mike Flynn having an interesting post on the nature of communication - I'll see if I can find the link). But we haven't really cracked any codes. One of the things that can cause misunderstandings between humans is differences in categorization of concepts - places where the two languages file things differently, as when the Dutch say a picture is "up the wall" instead of "on the wall." A creature that lives underwater and perceives its world through sonar signals will have a totally different way of perceiving the world. It may not even conceptualize the separation of objects as we do. What does it do to language concepts when the means of producing language (sound) is the same as the means for perceiving one's surroundings?
So effectively I think it would be hard to recognize alien language as language at all, and probably harder to try to "break it down," especially in a situation where the physiology of the aliens in question, and their environmental context (not to mention social context) were unknown.
This doesn't stop me from designing alien languages, obviously! As far as constructing the languages goes, I think it really depends on the author's intent with the story, and the nature of the primary language problem in the story. If decipherment is your primary problem, then you can really embrace problems of channel (auditory/visual etc.) and the identification of structure. If your primary problem is one of first contact and code cracking, then you can do some channel stuff, or you can focus on grammar or phonetic dificulties, pronunciation difficulties, etc. If your primary problem has to do with cultural issues and misunderstanding, then it's helpful to create a language and assume that humans have already cracked the code, which allows you to place the focus where it really belongs.
Wow, that became a long post! I hope you find it helpful. I welcome any questions, followup, "what-the-heck-did-that-mean-can-you-explain-this-bit-again-please," etc.