First, it's good to know what caste systems are. I suppose a general description would be something like this: caste systems are systems of societal organization that divide people into different structural categories (often ranked), each of which establishes behavioral expectations associated with membership.
Just in case that sounds vague (because it is, rather), I think it would be good to take a brief look at two examples from Earth: the Indian caste system, and the caste system of feudal Japan.
The Indian caste system has several categories. According to Wikipedia the four major "varna" are:
1. Brahmins (teachers, scholars and priests)
2. Kshatriyas (kings and warriors)
3. Vaishyas (agriculturists and traders)
4. Shudras (artisans, service providers)
By comparison, the divisions in feudal Japan (from both Wikipedia and my own schooling) were as follows:
1. samurai (warriors)
2. peasants (farmers)
At first glance, we can make several observations. First, the description of each group provides a sense of what kind of jobs you'd find these people in. Second, these systems don't cover the entire population. The Indian varnas don't include the Untouchables. The Japanese major divisions don't include the Court nobles (kuge), the Shogun and the daimyo at the top; or the Eta/Burakumin (filthy people: undertakers, slaughterers, tanners), and the Hinin (non-people: town guards, street cleaners, prostitutes, traveling minstrels and convicted criminals) at the bottom.
The idea of an undercaste is well-known in sf/f, but it's interesting to note that these groups usually don't "count" as part of a system; rather, they are perceived to be outside it. In a sense these groups are those who are considered to be exceptions to the system and not measurable within it; in the "exceptional" sense, it makes sense for the highest of the high to be excluded from a system like this as well.
According to the Wikipedia article, the Indian caste system has not always had the same degree of rigidity. Traditionally, although the political power lay with the Kshatriyas, historians portrayed that the Brahmins as custodians and interpreters of religious knowledge enjoyed much prestige and many advantages, and kings could come from any one of these groups. It was always possible for groups born into a lower caste to "rise to a higher position by adopting vegetarianism and teetotalism" i.e. adopt the customs of the higher castes. While theoretically "forbidden", the process was not uncommon in practice. This gave caste relations additional complexity. However, see my Indian friend Keyan's note below, where she connects the caste groups specifically with Hinduism (not Buddhism) and notes that marriage between castes was strictly forbidden. Indeed, one of the major elements defining castes is endogamy, or the fact that people aren't supposed to marry outside the group.
The system did become somewhat more rigid with the arrival of the English. Faced with an unfamiliar social contract, the English tried to equate the categories of the Indian caste system with their own class system and their own sense of how occupation related to social standing and intellectual ability. Unintentionally, they ended up further codifying the Indian castes with their census practices.
I think it's interesting to note that the strictness of a caste system can change over time because of historical influences - even when people are not really intending to cause change.
Another kind of change occurred in the Japanese system toward the end of its existence. The merchants began to have more power than they "should" have within the system because they handled the money, and the Samurai debts had been growing. Twice the government forgave all Samurai debts in order to restore the proper order - but you can imagine this didn't solve the underlying problem, which eventually led to the downfall of the system as a whole.
Here's a question that may occur to some of you: what advantages might there be to having a caste system (as opposed to having some other system)? I think Wikipedia makes a useful remark when it compares the Indian system to the medieval European guilds. Essentially, a system of this sort ensures division of labor and provides for apprentice training, thereby supporting economic activity (in addition to providing social groups that people can comfortably align with).
So, given these complex worldly examples, how do we go about implementing a caste system in a fictional world? I think it would be good to divide the process into three steps:
1. define structural divisions
Figure out what your structural divisions are called and how they are ranked. As you do this, make sure to give each division a concrete basis in societal function (like profession). Think about whether there are specific laws that apply to different groups concerning things like carrying weapons, attending schools, etc. Figure out where the financial support for each group comes from, and how large the population of each group is. These economic and demographic factors will have a significant influence later down the road.
2. determine the degree of mobility/fluidity in the system
A lot of fictional systems have zero legal mobility between groups. How is that enforced? What happens if a marriage occurs across caste borders? Will adopting certain forms of behavior allow someone to move up in caste so long as that person's background is kept quiet? You might also want to ask whether people in your system are in fact motivated to move up. In some systems such aspirations might be totally normal; in others it might be almost unheard of (my Varin system is one of the latter, in which many castes consider themselves better than those above them, and presumption is censured even among castemates).
3. elaborate on behaviors expected of, and accorded to, members of each groupThis is where your system will stop being a set of bland categories and start taking on real dimension. What do members of each group believe (possibly religion, but also values, ideals, and a sense of what makes a "good"member of their caste)? What kind of behaviors or manners define them? Do they differentiate themselves by elements of dress? Do they differentiate themselves by dialect? How aware are they of the other castes and their ways? What do they think of them?
There is room for an incredible amount of complexity here, especially in the area of personal details, background and beliefs. Dig in as deeply as you can, taking advantage of what you know about existing social divisions around you. Race is an obvious thing to compare to, but don't forget things as common as cliques at school. For example, literature and movies give us abundant examples of cases where not all "popular" people are happy being popular, nor are their lives easy just because everyone looks up to them.
Let's look for a second at some details from the Japanese feudal caste system:
"The shogunate carried out a number of significant policies. They placed the samurai class above the commoners: the agriculturists, artisans, and merchants. They enacted sumptuary laws limiting hair style, dress, and accessories. They organized commoners into groups of five, and held all responsible for the acts of each individual. To prevent daimyo from rebelling, the shoguns required them to maintain lavish residences in Edo (the capital) and live at these residences on a rotating schedule; carry out expensive processions to and from their domains; contribute to the upkeep of shrines, temples, and roads; and seek permission before repairing their castles." Furthermore, "only the samurai could have proper surnames."
It's at this point that I'm sorely tempted to jump into an extensive description of my own Varin caste system... but I'll refrain. Since I like to keep my sense of social groups as personal as possible, I'll just give you a quick sketch of the characteristics I've developed for the Imbati servant caste (featured in my forthcoming story, "The Eminence's Match"), using the format I explained above.
The Imbati of Varin:
1. The Imbati are ranked third out of seven among the Variner castes. They are called servants, but are highly valued: by profession they are lawyers, prison wardens and low-level magistrates (servants of the Courts), bureaucrats (servants of the State/civil servants), and valets/political assistants (servants of the nobles). They aren't legally allowed to carry weapons, but can receive training in weaponless martial arts. They receive extensive schooling and are paid well by the nobility for their service. They have a large population in all the Cities.
2. Varin has no legal mobility between groups. The Imbati are legally marked by tattoos on their foreheads (different Marks depending on their areas of service); they also typically wear black, but that is only a tradition and not a law. They could theoretically drop in status to marry, but are unlikely to because of the difficulty of removing the tattoos (which they are usually proud of anyway). They consider themselves to be the luckiest and most powerful group in Varin because they have great advisory power to the nobility, because they function as the major information conduit across the country, and because they have excellent health and education.
3. The Imbati follow the most common religion of Varin (a similarity with other castes) but unlike other castes, they value selfless service above all. Ideally they should strive to put love for their master (or for the needs of the Courts or State) above themselves. Not everyone does this, however; the strong-willed have to deal with holding these ideals in their own way, which can lead to internal conflict. Imbati also value personal autonomy. Because they are in charge of keeping secrets, they consider it impolite to ask questions, and this respect for the boundaries of the individual extends to an aversion to casual social touch. Their clothing, their manners and their speech thus clearly differentiate them from others. They learn systematically about all other castes because their bureaucratic and other functions bring them into contact with all levels - but their comprehension of other castes, though better than that of most groups, isn't very detailed.
I hope you find this post gives you some good background on caste systems, and some material to work with on your own - either for creating a caste system of your own or for comprehending the ones you see in the pages of your books.