No matter the size of the area within the castle walls, it did not include fields or peasants' homes and the vast majority of commoners lived outside the castle walls. Only a very few commoners, those directly in the employ and service of the daimyo or his retainers, lived within the walls.The Samurai stayed in the town surrounding the castle and the higher rank they were, the closer they stayed to the castle.
Many castle towns grow into Japanese cities over time. Tokyo and Kanazawa are two good examples of this. Merchants and artisans lived in special areas, while temple and entertainment districts were usually located just outside the cities, near the castle.
Farmers
The peasants who farmed the daimyo's estates were often poor but ranked next to the samurai because of the importance of their work. While the farmers grew a lot of rice, they could rarely eat it because they were forced to pay high taxes and the price of rice was expensive. The peasants instead ate barnyard grass or millet. Officials were assigned to manage the farmers' rice and land and about half of their harvests were taken away by the lord of the castle.
Merchants and craftsmen
Merchants and craftsmen were lower on the social scale than other classes. Craftsmen included carpenters, coopers (a person that makes barrels) and lumbermen (people who cut and prepare wood) and provided goods for trade in and between the towns. Merchants and traders produced luxury goods such as fine clothes and ivory carvings for rich samurai to buy. While merchants were in low regard and were often poor, the wealthier merchants were sometimes richer than the samurai. Some privileged merchants and craftsmen commissioned by the daimyo, were allowed to live adjacent to the highest ranking samurai.
Entertainers and shopkeepers
Over time, towns grew to include entertainment quarters, where off-duty samurai and other townspeople could eat and drink, relax in a bath-house, listen to music or go to the theatre. Theatre-goers could see popular plays performed by actors, or plays performed by puppeteers. Bookshops were popular with visitors and barbershops were common places for people to chat.
Other townspeople
Outsiders (named eta) were people with professions that were considered impure and they formed the lowest class. They lived separately from the other classes and they worked the land to make leather, or worked as minor guards and town cleaners. Outcasts included people who were poor or those punished for crimes they had committed.