Retaining walls are designed to prevent erosion and to keep earth from moving. They can also be built on the lower side of a slanted area and backfilled to create a more useful flat area. Retaining walls at the front of a property can be made to look attractive, but in less prominent places their appearance isn't important.
Because strength is more important than appearance for most retaining walls, a wide range of materials can be used. The least expensive material for building a very strong retaining wall is natural field stone that can be picked up off the ground. If you have a large property, especially in New England, stones won't be difficult to find. In exchange for the low price, however, you will need to put in a lot of labour. Another option is concrete bags laid on top of each other like bricks. You don't even need to take the concrete out of the bags, just stack them and soak them with water and the concrete will harden in place.
Talking some of your friends into spending an afternoon building a retaining wall is easily your cheapest option as far as labour is concerned. Turn it into a weekend-long event that involves other activities as well, and you may generate more interest. Alternatively, talk to a local temp agency or find some high school students who are interested in earning some money, and get them to help you. You can build a retaining wall alone, but if it is large, it will require a lot of labour.
You can keep the expense of a retaining wall low by building it right the first time so that you don't have to spend money maintaining and repairing it. Unlike freestanding walls, which need to be perfectly vertical, a good retaining wall tilts slightly backward in the direction of the ground that it's holding back. If you make the wall perfectly vertical, it is more likely to shift and possibly fall over. Include pipes or vents at the bottom of the wall where water can escape; otherwise your wall may act as a dam, trapping water behind it, which can endanger the wall, particularly in the winter when the trapped water freezes.
A well-built retaining wall should not require much maintenance. Inspect it once or twice a year to ensure that there is no deterioration or damage. If you find any erosion starting, fill gullies and dig small trenches to redirect water around the wall. Examine the base of the wall to ensure that the earth around it isn't being undercut by moving water.