Eating Local on a Small Budget

Can it be done? Man, I hope so.
We live in one of the best farming areas of the country. Our climate can support almost any fruit or vegetable (just not tropical ones, really), and we have a long growing season. Farmer's here sell their produce as well as meats, eggs, dairy, etc. at local stands and farmers markets. I absolutely love going to the farmers market and seeing all the different things farmers bring that they've grown, raised, produced, or baked. Food from the farmers market isn't grown in huge amounts, nor is it produced en masse. Chances are that whatever you end up taking home was carefully picked off a vine or branch, possible washed, cut, cooked, whatever by hand.
Because of the care and attention that goes into the products sold at the farmers market, the food there is obviously more expensive than it is at a grocery store.
Our food budget is fairly small; we're little people and don't eat a ton. But I'd really love to start eating locally and more organic foods. Does anyone have any suggestions for this?
It may prove to be quite a challenge in some respects. Take fruit, for example. Mr. Beardface's favorite fruit is a banana. Bananas fall into the tropical fruit category and therefore are one of a small group of fruits not grown locally. Those we'd have to get at a grocery store. As far as meats go, locally raised organic meat can be more than 3 times the price of grocery store meat. It will be tricky making room for that in our budget.
We'll see how it goes this week. We've got to get cat food, cat litter, AND paper towels. This week is going to be tight with food money, but I'm determined to have healthier food. And I'm pretty sure local food is the way to go.