Try to be in SLO on a Thursday night, as that's Farmer's Market night downtown. Besides all the fresh produce available to purchase, there's outdoor BBQ and a host of other events going on at the same time. They close down the main downtown street for this and it's very popular with locals and visitors alike.
Apple Farm isn't really in the best location for avoiding driving at night, especially if you want to walk into town. It's pretty far away for walking. Garden Street Inn is a good option right in town and puts you within steps of the downtown shops/restaurants. The Holiday Inn Express and Quality Suites on Monterey St. are also good options, but, again, a little bit of a walk into town, although not quite as bad as Apple Farm.
SLO is safe. We do have our share of problems, so just use the same cautions you would if you were in a large city. Stay aware of your surroundings and you'll be fine.
Woodstock's Pizza, Firestone Grill, Mo's BBQ are all good wallet friendly dining options in downtown SLO. There's a Pepe Delgado's Mexican eatery on Monterey St near the above hotels I mentioned.
Golf: You'll have to drive to get to the following courses: Morro Bay Golf Course inside Morro Bay State Park, Avila Beach Golf Course, Dairy Creek Golf Course, Sea Pines Golf Course, Cypress Ridge in Arroyo Grande, Blacklake in Nipomo.
Parking: There are a number of parking structures in SLO you can use to park the car and not worry about time limits. The first hour is usually free and, I believe, .75 cents an hour after the first. Very affordable.
Whale watching is available from Morro Bay. Virg's or Sub Sea Tours are your best options.