I can totally relate to feeling overwhelmed by driving while still getting used to the process! You're taking your own life into your hands and potentially others' as well, so it can be intimidating. There were so many times shortly after I got my license that I didn't want to go to a specific part of town because I knew I'd have to parallel park on a busy street, haha. It sounds like you're doing everything right in refreshing your driving skills. Before you know it you won't think twice about loading up your trunk with groceries or going for a hike someplace far away, it'll be second nature. :) 

Also, if you do go to driver's ed for a refresher course I can almost guarantee you won't be the oldest one there! My dad was an immigrant who didn't get his license until his 40s, and there are plenty of people in similar circumstances. Even if you were the oldest, so what? 

Congrats on getting your car and re-learning how to drive!