I've been driving since I was 18, but I remember that when I was just starting out, I was very nervous as well, not to the point of a panic attack, but rapid breathing, raised heart rate,... CHECK!.
Part of it was not feeling confident of being able to control the car in all circumstances. While I got over it on the public road, I think it might help you to drive a car in some kind of space where you will not encounter other cars or pedestrians etc. In that way, you will feel confident in handling the car (Ofcourse the agoraphobia think is something quite different)
Yeah I am really happy with how many empty roads there are here in MN. I am going to start saving for a car because my roommie/only friend here has a huge ass truck. I think once I have a car I will have the motivation to learn to drive it. :) (It snows a lot here so that'll make it even scarier haha)