General Fitness & Nutrition: Pain in the outside of my foot

I am an extremely novice runner.  Just finished logging just over 10 miles total.  I bought a pair of New Balance shoes to run in.  They feel fine up to about 2 miles.  When I hit the 2 mile mark the outside of my foot where it meets my heel starts hurting.  While it is sore when I wear shoes it hurts when I walk.  If I go bear foot there is no pain.  After I irritate it it seems to take a few days for it to go away. 

So far this is what I have come up with as possible problems:

• Bad shoe.
• I put too much weight on my heel when I run.
• I run on the side of my foot.
• I run with that side on the higher part of the crown of the road.

Has anyone else ever had this?  What solutions did you come up with to fix it?

To fix the bad shoe, what should I look for in my next pair of running shoes?  For the second bullet point should I get something with a higher heel or something to keep me on the front part of my foot?  For running on the side of my foot I am at a loss.  I was thinking of cutting up some insoles to relieve  the pressure but I think it will make it harder over the long run.  As for the last bullet point I mixed up the sides of the road I ran on today but I didn't pass 2 miles.

Thanks

 --Joe

Thanks for the response.  I did go to a local store and got the inserts and they helped a lot.  I tried them with the shoes they have in the store and now my shoes are junk :)

--Joe