Slow but Making Progress

This is such a slow process that it can be very frustrating, although I am begining to feel like I am making new habits, which is my longterm goal.  Whenever I walk 3 miles, the next day or two, my legs feel so tired and week.  I hope that will get better.  I want to walk 3 miles five days a week, especially while the weather is decent. 
When I walk my first mile is the hardest?  Does this make any sense?  After mile one it gets easier and I'm not sure why.