Why when I ride my horse he loses his hair?

I have a 9 year old american saddle bred. This has been happening for 3 or 4 years now. Every time I ride when I finish I hose him off and wipe any sweat off of him. While I do this I find that he starts loses hair on the back, sides and sometimes where the girth is. These very large places that look like I have taken a surg, clippers and shaved him down, the large areas don't get sore, itchy, pink, or even look like a sore might start. I don't ride him for about 2 or 3 weeks until it comes back. I rode 3 weeks ago in KY for 2 days. When I returned home my husband takes care of the horses because of my work. So when I got home from work a few days after this ride he told me to go to the barn and check my horse out. For the first time he had lost more hair than he had ever lost. I could rub the hair off of his back, sides, around where the bridle was and where the girth was at. We did ride in some very deep mud (1 foot deep or more). But that could not have caused it. . Now because it is worse that ever it has been 3 weeks and it is growing back, but I can not ride for about 2 or 3 more weeks. My vet is a very good friend of mine and over the past years, we have tried treating this as though it was rain rot, dry skin, lice. He says now that we need to do a scap. I don't think so. I feel that it is stress and a vit. problem. If I didn't have another horse to ride I could not ride for several weeks. Can someone help me?