By the name it sounds like it wouldn't be that tasty but believe it or not properly handled goat milk is sweet and delicious, tasting just like whole cow milk from the grocery store (if not better). Just like with anything if you let the expiration date expire it could be bitter, bad tasting, "goaty" or "bucky." If you decide to buy goat milk from the store it's available but take note that this will taste nothing like "real" raw goat milk right out of the goat.
The way your goat milk will taste will vary depending on what kind of goat you decide to get because what the goat eats could determine how it tastes. Research has shown that the strong flavor of plants such as onions and garlic could get into the milk if the goat has consumed any of it. Those two things have much more effect on the taste of the milk than eating fresh pasture and browse. With this in mind you might find great success and turn to goat milk instead of the dairy cow milk which will not be a bad choice at all.