Large number of people cannot say the difference between these two terms.
Handmade is used for something that is made by hand only, without any usage of machines. On the other hand, custom-made we use when something is made especially for somebody – by its measures/wishes or for special occasion. What is at the same time handmade and custom-made, such is, for example, clothing, in most cases has very high price.