It was started, primarily, by wealthy Europeans who traveled about collecting and stealing articles from indigenous peoples wherever they went, including body parts. These items ended up in grandiose dens of these wealthy who referred to their collections as "curiosity cabinets" otherwise known as trophies. Of course, not really taking a studied interest in the subjects and actual meaning of the items they brought home,they made shit up to suit the stories they fancied.
Eventually actual educated persons began to apply the scientific process of inquiry, about a hundred years ago, regarding the "others" thus birthing modern day Anthropology as we now know it.
I started my studies in Anthropology thinking that archeology was what I wanted to do. After a while, having many Native Americans as classmates, I came to see that it was not the noble direction I wanted to take. I chose to study cultures and languages instead, far more interesting and honest. I remember sitting at a dinner table with a family on the rez an elder asked what I was studying and if I was one of those "diggers". I was not surprised and I got where he going with the question... then he related this little tale about archeologists he'd encountered;
He said, " Once they come around here, looking for our relatives. They asked me where some of them were buried. I said to them, 'Tell me where your grandmother is buried so I can go dig her up.' They go around and dig up our people and make up stories about them when they don't know. They don't listen to us."
Instantly I felt good that I had chosen the direction of study that I had.