Cemetery is the much older word, going back to Roman times. Today, a cemetery refers to a large burial ground, typically not associated with a church.
The first citation in the Oxford English Dictionary for graveyard comes from 1767, and a graveyard is typically smaller than a cemetery and is often associated with a church. It is part of the churchyard.
Cemetery appears to be the more commonly used word today, perhaps because it’s been around longer, perhaps because people like the sound of it better, or perhaps because there are so many more people buried in cemeteries because they’re so much bigger than graveyards. It was actually the population growth in Europe that led to the creation of large cemeteries because the small churchyards could no longer hold all the dead, so I’m inclined to think their popularity as a resting place is also the reason the word it more popular.
And here’s a bonus—do you know why sailors called the late shift the “graveyard watch”? It’s not because you feel like you’re going to die, although that may be true while you’re adjusting to the odd hours. According to the Oxford English Dictionary, sailors called the shift from midnight to 4:00 AM the graveyard watch because of the silence and because of the number of disasters that occurred during these hours.
That’s your quick and dirty tip: Most people don’t know the difference between a cemetery and a graveyard and not all dictionaries even call out a difference, so you’re not likely to be criticized for using them wrong, but if you want to feel precise, you can use the word cemetery when you prowl around a huge urban burial ground and graveyard when you linger in the burial ground next to a small country church.