Their life wasn't horrible, but it certainly wasn't full of the love and affection normal parents are meant to provide their children. Alice would so often dream of a different life. One where she was free and not a just a pretty puppet for her parents to show off to their rich friends. 

She didn’t care if she lost the lavish life, the clothes and the riches, as long as she was free. The gardens she once loved she grew to detest. The parents she once looked up to she now knows the truth of their twisted ways. 

Yes, their lives seemed wonderful, but it was all a grand illusion created by the very people they were meant to trust. James wants to be seen as more than just a replacement for his father once he gets sickly or old. Alice wants more than pretty dresses, dolls and books. Edith just wants to be loved by her parents yet she might never receive it. Henry wants to be seen as more than just an object by the other servants and the adults, more than to work like a dog with no pay. Yes, there was a roof over their head, and yes, they were fed and cared for, could you really wonder why they’d want to leave?