I think there's a few things to think about.
One, grass is always greener. So there will simply be times when the "other thing" looks more appealing. Those times will always come and go. So look at the deep ocean swells or tides of your experiences and not the "chop" on the tops of your waves when it comes to your thoughts and emotions about a job. If you follow the chop you will be a mental wreck.
I think it's common for people to have a job and a side hobby. On one hand it's good to have a job you love and would be your hobby, but at the same time, doing it for a living has a way of ruining it for you as well. I had a friend when we were teenagers who loved to hang out with me and play at the public pool. Until that summer he got a job there and became a lifeguard. Guess who stopped ever wanting to hang out there and play?
We have ad agency people call us every few months wanted to have lunch out of nowhere and it almost always ends up with a "I'm sick of my job, can you guys make me a : director / producer / exec producer?". Yet they went to school and had all their experiences in the ad game, not the film game. But they have these pivital moments I guess where they get sick of it. I will note for the record that not one of them has ever gone on to ever produce / direct / exec produce. Whereas I've directed commercials for a 13 years now and a feature film before that and I daydream about being a quiet TD in a back room of a major effects house getting asked to program cool plugins and solutions for their artists. That's why I hang out here at mograph. I have a real job, but I admire all the things that you rugrats do. My point is. No matter what the job, you will always admire something else and focus a lot of attention on it. So you should ask yourself is it your personal interest or your career path destiny?
Another thing to think about is your age and where it's all leading you. Harry "Gray Machine" Frank had a brilliant post a few years back about age, being over the constant struggle of being a mograph artist and the long hours to appease someone else or someone else's career. When you're a teenager or in your 20's or early 30's it seems reasonable but there's a point where you grow out of your give-a-shit to do that. When you have a wife and kids you will then be questioning why you waste your life clocking in crazy grueling hours, missing time with them which is the most important commodity to you at this point. SO, understand from this 41 year old - old dog. Life moves fast as fuck. I barely remember my 30's they flashed by so quick. Make plans that set you up for how you will feel in your 40s and beyond. Work towards that always. Don't be satisfied because you got a chance to work on some AE comps or some sites for some folks. For that is just the beginning. Where is that leading to? What is your end game?