Yes, you should be very concerned. The passion for the idea is needed to push you through the really difficult times. If you don’t love what you’re doing it’s going to be harder to do everything a founder must do.
Finding a co-founder will be very, very difficult because the potential co-founders will see that you’re not passionate. Finding investors to fund the idea is going to be all but impossible because they’ll see right through you as well. It’s hard to fake passion and these types of people are going to see it missing.
Seth Godin has a great book called The Dip. It’s all about pushing through the trough when things are getting really hard and you lose your fire. He talks about how some ideas should be abandoned and how on others you must “push through the dip.”
Best of luck to you. My advice is to stop and find something that you love and that can become a business you enjoy running. :)