Q - Is it wrong to not want to date someone that you aren't physically attracted to, even though on a non-physical level you have a great time with?
A - Thanks for the question. I am guessing that you are not already dating this person, by the way you have phrased your question.
You are under no obligation to date anyone. Many Christians have been taught that all that really counts is what is inside. I would like to tweak that statement. What is "inside" is more important, but attraction does matter. There is such a thing as beauty and it really does make a difference in a relationship that could lead to marriage - which is what dating should be about.
Here is another post I did a while back about discernment and dating that I highly recommend to you. Another post that might be of interest is this one on how far is too far.
While physical attraction is less important than finding someone that you are attracted to in other areas (personality, interests, etc.) it does really matter in a relationship and you should not feel bad about not wanting to date someone that you are not physically attracted to.
One caution - be careful of how you are defining what "beauty" is. Most women are not made by God to look like small boys (thin waifs) and yet our society has held this up as the standard of beauty. Women are made to have curves and they come in different sizes, shapes, colors and all have different features that stand out. But, all are beautiful in some way. I am not saying that there are not some women more beautiful than others, but that we must be careful not to think that beauty can't come in different packages or levels.
Remember the most beautiful woman this world has ever seen may not have been the most beautiful women (by our culture's standards) physically - Mary.
I hope this helps.