Yo, wtf.I've always wondered this too.
Here's something for you to think about. Are the colors I see the same as what you see? For example, when I see a red apple, is the red I am seeing actually what you consider to be purple? (as in if we swapped bodies, you would see a whole different color when compared to your current body)
I don't know if this makes sense or not, but it has always been a question in my mind which no one will ever solve.