He asked you if you want to go to the gym, not if you want to start a sexual relationship. Don't jump to conclusions.
If something more is in the cards, let the hand play out and see what happens. Can you accept him as a friend and not as a potential lover? If so, go to the gym. If not, lay YOUR cards on the table and tell him where you stand.