Work relationships are dangerous for oh so many reasons. Here is my take on your situation.
Your are probably dealing with a wannabe player. He disappeared for awhile for the 4 weeks because he found another object of his affection. This probably went South because the guy is a jerk or he got what he wanted. You were probably seen as a backup or possibly item.
Somewhere in the conversation you scared him off or represented a threat to continued employment. Whatever the reason leave this opportunity alone because it has no legs. The only reason to continue is if you are interested in casual sex for the short term
“Better a true enemy than a false friend.”