I actually heard the remark in its context. It was at the end of a long radio interview with a black host who was quizzing Biden about a lot of topics, mostly surrounding the legislation in the past which was not advantageous for blacks, such as the Criminal Reform Bill, etc. In hindsight, a number of things he did in the past were appropriate and even favored at the time, but now that 20 years have gone by for some of those, the times they have a-changed. And it just doesn't age well.
By the end of the interview, though, Biden had pretty much established his "creds," and he and the host were exchanging pleasantries. One final question from the host was essentially, "What is it that you offer that Trump does not offer that will help minorities?"
Biden's shook his head and laughed with a touch of exasperation, as if to say, "If everything I've just said and the core of my record, including the time as VP under Obama, hasn't convinced you to vote for me, I don't know what more I can say to convince you." And then he topped it off with the remark that made me wince at the time: "You ain't black if you vote for Trump." Of course, what he MEANT by that was that he finds it hard to believe that any sane black person would look at Trump's record and his actions that have demeaned and discounted and vilified minorities over his career, from "Birtherism" to his recent comments, thinly-disguised "dog whistles" to his base... and then decide to vote for Trump. For Biden, that would constitute voting against one's self-interest. That's the way he sees it... but the way he SAID it... rut roh.