Well, lets consult the Oxford English Dictionary then, shall we?...
- prejudice, discrimination, or antagonism directed against a person or people on the basis of their membership in a particular racial or ethnic group, typically one that is a minority or marginalized.
"a program to combat racism"
- the belief that different races possess distinct characteristics, abilities, or qualities, especially so as to distinguish them as inferior or superior to one another.
"theories of racism"
...So no, Joe Biden saying Conservatism in America is antithetical to the progress of Black Americans was perhaps wrong in your opinion, but its not "racist". It's "party-ist" if anything.