In the early Empire, basically until after Constantine's Edict of Milan in 327 (tolerating christianity), there was a form of state religion which treated the emperor as divine. Jews and Christians both refused to offer sacrifices to images of rulers - think, in terms of civic meaning, of the pledge of allegiance - with the predictable consequence of seeing their loyalty to the state thrown in doubt.

That's largely where Roman anti-Semitism came from; that, and that awkward habit that the Jews had of revolting every other decade. The Romans didn't have an aversion to Jews comparable to later, medieval and modern anti-Semitism, which was mainly religiously based ("Jesus-killers who use the blood of christian children to make Matzohs") and then of course, "racially". There's been a synagogue in Rome since the late Republic, over 2,000 years ago; the Romans didn't want the Jews to stop being Jews, they just wanted them to worship the emperor like everyone else. From the polytheistic point of view, this whole monotheism business didn't make a lot of sense, hence the persecution.

It's always been astounding to me how many different kinds of people dislike Jews and for how many different reasons. It's as if Jews are a projection surface for every hatred under the sun.