Ese no es santo de mi devoción

Not be a fan of. To not be interested in maintaining a friendship with someone, implies that the person is not good. Generally there is bad blood, or a history between the two people.