Is this wrong to say about someone?

Okay, so like a few weeks ago I was at my cousins house and I asked her when was the last time she saw my father ( I have two dads: one that is like a sperm doaner because he is never around ; and one that is an actual father figure who provides for me when I need things.) and she said I haven't seen your father (she was talking about the one that is like a sperm doaner) in forever. And so I asked her who she was talking about and she said "Your Daddy" and so I said, " He ain't no daddy to me he is a sperm doaner." and then she said "Don't be like that it's not his fault that he doesn't come and see you" and in my head I was thinking "He lives right around the corner from me and he hasn't seen my since I was 12 (currently 14) he knows where I live so it is his fault that he doesn't come and see me" but do you think that I am wrong for thinking this way? or would you feel the same?