Ugly and Poor

This is a self-deprecating joke where the speaker implies that even if he had money, women would still find him unattractive. It turns the expectation around, because you would think money would make him more attractive. The punchline is that even money does not make him less ugly, and is now also considered poor.