The Port Mafia is full of members with various abilities. They would hunt him down as soon as he defected from the group. Couple that with my ability to nullify anyone else's powers with a single touch... Well, he wouldn't live for long.

I couldn't just let him kill me either. Things started to get interesting and he was going to kill Chuuya as well.

[ Chuuya falling into his hands would have been BAD but also Chuuya was going to be Dazai's servant so he couldn't just let him be killed. ]