"Fine then. We're sparring, but you're not against me. Let's go pick on your brother."
Other options:
no subject
"Fine then. We're sparring, but you're not against me. Let's go pick on your brother."