"The woman who he--" vague hand gesture. "Yeah, I remember, but that's nothing like--!" Oh sod it. There's no reason to try and argue how irrational the fear was when they've already established, precisely, that it is irrational.

Not thinking clearly is putting it lightly, but, "Look my point is he can't go off killing people just 'cause he's not thinking clearly! And anyway, you can't be with him every minute of the day."

It scarcely needs saying, he thinks, since obviously Riddler is here right now and Oswald is out somewhere on his own. It's not as though his relationship with Ed and Riddler mean that Oswald has stopped living an independent life.