He agrees he has brain damage but says he'd remember the incidents in the allegations if they'd happened