‘ And it was just devastating to me,” he says with a laugh