"The way I look at it is, you've got a cancer; it's got to be removed," Rivers said of his turbulent working relationship ...