On the topic of AI, the possibility of strong AI has around the same risk as the possibility of meeting extra-terrestrials. Any entity more intelligent than us is dangerous if it perceives itself as separate and has it's own agenda.
For that reason I favor cyborgization, or IA (intelligence amplification) rather than to build possible competition or replacement entities like AI which if it's strong AI we will have very limited ability to control if we don't build it right. In my opinion our only hope is to merge with AI so that it's not seen as a separate species but interacts more like a limb or an exocortex. A topic like AI ethics doesn't make any sense with our currently limited understanding and ability to process information to even have a clue about the consequences in the short or long term of our actions. I think we first need to amplify our ability to think and be ethical before we rush to build "strong AI", because if it is built now it will most likely be used as a weapon by the military or as a spy by the intelligence agency.
I have to say, I agree with Bostrom, we aren't ready for strong AI any time soon. Fortunately I don't think we are any where near being able to make strong AI. At best we might be able to make some sort of simulation of the brain of a rodent in five or ten years and from there learn enough to design better hardware and software (neuromorphic computing and a total breakthrough in software necessary).
That's exactly what I was talking about when I said
One might say we already are cyborgs, because we are so interconnected with the technology. Only the interface is kind of slow.
Talking about self-improvement.. My research already transformed my life so much. It influences my philosophy of life as well as how I spend my everyday life. When you think about questions like "what should the AI actually do" you can't evade the question "what should I actually do". And there is this "how" and "why" and you easily end up in philosophy even though you started from computer science / artificial intelligence...