There have been some self learning robotics labs that have been working on a robot named Cog for a long time now. I'm not sure what state it's development is at. In the early stages it was just a head two simple camera's and they had to "train it" to follow a light.
If you really want to learn about this stuff, go to MIT.
http://www.ai.mit.edu/projects/humanoid-robotics-group/cog/current-projects.html
I remember another where the robot did have the code to learn to walk, but the creators would spend 6-7 hours at a time helping 'teach' the robot to walk while it learned and adapted slowly like a child. Unfortunately at the time hard drive sizes were much smaller and the amount of data the robot learned in order to walk was so much that they didn't have storage for it and the robot would "forget" everything when they turned it off and they would have to train the robot from scratch the next time. This was 5-10 years ago so things have likely advanced quite a bit since then.
In terms of doing this with an operating system or a watchdog program that runs beside the operating system as a user and is allowed to make changes to it... well, that's a big task to get a program to understand the context of it's safety and the consequences of it's choices.
Personally I would start with game AI and pathfinding AI. And with getting your programs to make decisions based on weighing several educated assumptions as opposed to specific facts and static rule based decision making. You'll want to read up on Fuzzy Logic. To over simplify it, it's how to get a binary system that only understands yes and no to understand a 'maybe' or to get the computer to take an educated guess.
CodyBane++,
web application development