We are looking into using Amazon S3 and EC2 for scaling out a Java software solution for a new Web 2.0 service we are building.
The thing with Amazon S3 is that it is neither a fully CAS (Content Addressable Storage), distribued filesystem or a block-device. To give a brief background:
A full CAS solution allows for a simple file system like way where each object has a unique id and each object can be randomly accessed. Objects can vary in size, but the normal file system organization of data do not take place, each object just has a unique ID, and a data part. It works much like a large hash table that abstracts the actual storage for you. It holds no or very little meta-data. Most CAS solutions have custom API's and not many databases or other software components support CAS directly today.