Raster to Vector Conversion

There was a story on slashdot.org about different vector image editors. The story seemed to be brought up do to the ensuing hype about Firefox supporting SVG in a future release. Vector graphics support in web browsers is exciting to me since I have been working on raster to vector conversion software in Java for counted cross stitch patterns for so long. There appears to be a general consensus that a raster image can't be converted to a vector image. There isn't even an entry on wikipedia to any information regarding conversion. I've begun to think that there is a possible market for a service or product to do raster to vector conversion with the advent of browser support for SVG. There is an immense pool of raster images on the internet, but very limited support for converting them to vector images.
A search on Google displays a large selection of articles regarding region growing and image segmentation. The feeling I get is that the raster to vector conversion has been thought of as too processor intensive or RAM heavy to use brute force computation to solve. This might be no longer true with modern hardware. There is also an emphasis on real time processing of images which is highly time bound that doesn't fit the same market as the conversion tool I'm thinking of. Although admittedly, photo-realistic images in a vector format is not an ideal fit. But if you get past the idea that photos will get watered down by the conversion, or the file becomes too large, it just feels right to store all images as vector based. I can't back that feeling up with any evidence or hard numbers unfortunately.
Converting to vector images begs a couple of questions though. Number one is, can meaning be drawn for the vectorized image and stored for searching. I haven't solved this issue in my mind yet. It's easy to pull out lower end information from an image like colors and simple shapes. But pulling out "horse" from a picture of a horse is something else entirely. I know it can be done because my brain is doing it right now, but reproducing it in software is a something else. The second issue is video. This is the simpler of the two problems to solve. Video would be very processor intensive to encode it. The decoding process would be trivial though and could lead to smaller file sizes than MPEG possibly. I took a look at the format for MPEG and was stunned by how convoluted it is. I'm thinking that the industry is willing to take a hit on the resources and time it takes to encode a video in return for trivial decoding and a slightly smaller file size. Smaller, since only the difference between each frame would be stored to render the next one without dropping a refresher frame in every fourth time along with all the "super block" hoop jumping. If you look at what Macromedia has done with Flash, this approach just seems right too.
My primary inhibitor to working on this is hubris. I'm not Einstein in a patent office. I sell crafts on the internet. Who do I think I am to pull off something that others have dismissed as implausible? I've been looking around for someone who is doing this, and doing it well and I haven't seen it yet. If you have, let me know.