All I hear is people suggesting palette quanticization and color matching. There's another way, using lookup tables. I got the initial idea from a posting on Flipcode.com and worked out the implementation on paper during my boring classes :) My compiler's broken, so I don't know how good this'll look, but the idea is good.
Every color is composed of 3 components, RGB. Now make those XYZ and make a 3 dimensional lookup table: colortable contains a byte index into the VGA palette, which wil be set up with a formula so that the color closely approximates the color requested.
Here's the potential problem- the palette gives you 256 entries. Assuming the lookup table is a cube, you can have a max of (256)^(1/3) intensities per component. With 6 intensities you get 6^3=216 colors reserved for the lookup table, with 256-216=40 extra colors that can be modified. This gives you the color-blending capabilities of high-color mode with the palette tricks availible in mode 13. Unfortunately, if you have 6 intensities to get between 0 and 63, the values would go {0,13,25,38,50,63} That means that if you specify
RGB(6,45,60) you'll actually get RGB(0,50,63). Again, I don't know how this'll look.
This is getting long, so I'll leave the details of the pixel routines to the reader's imagination. A hint: If you're writing a "dword GetPixel(x,y)" function that returns the RGB values of a pixel in format {RRRRRRRR:GGGGGGGG:BBBBBBBB:AAAAAAAAA}
You'll want to make a dword lookup table 0..255 containing this value so you don't need to read from the palette registers every time you draw a transluscent pixel!