As far as I know there are no differences in the area of programming that would cause this. What version of the firmware is installed on the Mega 8 chip? You say the SSC-32 with the 168 will work when connected to a PC’s serial port directly, but not with the xbee module. What baud rate are you using? It may be possible that the 168 chip is defective. I’m not sure how we will be able to duplicate your problem on this end.
I looked over the datasheets for the Mega8 and Mega168 for clues. Here is what I think is going on.
Both the Mega8 and the Mega168 have the same guaranteed specification for the logic ‘1’ threshold: 0.6Vcc, which is 3.0V with a 5V supply. This is marginal when connected to a 3.3V signal.
The “typical” curves tell a more interesting story. With a 5V supply, the Mega8 has a typical logic ‘1’ threshold of about 1.8V; on the Mega168 it is about 2.5V. Both meet the same guaranteed spec, but something changed in the input circuit that makes the Mega168 threshold tend to be a little higher than the Mega8. Both will work fine for 5V logic, but the Mega8 will be more forgiving with 3.3V logic.
If the threshold on the Mega168 is higher than on the Mega8, the effective rise time of the signal would be changed, and it would be less likely to work at higher Baud rates. A good test would be to reduce the Baud rate to see if it begins to work at lower rates.
You could do a direct test of the logic ‘1’ threshold by connecting one of the ABCD pins to a pot and sweeping the voltage between 0 and 5V, all the while reading digital value of the pin. I would expect all the pins on the chip to have about the same logic threshold voltage.
The circuit you used to buffer the signal is a voltage follower, which would actually make the situation worse by reducing the voltage by the Vbe of the transistor. If you were to construct a similar circuit using a PNP transistor I think it might work. But you might need to reduce the 10k pullup resistor to increase the speed.
Another way to create a non-inverting level shifter is to use a logic gate from a family with a low threshold. The SN74AHCT08 looks like a reasonable candidate. (I chose this one because it is in stock at Digikey in a 14-pin DIP package in single quantities.)
You might test your level adapter by disconnecting it from the devices and connecting two 1.5v batterys (~3v) between the xbee tx input and ground, and put a multimeter between the ssc32 rx output and ground, and see how close the reading is to 5v.