Refrigerator outside-will temperature extremes damage it?

Could easily cause problems in cold weather. The refrigeration circuit depends on pressure differences to force refrigerant through the expansion device. In cold ambients the condenser pressure will be low, which means the compressor will need to draw a deeper vacuum in order to move the refrigerant. To solve this problem supermarket freezers with roof-top condensers will have refrigerant pumps to move the refrigerant in cold weather.

Without benefit of a refrigerant pump you could be looking at what is called a high pressure ratio which is hell on compressors because there can be a situation where there is not enough mass flow to carry away the waste heat of the compressor motor and the compressor itself. Many refrigerators have low suction cut-out to protect the compressor, but then the freezer don't work.

Though it sounds ridiculous you might need to build a box around the unit for winter with an incandescent light bulb on a line voltage thermostat to keep the ambient temperature of the refrigerator high enough.