Work Log of Apple Power Mac G5 Quad Liquid Cooling System Repair

Introduction

This text represents a work log of a cooling system repair done to a Power Mac G5 Quad (version 1 LCS with a single pump). I have also gathered links and related information to go with it as finding out the necessary bits was rather daunting; it is easier to have all the information in one place. The text, naturally, is written on the repaired Quad.

Note that all of the photos are links to a higher resolution versions of the same photos.

Changelog

Motivation

Symptoms of a liquid cooling system (LCS) failure include:

The LCS has very good cooling capacity. If idle CPU temps are high, it is likely that there is something wrong with the unit. Slightly above or below 40 °C would appear to be a reasonable expectation for idle temperatures.

Note that running "Thermal Calibration" from the Apple Service Diagnostics disc (version 2.6.3 for this machine), although tempting, does not seem like a wise action until the LCS is fixed, and probably not even then. Thermal calibration sets various values that depend on the performance of the cooling system and running it on an impaired cooling system will probably fail and could cause bad calibration data to be written leading to easier destruction of the CPUs, or it might even damage the CPUs while running. This, however, is just an opinion with no knowledge beyond publically available information.

In the present case, idle machine, even in "Reduced" CPU setting, ramped up the fans to maximum rpm slowly over several hours. Stressing the CPUs brought one core near 100 °C. An Apple service center nearby diagnosed the machine (for several days) and told there was nothing wrong with it and that the fan speeds and CPU temperatures were normal.

Prior art

Similar LCS repairs have been made by other people. Some of the results are even posted online. A list of useful links is found in the References section below.

Required tools and materials

The following tools and materials were used:

The r134a service port is an air conditioning service port. Port couplers can be found from A/C service equipment suppliers, and the couplers are rather expensive, but in my opinion a worthy buy for the project. Hoses and fittings are available from hydraulics stores. Alligator clips can be found from hobbyist electronics stores. Springs clamps, hose removal pliers, and fuel pump gauges are sold by automotive parts stores. Aspirator pumps are available at laboratory equipment suppliers.

R134a service port coupler
The service hose

Preparing for Assessing the Results

For assessing the results of the repair, a Linux-based boot CD was produced that runs a series of test and logs them. The CD uses a modified Linux kernel that sets all the fans and pumps to maximum rpm. If normal fan control was used, comparison of results would be meaningless. A stress testing program stresses the CPU cores one by one, and combined, and logs all of the cooling system sensor values while doing so. The logs can be saved to a USB stick, for example.

The stress testing program was run once on the original system to have the necessary "baseline" for comparison. Comparison results are included in the last section.

Take Apart

Apple's Power Mac G5 (Late 2005) Service Source can be used as a reference of how to pull the processor unit out of the case. Further take apart steps are listed here.

The following general guidelines might prove useful when taking apart and handling the LCS:

Before taking apart the LCS, it might be useful to test it outside the case using an external power source to power the pump. For example, it is quite easy to diagnose low coolant situation by turning the unit around and listening to the pump sound. The pump crackles and/or is very noisy when pumping air. See the section under Cleaning for further notes on how to power the pump with an external power source.

CPU Cards

Before removing the CPU cards, the metal support that attaches to the LCS "handle", where the serial number sticker is, needs to be removed first. It is held by two 2,5 mm hex screws. Removing a CPU card from the LCS happens by removing all, save one, of the black machine screws and removing the four spring screws that mount the CPU. The black screw at the center of the copper block is the one that does not need to be removed. A 2,5 mm hex driver is required.

It is probably best to remove the CPU mounting screws in a cross pattern and loosen them gradually, that is, by making several rounds. The pressure on the CPU will thus be released more evenly.

Removing the black screws is best to start from the copper block. Leaving two screws at furthest corners of the card and, of those, removing the top one last should allow easy removal of the CPU card, without risking scratching the CPU or the CPU block base.

After removing the CPU cards, the four captive screws that required ballhead hex driver are no longer captive. They should be removed at this point.

The LCS with CPU cards removed, bottom view
The LCS with CPU cards removed, top view

Service Port

The LCS has a R134a high side service port (the one with the black cap). The cap screws off the port easily. Removal of the valved port itself is less obvious as the hex nut at the base of the port is welded to the radiator. The port can be removed by holding the hex nut steady with a wrench while gripping the top part (which was covered by the cap) of the port with pliers and screwing it out.

R134a port with cap on
R134a port with cap removed
R134a port cap

Draining

The LCS should be drained before further disassembly. Draining can be done through the service port, by disconnecting one of the hoses or by opening one of the CPU blocks. Using the service port for draining will be much less messy than the other two ways.

The LCS was drained by connecting a plant sprayer handle through a hose to the service port and pumping most of the coolant out. Occasionally the system had to be "air fed" by opening the connection between the service port coupler and the hose because a plant sprayer doesn't pull much of a vacuum. When about 100 ml was drained, a hose connection was opened on the radiator and the rest of the coolant was poured out through from there. About 175 ml of coolant was recovered (and a few ml spilled). The LCS probably contained about 180 ml of coolant.

Drained coolant

CPU Blocks

The copper bases of the CPU blocks are fastened to the plastic tops by four screws. Before removing the screws, it is useful to mark the screws with a felt tip pen, so, that a cross of cardinal directions is overlaid (imagined) on top of the CPU block, aligned with the screws, and the outer edges of the heads of the screws are marked with a line along the cardinal directions. Having the screws marked will allow tightening them later to a similar extent as they originally were.

The screws are coated with yellow threadlocker adhesive (possibly Loctite). They should come off without damage, but the screwdriver needs to have just the right size Philips head to avoid stripping the screw heads.

With the CPU block screws removed, the copper base detaches easily. It might take a red O-ring and a red weird foamy pad along with it. The red O-ring, the red weird foamy pad, and the white maze can be removed.

CPU block with screws removed
CPU block with base removed
CPU block top and white maze
Inside of CPU block base and O-ring
Inside of CPU block base (the center square is composed of etched microchannels, open the photo in another window for a full res view)
CPU block O-ring and red foamy pad

Hoses

Removing the hoses can be done cleanly by using hose removal pliers and a hose removal pick. With the pliers placed at the end of the hose, a pick can be used to pry under the hose and lift it over the pliers. Then the pliers can be used with a pumping action to remove the hoses. The action needs to be slow enough for the rubber to react. If the hose seems stuck, it can first be loosened by gripping it with pliers (with a piece of cloth in between for protection) over the fitting and twisting it back and forth a bit.

However, the hoses are rather tight which makes the risk of damaging the barbs quite high. It is probably better to cut the hoses off by prying under the hose with something plastic to lift the hose a bit and cutting the lifted part longitudally with a carpet knife.

The inlets of the CPU blocks, with hoses coming from the radiator, contain a small screen filter, an AC orifice tube filter, actually. Care should be taken when removing the hoses to not damage the filters.

CPU block top showing some residue
Screen filter and white maze of a CPU block

It is easiest to detach the hoses from the radiator first, then the pump, the T-fitting, and finally the CPU blocks. Removing only the radiator connections would allow an easy route when not replacing the hoses: the pump and the CPU blocks are less likely to be clogged and to need clean up, than the radiator, and just cleaning up the radiator thorougly might improve the cooling system a fair bit. Back flushing might be enough to remove debris from the CPU block filters. Old coolant should, however, be flushed from the other parts of the system. If the ends or other parts of the hoses show cracking, the hoses should be replaced. A prime suspect would be the hose from the pump to the radiator.

CPU block top, hoses removed
The T-fitting

Pump

The pump was not taken apart as Torx heads of the required sizes were not handy. The pump is unlikely to be clogged so much that it would need opening. (See the separate pump exchange operation for more information.)

Clean Up

Powering the Pump

The pump was used for running water through the radiator while cleaning it. Any other other pump or direct faucet connection would do as well, or better.

The pump has four wires attached to a 12 pin connector. The pinout of the pins that are present on the connector is as follows (pin numbering dictated by the connector):

pin # usage
1 +12 V ?
3 Tachometer
5 Motor control ?
7 GND

The pump uses DC current and apparently needs a bit more than the rated 1.8 A for powering up. It was connected to a molex connector on a PC server PSU using alligator clips and a male molex from an old 486 heatsink assembly. Pin 3 of the pump was protected with a piece of plastic jacket from a wire (in the present case, that was already too late ).

The pump tachometer of the DDC-1 version used in the Dual G5s outputs 12 pulses per revolution. The DDC-3 tachometer is specified as 2 pulses per revolution, like normal PC fans.

The connections for powering the pump were +12 V to pins 1 and 5, GND to GND. Connecting +12 V to pin 5 runs the pump full rpm, so it is much louder than in normal use in the G5. Applying lower voltage to pin 5 will also lower the pump's speed.

The pump power contraption
Alligator clips connected to the pump connector
Alligator clips connected to the PSU

When running the pump it is important to ensure that it always has liquid in it. Running the pump dry will eventually damage it. The pump should not be oriented with the barbs pointing downwards or the pump cover facing downwards.

Radiator

The radiator is the most important part to clean up. It could be cleaned up using automotive cleaners for aluminium radiators. Not having any around, the following method was used:

  1. Boil cold tap water.
  2. Mix a 30:70 to 50:50 of white vinegar and hot water.
  3. Pour the mix in the radiator.
  4. Let sit until cool to the touch. Shake and pour the liquid out.
  5. Inspect the liquid against a bright light for debris.
  6. Go to step 1 until no apparent debris visible.
  7. Connect the pump to the radiator and run a few liters of water through it with all routing combinations.
  8. Rinse a few times with distilled water and inspect the water coming out for debris. If still not clean, back to step 1.

Setup for radiator cleaning with vinegar
Setup for radiator flushing using the pump

Cleaning the radiator took a lot of time. First rounds were made without vinegar, which proved to be too inefficient. Using stronger acids might damage the radiator.

Other Parts

The copper CPU block bases were cleaned up by soaking in white vinegar for a while (maybe 30 min) and then brushing with a toothbrush. Plastic, silicone, and rubber parts were be cleaned up by soaking in hot water spiked with a drop of dishwashing soap. Cleaning of all parts was finished by rinsing in distilled water.

Rebuild

Materials

The original hoses are polyester(?) reinforced rubber of some low-permeation rubber. They are moulded to shape. On the hose segments, there are faint imprints which are probably part numbers. As EPDM seems to be the industry norm in water/glycol cooling lines, it was chosen as the replacement hose material. Because the original routing has very tight bends necessitated by the very little room in the LCS, a non-moulded hose of similar dimensions and material as the original will kink. Therefore, a thicker walled hose had to be selected.

The specific hose that was used was Codan 3310 Air and Water Hose, 9.5 mm (3/8") I.D. and 16.3 mm O.D. with a minimum bend radius of 50 mm (which is exceeded grossly). Plastic tubing, such as the Tygon R-3400, might have been of better minimum bend radius, but it has worse temperature range. The Codan hose was also a lot cheaper than getting plastic tubing from some computer water cooling supply. Judging by marketing talk, the Gates Flexcord Plus seemed like the best material that would have been available for the application.

Thicker O.D. hose also needed new clamps. Spring clamps were selected because they would appear to have the best performance in a system with constant thermal changes. Norma FBS Constant Tension Springband clamps were the only spring clamps that were readily available. 16 mm size was chosen based on the hose O.D. The clamps have 12 mm band width, which does not leave much clearance in the LCS. Narrower clamps would be ideal. Clamp width is especially a problem on the CPU block fittings and the T-fitting. Clamps should be placed right behind the barb, not over it.

The original O-rings appeared to be silicone. EPDM replacement O-rings measuring 31,37 mm x 1,78 mm were purchased but not used as the originals seemed to be in good condition. EPDM looks like the standard practice for this application as well. Viton was not recommended for hot water applications by the O-ring supplier. Later, information that other O-ring types might not seat leak-free into the plastics tops was received - another reason to reuse the originals if they are not damaged.

Hoses

The original hoses had approximately the following dimensions:

Hose Length
Pump Outlet–Radiator 192 mm
Pump Inlet–T-fitting 170 mm
CPU A Outlet–T-fitting 192 mm
CPU A Inlet–Radiator 255 mm
CPU B Outlet–T-fitting 142 mm
CPU B Inlet–Radiator 175 mm

The same dimensions and routing were used for the replacements. The minimum bend radius of the hose was far exceeded on the T-fitting to pump inlet line. Another line that exceeds the bend radius visibly was the CPU A outlet to T-fitting line. The hoses did not kink, but flattened somewhat at the bends.

The hoses were first cut to length by wrapping a straight piece of paper around the cut point and gripping the point with pliers that have a straight side edge. Then a carpet knife was used to cut the hose, pressing the knife towards the pliers to get a straight, clean cut.

After cutting, the pieces of hose were cleaned with the aforementioned cleaning method and then clamps were inserted mid-hose.

Hoses cut and clamped ready for install

The CPU block tops were fitted with the orifice tube filters and reinstalled to the LCS assembly. The LCS was lifted to a pedestal of suitable width (around 7 cm, made of a thick magazine) so that the CPU block tops rested on the somewhat soft pedestal. Then the hoses connecting to the CPU block tops were installed on the fittings. After that it is useful to put the CPU top clamps in position, ensuring that there will be enough space for the CPU block screws to pass through from the other side. The narrower side of the spring clamp should be above the screw hole to allow clearance for the screw.

Then the pump–T-fitting line was installed along with positioning of its clamps. After that, the long CPU A–radiator hose was attached. The CPU A–radiator hose needs to be protected from the metal support stand; the hose comes in contact with the stand and any movement will cause the metal to eat into the hose. The rest of the hoses were installed in the following order: CPU A–T-fitting, CPU B–T-fitting, CPU B–radiator, pump–radiator. Then the rest of the clamps were positioned. The clamp on the pump output needs to be position so that the heatsink on the CPU card has proper clearance.

"Duct Tape Finish"

After rebuilding the coolant lines, zip ties were inserted to the tightest bends for kink protection. Then a protective "coating" for the middle stand was made by cutting a hose longitudally and inserting it over the edge of the stand. Finally another zip tie was inserted on the pump–T-fitting line to protect it from the metal piece that connects to the "handle" and the CPU card heatsinks.

Zip tied, almost fully rebuilt LCS

Suitably sized steel springs would have provided better kink protection. Protection for the hoses against the metal stand and the other metal part could have been made more cleanly by using pieces of plastic pipe or sturdier PVC hose over the affected hose segments, or using some other implements.

Rebuilt LCS

Refill

Initial refill testing was made with a translucent hose connecting the pump to the radiator. This setting allowed assessing the amount of air in the system. With the refill method perfected, the translucent hose was later replaced with EPDM hose.

Initial refill setup with a clear hose

The LCS would appear to have a capacity of 220 ml (the pump's expansion chamber excluded). During refill, 250 ml should be a sufficient amount with some margin.

The system was "filled to the top," excluding the pump's expansion chamber. Pressurising, or overfilling, it does not seem reasonable because the coolant will be heated up, and expand, during normal use. It looks like the volumetric expansion for a 20 °C to 50 °C temperature change in the coolant could be something between 2-5 ml, surroundings not taken into account. The pump's expansion chamber will deal with thermal expansion, which is also why underfilling does not seem necessary.

Coolant

The radiator of the LCS is aluminium and the CPU block base plates appear to be a gold plated copper blend (Oxygen-free phosphorus-containing copper, CuOFP, according to a more knowledgeable person). There is some stainless steel in the pump. Other parts of the system that contact water are plastic or rubber. Mixed metals bring about galvanic corrosion which can be very damaging, so, the coolant needs to have corrosion inhibition built in.

The original coolant is light green "Delphi A2 Cooling Fluid" with a fairly strong odour. It seems to "sediment" rather heavily. The coolant is mainly composed of propylene glycol and distilled water. Contrary to what is claimed around various web forums, it is not very toxic. The previous Dual G5 systems had coolant that is a different story.

Delphi A2 fluid sedimentation

Clear version of Thermochill EC6 was selected as the coolant as it has the necessary corrosion inhibition, even proven to automotive standards, it is non-toxic, and it doesn't contain any dyes. Dyes can break up and smudge or clog a cooling system. Other suggestions for corrosion inhibiting cooling fluid (after already selecting the EC6) encountered were the Sierra coolant and Glysantin G48, both automotive coolants. Apparently, a mix of 1/3 of Sierra propylene glycol antifreeze and 2/3 of distilled water was used in similar systems.

Vacuum Method

First refill attempts were made using a vacuum method. A 29 inch Hg vacuum was pulled through the service port using an aspirator pump. Then the service port valve was closed using the service port coupler. The vacuum hose was disconnected from the coupler and replaced with a coolant bottle contraption. The service port valve was opened using the service port coupler causing the coolant to be sucked in.

Several tries were made, but an air bubble of roughly 1-2 cm diameter was always left in the system. Either the vacuum was not deep enough or the coolant contraption leaked.

Definite benefits of the vacuum method would be easy and thorough evacuation of the system as well as quick refill without the need to separately bleed air. The vacuum should vaporize water and any other substance with lower vaporization temperature than water. With non-leaking connections and a deep enough vacuum there should be no possibility of excess air in the system.

Later, more information on generalities of vacuum refilling were given by an advisor. Vacuum refilling would require special equipment (more special than just the vacuum pump, but the pump needs to reach a rather high vacuum as well) and controlled temperature of both the target system and the liquid (or refrigerant). It does not seem feasible to do at home.

Syringe Method

As the vacuum method did not produce the expected results, it was augmented by bleeding the remaining air out with a syringe attached to the service port coupler through a 50 cm piece of hose.

Syringe attached to the LCS for bleeding

About 20 ml of coolant was first pulled in the syringe and the syringe then connected to the hose connecting to the service port coupler. Using the syringe, air from the hose and the (still closed) service port was pumped to the top of the syringe. At this point, the liquid level in the syringe (or hose) was made note of.

The LCS was positioned so that the service port was the highest point in the system. The service port valve was opened. Then the syringe plunger was pushed slowly and relesed, several times. Air bubbles emerged from the LCS and found their way to the syringe. When no more air would come out, the pump was run for a few seconds. The the syringe was used to pump air out again. This was continued until the pump sound was even. Several orientations of the LCS were tested with the pump running.

When the pump sound was even in several different orientations, the service valve was closed. Before closing the valve, the liquid level in the syringe was checked too see that it was lower than before. Bleeding took around 5 to 10 minutes in total.

In retrospect, the syringe would work well also for draining the system and refilling it with coolant. It is also, by far, the cheapest and probably easiest implement to handle the refill process. The syringe method works because the pump has an expansion chamber which allows enough movement to take place for the air bubbles to be transferred to the service hose. Once in the hose, the air has good clearance for going up.

Reassembly

Thermal Grease

Based on the appearance of the original thermal grease as well as online reviews, Shin-Etsu X23-7783D was selected as the new thermal grease. Shin-Etsu G751 should be applicable as well.

Original thermal grease on the CPU block

Rebuilding the CPU Blocks

The CPU blocks were rebuilt by seating the red O-ring, the red foamy pad, and the white plastic maze in the plastic CPU block top of each CPU block. The red foamy pads were slightly deformed from the outlet side, so they were turned 180° compared to the original position. The copper CPU block bases were then reinstalled by gradually tightening the four mounting screws in a circular pattern. There might be a designed, "correct" way to tighten the blocks so that the O-rings would be least likely to go out of place, but that of course isn't known.

Cleaning the CPUs and CPU Blocks

Before putting everything back together the CPUs and the CPU block bases needed to be cleaned. Old thermal grease was removed by using cotton swabs, toothpicks, and coffee filters dipped in isopropyl alcohol. The CPU blocks were actually roughly cleaned after detaching the CPU cards.

Reapplying Thermal Grease

The Shin-Etsu grease is very thick and putty-like and cannot be "worked" much. A rice grain sized blob was applied in the middle of each CPU. The grease can be softened by heating it up, by submerging in hot water inside a plastic bag for example. A more painstaking application of spreading the grease was once made, but it was of no benefit.

Reapplied thermal grease

Reattaching the CPU Cards

Before reattaching the CPU cards, the LCS mounting screws (that required the ballhead driver) were reinstalled. Then the CPU cards were reinstalled by first inserting two black machine screws to opposite corners of the cards and then the four CPU mounting screws using a cross pattern and gradual tightening. Then the rest of the black machine screws were put in place.

Results

Before the repair, the fans sped up to max rpm over several hours of idling, but after the repair, the fans stay at minimum (970 rpm for the intake and 1000 rpm for the exhaust) in normal usage while the CPU temperatures stay under 40 °C (ambient around 23 °C). A lengthy rendering with all the cores maxed out raise the fan speeds to around 1600 - 1700 rpm and the CPU temperatures up to 65 °C.

Following are a few test results from the stress testing program run before and after the repair. The stress testing program runs all of the tests with all the fans and the pump full speed.

CPUs idling for two hours, before repair (ambient 23,6 °C)
CPUs idling for two hours, after repair (ambient 22,8 °C)

After repair, maximum temperatures over ten degrees down. Before repair, a noticeable upward trend in CPU B; insignificant upward trend after repair.

CPUs fully loaded, before repair
CPUs fully loaded, after repair

Before repair, CPUs overheating and GHz clamped by the kernel (the section in graph with zero temperatures.) After repair, maximum temperatures down by 40 degrees C.

Time will tell how well the hose segments where the minimum bend radius is exceeded hold up. The effectiveness of the corrosion inhibtion package of the Thermochill EC6 coolant also remains to be seen. A maintenance interval of 6 months is the initial plan.

References

Repairs

Thermal Calibration

Coolant

Pump

Materials

Photos

Other

Text and photos copyright © 2012 Heikki Lindholm
Contact: h o l i n ( ADD AT HERE ) i k i . f i
Visits