First build in 15 years :-)
Objective: a data science / parallel computation / software dev machine, occasional game every now and again.
The CPUs are actually E5-2696 V4s, but have identical specs to the E5-2699 V4. Got them for a reasonable price from a vendor on Ebay. Admittedly a bit of a gamble but turned out fine. A GTX 1070 to get my feet wet with deep learning models and perhaps some gaming.
The Corsair 750D Airflow Edition doesn't officially support EEB form factor motherboards. I drilled two additional holes for motherboard standoffs. I went with this case because I like its minimalistic aesthetic and I didn't want a huge case on my desk. I replaced the standard case fans that came with the case with Bequit! Silent Wings 3 fans. There are two noctua fans in the top position. I originally planned to have the CPU coolers facing upwards with a push-pull fan configuration but this turned out to be practically impossible because of space limitations. I rotated them 90 degrees, and mounted the two leftover fans in the top instead.
The installation of Ubuntu 16.10 was a bit tricky for 2 reasons. First, the installer didn't recognize the Samsung 950 Pro nvme m.2 disk. It was possible to start up Ubuntu from the usb disk, but the necessary nvidia drivers were missing causing the mouse indicator not to move. I ended up solving the problem by acquiring another ssd, a Samsung 850 Pro, which connects to the motherboard with a trusty old-fashioned SATA cable. The installer recognized that one without problems. With some hassle navigating the installer using the keyboard, I was able to install the operating system, the nvidia drivers were installed using apt-get in a terminal.
I also ended up buying a small and cheap m.2 to PCIe 4x adapter card for the Samsung 950 pro, and installed it this way instead of using the slot on the motherboard for two reasons: the interface on the z10pe-d16 WS apparently is only 2x, so it doesn't make sense to install a fast drive in a slower slot. Secondly, the people at Puget systems recommend not installing a drive on the motherboard for thermal reasons because there is a high probability it will end up under the graphics card. According to them, an adapter card is recommended, and as a PC building newbie I simply complied with this advice ;-)
From a thermals perspective, having the hot air from one CPU cooler blow into the other one is a design flaw. The Noctuas are ~12cm wide, just a tad too large to have them blow upwards. To make sure thermal properties of the system are reasonable I ran
stress-ng -c 88 to keep all of the 88 threads busy. It turns out there is a big difference in CPU temperature under load. The first one ran comfortably around 55°C, whereas the second one hovered around 68°C. As long as it remains under 70° I do not worry too much. [UPDATE] it turns out the guys at Puget systems also use this fan configuration:
Another peculiar phenomenon is that the motherboard produces a soft but still annoying "chirping" noise when the CPUs are (almost) idle. Probably this is caused by some power/voltage management system. I figured out that by feeding one thread some busywork, the noise disappears. [UPDATE] I managed to get rid of the noise by disabling the power-efficiency setting in the BIOS IntelRCSetup section. Expectedly, when idling, the CPUs run averagely on a higher clock speed, and therefore a little hotter. I monitor each core's clock with
sudo watch -n 1 cat /sys/devices/system/cpu/cpu*/cpufreq/cpuinfo_cur_freq. The system is now inaudibly quiet when not under load, which is nice. When under load it emits a low-pitched hum, but still a lot quieter than my MBP with fans blazing.
As already pointed out in other builds with similar hardware, a downside of a system like this is the slow startup time, around a minute. On the other hand, as it's not a laptop it's probably going to stay on for long stretches of time anyway.
Overall, I'm extremely pleased with how it turned out. Ordering the Xeons on Ebay was a gamble, which fortunately turned out OK. The overall price I paid was lower than the estimate calculated by this website, I paid around 2300$ per CPU, customs included. So the total cost was around 2 beefy MBPs. IMHO very reasonable for a system like this. For an enthousiast build like this, it's worth keeping an eye on the second-hand market and strike when an opportunity presents itself.
Now all that remains is putting this rig to good use. In the short term it will be used to develop algorithms to visually explore high-dimensional data sets from the life sciences domain.