#scientific computing

10.09.2020

Today I offer you: [incoherent screaming].

Get to know Marc Schwalbach, computational scientist

1) What do you do?

I specialize in the field of computational sciences / scientific computing. This is essentially a combination of computer science, maths, and engineering that allows us to use computers to simulate physical phenomena, such as the flow of fluids around an airplane or weather forecasting. These computer simulations serve as a virtual lab with which we can analyze different designs and scenarios even before the first prototype is manufactured for physical experiments.

For example, in my PhD I combined both fluid and structural simulations to optimize the shape of a radial turbine to maximize its aerodynamic efficiency while ensuring its structural integrity.

2) Where do you work?

I currently work as a research engineer at CFD-Berlin in Germany, where most projects involve computational fluid dynamics and aeroacoustic optimization.

3) Tell us about the photos!

[Left:] My COVID-times PhD defense pose which was conducted via video call earlier this year.

[Right:] At a project meeting in Glasgow with our international partners in 2018

4) Tell us about your academic career path so far. 

After graduating from high school in the Philippines, I moved to Germany where I first attended a preparatory "Studienkolleg" that allows students with foreign high school degrees to apply for university. 

I received my B.Sc. and M.Sc. in Computational Engineering Science from RWTH Aachen University where I began specializing in computational  fluid dynamics and optimization. During this time, I got to spend a few months as a research intern at UBC in Canada with a DAAD RISE scholarship.

I then moved to the von Karman Institute in Belgium where I conducted my PhD research as a Marie Curie fellow and received my doctoral degree in Computer Science from the TU Kaiserslautern in Germany.

5) Anything else you’d like to share

In 2017, we announced that we were going to bring the design and manufacturing of our products in-house. The driving purpose was to leverage our understanding of our users’ needs in order to engineer better products for them. In 2018, we shipped the first fruits of our labor and, over the course of the last two years, shipped hundreds of updates to the Thelio line as we continuously integrated improvements into our manufacturing. Establishing our factory in Colorado made this possible, but we were just getting started.

Early this year, we set off to engineer our workstation version of a Le Mans Hypercar. It started with a challenge: Engineer a quad-GPU workstation that doesn’t thermal throttle any of the GPUs. Three GPUs is pretty easy. Stack the forth one in there and it’s a completely different animal. Months of work and thousands of engineering hours later we accomplished our goal. Every detail was scrutinized. Every part is of the highest quality. And new factory capabilities, like milling, enabled us to introduce unique solutions to design challenges. The result is Thelio Mega. A compact, high-performance quad-GPU system that’s quiet enough to sit on your desk.

We started with simple fan placement experiments to determine the best location for intake and exhaust fans and sizes. Computer fluid dynamics simulations assisted to dial in air flow ducts followed by hundreds of fan placement iterations and thermal tests.

*One of many CFD simulations.

We use gpuburn and stress-ng utilities to stress the components for each iteration until eventually finding the optimum fan position, size, speed, and duct design. Moving the side fans as little as 5mm up, down, left or right changes the thermal properties. Airflow shape has a considerable impact as well. The side intake panel has a duct on the back that directs air from the side and bottom fans into different areas of the GPUs. This helps limit inefficient turbulence and improves performance.

Inside the CPU duct, three fans of different, carefully chosen speeds pull and push air through the heat exchanger and exhaust it through the rear. The CPU duct fan positions were even more sensitive. 2mm of change in different directions improved or degraded performance.

Finally, we moved Thelio Mega into our acoustic testing booth to perfect the fan curves and make design tweaks for the quietest possible operation.

New in Thelio Mega are stabilizing feet and a PCI brace system, both milled from aluminum bar stock. The new PCI brace is particularly impressive. Nubs hold the GPUs steady for easier installation and a sliding brace locks them securely in place.

Our Thelio line is known for its natural and stained wood veneer, but we're particularly fond of the vent design flourishes that grace the system. If you have to make holes, you might as well make them interesting. The upper CPU vent is three celestial bodies representing the three-body problem in physics (and a fantastic novel by Liu Cixin). The GPU intake vent is embellished by rockets escaping the atmosphere. The rear CPU exhaust is the planetary alignment of the Solar System at the time of the Unix epoch.

Months of engineering was resulting in excellent performance and acoustic results. Often, quad GPU systems are terribly loud, unpleasant to work on, and throttle when under heavy GPU workloads. While working on Thelio Mega in quiet environments, it became clear that this was a new class of high-performance GPU compute workstation. But we wanted to be sure that we’re creating substantial value for the engineers and scientists that will get the most out of this system.

A common chassis used for quad-GPU setups is the Corsair Carbide Series™ Air 540 High Airflow ATX Cube Case. So we bought one. We ran thermal and acoustic tests on Thelio Mega, moved the components to the Corsair case and used the high-end Corsair H100i Pro 240mm liquid cooler in lieu of the custom Thelio Mega CPU cooling system.

Components Used Motherboard TRX40 AORUS XTREME AMD Ryzen Threadripper 3990X 64 GB 3200 MHz Kingston Memory 250 GB Samsung 970 Evo Plus NVMe drive 4 x NVIDIA GeForce RTX 2080ti (Gigabyte GV-N208TTURBO-11GC)

During our thermal engineering work, we found that 8 minutes is optimal for stressing the GPUs and iterating the design. If you get to 8 minutes without throttling, the system tends to be able to go much longer. We later expanded the tests to 16 hours. We used 10 minutes for CPU comparison. It wasn’t necessary to go longer. The Corsair case was unable to reach either 8 or 10 minutes without throttling.

GPU Cooling Test Results

Higher GPU fan percentage means higher load and louder operation. The NVIDIA RTX 2080Ti starts to throttle between 87 and 88 degrees C. Below 250 watts represents thermal throttling. The room temperature was 72 degrees Fahrenheit.

Thelio Mega GPU Burn - 8 Minutes

.tg {border-collapse:collapse;border-spacing:0;} .tg td{border-color:black;border-style:solid;border-width:1px;font-family:Arial, sans-serif;font-size:14px; overflow:hidden;padding:10px 5px;word-break:normal;} .tg th{border-color:black;border-style:solid;border-width:1px;font-family:Arial, sans-serif;font-size:14px; font-weight:normal;overflow:hidden;padding:10px 5px;word-break:normal;} .tg .tg-0pky{border-color:inherit;text-align:left;vertical-align:top}

GPU GPU Fan Usage GPU Temperature Watts 0 93% 83C 250 1 93% 83C 250 2 85% 78C 250 3 99% 86C 250

Corsair Case GPU Burn - 8 Minutes

.tg {border-collapse:collapse;border-spacing:0;} .tg td{border-color:black;border-style:solid;border-width:1px;font-family:Arial, sans-serif;font-size:14px; overflow:hidden;padding:10px 5px;word-break:normal;} .tg th{border-color:black;border-style:solid;border-width:1px;font-family:Arial, sans-serif;font-size:14px; font-weight:normal;overflow:hidden;padding:10px 5px;word-break:normal;} .tg .tg-0pky{border-color:inherit;text-align:left;vertical-align:top}

GPU GPU Fan Usage GPU Temperature Watts 0 100% 88C 220-230 1 100% 88C 210-230 2 87% 85C 250 3 100% 88C 170-180

CPU Cooling Test Results

Thelio Mega stress-ng -c 128 (stressing 128 threads for 10 minutes) CPU Temperature - 85C

Corsair Case w/ Corsair H100i Pro Liquid Cooler stress-ng -c 128 (stressing 128 threads for 10 minutes) CPU Temperature - throttling at 94.2C

So we’re doing great on GPU and CPU cooling. How about acoustics?

.tg {border-collapse:collapse;border-spacing:0;} .tg td{border-color:black;border-style:solid;border-width:1px;font-family:Arial, sans-serif;font-size:14px; overflow:hidden;padding:10px 5px;word-break:normal;} .tg th{border-color:black;border-style:solid;border-width:1px;font-family:Arial, sans-serif;font-size:14px; font-weight:normal;overflow:hidden;padding:10px 5px;word-break:normal;} .tg .tg-0pky{border-color:inherit;text-align:left;vertical-align:top}

Thelio Mega at Idle 31.8dB Corsair at Idle 45.7dB Thelio Mega CPU Stress 46.2dB Corsair CPU Stress 48.0dB Thelio Mega GPU Stress 53.5dB Corsair GPU Stress 62.3dB

For reference, 30-40 dB is a whisper to a quiet library. 50 is moderate rainfall. 60 is normal conversation and dishwashers. 70 is traffic and vacuums. Thelio Mega doesn’t get much louder than rainfall.

An additional benefit of developing an extreme-performance product, like Thelio Mega, is that the engineering that goes into this product can spread to other models in the Thelio line. Already, the engineering from Thelio Mega is being ported to Thelio Massive. And features from this work will move into Thelio Major as well. Engineering this kind of product pushes the envelope, and in doing so, improves the entire product line.

16.09.2020

Like come on, TensorFlow. Really? No Python 3.14 support? And yet to get working versions of other stuff I must use 3.14, etc.; it's just quite the clusterfuck. And don't even get me started on supported CUDA compute versions, the pain of actually getting those drivers working, or the fact that some software I need to use is stuck on the 2024 version of MATLAB or worse, on python 3.7 (which reached end-of-life in 2023...). It's really a mess. A mess you see shockingly less of in other ecosystems, even just looking at, like, analysis software that uses Java or the like -- most of that was written for Java 6 back in the day and can run on 25 now. Ahh well...