Efficient Access to Shared GPU Resources: Part 4

This is part 4 of a series of blog posts about GPU concurrency mechanisms. In part 1 we focused on the pros and cons of different solutions available on Kubernetes, in part 2 we dove into the setup and configuration details, and in part 3 we analyzed the benchmarking use cases.

In this part 4 we will focus on benchmarking results for time slicing of NVIDIA cards.

Setup

The benchmark setup will be the same for every use case:

• Time slicing disabled (GPU Passthrough)
• Time slicing enabled (the number denotes how many processes are scheduled on the same GPU):
  • Shared x1
  • Shared x2
  • Shared x4
  • Shared x8

Benchmarking time slicing can be complicated, because the processes need to start at the exact same moment. This means using a Deployment or a ReplicaSet will not work, because the pods are launched in a best effort manner, with some pods starting earlier than others.

The GPU alternates the execution in a round robin fashion. To benchmark, we start longer-running GPU processes in advance, to eliminate the need of start up synchronization. For example, to benchmark a script in a “Shared x4” GPU setup, we can:

• start 3 pods running the same script, but for a longer period of time.
• in the meantime, start the fourth pod. Make sure it starts and finishes while sharing the GPU with other 3.

Find more about the drivers installation, time slicing configuration, and environment setup.

FLOPS Counting

Floating Point Operations Per Second (FLOPS) is the metric used to show how powerful a GPU is when working with different data formats. To count FLOPS we rely on dcgmproftester, a CUDA based test load generator from NVIDIA. For more information, consult the previous blogpost.

fp16

fp32

fp64

fp16 Tensor Cores

Conclusions

• If time slicing is enabled, but only one process is using the GPU (shared x1), the time slicing penalty is negligible (<0.5%).
• When the GPU needs to do context switching (shared x2), there is a ~6% performance loss for fp32 and fp64, ~3% for fp16 and ~1.37% for fp16 on tensor cores.
• We don’t introduce an additional penalty when increasing the amount of processes sharing the GPU. The loss is the same for shared x2, shared x4, and shared x8.
• It is not yet understood why the performance loss is different depending on the data formats.

Compute-Intensive Particle Simulation

An important part of CERN computing is dedicated to simulation. These are compute-intensive operations that can significantly benefit from GPU usage. For this benchmarking we rely on the lhc simpletrack simulation. For more information, consult the previous blogpost.

Passthrough vs Shared x1

Shared x1 vs Shared x2

Shared x2 vs Shared x4

Shared x4 vs Shared x8

Conclusions

• The performance loss when enabling time slicing (shared x1) is very small (<2.5%).
• If the GPU needs to perform context switching (going from shared x1 to shared x2) the execution time triples. This means there is a performance loss of ~38%.
• Further increasing the number of processes (shared x4, shared x8), doesn’t introduce additional performance loss.

Machine Learning Training

For benchmarking we will use a pretrained model and fine-tune it with PyTorch. To maximize GPU utilization, make sure the script is not CPU-bound, by increasing the number of data loader workers and batch size. More details can be found in the previous blogpost.

Passthrough vs Shared x1

TrainingArguments:

• per_device_train_batch_size=48
• per_device_eval_batch_size=48
• dataloader_num_workers=8

Shared x1 vs Shared x2

TrainingArguments:

• per_device_train_batch_size=24
• per_device_eval_batch_size=24
• dataloader_num_workers=4

Shared x2 vs Shared x4

TrainingArguments:

• per_device_train_batch_size=12
• per_device_eval_batch_size=12
• dataloader_num_workers=2

Shared x4 vs Shared x8

TrainingArguments:

• per_device_train_batch_size=4
• per_device_eval_batch_size=4
• dataloader_num_workers=1

Conclusions

• The loss when performing ML training on a GPU with time slicing enabled is negligible (<0.7%).
• When scaling from shared x1 to shared x2 we get a ~2.2 times increase in overall computation (vs ideal 2). The time slicing loss is around 11% in this case.
• If the number of processes increases (shared x4, shared x8), the performance is not influenced a lot (0-3%).

Takeaways

• Considering the potential GPU utilization improvements, the penalty introduced by enabling time slicing but having only one process using the GPU (Shared x1) can be disregarded.
• There is a variable penalty introduced when the GPU needs to perform context switching (shared x1 vs shared x2).
• For more than 2 processes sharing the GPU, the execution time scales linearly (no extra penalty if we increase the number of processes sharing a GPU).
• The penalty introduced by time slicing can potentially be very big, depending on the use case:
  • When the GPU was running context switch sensitive workloads the penalty introduced by time slicing was about 38%.
  • In case the program is IO bound, CPU bound, heavy on tensor core utilization, etc, the penalty alleviates. For instance, with the ML traning the penalty dropped to ~11%.
• If the processes consume more memory than available, some of them will die because of OOM error. We cannot set limits or priorities for memory, we discussed earlier how we try to go around it (see more).

Time slicing can potentially introduce a big performance penalty. Even so, when applied to the correct use cases, it can be a powerful way of boosting GPU utilization. Consult the available overview to find more.

Next episode

In the next blog post, we will dive into the extensive MIG benchmarking. Stay tuned!

• ←Previous
Next→