Efficient Access to Shared GPU Resources: Part 5

This is part 5 of a series of blog posts about GPU concurrency mechanisms. In part 1 we focused on the pros and cons of different solutions available on Kubernetes, in part 2 we dove into the setup and configuration details, in part 3 we analyzed the benchmarking use cases, and in part 4 we benchmarked the time slicing mechanism.

In this part 5 we will focus on benchmarking MIG (Multi-Instance GPU) performance of NVIDIA cards.

Setup

The benchmark setup will be the same for every use case:

• Full GPU (MIG disabled) vs MIG enabled (7g.40gb partition)
• MIG enabled, different partitions:
  • 7g.40gb
  • 3g.20gb
  • 2g.10gb
  • 1g.5gb

Keep in mind that the nvidia-smi command will not return the GPU utilization if MIG is enabled. This is the expected behaviour, as NVML (NVIDIA management library) does not support attribution of utilization metrics to MIG devices. For monitoring MIG capable GPUs it is recommended to rely on NVIDIA DCGM instead.

Find more about the drivers installation, MIG configuration, and environment setup.

Theoretical MIG performance penalty

When sharing a GPU between multiple processes using time slicing, there is a performance loss caused by the context switching. As a result, when enabling time slicing but scheduling a single process on the GPU, the penalty can be neglected.

On the other hand, with MIG, the assumptions are very different. By just enabling MIG, a part of the Streaming Multiprocessors are lost. But there is no additional penalty introduced when further partitioning the GPU.

For instance, as can be seen in the image below, a whole A100 40GB NVIDIA GPU (the GPU used for the benchmarking that follows) has 108 Streaming Multiprocessors (SMs). When enabling MIG, 10 SMs are lost, which is the equivalent of 9.25% of the total number of compute cores.

As a result, when enabling MIG, it is expected to see a performance penalty of ~9.25%. As partitions are considered isolated, we shouldn’t have additional overhead when sharing the GPU between many users. We expect the scaling between partitions to be linear, meaning a 2g.10gb partition should perform 2 times better than a 1g.5gb because it has double the resources.

FLOPS Counting

Floating Point Operations Per Second (FLOPS) is the metric used to show how powerful a GPU is when working with different data formats. To count FLOPS we rely on dcgmproftester, a CUDA-based test load generator from NVIDIA. For more information, consult the previous blog post.

Full GPU (MIG disabled) vs MIG enabled (7g.40gb partition)

FLOPS counting per MIG partition

FLOPS scaling between MIG partitions

Memory bandwidth scaling between MIG partitions

Conclusions

• Enabling MIG:
  • For fp32 and fp64, Cuda Cores, the drop in performance is close to the theoretical one.
  • For fp16 on Cuda Cores and Tensor Cores, the loss is much smaller than the expected value. Might need further investigation.
  • There is no loss of memory bandwidth.
• The scaling between partitions:
  • On CUDA Cores (fp16, fp32, fp64) the scaling is converging to the theoretical value.
  • On Tensor Cores the scaling diverges a lot from the expected one (especially when comparing 7g.40gb and 3g.20gb). Might need further investigation.
  • The scaling of the memory bandwidth is based on powers of 2 (1, 2, 4, and 8 accordingly).

Compute-Intensive Particle Simulation

An important part of CERN computing is dedicated to simulation. These are compute-intensive operations that can significantly benefit from GPU usage. For this benchmarking, we rely on the lhc simpletrack simulation. For more information, consult the previous blog post.

Full GPU (MIG disabled) vs MIG enabled (7g.40gb partition)

Running simulation on different MIG partitions

Scaling between MIG partitions

Conclusions

• The performance loss when enabling MIG is very close to the theoretical 9.25%.
• The scaling between partitions converges to the ideal values.
• The results are very close to the theoretical assumptions because the benchmarked script is very compute-intensive, without many memory accesses, being CPU bound, etc.

Machine Learning Training

For benchmarking, we will use a pre-trained model and fine-tune it with PyTorch. To maximize GPU utilization, make sure the script is not CPU-bound, by increasing the number of data loader workers and batch size. More details can be found in the previous blog post.

Full GPU (MIG disabled) vs MIG enabled (7g.40gb partition)

• dataloader_num_workers=8
• per_device_train_batch_size=48
• per_device_eval_batch_size=48

7g.40gb vs 3g.20gb

• per_device_train_batch_size=24
• per_device_eval_batch_size=24
• dataloader_num_workers=4

When comparing a 7g.40gb instance vs a 3g.20gb one, the amount of cores becomes 2.33 (7/3) times smaller. This is the scale we are expecting to see experimentally as well, but the results are converging to 1.8 rather than 2.3. For machine learning training, the results are influenced a lot by the available memory, bandwidth, how the data is stored, how efficient the data loader is, etc.

To simplify the benchmarking, we will use the 4g.20gb partition instead of the 3g.20gb. This way all the resources (bandwidth, cuda cores, tensor cores, memory) are double when compared to 2g.10gb, and the ideal scaling factor is 2.

4g.20gb vs 2g.10gb:

• per_device_train_batch_size=12
• per_device_eval_batch_size=12
• dataloader_num_workers=4

2g.10gb vs 1g.5gb:

• per_device_train_batch_size=4
• per_device_eval_batch_size=4
• dataloader_num_workers=2

Conclusions

• The performance loss when enabling MIG is much smaller than the theoretical 9.25%. This can be caused by many reasons:
  • complex operations to be performed
  • being IO bound
  • executing at a different clock frequency
  • variable tensor core utilization, etc.
• The training time depends heavily on the number of CUDA Cores, Tensor Cores, but also the memory bandwidth, the data loader, the batch size, etc. Consult the previous blog for some ideas on how to profile and detect performance bottlenecks of models.
• The scaling between partitions is linear and is converging to the expected value. Especially on smaller partitions.

Takeaways

• When using MIG technology, the GPU partitions are isolated and can run securely without influencing each other.
• A part of the available streaming multiprocessors are lost when enabling MIG:
  • In the case of an A100 40GB NVIDIA GPU, 10/108 SMs, which means losing 9.25% of the available Cuda and Tensor cores.
  • Never enable MIG without actually partitioning the GPU (the 7g.40gb partition). It means losing performance without gaining GPU sharing.
• When enabling MIG:
  • The performance loss for compute-intensive applications can reach ~9.25%.
  • For machine learning training the loss alleviates as a result of memory accessing, being IO bound, being CPU bound, etc, and experimentally it will be much smaller than expected.
• The scaling between partitions is linear. Doubling the resources halves the execution time.
• For GPU monitoring rely on NVIDIA Data Center GPU Manager (DCGM). It is a suite of tools that includes active health monitoring, diagnostics, power and clock management, etc.
• A current limitation (with CUDA 11/R450 and CUDA 12/R525) that might be relaxed in the future is that regardless of how many MIG devices are created (or made available to a container), a single CUDA process can only enumerate a single MIG device. The implications are:
  • We cannot use 2 MIG instances in a single process.
  • If at least one device is in MIG mode, CUDA will not see the non-MIG GPUs.

Next episode

In the next blog post, we will use NVIDIA A100 GPUs and MIG to train in distributed mode a high energy physics (HEP) neural network on-prem Kubeflow. Stay tuned!

• ←Previous
Next→