dcgmi diag -r 4
Successfully ran diagnostic for group.
+---------------------------+------------------------------------------------+
| Diagnostic                | Result                                         |
+===========================+================================================+
|-----  Metadata  ----------+------------------------------------------------|
| DCGM Version              | 3.3.7                                          |
| Driver Version Detected   | 535.129.03                                     |
| GPU Device IDs Detected   | 2330,2330,2330,2330,2330,2330,2330,2330        |
|-----  Deployment  --------+------------------------------------------------|
| Denylist                  | Pass                                           |
| NVML Library              | Pass                                           |
| CUDA Main Library         | Pass                                           |
| Permissions and OS Blocks | Pass                                           |
| Persistence Mode          | Pass                                           |
| Environment Variables     | Pass                                           |
| Page Retirement/Row Remap | Pass                                           |
| Graphics Processes        | Pass                                           |
| Inforom                   | Pass                                           |
+-----  Integration  -------+------------------------------------------------+
| PCIe                      | Pass - GPUs: 0, 1, 2, 4, 5, 6, 7               |
|                           | Fail - GPU: 3                                  |
| Warning                   | GPU 3 Temperature 88 of GPU 3 exceeded user-s  |
|                           | pecified maximum allowed temperature 87 Verif  |
|                           | y that the user-specified temperature maximum  |
|                           |  is set correctly. If it is, check the coolin  |
|                           | g for this GPU and node: Verify that the cool  |
|                           | ing on this machine is functional, including   |
|                           | external, thermal material interface, fans, a  |
|                           | nd any other components.                       |
| Warning                   | GPU 3 Clocks are being throttled for GPU 3 be  |
|                           | cause of clock throttling starting 410.2 seco  |
|                           | nds into the test. clocks_throttle_reason_sw_  |
|                           | thermal_slowdown: the GPU or its memory have   |
|                           | reached unsafe temperatures. Check DCGM and s  |
|                           | ystem logs for errors. Reset GPU. Restart DCG  |
|                           | M. Rerun diagnostics.                          |
| Info                      | GPU 0 GPU to Host bandwidth:  55.10 GB/s, GPU  |
|                           |  0 Host to GPU bandwidth:  54.69 GB/s, GPU 0   |
|                           | bidirectional bandwidth: 100.47 GB/s, GPU 0 G  |
|                           | PU to Host latency:  1.961 us, GPU 0 Host to   |
|                           | GPU latency:  2.204 us, GPU 0 bidirectional l  |
|                           | atency:  3.376 us                              |
| Info                      | GPU 1 GPU to Host bandwidth:  55.08 GB/s, GPU  |
|                           |  1 Host to GPU bandwidth:  54.74 GB/s, GPU 1   |
|                           | bidirectional bandwidth: 100.51 GB/s, GPU 1 G  |
|                           | PU to Host latency:  1.930 us, GPU 1 Host to   |
|                           | GPU latency:  2.173 us, GPU 1 bidirectional l  |
|                           | atency:  3.366 us                              |
| Info                      | GPU 2 GPU to Host bandwidth:  55.11 GB/s, GPU  |
|                           |  2 Host to GPU bandwidth:  54.71 GB/s, GPU 2   |
|                           | bidirectional bandwidth: 100.50 GB/s, GPU 2 G  |
|                           | PU to Host latency:  1.933 us, GPU 2 Host to   |
|                           | GPU latency:  2.327 us, GPU 2 bidirectional l  |
|                           | atency:  3.378 us                              |
| Info                      | GPU 3 GPU to Host bandwidth:  55.09 GB/s, GPU  |
|                           |  3 Host to GPU bandwidth:  54.68 GB/s, GPU 3   |
|                           | bidirectional bandwidth: 100.48 GB/s, GPU 3 G  |
|                           | PU to Host latency:  1.922 us, GPU 3 Host to   |
|                           | GPU latency:  2.167 us, GPU 3 bidirectional l  |
|                           | atency:  3.372 us                              |
| Info                      | GPU 4 GPU to Host bandwidth:  55.10 GB/s, GPU  |
|                           |  4 Host to GPU bandwidth:  54.71 GB/s, GPU 4   |
|                           | bidirectional bandwidth: 100.53 GB/s, GPU 4 G  |
|                           | PU to Host latency:  1.904 us, GPU 4 Host to   |
|                           | GPU latency:  2.203 us, GPU 4 bidirectional l  |
|                           | atency:  3.044 us                              |
| Info                      | GPU 5 GPU to Host bandwidth:  55.10 GB/s, GPU  |
|                           |  5 Host to GPU bandwidth:  54.69 GB/s, GPU 5   |
|                           | bidirectional bandwidth: 100.49 GB/s, GPU 5 G  |
|                           | PU to Host latency:  1.890 us, GPU 5 Host to   |
|                           | GPU latency:  2.183 us, GPU 5 bidirectional l  |
|                           | atency:  3.056 us                              |
| Info                      | GPU 6 GPU to Host bandwidth:  55.11 GB/s, GPU  |
|                           |  6 Host to GPU bandwidth:  54.75 GB/s, GPU 6   |
|                           | bidirectional bandwidth: 100.51 GB/s, GPU 6 G  |
|                           | PU to Host latency:  1.920 us, GPU 6 Host to   |
|                           | GPU latency:  2.192 us, GPU 6 bidirectional l  |
|                           | atency:  3.034 us                              |
| Info                      | GPU 7 GPU to Host bandwidth:  55.10 GB/s, GPU  |
|                           |  7 Host to GPU bandwidth:  54.75 GB/s, GPU 7   |
|                           | bidirectional bandwidth: 100.07 GB/s, GPU 7 G  |
|                           | PU to Host latency:  1.885 us, GPU 7 Host to   |
|                           | GPU latency:  2.169 us, GPU 7 bidirectional l  |
|                           | atency:  3.107 us                              |
+-----  Hardware  ----------+------------------------------------------------+
| GPU Memory                | Pass - All                                     |
| Diagnostic                | Pass - All                                     |
| Pulse Test                | Pass - All                                     |
+-----  Stress  ------------+------------------------------------------------+
| Targeted Stress           | Pass - All                                     |
| Targeted Power            | Pass - All                                     |
| Memory Bandwidth          | Pass - All                                     |
| Memtest                   | Pass - All                                     |
| EUD Test                  | Skip - All                                     |
+---------------------------+------------------------------------------------+