How to prevent PyTorch Distributed Data Parallel (DDP) from freezing

I’ve recently faced an issue with Pytorch Distributed Data Parallel, in which the process hangs without any error message. I don’t exactly know what causes this, but I believe it is triggered by NCCL in the distributed setting.

March 8, 2022 less than 1 minute read

The following line solves the problem:

export NCCL_P2P_DISABLE=1

References

Distributed data parallel freezes without error message

Share on

Twitter Facebook LinkedIn

Jump multiple remote hosts using ProxyCommand (SSH Tunneling)

June 12, 2022 2 minute read

This article introduces how to ssh-jump on a remote intermediate server(s) to ssh-connect into a target server with a single command.

How to permanently add passphrase of private key to ssh-agent

May 10, 2022 3 minute read

Even if we’ve set a passwordless ssh login environment using private and public keys, we still need to type in the passphrase for the private key when loggin...

How to configure SSH without Passwords

May 9, 2022 4 minute read

This article introduces a secure private/public key-based SSH connection method to log into remote servers from a local machine. Here, we can think of the pu...

How to install/update/run npm, gem, and bundler for Jekyll-based minimal mistake management in macOS

April 11, 2022 1 minute read

As a minimal mistake Jekyll theme user, I use the following commands to set up and use npm, gem, and bundler for macOS.