Attack on an HF model split into multiple GPUs

[RealPolitiX]

Does textattack allow running the attack calculation on a single model doing distributed inference across multiple GPUs?

For example, one can use the argument device_map="auto", which distributes a large model to multiple GPUs, on HuggingFaceModelWrapper. However, it seems that if you split a single instance of a large model onto multiple GPUs, then when doing the attack, such as using attacker.attack_dataset(), then there will be a RuntimeError similar to the following (if two GPUs are present)
RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cuda:1!

[RealPolitiX]

Solved this problem by commenting out the following two lines

if torch.cuda.is_available():
     self.attack.cuda_()

in the attacker.py file. It seems that cuda_() is enforcing a specific mapping of the neural network to GPU memory.

[qiyanjun]

@RealPolitiX your workaround pointed at the right code, but removing cuda_() entirely isn't safe as a default (plenty of setups need it). Opened #831 with a more targeted fix: Attack.cuda_() now skips any model that already has an hf_device_map attribute (the marker transformers/accelerate sets when a model was loaded with device_map=...), since forcing .to(device) on an already-sharded model is what conflicts with its placement. Single-GPU/no-device_map usage is unaffected. Will close automatically once that merges.