How can I tell my programs to choose the available device automatically
It is very simple to make your program automatically choose the right device: DO NOT use cudaSetDevice().
When you don't explicitly specify to use which device, your CUDA program will first try to set up context on device0. If device0 is not available, it will try device1.
What will happen if I choose a device which is being used by another program.
Since all GPU devices have been configured as compute-exclusive mode, the second thread which tries to contact with GPU device will be denied. If you insist using this device by using cudaSetDevice() in your code, you will receive an error when running a CUDA API function.