https://en.wikipedia.org/wiki/Diffie-Hellman_key_exchange#Operation_with_more_than_two_parties
This seems like the right thing to me. You just have to make sure that the folks you're talking to are capable of maintaining the integrity of their private keys. If there is an actual threat one would imagine that you wouldn't want to speak to people incapable of this anyway.