In any case, simply giving someone your phone number for whatever reason should not entitle them to annoy you on WhatsApp.
Thankfully, WhatsApp has already considered this and has made it simple to prevent strangers from adding you to their chat groups. All you have to do is change a few settings in the privacy area of the app. This is how you get there:
In WhatsApp, tap the vertical ellipses in the top right corner, then Settings.
Select Privacy after going to Account (which should be the first option).
Scroll down to the Groups subhead, where you'll find a list of alternatives based on your privacy preferences to choose from.
Select My contacts from that list to allow only people you know to add you to WhatsApp groups. This is the way to go if you frequently give out your phone number to folks who aren't on your contact list. However, you may still have one or two obnoxious friends who can't stop themselves from forming weird gatherings and including you in them. There are a My contacts except... option for people like these, which does exactly what it says on the box.
There's a chance you don't like either of these alternatives and would prefer that people outside your contact list add you to their groups, perhaps for your company phone number or something else. In that situation, you can always use the Everyone setting option to eliminate all limits.
None of these options is foolproof, as group administrators can still send you private invitations and urge you to join. However, it should be sufficient to allow you to choose which organizations you wish to join and to keep you safe from unneeded danger.