If you are not ready to receive the serve, and make no attempt to return the serve, then it is a Let. (BWF Laws 9.4, 14.2, 14.2.1)
The only thing in the rules to say you were or were not ready is whether you attempt to return the serve.
So, if you stand in position to receive the serve, whether or not your head is just raised from being down, whether or not your hand is just lowered, and the fast serve takes place, if you make no attempt to return the serve because you were not ready, then it is deemed a LET.
Personally, my response to receiving such a quick serve would be to stay bolt still, and say ‘let – I wasn’t ready’. A few cycles of Lets and they should get the hint.
At a mixed ability club level I would not bother with raising a hand to ask the server to wait. The server simply cannot argue against your stated readiness if you make no attempt to return the serve (i.e. stay bolt still). However, any movement reaction from you in response to the fast serve (twitching or faltering) could be interpreted as an attempt to return the serve and the serve would be legitimate.
If your opponent is petty enough to require a physical display of readiness, I have heard the phrase “I am only ready when I look you in the eye” on court before! You can take your time then with foot placement, grip etc, and even looking at the opponent’s positioning, then un-nerve them with the death stare!
Just be aware that if you start spouting rules at your opponent, and start to take too long in your receiving preparations, they may too spout off – a player shall not: deliberately cause delay in, or suspension of, play (16.6)!
Relevant BWF Laws of Badminton:
14.2 It shall be a “let”, if:
14.2.1 the server serves before the receiver is ready (Law 9.4);
9.4 The server shall not serve before the receiver is ready. However, the receiver shall be considered to have been ready if a return of the service is attempted.