The negotiations between the server and client will be similar to the diagram given below,
ProtocolVersion:
Server sends the first message when a client connects to it. It sends the highest protocol version which it supports. Client responds to server in a similar way with the protocol version which it could work with.
Security:
Security type for the conversation. Server lists the security types to client and the client responds to it.
Security result:
After the negotiations for security is done, server responds to the client with a single word ( OK / failed ) to let the client know whether the security handshaking is successful.
ClientInit:
Client sends a value either zero to ask for an exclusive connection to the server or non-zero to ask the server share the desktop other clients too.
ServerInit:
The server sends back various data like frame buffer's height, width, pixel format and desktop name.
After all these negotiations, the client will start to send the FramebufferUpdateRequest to the server and server responds with the FramebufferUpdate. Client will also send other messages to server like KeyEvent, PointerEvent, and more.,
You could find a more detailed explanations about the protocol messages in wiki.
Further reading:
FBS files - To know how RFB data is stored in FBS files and what does the FBS files contain.
