I think what you are asking is how to reverse engineer the protocol.
Yes, reverse engineering is usually difficult.
One approach is to make the program do known things and watch the data. Repeat the operation and see what changes and what doesn't. Then do something slightly different and see what changes. You will gradually figure out the structure of the data by comparing the data packets to your input. You can gradually narrow down the layout of the packets.