Member Avatar for aeinstein

Hi All,

Just starting my online C programming class and my textbook states "In a computer language, a token is the smallest unit of the language that has a unique meaning. Thus, the reserved words, programmer-created identifiers, and all special mathematical symbols, such as + and -, are considered tokens of the C language."

Since the text previously states that the three type of C identifiers consist of keywords/reserved words, standard identifiers and programmer-created identifiers, then I'm presuming that both keywords and standard identifiers are also tokens, correct? And that punctuation marks ("()", "{}", ";" etc.), even though they represent attributes of the language, that they are considered to be part of the grammatical structure and are outside the definition of tokens, per se?

Regards,
Paul

Edited by aeinstein because: corrected misquote
  • 3 Contributors
  • 5 Replies
  • 187 Views
  • 1 Day Discussion Span
  • Latest Post Latest Post by aeinstein
Member Avatar for nezachem

> are outside the definition of tokens
No. The Standard defines token as either of:
keyword
identifier
constant
string-literal
punctuator
and punctuator in turn is either of
[ ] ( ) { } . ->
++ -- & * + - ~ !
/ % << >> < > <= >= == != ^ | && ||
? : ; ...
= *= /= %= += -= <<= >>= &= ^= |=
, # ##
<: :> <% %> %: %:%:

Member Avatar for aeinstein

TYVM for your reply nezachem. So the text is wrong in regards to mathematical operators being considered as tokens?

Paul

Member Avatar for nezachem

Umm... they are tokens.

Member Avatar for Adak

All operators (and thus all mathematics operators), are tokens. Your compiler help should have a list of them.

Member Avatar for aeinstein

Thanks for your replies nezachem and Adak, very helpful and very much appreciated.

Paul

PS: My apologies nezachem, I totally missed the quoted part of your reply, which is why I asked for clarification.

Edited by aeinstein because: n/a
Be a part of the DaniWeb community

We're a friendly, industry-focused community of developers, IT pros, digital marketers, and technology enthusiasts meeting, networking, learning, and sharing knowledge.