Member Avatar for Dani

I'm feeling a little dumb when it comes to the editor that we are using, and I was wondering if anyone knew of any good online tutorials for an introduction to tokenizers.

  • 3 Contributors
  • 3 Replies
  • 112 Views
  • 2 Days Discussion Span
  • Latest Post Latest Post by Airshow
Member Avatar for pritaeas

Not a tutorial, but I know PEAR has a tokenizer.

Member Avatar for Dani

Yeah, not what I need. I'm trying to make all these changes to the tokenizer used by the editor and it's confusing me because I have no experience with how a tokenizer works. :(

Member Avatar for Airshow

Dani,

I looked at the code behind the editor a couple of months ago and have to agree, it ain't the easiest to follow. I felt it needed to be taken apart and put back together in a more organised fashion.

If you want to take a step back from the actual code and remind yourself of the concepts and teminology in this area, then Wikipedia's entry on Lexical Analysis seems pretty good (with the usual provisos for Wikis). Some of the links may also prove fruitful.

Edited by Airshow
commented: Thanks +14
Be a part of the DaniWeb community

We're a friendly, industry-focused community of developers, IT pros, digital marketers, and technology enthusiasts meeting, networking, learning, and sharing knowledge.