Lexical Analysis Phase

[nemoo]

hey ppl,

i'm tring to write a program that works as scanner(Compiler's Lexical analysis phase).I have a general imagine on how i'll make the program read code from file and seperate words so as to generate the appropriate token class for each word but i've a very essential missing
i've no idea on how the program can generate the token class for each word
is there is a built in method in C# that takes a word an returns its token class??
is there is a database that conatins all the keywords,operators,functions,.... of C#??

please help me:(
thx,

[ddanbe]

To my knowledge there is no database as you describe.
Perhaps this code snippet might help you on the way: http://www.daniweb.com/code/snippet217185.html, the other two snippets may also be found in the code snippet section. Together they form a working program. Succes.

[nemoo]

thx 4 ur reply but i think the complete scanner is much more complex than that's of a calculator.
In the later there is only a few and limited number of expected inputs to the scanner but with a full scanner i think i must, for example, provide it with all the names of methods that exists in each namespace belongs to this language.Must not I??

btw i read in some where that there is a method that i can use in the following namespace

NameSpace Microsoft.VisualStudio.Package

but i ddnt found the source code for this namespace
so,please if u have it do u mind giving it to me??

[ddanbe]

You can extend my scanner calculator code and add more functions an if-statement etc. if you wish to do so.
Code about TokenInfo can be found here:http://www.koders.com/csharp/fid42657FA9A675FC16FADC38587FC012C8C37FD509.aspx?s=TokenInfo

[nemoo]

but ofcourse I don't know all the functions that exist in all the namespaces of this language :S:S

and in regards to the Microsoft NameSpace have u any idea about where could I find it??

[ddanbe]

Do you think I know all the methods in all the namespaces of .NET?
.NET is HUGE.
Use Add Reference... to add an extra reference to your project.
My advice is : keep it as simple as possible.
Start with scanning and evaluating simple expressions like 3 + 4 * 5.
You will find hat it is not so simple after all. This should evaluate to 23, not 35! Succes!

[nemoo]

thx for GOD I supposed only to create a simulation to the Lexical Analysis Phase not a complete compiler :lol:

sry but regards to this code
http://www.koders.com/csharp/fid4265...px?s=TokenInfo
there is alot of objects from TokenInfo Class
when i tried to do so the compiler didn't found this class although i used all the namespaces used in this code

[nemoo]

thx i found it in the same site

sry 4 interrupting u and i think that i'll keep ur advice in my mind :)

[Geekitygeek]

thx for GOD I supposed only to create a simulation to the Lexical Analysis Phase not a complete compiler :lol:

haha, i had wandered what you were doing. I read it and thought "A Lexical Analysis Scanner...for the .net framework...that'll be HUGE!" lol. I had no idea where to start trying to tokenise input based on the vast possiblities of the C# langauge so i figured i'd let someone who knew better help you out :p
As ddanbe pointed out, just handling the simple tokenisation of mathematic formulas is more complicated than it would seem and most Lexical Analysis Scanners use very complicated algorythms. You can replicate it at the simpelest level using a subset of the possible keywords and namespaces, then use regular expressions to match tokens.

[ddanbe]

@Ryshad
Totally agree. It is a dream of many to write a new language. But one should always have dreams! No dreams, no life. I can very well live with my small console calculator I managed to put together, after reading and studying alot of stuff, experimenting with small code pieces etc. It was fun to do!