Member Avatar for katerinaaa

Hi,
I would like to ask if anyone know how can I create a tokenizer for a txt file in C++.
I find it difficult because there are not only words but there are also numbers and <p id> tags.

I have attached the file that is needed to be tokenized.

Could anyone help me ?

Thanks a lot

  • 4 Contributors
  • 4 Replies
  • 179 Views
  • 16 Hours Discussion Span
  • Latest Post Latest Post by katerinaaa
Member Avatar for Narue

The file wasn't attached.

Member Avatar for twomers

Maybe that's the point ...

Member Avatar for katerinaaa

The file is something like that :

<P ID=1>
CONTENTS
</P>
<P ID=2>
VOLUME I
</P>
<P ID=3>
BOOK FIRST.--A JUST MAN
</P>
<P ID=4>
CHAPTER
I. M. Myriel
II. M. Myriel becomes M. Welcome
III. A Hard Bishopric for a Good Bishop
IV. Works corresponding to Words
V. Monseigneur Bienvenu made his Cassocks last too long
VI. Who guarded his House for him
VII. Cravatte
VIII. Philosophy after Drinking
IX. The Brother as depicted by the Sister
X. The Bishop in the Presence of an Unknown Light
XI. A Restriction
XII. The Solitude of Monseigneur Welcome
XIII. What he believed
XIV. What he thought
</P>
<P ID=5>
BOOK SECOND.--THE FALL
</P>
<P ID=6>
I. The Evening of a Day of Walking
II. Prudence counselled to Wisdom
III. The Heroism of Passive Obedience
IV. Details concerning the Cheese-Dairies of Pontarlier
V. Tranquillity
VI. Jean Valjean
VII. The Interior of Despair
VIII. Billows and Shadows
IX. New Troubles
X. The Man aroused
XI. What he does
XII. The Bishop works
XIII. Little Gervais
</P>

<P ID=1>
CONTENTS
</P>
<P ID=2>
VOLUME I
</P>
<P ID=3>
BOOK FIRST.--A JUST MAN
</P>
<P ID=4>
CHAPTER
    I.  M. Myriel
   II.  M. Myriel becomes M. Welcome
  III.  A Hard Bishopric for a Good Bishop
   IV.  Works corresponding to Words
    V.  Monseigneur Bienvenu made his Cassocks last too long
   VI.  Who guarded his House for him
  VII.  Cravatte
 VIII.  Philosophy after Drinking
   IX.  The Brother as depicted by the Sister
    X.  The Bishop in the Presence of an Unknown Light
   XI.  A Restriction
  XII.  The Solitude of Monseigneur Welcome
 XIII.  What he believed
  XIV.  What he thought
</P>
<P ID=5>
BOOK SECOND.--THE FALL
</P>
<P ID=6>
    I.  The Evening of a Day of Walking
   II.  Prudence counselled to Wisdom
  III.  The Heroism of Passive Obedience
   IV.  Details concerning the Cheese-Dairies of Pontarlier
    V.  Tranquillity
   VI.  Jean Valjean
  VII.  The Interior of Despair
 VIII.  Billows and Shadows
   IX.  New Troubles
    X.  The Man aroused
   XI.  What he does
  XII.  The Bishop works
 XIII.  Little Gervais
</P>
<P ID=7>
BOOK THIRD.--IN THE YEAR 1817
</P>
<P ID=8>
    I.  The Year 1817
   II.  A Double Quartette
  III.  Four and Four
   IV.  Tholomyes is so Merry that he sings a Spanish Ditty
    V.  At Bombardas
   VI.  A Chapter in which they adore Each Other
  VII.  The Wisdom of Tholomyes
 VIII.  The Death of a Horse
   IX.  A Merry End to Mirth
</P>
<P ID=9>
BOOK FOURTH.--TO CONFIDE IS SOMETIMES TO DELIVER INTO A PERSON'S POWER
</P>
<P ID=10>
    I.  One Mother meets Another Mother
   II.  First Sketch of Two Unprepossessing Figures
  III.  The Lark
</P>
<P ID=11>
BOOK FIFTH.-- THE DESCENT
</P>
<P ID=12>
    I.  The History of a Progress in Black Glass Trinkets
   II.  Madeleine
  III.  Sums deposited with Laffitte
   IV.  M. Madeleine in Mourning
    V.  Vague Flashes on the Horizon
   VI.  Father Fauchelevent
  VII.  Fauchelevent becomes a Gardener in Paris
 VIII.  Madame Victurnien expends Thirty Francs on Morality
   IX.  Madame Victurnien's Success
    X.  Result of the Success
   XI.  Christus nos Liberavit
  XII.  M. Bamatabois's Inactivity
 XIII.  The Solution of Some Questions connected with the
            Municipal Police
</P>
<P ID=13>
BOOK SIXTH.--JAVERT
</P>
<P ID=14>
    I.  The Beginning of Repose
   II.  How Jean may become Champ
</P>
<P ID=15>
BOOK SEVENTH.--THE CHAMPMATHIEU AFFAIR
</P>
<P ID=16>
    I.  Sister Simplice
   II.  The Perspicacity of Master Scaufflaire
  III.  A Tempest in a Skull
   IV.  Forms assumed by Suffering during Sleep
    V.  Hindrances
   VI.  Sister Simplice put to the Proof
  VII.  The Traveller on his Arrival takes Precautions
            for Departure
 VIII.  An Entrance by Favor
   IX.  A Place where Convictions are in Process of Formation
    X.  The System of Denials
   XI.  Champmathieu more and more Astonished
</P>
<P ID=17>
BOOK EIGHTH.--A COUNTER-BLOW
</P>
<P ID=18>
    I.  In what Mirror M. Madeleine contemplates his Hair
   II.  Fantine Happy
  III.  Javert Satisfied
   IV.  Authority reasserts its Rights
    V.  A Suitable Tomb
</P>
<P ID=19>
VOLUME II
</P>
<P ID=20>
BOOK FIRST.--WATERLOO
</P>
<P ID=21>
CHAPTER
    I.  What is met with on the Way from Nivelles
   II.  Hougomont
  III.  The Eighteenth of June, 1815
   IV.  A
    V.  The Quid Obscurum of Battles
   VI.  Four o'clock in the Afternoon
  VII.  Napoleon in a Good Humor
 VIII.  The Emperor puts a Question to the Guide Lacoste
   IX.  The Unexpected
    X.  The Plateau of Mont-Saint-Jean
   XI.  A Bad Guide to Napoleon; a Good Guide to Bulow
  XII.  The Guard
 XIII.  The Catastrophe
  XIV.  The Last Square
   XV.  Cambronne
  XVI.  Quot Libras in Duce?
 XVII.  Is Waterloo to be considered Good?
XVIII.  A Recrudescence of Divine Right
  XIX.  The Battle-Field at Night
</P>
<P ID=22>
BOOK SECOND.--THE SHIP ORION
</P>
<P ID=23>
    I.  Number 24,601 becomes Number 9,430
   II.  In which the reader will peruse Two Verses which are
          of the Devil's Composition possibly
  III.  The Ankle-Chain must have undergone a Certain Preparatory
        Manipulation to be thus broken with a Blow from a Hammer
</P>
<P ID=24>
BOOK THIRD.--ACCOMPLISHMENT OF THE PROMISE MADE TO THE DEAD WOMAN
</P>
<P ID=25>
    I.  The Water Question at Montfermeil
   II.  Two Complete Portraits
  III.  Men must have Wine, and Horses must have Water
   IV.  Entrance on the Scene of a Doll
    V.  The Little One All Alone
   VI.  Which possibly proves Boulatruelle's Intelligence
  VII.  Cosette Side by Side with the Stranger in the Dark
 VIII.  The Unpleasantness of receiving into One's House a Poor
          Man who may be a Rich Man
   IX.  Thenardier at his Manoeuvres
    X.  He who seeks to better himself may render his Situation Worse
   XI.  Number 9,430 reappears, and Cosette wins it in the Lottery
</P>
<P ID=26>
BOOK FOURTH.--THE GORBEAU HOVEL
</P>
<P ID=27>
    I.  Master Gorbeau
   II.  A Nest for Owl and a Warbler
  III.  Two Misfortunes make One Piece of Good Fortune
   IV.  The Remarks of the Principal Tenant
    V.  A Five-Franc Piece falls on the Ground and produces a Tumult
</P>
<P ID=28>
BOOK FIFTH.--FOR A BLACK HUNT, A MUTE PACK
</P>
<P ID=29>
    I.  The Zigzags of Strategy
   II.  It is Lucky that the Pont d'Austerlitz bears
          Carriages
  III.  To Wit, the Plan of Paris in 1727
   IV.  The Gropings of Flight
    V.  Which would be Impossible with Gas Lanterns
   VI.  The Beginning of an Enigma
  VII.  Continuation of the Enigma
 VIII.  The Enigma becomes Doubly Mysterious
   IX.  The Man with the Bell
    X.  Which explains how Javert got on the Scent
</P>
<P ID=30>
BOOK SIXTH.--LE PETIT-PICPUS
</P>
<P ID=31>
    I.  Number 62 Rue Petit-Picpus
   II.  The Obedience of Martin Verga
  III.  Austerities
   IV.  Gayeties
    V.  Distractions
   VI.  The Little Convent
  VII.  Some Silhouettes of this Darkness
 VIII.  Post Corda Lapides
   IX.  A Century under a Guimpe
    X.  Origin of the Perpetual Adoration
   XI.  End of the Petit-Picpus
</P>
<P ID=32>
BOOK SEVENTH.--PARENTHESIS
</P>
<P ID=33>
    I.  The Convent as an Abstract Idea
   II.  The Convent as an Historical Fact
  III.  On What Conditions One can respect the Past
   IV.  The Convent from the Point of View of Principles
    V.  Prayer
   VI.  The Absolute Goodness of Prayer
  VII.  Precautions to be observed in Blame
 VIII.  Faith, Law
</P>
<P ID=34>
BOOK EIGHTH.--CEMETERIES TAKE THAT WHICH IS COMMITTED THEM
</P>
<P ID=35>
    I.  Which treats of the Manner of entering a Convent
   II.  Fauchelevent in the Presence of a Difficulty
  III.  Mother Innocente
   IV.  In which Jean Valjean has quite the Air of having read
          Austin Castillejo
    V.  It is not Necessary to be Drunk in order to be Immortal
   VI.  Between Four Planks
  VII.  In which will be found the Origin of the Saying: Don't
          lose the Card
 VIII.  A Successful Interrogatory
   IX.  Cloistered
</P>
<P ID=36>
VOLUME III
</P>
<P ID=37>
BOOK FIRST.--PARIS STUDIED IN ITS ATOM
</P>
<P ID=38>
     I.  Parvulus
    II.  Some of his Particular Characteristics
   III.  He is Agreeable
    IV.  He may be of Use
     V.  His Frontiers
    VI.  A Bit of History
   VII.  The Gamin should have his Place in the Classifications
            of India
  VIII.  In which the Reader will find a Charming Saying of the
            Last King
    IX.  The Old Soul of Gaul
     X.  Ecce Paris, ecce Homo
    XI.  To Scoff, to Reign
   XII.  The Future Latent in the People
  XIII.  Little Gavroche
</P>
<P ID=39>
BOOK SECOND.--THE GREAT BOURGEOIS
</P>
<P ID=40>
     I.  Ninety Years and Thirty-two Teeth
    II.  Like Master, Like House
   III.  Luc-Esprit
    IV.  A Centenarian Aspirant
     V.  Basque and Nicolette
    VI.  In which Magnon and her Two Children are seen
   VII.  Rule: Receive No One except in the Evening
  VIII.  Two do not make a Pair
</P>
<P ID=41>
BOOK THIRD.--THE  GRANDFATHER AND THE GRANDSON
</P>
<P ID=42>
     I.  An Ancient Salon
    II.  One of the Red Spectres of that Epoch
   III.  Requiescant
    IV.  End of the Brigand
     V.  The Utility of going to Mass, in order to become a
            Revolutionist
    VI.  The Consequences of having met a Warden
   VII.  Some Petticoat
  VIII.  Marble against Granite
</P>
<P ID=43>
BOOK FOURTH.--THE FRIENDS OF THE ABC
</P>
<P ID=44>
     I.  A Group which barely missed becoming Historic
    II.  Blondeau's Funeral Oration by Bossuet
   III.  Marius' Astonishments
    IV.  The Back Room of the Cafe Musain
     V.  Enlargement of Horizon
    VI.  Res Angusta
</P>
<P ID=45>
BOOK FIFTH.--THE EXCELLENCE OF MISFORTUNE
</P>
<P ID=46>
     I.  Marius Indigent
    II.  Marius Poor
   III.  Marius Grown Up
    IV.  M. Mabeuf
     V.  Poverty a Good Neighbor for Misery
    VI.  The Substitute
</P>
<P ID=47>
BOOK SIXTH.--THE CONJUNCTION OF TWO STARS
</P>
<P ID=48>
     I.  The Sobriquet; Mode of Formation of Family Names
    II.  Lux Facta Est
   III.  Effect of the Spring
    IV.  Beginning of a Great Malady
     V.  Divers Claps of Thunder fall on Ma'am Bougon
    VI.  Taken Prisoner
   VII.  Adventures of the Letter U delivered over to Conjectures
  VIII.  The Veterans themselves can be Happy
    IX.  Eclipse
</P>
<P ID=49>
BOOK SEVENTH.--PATRON MINETTE
</P>
<P ID=50>
     I.  Mines and Miners
    II.  The Lowest Depths
   III.  Babet, Gueulemer, Claquesous, and Montparnasse
    IV.  Composition of the Troupe
</P>
<P ID=51>
BOOK EIGHTH.--THE WICKED POOR MAN
</P>
<P ID=52>
     I.  Marius, while seeking a Girl in a Bonnet encounters a
            Man in a Cap
    II.  Treasure Trove
   III.  Quadrifrons
    IV.  A Rose in Misery
     V.  A Providential Peep-Hole
    VI.  The Wild Man in his Lair
   VII.  Strategy and Tactics
  VIII.  The Ray of Light in the Hovel
    IX.  Jondrette comes near Weeping
     X.  Tariff of Licensed Cabs, Two Francs an Hour
    XI.  Offers of Service from Mis
Be a part of the DaniWeb community

We're a friendly, industry-focused community of developers, IT pros, digital marketers, and technology enthusiasts meeting, networking, learning, and sharing knowledge.