Member Avatar for ultimate_fusion

I have to read a file in format char, int string.
I can easily do it in the form - int int, string e.g. 12 34 the cat sat on the mat(which is what I want to process and output)
code below.

tokenType = inputStream.nextToken();
		while (tokenType != StreamTokenizer.TT_EOF)
		{
		   
			firstNo= (float)inputStream.nval; inputStream.nextToken();
			secondNo = inputStream.nval;
                                      thirdstring = inputStream.Sval;
			System.out.println(firstNo+"\t"+secondNo+"\n"+thirdstring );			
			tokenType = inputStream.nextToken();
			
			
      }
		}

but when theres a character at the begining or a string, int, string I can only get the first word(shown below) of the string instead of the whole title.

in.nextToken();

		while (in.ttype!=StreamTokenizer.TT_EOF)
		{
			if(in.ttype==StreamTokenizer.TT_WORD){
				type=in.sval;
			}
			else{
				System.out.println("Wrong file format");
			}
			
				if(in.nextToken()==StreamTokenizer.TT_NUMBER){
				v=in.nval;
			}
			else{
				System.out.println("Wrong file format");
			}
		
				if(in.nextToken()==StreamTokenizer.TT_WORD){
					   
		                              \\here
				     show=in.sval;
				     
				    in.nextToken();
                                       }
				
				   
				    
				    }

it will get the first word of the last string.
I want to put the foloowing code where "\\here" is to get the rest of string

show+=in.sval;
show+=" ";
 if(in.nextToken()==StreamTokenizer.TT_EOL)
{
    break;
}
show=" ";

but it doesn't like EOL, nothing gewts printed out???

  • 2 Contributors
  • 1 Reply
  • 623 Views
  • 7 Hours Discussion Span
  • Latest Post Latest Post by server_crash

Dani AI

Generated

Short answer: StreamTokenizer will split an unquoted multi‑word title into separate TT_WORD tokens and it does not emit TT_EOL by default — so your TT_EOL check never becomes true, and repeatedly calling nextToken() inside if tests advances/consumes tokens unexpectedly. Two practical fixes follow.

Example 1 — keep StreamTokenizer but make EOL significant and read tokens into a buffer (call nextToken() once and test the returned ttype each time):

StreamTokenizer st = new StreamTokenizer(new BufferedReader(new FileReader("in.txt")));
st.eolIsSignificant(true);

int t = st.nextToken();            // first token
String type = (t == StreamTokenizer.TT_WORD) ? st.sval : String.valueOf((char)t);

t = st.nextToken();                // number
if (t != StreamTokenizer.TT_NUMBER) throw new IOException("Bad format");
int value = (int)st.nval;

StringBuilder title = new StringBuilder();
t = st.nextToken();                // start of title
while (t != StreamTokenizer.TT_EOL && t != StreamTokenizer.TT_EOF) {
    if (t == StreamTokenizer.TT_WORD || t == '"' || t == '\'') {
        if (title.length() > 0) title.append(' ');
        title.append(st.sval);
    } else if (t == StreamTokenizer.TT_NUMBER) {
        if (title.length() > 0) title.append(' ');
        title.append(Double.toString(st.nval));
    } else {
        title.append((char)t);
    }
    t = st.nextToken();
}
String show = title.toString();

Example 2 — much simpler and more robust: read line-by-line and split into 3 parts. This avoids tokenizer quirks:

BufferedReader br = new BufferedReader(new FileReader("in.txt"));
String line;
while ((line = br.readLine()) != null) {
    String[] p = line.trim().split("\\s+", 3);
    if (p.length < 3) { /* handle */ continue; }
    String type = p[0];
    int value = Integer.parseInt(p[1]);
    String title = p[2];
}

Notes: check sval/nval (lowercase), avoid calling nextToken() inside conditionals, and remember quoted strings produce a token whose ttype is the quote character and whose sval holds the whole quoted string. If you want a very small change to your current code, enable eolIsSignificant(true) and switch to a single nextToken() per test. This addresses the behaviour you saw.

Member Avatar for server_crash

I don't think you can differentiate between different data types using a StringTokenizer. You would have to use a StringBuffer to hold all the stuff and then do some parsing.

Although a StringTokenizer doesn't differentiate, a StreamTokenizer does. It can distinquish between Strings, ints, and all that good stuff.

Be a part of the DaniWeb community

We're a friendly, industry-focused community of developers, IT pros, digital marketers, and technology enthusiasts meeting, networking, learning, and sharing knowledge.