Change the next token of a stringtokenizer

[soUPERMan]

StringTokenizer st = new StringTokenizer(line);
				while (st.hasMoreTokens())
				{	
					String wordIn = st.nextToken();
					if (wordIn.endsWith("."))
						wordIn = wordIn.replace('.', ' ').trim();	
					if (wordIn.endsWith("?"))
						wordIn = wordIn.replace('?', ' ').trim();
					if (wordIn.endsWith(","))
						wordIn = wordIn.replace(',', ' ').trim();
					String result = bd.searchBase(bd.getWords(), wordIn, 0, bd.getWords().length -1);
					if (result != null)
						System.out.println("Word found \n");
					else
						System.out.println(baseSuggestions(wordIn));
						
					System.out.println("Press <Enter> for next word");
					key.nextLine();
					
				}

in the above code, a stream of words are extracted by the stringtokenizer separated by empty space, but when the word has the following characters (". , ?"), i want to convert the next word to lowercase, how do i go about that?

Thanks.

[masijade]

matches and toLowerCase ?

See the API docs for String

[soUPERMan]

matches and toLowerCase ?

See the API docs for String

What im asking is, for example the stream of words are: This is a new line in a sentence. And this is another

The word "sentence." will be extracted by the stringtokenizer together with the "." character...i want after that, the next word "And" to be changed to lowercase, im not sure how to exactly.

[masijade]

if (token.matches(regex)) {
  nextToken.toLowerCase()
}

as pseudo code.

try it. If it doesn't work post your attempt and we will help you correct it.

Once again, see the API docs for String.

[soUPERMan]

if (token.matches(regex)) {
  nextToken.toLowerCase()
}

as pseudo code.

try it. If it doesn't work post your attempt and we will help you correct it.

Once again, see the API docs for String.

This is what i did

StringTokenizer st = new StringTokenizer(line);
				while (st.hasMoreTokens())
				{	
					//String wordIn = st.nextToken();
					if (checkEnd(st.nextToken()) == true) // check if the word is at the end of the sentence
					{
					String wordIn = replaceEnd(st.nextToken()); // remove the punctuation 
					st.nextToken().toLowerCase(); //convert the next token to lowercase
					String result = bd.searchBase(bd.getWords(), wordIn, 0, bd.getWords().length -1);
					if (result != null) // word is found
						System.out.println("Word found \n");
					else 
						System.out.println(baseSuggestions(wordIn)); // base dictionary suggestions
					
					
					
					}
					else
					{
						String wordIn = st.nextToken();
						String result = bd.searchBase(bd.getWords(), wordIn, 0, bd.getWords().length -1);
						if (result != null)
							System.out.println("Word found \n");
						else
							System.out.println(baseSuggestions(wordIn));
					}		
					System.out.println("Press <Enter> for next word");
					key.nextLine();
					
				}

The methods

public static String replaceEnd(String wordIn)
	{
		if (wordIn.endsWith("."))
			return (wordIn.replace('.', ' ').trim());	
		else if (wordIn.endsWith("?"))
			return (wordIn.replace('?', ' ').trim());
		else if(wordIn.endsWith(","))
			return (wordIn.replace(',', ' ').trim());
		else 
			return wordIn;
	}
	
	public static boolean checkEnd(String wordIn)
	{
		char last =  wordIn.charAt(wordIn.length() - 1);
		if ((int)last == 33 || (int)last ==  44 || (int)last == 46 || (int)last == 58 || (int)last == 63)
			return true;
		else 
			return false;
	}

It doesn't seem to work, the next token still has an uppercase :( , did i write it incorrectly?

[soUPERMan]

i changed this part

while (st.hasMoreTokens())

to

while (st.nextToken() != null)

Seems to respond

[soUPERMan]

disregard that...false alarm

[masijade]

A String is immutable. You can "change" the String, but that really creates a new String. It is impossible to change a Strings value, so you need to read the Tokens and save them into a new String.

tokenizer = new tokenizer(line);
StringBuilder newLine = new StringBuilder();
while(hasmore) {
  if (matches) {
    newLine.append(token.toLowerCase);
  } else {
    newLine.append(token);
  }
}
line = newLine.toString();

Edit: Again as quasi pseudocode.

[soUPERMan]

A String is immutable. You can "change" the String, but that really creates a new String. It is impossible to change a Strings value, so you need to read the Tokens and save them into a new String.

tokenizer = new tokenizer(line);
StringBuilder newLine = new StringBuilder();
while(hasmore) {
  if (matches) {
    newLine.append(token.toLowerCase);
  } else {
    newLine.append(token);
  }
}
line = newLine.toString();

Edit: Again as quasi pseudocode.

I tried what u said, but i think i might be going wrong somewhere, here's the code:

import java.util.*;
import java.io.*;

public class TestShit{
	public static void main(String[] args){
		
		String fname = ("today.txt");
		
		
		StringBuilder sb = new StringBuilder();
		
		try 
		{
			FileReader fr = new FileReader(fname);
			BufferedReader br = new BufferedReader(fr);
			String line = br.readLine();
			
			while (line != null)
			{
				StringTokenizer st = new StringTokenizer(line," ");
				while (st.hasMoreTokens())
				{
				String wordIn = st.nextToken();
				if (checkEnd(wordIn) == true) // check if the word is at the end of the sentence
					{
						//sb.append(replaceEnd(wordIn) + " "); // remove the punctuation 
						System.out.println(replaceEnd(wordIn) + " ");
						st.nextToken().toLowerCase();
					}
				else {
					sb.append(wordIn);
					System.out.println(wordIn + " ");
				}
				}
				line = br.readLine();
			}
			//System.out.println(sb.toString());
		}
		catch (FileNotFoundException fnfe)
		{
			System.out.println(fname + " is not found");
		}
		catch (IOException ioe)
		{
			System.out.println("Error reading file");
		}
		

	}
	
	public static String replaceEnd(String wordIn)
	{
		
		if (wordIn.endsWith("."))
			wordIn = (wordIn.replace('.', ' ').trim());	
		else if (wordIn.endsWith("?"))
			wordIn = (wordIn.replace('?', ' ').trim());
		else if(wordIn.endsWith(","))
			wordIn = (wordIn.replace(',', ' ').trim());
		else 
			wordIn = wordIn;
			
		return wordIn;
	}
	
	public static boolean checkEnd(String wordIn)
	{
		char last =  wordIn.charAt(wordIn.length() - 1);
		if ((int)last == 33 || (int)last ==  44 || (int)last == 46 || (int)last == 58 || (int)last == 63)
			return true;
		else 
			return false;
	}	
}

Thanks for the help

[masijade]

this

st.nextToken().toLowerCase();

produces a new String object. what are you doing with it? It also, of course, consumes the next token in the String.

[soUPERMan]

this

st.nextToken().toLowerCase();

produces a new String object. what are you doing with it? It also, of course, consumes the next token in the String.

oh yeah, i noticed it consumes the next token. How do i stop that? How do i instead assign it to the next token to be read? I tried google but i not getting any desirable results. :(

[masijade]

You can't.

I am not going to take forever to describe this so think about it a bit. Rather than applying the "toLowerCase" and "nextToken" immediately, set a boolean variable to true, then, on the next iteration, when that variable is true, do the "toLowerCase" and set the variable to false.

But, it's not as though you're doing much with the stuff, so I see no reason why you can't simply perform the actions you're performing then and there.