Java mail api - Page 3

[JamesCherrill]

Hi - just a quick correction and apology.
In the latest versions I suggested adding a null to the queue to signal a shutdown. I should have tested that first - it doesn't work with a LinkedBlockingQueue. Sorry.
Please refer to the earlier example of queue code that I posted where I added a special String value "Please shut down now", then tested for that value in the queue processor (rather than testing for null). That will work OK.
My apologies.

[newbie14]

Dear James,
I am almost ready to go. I am using linux environment. So what are the benchmarks value do you want me to collect from my existing system for comparison?

[JamesCherrill]

Personally I see no point benchmarking the original one-connection-per-inbound-message code, since there's no way that can match opening a small number of database connections and re-using those for all transactions.
The simplest way to see the effect of 1,2,... database servers is to monitor the size of the queue. As long as the queue doesn't grow indefinitely then everything is ok, but as you improve the database throughput by approaching the ideal number of processors, you should see the average queue size getting smaller.

[newbie14]

Dear James,
Where and how to check queue size? Just to give further description this java programme I run on server and db server is separate server altogether. Thank you.

[JamesCherrill]

Where and how to check queue size?

You shouldn't be asking questions like that before trying for yourself. Please look at the API doc for LinkedBlockingQueue and see if you can find a way to get the queue size, it's easy. It really doesn't matter exactly where you check it as the queue is accessible to all parts of your program. Maybe each time you add something? each time you take an element? In a separate timer thread once every 10 seconds? Its up to you.

Just to give further description this java programme I run on server and db server is separate server altogether. Thank you.

That's OK. You may find that you get the best performance with a thread pool a bit larger than I suggested for running the database on the same server, but your testing will reveal that. Opening a connection to a remote server will take longer than on the same machine, so this really confirms that you are now using the best architecture.

[newbie14]

Dear James,
Sorry for the queue question I was thinking it would be done via some linux command to find out the queue size. In fact I go another idea. What I am planning is something below please give me your comments? I plan to record the queue size each time I am receving a message into a text file is that ok? Beside that I have one problem here private LinkedBlockingQueue<Message> eMailQueue = new LinkedBlockingQueue<Message>(); I am getting error because I have not formatted into a Message variable. How do you want me to format it into a message format?

public void run()
      {
         // this is just like the QueueProcessor example I gave you
         // open database connection
      	try
      	{
	         dbconn = DriverManager.getConnection("jdbc:mysql://localhost:3306/***?"+"user=***&password=***");
		     stmt = dbconn.createStatement();
	         while (true) 
	         {
                
	            try 
                     {

                      message = databaseQueue.take();
		    if (message.equals(null)) {
		      System.out.println("QueueProcessor is shutting down");
		      break; // exit while loop, ends run() method
		    }
		    System.out.println("Processing " + message);
		    

                       //File processing.

                       Date dateIn = new Date();
	              Date date = Calendar.getInstance().getTime(); 
                       String today = formatter.format(date); 
                       String filename= "MyDataFile"+today+".txt"; 
                       boolean append = true;
                
                       FileWriter fw = null;
		                 try
		                 {
                                   fw = new FileWriter(filename,append); 
			        fw.write(message+"  "+dateFormat.format(dateIn)+"\n");//appends the string to the file 
			       Date dateOut = new Date();
			       fw.write("Queue Size"+"  "+databaseQueue.size()+"\n");			       }
			       catch (IOException ex)  
			       { 
			          //ex.printStackTrace(new PrintWriter(sWriter));
			          System.out.println("MyError:IOException has been caught in in the file operation"+ex.toString());
			          ex.printStackTrace(System.out);
			       }  
                          } 
	            catch (Exception e) 
	            {
	            e.printStackTrace();
	            }            
                    }//while true

[JamesCherrill]

1. Write to file is fine. Suggest you put that code into a little method of its own (pass the message string as a parameter)and just call it from your while loop. If you keep putting all your code directly in the while loop it will become impossible for anyone else to to understand it.
2. LinkedBlockingQueue<Message>
Message here is a class, not a variable. It's saying that the queue is a queue of Message objects. In my code I started with
import javax.mail.Message;
so that's the Message class its referring to.
My assumption was that you would create the actual message object in your database code (where you still have access to all the info you need to create the message), then hand it off via the queue to a mail processor that would do the send (which is slow because you have to wait for the mail smtp/imap server)

[newbie14]

Dear James,

1. Yes I will call a separate method for it. I am also worried now I have more queries to be process but I guess I cannot separate them into each function then I will have problem with connections again right.

2.
Below is what I have manage to run as a separate code.So can you guide me how to adapt to yourstyle of the producer and consumer method.

Properties props = new Properties();
		props.put("mail.smtp.host", "smtp.gmail.com");
		props.put("mail.smtp.socketFactory.port", "465");
		props.put("mail.smtp.socketFactory.class",
				"javax.net.ssl.SSLSocketFactory");
		props.put("mail.smtp.auth", "true");
		props.put("mail.smtp.port", "465");
 
		Session session = Session.getDefaultInstance(props,
			new javax.mail.Authenticator() {
				protected PasswordAuthentication getPasswordAuthentication() {
					return new PasswordAuthentication("*****","******");
				}
			});
 
		try {
 
			Message message = new MimeMessage(session);
			message.setFrom(new InternetAddress("******"));
			message.setRecipients(Message.RecipientType.TO,
					InternetAddress.parse("******"));
			message.setSubject("Testing Subject");
			message.setText("Dear Mail Crawler," +
					"\n\n No spam to my email, please!");
 
			Transport.send(message);
 
			System.out.println("Done");
 
		  } 
		  catch (MessagingException e)
		  {
			throw new RuntimeException(e);
		   }

[JamesCherrill]

Sorry don't understand your point 1. When I suggested a separate method I just wanted to split the code into more readable pieces. I wasn't thinking of creating any new processes or threads or using any more connections for that. The number of database connections as always equal to the number of database processors objects you start, regardless of how many transactions there are or how long each one takes.
Point 2.
This code splits into three parts:
1. Mail initialisation: lines 1-14.
This needs to be executed once, so it goes in the mail processor's run method, before entering the while loop
2. Building the mail message: lines 18-24
This needs data from your database, so it needs to be in the database processor, after doing the SQL. Then put the constructed message onto the email queue.
3. Sending the message: line 26
This will take some time, so that's why we have the separate email processor/thread. This goes inside the while loop of the email processor (ie take next message / send message / repeat until shutdown is requested.

These are all illustrated by the comments in the template code I posted earlier - please review those now.

[newbie14]

Dear James,

1. I have created a separate method in the database processor class itself. I think I confuse you what I was trying to tell you is that I have quite a number of queries in the run function which I think that can be avoided to be further made it into separate functions. I got a compile error here Message emailMessage = new Message();

2. Below is my latest email processor code. Any comments. What should I put for tidy up section. Another thing for the transport do I need to put in a try and catch same goes for the Building the mail message: lines 18-24.

class MailProcessor implements Runnable {

      // sends emails that were placed on eMailQueue
     
      public void run() {

      Properties props = new Properties();		
      props.put("mail.smtp.host", "smtp.gmail.com");		
      props.put("mail.smtp.socketFactory.port", "465");		
      props.put("mail.smtp.socketFactory.class","javax.net.ssl.SSLSocketFactory");		
      props.put("mail.smtp.auth", "true");		
      props.put("mail.smtp.port", "465"); 		
      Session session = Session.getDefaultInstance(props,new javax.mail.Authenticator() 
      {				
      	protected PasswordAuthentication getPasswordAuthentication() 
      	{					
      	return new PasswordAuthentication("*****","******");				
      	}		
      });
         
         // this is also just like the QueueProcessor example I gave you

         //

         // do any Mail initialisation that maybe needed

         while (true) {

            // take next email Message from eMailQueue and send it

            // (if message == null app is closing down, so break;)
            
                   try 
	               {
	            	int count=0;
	            	Message emailMessage = new Message();
		            emailMessage = eMailQueue.take();
		            if (emailMessage.equals(null)) {
		               System.out.println("Email Processor is shutting down");
		            break; // exit while loop, ends run() method
		            }
		            
		            Transport.send(emailMessage);
		           }
		           // System.out.println("Processing " + message);
		           catch (IOException ex)  
			       { 
			          //ex.printStackTrace(new PrintWriter(sWriter));
			          System.out.println("MyError:IOException has been caught in in the file operation"+ex.toString());
			          ex.printStackTrace(System.out);
			       }         

         }

         // tidy up

      }

   }

[JamesCherrill]

1. You don't need to create a new Message on that line because you take one off the queue on the next line. Did you include javax.mail...?
2. You probably don't need any tidy up for mail; I'm not sure. YOur latest code looks OK to me - some details to fix, but nothing disasterous.
You don't need to ask me about try/catch. Firstly the API doc tells you whether a method call can throw an Exception and if so which. Secondly the compiler will tell you if you have missed one.
The shutdown thing with a null was my mistake, as I posted earlier. We need to think of a better way to add something to the queue which the processor can easily identify as a shutdown request. I'll leave that to you to come up with an idea.

I'm off to Spain for a brief holiday in the morning, so I won't be posting here until next week. If you hit problems re-read my earlier posts - I deliberately left hints relevant to the problems you were likely to hit next.
See you next week.
J

[newbie14]

Dear James,
Where are you from actually?
1. I dont get you where are you referring this to " You don't need to create a new Message on that line because you take one off the queue on the next line. " Are you referring to this one is it Message emailMessage? If I dont put that then my emailMessage is not recognise as Message Object. Yes I have included all the javax.mail relevant libraries.

[newbie14]

Dear James,
I got one problem now I get error for this Message message = new MimeMessage(session); as below because I have now separate the session settings separately. How to overcome this ya?

GPSDemo.java:437: cannot find symbol
symbol : variable session

[JamesCherrill]

1. I'm English, from London, but I've lived the past 12 years in South-West France
2. The code I referred to was
Message emailMessage = new Message();
emailMessage = eMailQueue.take();
because the new message you create in the first line is immediately discarded when you make the assignment on the next line.
3. session settings. OK, I confess I'm not a javamail expert. I missed the point that you need those session settings to create a Message.
You could do things to share the session object back from the email processor to the database processor, but IMHO that would be bad architecture because it creates a two-way dependency between logical functions where before it was only 1 way.
My best guess would be to put the info for the message into a single tab-delimited String (something like "[e-mail]\tHere is the subject\tHere is the body of the email") and put that on the email queue. So the database processor needs no knowledge of how to construct an email other than that it needs a recipient, subject, and body. Then the email processor can take that String, use split("\t") to get the 3 components, and do all the email stuff.

[newbie14]

Dear James,
How was your trip?What do you work as? I guess you are some kind of an expert consultant in java is it? So back to my problem I can do like this in my database process function string emailDetails="[e-mail]\tHere is the subject\tHere is the body of the email" and send via emailQueue.add(emailDetails);.
In this case move the session into the email processor itself will be better is it? So I guess I have to change this private LinkedBlockingQueue<Message> eMailQueue = new LinkedBlockingQueue<Message>(); to <String> right?

[JamesCherrill]

Hi. Trip was great thanks.
I just do as much (or as little) consulting as I choose now, but I used to be a design consultant with Hewlett-Packard's software engineering systems division specialising in O.O and GUI (and many other software-related things before that).
Anyway, I thought the the session variable was already in the mail processor? Apart from that, "yes" to all your other questions.

[newbie14]

Dear James,
Sorry got mixed up about the session settings. Ok I will do the changes and monitor. Actually why we cant just call a function with the database processor to run the email functiotnality?

[JamesCherrill]

Because then it would be running on a database processor thread and hold up database processing on that thread while the email did it's SMTP/IMAP network stuff (or whatever). You will probably also find that you get the best database throughput with a number of concurrent threads, but the best email throughput with a much smaller number (I'd guess "one").
Having said that there's no absolute reason why that would not work. I'm just treating this whole exercise as a coaching session in software architecture, and this kind of de-coupling of slow-running tasks via queues and thread pools is really good architecture. The bigger the scale, the better it works.

[newbie14]

Dear James,
Really appreciate your coaching in fact is something very valuable knowledge from a well experience person in the field. I really appreciate you. I would like to ask so incase in future I would like to also have a sms functionality do the same is it create a new sms processor is it?

[JamesCherrill]

Yes. Easy isn't it? This architecture scales well both for transaction volumes and number/complexity of processes.
The main case where you might not use this is where you have a process that has little or no wait time (ie no network accesses, little or no I/O). Database processing, email, SMS, remote server accesses in general, and any kind of user transaction all fit well.
Enough questions for now!

[newbie14]

Dear James,
Ok thank you let me wrap up all my codes and do a full test and see how things work. Thank you once again.