Hi Daniweb.
I've been working on a project for quite a while, and posted many problems that I encountered on Daniweb. Thank you to all the users who helped me. Every single one of those problems where solved in less than 24 hours after posting on here.
My project mostly done now. I have one more problem, but I'm not asking for a solution here this time. I'm going to post the entire program (It may seems very simple to you, but I'm very new to Java, and have been learning throughout this project)
My problem is this: The first job in my settings file is being ignored. It is only processing files for the second job. It's a threading issue, I know. Can anyone point the problem out for me, please?
Code:
/*
* This program was designed with multithreading in mind.
* It will handle multiple files simultaniously, and provide logs.
*
* @author Lodewyk Duminy, Actnet Print Concepts, Cape Town
*/
package multithreadsprocessor20;
import org.apache.log4j.PropertyConfigurator;
import java.io.BufferedWriter;
import java.io.File;
import java.io.FileWriter;
import java.io.FilenameFilter;
import java.io.IOException;
import org.apache.log4j.Logger;
import java.util.ArrayList;
import java.util.regex.Pattern;
import javax.xml.parsers.DocumentBuilder;
import javax.xml.parsers.DocumentBuilderFactory;
import javax.xml.parsers.ParserConfigurationException;
import org.w3c.dom.Document;
import org.w3c.dom.Node;
import org.w3c.dom.NodeList;
import org.xml.sax.SAXException;
public class MultiThreader {
public static Logger logger = Logger.getLogger(MultiThreader.class);
/*
* Declaration of all the variables that will contain values
* supplied by the settings file. These will be used throughout
* the program
*/
public static String parentDirLocation;
public static String logDir;
public static String settingsLocation;
public static int totalThreads;
public static int priorityThreads;
public static int normalThreads;
/*
* @param args Input from the user. The first argument has to be the location of the settings file, the second the name of the drive that this is run from
*/
public static void main(String[] args) {
PropertyConfigurator.configure("log4j.properties");
new Settings().globalSettings(args[0]);
logger.info("Read global settings from " + args[0]);
settingsLocation = args[0];
while (true) {
logger.info("Starting main scanning loop.");
/*
* Get a list of jobs, then iterate through them, get their settings,
* and pick up files for them to process.
*
* Assumptions:
* 1. There are jobs in the settings file
*/
String[] jobList = new Settings().jobList(args[0]);
logger.info("Job found");
for (int numJobs = 0; numJobs < jobList.length; numJobs++) {
/*
* Get each job's settigns, then call filePickup() for it
*/
String jobName = jobList[numJobs];
logger.info("Starting job " + jobName);
try {
Thread.sleep(6000);
} catch (InterruptedException ie) {
}
new MultiThreader().jobSpawner(jobName);
}
}
}
void jobSpawner(String jobName) {
new Thread(new JobThread(jobName)).start();
}
}
class Threadcounter {
private int threadCnt = 0;
private int totThreads = MultiThreader.totalThreads;
/*
* @return false if the maximum number of threads are alive
*/
public boolean threadCntr() {
if (threadCnt >= totThreads) {
return false;
} else {
return true;
}
}
/*
* @param cnt is the amount that the thread count will be modified with.
*
* 'cnt' should never be more or less than 1
*/
public void cntModifier(boolean cnt) {
if (cnt) {
threadCnt++;
} else if (!cnt) {
threadCnt--;
}
}
}
class JobThread implements Runnable {
private String jobName;
private String filemask;
private String priorityDirs;
private String normalDirs;
private String retain;
private String jobbatchparams;
private String jobbatchFilepath;
private String overrideFolder;
JobThread(String job) {
this.jobName = job;
}
public void run() {
jobSettings(MultiThreader.settingsLocation, this.jobName);
System.out.println("job: " + this.jobName);
scan();
}
void scan() {
/*
* This function makes it possible to track the total amount of threads
*/
if (!priorityDirs.equals("") && !priorityDirs.equals(null)) {
MultiThreader.logger.info("Scanning high priority files");
/*
* Scan priority folder for files, use filepickup
* filePickup will scan the high priority dir(s)
*/
filePickup("high", MultiThreader.priorityThreads);
} else {
MultiThreader.normalThreads = MultiThreader.totalThreads;
}
if (!normalDirs.equals("") && !normalDirs.equals(null)) {
MultiThreader.logger.info("Scanning low priority files");
/*
* Scan normal folders for files, use filepickup
* filepickup will scan the low priority dir(s)
*/
filePickup("low", MultiThreader.normalThreads);
}
}
/*
* @param settingsLoc is the location of the settings file that will supply the job's settings.
* @param jobName is the name of the job who's settings need to be read in.
*
* @exception IOException will be thrown if the file can t be opened to be read.
* @exception SAXException will be thrown if the .xml file that needs to be read hasn't been written correctly, ie it has an XML error.
* @exception ParserConfigurationException indicate a serious configuration error.
*/
private void jobSettings(String settingsLoc, String jobName) {
/*
* Read the job's settings
*/
try {
DocumentBuilderFactory factory = DocumentBuilderFactory.newInstance();
DocumentBuilder docBuilder = factory.newDocumentBuilder();
Document doc = docBuilder.parse(new File(settingsLoc));
NodeList nodelist_topLevel = doc.getElementsByTagName("*"); // Lists all the nodes
/*
* Read the job settings, where the "job" node's content matches jobName
*/
for (int nodes = 0; nodes < nodelist_topLevel.getLength(); nodes++) {
Node node_topLevel = (Node) nodelist_topLevel.item(nodes);
if (node_topLevel.getNodeName().equalsIgnoreCase("job")) {
NodeList nodeList_secondLevel = node_topLevel.getChildNodes(); // Lists all the noes under "global"
for (int globalNodes = 0; globalNodes < nodeList_secondLevel.getLength(); globalNodes++) {
Node node_secondLevel = nodeList_secondLevel.item(globalNodes);
String nodeName = node_secondLevel.getNodeName();
/*
* Set the global variables after checking the node names
*/
if (nodeName.equalsIgnoreCase("filemask")) {
this.filemask = node_secondLevel.getTextContent();
} else if (nodeName.equalsIgnoreCase("prioritydirs")) {
this.priorityDirs = node_secondLevel.getTextContent();
} else if (nodeName.equalsIgnoreCase("normaldirs")) {
this.normalDirs = node_secondLevel.getTextContent();
} else if (nodeName.equalsIgnoreCase("retain")) {
this.retain = node_secondLevel.getTextContent();
} else if (nodeName.equalsIgnoreCase("jobbatchparams")) {
this.jobbatchparams = node_secondLevel.getTextContent();
} else if (nodeName.equalsIgnoreCase("jobbatchfilepath")) {
this.jobbatchFilepath = node_secondLevel.getTextContent();
} else if (nodeName.equalsIgnoreCase("overridefolder")) {
this.overrideFolder = node_secondLevel.getTextContent();
}
}
}
}
} catch (ParserConfigurationException pce) {
MultiThreader.logger.error("Parser config Exception caught: " + pce);
} catch (SAXException sax) {
MultiThreader.logger.error("SAX Exception caught: " + sax);
} catch (IOException io) {
MultiThreader.logger.error("IO Exception caught: " + io);
}
}
/*
* @param priority This specifies what priority the job has, and will use the appropriate thead count
* @param threadCap This is the maximum amount of allowed threads
*
* @exception InterruptedException Raised if any of the .sleep() functions are interrupted.
*/
public void filePickup(String priority, int threadCap) {
/*
* threadCap is the total amount of threads that will be allowed to be open at the same time
*/
try {
ArrayList folders = new ArrayList();
/*
* Coding in here seems redundant, but apparently
* you can't split right into an ArrayList, which forces me
* to split into an array, then iterate through it and
* populate the ArrayList from there.
*/
if (priority.equals("high")) {
/*
* Get the list of high priority folders
*/
String[] foldersTemp = priorityDirs.split(",");
for (int x = 0; x < foldersTemp.length; x++) {
folders.add(foldersTemp[x]);
}
} else if (priority.equals("low")) {
/*
* Gets the list of low priority folders
*/
String[] foldersTemp = normalDirs.split(",");
for (int x = 0; x < foldersTemp.length; x++) {
folders.add(foldersTemp[x]);
}
}
/*
* Iterate and scan through the folders[]
*/
for (int folderScan = 0; folderScan < folders.size(); folderScan++) {
/*
* Get the folder's absolute path and scan it for files
* Make sure that the total number of allowed threads is not exceeded
*/
File file = new File(folders.get(folderScan).toString());
File[] files = file.listFiles();
MultiThreader.logger.info("Found " + files.length + " files");
for (int totalFiles = 0; totalFiles < files.length; totalFiles++) {
if (files[totalFiles].isFile()) {
String fileName = files[totalFiles].getName();
FilenameFilterRegex FF = new FilenameFilterRegex();
FF.aregex(this.filemask);
File fakeFile = new File("");
if (FF.accept(fakeFile, fileName)) {
MultiThreader.logger.info("Handling " + fileName);
while (true) {
Threadcounter tc = new Threadcounter();
if (tc.threadCntr() == false) {
// Keep looping
Thread.sleep(5000);
} else {
new Thread(new ThreadSpawner(fileName, file.getAbsolutePath(), retain, jobbatchFilepath, jobbatchparams, overrideFolder)).start();
tc.cntModifier(true);
Thread.sleep(5000);
break;
}
}
}
}
Thread.sleep(2000);
}
}
} catch (InterruptedException ie) {
MultiThreader.logger.error("Interrupted exception caught: " + ie);
}
}
public static class FilenameFilterRegex implements FilenameFilter {
private String strRegex = "";
public String aregex(String straRegex) {
if (straRegex.indexOf("*") != -1) {
}
straRegex = straRegex.replace("*", ".*");
strRegex = straRegex;
return (strRegex);
}
/*
* @param dir This isn't used, but is still kept in case future versions have a use for it.
* @param name This is the name of the file that has to be checked with regex
*
* @return true if the file's name matches the required pattern
*/
public boolean accept(File dir, String name) {
/*
* return Pattern.matches(".*\\.(jpg|jpeg|gif|png|bmp)", name);
* if only one extension to check : "\\.jpg"
*/
return Pattern.matches(strRegex, name);
}
}
}
class ThreadSpawner implements Runnable {
private String file;
private String folder;
private String retain;
private String jobbatchFilepath;
private String jobbatchparams;
private String overrideFolder;
//private MultiThreader ts;
/*
* @param TS This is just a required parameter for the run() to be called
* @param fileName is the file that will be handled by the thread
* @param origFolder is the location of the source file, which will be moved
*/
ThreadSpawner(/*MultiThreader TS,*/String fileName, String origFolder, String retainTemp, String jobbatchFilepathTemp, String jobbatchparamsTemp, String overrideFolderTemp) {
//this.ts = TS;
this.file = fileName;
this.folder = origFolder;
this.retain = retainTemp;
this.jobbatchparams = jobbatchparamsTemp;
this.jobbatchFilepath = jobbatchFilepathTemp;
this.overrideFolder = overrideFolderTemp;
}
/*
* This is the actual thread.
*
* The source files will be moved and the OverrideFile method will be called
* to write an override file
*
* 'ExecuteCommand' will be called and given a set of Strings as variables
*/
public void run() {
File fromFile = new File(this.folder + "\\" + this.file);
File toFile = new File(this.retain + "\\" + this.file);
String returnFile = OverrideFile(fromFile, toFile);
MultiThreader.logger.info("Return file written - " + returnFile);
/*
* Use return file
*/
int intRC1 = ExecuteCommand("\"" + jobbatchFilepath + "\"", " -j " + "\"" + jobbatchparams + "\" -q", "\"" + returnFile + "\"", "WAIT");
if (intRC1 == 1 || intRC1 == 0) {
Threadcounter tc = new Threadcounter();
tc.cntModifier(false);
}
}
/*
* @param Arg0 is the job batchfile
* @param Arg1 contains the job parameters
* @param Arg2 is the location of the return file
* @param strRunmode is the runmode, and can be
* 1. WAIT
* 2. NOWAIT
*
* @return int is the exit value of the command prompt
*/
public int ExecuteCommand(String Arg0, String Arg1, String Arg2, String strRunmode) {
/*
* Executes batch commands and waits for results, captures output and writes to console and log using streamGobbler
*/
int exitVal = 0;
try {
String[] cmd = new String[3];
cmd[0] = "cmd.exe";
cmd[1] = "/C";
cmd[2] = Arg0 + " " + Arg1 + " " + Arg2;
String cmdLine = "";
for (int testx = 0; testx < 3; testx++) {
cmdLine = cmdLine + " - " + cmd[testx];
}
MultiThreader.logger.info("Processing file with command line: " + cmdLine);
Runtime rt = Runtime.getRuntime();
Process proc = rt.exec(cmd);
if (strRunmode.toUpperCase().equals("NOWAIT")) {
exitVal = 0;
/*
* Return Control Immediately
*/
} else {
exitVal = proc.waitFor();
}
} catch (Throwable t) {
MultiThreader.logger.error("Exception caught: " + t);
}
return (exitVal);
}
/*
* @param original is the location the original file will be moved from
* @param copy is the location the original file will be moved to
*
* @return String is the location of the return file
*
* @exception IOException is thrown if the return file cannot be written
*/
public String OverrideFile(File original, File copy) {
String overrideName = original.getName();
/*
* Get the file location
*/
File origLoc = new File(original.getAbsolutePath().toString());
File copyLoc = new File(copy.getAbsolutePath().toString());
String returnFile = "";
System.out.println("origional: " + origLoc);
System.out.println("copy: " + copyLoc);
try {
System.out.println("...");
if (origLoc.renameTo(copy)) {
System.out.println("origional file moved");
/*
* Contents of the return file will be generic, with only the name and
* the location of the source file differing between files.
*/
String contents = "<?xml version=\"1.0\" encoding=\"UTF-8\" ?>\n" +
"<JobDatasourceOverride>\n" +
"<JobStepDatasourceOverride jobStepID=\"0\" jobStepName=\"1\" defineAssembleDatasource=\"\" definedTriggerDatasource=\"\" overridenAssembleDatasource=\"\" overridenAssembleDataFile=\"" + copyLoc + "\" overridenTriggerDatasource=\"\" overridenTriggerDataFile=\"\" />\n" +
"</JobDatasourceOverride>";
returnFile = overrideFolder + "\\" + overrideName + ".xml";
MultiThreader.logger.info("Return file name: " + returnFile);
/*
* Write out the return file
*/
BufferedWriter out = new BufferedWriter(new FileWriter(returnFile));
out.write(contents);
out.flush();
out.close();
} else {
/*
* Log that file could not be moved, return blank return location
*/
returnFile = "";
}
} catch (IOException io) {
MultiThreader.logger.error("IO Exception caught: " + io);
}
return returnFile;
}
}
class Settings {
/*
* @param settingsLoc is the location of the settings file that will supply the global settings
*
* @exception IOException will be thrown if the file can t be opened to be read.
* @exception SAXException will be thrown if the .xml file that needs to be read hasn't been written correctly, ie it has an XML error.
* @exception ParserConfigurationException indicate a serious configuration error
*/
void globalSettings(String settingsLoc) {
try {
DocumentBuilderFactory factory = DocumentBuilderFactory.newInstance();
DocumentBuilder docBuilder = factory.newDocumentBuilder();
Document doc = docBuilder.parse(new File(settingsLoc));
/*
* Lists all the nodes
*/
NodeList nodelist_topLevel = doc.getElementsByTagName("*");
/*
* Read Global settings
*/
for (int nodes = 0; nodes < nodelist_topLevel.getLength(); nodes++) {
Node node_topLevel = (Node) nodelist_topLevel.item(nodes);
if (node_topLevel.getNodeName().equalsIgnoreCase("global")) {
/*
* Lists all the nodes under "global"
*/
NodeList nodeList_secondLevel = node_topLevel.getChildNodes();
for (int globalNodes = 0; globalNodes < nodeList_secondLevel.getLength(); globalNodes++) {
Node node_secondLevel = nodeList_secondLevel.item(globalNodes);
String nodeName = node_secondLevel.getNodeName();
/*
* Set the global variables after checking the node names
*/
if (nodeName.equalsIgnoreCase("totalthreads")) {
MultiThreader.totalThreads = Integer.parseInt(node_secondLevel.getTextContent());
} else if (nodeName.equalsIgnoreCase("prioritythreads")) {
MultiThreader.priorityThreads = Integer.parseInt(node_secondLevel.getTextContent());
} else if (nodeName.equalsIgnoreCase("logdir")) {
MultiThreader.logDir = node_secondLevel.getTextContent();
}
MultiThreader.normalThreads = MultiThreader.totalThreads - MultiThreader.priorityThreads;
}
}
}
} catch (ParserConfigurationException pce) {
MultiThreader.logger.error("Parser config Exception caught: " + pce);
} catch (SAXException sax) {
MultiThreader.logger.error("SAX Exception caught: " + sax);
} catch (IOException io) {
MultiThreader.logger.error("IO Exception caught: " + io);
}
}
/*
* @param settingsLoc is the location of the settings file that will supply the global settings
*
* @return String[] is an array of all the jobs in the esttings file.
*
* @exception IOException will be thrown if the file can t be opened to be read.
* @exception SAXException will be thrown if the .xml file that needs to be read hasn't been written correctly, ie it has an XML error.
* @exception ParserConfigurationException indicate a serious configuration error
*/
String[] jobList(String settingsLoc) {
ArrayList jobArrayList = new ArrayList();
/*
* Read all the job names in the settigns file
*/
try {
DocumentBuilderFactory factory = DocumentBuilderFactory.newInstance();
DocumentBuilder docBuilder = factory.newDocumentBuilder();
Document doc = docBuilder.parse(new File(settingsLoc));
NodeList list = doc.getElementsByTagName("*");
/*
* Read the settigns after finding the correct job
*/
for (int i = 0; i < list.getLength(); i++) {
Node node = (Node) list.item(i);
/*
* This dives into the JOB sets, and searches for the correct set
*/
if (node.getNodeName().equalsIgnoreCase("JOB")) {
NodeList list2 = node.getChildNodes();
Node nameName = (Node) list2.item(1);
if (nameName.getNodeType() == Node.ELEMENT_NODE && nameName.getNodeName().equals("NAME")) {
/*
* Get the settings
*/
String job = nameName.getTextContent();
jobArrayList.add(job);
}
}
}
} catch (ParserConfigurationException pce) {
MultiThreader.logger.error("Parser config Exception caught: " + pce);
} catch (SAXException sax) {
MultiThreader.logger.error("SAX Exception caught: " + sax);
} catch (IOException io) {
MultiThreader.logger.error("IO Exception caught: " + io);
}
/*
* Caste jobArrayList to an array and return it
*/
String[] jobArray = new String[jobArrayList.size()];
for (int jobNumber = 0; jobNumber < jobArrayList.size(); jobNumber++) {
jobArray[jobNumber] = jobArrayList.get(jobNumber).toString();
}
return jobArray;
}
}Settings (in .xml file):
<SETTINGS>
<GLOBAL>
<TOTALTHREADS>10</TOTALTHREADS>
<PRIORITYTHREADS>5</PRIORITYTHREADS>
<LOGDIR>C:\MultiThreader\log</LOGDIR>
</GLOBAL>
<JOB>
<NAME>One</NAME>
<ACTIVESTATUS>Active</ACTIVESTATUS>
<FILEMASK>*.txt</FILEMASK>
<PRIORITYDIRS>C:\MultiThreader\files</PRIORITYDIRS>
<NORMALDIRS></NORMALDIRS>
<STARTTIME>60000</STARTTIME>
<STOPTIME>60000</STOPTIME>
<MAXRUNTIME>600000</MAXRUNTIME>
<KILLNOTIFY>Kill</KILLNOTIFY>
<RETAIN>C:\MultiThreader\retain</RETAIN>
<JOBBATCHPARAMS>FC</JOBBATCHPARAMS>
<JOBBATCHFILEPATH>C:\jboss\server\default\deploy\xPression.ear\BatchRunner.bat</JOBBATCHFILEPATH>
<OVERRIDEFOLDER>C:\MultiThreader\override</OVERRIDEFOLDER>
<DBUSER>xpression</DBUSER>
<DBPWORD>xpression</DBPWORD>
<DBDRIVER>com.microsoft.jdbc.sqlserver.SQLServerDriver</DBDRIVER>
<SOURCEURL>jdbc:sqlserver:/LOCALHOST:1433;selectMethod=cursor;databaseName=xPress_CR;</SOURCEURL>
</JOB>
<JOB>
<NAME>Two</NAME>
<ACTIVESTATUS>Active</ACTIVESTATUS>
<FILEMASK>*.txt</FILEMASK>
<PRIORITYDIRS>C:\MultiThreader\files2</PRIORITYDIRS>
<NORMALDIRS></NORMALDIRS>
<STARTTIME>60000</STARTTIME>
<STOPTIME>60000</STOPTIME>
<MAXRUNTIME>600000</MAXRUNTIME>
<KILLNOTIFY>Kill</KILLNOTIFY>
<RETAIN>C:\MultiThreader\retain</RETAIN>
<JOBBATCHPARAMS>FC</JOBBATCHPARAMS>
<JOBBATCHFILEPATH>C:\jboss\server\default\deploy\xPression.ear\BatchRunner.bat</JOBBATCHFILEPATH>
<OVERRIDEFOLDER>C:\MultiThreader\override</OVERRIDEFOLDER>
<DBUSER>xpression</DBUSER>
<DBPWORD>xpression</DBPWORD>
<DBDRIVER>com.microsoft.jdbc.sqlserver.SQLServerDriver</DBDRIVER>
<SOURCEURL>jdbc:sqlserver:/LOCALHOST:1433;selectMethod=cursor;databaseName=xPress_CR;</SOURCEURL>
</JOB>
</SETTINGS>Many of the settings aren't used in my project. That's because another project is using them, so they can be ignored.
The settings are read in fine, the files are being moved correctly, etc. The ONLY problem I'm facing at the moment is that the thread that no many how many jobs I put in there, it only handles the threads for the last one.
