Need advice on a future application

[MrDiaz]

Hi everyone

I've been trying to develop a mid-scale (size wise) application in the past few days. Something I'm doing for fun, to get some experience. My application is some sort of log system, it can have different users, each user has its own log entries (or records) and so on. I've been creating this app using text files, users being folders. In my opinion this is very ineffective, so I've been thinking in the use of databases for my data.

I was wondering, what would be best for me to use to store data.

- MS Access
- MySQL
- Text files
- Other alternatives

I guess you can call my app a C.R.U.D application, or sort of. I want my data to be kept forever pretty much. What do you guys suggest I should do this?

[sillyboy]

I'm not sure it matters too much. Sure a database is probably a cleaner environment to use, but if it just for fun, who cares? Then again, it is probably worthwhile getting some experience coding /w databases too.

I would say try MySQL as is it free and widely used.

[verruckt24]

If you are developing a logging tool, flat files are the best. Since an application using your tool might be logging hundreds of entries in a matter of a few seconds and needless to say you are certainly not going to achieve that kind of speed with logging into a database system.

On the other hand you could implement multiple alternatives and let the user select one of them.

Also if you absolutely want to have the log entries put into a database system what you could do is use a flat file at runtime to log in entries as they are generated by the application. But these entries are written in a formatted manner to the file, say comma-separated so that you could push them into the db say on the issue of a single command at a later time. Check the "LOAD DATA INFILE" option for MySQL back ends.

This option has the advantage of speed with the logging into the flat file at runtime (though this file won't be of much help to read from) and the added option of having the entries into a db for viewing them through an interface. Once the log entries are in a db you can build up the view for each user accordingly through the use of different queries.

Note : LOAD DATA INFILE of MySQL is much like the bcp (Bulk copy) of MS-SQL Server, which inserts records into the db in bulk and hence is much much faster than single inserts.

[~s.o.s~]

Is this a web application or a desktop application?

If it's a standalone desktop application, then consider using a serverless & embeddable database which is most suitable for replacing such ad-hoc file/folder storage. An embeddable database like SQLite uses a single file for storage which is *really* minimalistic.

If it's a small scale web application, consider using a small footprint pure Java database like Derby since it would help you get started in a small amount of time without any server setup headache. Of course, MySQL, PostgreSQL etc. also are good production ready alternatives.

> Since an application using your tool might be logging hundreds of
> entries in a matter of a few seconds and needless to say you are
> certainly not going to achieve that kind of speed with logging into
> a database system

That statement can be thought of as "premature optimization" in the absence of proper profiling. Sure, it might be so that the performance of raw writes is better than using a database but is that kind of performance really required? Embeddable databases can be thought of as flat files which come pre-packaged with tools for processing the file data.

What happens if there arises a need for pulling out entries between a given date range? Implementing a custom logic which iterates through all the log entries might prove out to be really troublesome esp with the sheer amount of data a logging tool might generate.

IMO, a data rightly belongs to a flat file if:
- The user requires direct manipulation of the data
- The volume of data is not humongous [again this is a relative term]
- The data is required to be in format capable of being processed by outside applications in a platform neutral manner. [like XML, JSON etc.]

[JamesCherrill]

Either way, you could (should?) make the storage mechanism "pluggable" by defining an abstract class that handles all the store/retrieve operations and then implement flat file and/or SQL versions as required.

[verruckt24]

@s.o.s : I was thinking more of on the lines that the user was building a logging tool one like apache log4j.

@OP : And if you are doing that, more than anything else you should look at the tool as the pipe from source to sink. Where the sink should be pluggable to accomodate the needs of various users/systems.

[MrDiaz]

To be more specific, the app is not really a log system. What I'm doing is a Pilot Log Book for students.

- There can be multiple students (users)
- Each student has daily entries (with a bunch of records)

I was taking a look at http://www.netbeans.org/kb/docs/java/gui-db-custom.html and it looks pretty good. What you guys think about using this?

[kvprajapati]

Dear,

Think about XML. You said that your application is sort of log system; Use JDOM API instead of Java's DOM or SAX.

[JamesCherrill]

Reading the entirely sensible suggestions above, I'm struck by the thought that this is a bit front-backward for an OO language design.
Maybe you should really pin down the requirements by formally coding the interface that this must support. For example, and this is just by way of illustration...

class LogEntry {
   public LogEntry(String user, Date timestamp, int errorlevel, String message) {
   // etc
   }
   // usual accessors
}
abstract class Log {
   public void write(LogEntry e);
   public LogEntry[] getEntries(LogEntry searchTemplate, Date from, Date to);
   public void purge(Date olderThan);
   // etc:  what do you really want to do with these logs?
}

Then you can attach an estimate of volumes and frequencies for each of these calls, then, and only then, you will be in a good position to evaluate alternative implementations.

[MrDiaz]

Hi James, I have all my methods/class/variables set up. My program currently works using text files, that as I mentioned on my first post I don't think is a clever way of doing it.

I'd like to use something more organized, long-lasting type of data.

[JamesCherrill]

OK That's great! Can you show us the public method signatures?

[MrDiaz]

I have like 100, setters/getters for my variables. Some methods to load/save data on my files.

[JamesCherrill]

No need for all the accessors, but I thought it would be useful to see how complex/variable the log records are, and how complex the retrieval or query operations are. For example, high variability of format, low query complexity may lean you towards XML, but fixed formats and complex queries may lean you towards SQL (you get the idea) (ps I'm not trying to start an XML/SQL flame war here).

[MrDiaz]

This is a small sample of my store method, will take the data the user entered on the GUI and save it to a text file.

public void store(String fileName) throws IOException {
        Day newDay = new Day();
        setValues(newDay); 
        addDay(newDay);
        try {
            PrintWriter  out = new PrintWriter (new FileWriter(fileName));
            for (Day day:listOfDays){
                out.println(day.getDate() + "," + day.getAircraftType() + "," +
                        day.getAircraftIndent() + "," + day.getRouteOfFlight() +
                        "," + day.getDuration() + "," + day.getDayLandings() + "," +
                        day.getNightLandings() + "," + day.getInstrument() + "," +
                        day.getSimInstrument() + "," + day.getNightHours() + "," +
                        day.getSimulator() + "," + day.getXc() + "," +
                        day.getSolo() + "," + day.getPic() + "," + day.getSic() + "," +
                        day.getDual() + "," + day.getIp());
            }
            out.close();
    } catch (IOException e) {
    }

[JamesCherrill]

Ah thanks - that's a lot more data than I was imagining. Are there more like this or is this the main/only such record format?
Also: are you looking for general comments on program design, or just choice of data storage technique?

[MrDiaz]

There's a little more data, users are not included there. I also have a config file for application position, first time users, small things like that.

Problem with using those text files is that it makes updating/deleting/adding records a little more difficult, I'm not too happy with that. If you can give me some ideas on program design I'll glady take those too! Thanks James

[JamesCherrill]

A couple more questions: you say update/delete records, but a "log" typically records things as they happen, and those historical records don't get updated. You also said "want my data to be kept forever", so where does delete come in?
As far as structure is concerned (what follows is my opinion, OK?) for a "mid-scale" I'd expect to see a GUI layer sitting on top of a domain model layer, sitting on top of a storage/persistence layer, with a high degree of isolation between them. In that model I'd expect the store method to look more like public void store(Day d) {...} What's setValues? Not a GUI, I hope.
Here's a quick test or two for good layering and isolation:
If you implement web browser HTML interface to run in parallel with your existing GUI, what existing code needs to be modified (optimum answer: one call in main()).
If you implement a new data storage mechanism, such as SQL instead of text files, what existing code needs to be modified (ideal answer: maybe 1 line somewhere) and how much code in the text file version needs to be duplicated in the SQL version (ideally: none).
Anyway, to go back to your original question, from what I've now heard I would definitely go with an SQL database.Java DB is standard from JDK 1.6, and is surely the easiest SQL to get up and running ( http://developers.sun.com/javadb/ ) and mySQL is probably the best choice if you want a more full-function BD environment. Moving from Java DB to mySQL should only need one or two lines to be changed in your code.

[MrDiaz]

Hi james, great response. I will try to explain a bit better what my app does. When I mentioned a log system, wasn't necessarily a log systems for users or similar. This is what my app does.

Is a program that stores daily information, on what student pilots do. Every time they finish the day, they need to record the activities they did, how many hours they spent flying, did they use a simulator? Was it solo flying? Dual flying? And so on, there are a few variables. My application does indeed uses a GUI, filled with text fields and spinners.

setValues() takes all the data from the controls on my UI and sets these values to an Object "Day". So each day has the same properties (hours, landings, solo flying, etc.)

I think I'm going to use the Java DB. Quick question, how easy would it be to deploy my application on other systems if I were to use the JavaDB? Would it need third party apps for it to work? How exactly would that work

[BestJewSinceJC]

setValues() takes all the data from the controls on my UI and sets these values to an Object "Day". So each day has the same properties (hours, landings, solo flying, etc.)

I think I'm going to use the Java DB. Quick question, how easy would it be to deploy my application on other systems if I were to use the JavaDB? Would it need third party apps for it to work? How exactly would that work

I don't think your explanation of setValues is clear. Does setValues get each field from the GUI, then construct an Object called Day based on those values? If so, setValues seems like a strange method name, but if not, I don't get it. As for your second thing, I'll let someone more knowledgeable about it answer, but it is definitely possible. You could basically have an app from any computer that knows the location of the database and has access to it update and read from the database, as far as I know.

[MrDiaz]

yes setValues() gets the values from the GUI fields and sets the values for a Day object.

As to deploy the app to other systems, I would have to move the database to that system as well? Don't really care of it goes empty, just need to make sure the database is created

[MrDiaz]

I would post my code but there are quite a few thousand lines. Is there a better place to upload the code to?

[BestJewSinceJC]

Oh, I thought you were talking about updating the same database from multiple different places. Which has got to be possible via the internets. And I do not think that posting your entire program will be necessary.
:)

[MrDiaz]

Yeah that is not really necessary. I just want to be able to run the program on any machine I deploy it too.

[JamesCherrill]

If I understand right, your data volume looks like max 1 record per user per day, so there's no need for heavyweight database technogy. Java DB will probably be perfect here.
Deployment of the database comes in 2 main parts - installing the db software (trivial with Java DB, depends on licences etc for other databases), - and installing your empty/startup database, which mainly comes down to where to put it. System.getProperty("user.home" is a good start.
=======================================
Re software design. I don't mean this in any rude way, but your design, with the file IO calling the GUI, could certainly be improved. If you're interested in using this as a learning vehicle, and you're ready to take a quantum leap in your architectural skills, I'd be glad to help. If you have other priorities right now, that's OK too.
J.

[MrDiaz]

Sure thing James, go ahead and tear me apart. I'm always up for learning, heck is why I'm doing this in the first place! I'd really appreciate it if you're willing to teach me

[JamesCherrill]

Hi MrDiaz - going to dinner now, but I'll get started tomorrow. I'll be 100% constructive!

[JamesCherrill]

Good morning.
Right now you have what is called "spaghetti code". Everything potentially can be affected by everything else. Which means any change can break any other part of the code (eg a change in setValues - how the GUI stores its values into a Day - could break the code for saving values to file). Also, if you want to re-use any existing parts to make an extension of the app, you have to take the whole thing, because it's all tied together.

So the goal is to break the app into a number of parts, each of which has very clearly defined responsibilities, and the smallest simplest possible interface to any other part. We'll also try to structure those parts so that dependencies between them are as understandable as possible, and so that no knowledge of the internals of any part is needed to use it from another part.
A "part" could be a class or a small collection of classes. How comfortable are you with setting up packages for your code? Provided that's ok, we can make good use of them.

Finally, a quick start on scope & requirements: I've seen one Use Case so far: user enters a Day of his hours etc [computer validates them?] , computer stores them.
Or is it: operator enters hours for a number of users... ?
What about the other Use Cases? Are there cases like: user reviews hours for last week/month? User requests total hours for last year? User needs to correct errors on previous Day's figures? Instructor requests statictical analysis of multiple user's hours? etc - you get the idea.

[MrDiaz]

I haven't set up many use cases yet, I wanted to sort out the database issue first before moving onto anything else. As to what you mentioned previously, the way I have it set up right now allows me to modify the app whenever I want to without changing any code. You see, I have a class called Day. This class stores all the daily information found on the components. I have an ArrayList<Day> that stores all my days. So to store all these values into a text files, I just use a for: each iterate through the array and each Day object inside my arrayList is just a new line on the text file.

[JamesCherrill]

OK man, that's cool.