Member Avatar for naief

Hi all

I need to put a program on the server that frequently get data from the web and update the database with these data and this program should do this automatically.

Can anyone put me on the Beginning of the road. I read in some websites about servlet context listener ... Is it what I need and if so how can I make it works on the server automatically.

regards

  • 3 Contributors
  • 3 Replies
  • 114 Views
  • 19 Hours Discussion Span
  • Latest Post Latest Post by naief

Dani AI

Generated

Two practical, proven ways to run a periodic fetch-and-update job for a weather XML feed and MySQL backend (the scenario described):

  • Run a small standalone JVM process on the server and schedule it with the OS (cron or systemd) or a cloud scheduler. This keeps the job independent of your webapp lifecycle and is easiest to reason about.
  • Or run a scheduler inside the webapp (started at deploy time and stopped on undeploy). That is what people often mean when they mention a lifecycle listener; it works, but needs careful lifecycle and thread handling to avoid leaks.

Implementation checklist and best practices (build once, run forever)

  • Fetch robustly: set connect/read timeouts, retry with exponential backoff, and respect remote rate limits. Prefer streaming XML parsing for feeds (use StAX for low memory use) (StAX tutorial).
  • Persist safely: use a connection pool (eg. HikariCP) and make writes idempotent — unique constraints, upserts, or last-modified timestamps prevent duplicates and race conditions (HikariCP).
  • Scheduling choices: simple OS cron is reliable for many cases (example: */15 * * * * /usr/bin/java -jar /opt/fetcher/fetcher.jar) and easy to monitor (crontab docs). For in-process scheduling use the JDK scheduler or a full scheduler like Quartz if you need persistence, clustering, or advanced calendars (ScheduledExecutorService, Quartz).
  • On where to poll: ’s file-polling idea is valid. If the provider supports push/webhooks, prefer that. Otherwise a polling job that marks processed items (or relies on DB uniqueness) works well.

Operational tips

  • Log each run, record last-success time, alert on repeated failures, and test with short intervals before production. If you run inside a servlet container, follow proper lifecycle shutdown to avoid orphan threads (that’s what was getting at).

Recommended Answers

All 3 Replies

Member Avatar for ndeniche

I once did this for a vote counting project on national elections. The users would send a SMS with a pre-established string format to a certain phone number, and a text file would be downloaded from the phone company owning the phone number receiving the SMSs to my server. The server would have a running program constantly processing the text file checking for new data entry, in which case it would update the database.

In your case, if I'm not mistaken, it's a similar case: you get data from the web, maybe it could update a text file or a separate database (which could contain some kind of sentinel variable, such as "checked" or something in a column of the table) which is constantly checked by the program running on the server. This program would then process the "unchecked" data in the file or database and add it along with the variables you need to insert to the database.

Hope this is of any help.

Member Avatar for stultuske

do you know what threads are?

Member Avatar for naief

Thank you guys for your response ..

Nichito you are right. I'm going to parse an xml of a weather web-site and get the the information like temperatures etc.. and update my database (mySQL in my case) so that I can analyze the temperatures and construct graphs ..etc

Stultuske do you mean by Thread the JAVA class that implements interface? or you mean threads in server context.
If you mean the Java class please direct me to the way that I can benefit from it in my case.

again thank you all

Be a part of the DaniWeb community

We're a friendly, industry-focused community of developers, IT pros, digital marketers, and technology enthusiasts meeting, networking, learning, and sharing knowledge.