Java Stock Trend Analysis Program :)

[joshSCH]

Alright guys.. I'm back in the java forums for help/suggestions. I'm sure one of the Daniweb java programming geniuses will be able to decide if this idea is too difficult or needs to be revised.

I was thinking of writing a stock trend analysis program. There will be a stock object that contains an array of 10 doubles, each of which contain a dollar amount for the value of a specific stock. Every 6 seconds, a double is recorded from a quote streaming program provided by a broker. After all 10 doubles have been entered into the array (1 minute later), the program will evaluate the current angle of the slope, the curve's concavity, the integrated area of the curve between position 0 of the array and position 9. Based on this information, the stock is then sorted by whether or not it is a priority*, then by the angle of the slope.

Each object stock will contain:
int open- value at which the specific stock opened at
int volume- volume of the current traded shares of the day
int integrate- value of the integral from array position 0 to position 9
int slope- slope of the curve measured between positions 8 and 9 of the array.. this will be an angle between -90 degrees and 90 degrees.

String symbol- that specific stock's symbol
boolean priority- evaluates to true if the stock's concavity has just shifted, the value of position 9 in the array is less than the opening stock value, and if the slope is rising/falling depending on concavity.

On execution, the program will immediately retrieve information regarding every stock, (open, symbol), and saves every stock to an arraylist of stock objects

Next, the program will output everything to a graphical interface (probably a table)

Finally, the program will begin retrieving stock data from the broker, and update the table every minute.

What do yall think? I could easily write everything.. except the streamreader that communicates with the broker.. I have yet to discover how to do this.. The brokerage firm I use actually uses java for its instant quote streamer data, so hopefully that will make things less complicated.. but if you have any suggestions, know how to write a streamreader, or know of a program like this that has already been created then please post :)

[peter_budo]

>Next, the program will output everything to a graphical interface (probably a table)
would be nicer if you can give outout in form of graph with something like JGraph, maybe??? If so, you would like to keep your calculated values in DB (in case let say, in middle of the day system crash, you will only loose data which been currently processed and data send to you while recovering from crash)
I think I can be good exercise

[joshSCH]

ah good advice.. I wasn't even thinking about what would happen in case of a crash. Yea, I'll have to see what swing component would work best..

I really have no clue how to communicate with my broker's stock streamer applet..

[joshSCH]

The Scottrader Streaming Quotes applet connects via socket (ports 80 and 443).

How can I communicate with this applet?

[~s.o.s~]

Maybe these links would give you an idea.

[joshSCH]

thanks.. but from what I've read, I would have to know specifics about the program to invoke specific methods and such.. and I seriously doubt they will let me have a look at the source :(

..but, I believe there is a way via secure http that I can access the information that I want.. I am currently looking into this, and it seems relatively easy..

if anyone finds out a way in which one can communicate with a java program w/out directly knowing its source code and methods, then please inform me. Thanks :)

[jwenting]

I could tell you a lot about that, but can't because of contractual obligations.
My work involves (among other things) maintaining a piece of software that generates analysis charts from streaming stockmarket information...

We use our own systems for capturing (which I don't maintain), filtering/retrieving from the database/stream (which I help maintain), and presenting (which I do maintain) the data.

It's far from a trivial task to do all that. Our software is maintained by a team of some 25 people fulltime, plus another dozen or so to maintain the database and hardware.
And that doesn't include the people sending us the information (stock exchanges, banks, investment firms, etc.).

We use JFreeChart (on the Java side, we also have .NET and C++ software) for charting, with some ten thousand lines of custom code to generate the data to be charted and customise the charts every which way.

I do hope you define "every stock" as "every stock from a list we're interested in"?
We DO have every stock (from the exchanges we capture, something like a hundred of them I think) and there's tens of thousands of the buggers.
Add options, futures, warrants, bonds, CDS, CDX, currency futures, etc. etc. and we are now working with roughly 2 million instruments.

The only way to talk to a Java program (or any program for that matter) is to know its public API (if it has one) and how to talk to it (RMI, SOAP, COM, etc. etc.).
For this kind of information that's almost certainly quite well hidden, and the public interface you see in that applet won't have any means to talk to it from software. It will in fact itself talk to another piece of software on their servers to retrieve the information over a secure channel, probably RMI or SOAP.

To give you an idea of the cost of that data:
We market our data at an initial price of several hundred dollars per workstation per year plus a flat fee of I think $1 per instrument per workstation per month.
Things are sold in packages (an instrument type for an exchange for example).
The money involved is tremendous. Deals can run into the hundreds of thousands of dollars, even millions of dollars, per year.

If you work for a company that has a business need to get the data, you'll be able to get budget to get the data legally.
If not you're likely going to have to resort to some simulated system instead, which would be good to have anyway for testing purposes so you can have predictable data for testing.

If you're serious about getting such data and can get the budget I may be able to bring you into contact with some of our account managers. But only if you're dead serious please, this business is no joke and me (and them) are busy enough as it is.

[joshSCH]

I could tell you a lot about that, but can't because of contractual obligations.
My work involves (among other things) maintaining a piece of software that generates analysis charts from streaming stockmarket information...

We use our own systems for capturing (which I don't maintain), filtering/retrieving from the database/stream (which I help maintain), and presenting (which I do maintain) the data.

It's far from a trivial task to do all that. Our software is maintained by a team of some 25 people fulltime, plus another dozen or so to maintain the database and hardware.
And that doesn't include the people sending us the information (stock exchanges, banks, investment firms, etc.).

We use JFreeChart (on the Java side, we also have .NET and C++ software) for charting, with some ten thousand lines of custom code to generate the data to be charted and customise the charts every which way.

I do hope you define "every stock" as "every stock from a list we're interested in"?
We DO have every stock (from the exchanges we capture, something like a hundred of them I think) and there's tens of thousands of the buggers.
Add options, futures, warrants, bonds, CDS, CDX, currency futures, etc. etc. and we are now working with roughly 2 million instruments.

The only way to talk to a Java program (or any program for that matter) is to know its public API (if it has one) and how to talk to it (RMI, SOAP, COM, etc. etc.).
For this kind of information that's almost certainly quite well hidden, and the public interface you see in that applet won't have any means to talk to it from software. It will in fact itself talk to another piece of software on their servers to retrieve the information over a secure channel, probably RMI or SOAP.

To give you an idea of the cost of that data:
We market our data at an initial price of several hundred dollars per workstation per year plus a flat fee of I think $1 per instrument per workstation per month.
Things are sold in packages (an instrument type for an exchange for example).
The money involved is tremendous. Deals can run into the hundreds of thousands of dollars, even millions of dollars, per year.

If you work for a company that has a business need to get the data, you'll be able to get budget to get the data legally.
If not you're likely going to have to resort to some simulated system instead, which would be good to have anyway for testing purposes so you can have predictable data for testing.

If you're serious about getting such data and can get the budget I may be able to bring you into contact with some of our account managers. But only if you're dead serious please, this business is no joke and me (and them) are busy enough as it is.

Wow, I do appreciate all that information and time you put into the post.

By "every stock", I meant every single one that is traded daily (even the pink sheets). I planned on having a txt file with all the stock symbols on it, where a program would read them all in... however, if there are that many then perhaps I should limit myself.. I have no clue of how much ram and processor resources that this program will need..

hmm.. I found the URL that executes the java applets (there are actually 3-4 programs that open up simultaneously), and I found this code in the html:

<APPLET height=1 archive=scottraderv4_2_2.jar width=1 code="MainClassApplet.class">

So, I presume the java main class is MainClassApplet.class. Also, the applet connects via socket. Is there not a way to somehow communicate with this applet? If not, is there a way in which I can manipulate and communicate with the server? Since the app does use socket, perhaps there is a way I can ask the server for info myself..

And about your proposal, I'm simply some teen who is just 'discovering' the stock market.. so I don't wish to waste your time or your colleague's time either.. I just thought this program would help my trading strategies a bit.. but, if someone did want to obtain your software, about how much would it cost for the simplest form?

[~s.o.s~]

If this is just a simple and plain educational java project, I would suggest you not to think a lot about it. After all, the aim here is to learn and not to make a professional grade software. I assume you wouldn't have that kind of money to throw around.

Create a dummy 'stock broker' database which would be used pull out data. I am pretty sure the people who are to see this project don't expect you to connect to a real 'Stock broker' when you explain to them the constraints faced.

Other than that, its your call in the end.

[jwenting]

best would indeed be to make your own dummy dataset and present that.
If you feel interested you can make some sort of generator even that creates new data based on some trend information and previous values as a simulator, so it appears to have life data.

By having that pretty much external to your main system you can have it so that you could relatively easily create a connector to have your system talk to real datasources instead if and when you get access to them.

And no, you wouldn't have the money it takes as a student.
Even a small contract to supply once daily updates for a few hundred instruments could set you back $10K.

[joshSCH]

best would indeed be to make your own dummy dataset and present that.
If you feel interested you can make some sort of generator even that creates new data based on some trend information and previous values as a simulator, so it appears to have life data.

By having that pretty much external to your main system you can have it so that you could relatively easily create a connector to have your system talk to real datasources instead if and when you get access to them.

eh, yea.. I think I can easily communicate with a url and obtain the same streaming quotes anyway.. I basically need to know the same technique as that 'autoposter' program that I talked about in another thread.. I'll check it out, and see what I can do. Thanks.

And no, you wouldn't have the money it takes as a student.
Even a small contract to supply once daily updates for a few hundred instruments could set you back $10K.

whoa.. Do your clients generally think the software helps them a lot? What exactly do most of your clients use it for? jw

[jwenting]

Our clients are major investment banks using the data to power the terminals used by their traders, the websites showing their clients data, etc. etc.

For them the cost is peanuts (and it's a fraction of what our major competitor charge for similar services, IF those competitors have the data (some of it is unique to us)).
A single trader could make more money in a day for them than the entire cost of the data for a year.

So we don't deal (directly at least) with the smalltime investor working on his bedroom PC. We deal with the BIG traders like JP Morgan, banks like HSBC, guys like that (not necessarilly them, but you get the idea).
If a company trades for billions a day (with revenue in tens of millions a day) a few thousand a month for a data license isn't going to bother them.

[joshSCH]

Hmm.. Well, I don't want to use 'dummy' information as the entire purpose of this program is to provide me with a day trading tool.

I have found another way to obtain quote data besides the java applet.. All you have to do is type in the symbol on their website, and they display information about the stock using javascript (I think). I haven't tested if these are accurate quotes, but I will tomorrow.

So, any1 have advice regarding this.. ?

[jwenting]

you can try sending and receiving http requests and parsing the results.
It's possible, but messy.

Or try to contract them to provide you with the data through some other means (like SOAP).

[joshSCH]

Okay, the quotes online via their website are also streaming so I don't actually need to communicate with an applet..

Yea, I've tried contacting them by e-mail, but to no reply :-/

The applet first attempts a socket connection, but an http connection is the secondary option..

I've tried using URLConnection, but the stream reader cannot input the data I'm looking for because 1, it uses javascript or somethn to cmmunicate with the server.. and 2, I cannot manage to log into my account using secure http..

Jwenting, does your software get information from the same server? Or can your customers choose which brokerage firm to go through, and your company is able to retrieve information dynamically, from anyone? Basically, just wondering the best way to retrieve my data.. I'm a bit confused as to how these web pages function.. I know some html, but I don't know much about scripts within the html.. When I view the source of the page, it looks like it executes javascript or somethn.. :?:

[jwenting]

the site if almost certainly using Ajax to send XML data over http and get XML back.
Run the browser through some sort of package sniffer to get the request and response structure and you may be able to have your own software send and receive the same thing (but were I the guy programming that site I'd defend against such things, using stuff like encryption, sessions, etc. etc.).

[joshSCH]

Oh, sorry.. I accidentally posted before I finished.. had to edit.

[Ezzaral]

If you are using Firefox (which of course you SHOULD be!! :) ), you might find this add-on useful in picking through how a site is doing things:
Web Developer : https://addons.mozilla.org/en-US/firefox/addon/60

It adds a toolbar that lets you inspect all kinds of things going on in a page.

[joshSCH]

Thanks much Ezzaral for that plugin :)

Excuse me if this post appears idiotic and/or noobish (As I'm sure it will)

Okay, first of all.. What packet sniffer would you recommend I use?

Does the attached link have any useful information at all?

I think I found the page that opens up specific data regarding each symbol:
https://trading.scottrade.com/quotesresearch/ScottradeResearch.aspx?symbol=HDY

.. and a few other interesting links:
https://trading.scottrade.com/home/Default.aspx?SORT=SYMBOL&ORDER=ASC&LastCOL=
javascript:__doPostBack('WatchLists1$watchGrid$_ctl1$_ctl1','')

This doesn't really mean much to me, as I have only basic knowledge of html, and only know of the java programming language. However, these appear to be important links to the server..

[Ezzaral]

Since you have an address that accepts the symbol as a request parameter, it should be trivial to scrape the quote from the HTML response. Take a look at this article I ran across in a quick search for "stock quote web service". The article is using C# but given the similarity to Java you shouldn't have any trouble with it:
http://dotnetjunkies.com/Article/A3E8CA89-9AA1-4C3B-BCC6-901C92A523E9.dcik

[Ezzaral]

I was playing around a bit scraping quotes from Yahoo and found a URL that will give you the quote as CSV text. You might see if this bit of code below helps you out:

try {
    // Create a URL for the desired symbol
    String symbol = "FFIDX";
    URL url = new URL("http://download.finance.yahoo.com/d/quotes.csv?s="+symbol+"&f=sl1d1t1c1ohgv&e=.csv");
    // Read all the text returned by the server
    BufferedReader in = new BufferedReader(new InputStreamReader(url.openStream()));
    String inputLine;
    StringBuilder doc = new StringBuilder();
    while ((inputLine = in.readLine()) != null){
        doc.append(inputLine);
    }
    if (doc!=null){
        String[] parsed = doc.toString().split(",");
        System.out.println("Symbol: "+parsed[0]);
        System.out.println("Price: "+parsed[1]);
    }

    in.close();
} catch (MalformedURLException e) {
} catch (IOException e) {
}

[joshSCH]

Wow.. yea, I was working with approximately that same code trying to view the data.. However, you managed to find a csv, and easily extract the same data I've been searching for :)

If only yahoo had streaming quotes... :-/

I'll see if I can manage to follow your example using my broker's website..

[joshSCH]

hmm.. I just ran across JScrape, a java screen scraping API. Perhaps I can use this to read in the information! :)

[joshSCH]

Ah, hell..

Anyone know how to connect to a https website? I've got a username and password, but I have not a clue in the world as to how to code for this.. I've checked online, but to no avail :(

Also, on that JScrape:http://www.apsquared.net/JScrape.html

The program doesn't work for me.. I think there is a problem with the custom libraries.. how do I load them so I can import them properly?

[Ezzaral]

Ah, hell..

Anyone know how to connect to a https website? I've got a username and password, but I have not a clue in the world as to how to code for this.. I've checked online, but to no avail :(

This is a short example I found over in another forum (I've never used HTTPS myself):

import java.net.URL;
import java.io.*;
import javax.net.ssl.HttpsURLConnection;

public class Test
{
    public static void main(String[] args) throws Exception
    {
        String httpsURL = "https://your.https.url.here/";
        URL myurl = new URL(httpsURL);
        HttpsURLConnection con = (HttpsURLConnection)myurl.openConnection();
        InputStream ins = con.getInputStream();
        InputStreamReader isr=new InputStreamReader(ins);
        BufferedReader in =new BufferedReader(isr);

        String inputLine;

        while ((inputLine = in.readLine()) != null)
            System.out.println(inputLine);

        in.close();
    }
}

Also, on that JScrape:http://www.apsquared.net/JScrape.html

The program doesn't work for me.. I think there is a problem with the custom libraries.. how do I load them so I can import them properly?

I assume you downloaded the "Lite" version? All of the libraries they mention are freely available, however on the site there is a "JScrape Full" link to download a version that includes all of the third party libs.

[joshSCH]

This is a short example I found over in another forum (I've never used HTTPS myself):

import java.net.URL;
import java.io.*;
import javax.net.ssl.HttpsURLConnection;

public class Test
{
    public static void main(String[] args) throws Exception
    {
        String httpsURL = "https://your.https.url.here/";
        URL myurl = new URL(httpsURL);
        HttpsURLConnection con = (HttpsURLConnection)myurl.openConnection();
        InputStream ins = con.getInputStream();
        InputStreamReader isr=new InputStreamReader(ins);
        BufferedReader in =new BufferedReader(isr);

        String inputLine;

        while ((inputLine = in.readLine()) != null)
            System.out.println(inputLine);

        in.close();
    }
}

Hmm.. but where do I enter my username/password?

I assume you downloaded the "Lite" version? All of the libraries they mention are freely available, however on the site there is a "JScrape Full" link to download a version that includes all of the third party libs.

No, I downloaded the full version.. the YahooStock.java program still doesn't execute. Even with the libraries that were installed.. I think there is a problem with it finding the libraries..

[lucky1981_iway]

As per my understanding there are 2 types of SSL connection

1) One-Way SSL Authentication URL Outbound SSL Connection
here server is responsible for accepting your connection.(it's policies are set to accept connection from anoymous client)

2) Two-Way SSL Authentication URL Outbound SSL Connection
In this a Java client performs these minimum functions:

• Initializes an SSLContext with client identity, a HostnameVerifierJSSE, a TrustManagerJSSE, and a HandshakeCompletedListener.
• Creates a keystore and retrieves the private key and certificate chain
• Uses an SSLSocketFactory
• Uses HTTPS to connect to a JSP served by an instance of Server

So i don't think you need any user name and password to connect if it's one way ssl and server is opened for connection.

[Ezzaral]

Hmm.. but where do I enter my username/password?

You will probably have to send those as a POST request and store the cookie or session ID that is returned by the server. Take a look at the page source to see if they are using a form for login and what parameters are being sent. I'm just speculating here, as I've never played around with http and web requests all that much.

As far as the ssl connection, see the following Sun info on Secure Socket Extension:
http://java.sun.com/j2se/1.4.2/docs/guide/security/jsse/JSSERefGuide.html

No, I downloaded the full version.. the YahooStock.java program still doesn't execute. Even with the libraries that were installed.. I think there is a problem with it finding the libraries..

What editor are you using? You will need to add them to the project classpath and how to do so is specific to the editor.

[joshSCH]

I use JCreator.

And sorry if I confused anyone..
But, I need to connect to my broker's website via https before I can access the applet. So, first I need to write a program that will connect, and sign me in my account. Then I need one that will communicate with the applet. (The applet uses socks port btw).

[joshSCH]

Okay, I have scrapped they idea of an applet to applet communication process as the code is to arduous to go through.

However, I can achieve the same results from logging in, and simply using the web server. I have acquired JScraper to screen scrape the data for me, and I have finally figured out how to make it work in JCreator. Okay, so all is good with that.. but, I do not know how to use it specifically with Scottrade.

Like Ezzaral said, I need to figure out the POST parameters to login, then code it into java.. I think I may know the parameters, but you html gurus may want to double check me:
https://trading.scottrade.com/default.aspx

I am searching for ways to connect to the https server and log in.. but if anyone can help me, it would be greatly appreciated.