Parsing html form.

[croft]

Let me tell you this from the start =)

We play a game called eve.
In that game wheneve you kill someone you will recive an email wich i will post belove here for an idea how it looks like.

This game alsoo has an ingame browser html 1.4 compliant. not the biggest dream but it works =)

I want to create a form. only 1 form where you paste the entire content of the 'kill mail' and hit a submit button. That will be placed in mysql database. From there i want to be able to show the latest kills. top killer. top looser and so on on a webpage.

Can anyone help me get this thing going ?
Im pretty sure i can get the info from the database when its in there but i seam to have problems getting the information in to the database.

Here is 1 basic 'kill mail' i recived a couple of days agoo.
Would be much apriciated if someone could assist me with this.

2005.06.04 03:35:00

Victim: name of killed person here
Alliance: Unknown
Corporation: Storm Guard Elite
Destroyed Type: Kestrel
Solar System: H74-B0
System Security Level: 0.0

Involved parties:

    Name: name of killer here (laid the final blow)
    Security Status: 5.0
    Alliance: Firmus Ixion
    Corporation: Corp name here
    Ship Type: Megathron
    Weapon Type: 425mm Railgun II



Destroyed items:

    Type: Standard Missile Launcher I (Fitted - High slot)

    Type: Standard Missile Launcher I (Fitted - High slot)

    Type: Warp Disruptor I (Fitted - Medium slot)

    Type: Standard Missile Launcher I (Fitted - High slot)

    Type: Sabretooth Light Missile I (Cargo)
    Quantity: 100

    Type: Bloodclaw Light Missile I (Cargo)
    Quantity: 50

    Type: Bloodclaw Light Missile I (Cargo)
    Quantity: 50

    Type: Sensor Booster I (Fitted - Medium slot)

    Type: Barton Reactor Capacitor Recharger I (Cargo)
    Quantity: 1

    Type: Medium Automated Structural Restoration (Cargo)
    Quantity: 1

    Type: Flameburst Light Missile I (Fitted - High slot)
    Quantity: 36

    Type: Bloodclaw Light Missile I (Fitted - High slot)
    Quantity: 36

[Troy]

So you want a process that will parse these emails to get specific bits of data, store this data in a database, then have an interface to display the data in a meaningful format?

If what I describe above is what you want, then you'd go about it by creating a PHP script that will use string manipulating and maybe regex (regular expressions) to parse the text of the email--breaking out all the individual data pieces you want. Then you build a sql statement to insert the data into your table. Finally, you build a script that selects the data and displays the summary data however you want.

The question now is, where do you need help? Are you a PHP programmer? Do you know how to connect to a mysql database in PHP? Are you familiar with using basic string manipulation functions in any programming language?

Also, do you have any control over the program that originally sends out these emails? Can you have that system send every email you care about to a special email address, you can have your server automatically check that email account to get the data. This way you would not have to copy & paste the email text and submit it to your script. If this is a controlled group of friends you are talking about, then you could have them all setup auto-forwarding rules to automatically forward their game emails to this special email account. It's an idea anyway.

[croft]

So you want a process that will parse these emails to get specific bits of data, store this data in a database, then have an interface to display the data in a meaningful format?

This is exactly what i want =)

I am not a programmer. I know some stuff but very basic things. I have never written a php script myself. I do know how to connect to a database. And the database is no problem for me. The problem is in the coding of the parse script. I have no idea.

I have no control what so ever over the email program.
If anyone can help out it would be greatly apriciated =)

[Troy]

Who loves you, croft? That's right, Troy does! Here is complete, working, tested code to parse your email body into the various data pieces. Just create the two files below, and place on a PHP enabled web server. Then surf to the eve_submit.htm web page.

NOTE: Text parsing is a tricky business. One small change in the text format may break your parsing script.

Create this HTML page: eve_submit.htm

<html>
  <head>
  <title></title>
  </head>
  <body>
  
  <form method="POST" action="eve_parse.php">
  <textarea name="email_body" cols="80" rows="20"></textarea>
  <br /><input type="submit" />
  </form>
  
  </body>
</html>

Create this PHP script: eve_parse.php

<?php

/*
Put the text that was passed in via HTTP into a variable.
*/
$email_body = $_REQUEST['email_body'];

/*
Create an array to hold the key -> value pairs.
*/
$data = array();

/* Create an array where each line of text is in an array element. */
$lines = split("\n", $email_body);

/*
Since the timestamp is the very first line of the text, go ahead and put this
into a variable.
 */
$data['timestamp'] = trim($lines[0]);

/*
Now iterate through the lines breaking each into it's key->value set.
Do this until we reach the "Destroyed items" section.
*/
for ($line_idx=1; $line_idx<count($lines); $line_idx++) {
  $line = $lines[$line_idx];
  $pair = split(":", trim($line));
  $pair[0] = trim($pair[0]);
  $pair[1] = trim($pair[1]);
  if ($pair[0] == "Destroyed items") { break; }
  if ((strlen($pair[0]) > 0) && (strlen($pair[1]) > 0)) {
    $data[$pair[0]] = $pair[1];
  }
} 

/*
Now iterate through the destroyed items.
*/
$data['destroyed items'] = array();
for ($line_idx=$line_idx+1; $line_idx<count($lines); $line_idx++) {
  $line = $lines[$line_idx];
  $pair = split(":", trim($line));
  $pair[0] = trim($pair[0]);
  $pair[1] = trim($pair[1]);
  if ((strlen($pair[0]) > 0) && (strlen($pair[1]) > 0)) {
    if ($pair[0] == "Type") {
      $item = array();
      $item[] = $pair[1];

      $line_idx++;
      $line = $lines[$line_idx];
      $pair = split(":", trim($line));
      $pair[0] = trim($pair[0]);
      $pair[1] = trim($pair[1]);
      if ((strlen($pair[0]) > 0) && (strlen($pair[1]) > 0)) {
        $item[] = $pair[1];
      } else {
        $item[] = 1;
      }
      $data['destroyed items'][] = $item;
    }
  } 
} 

/*
Dump the data array contents to the browser for testing purposes.
*/
echo "<pre>";
print_r($data);
echo "</pre>";

/*
You can directly access any item within the data array like so:
echo $data['Victim'];

To grab the values for the fourth destroyed item:
echo "Item #4 Name: ".$data['destroyed_items'][3][0];
echo "Item #4 Qty: ".$data['destroyed_items'][3][1];
(Remember, PHP arrays are zero-indexed, so counting starts at zero.)

Now you can insert whatever you want into your database.
*/
?>

The program, when passed the email example you posted, outputs:

Array
(
    [timestamp] => 2005.06.04 03:35:00
    [Victim] => name of killed person here
    [Alliance] => Firmus Ixion
    [Corporation] => Corp name here
    [Destroyed Type] => Kestrel
    [Solar System] => H74-B0
    [System Security Level] => 0.0
    [Name] => name of killer here (laid the final blow)
    [Security Status] => 5.0
    [Ship Type] => Megathron
    [Weapon Type] => 425mm Railgun II
    [destroyed items] => Array
        (
            [0] => Array
                (
                    [0] => Standard Missile Launcher I (Fitted - High slot)
                    [1] => 1
                )

            [1] => Array
                (
                    [0] => Standard Missile Launcher I (Fitted - High slot)
                    [1] => 1
                )

            [2] => Array
                (
                    [0] => Warp Disruptor I (Fitted - Medium slot)
                    [1] => 1
                )

            [3] => Array
                (
                    [0] => Standard Missile Launcher I (Fitted - High slot)
                    [1] => 1
                )

            [4] => Array
                (
                    [0] => Sabretooth Light Missile I (Cargo)
                    [1] => 100
                )

            [5] => Array
                (
                    [0] => Bloodclaw Light Missile I (Cargo)
                    [1] => 50
                )

            [6] => Array
                (
                    [0] => Bloodclaw Light Missile I (Cargo)
                    [1] => 50
                )

            [7] => Array
                (
                    [0] => Sensor Booster I (Fitted - Medium slot)
                    [1] => 1
                )

            [8] => Array
                (
                    [0] => Barton Reactor Capacitor Recharger I (Cargo)
                    [1] => 1
                )

            [9] => Array
                (
                    [0] => Medium Automated Structural Restoration (Cargo)
                    [1] => 1
                )

            [10] => Array
                (
                    [0] => Flameburst Light Missile I (Fitted - High slot)
                    [1] => 36
                )

            [11] => Array
                (
                    [0] => Bloodclaw Light Missile I (Fitted - High slot)
                    [1] => 36
                )

        )

)

[croft]

Thats great.
There is a couple of problems tho.

Array
(
[timestamp] => 2005.06.04 03:35:00
[Victim] => name of killed person here
[Alliance] => Firmus Ixion
[Corporation] => Corp name here
[Destroyed Type] => Kestrel
[Solar System] => H74-B0
[System Security Level] => 0.0
[Name] => name of killer here (laid the final blow)
[Security Status] => 5.0
[Ship Type] => Megathron
[Weapon Type] => 425mm Railgun II
[destroyed items] => Array

The victims alliance turns up as firmus ixions when the killer alliance is firmus ixion. The corp name for the killer and killed. The allliance for killer and killed.

Also. This wont add anything to a database unless you do it manually ? Or am i wrong here. Like me the players have no clue about this stuff and or how to add these things in to the database.

[Troy]

Ah, yes. I missed that there is in fact a field that is duplicated between the "Victim" and "Involved Parties" sections. So you'd have to manipulate the code that breaks out those sections similar to how I broke out the destroyed items.

No, the code I provided does the retrieval and parsing. It does not insert the data into a database. I thought you only needed help with the parsing part because you said:

I do know how to connect to a database. And the database is no problem for me.

Here is the code example I give PHP newbies to learn how to work with a database.

<?php

$server = "localhost"; // Name or IP of database server.
$user   = ""; // username
$pwd    = ""; // password
$db     = ""; // Name of database to connect to.

if (!$cnn = mysql_connect($server,$user,$pwd )) {
  die("mysql_connect() failed");
}

if (!mysql_select_db($db,$cnn)) {
  die("Could not select database named ".$db);
}

/* Build your SQL statement however you need. */
$sql = "select * from mytable";

/* Execute the query. */
if (!$res = @mysql_query($sql)) {
  die(mysql_error());
  return false;
}

/*
If the SQL statement above was an INSERT, UPDATE, or DELETE,
you'd be done now. If it was a SELECT, then this next part shows
you how to work with the recordset.
*/

/* Create an array of arrays out of the recordset. */
while ($row = mysql_fetch_array($res, MYSQL_ASSOC)) {
  $data[] = $row;
}

/* Now iterate through the recordset creating a simple table. */
echo "<style>table.dump { font-family:Arial; font-size:8pt; }</style>";
echo "<table class=\"dump\" border=\"1\" cellpadding=\"1\" cellspacing=\"0\">\n";
echo "<tr>";
echo "<th>#</th>";
foreach($data[0] as $key=>$val) {
  echo "<th><b>";
  echo $key;
  echo "</b></th>";
}
echo "</tr>\n";
$row_cnt = 0;
foreach($data as $row) {
  $row_cnt++;
  echo "<tr align='center'>";
  echo "<td>".$row_cnt."</td>";
  foreach($row as $val) {
    echo "<td>";
    echo $val;
    echo "</td>";
  }
  echo"</tr>\n";
}
echo "</table>\n";    

?>

So now, croft, you have most of the code you need to do your project. You'll have to tweak and put the pieces all together, but almost everything you need is in these two code examples.

If you have additional specific questions regarding this project, post them here and I or somebody else will try to help. If you just want to pay someone to code the project for you, there are plenty of PHP gurus who'd be happy to code this according to your specs. Just let me know.

[croft]

Its fine =)

I went out and bought me a book today. php and mysql bible. Have no idea if its any good but it was the only one i could find.
Now im gonna dive in it and check stuff out.

[wds]

This is interesting and related to something I'm looking for. I want to be able to easily parse data from an HTML file. I have seen several classes people made in PHP and Java, but I was wondering if anyone did this already and could recommend a method? Thanks! Wes

[Troy]

...I have seen several classes people made in PHP and Java, but I was wondering if anyone did this already and could recommend a method? Thanks! Wes

The classes you've already seen are the recommended methods from the people who already did this.

You haven't asked a specific question. You've asked about a general technique, in which case, my code post above is a big part of your answer.

Let us know how we can help you in your journey. :cool:

[croft]

This is the code i ended up with at the end.
Thanks for the help.

<?php

//killmail lines
$lines = split("\n", $email_body);

//get the timestamp
$timestamp = $lines[0];

/*Outline the plan....lets create three arrays:

	Victim array:
	* $victim[Victim]
	* $victim[Alliance] (this will be null/blank if it doesn't exist)
	* $victim[Corporation]
	* $victim[Destroyed Type]
	* $victim[Solar System]
	* $victim[Security Level]
	
	For each element (n) in the $involved array above you are dealing with an attacker
	* $involved[n][Name]
	* $involved[n][Security Status]
	* $involved[n][Alliance] (this will be null/blank if it doesn't exist)
	* $involved[n][Corporation]
	* $involved[n][Ship Type]
	* $involved[n][Weapon Type]
	* $involved[n][Final Blow] (this is a boolean true/false)
	
	The destoyed items array will have 3 fields for each element (n)
	* $destroyed[n][Type]
	* $destroyed[n][Location]
	* $destroyed[n][Quantity]

*/

//lets find the location of the involved parties and destroyed items lines
for ($i = 0;$i < count($lines);$i++){
	if(trim($lines[$i]) == "Involved parties:") {
		$inv_loc = $i;
	}else if(trim($lines[$i]) == "Destroyed items:") {
		$dest_loc = $i;
	}
}

//get victim info, start at line 2 go up to where we found the involved parties
for($i = 2;$i < $inv_loc;$i++){
	if ( (trim($lines[$i]) != "")) {//working on a good line
		unset($pair);
	    $pair = explode(": ", $lines[$i]);
		$victim[$pair[0]] = $pair[1];
	}	
}

//lets assemble our killers by looking between the involved parties line and the destroyed items line
$inv_loc++; //add one so we start after the title line
for($i = $inv_loc;$i < $dest_loc;$i++){
	$line = trim($lines[$i]);
	if( ($line != "") && ($line != "**** Truncated - mail is too large ****") ){//working on a good line
		unset($pair);
		$pair = explode(": ", $line);
		
		//clear array if on new killer
		if($pair[0] == "Name") {
			unset($killer);
			if(substr_count($pair[1], "(laid the final blow)")) {
				//found final killer get rid of final blow text
				$pair[1] = substr($pair[1], 0, strlen($pair[1]) - 22);
				$killer['Final Blow'] = 1;
			}else {//not final killer
				$killer['Final Blow'] = 0;
			}
		}
		
		//add data onto killer array
		$killer[$pair[0]] = $pair[1];
		
		if($pair[0] == "Weapon Type") {
			//finished, place the killer array onto the $involved stack
			$involved[] = $killer;
		}
	}
}

//lets assemble the items that were destroyed
$dest_loc++;
for($i = $dest_loc; $i < count($lines); $i++){
	$line = trim($lines[$i]);

	if( ($line != "") && ($line != "**** Truncated - mail is too large ****") ){//working on a good line
		unset($pair);
		$pair = explode(": ", $line);

		if($pair[0] != "Quantity") {//we only want the Type lines
			unset($item);
			//pull off the item name
			$item_name = substr($pair[1], 0, strpos($pair[1], "("));
			//pull off the location of the item; first pull from the last occurance of a ( then trim off the parenthesis
			$item_loc = ltrim(rtrim(strrchr($pair[1], "("), ")"), "(");
			$item['Type'] = $item_name;
			$item['Location'] = $item_loc;
			//check if this item has a quantity in the next line
			$k = $i + 1;
			if (substr_count($lines[$k], "Quantity: ")) {
				$line = trim($lines[$k]);
			    unset($pair);
				$pair = explode(": ", $line);
				$item['Quantity'] = $pair[1];
			}
			//add this item to the stack
			$destroyed[] = $item;
		}
			
	}
	
}

//now we're done lets echo out our data to see if we got it all right
echo "Victim: " . $victim['Victim'] . "<br>";
echo "Alliance: " . $victim['Alliance'] . "<br>";
echo "Corporation: " . $victim['Corporation'] . "<br>";
echo "Ship Lost: " . $victim['Destroyed Type'] . "<br>";
echo "Location: " . $victim['Solar System'] . "<br>";
echo "Security Level: " . $victim['Security Level'] . "<br>";
echo "<br>";

echo "There were " . count($involved) . " known involved parties.<br>";
echo "<br>";

foreach ($involved as $killer) {
	if ($killer['Final Blow']) {
	    echo "<b>The final blow was laid by:</b><br>";
		echo "<blockquote>";
		echo "Name: " . $killer['Name'] . "<br>";
		echo "Security Status: " . $killer['Security Status'] . "<br>";
		echo "Alliance: " . $killer['Alliance'] . "<br>";
		echo "Corporation: " . $killer['Corporation'] . "<br>";
		echo "Ship Type: " . $killer['Ship Type'] . "<br>";
		echo "Weapon Type: " . $killer['Weapon Type'] . "<br>";
		echo "</blockquote>";
		echo "<br>";
	}
}

echo "<b>Other partied involved were:</b><br>";
echo "<blockquote>";

foreach ($involved as $killer) {
	if(!$killer['Final Blow']) {
		echo "Name: " . $killer['Name'] . "<br>";
		echo "Security Status: " . $killer['Security Status'] . "<br>";
		echo "Alliance: " . $killer['Alliance'] . "<br>";
		echo "Corporation: " . $killer['Corporation'] . "<br>";
		echo "Ship Type: " . $killer['Ship Type'] . "<br>";
		echo "Weapon Type: " . $killer['Weapon Type'] . "<br>";
		echo "<br>";
	}
}
echo "</blockquote>";

echo "<br>";
echo "<b>Items destroyed in this ship loss:</b><br>";
echo "<blockquote>";

echo "<u>High Slot Items:</u><br>";
foreach ($destroyed as $item) {
	if ($item['Location'] == "Fitted - High slot") {
		//only high slot items have ammo
		if($item['Quantity'] == ""){
			echo $item['Type'] . "<br>";
		}else {
			echo $item['Quantity'] . " x " . $item['Type'] . "<br>";
		}
	}	
}

echo "<br>";
echo "<u>Medium Slot Items:</u><br>";
foreach ($destroyed as $item) {
	if ($item['Location'] == "Fitted - Medium slot") {
	    echo $item['Type'] . "<br>";
	}	
}

echo "<br>";
echo "<u>Low Slot Items:</u><br>";
foreach ($destroyed as $item) {
	if ($item['Location'] == "Fitted - Low slot") {
	    echo $item['Type'] . "<br>";
	}	
}

echo "<br>";
echo "<u>Cargo Items:</u><br>";
foreach ($destroyed as $item) {
	if ($item['Location'] == "Cargo") {
		//looks like Eve counts weird here but meh, lets show quantity all the time setting nulls to 1
		if ($item['Quantity'] == "") $item['Quantity'] = 1;
	    echo $item['Quantity'] . " x " . $item['Type'] . "<br>";
	}	
}
echo "</blockquote>";
?>

[Troy]

Great job, croft!

Now, consider if there are ways to automate the entire process even more! Right now, you plan to have a web page where each person, after receiving their kill email, will copy & paste the text into a form and submit it to your parsing script. What if instead, those people could just forward the email to a special account that would be checked by your parse script? This would be easier for people I think than browsing to a page and pasting in the text. At that point, they could even setup an auto-forward rule (if their email client has this feature like Outlook and Outlook Express do). With an auto-forward rule, they would not have to do anything! The kill mail would arrive then be automatically forwarded to your parsing script!

To accomplish this, you'd write a PHP script that uses POP3 to check a mail account that you create specifically for this purpose. The only thing left would be to schedule that script to routinely check the POP3 account. You can do this scheduling a number of ways depending on what access you have to the web server. If you have shell access and cron privs, you could create a cron job that runs your PHP script every 10 minutes or something like that. If you have a webpage that is frequently accessed anyway, you can add code to it that runs your POP3 checking and parsing script whenever people hit it. You have options.

Keep this thread going if you want to develop this further. It's the creative process that I love about programming. It's fun coming up with creative, logical solutions to complex situations.

[croft]

Using the auto email thingy is not an optimal way to go. Then they have to alt/tab out of the game start up outlook or whatever and mail it to a new adress. Plus you can not forward mail from eve mail client to outlook or any other program.

As it is now they just klick the ingame browser and go to the page post the mail and its done. Faster and easyer. Ofcourse i will add a bit security to it but not untill this is done.

[Troy]

...you can not forward mail from eve mail client...

Ah, so "eve" has it's own built in mail system for communicating with players. I was thinking that the game was sending standard email to the players normal email accounts. Gotcha.

Yeah, not sure of any way to automate this further. Again, good work, croft.

[Aradon]

Hi,

I too am trying to develop my own EVE kill mail parser, was wondering how this works, not literally how it works but can i see it working?

Cheers

[sonicj]

If you just want to pay someone to code the project for you, there are plenty of PHP gurus who'd be happy to code this according to your specs. Just let me know.

I am curious what a project like this would cost, and who I would contact for this.

[Troy]

A few hundred dollars most likely. You can post programming contract gigs in the IT Water Cooler forum. http://www.daniweb.com/techtalkforums/forum95.html

[digital-ether]

Interesting stuff.

You could automate it further by screen scraping the mail pages. Just have PHP download the latest mail periodically and do everything in an automated way.

However, this would require that PHP know the passwords of the users it wants to download mail for and also authenticate with the website.