Member Avatar for theweirdone

Hi,
I'm trying to extract certain things from a web page. The website is TVRage.com, and the example I'm using at the moment is . So far I've managed to get the title of the show using this code:

#!/usr/bin/env python

import urllib

def save_page(site=""):
	mypath = site
	mylines = urllib.urlopen(mypath).readlines() 
	
	f = open('temp2.txt', 'w')
	for item in mylines:	
		f.write(item)

	f.close()

def find_title(temp="temp2.txt"):
	f = open(temp, "r")
	site = f.read()
	f.close()
	
	search1 = "<title>"
	search2 = " (Episode"
	starter = site.find(search1)
	ender   = site.find(search2)
	#print "Starts at %s and ends at %s" % (starter, ender) Just gives the indexes
	print site[(starter+19):ender]

Now I'm trying to get episode numbers, dates, and titles, the only problem is I can't figure out how to extract them from the html. So far I've tried this code to no effect:

def find_episodes(temp="temp2.txt"):
	f = open(temp, "r")
	site = f.read()
	f.close()
	
	for line in site:
		if '/Warehouse_13/episodes/1064905360' in line:
			print line
		else:
			print "We got nothing."

Any suggestions would help tremendously.

  • 3 Contributors
  • 2 Replies
  • 695 Views
  • 1 Year Discussion Span
  • Latest Post Latest Post by Aung Myat

Dani AI

Generated

A more robust approach is to stop treating the page as plain text and use an HTML parser. is right to save the HTML for inspection; ’s line-by-line partitioning can work for a one-off, but it breaks easily when markup, whitespace, or attributes change. Use a parser (BeautifulSoup or lxml) to locate the episode container, then extract link text, nearby cells, or sibling nodes for dates and numbers. That makes your code resilient to formatting changes.

Example: fetch the page, find a table/section that holds episodes, then pull titles, dates and resolved URLs. Replace the container selector with what you discover in the browser devtools.

import requests
from bs4 import BeautifulSoup
from urllib.parse import urljoin
import re

r = requests.get(page_url, headers={'User-Agent': 'script/1.0'})
r.raise_for_status()
soup = BeautifulSoup(r.text, 'html.parser')

episodes = []
for tr in soup.select('table tr'):
    tds = tr.find_all('td')
    if len(tds) < 2:
        continue
    a = tr.find('a', href=True)
    if not a:
        continue
    title = a.get_text(strip=True)
    # look for a nearby date-like string (month names or ISO dates)
    date = next((s for s in tr.stripped_strings if re.search(r'\b(?:Jan|Feb|Mar|Apr|May|Jun|Jul|Aug|Sep|Oct|Nov|Dec|\d{4}-\d{2}-\d{2})\b', s)), None)
    episodes.append({'title': title, 'date': date, 'url': urljoin(r.url, a['href'])})

If you need continuous monitoring (as asked), poll at a sensible interval, persist the last-seen value to disk, and trigger your task only when the parsed number increases. Example pattern: fetch → parse number → compare with saved value → act → update saved value. Use exponential backoff on failures, a polite User-Agent, and a delay (seconds to minutes) to avoid overloading the site. Always check the site’s robots.txt/terms and prefer an official API if available.

Recommended Answers

All 2 Replies

Member Avatar for Beat_Slayer

Like this?

#!/usr/bin/env python

import urllib

def save_page(site=""):
    mypath = site
    f = open('temp2.txt', 'w')
    for item in urllib.urlopen(mypath).readlines():    
        f.write(item)
    f.close()

def find_title(temp="temp2.txt"):
    f = open(temp)
    site = f.readlines()
    f.close()
    for item in site:
        if item.find('<title>') != -1:
            before_html, tag_before, rest_html = str(item).partition('<title>')
            title, tag_after, after_html = rest_html.partition('</title>')
    print 'Title:', title

def find_episodes(temp="temp2.txt"):
    f = open(temp)
    site = f.readlines()
    f.close()
    for item in site:
        if item.find('''onmouseover="showToolTip2(event,'View Trailer');return false;" onmouseout="hideToolTip2();" ></a> <a href='/Warehouse_13/episodes/''') != -1:
            before_html, tag_before, rest_html = str(item).partition('''onmouseover="showToolTip2(event,'View Trailer');return false;" onmouseout="hideToolTip2();" ></a> <a href='/Warehouse_13/episodes/''')
            title, tag_after, after_html = rest_html.partition('</a> </td>')
            print 'Episodes:', title[12:]

save_page()
find_title()
find_episodes()

Happy coding.

commented: this is very nice. it helps you avoid regexes, which is awesome. +0
Member Avatar for Aung Myat

Hello

I am totally new to python and would like to develop a script.

The followings are my requirements:

1) I want to extract a number from the webpage and constantly monitor the number change.
2) Once there is a change in number, the script should compare with the number extracted earlier.
3) If new number is greater, the script should trigger to do some tasks. (i.e something like API for interface with another script).

Please see the attached HTML image to understand more.
Thank you very much in advance.

Be a part of the DaniWeb community

We're a friendly, industry-focused community of developers, IT pros, digital marketers, and technology enthusiasts meeting, networking, learning, and sharing knowledge.