snippsat 661 Master Poster

To set it up more,correct and fix error.
Capitale letters in method == 'POST'
Have remove GET to avoid to confusion.

foo\
app.py
    templates\
    index.htm

app.py

from flask import Flask,redirect,url_for,request,render_template

app = Flask (__name__)
@app.route('/')
def my_form():
    return render_template("index.html")

@app.route('/suc/<name>')
def suc(name):
    return "hello Boss %s" % name

@app.route('/login', methods=['POST'])
def log():
    if request.method == 'POST':
        user = request.form['nm']
        return redirect(url_for('suc', name=user))

if __name__ == '__main__' :
    app.run(debug=True)

index.html

<html>
   <body>
      <form action = "http://localhost:5000/login" method="post">
         <p>Enter Name:</p>
         <p><input type="text" name="nm" /></p>
         <p><input type="submit" value="submit" /></p>
      </form>
   </body>
</html>
snippsat 661 Master Poster

It's part of standar library,so it should work.
Here a run in Python 3.4.

>>> from http import cookies
>>> a_cooike = cookies.SimpleCookie()
>>> a_cooike['Base'] = 'cream'
>>> print(a_cooike)
Set-Cookie: Base=cream

The default placement for python 3.5 is terrible,what are developers thinking about?
C:\Users\hohait\AppData\Local\Programs\Python\Python35-32
I know the answer and it security relatet,but i don't like it.

Follow this,the you get better placement C:\Python35-32
Make sure under installation that you mark for Path and pip.
Test pip and upgrade pip.
Here a run and use virtualenv,so you see how pip work from clean setup.

C:\test\Scripts
λ activate

(test) C:\test\Scripts
λ python -m pip install --upgrade pip
Requirement already up-to-date: pip in c:\test\lib\site-packages

(test) C:\test\Scripts
λ pip -V
pip 8.1.2 from c:\test\lib\site-packages (python 3.4)

(test) C:\test\Scripts
λ pip install requests
Collecting requests
  Downloading requests-2.11.0-py2.py3-none-any.whl (514kB)
    100% |################################| 522kB 1.0MB/s
Installing collected packages: requests
Successfully installed requests-2.11.0

(test) C:\test\Scripts
λ python
Python 3.4.2 (v3.4.2:ab2c023a9432, Oct  6 2014, 22:15:05) [MSC v.1600 32 bit (Intel)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import requests
>>> requests.__version__
'2.11.0'

#And also test cookies
>>> from http import cookies
>>>
snippsat 661 Master Poster

I like python and I wanted to give it a try at web programming but the setup and management is too difficult. I'll just go back to php.

It's not so difficult,and you do not need stuff like LAMP.
Django and Flask has build in web-server this you use for all development local.

Try Flask is as easy as it get,try hello world example on the front page.
Run the 7 lines hello.py then it's a server,and you go to http://localhost:5000/ in browser to see result.

I use Flask for all my web-development,the simplicity and power it has to build all from small to bigger web apps is great.
Just a example of an simple image viewer,
i just iterate over some of my images and use CSS to organize little.
And i delpoy to web,you can see result here Image viewer

So it's like 40 lines of code and never used LAMP or any other stuff,just the build in web-server.
I talk more deploy and this stuff in this post.

server.py

from flask import Flask, render_template, jsonify
import os

app = Flask(__name__,static_url_path='')    
@app.route('/')
def img():
    pics = os.listdir('mysite/static/images')
    return render_template('img1.html', pics=pics)

if __name__ == '__main__':
  app.run()

index.html

 <!DOCTYPE html>
<html>
<head>
  <style>
    body {
      background: radial-gradient(#E0D9C6, #4B3A40);
    }
    h1 {
      text-align: center;
      font-weight: bold;
      color: #663333;
      font-size:60px;
    }
    .thumbnail {
      float: left;
      width: auto;
      margin: 2% 2% 30px 7%;
      height: 600px;
    }
  </style>
</head>
  <body>
     <h1>Image viewer</h1> …
snippsat 661 Master Poster

Command you has to be string and you concatenate "line" inside os.system().
So if i have i list of words and want to now lenght of each word,
i can use linux command expr length.
The command get executed on each line of words.txt.

import os
import time

f = open('words.txt')
for line in f:
    line = line.strip()
    time.sleep(5)
    os.system('expr length ' + line)
f.close()    
snippsat 661 Master Poster

The problem can be solved before give data to Pandas.
As mention over there are lot of cases to think about.
I would have made up rules with regex.
Example.

 >>> import re

>>> s = "Richard Wayne Van Dyke"
>>> re.split(r"\s*(?:(?:[a-zA-Z]')|(Van \w.*))", s, re.I)[:-1]
['Richard Wayne', 'Van Dyke']
>>> s = 'Wayne Van Dyke'
>>> re.split(r"\s*(?:(?:[a-zA-Z]')|(Van \w.*))", s, re.I)[:-1]
['Wayne', 'Van Dyke']
>>> s = 'Edda Kathleen Van Heemstra Hepburn-Ruston'
>>> re.split(r"\s*(?:(?:[a-zA-Z]')|(Van \w.*))", s, re.I)[:-1]
['Edda Kathleen', 'Van Heemstra Hepburn-Ruston']

>>> s = "Gary Del Barco"
>>> re.split(r"\s*(?:(?:[a-zA-Z]')|(Del \w.*))", s, re.I)[:-1]
['Gary', 'Del Barco']

>>> s = 'Dave Allen Smith'
>>> re.split(r"\s*(?:(?:[a-zA-Z]')|(Del \w.*))", s, re.I)[:-1]
[]

So key word like Van and Del,in this regex will take key word and all word after as last name.
If need more rule than this,you have to make own make rule for problem names.
If pass in name that's not in regex,a empy list is returned.

mattrweaver commented: Thanks for the responses. I will be working with a limited set of data periodically, so I can easily write new rules for problem names. +0
snippsat 661 Master Poster

Is that what gets returned if the letter is not in the dictionary? So if it can't find the letter it adds 0?

Yes,and you can also return a more meaningful message.
Here in function and return value out.

def record(word):
    return {
        "t": 1, "f": 1, "c": 1,
        "b": 2, "h": 2, "y": 2,
        "q": 3, "w": 3, "z": 3
    }.get(word, "Not in record")

if __name__ == '__main__':
    word = input("Please enter a word: ")
    print(record(word))
snippsat 661 Master Poster

Beautiful Soup is for python . i am using windows and vb.net

You should find a parser for vb.net,like Html Agility Pack

i have been doing some research.
and found out about Regular expressions.....

Regex i bad tool for html,read this funny answer bye bobince .

gives me pratice coding.......

Yes and lot frustration if you gonne to this every time you want to parse stuff.
Beautiful Soup was mention so i can use some minutes and make a demo.

from bs4 import BeautifulSoup

soup = BeautifulSoup(html)
h1_tag = soup.find('h1')
table_tag = soup.find_all('table')
span_tag = soup.find_all('span', style=True)
span_tag = span_tag[0].find_all('div')

print(h1_tag.text)
for item in table_tag:
    print(item.text)
for item in span_tag:
    r = str(item).replace('<br/>', '\n')
    print(r.replace('<div>', '').replace('</div>', '').strip())

Output:

Friday 31 - 1 - 2014

Created:31/01/2014 2:32 PM
Updated:31/01/2014 7:03 PM
Location:37°35'13 S  145°7'41 E

Shift 143
Car 7008
Bus 280
564 / 9
Car 7008
Car 7011
Bus 361
572 / 20
snippsat 661 Master Poster

I just started to learn python few days ago. This is a small program I made to practise.

Ok,some points.

names = ['(1) Sphere','(2) Right Circular Cone','(3) Cylinder']
x=0
while x < 3:
    print(names[x])
    x +=1

This is not the way to loop in Python.
You just iterate over list like this.

names = ['(1) Sphere','(2) Right Circular Cone','(3) Cylinder']
for name in names:
    print(name)

Now all code is global space,in future think of stucture code better.
Eg using functions.

import math

def cylinder():
    '''Calculate volume of a cylinder'''
    R = int(input('\n'+"Enter value for r : "))
    H = int(input("Enter value for h : "))
    result = math.pi * (R*R) * H
    return result

Test.

>>> cylinder()
Enter value for r : 5
Enter value for h : 10
785.3981633974483

>>> help(cylinder)
Help on function cylinder in module __main__:

cylinder()
    Calculate volume of a cylinder

So you could had have 3 functions,this mean that code is isolatet in function.
As you see it help readability of code,and code reuse(you can now put this function in other code and call it).

snippsat 661 Master Poster

I've seen the Full stack python, it also gives you many options to choose & but not how do I choose them

Yes it can be confusing,Nginx + Gunicorn,Tornado,uWSGI?
Here is little info about Heroku and PythonAnywhere.

@Heroku
Do Python applications run behind Nginx?
So on Heroku you do not use Nginx,you just have to setup a server like Gunicorn.
You see how in Getting Started with Python on Heroku

PythonAnywhere is very easy,and do like all deploy stuff for you.
You use UI and upload files and do a coulpe of config choices.
Then you have website,and PythonAnywhere run that site with,
Nginx to do routing and uWSGI to run the apps.

snippsat 661 Master Poster

In the first question, ctually asking which directory on linux should I choose to place a project? Like /home/user/ or /var/www etc?

Where I should put my python scripts in Linux?.

In the 3rd question, you said you'd use something like Gunicorn,Tornado,uWSGI or Apache. The question how do you decide between so many options. When do you want to use Gunicorn & when do you prefer mod_wsgi on Apache like that?

I like Gunicorn easy to setup,and default options works good for most cases.
I you search eg: Gunicorn vs uWSGI you get a lot of opinions.

Look at Full Stack Python,there is a lot of info on deploy and run a Python web application.
For smaller personal projects you don't have to think to much about the deploy side.
For bigger commercial sites there is a lot of tought/work on the deploy side.

Gribouillis commented: awesome +0
snippsat 661 Master Poster

Where should I host a python project on linux? I'm sure there'd be multiple choices. So, Where would you host it & on what circumstances?

Github,Bitbucket...
If you mean web-host,pythonanywhere,Heroku,Webfaction,Digital ocean...

I understand that virtualenv creates a virtual environment separate from others. But, why should one need it? Do we use it in production too, if so, why?

You pip install stuff into virtualenv,
so you always get newest versions of modules/package.
It's isolatet,so it don't mess with your OS version of Python.
As you see in this Getting Started tutorial is normal to use virtualenv when using a web-host.

If I want to create a web application using python? What server should I choose & on what basis should I choose them for a given project.

Flask and Django has build in webserver,
this is good for working/testing local and building a web application.

When finish and want to deploy to web,
then think of(shall i use Gunicorn,Tornado,uWSGI or running mod_wsgi (Apache)?
Host like Pythonanywhere,Heroku make it easier to deploy Python code.

Gribouillis commented: great answer +14
snippsat 661 Master Poster

I am trying to convert a file to a dictionary here is the contents of the file,

You can data keep structure bye using serialization,
then you don't have to struggle to recreate structure from a string.
JSON is preferably over Pickle.

import json

record = {
'Name': 'DragonMastur',
'Hand': None,
'Inventory': {"Gold": 0, "Brush": 0, "Twig": 0, "Sapling": 0,
    "Metal Axe": 0, "Grass": 0, "Metal Pickaxe": 0, "Metal": 0,
    "Log": 0, "Diamond": 0, "Stick": 0, "Stone": 0, "Wheat": 0,
    "Stone Axe": 0, "Stone Pickaxe": 0},
'Health': 100
    }

with open("my_file.json", "w") as f_out:
    json.dump(record, f_out)
with open("my_file.json") as f_obj:
    saved_record = json.load(f_obj)

Test that saved_record still works as a dictionary.

>>> type(saved_record)
<class 'dict'>
>>> saved_record['Name']
'DragonMastur'
>>> saved_record['Inventory']
{'Brush': 0,
 'Diamond': 0,
 'Gold': 0,
 'Grass': 0,
 'Log': 0,
 'Metal': 0,
 'Metal Axe': 0,
 'Metal Pickaxe': 0,
 'Sapling': 0,
 'Stick': 0,
 'Stone': 0,
 'Stone Axe': 0,
 'Stone Pickaxe': 0,
 'Twig': 0,
 'Wheat': 0}
>>> saved_record['Inventory']['Gold']
0
snippsat 661 Master Poster

A must read

Or a better read
You should really look into and use Requests Shailang.

rproffitt commented: Nice. Something I didn't know, thanks. +5
snippsat 661 Master Poster

I have to say that i don't like this tagging system at all.
Maybe it's get better when i get used to it,but i am not sure.

In Python part of the old forum we had sticky thread,
do we really need to search for those threads now?
Or can post be sticky again?
Here is a picture.

snippsat 661 Master Poster

really don't even know which webpage language I should use HTML, PHP, SQL something else?.

You can of course use Python.
Micro-framework like Flask can be a good place to start.
Python has bigger Framework like Django, web2py or CMS like Django-cms.
These has a lot of tools,and can be a overwhelming when you shall learn basic web stuff.

Your question show that this is all new.
For web-development HTML,CSS,JavaScript(jQuery) is something that's has to be studied in some degree.
E.g you run Flask as server and Write HTML and CSS to show stuff in a webpage.

I'm assuming I can simply pipe the python code to an file of some sort, but is there a better approach to getting started for this type of application?

First you can test local and try to upload file to Flask server.
If you want this to be a webpage that is running public on internet,
you need a host like Pythonanywhere, Heroku.

snippsat 661 Master Poster

If you search for cola you see this url.
https://datacvr.virk.dk/data/visninger?soeg=cola&type=Alle
Generate own search url's.

>>> url = 'https://datacvr.virk.dk/data/visninger?soeg={}&type=Alle'
>>> search_lst = ['car', 'train']
>>> for item in search_lst:
...     print(url.format(item))
...     
https://datacvr.virk.dk/data/visninger?soeg=car&type=Alle
https://datacvr.virk.dk/data/visninger?soeg=train&type=Alle

E.g of take out some data,here all title text when search for cola.

import requests
from bs4 import BeautifulSoup

url = 'https://datacvr.virk.dk/data/visninger?soeg=cola&type=Alle'
page = requests.get(url)
soup = BeautifulSoup(page.content)
tile_tag = soup.find_all('h2', {'class': 'name'})
for name in tile_tag:
    print(name.text.strip())

"""Output-->
COCA-COLA NORDIC SERVICES ApS
Cola klubben
Coca-Cola Service S.A.
The Coca-Cola Company
CARLSBERG DANMARK A/S
Abby Thai Kropsmassage v/Ornanong Johansen
Karim Rahima
Coca-Cola Service S.A.
CARLSBERG DANMARK A/S Coca -Cola Tapperierne
COCA-COLA NORDIC SERVICES ApS
"""
Slavi commented: Thank you +6
snippsat 661 Master Poster

It work's for me when i do a little test,using Pyhon 3.4.2.

mytxt1.txt:

http://ascii.cl?parameter=%22Click+on+%27URL+Decode%27%21%22
login=bob%40%3CSCRipt%3Ealert%28Paros%29%3C%2FscrIPT%3E.parosproxy.org
http://ascii.cl?parameter=%22Click+on+%27URL+Decode%27%21%22
login=bob%40%3CSCRipt%3Ealert%28Paros%29%3C%2FscrIPT%3E.parosproxy.org

outtxt1.txt:

http://ascii.cl?parameter="Click+on+'URL+Decode'!"
login=bob@<SCRipt>alert(Paros)</scrIPT>.parosproxy.org
http://ascii.cl?parameter="Click+on+'URL+Decode'!"
login=bob@<SCRipt>alert(Paros)</scrIPT>.parosproxy.org
snippsat 661 Master Poster

Also, strings in Python 3 are unicode so enocde and decode are not necessary.

That's only true if text is already inside Python 3.
Here we are talking about taking text from outside into Python 3,
then we must define a encoding like utf-8,latin-1...,
or it will give an error or become a byte string.

Because we must read with correct encoding when taking text into Python 3,
with open() has new stuff like errors='ignore', errors='replace'

with open('some_file', 'r', encoding='utf-8', errors='ignore') as f:
    print(f.read())

So this statement.
In Python 3 are all strings are sequences of Unicode character
Yes this is true,
but then all text taken in from outside must have been correct encoded into Python 3.

snippsat 661 Master Poster

Here is a hint.

>>> user_input = 'The quick brown fox jumps over a lazy dog'
>>> word_lst = user_input.split()
>>> word_lst
['The', 'quick', 'brown', 'fox', 'jumps', 'over', 'a', 'lazy', 'dog']
>>> len(word_lst)
9
>>> [len(c) for c in word_lst]
[3, 5, 5, 3, 5, 4, 1, 4, 3]
sum([len(c) for c in word_lst])
33

If you wonder about [len(c) for c in word_lst],so is this called list comprehensions.
list comprehensions is used very much in Python.
To write it in a ordinary loop, it looks like this.

>>> char_count = []
>>> for c in word_lst:
...     char_count.append(len(c))
...     
>>> char_count
[3, 5, 5, 3, 5, 4, 1, 4, 3]
snippsat 661 Master Poster

snippsat I understand the inputting it into a gui or >>> but I need it to take any user input and not just set filename. Unless I set filename to equal a list of file extensions.

from os.path import splitext

def parse_extension(filename): 
    if filename == splitext(filename)[-1]:
        print 'Please Enter a FileName: ' #Think of what you are doing here
    elif filename != splitext(filename)[-1]:
        print splitext(filename)[-1]

filename = raw_input('Please Enter a File Name: ')
parse_extension(filename)

I dont understand how the import os feature works

os is part of Python standard library,just import os and use it.
Doc Miscellaneous operating system interfaces
Tutorial OS Module

snippsat 661 Master Poster

There is also a markdown2 module.

import markdown2

markdown = """\
# Hello, how are you
I'm fine, thank you

I know that it is not so **hard**
but for me it's not so *easy*

What is ##this?

How about `*this*`?

**it *wor`k`s!***

Let me put a bold link: [**stuff**](http://41.media.tumblr.com/49a58542fd70b8ca39b5bd0d9c9c53aa/tumblr_nob40mvTN41tb9nzio1_500.jpg)
## Okay, part two

But there's no part two!

###### So long!"""

print(markdown2.markdown(markdown))

"""Output-->
<h1>Hello, how are you</h1>

<p>I'm fine, thank you</p>

<p>I know that it is not so <strong>hard</strong>
but for me it's not so <em>easy</em></p>

<p>What is ##this?</p>

<p>How about <code>*this*</code>?</p>

<p><strong>it <em>wor<code>k</code>s!</em></strong></p>

<p>Let me put a bold link: <a href="http://41.media.tumblr.com/49a58542fd70b8ca39b5bd0d9c9c53aa/tumblr_nob40mvTN41tb9nzio1_500.jpg"><strong>stuff</strong></a></p>

<h2>Okay, part two</h2>

<p>But there's no part two!</p>

<h6>So long!</h6>
"""
snippsat 661 Master Poster

A good technique to understand code is to break it up in smaller parts.
For this is interactive interpreter great to us,
also us print() in code on places you wonder what's happends.
So lets say you wonder about deque(maxlen=history).

>>> from collections import deque
>>> lst = [1, 2, 3, 4, 5, 6, 7]
>>> d = deque(maxlen=3)
>>> for item in lst:
...     d.append(item)
...     print(d)
...     
deque([1], maxlen=3)
deque([1, 2], maxlen=3)
deque([1, 2, 3], maxlen=3)
deque([2, 3, 4], maxlen=3)
deque([3, 4, 5], maxlen=3)
deque([4, 5, 6], maxlen=3)
deque([5, 6, 7], maxlen=3)

So here it easy to see that it fill up a list,
up to 3 elements max and then it start to throw out the first vaules in.

snippsat 661 Master Poster

It's a little old way of doing this,but it work if fixing as Girb posted.
You should not make a seperate line for count.
Do it like this.

my_dict = {}
s = 'hello world hello'
for word in s.split():
    if word in my_dict:
        my_dict[word] += 1
    else:
        my_dict[word] = 1

print(my_dict) #{'world': 1, 'hello': 2}

The next step in improving this,can be to use get() method.

my_dict = {}
s = 'hello world hello'
for word in s.split():
    my_dict[word] = my_dict.get(word, 0) + 1

print(my_dict) #{'world': 1, 'hello': 2}

Your code with full Python power,here i use colletions.Counter.
And regex to remove punctuation.
You should also look at PEP-8,e.g not use capitale letter in variable name.

from collections import Counter
import re    

with open('count_test.txt') as f:
    text = f.read().lower()
words = re.findall(r'\w+', text)

print(Counter(words))
print(Counter(words).most_common(2))
Slavi commented: love the get() +6
snippsat 661 Master Poster

Good effort,i give you a up vote:)
But have to mention your code style again,it's not so pretty.
I link to PEP-8 in your post,
just to show a doc that can help you in the future regarding code style.

Here a version i made,
Look a little at how code is formatet,
and names i use in code(No capitale letters,because that is used in class name).

#Testet work in Python 2 and 3
from __future__ import print_function
from random import randint
#Fix so input() work 2 and 3
try:
    input = raw_input
except NameError:
    pass

def random_hex_colors():
    return "#{:06x}".format(randint(0,0xFFFFFF)).upper()

def gen_html_tag(random_hex_colors, number_of_tags, width, filler_text):
    return \
    '\n'.join('<tr><td>{0}</td><td style ='
     '"background-color: {0};width:{1}px">{2}</td></tr>'
    .format(random_hex_colors(), html_width, text_filler)
    for i in range(number_of_tags))

def write_html(tag_gen, file_name):
    with open('{}.html'.format(file_name), 'w') as f_obj:
        f_obj.write('''\
<!DOCTYPE html>
<head>
<title>Random Color Chart</title>
</head>
<table border = "1">
<tr><th>Color Name</th><th>Color</th></tr>
{}
</table>'''.format(tag_gen))

if __name__ == '__main__':
    #Test without user input
    number_of_tags = 3
    html_width = 300
    text_filler = 'test'
    file_name = 'color_test'        
    #Call functions
    tag_gen = gen_html_tag(random_hex_colors,
    number_of_tags, html_width, text_filler)
    write_html(tag_gen, file_name)
snippsat 661 Master Poster

You have been told before to before to watch your coding style.
Stop using global.

There is a also build module that solve this in Python 3.

from functools import lru_cache

@lru_cache(maxsize=None)
def fib(n):
    if n < 2:
        return n
    return fib(n-1) + fib(n-2)

Test all computation are instantly.

>>> fib(100)
354224848179261915075
>>> fib(500)    139423224561697880139724382870407283950070256587697307264108962948325571622863290691557658876222521294125
>>> fib(700)    87470814955752846203978413017571327342367240967697381074230432592527501911290377655628227150878427331693193369109193672330777527943718169105124275

Maybe post it as question first before making a Code snippets,
to get advice about style and other way to solve it.

The author of lru_cache Raymond Hettinger,
had a whole talk about PEP-8 in this year PyCon.
Raymond Hettinger - Beyond PEP 8 -- Best practices for beautiful intelligible code -

Gribouillis commented: great +14
snippsat 661 Master Poster

You use subprocess module.

import subprocess

calc_program = "path_to_program"
subprocess.call([clac_program])
snippsat 661 Master Poster

It's started (:
Find some videos that are interesting,feel free to post link or discuses it here.

I was looking most forward to David Beazley talks.
Live coding with Concurrency as topic,doesn't get much better than this.
David Beazley - Python Concurrency From the Ground

The main talk David Beazley has is about Modules and Packages.
Maybe not the most interesting topic,haven't watch it all(long),but i guess it will be good.
David Beazley - Modules and Packages: Live and Let Die!

snippsat 661 Master Poster

because my anaconda distribution doesn't seem to have linprog (?)

You can use pip install or conda install(Scripts folder) to get new stuff into Anaconda.
For this is best to use conda install scikit-learn,then you get all new packages that scikit-learn need.
Look like this when run conda install scikit-learn.

The following packages will be downloaded:   

package                    |            build
    ---------------------------|-----------------
    conda-3.10.0               |           py27_0         207 KB
    conda-env-2.1.3            |           py27_0          54 KB
    nose-1.3.4                 |           py27_1         233 KB
    numpy-1.9.2                |           py27_0        23.2 MB
    requests-2.6.0             |           py27_0         590 KB
    scikit-learn-0.16.0        |       np19py27_0         3.5 MB
    scipy-0.15.1               |       np19py27_0        71.3 MB
    ------------------------------------------------------------
                                           Total:        99.0 MB

As you see you get scipy version 0.15.1.

You can even create a virtual environment,with all stuff you need.
conda create -n test scikit-learn
Create a folder test(virtual environment) that has Python and scikit-learn installed.

Gribouillis commented: great info +14
snippsat 661 Master Poster

Making hello world movie with great MoivePy
Results can be seen here

This make a 10 sec long videoclip showing hello world,using codec H264.

import moviepy.editor as moviepy

hello_world = moviepy.TextClip('Hello World!',font="Amiri-bold", fontsize=100, color='blue', size=(800,600))
hello_world = hello_world.set_duration(10)
hello_world.write_videofile('hello_world.avi', fps=24, codec='libx264')

Here a circle fade to text The End.

from moviepy.editor import *
from moviepy.video.tools.drawing import circle

clip = VideoFileClip("hello_world.avi", audio=False).\
           subclip(0,10).\
           add_mask()
w,h = clip.size

# The mask is a circle white vanishing radius r(t) = 800-200*t
clip.mask.get_frame = lambda t: circle(screensize=(clip.w,clip.h),
                                       center=(clip.w/2,clip.h/4),
                                       radius=max(0,int(800-200*t)),
                                       col1=1, col2=0, blur=4)

the_end = TextClip("The End", font="Amiri-bold", color="red",
                   fontsize=200).set_duration(clip.duration)
final = CompositeVideoClip([the_end.set_pos('center'),clip],
                           size =clip.size)
final.write_videofile('the_end.avi', fps=24, codec='libx264')
snippsat 661 Master Poster

psutil is good.
Here a run on my windows and Mint(VirtualBox)

>>> import psutil
>>> psutil.disk_partitions()
[sdiskpart(device='C:\\', mountpoint='C:\\', fstype='NTFS', opts='rw,fixed'),
 sdiskpart(device='D:\\', mountpoint='D:\\', fstype='', opts='cdrom'),
 sdiskpart(device='E:\\', mountpoint='E:\\', fstype='NTFS', opts='rw,fixed'),
 sdiskpart(device='F:\\', mountpoint='F:\\', fstype='UDF', opts='ro,cdrom'),
 sdiskpart(device='G:\\', mountpoint='G:\\', fstype='NTFS', opts='rw,fixed')]
>>> psutil.disk_usage('C:\\')
sdiskusage(total=255953203200L, used=177126027264L, free=78827175936L, percent=69.2)

Mint:

>>> import psutil
>>> psutil.disk_partitions()
[partition(device='/dev/sda1', mountpoint='/', fstype='ext4', opts='rw,errors=remount-ro')]
>>> psutil.disk_io_counters()
iostat(read_count=32177, write_count=6804, read_bytes=833645568, write_bytes=317702144, read_time=123880, write_time=124128)
snippsat 661 Master Poster

Here are the diffrent ways,
and also what i would call the prefered way these day with Requests.

Python 2:

from urllib2 import urlopen

page_source = urlopen("http://python.org").read()
print page_source

Python 3:

from urllib.request import urlopen

page_source = urlopen('http://python.org').read().decode('utf_8')
print(page_source)

For Python 3 to get str output and not byte we need to decode to utf-8.

Here with Requests,work for Python 2 and 3:

import requests

page_source = requests.get('http://python.org')
print(page_source.text)

Basic web-scraping we read in with Requests and parse with BeautifulSoup.

import requests
from bs4 import BeautifulSoup    

page_source = requests.get('http://python.org')
soup = BeautifulSoup(page_source.text)
print(soup.find('title').text) #--> Welcome to Python.org
snippsat 661 Master Poster

print link['title'] what is title? The alt of the <img> tag?

Have you looked at Firebug or Chrome DevTools?
Links i gave you in post.
Then is easy to so see what <title> of the <img> is.

except KeyError: what does keyError mean here? What the keyword keyError is for?

Not all <img> on this page has a title tag,so it trow a keyError.
This is source code that gets called in BeautifulSoup.

def __getitem__(self, key):
    """tag[key] returns the value of the 'key' attribute for the tag,
       and throws an exception if it's not there."""
    return self.attrs[key]

With except KeyError: pass,just ignore those images.
Can be more specific with class="col-md-6,
so it only search for names on images we need.
Then it will not trow an error.

from bs4 import BeautifulSoup
import urllib2

url = 'http://www.thefamouspeople.com/singers.php'
html = urllib2.urlopen(url) #Don not use read()
soup = BeautifulSoup(html)
tag_row = soup.find_all('div', {'class':'col-md-6'})
for item in tag_row:
    print item.find('img')['title']
snippsat 661 Master Poster

David W it's about time to learn string formatting :)
Splitting up with , and + something + is not so nice.

print('"{}".istitle() is {}'.format(item, item.istitle()))

Python String Format Cookbook

string formatting specifications(Gribouillis)

snippsat 661 Master Poster

So now i want my script to find every word that start with "A" in that url page >and print it for me. Then i should find a way to ask my crawle just save those >words starting with "A" that are singers names.
Very difficult!!!

That would be nightmare,and you would have to clean up a lot of rubbish text.
This is just one word that start with A Ajax.Request(,
that i saw when i quickly looked at source you get from that page.

You have to find a tag that give you info you want.
so tag <img> with <title>,will give you a fine list.

from bs4 import BeautifulSoup
import urllib2

url = 'http://www.thefamouspeople.com/singers.php'
html = urllib2.urlopen(url) #Do not use read()
soup = BeautifulSoup(html)
link_a = soup.find_all('img')
for link in link_a:
    try:
        print link['title']
    except KeyError:
        pass

"""Ouptput--> here just 3 names befor it changes to B
Ashlee Simpson
Avril Ramona Lavigne
Axl Rose
Barbra Streisand
Barry Manilow
Barry White
"""
snippsat 661 Master Poster

he's gone too far=D

Yes of course,to make it great humoristic read.
Regex can be ok to use some times,like you only need a singel text/value.

Both BeautifulSoup and lxml has build in support for regex.
Sometime it ok to use regex as helper to parser,when parsing dynamic web-sites
you can get a at lot of rubbish text.

snippsat 661 Master Poster

Use regular expressions Click Here

No no no just to make it clear :)
Have to post this link again.
Use a parser Beautifulsoup or lxml.

from bs4 import BeautifulSoup

html = '''\
<head>
  <title>Page Title</title>
</head>
<body>
  <li>Text in li 1</li>
  <li>Text in li 2</li>
</body>
</html>'''

soup = BeautifulSoup(html)
tag_li = soup.find_all('li')
print tag_li
for tag in tag_li:
    print tag.text

"""Output-->
[<li>Text in li 1</li>, <li>Text in li 2</li>]
Text in li 1
Text in li 2
"""
Slavi commented: Great read, but this 'Every time you attempt to parse HTML with regular expressions, the unholy child weeps the blood of virgins.. he's gone too far=D +6
snippsat 661 Master Poster

so why it didn't print the url of each website into output?!

Because this is a dynamic site using JavaScript,jQuery....
The problem is that JavaScript get evaluatet by DOM in browser.
To get links we need something that automates browsers like Selenium.

Can show you one way,here i also use PhantomJS,for not loading a browser.

from bs4 import BeautifulSoup
from selenium import webdriver

driver = webdriver.PhantomJS(executable_path='C:/phantom/phantomjs')
driver.set_window_size(1120, 550)
driver.get('https://duckduckgo.com/?q=3D&t=canonical&ia=meanings')
page_source = driver.page_source
soup = BeautifulSoup(page_source)
link_a = soup.find_all('a')
for link in set(link_a):
    if 'http' in repr(link):
         try:
            print link['href']
         except KeyError:
            pass

Output: here first 6 links of out 100 links.

http://3d.si.edu/
https://en.wikipedia.org/wiki/3-D_film
http://3d.about.com/
http://www.boxofficemojo.com/genres/chart/?id=3d.htm
http://www.urbanfonts.com/fonts/3d-fonts.htm
http://www.3dcontentcentral.com/

This is more advanced web-scraping,
and you need to study(understand) site and know what tools to use.

Gribouillis commented: very good help +14
snippsat 661 Master Poster

Change last part to this.

lst = []
for k in d:
    high = int(max(d[k]))
    lst.append((k, high))

score_lst = sorted(lst, key=lambda tup: tup[1], reverse=True)
for name,score in score_lst:
    print('{} {}'.format(name, score))

It's ok to throw away lambda,and use itemgetter as shown by slavi.

snippsat 661 Master Poster

Use the new bs4,do not call old BeautifulSoup.
Do not use read(),BeautifulSoup detect encoding and convert to Unicode.

As mention you need take out href attributes,
and you most learn to study webpage with Firebug or Chrome DevTools.
So then you see that you only need adresses that start with http and have href attributes.

from bs4 import BeautifulSoup # Use bs4
import urllib2

url = urllib2.urlopen('http://www.python.org') # Do not call read()
soup = BeautifulSoup(url)
with open('python-links.txt.', 'w') as f:
    for link in soup.find_all('a'):
        if link['href'].startswith('http'):
            f.write('{}\n'.format(link['href']))
snippsat 661 Master Poster

As Andrae posted,but need to close file object.

f = open('url.txt', 'w') #Opens the file to write text
f.write(url) #Writes the file to the opened text file
f.close()

Or the best approch is to use with open(),no need to close file object.

import urllib

url = 'http://www.python.org'
text = urllib.urlopen(url).read()
with open('url.txt', 'w') as f:
    f.write(text)
snippsat 661 Master Poster
>>> answer = ('7', 'Q', '5', 'H', '9', '4', '3', '8', 'L') 
>>> print(' '.join(answer))
7 Q 5 H 9 4 3 8 L
>>> help(''.join)
Help on built-in function join:

join(...) method of builtins.str instance
    S.join(iterable) -> str

    Return a string which is the concatenation of the strings in the
    iterable.  The separator between elements is S.
snippsat 661 Master Poster

Don't call both function main,use name that make sense.

import random

def generate_numbers():
    one = 1
    thirteen = 13
    subtract = 1
    number_gen = open('numbers.txt', 'w')
    for number in range(one,thirteen,subtract):
        numbers = random.randint(1,100)
        number_gen.write(str(numbers) + ' ')
    number_gen.close()

def read_numers(numb):
    numbers_read = open(numb)
    numbers = [float(n) for n in numbers_read.read().split()]
    numbers_read.close()
    for n in numbers:
        print(n)

generate_numbers()
numb = 'numbers.txt'
read_numers(numb)

A more pythonic approch.

def generate_numbers():
    with open('numbers.txt', 'w') as f_out:
        f_out.write(' '.join(str(random.randint(1,100)) for i in range(12)))

def read_numers(numb):
    with open(numb) as f:
        numbers = [float(n) for n in f.read().split()]
        for n in numbers:
            print(n)

generate_numbers()
numb = 'numbers.txt'
read_numers(numb)
snippsat 661 Master Poster

Here is a big hint.

>>> import random
>>> s = 'i am a programmer'
>>> ''.join(random.sample(s, len(s)))
'm mraep mriago ra'
>>> #no whitespace
>>> s_1 = ''.join(s.split())
>>> ''.join(random.sample(s_1, len(s_1)))
'omagripramamer'
snippsat 661 Master Poster

A look at Dataset and easy and Pythonic way to create a database.
Other candidates in this categorie,i will mention pyDAL and Peewee.
I did some test of Dataset in this post to look at.

So here a common task some web-scraping of food recipes(just a task i did help someone with).
A task you may not use a database for(to much work).

So let see how Dataset will work for this task.
Requirement Requests and BeautifulSoup.
First clean code without Dataset.

import requests
from bs4 import BeautifulSoup

start_page = 1
end_page = 3
for page in range(start_page, end_page+1):
    url = 'http://www.taste.com.au/search-recipes/?q=&%3Btag[]=13&%3Btag[]=28&%3B=&sort=title&order=asc&page={}'.format(page)
    url_page = requests.get(url)        
    soup = BeautifulSoup(url_page.text)
    tag_div = soup.find_all('div', {'class': "content-item tab-content current"})[0]\
    .find_all('div', {'class': 'story-block'})
    print('--- Page {} ---'.format(page))
    for content in tag_div:
        print(url_page.status_code, content.find('a')['href'])

In code snippet i bring in Dataset,and take out some data.

snippsat 661 Master Poster

for sses in assesses, it output me 1 instead 2! What is the secret?

In your first post you dont't have "assesses" and "sses" test.
str.count() don't count overlapping occurrences.
So if you need to count overlapping occurrences you can not use str.count().
Help and doc do mention that it return non-overlapping occurrences.

>> help(str.count)
Help on method_descriptor:

count(...)
    S.count(sub[, start[, end]]) -> int

    Return the number of non-overlapping occurrences of substring sub in
    string S[start:end].  Optional arguments start and end are interpreted
    as in slice notation.

To fix my regex soultion,to count overlapping occurrences.

>>> import re
>>> text = "assesses"
>>> sub = "sses"
>>> len(re.findall(r'(?={})'.format(sub), text))
2
snippsat 661 Master Poster

A test with dataset,database as simple as it get.
So here i have a player list that i store in a Sqlite(comes with Python) database.

import dataset

player_lst = [
    {'name': 'Tom', 'score': 250},
    {'name': 'Kent', 'score': 150},
    {'name': 'Paul', 'score': 500}
    ]

db = dataset.connect('sqlite:///mydatabase.db')
table = db['Score_table']
for item in player_lst:
    table.insert(item)

Test it out,by taking out some data.

>>> users = db['Score_table'].all()
>>> for user in db['Score_table']:
...     print user
...     
OrderedDict([(u'id', 1), (u'score', 250), (u'name', u'Tom')])
OrderedDict([(u'id', 2), (u'score', 150), (u'name', u'Kent')])
OrderedDict([(u'id', 3), (u'score', 500), (u'name', u'Paul')])

>>> table.find_one(name='Tom')
OrderedDict([(u'id', 1), (u'score', 250), (u'name', u'Tom')])
>>> table.find_one(name='Tom')['score']
250

It also have all the power of SQL queries if needed.

>>> [i for i in db.query("SELECT MAX(score) FROM Score_table")]
[OrderedDict([(u'MAX(score)', 500)])]
>>> [i for i in db.query("SELECT MAX(score) FROM Score_table")][0]['MAX(score)']
500
Gribouillis commented: cool link +14
snippsat 661 Master Poster
>>> text = "trans panamanian bananas"
>>> text.count('an')
6

--

>>> import re
>>> len(re.findall(r'an', text))
6
snippsat 661 Master Poster

this is the HTML line which returns as soup - I'm after the 1.41 only - which
I hope to return as valueTable

Using .next_sibling can be better.

from bs4 import BeautifulSoup

html = '''\
<td class="yfnc_tablehead1" width="74%">Price/Book (mrq):</td><td class="yfnc_tabledata1">1.41</td>'''

soup = BeautifulSoup(html)
tag = soup.find('td', {'class': 'yfnc_tablehead1'})

Test with parent and nextSibling.

>>> tag
<td class="yfnc_tablehead1" width="74%">Price/Book (mrq):</td>
>>> tag.parent
<td class="yfnc_tablehead1" width="74%">Price/Book (mrq):</td><td class="yfnc_tabledata1">1.41</td>
>>> tag.parent.text
'Price/Book (mrq):1.41'    

>>> tag.nextSibling
<td class="yfnc_tabledata1">1.41</td>
>>> tag.nextSibling.text
'1.41'
>>> float(tag.nextSibling.text) + 1
2.41
snippsat 661 Master Poster

You should use after
Tkinter is running an infinite loop(the event loop).
When you use a while loop and time sleep,then this can lock up(block GUI).

That's why all GUI toolkit has some kind of timer/schedule/thread task,
that dont interfer with the running event loop.

In Wxpython that i have used most,there is wx.Timer, wx.CallLater.
Here is an example i found with Tkinter,that use this method.

import Tkinter as tk

class ExampleApp(tk.Tk):
    def __init__(self):
        tk.Tk.__init__(self)
        self.label = tk.Label(self, text="", width=10)
        self.label.pack()
        self.remaining = 0
        self.countdown(10)

    def countdown(self, remaining = None):
        if remaining is not None:
            self.remaining = remaining    
        if self.remaining <= 0:
            self.label.configure(text="time's up!")
        else:
            self.label.configure(text="%d" % self.remaining)
            self.remaining = self.remaining - 1
            self.after(1000, self.countdown)

if __name__ == "__main__":
    app = ExampleApp()
    app.mainloop()
snippsat 661 Master Poster

the "Price Book or class="yfnc_tabledata1" is in the return respData which is >the source code downloaded from yahoo.ca.

Ok i understand,it's just that i cant find it if search through "respData" or url.