自己动手做一个搜索引擎（三）——利用web.py建立搜索引擎

Web Frameworks for Python

A Web framework is a collection of packages or modules which allow developers to write Web applications or services without having to handle such low-level details as protocols, sockets or process/thread management.

The majority of Web frameworks are exclusively server-side technology, although, with the increased prevalence of AJAX, some Web frameworks are beginning to include AJAX code that helps developers with the particularly tricky task of programming (client-side) the user’s browser. At the extreme end of the client-side Web Frameworks is technology that can use the web browser as a full-blown application execution environment (a la gmail for example): see Web Browser Programming for details.

As a developer using a framework, you typically write code which conforms to some kind of conventions that lets you “plug in” to the framework, delegating responsibility for the communications, infrastructure and low-level stuff to the framework while concentrating on the logic of the application in your own code. This “plugging in” aspect of Web development is often seen as being in opposition to the classical distinction between programs and libraries, and the notion of a “mainloop” dispatching events to application code is very similar to that found in GUI programming.

Generally, frameworks provide support for a number of activities such as interpreting requests (getting form parameters, handling cookies and sessions), producing responses (presenting data as HTML or in other formats), storing data persistently, and so on. Since a non-trivial Web application will require a number of different kinds of abstractions, often stacked upon each other, those frameworks which attempt to provide a complete solution for applications are often known as full-stack frameworks in that they attempt to supply components for each layer in the stack.

Many frameworks now provide an element of customization in their support for the above activities and abstractions, utilizing components in that they provide abstractions only for certain specific things. As a result, it can be possible for you to build your own full-stack framework almost entirely from existing components.

Popoular Frameworks

web.py

Although there are so many popular and easy frameworks, we just need to learn web.py and use it to build a search engine.

Installation

web.py 0.38 is the latest released version of web.py. You can install it by running:

pip install web.py

Or to get the latest development version from git:

git clone git://github.com/webpy/webpy.git
ln -s `pwd`/webpy/web .

URL Handling

The most important part of any website is its URL structure. Your URLs aren’t just the thing that your visitors see and email to their friends, they also provide a mental model of how your website works. On popular sites like del.icio.us, the URLs are even part of the user interface. web.py makes it easy to make great URLs.

To get started with your web.py application, open up a new text file (let’s call it code.py) and type:

import web

This imports the web.py module.

Now we need to tell web.py our URL structure. Let’s start out with something simple:

urls = (
  '/', 'index'
)

The first part is a regular expressions that matches a URL, like /, /help/faq, /item/(\d+), etc. (i.e. \d+ would match a sequence of digits). The parentheses say to capture that piece of the matched data for use later on. The second part is the name of a class to send the request to, like index, view, welcomes.hello (which gets the hello class of the welcomes module), or get_\1. \1 is replaced by the first capture of your regular expression; any remaining captures get passed to your function.

This line says we want the URL / (i.e. the front page) to be handled by the class named index.

Get and POST: the difference

Now we need to write the index class. While most people don’t notice it just browsing around, your browser uses a language known as HTTP for communicating with the World Wide Web. The details aren’t important, but the basic idea is that Web visitors ask web servers to perform certain functions (like GET or POST) on URLs (like / or /foo?f=1).

GET is the one we’re all familiar with, the one used to request the text of a web page. When you type harvard.edu into your web browser, it literally asks the Harvard web server to GET /. The second-most famous, POST, is often used when submitting certain kinds of forms, like a request to purchase something. You use POST whenever the act of submitting a request does something (like charge your credit card and process an order). This is key, because GET URLs can be passed around and indexed by search engines, which you definitely want for most of your pages but definitely don’t want for things like processing orders (imagine if Google tried to buy everything on your site!).

In our web.py code, we make the distinction between the two clear:

class index:
    def GET(self):
        return "Hello, world!"

This GET function will now get called by web.py anytime someone makes a GET request for /.

Now we need to create an application specifying the urls and a way to tell web.py to start serving web pages:

if __name__ == "__main__":
    app = web.application(urls, globals())
    app.run()

First we tell web.py to create an application with the URLs we listed above, looking up the classes in the global namespace of this file. And finally we make sure that web.py serves the application we created above.

Now notice that although I’ve been talking a lot here, we only really have five or so lines of code. That’s all you need to make a complete web.py application.

For easier access, here’s how your code should look like:

import web

urls = (
    '/', 'index'
)

class index:
    def GET(self):
        return "Hello, world!"

if __name__ == "__main__":
    app = web.application(urls, globals())
    app.run()

Start the Server

If you go to your command line and type:

$ python code.py
http://0.0.0.0:8080/

You now have your web.py application running a real web server on your computer. Visit that URL and you should see “Hello, world!” (You can add an IP address/port after the “code.py” bit to control where web.py launches the server. You can also tell it to run a fastcgi or scgi server.)

Note: You can specify the port number to use on the command line like this if you can’t or don’t want to use the default:

$ python code.py 1234

Templating

Writing HTML from inside Python can get cumbersome; it’s much more fun to write Python from inside HTML. Luckily, web.py makes that pretty easy.

Let’s make a new directory for our templates (we’ll call it templates). Inside, make a new file whose name ends with HTML (we’ll call it index.html). Now, inside, you can just write normal HTML:

<em>Hello</em>, world!

Or you can use web.py’s templating language to add code to your HTML:

$def with (name)

$if name:
    I just wanted to say <em>hello</em> to $name.
$else:
    <em>Hello</em>, world!

As you can see, the templates look a lot like Python files except for the def with statement at the top (saying what the template gets called with) and the $s placed in front of any code. Currently, template.py requires the $def statement to be the first line of the file. Also, note that web.py automatically escapes any variables used here, so that if for some reason name is set to a value containing some HTML, it will get properly escaped and appear as plain text. If you want to turn this off, write $:name instead of $name.

Now go back to code.py. Under the first line, add:

render = web.template.render('templates/')

This tells web.py to look for templates in your templates directory. Then change index.GET to:

name = 'Bob'
return render.index(name)

(‘index’ is the name of the template and ‘name’ is the argument passed to it)

Visit your site and it should say hello to Bob.

But let’s say we want to let people enter their own name in. Replace the two lines we added above with:

i = web.input(name=None)
return render.index(i.name)

Visit / and it should say hello to the world. Visit /?name=Joe and it should say hello to Joe.

Of course, having that ? in the URL is kind of ugly. Instead, change your URL line at the top to:

'/(.*)', 'index'

and change the definition of index.GET to:

def GET(self, name):
    return render.index(name)

and delete the line setting name. Now visit /Joe and it should say hello to Joe.

Forms

The form module of web.py allows the ability to generate html forms, get user input, and validate it before processing it or adding it to a database.

The form module defines 2 primary classes: the Form class, and the Input class. Forms are instantiated with one or more inputs, and optional validators. Inputs are instantiated with a name variable, and optional arguments and validators. The Input class is subclassed into the following html inputs (html type in parens):

• Textbox - free form single line input (input type=”text”)
• Password - free form single line that hides input (input type=”password”)
• Textarea - free form multi line input (textarea)
• Dropdown - mutually exclusive input for lists (select and options)
• Radio - mutually exclusive input for a few options (input type=”radio”)
• Checkbox - binary input (input type=”checkbox”)
• Button - submit the form (button) A basic login form would look like this:

login = form.Form(
    form.Textbox('username'),
    form.Password('password'),
    form.Button('Login'),
)

This defines a basic form. Once defined, you should call it again to get an copied instance, and then you can call the render method on it, like so:

f = login()
print f.render()

This outputs the following HTML:

<table>
    <tr><th><label for="username">username</label></th><td><input type="text" id="username" name="username"/><div class="post" style="display: none;"></div></td></tr>
    <tr><th><label for="password">password</label></th><td><input type="password" id="password" name="password"/><div class="post" style="display: none;"></div></td></tr>
    <tr><th><label for="Login"></label></th><td><button id="Login" name="Login">Login</button><div class="post" style="display: none;"></div></td></tr>
</table>

Exercise

使用web.py，结合前面学习的HTML, Lucene, 中文分词等知识点，根据上次实验爬取的网页，建立一个简单的搜索引擎。

You can see my code here