pattern

Pattern is a web mining module for the Python programming language.

It has tools for data mining (Google, Twitter and Wikipedia API, a web crawler, a HTML DOM parser), natural language processing (part-of-speech taggers, n-gram search, sentiment analysis, WordNet), machine learning (vector space model, clustering, SVM), network analysis and <canvas> visualization.

The module is free, well-document and bundled with 50+ examples and 350+ unit tests.

Download

Pattern 2.6 | download (.zip, 23MB) Requires: Python 2.5+ on Windows | Mac | Linux Licensed under BSD Latest releases: 2.6 | 2.5 | 2.4 | 2.3 | 2.2 | 2.1 | 2.0 Authors:  Tom De Smedt (tom at organisms.be)  Walter Daelemans  Reference: De Smedt, T. & Daelemans, W. (2012). Pattern for Python. Journal of Machine Learning Research, 13: 2031–2035. SHA256 checksum of the .zip: 28213f05d94a86d2de1d8a03525d456a9e68dc3b563dc2481ad08fe3db180d02

Modules pattern.web pattern.db pattern.en | es | de | fr | it | nl pattern.search pattern.vector pattern.graph  Helper modules pattern.metrics canvas.js Command-line Command-line interface

Contribute Developer documentation GitHub repository Google group

Installation

Pattern is written for Python 2.5+ (no support for Python 3 yet). The module has no external dependencies, except LSA in the pattern.vector module, which requires NumPy (installed by default on Mac OS X). 

To install Pattern so that the module is available in all Python scripts, from the command line do:

> cd pattern-2.6
> python setup.py install 

If you have pip, you can automatically download and install from the PyPi repository:

> pip install pattern

If none of the above works, you can make Python aware of the module in three ways:

• Put the pattern subfolder in the .zip archive in the same folder as your script.
• Put the pattern subfolder in the standard location for modules so it is available to all scripts:
  c:\python27\Lib\site-packages\ (Windows),
  /Library/Python/2.7/site-packages/ (Mac),

  /usr/lib/python2.7/site-packages/ (Unix).
  
• Add the location of the module to sys.path in your Python script, before importing it:

>>> import sys; sys.path.append('/users/tom/desktop/pattern')
>>> from pattern.web import Twitter 

Quick overview

pattern.web

The pattern.web module is a web toolkit that contains API's (Google, Gmail, Bing, Twitter, Facebook, Wikipedia, Wiktionary, DBPedia, Flickr, ...), a robust HTML DOM parser and a web crawler.

>>> from pattern.web import Twitter, plaintext
>>>
>>> twitter = Twitter(language='en') 
>>> for tweet in twitter.search('"more important than"', cached=False):
>>>    print plaintext(tweet.text)

'The mobile web is more important than mobile apps.'
'Start slowly, direction is more important than speed.'
'Imagination is more important than knowledge. - Albert Einstein'
... 

pattern.en

The pattern.en module is a natural language processing (NLP) toolkit for English. Because language is ambiguous (e.g., I can ↔ a can) it uses statistical approaches + regular expressions. This means that it is fast, quite accurate and occasionally incorrect. It has a part-of-speech tagger that identifies word types (e.g., noun, verb, adjective), word inflection (conjugation, singularization) and a WordNet API.

>>> from pattern.en import parse
>>>  
>>> s = 'The mobile web is more important than mobile apps.'
>>> s = parse(s, relations=True, lemmata=True)
>>> print s

'The/DT/B-NP/O/NP-SBJ-1/the mobile/JJ/I-NP/O/NP-SBJ-1/mobile' ... 

The text has been annotated with word types, for example nouns (NN), verbs(VB), adjectives (JJ) and determiners (DT), word types (e.g., sentence subject SBJ) and prepositional noun phrases (PNP). To iterate over the parts in the tagged text we can construct a parse tree.

>>> from pattern.en import parsetree
>>>  
>>> s = 'The mobile web is more important than mobile apps.'
>>> s = parsetree(s) 
>>> for sentence in s: 
>>>     for chunk in sentence.chunks:
>>>         for word in chunk.words:
>>>             print word,
>>>         print

Word(u'The/DT') Word(u'mobile/JJ') Word(u'web/NN')
Word(u'is/VBZ')
Word(u'more/RBR') Word(u'important/JJ')
Word(u'than/IN')
Word(u'mobile/JJ') Word(u'apps/NNS')

Parsers for Spanish, French, Italian, German and Dutch are also available:
pattern.es | pattern.fr | pattern.it | pattern.de | pattern.nl

pattern.search

The pattern.search module contains a search algorithm to retrieve sequences of words (called n-grams) from tagged text.

>>> from pattern.en import parsetree
>>> from pattern.search import search
>>>  
>>> s = 'The mobile web is more important than mobile apps.'
>>> s = parsetree(s, relations=True, lemmata=True)
>>>
>>> for match in search('NP be RB?+ important than NP', s):
>>>     print match.constituents()[-1], '=>', \
>>>           match.constituents()[0]

Chunk('mobile apps/NP') => Chunk('The mobile web/NP-SBJ-1')

The search pattern NP be RB?+ important than NP means any noun phrase (NP) followed by the verb to be, followed by zero or more adverbs (RB, e.g., much, more), followed by the words important than, followed by any noun phrase. It will also match "The mobile web will be much less important than mobile apps" and other grammatical variations.

pattern.vector

The pattern.vector module is a toolkit for machine learning, based on a vector space model of bag-of-words documents with weighted features (e.g., tf-idf) and distance metrics (e.g., cosine similarity, infogain). Models can be used for clustering (k-means, hierarchical), classification (Naive Bayes, Perceptron, k-NN, SVM) and latent semantic analysis (LSA).

>>> from pattern.web    import Twitter
>>> from pattern.en     import tag
>>> from pattern.vector import KNN, count
>>> 
>>> twitter, knn = Twitter(), KNN()
>>> 
>>> for i in range(1, 10):
>>>     for tweet in twitter.search('#win OR #fail', start=i, count=100):
>>>         s = tweet.text.lower()
>>>         p = '#win' in s and 'WIN' or 'FAIL'
>>>         v = tag(s)
>>>         v = [word for word, pos in v if pos == 'JJ'] # JJ = adjective
>>>         v = count(v) 
>>>         if v:
>>>             knn.train(v, type=p)
>>> 
>>> print knn.classify('sweet potato burger')
>>> print knn.classify('stupid autocorrect')

'WIN'
'FAIL'   

This example trains a classifier on adjectives mined from Twitter. First, tweets with hashtag #win or #fail are mined. For example: "$20 tip off a sweet little old lady today #win". The word part-of-speech tags are parsed, keeping only adjectives. Each tweet is transformed to a vector, a dictionary of adjective → count items, labeled WIN or FAIL. The classifier uses the vectors to learn which other, unknown tweets look more like WIN (e.g., sweet potato burger) or more like FAIL (e.g., stupid autocorrect).

pattern.graph

The pattern.graph module provides a graph data structure that represents relations between nodes (e.g., terms, concepts). Graphs can be exported as HTML <canvas> animations (demo). In the example below, more central nodes (= more incoming traffic) are colored in blue.

>>> from pattern.web    import Bing, plaintext
>>> from pattern.en     import parsetree
>>> from pattern.search import search
>>> from pattern.graph  import Graph
>>>  
>>> g = Graph()
>>> for i in range(10):
>>>     for result in Bing().search('"more important than"', start=i+1, count=50):
>>>         s = r.text.lower() 
>>>         s = plaintext(s)
>>>         s = parsetree(s)
>>>         p = '{NP} (VP) more important than {NP}' 
>>>         for m in search(p, s):
>>>             x = m.group(1).string # NP left
>>>             y = m.group(2).string # NP right
>>>             if x not in g:
>>>                 g.add_node(x)
>>>             if y not in g:
>>>                 g.add_node(y)
>>>             g.add_edge(g[x], g[y], stroke=(0,0,0,0.75)) # R,G,B,A
>>>  
>>> g = g.split()[0] # Largest subgraph.
>>>  
>>> for n in g.sorted()[:40]: # Sort by Node.weight.
>>>     n.fill = (0, 0.5, 1, 0.75 * n.weight)
>>>  
>>> g.export('test', directed=True, weighted=0.6) 

Some relations (= edges) could use some extra post-processing, e.g., in nothing is more important than life, nothing is not more important than life.

Case studies 

Case studies with hands-on source code examples.