What is
Python really?
- Python is an interpreted language. That means that, unlike languages like C and its variants, Python does not need to be compiled before it is run. Other interpreted languages include PHP and Ruby.
- Python is dynamically typed, this means that you don't need to state the types of variables when you declare them or anything like that. You can do things like x=111 and then x="I'm a string" without error
- Python is well suited to object orientated programming in that it allows the definition of classes along with composition and inheritance. Python does not have access specifiers (like C++'s public, private), the justification for this point is given as "we are all adults here“
- in Python, functions are first-class objects. This means that they can be assigned to variables, returned from other functions and passed into functions. Classes are also first class objects
- Writing Python code is quick but running it is often slower than compiled languages. Fortunately， Python allows the inclusion of C based extensions so bottlenecks can be optimized away and often are. The numpy package is a good example of this, it's really quite quick because a lot of the number crunching it does isn't actually done by Python
- Python finds use in many spheres - web applications, automation, scientific modeling, big data applications and many more. It's also often used as "glue" code to get other languages and components to play nice
How
python code is executed
a = "hello“
There are
four steps that python takes when you hit return: lexing, parsing, compiling,
and interpreting.
- Lexing is breaking the line of
code you just typed into tokens.
- The parser takes
those tokens and generates a structure that shows
their relationship to each other (in this case, an Abstract
SyntaxTree).
- The compiler then
takes the AST and turns it into one (or more) code
objects. (function objects, code objects, and bytecode)
- Finally,
the interpreter takes each code object (It
contains information that this interpreter needs to do its job) executes the
code it represents.
**to
understand how Lexing, parser & AST work refer
Python
Variable and Types
·
Python is
completely object oriented
·
you do
not need to declare variable before using them or declare their type
- Every variable
in Python is an Object
- Numbers
Python Support 2 type of
numbers Integer and Floating point number
To define an Integer use x = 7
To define a floating point
number use x = 7.0
- Object Types
Python automatically assign
object type based on the assignment
for example
> x = 7
> type(x)
<type 'int'>
> x =
'abc' > type(x)
<type 'str'>
> x = 7.7
> type(x)
<type 'float'>
Python
Strings
- String
String constants are delimited with “ ” (double
quotes) or ‘ ‘ (single quotes)
>> h = “Hello
World” or >> h = ‘Hello World’
String constant can
contain new line by using double quotes or single quotes.
>> h = “”” Hi
How
are you”””
- String
Concatenation
Use “+” (plus)
>> s = “hi”
>> t = “How are you”
>> print(s + t)
Hi How are you
- Strings are
Immutable
String cannot be modified like in programming
language (like c)
>> h = “Hello world”
>> h[2] = ‘g’
TypeError: ‘str’ object does not support
item assignment.
- String Indexing
and Splicing
String can be indexed like in C
>> h = “hello world”
>> h[0] #
note index starts from zero
H
String
can be sliced
>>h[2:5]
llo
>>h[2:]
llo world
>>h[:2]
he
- String
Interpolation
The mod(%) operator in string is used for
formatting or to insert variable in a string
>> print “There are %d orange in the
basket” % 32
There are 32 orange in the basket
>> print “There are %d %s and %d %s
in the basket” % (32, ‘orange’, 12, ‘apple’)
There are 32 orange and 12 apple in the
basket
Python Strings - Useful Functions
S = “Jack and Jill”
- capitalize()
Capitalizes first letter of the string
>> s.capitalize()
>> Jack And Jill
- count(str,
beg=0, end=len(string))
Count how many times str occurs in a string or
in a substring (if beg and end is given)
>> s.count(‘J’)
2
- find(str, beg=0,
end=len(string))
Determine if str occurs in string or in a
substring of string if starting index beg and ending index end are given return
index if found else -1
>> s.find(‘and’)
>> 5
>> s.find(‘j’)
>> -1
- len()
Return the length of the string
>> s.len()
>> 13
- stripp()
Remove all trailing and leading white space
>> line = “ hi I am with space “
>> line.stripp()
>> hi I am with space
- split()
Splits string according to delimiter str (space
if not provided) and return list of substrings: split into at most num
substrings.
>> s.split()
>> [‘Jack’, ‘and’, ‘Jill’]
- join()
Merges (concatenates) the string representations
of elements in sequence seq into a string, with separator string
>> [‘Jack’, ‘and’, ‘Jill’]
>> Jack and Jill
- replace
Replaces all occurrences of old in string with
new or at most max occurrence if max given
>> value = “aabc”
>> print(value.replace(‘bc’, ‘yz’))
>> aayz
>> print(value.replace(‘a’, ‘x’,
1)) # replace only first occurance
>> xabc
Python List
A List
stored element one after another, in a linear collection.
It handle
any type of element including numbers, string, tuple, list and other
collections
- List are denoted
with []
>> a = [“hi’, ‘How’, ‘are’, ‘you’, ‘9’]
- Like string list
can be sliced, concatenated
>> a[1:3]
>> [‘How’, ‘are’,]
- unlike string,
individual element can be Modified
>> a[2] = “ARE”
>> a
>> [“Hi”, “How”, ‘ARE”, ‘you’, 9]
- Also element can
be Removed
>> a[1:3] = []
>> a
>>[‘Hi’, you’, 9]
- Can be Added anywhere
in the list
>> a[1:1] = [‘a,’, ’b’, ’c’]
>> a
>> [‘Hi’, ‘a,’, ’b’, ’c’ , ‘you’, 9]
- to get Length of
the list , use len
>> len(a)
>> 8
Python List Functions
- append
add element to list
>> a = [2]
>> a.append(1)
>> a
>> [2, 1]
- insert
An element can be appended to another list. With
insert() we can add to the first part or somewhere in the middle of the list
>> a = [‘jack’, ‘jill’]
>> a.insert(1, ‘and’)
>> print(a)
>> [‘jack’, ‘and’, ‘jill’]
- extend
A List can be appended to another list with extend.
So we can extend one list to include another list at its end.
>> a = [1,2,3]
>> b = [4,5,6]
>> a.extend(b)
>> print a
>> [1, 2, 3, 4, 5, 6] # a.ppend(b) will give [1,2,3, [4,5,6]]
- membership
Check whether element exists in the list
>> items = [‘book’, ‘computer’,
‘keys’, ‘mug’]
>> ‘computer’ in items
>> True
- reverse
Reverse the order of the list of the element
>> items.reverse()
>> [‘mug’, ‘keys’, ‘computer’,
‘book’]
- remove
This takes away the first matching element in
the list.
>> items.remove(‘keys’)
>> print(items)
>> [‘mug’, ‘computer’, ‘book’]
- del
This removes element by index or a range of
indices
>> del items[2:]
>> [‘mug’, ‘computer’]
- sort
Sort list elements
>> items = [‘keys’, ‘computer’, ‘book’,
‘mug’]
>> items.sort()
>> [‘book’, ‘computer’, ‘keys’, ‘mug’]
- sort, key
Sometimes elements in a list must be sorted in a
specific way. Here we sort list string elements based on their last characters,
and then their second characters
>> def lastchar(s):
Return s[-1]
>> values = [‘abc’, ‘bca’, ‘cab’]
>> values.sort(key=lastchar)
>> print(values)
>> ['bca', 'cab', 'abc']
- index
Find the index of the element
>> values = [‘abc’, ‘bca’, ‘cab’]
>> values.index(‘bca’)
>> 1
- pop
Remove item at position I and return it. If no
index I is given then remove the first item in the list
>> values = [‘abc’, ‘bca’, ‘cab’]
>> value.pop(1)
>> [‘abc’, ‘cab’]
- count
Return the number of time x appears in the list
>> values = [‘abc’, ‘bca’, ‘cab’]
>> values.count()
>> 3
Python Sets
- Sets are the
list with no duplicate entries. Lets say you want to
collect a list of words used in the paragraph
> print(set("My name is John and John is My name".split()))
>
set(['My', 'name', 'is', 'John', 'and'])
- Sets are a powerful tool
in python since they have the ability to calculate difference and intersection between
other sets. For example, say you have a list of participants in event A
and B
> a = set(['jake', 'john',
'Eric']) >
b = set(['john', 'Jill'])
- To find out
which member attended both events, you may use the "intersection"
method.
> a.intersection(b) or >
b.intersection(a) will give > set(['john'])
- To find out
which member attended only one of the events, use the "symmetric_difference" method
> a.symmetric_difference(b) will
give > set(['jill', 'jake', 'Eric'])
- To find out
which member attended only one event and not the other, use the "difference" method
> a.difference(b) will give
> set(['jake', 'Eric'])
> b.difference(a) will give
> set(['jill'])
- To receive a
list of all participants use the "union" method
> a.union(b) will give >
set(['jill', 'jake', 'john', 'Eric'])
Python Tuple
- A tuple is
an immutable list. A tuple can not be changed in any way
once it is created.
- Tuple don't
have any methods. No append, pop, index, remove or extend
- Tuple is defined
in same way as list, except that the whole set of elements are enclosed
in parentheses instead of square brackets.
- First element of
non empty tuple is access using index 0. t[0]
- Negative indices
count from the end of tuple, just like list.
- Slicing works
too, just like list. Note when you slice list you get new list, when you
slice a tuple you get a new tuple.
- Can use "in" to
see if an element exists in the tuple. e.g. > 'z' in ('a',
'b', 'z') will give > True
- Tuple are faster than
list. if you are defining constant and all you gonna do is iterate then
better to use Tuple instead of list.
- Tuple can be
used as key in dictionary
Python Tuple Operation
- Length
>> len(1,2,3)
>> 3
- Concatenation
>> (1, 2, 3) + (4,5,6)
>> (1,2,3,4,5,6)
- Repetition
>> (‘hi’) * 4
>> (‘hi’, ‘hi’, ‘hi’, ‘hi’)
- Membership
>> 3 in (1, 2, 3)
>> True
- iteration
>> for x in (1,2,3):
Print x
Python Dictionary
- A dictionary is
a data structure that associates a key with a value.
- Defining a
dictionary
>> ages = {}
- Adding key and
values
>> ages(‘Shahruk’) = 50
>> ages(‘Salman’) = 48
>> ages(‘Amir’) = 52
>> ages
>> {‘Shahruk’: 50, ‘Salman’: 48,
‘Amir’: 52}
- del() Delete a
record from a dictionary
>> del ages[‘Amir’]
>> ages
>> {‘Shahruk’: 50, ‘Salman’: 48}
- has_key() -
returns True is exists else False
>> ages.has_keys(‘Shahruk’)
>> True
- keys() - return
all keys of the dictionary as a list
>> ages.keys()
>>[‘Shahruk’, ‘Salman’]
- values() -
return all values of the dictionary as a list
>> ages.values()
>>[50, 48]
- update() -
update one dictionary with other dictionary
>> pet1 = {‘cat’: ‘feline’, ‘dog’: ‘canine}
>> pet2 = {‘dog’: ‘animal’, ‘parakeet’:
‘bird’}
>> pet1.update(pet2)
>> print(pet1)
>> {‘parakeet’: ‘bird’, ‘parakeet’: ‘bird’,
‘cat’: ‘feline’}
- copy - create a
copy of dictionary. Keep original dictionary
as it is.
>> modifiedPet = pet2.copy()
>> modifiedPet
>> {‘dog’: ‘animal’, ‘parakeet’: ‘bird’}
- fromKeys() -
create a dictionary from list of key with same values for all keys
>> keys = [‘bird’, ‘plant’, ‘fish’]
>> d = dict.fromKeys(keys, 5)
>> print d
>> {‘plant’: 5, ‘bird’: 5, ‘fish’: 5}
- dict() - built
in function can construct a dictionary from list of tuples.
>>pairs = [(‘cat’, ‘meow’), (‘dog’, ‘woof’)]
>> d = dict(pairs)
>> print d
>> {‘cat’:
‘meow’, ‘dog’: ‘woof’}
- Create
dictionary from 2 lists
>> keys = [‘a’, ‘b’, ‘c’]
>> values = [1, 2, 3]
>> d = dict(zip(keys, values))
>> d
>> {‘a’: 1, ‘b’: 2, ‘c’: 3}
- keyError, get -
keyError occur when key being access does not exists in dictionary. To
avoid the keyError use "get"
>> animal ={‘cat’: 1, ‘dog’: 2}
>> animal[‘fish’]
Traceback (most recent call last)
File “<pyshell#1>, line 1, in
<module>
Animal[‘fish’]
KeyError: ‘fish’
- items() - gives
list for keys & value tuple
>>animal.items()
>> [(‘cat’, 1), (‘dog’, 2)]
Python Datetime
- A datetime
provide number of ways to deal with dates, times and time interval
- datetime
contains following types
- datetime -
represent a date and time during that day
- date - represent
just a date between year 1 and 9999
- time type -
represent time independent of the date
- timedelta type -
represent the difference between two time or date object
- tzinfo type-
used to implement timezone support for the time and datetime obj.
- Examples
>> import datetime
>> now = datetime.datetime(2016, 9,
12, 21, 41, 43)
>> print now >> 2016-09-12 21:41:43
>> print repr(now) >> datetime.datetime(2016, 9, 12,
21, 41, 43)
>> print now.year, now.month, now.day >> 2016 9 12
>> print now.hour, now.minute,
now.second >> 21 41 43
>> print now.microsecond >> 0
>> print.now.strftime(‘%Y - %m - %d
%H:%M:%S’)
>> ‘2016-09-12 21:41:43’
>>print datetime.datetime.strptime(’12-09-2016’,
‘%d-%m-%Y’)
>> datetime.datetime(2016, 9, 12, 0,
0)
Python Functions
In
python, function is a group of related statements that perform a specific
task
Functions
help break our program into smaller and modular chunks
As
our program grows larger and larger, functions make it more organized and
manageable. Furthermore, it avoids repetition and make code resuable.
Syntax of
Function
def function_name(parameter):
"""docstring"""
statement(s)
return
Above
sown is a function definition which consists of following components
- Keyword def mark
the start of function header
- A function name
is uniquely identify it
- Parameter
(arguments) through which we pass values to a function. they are optional
- A colon
":" marks the end of the function header
- Optional documentation
string (docstring) to describe what the function does
- One or more
valid statements
- optional return
statement to return value from function
- Example
def greet(name):
"""this
function greet to the person"""
print("Hello %s
Good Morning" % name)
- calling function
Once we have defined a function we can call it from
another function, program or even the python prompt. To call a function we
simply type the function name with appropiate parameters
> greet('Prahsant')
> Hello Prashant Good Morning
Python Function Argument
def
greet(name, msg):
"""this function greet to the person"""
print("Hello %s %s" % (name, msg))
greet('Monica',
'Good Evening!')
Here function
has 2 parameters. Since we passed 2 parameters, it ran successfully and we do
not get error.
But we
call function with different number of arguments it will complain
>
greet("Monica")
Type Error
greet() missing 1 required positional argument 'msg'
Default
Argument
def
greet(name, msg='Good Morning'):
"""this function greet to
the person"""
print("Hello %s %s" % (name, msg))
>
greet('Kate')
Hello Kate
Good Morning
>
greet('Diya', 'How are you!')
Hello Diya
How are you!
Note:
For
Positional arguments with Keyword arguments during a function call
Keyword
arguments must be follow positional argument
Having a
positional argument after keyword argument will result into errors
Multiple Function Argument
(*args & **kwargs)
Every
function in python receives a predefined number of arguments if declared
normally like this
def
myfunction(first, second, third):
# some statements
It is
possible to declare function which receive a variable number of arguments
def foo
(first, second, third, *args):
print ("First %s Second %s Third %s" % (first, second, third))
print ("And all the rest ... %s" % list(args))
The
"args" variable is a list of variables, which receives all arguments
which were given to the "foo" function after the first arguments. So
calling foo(1,2,3,4,5) will print out
First 1
Second 2 Third 3
And all
the rest ... [4,5]
it is
also possible to send the functions arguments by keyword, so that the order of
the argument does not matter, using following syntax
def bar
(first, second, third. **options):
if options.get("action") == 'sum':
print "The sum is %d" % (first, second, third)
if
option.get("number") == "first":
return first
result =
(1,2,3,action='sum'. number="first")
print
"Result %d" % result
Code Introspection
Code
introspection is the ability to examine classes, functions and keywords to know
what they are, what they do and what they know.
Python
provides several functions and utilities for code introspection.
help()
dir()
hasattr()
id()
type()
repr()
callable()
issubclass()
isinstance()
__doc__
__name__
Anonymous Function / Lambda
In
python, anonymous function is a function that is defined without a name by
using lambda keyword
While
normal function are defined using the keyword
Lambda
function can have any number of arguments but just one expression.
The
expression is evaluated and returned.
Lambda
function can be used wherever function objects are required.
dd =
lambda x: x ** 2
print(dd(5)
gives
-> 25
Filter, Map & Reduce
Filter
The filter()
function in python takes a function and list of arguments.
Function
is called with all items in the list and a new list is returned which contains
item for which the function evaluate to True
example -
Program to filter out even items from a list using filter() and lambda function
my_list =
[1, 5, 4, 6, 8, 11, 3, 12]
new_list
= filter(lambda x: x%2 ==0, my_list)
print(new_list)
[2,4,6,8,12,16]
Map
The Map()
function in python takes in a function and a list. The function is called with
all item in the list and a new list is returned which contains item return by
that function for each item.
new_list
= map(lambda x: x ** 2, my_list)
print(new_list)
[1, 25,
16, 36, 64, 121, 9, 144]
Reduce
The
function reduce(func, seq) continuously applies the function func to a sequence
seq. It return a single value
reduce(lambda
x,y: x + y, my_list)
> 50
func =
lambda x,y: x if x > y else y
reduce(func,
my_list)
> 12
List comprehensions
- List
comprehension is very powerful tool, which create a new list based on
another list in a single readable line
- List
comprehension is an elegant way to define and create a list in python
- List
comprehension is complete substitute for the lambda function as well as
the functions map(), filter() and reduce()
For
example, get even number form list
numbers =
[1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11]
even_numbers
= []
using for
loop
for
number in numbers:
if number % 2 == 0:
even_number.append(number)
print
even_number
using
filter()
even_number
= filter(lambda x: x%2 ==0, numbers)
using
list comprehension
even_numbers
= [ number for number in numbers if number%2 == 0]
Cross
product of two set
colours =
['blue', 'red']
things =
['house', 'car', 'tree']
coloured_things
= [(x,y) for x in colour for y in things]
print(coloured_things)
[('blue',
'house'), ('blue', 'car'), ('blue', 'tree'), ('red', 'house'), ('red',
'car'), ('red', 'tree')]
Generators
Generators
are very easy to implement, but a bit difficult to understand.
Generators
are used to create iterators, but with a different approach. Generators are
simple functions which return an iterable set of items, one at a time, in a
special way.
When an
iteration over a set of item starts using the for statement, the generator is
run. Once the generator's function code reaches a "yield" statement,
the generator yields its execution back to the for loop, returning a new value
from the set. The generator function can generate as many values (possibly
infinite) as it wants, yielding each one in its turn.
Here is a
simple example of a generator function which returns 7 random integers:
import
random
import
random
random.seed(1)
# seed generate same number everytime
def
lottery():
# returns 6 numbers between 1 and 40
for i in range(6):
print("I am inside")
yield random.randint(1, 40)
print("I am inside but outside for loop")
# returns a 7th number between 1 and 15
yield random.randint(1,15)
# Testing
for
random_number in lottery():
print("I am outside")
print ("And the next number is... %d!" % random_number)
I am
inside
I am
outside
And the
next number is... 9!
I am
inside
I am
outside
And the
next number is... 37!
I am
inside
I am
outside
And the
next number is... 5!
I am
inside
I am
outside
And the
next number is... 17!
I am
inside
I am
outside
And the
next number is... 8!
I am
inside
I am
outside
And the
next number is... 32!
I am
inside but outside for loop
I am
outside
And the
next number is... 13!
import
time
def
fib():
a, b = 1 ,1
while 1:
time.sleep(5)
yield a
a, b = b , a+b
import
types
if
type(fib()) == types.GeneratorType:
print("Nice fib is a generator")
counter =
0
for n in
fib():
print(n)
counter += 1
if counter == 3:
break
Nice fib is a generator
1
1
2
Serialization
Python provides built-in JSON libraries to encode and decode
JSON.
In order to use the json module, it must first be imported:
import json
There are two basic formats for JSON data.
Either in a string or the object data structure.
The object data structure, in Python, consists of lists and
dictionaries nested inside each other.
The object data structure allows one to use python methods
(for lists and dictionaries) to add, list,search and remove elements from the
data structure. The String format is mainly used to pass the data into another
program or load into a data structure.
To load JSON back to a data structure, use the "loads"
method.
This method takes a string and turns it back into
the json object data structure:
print json.loads(json_string)
To encode a data structure to JSON, use the
"dumps" method. This method takes an object and returns a String:
json_string = json.dumps([1, 2, 3, "a",
"b", "c"])
Python supports a Python proprietary data
serialization method called pickle (and a faster alternative called cPickle).
def add_employee(salaries_json, name, salary):
salaries = json.loads(salaries_json)
salaries[name] = salary
return
json.dumps(salaries)
# test code
salaries = '{"Alfred" : 300, "Jane"
: 400 }'
new_salaries = add_employee(salaries,
"Me", 800)
decoded_salaries = json.loads(new_salaries)
print decoded_salaries["Alfred"]
print decoded_salaries["Jane"]
print decoded_salaries["Me"]
print decoded_salaries
Partial functions
You can
create partial functions in python by using the partial function from the
functools library.
Partial
functions allow one to derive a function with x parameters to a function with
fewer parameters and fixed values set for the more limited function.
Import required:
from
functools import partial
Example:
from functools import partial def multiply(x,y): return x * y
# create
a new function that multiplies by 2
dbl =
partial(multiply,2)
print
dbl(4)
Decorators
A
decorator is a special kind of function that either takes a function
and returns a function, or takes a class and returns a class. The @ symbol
is just syntactic sugar that allows you to decorate something in a way
that's easy to read.
Decorators
allow you to make simple modifications to callable objects like functions,
methods, or classes. The syntax
@decorator
def functions(arg):
return "Return"
Is
equivalent to:
def function(arg):
return "Return"
# this
passes the function to the decorator, and reassigns it to the functions
function=decorator(function)
As you
may have seen, a decorator is just another function which takes a functions and
returns one. For example you could do this:
def repeater(old_function):
def new_function(*args, **kwds):
old_function(*args, **kwds) #
we run the old function
old_function(*args, **kwds) #
we do it twice
# we have to return the new_function, or it wouldn't
reassign it to the value
return new_function
This
would make a function repeat twice.
@repeater
def Multiply(num1, num2):
print num1*num2
Multiply(2, 3)
6
6
Decorator
using the inputs
Let's say
you want to multiply the output by a variable amount. You could do
def Multiply(multiplier):
def Multiply_Generator(old_function):
def new_function(*args, **kwds):
return
multiplier*old_function(*args, **kwds)
return new_function
return Multiply_Generator #it returns the new
generator
Now, you
could do
@Multiply(3) # Multiply is not a generator, but
Multiply(3) is
def Num(num):
return num
Function Decorators
A
function decorator is applied to a function definition by placing it on the
line before that function definition begins. For example:
@myDecorator
def
aFunction():
print("inside aFunction")
When
the compiler passes over this code, aFunction() is compiled and the resulting function object
is passed to the myDecorator code,
which does something to produce a function-like object that is then substituted
for the original aFunction()
What
does the myDecorator code look like?
def
myDecorator(f):
def new_f():
print("Inside myDecorator")
f()
print("Finished Executing function")
return new_f
@myDecorator
def
aFunction():
print("inside aFunction")
print("Finsih
decorating")
aFunction()
#
output
Finsih decorating
Inside myDecorator
inside aFunction
Finished Executing function
What
should the decorator do?
Well,
it can do anything but usually you expect the original function code to be used
at some point
When aFunction() is called after it has been decorated,
we get completely different behavior; the my_decorator method is called instead of the
original code. That’s because the act of decoration replaces the original function object with
the result of the decoration – in our case, the my_decorator object replaces aFunction. Indeed, before decorators were added you
had to do something much less elegant to achieve the same thing:
def
aFuncton() ....
myDecorator(aFunction)
With
the addition of the @ decoration operator, you now get
the same result by saying:
@myDecorator
def
aFunction(): pass
@ is just a little syntax sugar
meaning “pass a function object through another function and assign the result
to the original function.”
def
entry_exit(f):
def new_f():
print("Entering", f.__name__)
f()
print("Exited", f.__name__)
return new_f
@entry_exit
def
func1():
print("inside func1()")
@entry_exit
def
func2():
print("inside func2()")
func1()
func2()
print(func1.__name__)
# output
Entering func1
inside func1()
Exited func1
Entering func2
inside func2()
Exited func2
new_f
new_f() is defined within the body
of entry_exit(), so it is created and returned
when entry_exit() is called. Note that new_f() is a closure, because it
captures the actual value of f.
Once new_f() has been defined, it is returned
from entry_exit() so that the decorator mechanism
can assign the result as the decorated function.
The
output of the line print(func1.__name__) is new_f, because the new_f function has been substituted for the
original function during decoration.
you can change the
name of the decorator function before you return it
new_f.__name__ =
f.__name__
Decorator
Function with Arguments
def
decorator_function_with_arguments(arg1, arg2, arg3):
def wrap(f):
print("Inside wrap()")
def wrapped_f(*args):
print("Inside wrapped_f()")
print("Decorator arguments:",
arg1, arg2, arg3)
f(*args)
print("After f(*args)")
return wrapped_f
return wrap
@decorator_function_with_arguments("hello",
"world", 42)
def
sayHello(a1, a2, a3, a4):
print('sayHello arguments:', a1, a2, a3, a4)
print("After
decoration")
print("Preparing
to call sayHello()")
sayHello("say",
"hello", "argument", "list")
print("after
first sayHello() call")
sayHello("a",
"different", "set of", "arguments")
print("after
second sayHello() call")
#
Output
Inside
wrap()
After
decoration
Preparing
to call sayHello()
Inside
wrapped_f()
Decorator
arguments: hello world 42
sayHello
arguments: say hello argument list
After
f(*args)
after
first sayHello() call
Inside
wrapped_f()
Decorator
arguments: hello world 42
sayHello
arguments: a different set of arguments
After
f(*args)
after
second sayHello() call
The return value of
the decorator function must be a function used to wrap the function to be
decorated.
That is, Python will
take the returned function and call it at decoration time, passing the function
to be decorated.
That’s why we have
three levels of functions; the inner one is the actual replacement function.
Because of closures,
wrapped_f() has access to the decorator arguments arg1, arg2 and arg3, without
having to explicitly store them as in the class version(see below).
.
Class as Decorator
Decorator
as a class decoration
mechanisms instead of function is more powerful
The only constraint upon the object returned by the decorator is
that it can be used as function, which basically means it must be callable.
Thus any classes we use as decorator must implement __call__
class
my_decorator(object):
def __init__(self, f):
print("inside
my_decorator.__init__()")
f() # Prove that function definition
has completed
def __call__(self):
print("inside
my_decorator.__call__()")
@my_decorator
def
aFunction():
print("inside aFunction()")
print("Finished
decorating aFunction()")
aFunction()
#
Output
inside
my_decorator.__init__()
inside
aFunction()
Finished
decorating aFunction()
inside
my_decorator.__call__()
Notice that the constructor for my_decorator is executed at the point of decoration of the function.
Since we can call f() inside __init__(), it shows that the creation of f() is complete before the decorator is called.
Note also that the decorator constructor receives the function
object being decorated. Typically, you’ll capture the function object in the
constructor and later use it in the __call__() method (the fact that decoration and calling are two clear
phases when using classes is why I argue that it’s easier and more powerful
this way).
When aFunction() is called after it has been decorated, we get
completely different behavior; the my_decorator.__call__() method is called instead of the original code.
That’s because the act of decoration replaces the original
function object with the result of the decoration – in our case, the my_decorator object replaces aFunction.
Class as
Decorators without Arguments
If we create a
decorator without arguments, the function to be decorated is passed to the
constructor, and the __call__() method is called whenever the decorated
function is invoked:
class
decorator_without_arguments(object):
def __init__(self, f):
"""
If there are no decorator arguments, the function
to be decorated is passed to the constructor.
"""
print("Inside __init__()")
self.f = f
def __call__(self, *args):
"""
The __call__ method is not called until the decorated function is called.
"""
print("Inside __call__()")
self.f(*args)
print("After self.f(*args)")
@decorator_without_arguments
def
sayHello(a1, a2, a3, a4):
print('sayHello arguments:', a1, a2, a3, a4)
print("After
decoration")
print("Preparing
to call sayHello()")
sayHello("say",
"hello", "argument", "list")
print("After
first sayHello() call")
sayHello("a",
"different", "set of", "arguments")
print("After
second sayHello() call")
Any
arguments for the decorated function are just passed to __call__(). The output
is:
Inside
__init__()
After
decoration
Preparing
to call sayHello()
Inside
__call__()
sayHello
arguments: say hello argument list
After
self.f(*args)
After
first sayHello() call
Inside
__call__()
sayHello
arguments: a different set of arguments
After
self.f(*args)
After
second sayHello() call
Notice
that __init__() is the only method called to
perform decoration, and__call__() is called every time you call the
decorated sayHello().
Class as Decorators with
Arguments
The
decorator mechanism behaves quite differently when you pass arguments to the
decorator
Let’s
modify the above example to see what happens when we add arguments to the
decorator:
class
decorator_with_arguments(object):
def __init__(self, arg1, arg2, arg3):
"""
If there are decorator arguments, the function
to be decorated is not passed to the constructor!
"""
print("Inside __init__()")
self.arg1 = arg1
self.arg2 = arg2
self.arg3 = arg3
def __call__(self, f):
"""
If there are decorator arguments, __call__() is only
called
once, as part of the decoration process! You can only give
it a single argument, which is the function object.
"""
print("Inside __call__()")
def wrapped_f(*args):
print("Inside wrapped_f()")
print("Decorator arguments:",
self.arg1, self.arg2, self.arg3)
f(*args)
print("After f(*args)")
return wrapped_f
@decorator_with_arguments("hello",
"world", 42)
def
sayHello(a1, a2, a3, a4):
print('sayHello arguments:', a1, a2, a3, a4)
print("After
decoration")
print("Preparing
to call sayHello()")
sayHello("say",
"hello", "argument", "list")
print("after
first sayHello() call")
sayHello("a",
"different", "set of", "arguments")
print("after
second sayHello() call")
From
the output, we can see that the behavior changes quite significantly
Inside
__init__()
Inside
__call__()
After
decoration
Preparing
to call sayHello()
Inside
wrapped_f()
Decorator
arguments: hello world 42
sayHello
arguments: say hello argument list
After
f(*args)
after
first sayHello() call
Inside
wrapped_f()
Decorator
arguments: hello world 42
sayHello
arguments: a different set of arguments
After
f(*args)
after
second sayHello() call
Now the process of decoration calls the
constructor and then immediately invokes__call__(), which can only take a single argument (the
function object) and must return the decorated function object that replaces
the original. Notice that __call__() is now only invoked once, during
decoration, and after that the decorated function that you return from __call__() is used for the actual calls.
Although this behavior makes sense –
the constructor is now used to capture the decorator arguments, but the
object __call__() can no longer be used as the
decorated function call, so you must instead use __call__() to perform the decoration – it is
nonetheless surprising the first time you see it because it’s acting so much
differently than the no-argument case, and you must code the decorator very
differently from the no-argument case.
Profile
Profiling
a Python program is doing a dynamic analysis that measures the execution time
of the program and everything that compose it. That means measuring the time
spent in each of its functions. This will give you data about where your
program is spending time, and what area might be worth optimizing.
It's
a very interesting exercise. Many people focus on local optimizations, such as
determining
e.g.
which of the Python functions range or xrange is going to be faster.
It
turns out that knowing which one is faster may never be an issue in your
program, and that the time gained by one of the functions above might not be
worth the time you spend researching that, or arguing about it with your
colleague.
Trying
to blindly optimize a program without measuring where it is actually spending
its time is a useless exercise. Following your guts alone is not always
sufficient. Python includes a profiler called cProfile. It not only gives the
total running time, but also times each function separately, and tells you how
many times each function was called, making it easy to determine where you
should make optimizations.
Python
includes a profiler called cProfile. It not only gives the total running time,
but also times each function separately, and tells you how many times each
function was called, making it easy to determine where you should make
optimizations.
Place the
following functions below in order of their efficiency. They all take in a list
of numbers between 0 and 1. The list can be quite long. An example input list
would be
[random.random()
for i in range(100000)]. How would you prove that your answer is correct?
def
f1(lIn):
l1 = sorted(lIn)
l2 = [i for i in l1 if i<0.5]
return [i*i for i in l2]
def
f2(lIn):
l1 = [i for i in lIn if i<0.5]
l2 = sorted(l1)
return [i*i for i in l2]
def
f3(lIn):
l1 = [i*i for i in lIn]
l2 = sorted(l1)
return [i for i in l1 if i<(0.5*0.5)]
Answer
Most to
least efficient: f2, f1, f3. To prove that this is the case, you would want to
profile your code. Python has a lovely profiling package that should do the
trick.
import
cProfile
lIn =
[random.random() for i in range(100000)]
cProfile.run('f1(lIn)')
cProfile.run('f2(lIn)')
cProfile.run('f3(lIn)')
For
completion's sake, here is what the above profile outputs:
>>>
cProfile.run('f1(lIn)')
4 function calls in 0.045 seconds
Ordered by: standard name
ncalls tottime percall cumtime percall
filename:lineno(function)
1 0.009 0.009
0.044 0.044 <stdin>:1(f1)
1 0.001 0.001
0.045 0.045 <string>:1(<module>)
1 0.000 0.000
0.000 0.000 {method 'disable' of '_lsprof.Profiler' objects}
1 0.035 0.035
0.035 0.035 {sorted}
>>>
cProfile.run('f2(lIn)')
4 function calls in 0.024 seconds
Ordered by: standard name
ncalls tottime percall cumtime percall
filename:lineno(function)
1 0.008 0.008
0.023 0.023 <stdin>:1(f2)
1 0.001 0.001
0.024 0.024 <string>:1(<module>)
1 0.000 0.000
0.000 0.000 {method 'disable' of '_lsprof.Profiler' objects}
1 0.016 0.016
0.016 0.016 {sorted}
>>>
cProfile.run('f3(lIn)')
4 function calls in 0.055 seconds
Ordered by: standard name
ncalls tottime percall cumtime percall
filename:lineno(function)
1 0.016 0.016
0.054 0.054 <stdin>:1(f3)
1 0.001 0.001
0.055 0.055 <string>:1(<module>)
1 0.000 0.000
0.000 0.000 {method 'disable' of '_lsprof.Profiler' objects}
1 0.038 0.038
0.038 0.038 {sorted}
Why do we
care?
Locating
and avoiding bottlenecks is often pretty worthwhile. A lot of coding for
efficiency comes down to common sense - in the example above it's obviously
quicker to sort a list if it's a smaller list, so if you have the choice of
filtering before a sort it's often a good idea. The less obvious stuff can
still be located through use of the proper tools. It's good to know about these
tools.
Python Garbage Collection
Python
maintains a count of the number of references to each object in memory.
If a
reference count goes to zero then the associated object is no longer live and
the memory allocated to that object can be freed up for something
else occasionally things called "reference cycles" happen.
The
garbage collector periodically looks for these and cleans them up.
An
example would be
if you
have two objects o1 and o2 such that
o1.x ==
o2 and o2.x == o1.
If o1 and
o2 are not referenced by anything else then they shouldn't be live.
But each
of them has a reference count of 1.
Certain
heuristics are used to speed up garbage collection.
For
example, recently created objects are more likely to be dead. As objects are
created, the garbage collector assigns them to generations.
Each
object gets one generation, and younger generations are dealt with first.
Python's
memory allocation and deallocation method is automatic. The user does not have
to preallocate or deallocate memory by hand as one has to when using dynamic
memory allocation in languages such as C or C++. Python uses two strategies for
memory allocation reference counting and garbage collection.
Prior to
Python version 2.0, the Python interpreter only used reference counting for
memory management.
Reference
counting works by counting the number of times an object is referenced by other
objects in the system.
When references to an object are removed, the
reference count for an object is decremented.
When the reference count becomes zero the object is
deallocated.
Reference
counting is extremely efficient but it does have some caveats. One such caveat
is that it cannot handle reference cycles. A reference cycle is when there is
no way to reach an object but its reference count is still greater than zero.
The easiest way to create a reference cycle is to create an object which refers
to itself as in the example below:
def make_cycle():
list_cycle = [ ]
list_cycle.append(list_cycle)
make_cycle()
Because
make_cycle() creates an object "list_cycle" which refers
to itself, the object "list_cycle" will not automatically be
freed when the function returns. This will cause the memory that "list_cycle"
is using to be held onto until the Python garbage collector is invoked.
Automatic
Garbage Collection of Cycles
Because
reference cycles take computational work to discover, garbage collection must
be a scheduled activity. Python schedules garbage collection based upon a
threshold of object allocations and object deallocations. When the number of
allocations minus the number of deallocations are greater than the threshold
number, the garbage collector is run. One can inspect the threshold for new
objects (objects in Python known as generation 0 objects) by loading the
gc
module
and asking for garbage collection thresholds:
import gc
print "Garbage collection thresholds: %r" %
gc.get_threshold()
Garbage collection thresholds: (700, 10, 10)
Here we
can see that the default threshold on the above system is 700. This means when
the number
of
allocations vs. the number of de-allocations is greater than 700 the automatic
garbage collector will run.
ClassMethod
@classmethod
function also callable without instantiating the class, but its definition
follows sub class, not parent class via inheritance thats because the first
argument for @classmethod function must always be a cls (class)
use @classmethod to
change the behavior of the method based on which subclass is calling the
method.
Class
method are Mutable.
class Date(object):
def __init__(self, day=0, month=0,
year=0):
self.day = day
self.month = month
self.year = year
Let's
assume that we want to create a lot of Date class instances having date
information coming from outer source encoded as a string of next format
('dd-mm-yyyy'). We have to do that in different places of our source code in
project. So what we must do here is:
Parse a string to receive day,
month and year as three integer variables or a 3-item tuple
consisting of
that variable.
Instantiate Date by passing
those values to initialization call.
This will
look like:
day, month, year = map(int, string_date.split('-'))
date1 = Date(day, month, year)
For this
purpose, C++ has such feature as overloading, but Python lacks that feature- so
here's when classmethod applies
@classmethod
def from_string(cls, date_as_string):
day, month, year = map(int,
date_as_string.split('-'))
date1 = cls(day, month, year)
return date1
date2 = Date.from_string('11-09-2012')
Static Method
@staticmethod function
is nothing more than a function defined inside a class. It is callable without
instantiating the class first.
It’sdefinition
is immutable via inheritance
using
static you would want the behavior to remain unchanged across subclasses.
StaticMethod
are immutable
class Date:
def __init__(self, month, day, year):
self.month = month
self.day = day
self.year = year
def display(self):
return
"{0}-{1}-{2}".format(self.month, self.day, self.year)
@staticmethod
def millenium(month, day):
return Date(month,
day, 2000)
new_year
= Date(1, 1, 2013)
# Creates a new Date object
millenium_new_year
= Date.millenium(1, 1) # also creates a Date object.
new_year.display()
#
"1-1-2013“
millenium_new_year.display()
# "1-1-2000“
isinstance(new_year,
Date)
# True
isinstance(millenium_new_year,
Date) #True
ClassMethod and StaticMethod
class Hero:
@staticmethod
def say_hello():
print("Helllo...")
@classmethod
def say_class_hello(cls):
if(cls.__name__=="HeroSon"):
print("Hi Kido")
elif(cls.__name__=="HeroDaughter"):
print("Hi Princess")
class HeroSon(Hero):
def say_son_hello(self):
print("test hello")
class HeroDaughter(Hero):
def say_daughter_hello(self):
print("test hello
daughter")
testson =
HeroSon()
testson.say_class_hello()
#
prints Hi Kido
testson.say_hello()
# Helllo...
testdaughter
= HeroDaughter()
testdaughter.say_class_hello()
# Hi Princess
testdaughter.say_hello()
# Helllo...
Property
One
simple use case will be to set a read only instance attribute , as you know leading
a variable name with one underscore _x in python usually mean it's private
(internal use) but sometimes we want to be able to read the instance attribute
and not to write it so we can use property for this:
class C(object):
def __init__(self, x):
self._x = x
@property
def x(self):
return self._x
>>>
c = C(1)
>>>
c.x
1
>>>
c.x = 2
AttributeError
Traceback (most recent call last) AttributeError: can't set
attribute Property
class C(object):
def __init__(self):
self._x = None
def getx(self):
return self._x
def setx(self, value):
self._x = value
def delx(self):
del self._x
x = property(getx, setx, delx, "I'm the
'x' property.")
Property
using "property" decorator
class C(object):
def __init__(self):
self._x = None
@property
def x(self):
"""I'm
the 'x' property."""
return self._x
@x.setter
def x(self, value):
self._x = value
@x.deleter
def x(self):
del self._x
Iterator
There are
4 ways to build a iterative function
- create a
generator (uses the yield keyword)
- use a generator
expression (genexp)
- Make class an
iterator (defines __iter__ and __next__)
- create a
function that python can iterator over on its own (define __getitem__)
# 1.
using generator
def uc_gen(text):
for char in text:
yield char.upper()
#
2 generator expression
def uc_genexp(text):
return (char.upper() for char in text)
# 3 class
class uc_iter():
def __init__(self, text):
self.text = text
self.index=0
def __iter__(self):
return self
def next(self):
try:
result =
self.text[self.index].upper()
except indexError:
raise
StopIteration
self.index += 1
return result
def __reversed__(self):
return reversed(self.text)
# 4
getitem method
class uc_getitem():
def __init__(self, text):
self.text = text
def __getitem__(self, index):
result =
self.text[index].upper()
return result
def __len__(self):
return len(self.text)
# usage
for iterator in uc_gen, uc_genexp, uc_iter, uc_getitem:
for ch in iterator('abcde'):
print ch
iterator Class
Iterator
objects in python conform to the iterator protocol, which basically means they
provide two methods __iter__() and next()
- The __iter__
returns the iterator object and is implicitly called at the start of the
loop
- The __next__()
method returns the next value and is implicitly called at each loop
increment
- __next__()
raises a StopIteration exception when there are no more value to return,
which is implicitly captured by looping constructs to stop iterating
- __reversed__
will need this method if we want to implement reversed
class Counter:
def __init__(self, low, high):
self.current = low
self.high = high
def __iter__(self):
return self
def __next__(self): # In PY2 def
next(self):
if self.current >
self.high:
raise
StopIteration
else:
self.current +=1
return self.current - 1
# usage
[c for c
in Counter(3,8)]
# output
[3, 4, 5, 6, 7, 8]
Context
Manager
The most
common and important use of the context manager is to roperly manage the
resources
The act
of opening a file, consume a resource (called a file descriptor) and this
resource is limited by OS. That is to say, there are a maximum number of files
a process can have open at one time
files = []
for x in range(100000):
files.append(open('foo.txt', 'w'))
This will
give error
OSError
[Errno 24] Too many open files: 'foo.txt'
So
simpler and better way of completing program is to close file.
files = []
for x in range(100000):
f = open('foo.txt', 'w')
f.close()
files.append(f)
it is
difficult to make sure that close() is called on every file opened, especially
if the file is in a function that may raise an exception or has multiple return
path
In other
languages, developers are forced to use try...except...finally every time they
work with a file (or any other type of resource that needs to be closed like
socket, db connection)
for such
cases python has provided context manager which is achived by using
"with"
with open("somefile.txt", 'r') as infile:
for line in infile:
print ('> {}'.format(line))
Think of
"with" as creating a mini-function
We can
use the variable freely in the indented portion, but once the clock end the
variable (infile in this example) goes out of scope
It
automatically calls a special method that contains the code to clean up the
resource.
But where
is the code that is actually being called when the variable goes out of scope.
Context Manager Class
Lets try
to implement a context manager using a class
A context
manager must implement two methods to collaborate properly with the
"With" statement
__enter__
This
method is called on entry to the with statement. The value returned by this
method function will be the value assigned to the as variable
__exit__
This
method is called on exit from the with statement.
it does
any cleanup work and returns nothing.
If the
type, value or traceback parameter have a values of None, then this is a normal
conclusion of the statement
If it is
not a None the exception is raised
lets see
a example
class File():
def __inti__(self, filename, mode):
self.filename = fileanme
self.mode = mode
def __enter__(self):
self.open_file =
open(self.filename, self.mode)
return self.open_file
def __exit__(self, *args):
self.open_file.close()
files = []
for _ in range(100000):
with File('foo.txt','w') as infile:
infile.write('foo')
files.append(infile)
One more
example of making class as context manager
class SessionContextManager(object):
def __enter__(self):
self.session = Session()
self.session.execute("begin transaction")
return self.session
def __exit__(self, exc_type, exc_value,
exc_traceback):
if exc_type:
self.session..rollback()
return # propogate exception
self.session.commit()
self.session.close()
Context Manager using contextlib
Given the
context manager are so helpfull, they were added to the standard Library in
number of places
LockObjects
in threading are contect manager
zipfile.Zipfiles
subprocess.Popen
tarfile.TarFile
telnetlib.Telnet
pathlib.Path
One nice
way of creating a context manager using function is to use decorator
@contextmanager
To use
it, decorate a generator function that calls yield exactly onces
Everything
before the call to yield is considered the code for __enter__()
Everything
after Yield is considered as code for __exit__()
from contextlib import contextmanager
@contextmanager
def SessionContextManager()
session = Session()
try:
session.execute('begin
transaction')
yield session
except Exception:
session.rollaback()
raise
else:
session.commit()
finally:
session.close()
making
class as context manager decorator
from contextlib import ContextDecorator
class makeparagraph(ContextDecorator):
def __enter__(self):
print('<p>')
return self
def __exit__(self, *exc):
print('<p>')
return False
@makeparagraph()
def emit_html():
print('Here is some non-HTML')
emit_html()
#output
<p>
Here is some non-HTML
<p>
