So I have been playing around with some Python code for a while and I want to teach you some tricks to improve the speed of your Python based software. The performance can almost be felt immediately, especially in project where you have to crunch a lot of numbers, computational problems, speed is essential.
1) Numpy
You should use the Numpy package, always for array management. It is at least 50% quicker in my experience, and we are talking about big computations like 1,000,000+ iterations. It’s also reportedly quicker than Pandas, so it’s definitely very important to use this instead of the native language.
For example an array declaration:
ARRAY=[10]
vs Numpy
ARRAY = numpy.zeros(10, dtype=float)
Unfortunately I don’t think you can create unassigned arrays, so you must get the size first, and you should specify the type as well. So in this case it’s faster since the type of the array is set beforehand and you don’t need to set very single element to float after it, so this makes it ultra fast.
So it’s an efficient package, low level C based, and actively maintained and bug-tested, so it’s very suitable for big calculations.
2) Don’t convert variable types
You should avoid converting variable types after declaration as much as possible, I have observed that converting variables slows down the code by 60% or more.
For instance people might enforce the float type on a variable, because I think Python 2.x.x was using fewer decimals than the float type by default so many people could have, including me, used float types on a variable uselessly like say:
A=777
B=666
print(float(A)/float(B))
Now this particular code in Python 2.x.x and certainly in 3.x.x is very inefficient and slow. I know for a fact since I used to divide numbers like this myself, like a novice, not realizing how inefficient this code was. Or perhaps you just turned the denominator into float, which is 1 less operation, but still very inefficient.
Instead the correct code for this operation is this:
A=777.0
B=666.0
print (A/B)
Yes it’s that simple, and it’s bare minimum 60% quicker. Basically if you add a . digit mark and a zero after that, then it’s automatically declared as float, so the fraction can be alreadily calculated.
Now of course this is a trivial example, but with arrays it’s literally impossible to avoid this without Numpy. Take the this code for instance:
TEXT="777,666"
ARRAY=TEXT.strip().split(',')
FRAC=ARRAY[0]/ARRAY[1]
print(FRAC)
Say we want to read in a .csv file delimited by a comma and we divide the first column by the second one, so 777/666. Well of course this code doesn’t run, it returns an error. Why? Because it recognizes the input as string, so we must convert it into float here, there is no alternative. The correct code would look like this:
TEXT="777,666"
ARRAY=TEXT.strip().split(',')
FRAC=float(ARRAY[0])/float(ARRAY[1])
print(FRAC)
So you can’t escape the float conversion, so as I thought, but this code is literally 100-200% slower than it should have been. Why? Because we transform the variable, and we use the default array.
So the efficient version of this code like this:
import numpy
TEXT="777,666"
ARRAY=numpy.array(TEXT.strip().split(','), dtype=float)
FRAC=ARRAY[0]/ARRAY[1]
print(FRAC)
This code should be massively quicker than anything else, because we already make the array float, so the string gets inserted there as a float so we don’t need to convert again during the computation.
This performance boost can be felt especially in long calculations, so if you are doing big math computations, this is definitely the way to do it.
3) Style vs Performance
It’s well known that repetitive code can be put inside a function and using the function as a reference to invoke it makes the code shorter, more stylish and easier to debug, however it slows it down.
So if you want ultra fast code then you must write everything out, can’t use many function and can’t use nested loops, I found them very inefficient and slow.
So for instance:
array=[1,1,1,1,1,1,1,1,1,1]
sum=0
for a in range(0,10):
sum+=array[a]
print(sum) # it will be 10
Is much much slower than writing it out:
array=[1,1,1,1,1,1,1,1,1,1]
sum=array[0]+array[1]+array[2]+array[3]+array[4]+array[5]+array[6]+array[7]+array[8]+array[9]
print(sum) # it will be 10
I mean literally this could be like a 200% speed boost if you write it out like this instead of using loops. So whenever you can, and style doesn’t matter, writing things out like this can speed it up by a huge degree.
Of course if you write a too long code, then the probability of making mistakes grows, for instance I could have easily switch the 4 with a 5 accidentally, of course for an addition it doesn’t matter since it’s commutative, but for a division or a more complex calculation it does.
So it’s a pain in the ass, but you could perhaps verify the code by some other means, but if performance is a must, then writing the longer version of the code is always quicker in my experience.
Conclusion
So these are my observations, I have been doing statistical analysis on BTC markets, and believe me the computation takes a lot of time, sometimes up to 3 hours per model, so using every bit of efficiency helps cut down the computation speed by a lot. Hopefully this information can help you too.
Sources:
- Python logo by Python Software Foundation, Python and the Python logo is a trademark of Python Software Foundation
I can't agree with what you wrote in "Style vs Performance". Your example seems like a classic example of premature optimization.
Especially this sentence:
Style matters a lot. Assuming that what you write is not a single-use program that'll be forgotten after execution, your code will be read much, much more often than written or refactored. This is even more true when you collaborate with others - then you can either focus on style first, or write terribly styled code and spend thrice as time on writing comments or docs, or be a jerk to your coworkers and write just the terribly styled code.
And as for your example, I measured it:
$ python -m timeit 'array=[1,1,1,1,1,1,1,1,1,1] sum=0 for a in range(0,10): sum+=array[a]' 1000000 loops, best of 3: 0.802 usec per loop $ python -m timeit 'array=[1,1,1,1,1,1,1,1,1,1] sum=array[0]+array[1]+array[2]+array[3]+array[4]+array[5]+array[6]+array[7]+array[8]+array[9]' 1000000 loops, best of 3: 0.333 usec per loop0.802µs vs 0.333µs. Yeah, it's slower. Does it matter? Most likely not. What matters is that second example is much less readable and more error-prone (which you even admit yourself).
Yes but it matters to me though, I was analyzing the market and speed is very crucial, a default calculation takes like 2 hours.
Besides the long string can be automatically written too:
string="sum=array[0]" for i in range(1,9+1): string+="+array["+str(i)+"]" print (string)This way we avoid typos and mistakes.
...and then you'd have to
evalthat long string to get an array which would be even slower, not to say completely unsafe 😉If your calculation takes 2 hours, then I'd say replacing
forloops with manually typed array accesses is not the best place to optimize. Most likely something else takes long to finish.Have you tried profiling your code?
Have you thought about converting your analysis to c++?
Python is loads easier to code, but c++ and similar level languages blow it out of the water in terms of speed, since Python has an interpreted implementation compared to a compiled implementation of c++ style languages.
It would complicate the compilation process. For now I like Python, besides I have already made my code as efficient as possible.