Posts tagged Python
Hello world!
- 01 abril 2024
Welcome to my new blog! I’ll gradually describe all my adventures with self-hosting of various services, including blog/website, files and photos.
I started my website/blog in 2014. It was served using wordpress that was set up on virtual machine (VM) in google cloud.
I felt for a long time that running a dedicated VM for a blog is an overkill. There was huge overhead in maintaining both, the VM (regular system updates and backups) as well as wordpress itself (updates and database backups).
Python code profiling and accelerating your calculations with numba
- 31 agosto 2020
You wrote up your excellent idea as Python program/module but you are unsatisfied with its performance. The chances are high most of us have been there at least once. I’ve been there last week.
I found excellent method for outlier detection (Enhanced Isolation Forest). eIF was initially written in Python and later optimised in Cython (using C++). C++ is ~40x faster than vanilla Python version, but it lacks the possibility to save the model (which is crucial for my project). Since adding model saving to C++ version is rather complicated buisness, I’ve decided to optimise Python code. Initially I hoped for ~5-10x speed improvement. The final effect surprised me, as rewritten Python code was ~40x faster than initial version matching C++ version performance!
How is it possible? Speeding up your code isn’t trivial. First you need to find which parts of your code are slow (so-called code profiling). Once you know that, you can start tinkering with the code itself (code optimisation).
Connecting to MySQL without password prompt
- 13 enero 2016
If you are (like me) annoyed by providing password at every mysql login, you can skip it. Also it makes easier programmatic access to any MySQL db, as not passwd prompting is necessary :)
Create ~/.my.cnf file:
And login without -p parameter:
Multiprocessing in Python and garbage collection
- 14 junio 2014
Working with multiple threads in Python often leads to high RAM consumption. Unfortunately, automatic garbage collection in child processes isn’t working well. But there are two alternatives:
When using Pool(), you can specify number of task after which the child will be restarted resulting in memory release.
If you use Process(), you can simply delete unwanted objects and call gc.collect() inside the child.
Note, this may slow down your child process substantially!
TAR random access
- 23 abril 2014
I was often challenged with accessing thousands/millions files from network file system (NFS). As I update some of the stored files once in a while, I have decided to store these files in multiple TAR archives. The data complexity was therefore reduced. But still, there was an issue with random access to the files within each archive.
First, I had a look at tar indexer. Its simplicity is brilliant.
Yet, it stores index in raw text file and it can handle only single tar file.
Therefore, I have ended up writing my own tar_indexer
tool using sqlite3 for index storing and allowing indexing of multiple tar archives.
This can be easily incorporated into any Python project.
Note, only raw (uncompressed) tar files are accepted as native tar.gz cannot be random accessed.
But you can compress each file using zlib before adding it to tar. At least, this is what I do.