Django/Python app memory usage - examples
Examples of memory usage problems (currently: whoosh searching, sitemaps generation)
WhooshAfter launching my new website - JobMaster.rk.edu.pl, that contained some technologies used by me for the first time I've encountered few memory usage issues on the production server (Nginx, FastCGI). Whoosh used for full text searching was leaking memory when searching. The code was based on arnebrodowski.de example. The problem was with the searching view, line with: searcher = ix.searcher(), which was leaking at every search ("memstat -v" was used to check memory usage). Solution for this was to move to settings.py so this object won't be created for every request (as you can see it can be "reused", as it doesn't depend on the search term). So in settings.py I've placed all "reusable" code: The "reuse" solution came up when I was testing whoosh with a simple server-client setup on localhost. Here is whoosh-client.py: The leaking server, whoosh-server-bad.py: And the not leaking whoosh-server.py with "ix.searcher()" moved from the while loop: Start the bad server in one terminal, then in second use ps -aux to check memory usage, and in next terminal launch whoosh-client. The memory used by server should increase every client run. If you use the good server memory usage won't go up.
SitemapsIt looks like if you use Django built-in Sitemaps framework to make big sitemaps it will use a lot of RAM for something. This time I had really big sitemap - about 9000 elements, and 1,7MB sitemap.xml file. However when I requested the sitemap after server restart - memory usage boosted from 6 to 105MB. Every next request did not increase the memory usage. Solution was to server static sitemap.xml file.