django

Archived Posts from this Category

django db mocks - thoughts

Posted by Andrey Khavryuchenko on 30 Jun 2007 | Tagged as: Blog, django

I hate hitting database in my unit tests.

Even if it’s tiny, nearly empty and easy to fill. Things only become worse when you have to test lots different cases that contradict each other and yet are small enough to justify a separate database fixture.

Thus, tonight I’m fighting my laziness to decipher django.db.models enough to be able to create slim and easy db mocks inside a nose unit test.

So far is far too late for my brain to spin at full speed and no code was produced. Yet I’ve come down to two distinct ways to implement this:

  1. Completely mock django.db.models.query._QuerySet to use in-test dataset and respond to various queries
  2. Construct in-memory python SQLite and replace application connection with the one of the unit test.

Definitely the second way is faster to either implement or fail. So will it be.

Watch for updates with working code..

3 hints re migrating django app to unicode

Posted by Andrey Khavryuchenko on 06 Jun 2007 | Tagged as: Blog, django

Last two days I’ve spent fixing Djiggit wrt international, esp cyrillics feeds.

Here are 3 points that I’ve learned:

  1. Read and follow instructions at docs/unicode.txt and communicate on the django-users list
  2. Use latest db adaptors
  3. Beware of your database encoding

Long story.

It took me several hours to trace why cyrillics is stored as ‘????’ in the database. Starting from top - my code - to the database itself.

Distilled down for mysql:

  • make sure you’re using latest python-MySQLdb: earlier versions have known problems with utf8
  • make sure that your db server is started with using utf8 charset in mind
  • backup your database as json using

     manage.py dumpdata > .json
  • recreate your database making sure it has “utf8″ character set and collation:

    mysql> show variables like 'ch%';
    +--------------------------+----------------------------+
    | Variable_name            | Value                      |
    +--------------------------+----------------------------+
    | character_set_client     | latin1                     |
    | character_set_connection | latin1                     |
    | character_set_database   | utf8                       |
    | character_set_filesystem | binary                     |
    | character_set_results    | latin1                     |
    | character_set_server     | utf8                       |
    | character_set_system     | utf8                       |
    | character_sets_dir       | /usr/share/mysql/charsets/ |
    +--------------------------+----------------------------+
    8 rows in set (0.00 sec)
  • load your data back from json dump:

    manage.py loaddata 

Happy hacking!

django not ready for unicode out-of-box?

Posted by Andrey Khavryuchenko on 05 Jun 2007 | Tagged as: Blog, django

Djiggit is build on the Django unicode branch and yet it require non-obvious solutions to everyday tasks.

Imagine you’re trying to present a (cyrillic) tag in url-ready form. Usually this is solved by

{{ tagname|urlencode }}

But when the ‘tagname’ contains non-ascii symbols, urlencode barfs:

KeyError at /
u'\u0420'
Request Method:     GET
Request URL:    http://localhost:8088/
Exception Type:     KeyError
Exception Value:    u'\u0420'
Exception Location:     /usr/lib/python2.4/urllib.py in quote, line 1117
Template error

My current (temporary) solution is to encode this manually, in python:

urllib.quote(tagname.encode('utf8')

But this is not DRY and makes me think there’s a better way.

Perhaps it’s time to add a custom filter and push it into django codeline.

testing django with twill and nosetests

Posted by Andrey Khavryuchenko on 04 Jun 2007 | Tagged as: Blog, django

Just a little writeup..

Now have a pleasure of not starting django development webserver, nor using apache to do http-level tests in my project.

Thanks to twill and its wsgi integration.

Works smoothly now, but caused me little cursing when twill refused to see rss feed.

After digging the twill code I’ve changed a single line - still don’t know how correct is this:

--- /usr/lib/python2.4/site-packages/twill-0.9b1-py2.4.egg/twill/wsgi_intercept.py.orig 2007-06-04 21:10:37 +0300
+++ /usr/lib/python2.4/site-packages/twill-0.9b1-py2.4.egg/twill/wsgi_intercept.py      2007-06-04 21:10:45 +0300
@@ -265,7 +265,7 @@
                 for data in self.write_results:
                     self.output.write(data)

-            if generator_data:
+            if generator_data is not None:
                 self.output.write(generator_data)
                 for data in self.result:
                     self.output.write(data)

Nevertheless it didn’t break anything and I have a luxury to test rss feeds too:

# -*- coding: utf-8 -*-
# $Id: tests.py 326 2007-06-04 15:58:48Z akhavr $

from nose.tools import *
from twill.commands import *
from twill import add_wsgi_intercept

def setup():
    import os
    os.environ["DJANGO_SETTINGS_MODULE"] = "web.settings"

    from django.core.servers.basehttp import AdminMediaHandler
    from django.core.handlers.wsgi import WSGIHandler

    app = AdminMediaHandler(WSGIHandler())
    add_wsgi_intercept("127.0.0.1", 9876, lambda: app)

@with_setup(setup)
def test_rss():
    'test rss feed'
    go('http://127.0.0.1:9876/kds-djangofeed/feed/rss/')
    code('200')
    find('Django Feed Planet.')
    return

FeedJack, Django and select_related

Posted by Andrey Khavryuchenko on 28 May 2007 | Tagged as: Blog, django

Lately I wanted to collect my django feeds into a single web-viewable channel.

There’s more than a single solution to do this. Being django fellow, I’ve decided to utilize FeedJack.

So… Well, I guess, it might work flawlessly for other people, but in my cause it failed miserably after adding two feeds:

  • official django blog http://www.djangoproject.com/weblog/ and
  • djangosnippets http://www.djangosnippets.org/

Just after adding djangosnippets, it messed up the order of posts terribly.

What’s going on? The code shows.

FeedJack urls.py says:

urlpatterns = patterns('',
    [...]
    (r'^$', views.mainview),
)

That, after couple hops, translates to the fjlib.get_paginator:

def get_paginator(site, sfeeds_ids, page=0, tag=None, user=None):
    """ Returns a paginator object and a requested page from it.
    """

    if tag:
        try:
            localposts = models.Tag.objects.get(name=tag).post_set.filter(\
              feed__in=sfeeds_ids)
        except:
            raise Http404
    else:
        localposts = models.Post.objects.filter(feed__in=sfeeds_ids)

    if user:
        try:
            localposts = localposts.filter(feed=user)
        except:
            raise Http404
    if site.order_posts_by == 2:
        localposts = localposts.order_by('-date_created', '-date_modified')
    else:
        localposts = localposts.order_by('-date_modified')

    paginator = ObjectPaginator(localposts.select_related(), 
                                site.posts_per_page)
    try:
        object_list = paginator.get_page(page)
    except InvalidPage:
        if page == 0:
            object_list = []
        else:
            raise Http404
    return (paginator, object_list)

Quick introspection reveals that localpost order isn’t broken just until the FeedJack creates the paginator with:

    paginator = ObjectPaginator(localposts.select_related(), 
                                site.posts_per_page)

After that, the post order is messed up and the cause is optimizing call to select_related.

I’m not yet sure if it’s a feature or a bug, but certainly, the premature optimization is the root of all evils.

  • Django - unicode branch
  • python 2.4
  • FeedJack 0.9.9

Good luck!

PS. Finally it was solved by leaving select_related in and adding sorting by id to preserve the original order.