I hate hitting database in my unit tests.

Even if it’s tiny, nearly empty and easy to fill. Things only become worse when you have to test lots different cases that contradict each other and yet are small enough to justify a separate database fixture.

Thus, tonight I’m fighting my laziness to decipher django.db.models enough to be able to create slim and easy db mocks inside a nose unit test.

So far is far too late for my brain to spin at full speed and no code was produced. Yet I’ve come down to two distinct ways to implement this:

1. Completely mock django.db.models.query._QuerySet to use in-test dataset and respond to various queries
2. Construct in-memory python SQLite and replace application connection with the one of the unit test.

Definitely the second way is faster to either implement or fail. So will it be.

Watch for updates with working code..

Bookmark to:

No Comments »

Last two days I’ve spent fixing Djiggit wrt international, esp cyrillics feeds.

Here are 3 points that I’ve learned:

1. Read and follow instructions at docs/unicode.txt and communicate on the django-users list
2. Use latest db adaptors
3. Beware of your database encoding

Long story.

It took me several hours to trace why cyrillics is stored as ‘????’ in the database. Starting from top - my code - to the database itself.

Distilled down for mysql:

• make sure you’re using latest python-MySQLdb: earlier versions have known problems with utf8
• make sure that your db server is started with using utf8 charset in mind
• backup your database as json using

     manage.py dumpdata > <dumpfile>.json

• recreate your database making sure it has “utf8″ character set and collation:

    mysql> show variables like 'ch%';
    +--------------------------+----------------------------+
    | Variable_name            | Value                      |
    +--------------------------+----------------------------+
    | character_set_client     | latin1                     |
    | character_set_connection | latin1                     |
    | character_set_database   | utf8                       |
    | character_set_filesystem | binary                     |
    | character_set_results    | latin1                     |
    | character_set_server     | utf8                       |
    | character_set_system     | utf8                       |
    | character_sets_dir       | /usr/share/mysql/charsets/ |
    +--------------------------+----------------------------+
    8 rows in set (0.00 sec)

• load your data back from json dump:

    manage.py loaddata <dumpfile>

Happy hacking!

Bookmark to:

No Comments »

Djiggit is build on the Django unicode branch and yet it require non-obvious solutions to everyday tasks.

Imagine you’re trying to present a (cyrillic) tag in url-ready form. Usually this is solved by

{{ tagname|urlencode }}

But when the ‘tagname’ contains non-ascii symbols, urlencode barfs:

KeyError at /
u'\u0420'
Request Method:     GET
Request URL:    http://localhost:8088/
Exception Type:     KeyError
Exception Value:    u'\u0420'
Exception Location:     /usr/lib/python2.4/urllib.py in quote, line 1117
Template error

My current (temporary) solution is to encode this manually, in python:

urllib.quote(tagname.encode('utf8')

But this is not DRY and makes me think there’s a better way.

Perhaps it’s time to add a custom filter and push it into django codeline.

Bookmark to:

No Comments »

Just a little writeup..

Now have a pleasure of not starting django development webserver, nor using apache to do http-level tests in my project.

Thanks to twill and its wsgi integration.

Works smoothly now, but caused me little cursing when twill refused to see rss feed.

After digging the twill code I’ve changed a single line - still don’t know how correct is this:

--- /usr/lib/python2.4/site-packages/twill-0.9b1-py2.4.egg/twill/wsgi_intercept.py.orig 2007-06-04 21:10:37 +0300
+++ /usr/lib/python2.4/site-packages/twill-0.9b1-py2.4.egg/twill/wsgi_intercept.py      2007-06-04 21:10:45 +0300
@@ -265,7 +265,7 @@
                 for data in self.write_results:
                     self.output.write(data)

-            if generator_data:
+            if generator_data is not None:
                 self.output.write(generator_data)
                 for data in self.result:
                     self.output.write(data)

Nevertheless it didn’t break anything and I have a luxury to test rss feeds too:

# -*- coding: utf-8 -*-
# $Id: tests.py 326 2007-06-04 15:58:48Z akhavr $

from nose.tools import *
from twill.commands import *
from twill import add_wsgi_intercept

def setup():
    import os
    os.environ["DJANGO_SETTINGS_MODULE"] = "web.settings"

    from django.core.servers.basehttp import AdminMediaHandler
    from django.core.handlers.wsgi import WSGIHandler

    app = AdminMediaHandler(WSGIHandler())
    add_wsgi_intercept("127.0.0.1", 9876, lambda: app)

@with_setup(setup)
def test_rss():
    'test rss feed'
    go('http://127.0.0.1:9876/kds-djangofeed/feed/rss/')
    code('200')
    find('<rss version="2.0"><channel><title>Django Feed Planet.</title>')
    return

Bookmark to:

14 Comments »

Lately I wanted to collect my django feeds into a single web-viewable channel.

There’s more than a single solution to do this. Being django fellow, I’ve decided to utilize FeedJack.

So… Well, I guess, it might work flawlessly for other people, but in my cause it failed miserably after adding two feeds:

• official django blog http://www.djangoproject.com/weblog/ and
• djangosnippets http://www.djangosnippets.org/

Just after adding djangosnippets, it messed up the order of posts terribly.

What’s going on? The code shows.

FeedJack urls.py says:

urlpatterns = patterns('',
    [...]
    (r'^$', views.mainview),
)

That, after couple hops, translates to the fjlib.get_paginator:

def get_paginator(site, sfeeds_ids, page=0, tag=None, user=None):
    """ Returns a paginator object and a requested page from it.
    """

    if tag:
        try:
            localposts = models.Tag.objects.get(name=tag).post_set.filter(\
              feed__in=sfeeds_ids)
        except:
            raise Http404
    else:
        localposts = models.Post.objects.filter(feed__in=sfeeds_ids)

    if user:
        try:
            localposts = localposts.filter(feed=user)
        except:
            raise Http404
    if site.order_posts_by == 2:
        localposts = localposts.order_by('-date_created', '-date_modified')
    else:
        localposts = localposts.order_by('-date_modified')

    paginator = ObjectPaginator(localposts.select_related(), 
                                site.posts_per_page)
    try:
        object_list = paginator.get_page(page)
    except InvalidPage:
        if page == 0:
            object_list = []
        else:
            raise Http404
    return (paginator, object_list)

Quick introspection reveals that localpost order isn’t broken just until the FeedJack creates the paginator with:

    paginator = ObjectPaginator(localposts.select_related(), 
                                site.posts_per_page)

After that, the post order is messed up and the cause is optimizing call to select_related.

I’m not yet sure if it’s a feature or a bug, but certainly, the premature optimization is the root of all evils.

• Django - unicode branch
• python 2.4
• FeedJack 0.9.9

Good luck!

PS. Finally it was solved by leaving select_related in and adding sorting by id to preserve the original order.

Bookmark to:

No Comments »

FRIM: Another Way to Gather Data

In FRIM, the team writes 3×3 sticky notes about the events, impediments, and boons* of the iteration. (If you want to get fancy, you could use different color sticky notes for events, impediments, and boons.) Team members write sticky notes to include as many events, impediments and boons as they can remember. They may work individually or in pairs or triads if you have a larger team.

The retrospective leader draws a large 6×6 grid on the whiteboard or a flip chart-papered wall. Team members post their sticky notes on the grid according to the frequency of the event and its impact on the team.

Impact (vertical dimension)
5 = Maximum Impact
4 = Significant Impact
3 = Moderate Impact
2 = Some Impact
1 = Little Impact
0 = No Impact

Let each team member devise his or her own concept of relative impact, or, if time allows, hold a brief team discussion to define “maximum impact” vs. “moderate” or “little”.

Frequency (horizontal dimension)
5 = More than daily
4 = Daily
3 = Every 2-3 days
2 = 1 or 2 times in a Iteration
1 = Once or fewer times in a Iteration
0 = 2 or 3 times a Release/Rarely

When all the notes have been posted, review the overall story. Start by reading the notes in the top, right-hand cell first (5I-5F), then work across and down, focusing on impact first (5I-4F, 5I-3F, 5I-2F…) and frequency second (4I-5F. 4I-4F, 4I-3F…). As a group, discuss commonalities or patterns. Refer to the grid as you shift into a discussion of the insights gained by telling the story of the iteration.

How pity that some teams decide that even simplistic “what are the causes? what should be done differently?” is too difficult for them.

Bookmark to:

No Comments »

Effects of Defects: Grey Scope Creep (Agile Advice)

Fresh features are marked done, and then disappear somewhere in QA to eventually fire back at unknown time with unknown bugs. Grey scope creep. Stop it. Instead, insist on taking less but making it “done” within an iteration. Done reads fully developed, thoroughly tested, debugged, fixed including regression, and accepted by product owner.

That’s it. No comments necessary.

Bookmark to:

No Comments »

Software development and teaching it requires something more than a one pair of eyes to watch the code.

Essentially, the software development process is the process of translation from customer’s ideas down to code done collectively. That’s why code should be written more for reading by other people than for computer execution.

Unfortunatelly, our teaching system and existing developers rarely seem to understand this simple idea. At least, not until they will try to read and get some oldish perl code like this:

sub makesmblock { my($isblock2,@mh)=@_; my ($smi,$xmnum);  $xmnum=1;
foreach $xbit(@mh) { ($bit,$fpnum)=split(/\і/,trim($xbit)); if($bit) {
#print "$bit<br /><br />";
if(($fpnum==1 && !($nocf)) || ($fpnum==1 && ($PROCESS{'domode'} || $PROCESS{try})&& !($ori_ha{'notrycontactform'}))){$bit.=qq~\|$transm41\|$my_http\?cf=3$ltag$vtag~; }
if(($fpnum==1 && $sitemap==2 && !($nosm)) || ($fpnum==1 && ($PROCESS{’domode’} || $PROCESS{try})&& !($ori_ha{’notrysitemap’}))){$bit.=qq~\|$transm39\|$my_http\?sm=3$ltag$vtag~; }

(real code from real product)

Systematic code review is possible either by

• scheduled real-life code reviews
• pair development
• distributed code review

Scheduled meetings with the only purpose of code review is a no-no in our software development team. As well, as in any team that doesn’t sit in a single office, I guess. Pair development is equally hard for geographically distributed team. Thus, we are left with only one option of distributed code review.

We are using trac for our project management for years with great success. That’s why I, naturally, tried to use trac peer review plugin.

No way

Given latest stable trac 0.10.3 you get html parse errors (#1035) due to crappy html, generated by plugin.

And even after you mask those errors with patch, you discover another defect (#938) that makes impossible to use this plugin at all.

Looks like its time to look for an external code review system. At least until someone fixes (rewrites) PeerReviewPlugin or writes another one.

Bookmark to:

No Comments »

Since it is not possible to predict every detail, every project plan, not just a software development one, needs a feedback loop.

The After Action Review is such feedback link, that was missing in the outline of basic software development process. The third component.

The first two key components are:

• Well Formed Outcome
• Decomposition also known as the “Separation of Concerns”.

Originally it was employer by US Army to adapt to a fast-changing tactics of its opponents. Lately it was incorporated in all army actions and found its way into business.

In simplest form it consists just of four questions, that could be explored either individually or team-wide. These questions are:

1. What was the goal?
2. What are the actual results?
3. Why there is a difference? (Or “Why we were successful?” if no difference)
4. What decisions we can make basing on (3)?

In software development we apply them at every measurable activity step:

• day plan and results
• week plan and results
• milestone
• financial goals

We were surprised how quickly the AAR highlights the open problems in the software project and in the software development process and how quckly and easily we were able to come up with useful and implementable decisions. Decisions that do work and improve the value we bring to a Customer.

It is such simple and powerful tool that not much could be written about it. You have to experience it: try doing daily AAR for at least of couple a weeks.

Apply the AAR and the decisions made rigirously and the world would turn around you.

More information about After Action Review:

• Wikipedia
• US Army’s Leader’s Guide to After-Action Reviews
• Knowledge Management Library,

Bookmark to:

No Comments »

Based on RFC 1925. It’s almost verbatim - most of truths are common.

1. It Has To Work.
2. No matter how hard you push and no matter what the priority, you can’t increase the speed of light.
   • (2a) (corollary). No matter how hard you try, you can’t make a baby in much less than 9 months. Trying to speed this up might make it slower, but it won’t make it happen any quicker.
3. With sufficient thrust, pigs fly just fine. However, this is not necessarily a good idea. It is hard to be sure where they are going to land, and it could be dangerous sitting under them as they fly overhead.
4. Some things in life can never be fully appreciated nor understood unless experienced firsthand. Some things in software development can never be fully understood by someone who neither builds an actual custom software product nor administers a such one.
5. It is always possible to aglutenate multiple separate problems into a single complex interdependent solution. In most cases this is a bad idea.
6. It is easier to move a problem around (for example, by moving the problem to a different part of the overall software architecture) than it is to solve it.
   • (6a) (corollary). It is always possible to add another level of indirection.
7. It is always something
   • (7a) (corollary). Good, Fast, Cheap: Pick any two (you can’t have all three).
8. It is more complicated than you think.
9. For all resources, whatever it is, you need more.
   • (9a) (corollary) Every software development project always takes longer to do than it seems like it should.
10. One size never fits all.
11. Every old idea will be proposed again with a different name and a different presentation, regardless of whether it works.
    • (11a) (corollary). See rule 6a.
12. In software design, perfection has been reached not when there is nothing left to add, but when there is nothing left to take away.

Bookmark to:

No Comments »