Basie Blog

I have been putting some thought into hide-not-delete, and here are my thoughts:

• To alleviate confusion, I will call entities which have been hidden-because-they-should-be-deleted “retired“. For example, a ticket will no longer be deleted, it will be retired.
• An abstract model class should be created, which adds, among other things, a retired = BooleanField(). This will be used by the model managers, as I describe below.
• Two new model managers should be created.
  • One manager will filter out retied entities (ie, it will filter queries byretired=false). This manager will replace Model.objects (ie, it will be the one most frequently used). The QuerySet it returns will also have a retire() method, which will be identical to delete(), except (of course) it will not actually remove any rows. This is the manager which will, for example, return ObjectDoesNotExist in response to Ticket.objects.get(id=42) and will hide retired wiki pages from WikiPage.objects.all().
  • A second manager which will have the use_for_related_fields property set and will return a QuerySet that will not filter out retired entities – instead, if the QuerySet encounters a retired entity, it will instead return a special instance of the appropriate class. For example, if a query would normally return a User, but the instance User is retired, an instance of RetiredUser will be returned instead. This is so that code which depends on a user existing will still function – it just won’t have information about the particular user which has been retired. This is, for example, how Reddit handles deleted accounts and comments.
• If all goes according to plan (which it rarely does, but I’m optimistic), all the calls to .delete() in the existing code will be replaced by a call to .retire(), and everything will “just work”.

One thing, however, is that I haven’t mentioned the admin interface. That’s because I don’t know the first thing about how it works. If you do, would you mind replying, telling me how you think the scheme described above will interact with it?

Status

• Successfully installed Basie (Mac OS X 10.6)
• Filed a bug, http://beta.basieproject.org/dev/tickets/#62

Blocks

• None

Next Steps

• Find and Post another bug
• Fix a bug, currently working on http://beta.basieproject.org/dev/tickets/#62
• Meet up for Coding Sprint!

• Status

- finished installing basie with the most recent revision
- submitted 2 bugs through email to basie-devs
- started investigation to allow users select their time zone in user preferences

• Next Steps

- continue investigation and start implementation of user preference time zones

• Roadblocks

- need to further investigate structure of source code tree in order to understand what changes need to be made

Just before Christmas, I put up a post discussing possible goals for the term. It’s time to make those more concrete, if only so that people know what they’re going to be graded on. Our high-level goals are:

1. Have fun and learn lots.
2. Impress people (at PyCon 2010 in Feb February and afterward) so that they’ll use Basie and/or help us develop it. This means having something that’s actually really truly useful and usable Real Soon.

Unfortunately, there’s tension between #1 and #2. We didn’t get Version 0.6 out the door last term (more about that below). That must be our first task this term, which means the first order of business is going to be finding and fixing bugs, tidying up Basie’s visual appearance, improving its performance, making sure that we hide things instead of erasing them, and so on. This isn’t as exhilarating as writing entirely new components from scratch, but on the upside, it’s a great way to learn how the existing code works.

Just to get us started, here’s a list of the things I believe need work:

1. The RSS feed for the ticket app is broken.
2. The wiki’s RSS feed points to the ticket’s RSS feed.
3. The titles for the mailing list and event RSS feeds aren’t in English.
4. Doing a ‘diff’ on a directory in the repository browser blows up.
5. No one has managed to get Basie running on Windows (which means some developers who might help us won’t).
6. We aren’t self-hosting (i.e., we’re not using Basie ourselves on a regular basis).
7. We don’t have a “sandbox” that’s torn down and refreshed periodically. (For people who haven’t run into the term before, a sandbox is a place where children play; in web terms, it’s an instance of an application that people can mess around in without doing any real damage.)
8. The script to import mail messages runs very, very slowly (which probably means there’s a performance problem somewhere that will bite us in Basie itself).
9. Setup on Mac is harder than it should be.
10. Our HTML, CSS, and JavaScript are inconsistent (see Chris Van Wiemeersch’s detailed email message and follow-ups from Jeff Balogh and Antoni Aloy—which, by the way, would make a blog post and/or additions to the style guide).
11. Tickets take a suspiciously long time to display (possibly because lots of unneeded data is being formatted, then hidden?).
12. We don’t have a short screencast demonstrating what Basie is, or how to set it up.
13. Sizing and layout need work all over the place (I sometimes feel like I’m looking at the “large print” edition—which, given my age, is probably not a bad thing).
14. The REST API has fallen behind the application itself, and needs documenting with lots of examples.
15. The tool Ian Lienert wrote to import projects from DrProject has fallen out of date, and we haven’t even started one to import data from Trac.
16. Basie is slow compared to Trac or DrProject (though this might reflect the server the current sandbox is on, rather than the app itself).
17. Channel management in IRC is hard enough that it might be simpler to run our own IRC server (or switch to Jabber/XMPP, or…).
18. Managing permissions and tagging (and other things that cut across components) has become so complicated that some polymorphism might be in order.

I’m sure there are more—if current or former team members want to add items as comments to this post, it would be welcome.

Now comes the hard part: figuring out what went wrong last time so that we don’t make those mistakes again. In this case, the blame for not getting Version 0.6 out last term lies squarely with me. As the project manager, it’s my responsibility to make sure (a) that developers are working on things that actually matter, (b) that their goals are realistic, and (c) that they’re making progress. I think I did a pretty good job of (a), but I fell down on (b) and (c). Some time in late October, I should have scaled down what we were trying to do, since finishing 100% of a small job is better than finishing 75% of a large one. I should also have been much stricter in the last three or four weeks about tracking who had finished what, and who was waiting on whom so that we didn’t suddenly “discover” at the end of term that we weren’t going to have a screencast because the sandbox hadn’t been refreshed because a couple of patches were waiting for review because they’d been held up waiting for answers to questions that—but you get the picture.

So here’s what I propose we do this term:

1. Every member of the team will post a punchline status report to this blog before they go to sleep on Monday. This post (from the MarkUs team) shows you what we’re after: Status, Next Steps, and Roadblocks, with bullet points for each.
2. Everyone will read these reports before our online Tuesday meeting so that we can spend our time discussing technical issues.
3. Everyone will put something—even if it’s very small—onto ReviewBoard each week, and do at least one review. This ought to mean a minimum of half a dozen small commits per week.
4. We’ll refresh the sandbox instance weekly (I’ll talk to Zuzel and Dan about the best schedule) so that we can always see what state the code is in.
5. For the next four weeks, we will work toward getting Version 0.6 into a usable, releasable state.
6. Everyone will then get to spend the second half of the term working on whatever appeals to them most.

Please let me know what you think—as with almost everything else in this course, it’s negotiable, but PyCon is our big chance to make a good impression on the world, and I think this plan gives us the best shot at making the most of it.

…or maybe not: we just had our tenth help request from someone trying to set Basie up.  That’s, what, a zillion percent growth in the last two months?

ReviewAnywhere is a code review tool written by David Wolever (a Basie contributor)  that takes a document and allows for comments to be placed over top.  It’s got some cool AJAX functionality which makes it a pleasure to work with, and appears to be quite pluggable.

If David reads this, maybe he’ll provide a link to his project at wolever.net, but the server appears to be down right now.

The reason I’m writing about this on the Basie blog is that this can be plugged in to a code review tool whenever we get one built.  What is needed is:

1. An app within Basie to upload diffs and manage these diffs.

2. An SVN backend that can process what these diffs mean in the context of the HEAD revision of the SVN repo in question.

3. A nice looking frontend that can display these diffs in a way that works with ReviewAnywhere.

4. David W. to be so kind as to work with someone on this next term.

Anyway, this was a bit of a brain dump so that the next person who comes across the code review concept has a great starting point.

With the feature freeze already set for the end of the week, the Basie team has gathered online to review just how well we are coming into the end-of-term phase of the project.

With the priority set on getting everything finished, Bill took charge of the meeting with a set of questions:

1. What was my original commitment in terms of functionality to be tested/reviewed/committed for v0.6?
2. Where am I in terms of reaching the original goal? Will I pull it off?
3. What help do I need from other team members (if any) to finish?

IRC Integration

Andrew is getting close to finishing the back-end communication with the channels. It turned out that the freenode network allows for a maximum of 20 concurrent channel connections, so large multi-project installs of Basie might call for multiple bots. Though with timely peer-reviews, the project should be on track.

Tony is finishing up with Django half of IRC integration: database, webviews, search. Prettiness (CSS) is a feature that was easily pushed aside. The focus is on functionality, so Tony will get his posted patches committed and help with reviews. Lack of having a sense of design might be called a bug for next week.

Milestones

“Add milestones, edit milestones, linkable milestones (i.e. to calendar, tickets), progress bars.” Calendar parts are solid, though the display of the dashboard events still needs some work. There will be peer-review requests; but with no blocking items, milestones seem to be on a good track to completion.

Tagging

After some peer-review sessions, the bulk of the code is about to get shipped out of the ReviewBoard. Then just about anything with get_absolute_url() and a project property will be able to easily get tagged. (This is awesome.) Some nice-to-haves fancy addons might not make it in, but the core functionality does work.

An issue of bad tag names came up. We were talking about proper HTML escapes, security, and relevant limitations on the mailing list. Having to support international character sets also make things much more complicated. All of this could use some more review work, but otherwise no blocking items.

Dashboard

Maybe “70%” done, with large chunks of development code being put up on the ReviewBoard. No blocking items, but will certainly require some attention from peers.

—In Conclusion—

We seem to be in a good shape. Still talking with each other, and generally seem to be on track with finishing our tasks for this term. Or we are just keeping up such an illusion… One obvious trend is that peer-reviewing each other’s code into the production state will become a much more vital part, as the school term comes to an end. We should aim to finish the term with a clean codebase.

The complete IRC transcript can be found in the archives.

Having been chasing the broken test suite since the beginning of the term, we’ve finally tracked down that the data left over after some interactive doctest tests was causing a lot of problems. Zuzel has put together a patch that would explicitly clear stagnant data.


>>> call_command('flush', verbosity=0, interactive=False)


Now the tests set up and have only their own data, and we avoid a cascade of problems resulting from “IntegrityError: column slug is not unique” type of errors.

The Basie team is proud to announce that all of the tests now pass.

We are quite excited about that. Among the many reasons, this will allow to easily regression test new code commits.

Ran 279 tests in 739.022s
OK

(no failures!)

Now that we no longer have errors and failures, and the test suite has seen quite a few changes over the last couple of weeks, we can also revisit our state of test coverage. I’ve left out apps with 90+% coverage, and colour-coded the rest.

apps.a3c 59 53 89% 83, 100-107
apps.a3c.api 114 92 80% 29-30, 46, 57, 61, 65-70, 91, 103, 107, 111-116, 121, 138, 156, 166
apps.a3c.views 34 25 73% 23, 28-29, 33-34, 40-41, 46-47
apps.about.views 12 7 58% 5, 8, 11, 14, 17
apps.audit.models 153 111 72% 70-71, 77-89, 155-156, 165, 210-213, 226-234, 239-260, 264-276
apps.chatlog.models 24 21 87% 27, 43-47
apps.chatlog.views 17 7 41% 11-24, 27-34
apps.dash 7 6 85% 11
apps.django_evolution 11 9 81% 3, 6
apps.django_evolution.db.common 120 20 16% 8-11, 14-20, 23-26, 29-30, 33-38, 41-82, 85-90, 94-98, 101-102, 105, 108-128, 131-136, 139-143, 146, 149-154, 157-166
apps.django_evolution.db.mysql 58 10 17% 8-42, 45-50, 53-63, 67-69, 72-81, 84-89
apps.django_evolution.db.postgresql 9 4 44% 7-13
apps.django_evolution.db.sqlite3 149 19 12% 10-25, 28-35, 38-45, 48-60, 63-79, 83, 86-99, 102-134, 137-159, 162-175, 178, 181, 184, 187-200
apps.django_evolution.diff 117 45 38% 10-11, 15-17, 20, 23, 34-38, 71, 74, 84-85, 90-93, 99-104, 113-122, 131, 147, 151-167, 171-205
apps.django_evolution.evolve 45 22 48% 19, 36-60
apps.django_evolution.management 62 41 66% 3-4, 30, 39-40, 42-45, 49, 66-68, 73-74, 77-80, 95-98
apps.django_evolution.management.commands.evolve 158 25 15% 6-7, 47-225
apps.django_evolution.models 20 16 80% 14-16, 27
apps.django_evolution.mutations 308 65 21% 23-34, 45-74, 77, 80-86, 89, 92, 95, 107-109, 112, 121-123, 127, 134, 143, 147-149, 152, 156-159, 163, 168-169, 172, 175-199, 202-218, 222-226, 229-237, 240-255, 258-261, 264-275, 278-295, 300-304, 307-314, 317-335, 338-370, 374-377, 380-392, 396-407, 411-432, 436, 439, 442-444, 447-461, 465, 468, 471-475
apps.django_evolution.utils 13 3 23% 5-9, 16-22
apps.django_nose.nose_runner 43 11 25% 33-42, 46-48, 52-54, 58-83
apps.errors.log 22 13 59% 14, 28-35
apps.events.feed 25 15 60% 14-15, 18-22, 26, 29, 32-33, 36
apps.ircbot.plugins.BasieLogger.config 6 4 66% 9-10
apps.linkmedia.management.commands.linkmedia 35 9 25% 31-62
apps.lists 36 27 75% 20-23, 25, 30-32, 49-50
apps.mail.api 45 40 88% 15, 32, 43, 50, 60
apps.mail.bll 52 42 80% 18, 24-27, 33-34, 47, 56, 64
apps.mail.feed 23 15 65% 14-15, 18, 21, 24, 27-28, 31
apps.mail.views 38 27 71% 20, 43, 46, 53-60
apps.manage.views 48 37 77% 18, 50, 55, 70, 75, 84-85, 89-90, 94-95
apps.milestones.api 41 11 26% 10-15, 20-26, 31-36, 40-45, 50-54, 59-62
apps.milestones.bll 37 9 24% 9-15, 20-23, 27-30, 34, 39-50, 55-61
apps.milestones.forms 12 10 83% 15-16
apps.milestones.views 34 12 35% 16-39, 44-52
apps.restapi.responder 41 28 68% 48, 74-85
apps.search 15 11 73% 14-17
apps.search.index 16 10 62% 40-45, 50-51
apps.search.searching 33 28 84% 33-39, 54-57
apps.tickets.api 41 11 26% 11-16, 21-28, 35-40, 44-49, 54-59, 64-67
apps.tickets.bll 48 13 27% 9-18, 23-26, 31-32, 40-57, 62-68, 71-77
apps.tickets.dash_data 30 26 86% 12-15
apps.tickets.feed 23 15 65% 14-15, 18, 21, 24, 27-28, 31
apps.util 27 21 77% 23, 41-42, 69-71
apps.vcs.dash_data 17 15 88% 19-20
apps.vcs.diff_match_patch 947 144 15% 93, 143-216, 231-272, 282-286, 299-392, 405-437, 450-482, 537, 554-621, 630-669, 680-769, 778-846, 875-892, 901, 937-943, 946-952, 957, 970-989, 1000-1015, 1026-1030, 1041-1045, 1059-1070, 1086-1124, 1139-1152, 1167-1259, 1270-1275, 1287-1313, 1327-1395, 1408-1468, 1480-1525, 1534-1599, 1610-1613, 1628-1690, 1700-1704, 1714-1738
apps.vcs.forms 19 9 47% 13-17, 20-24
apps.vcs.management.commands.update_authz 7 5 71% 10-11
apps.vcs.models 90 79 87% 46, 66-72, 84-85, 100, 116-117, 137-139
apps.vcs.svn_backend 229 119 51% 21-22, 48-64, 69-71, 76-84, 90-94, 99-105, 124, 131-135, 139-151, 157-178, 182-187, 205-228, 255, 258, 264, 278, 284, 287, 305-309, 318, 325-334, 341, 345-353, 356, 359-363, 366, 383, 419
apps.vcs.views 153 129 84% 26-27, 74, 95, 100, 119-120, 123-129, 132, 144-147, 152, 169-170, 186-187, 191, 216, 337
apps.wadofstuff.django.serializers.python 63 43 68% 56, 81-83, 89-97, 107-112, 135-140
apps.wiki.feed 23 15 65% 14-15, 18, 21, 24, 27-28, 31
apps.wiki.macros 26 21 80% 23-26, 31-33
apps.wiki.markup.basie_creole2html 50 44 88% 26, 29, 40-41, 50-52
apps.wiki.markup.links 34 30 88% 35, 49, 60, 64
apps.wiki.markup.wikicreole.creole 211 130 61% 79, 161-172, 208-212, 217-218, 221-245, 250-251, 263, 270, 275-296, 299-307, 316-317, 322-326, 329-333, 336-337, 340-342
apps.wiki.markup.wikicreole.creole2html 91 51 56% 71-72, 89, 95, 98, 101, 104, 107, 110, 113, 116, 119, 126, 129-141, 145-154, 158, 161, 164, 169, 188-190
apps.wiki.utils 12 10 83% 20-21
apps.wiki.views 94 84 89% 66-67, 75, 79-80, 105, 111, 165-198

As most of you now know, I’m in a somewhat unique position where I have to meet the requirements of a course here at MUN through my work on Basie. I sat down with my local Prof and we came up with a grading scheme that will satisfy the requirements for this course as well as accurately represent my work on Basie in the way that the majority of you are being graded. This makes things a little more complicated for Ian and I, but I believe that Ian should be able to be graded based on the Team Calendar grading scheme without having to worry about my specific requirements. Without further ado, here is what we came up with:

Process “document” – 15% (individual)
-an examination of the differences between the project management process and traditional project management elements of a software project. This will also identify how the end product can be evaluated(validated and verified) in terms of its intended purpose. Includes critique of the current process we use for Basie. This is the document I have been working on, and am in the process of revising based on feedback from the rest of the team. This will be especially useful for determining how the end product matches the requirements.

Requirements artifacts – 10% (team)
collection of blogs, documents, diagrams, discussions, etc that describe the intended outcomes of the process. I have been receiving help from the team on this one as well, they have pointed me to blogs and discussions and given me their views on how our requirements are decided.

System Demonstration – 25%
(self-explanatory)

Architecture Document – 15% (team)
We decided that this would describe how Basie works in terms of the already existing django architecture.

Module Section (individual)
This is broken into two components:
-Performance Evaluation: We decided this would be a good place to use what Bill proposed on the Basie blog. At this point I imagine it would be more appropriate to use Ian’s scheme mentioned above since we are team Calendar, but the two are similar enough. The idea is that my scheme contains at least something that everybody elses does. – 15%

-Product: code and description of code (maybe class diagrams , etc) – 20%

So there is my scheme. The weights associated may seem arbitrary but they were chosen to line up with the requirements of my course here. If anybody is interested the course outline can be found in this doucument: Undergraduate Handbook (course number 4770.) I would be interested to see if any other students taking UCOSP are in the same scenario as me where they have to meet specific requirements for a course that are set before they have to decide their own grading scheme for UCOSP. It was a bit challenging to do this, but it was definitely a good learning experience for us. I imagine we all had difficulties coming up with our scheme, after all, having a grading scheme handed to us on a silver platter is just one luxury we have as students : )

I left it at ‘Calendar’ because there might be some variation from Josh, given his MUN-specific requirements. Essentially I’ve decided to go with Tony’s take on grading. That is:

20% — Participation

40% — Work

30% — “Team Citizenship”

10% — ScreenCast