Basie Blog

Tasks for this week + next 2 weeks (tentative):

Everyone:
- update pinax app and add both svn + hg repo (follow Nikita’s instructions)

Front-end team:
- file: missing choosing revision number, view history/download file links (no download link if binary), make ‘pretty’
- dir: make revision link, (no download link if binary), make ‘pretty’ (alignment is weird, longer dates mashing with revision #, fix layout when revision is hash (hg), etc…) — please test with different repo types, browser window sizes, etc
- hist: make revision link, make diff radio button (see mockup!) — no diff radio button if binary file
- diff: keep implementing

Hg:
- fix history for dir, history for some files (filename issue?)
- filtering revision in history
- work with Svn team on return type for diff
- iron out the hash vs rev # issue (? not sure if it is actually an issue, to be discussed offline)

Svn:
- work with Hg team on return type for diff
- continue work on diff
- error checking for history

In general:
- make things fail gracefully (non-existant path in repo, non-existant revision # in repo, etc) –> needs to generate some “no such path/rev#” in template in such cases
- support revision back-end (for both hg + svn) for template
- urls fix/unify by tonight, please let everyone know what the new convention(s) is/are
- possibly make “group aware” (?) if time

Screencast:
- script by Tuesday 9PM

Feedback on script:
- due by Thursday 9PM if you want any say in it

Next meeting dictator: Joseph

Week ending July 30th (tentative):
- first version of video must be done by Tue
- everyone download and check, must reply with comments by Thursday 9PM
- everyone read each others’ comments before meeting
- diff template must be finished
- testing all around
- patch view back-end
- add link to diff template for patch

Week ending Aug 6th (tentative):
- video ver 2
- fine-tuning
- moving templates to app
- other wrap-up

This is a quick snapshot at what our File View template can provide right now:

Some of the highlights include:

• compatibility with Pinax’s CSS and templates
• Breadcrumb support to show users the current file being viewed, its path, and its revision
• One of our most requested features: Syntax Highlighting and Line Numbering!

cheers,

Johan

Colin, Dave, and I have done a few usability tests over the past few days using simple paper prototypes. We tested the initial mockups on three CS undergrads. Here are the highlights of the testing from two CS undergrads who are currently doing PEY at IBM

• Participants were able to use the breadcrumbs to navigate the repo, but they often asked “Is there a Back button?”
• Some participants found the Revision column which shows revision numbers as odd, they think that clicking on those revision numbers will show the file/dir at that revision, which as one pointed out would be pretty useless since it would just show him the latest revision (but isn’t that what he’s supposed to be seeing right now anyway?)
• One of the participants find it pointless to attach revision number to directories, since what he’s interested in are the files.
• One of the participants prefer that the diff view shows the content of the entire file, just like the diff view in Eclipse
• Some mishap with the UI elements for generating diffs. For example the “Diff” button seems to be hard to find and should be relocated.
• Language is strange at time. Inconsistent to use ‘expand’ in one case but its opposite, collapse, is instead denoted as ^. Author is more familiar than committer.
• Unclear what format dates should be entered in during revision search. How to use the radio buttons to generate a diff could be confusing.

The points gleaned from the testing at U of T:

• The participant thought that the revision number column in the directory view was useless. When asked what they thought would happen on clicking it, they thought it would show the corresponding file or directory at that revision (as one of the IBM participants thought).
• The most important feature they felt was missing was a way to find all the commits made by a specific user. They suggested that the “entries in the author” column should be clickable, and should lead to a list of commits made by that user, in reverse chronological order.
• When asked what they would want or expect on doing a diff of two revisions (as is supposed to be possible in our ‘repository-revision’ mockup),  they suggested it should give a diff of all the files in the chosen directory (recursively), between the two chosen revisions.

Front-end team:
- Dave will post mock-up for the revision page
- Noam will start directory template
- Johan will blog about findings from usability testing @PEY + start file contents template
- Colin will blog about findings from usability testing @UT + work with Joseph and Misa on integrating front+back

HG team:
- Nikita will work on setup.py and testing
- Joseph will work on integrating + other code/testing if there is time

Svn team:
- Todd will work on testing and making changes to how svn-side is pointed to svn repo
- Joel will continue work on get_history (+URL) and updating/fixing class diagram
- Misa will work on integrating + other code/testing if there is time

Next meeting leader: Todd

What Joseph and I have so far:

* wrote two manage.py commands, getkiln and krunserver. The first one fetches and bootstraps kiln [you still need to get all the dependences, it's not that sophisticated]. It will also set up a sample hg repository (kiln backend repository itself). The second one behaves just like django’s runserver, except it also lunches kiln app on http://127.0.0.1:8001. You have to run the first one once to bootstrap kiln, you have to run the second one instead of runserver if you want kiln stuff to work at all

* we’ve implemented part of proposed API (getting file contents, file history, getting directory listing), there are some things that we’ve done differently but I’ll have a chance to discuss this in a separate email before the meeting on Wednesday. My implementation of get_file adds a dependency – python-magic which is a module that uses magic numbers to guess file type by it’s contents. The method returns both file contents and its MIME type (this actually lets the view to decide what to do with the file and not blindly serve it).

* kiln path, kiln repo folder, project name is defined in settings.py

Here are a few screenshots (these views are just to verify that our backend communicates with kiln properly, we did not do anything fancy as this will be part of the implementation the front-end team is going to do) :

Front-end team’s commitments:

• Polish up the existing mock-ups so that they’re suitable for demoing.
• Find guinea pigs for usability testing (preferably among the application’s target audience: second or third year compsci students).

Back-end teams’ commitments:

• By Tuesday, finalize the class design of the API.
• By Friday, each team will have written at least two views (that’s ‘controllers’ if you don’t speak Django).

Still under discussion:

• Will the URL for a repository make explicit the type (HG or SVN) of the repository? The answer to this depends on whether or not we think it’s reasonable for one project to have more than one repository.

Summary for the meeting:

- Both the SVN and HG back-end team will work together to simplify and consolidate the API and start coding when that is done.

- The front-end team will start prototyping for user testing. Using Django template

- Set up a starter pinax app, and submit it to Basie review board.

- URL:
A) the format for a non-diff page is /action-type/R.#/dir/path  (where R.# is omitted if it is a HEAD revision)
B) the format for a revision history of a file/directory is /history/query-string/dir/path
C) we still need to agree on the URL for diff:
1) /action-type/R.#+R.#/dir/path (format follows A) and B))
2) /action-type/dir/path/R.#+R.# (more ‘intuitive’ URL)

If you’re on the Basie mailing list, you may have noticed some lively debate over how to structure URLs for Basie’s repository browser. Greg has asked me to summarize the positions of both camps and their pros and cons. You can vote for your preferred option in the comments of this post.

Before presenting the two options, let me describe the motivating problem. We can agree that the URL to view the directory /spam/eggs in the project Vikings should look something like:

http://basie.org/Vikings/view-source/spam/eggs/

(In the future, I’ll omit everything before ‘view-source’ for the sake of brevity). Now the question is, how should the URL change if the user wants to see the same directory, but with one or more differences in content or presentation, such as:

• with files sorted by age/size/name
• at a different revision

There are at least two answers to this question.

Read the rest of this entry »

First of all, let me apologize for taking so long in writing this post. I don’t have full time internet access where I’ve been staying, so everything takes a little bit longer than I would like.

Anyway, I was lucky enough to attend Djangocon.eu in Berlin this last week. Being my first conference, it was a bit overwhelming but it was a lot of fun. It was a pretty intense week, I met a bunch of people and I learned a lot. Be sure to check out all the talks on blip.tv (especially Idan Gazit’s talk ‘Design for Developers’, as we’re diving head first into the code browser design).

Bad Pony!

On day 2, Russell Keith-Magee gave an interesting talk on the Django development process. By looking at why some feature proposals are rejected (or put on hold), the talk sheds some light on how the Django community is structured, why certain decisions are made, and how developers can go about contributing to it. From the point of view of Basie, it was an interesting look at how a large open source project manages it’s development and the tools that they use (trac).

I’m going to skip over the majority of Russell’s talk and get right to the Basie specific stuff. (Check out Russell’s talk above or a summary here.)

Django is a large project with a lot of code, a lot of contributors, and even more users. Managing an open source project of this size isn’t easy, especially with the relatively small number of core committers. As such, it can be difficult for developers who are new to the community to build up a reputation in that community. And in open source, reputation can go a long way – if a core committer recognizes your name, they’ll be more likely to take a look at your tickets and patches.

In the current system, Django uses trac to manage the code, tickets and patches. Reputation is managed ‘offline’ – it’s boils down to whether or not people recognize your name as someone who is contributing to the community. Since the core team has limited resources, they need to be selective as to where they put their time. And invariably, their time will go towards dealing with issues that come from people that they know and trust – i.e. from contributing members of the community.

There was a lot of great project management discussion during the question period and  Jacob Kaplan-Moss (Django co-creator and core developer) came up to help field some of the questions. The first question had to do with the ticket triage process. During the ticket triage stage, the ticket’s status (eg. duplicate, won’t fix, or works for me) is a bit too terse and is often misinterpreted. To help developers who are new to the process, Jacob said that they were drafting a set of ‘form letters’ to help clarify what each ticket status means and what the next step in each stage is, including what you should do if you think the decision was wrong. These letters will reside on a page of their wiki, but I think it would be nice you had the option to send these letters automatically when a ticket is triaged.

The next question directly touched on some of the shortcomings of trac (or specifically the outdated version that they are currently running). Feature proposals for upcoming versions of Django aren’t handled via trac, but on a wiki page where community members propose tickets to be addressed in the upcoming release. Russell proposed integrating a voting system into trac (via a plugin) that allows developers to vote on tickets that are affecting them personally. It would then generate a leader-board of the top outstanding tickets that need addressing. In addition to voting on tickets, he proposed that they need a way of voting on patches. For example, if 40 people are using a particular patch – the core team would like to know about it. On top of keeping tabs on who’s writing good patches, they’d also like to be able to keep track of the people who are good at picking good patches.

What I think that this all boils down to is integrating a voting and karma system into the project management system. Obviously, this wouldn’t be as useful to smaller projects with only a handful of developers, but if Basie wants to make itself more attractive to larger projects then this would be a great asset. Taking a lesson from social news sites like reddit or digg, we could attach karma scores to tickets, patches and users (or anything else!). Instead of just adding your email to the cc list, users could follow tickets by upvoting. We could then display these tickets on each user’s profile page, which is a lot nicer than just saving a bookmark to the ticket (or in 30-pages of paper notes like Jacob does!). This has the potential of harvesting a tonne of useful information which would be invaluable for larger projects. Apparently some of this is being addressed in an upcoming version of trac (ticket voting?), but I think Basie could set itself apart by integrating social networking features at its core.

Tortoise is a line of version control system tools. There’s a Tortoise tool for SVN, another for Mercurial, another for Bazaar, and so on. I’m taking a look at the SVN tool for Windows.

Shell-facing UI

The first thing I notice about its interface is that it’s a shell extension. The advantage to this is the user already has a familiar browsing structure and set of icons. The disadvantage is the tooltip for the folder is the Windows one, rather than the Tortoise one. This means that the icons that TortoiseSVN have to contain all the user needs to know.

I remember from my first day using the tool my initial impressions of what the icons meant were wrong and I had to pull up the legend. The green circle with a checkmark was easy for me to understand. My files were in sync with the repository. The icons with the exclamation marks were less so.

The icon for conflict mimics the icon that Windows uses for errors, that is a triangle containing an exclamation mark. What did the red circle with the exclamation mark mean? Is it as bad? No, it just means modified. These are the files that will get submitted when I do a commit. Fortunately I didn’t need the legend for very long. When in other parts of the program, like the change log, you can hover over the icons and get a tooltip explaining what they mean. It didn’t take very long to learn.

Other parts of the program are accessed through the context menu that you get when you right click on a directory managed by TortoiseSVN. It’s got three lines in the menu: Commit, update, and TortoiseSVN. The first two are the most commonly used SVN commands so it’s great that they’re so easily accessed. The latter opens up to all of the less used TortoiseSVN commands.

The menu that opens up contains 25 items, related ones are grouped, and sections are separated by horizontal rules.

Non-shell elements

The first thing that I did was point to one of the directories that’s got the modified icon. I select update. Here’s what I see. The dialogue at the top is what popped up, the dialogue at the bottom is what I got when I clicked “show log.”

I’m not surprised by the original message. That’s what you’d get if you svn update through the command line and there’s a modified file. The svn log window is what you’d expect from a svn log command on the command line, just presented differently. It makes it easier to input parameters but doesn’t add any new information. This seems to be the case for most of TortoiseSVN. There may be more added on functionality in features like merge, but those aren’t the features I touched as a student working on a collaborative school project. I’m under the impression that TortoiseSVN is more of a front end than added functionality.

Browser and graphs

Take a look at the graph in Revision Explorer presented by TortoiseHg in this screenshot taken by Joel.

Compare it to the graph presented by TortoiseSVN to me.

To be fair, let’s also compare two windows with the same title. Here’s the window in TortoiseSVN called Repository Explorer, compare it to Joel’s screenshot. The TortoiseSVN Repository Explorer doesn’t have a graph in it, but contains a pane with a table of version control metadata like revision number and modified date. It has a directory browser where TortoiseHg does not.

I was a little confused at first because the TortoiseSVN graph is called Revision Graph but I only see two nodes in there. I know that the files contained in that directory were the most heavily modified files in the repository. At least ten people touched them and there were dozens upon dozens of commits to them.

After going through TortoiseSVN’s help page on revision graphs I see that the graph isn’t meant to do what I thought it would.

Sometimes you need to know where branches and tags were taken from the trunk, and the ideal way to view this sort of information is as a graph or tree structure. That’s when you need to use TortoiseSVN → Revision Graph…

This command analyses the revision history and attempts to create a tree showing the points at which copies were taken, and when branches/tags were deleted.

It’s not meant to be a very granular view but instead a quick way to find where each copy came from.

TortoiseHg’s tool is also meant to help you with maintenance but its default filters are broader than TortoiseSVN’s.

The changelog tool (also known as the repository explorer) is used to visualize the revision history of your repository and to perform any maintenence tasks that involve changesets. It presents a graph of the revision history, showing the parent/child relationship of each change. At each revision you can view the files that were modified and the contents of those changes.

TortoiseHg has configurable filters and its defaults show all changes. TortoiseSVN does allow you to check “Show all revisions” but it’s still a bit confusing. I couldn’t work out how to refresh the view to get all the revisions, and closing and reopening the browser showed me the same thing.

Which approach is more useful to Basie users? My immediate guess is TortoiseHg’s wide open filters since students rarely create branches or tags. We can still support filters out of consideration to the use case where students build off an established open source project and would want to find the right tag or branch to import.

The TortoiseSVN approach is great for release management and importing, both of which are extremely rare occurrences for student projects. The TortoiseHg approach with its default filters makes it easier to get up to speed after you’ve spent a couple days away from coding because you were working on other courses. The downside to the TortoiseHg approach is that it works based on commits in general and it’s harder to narrow down to a chunk of the project that you’re working on.

So how about a hybrid that lets you browse to the couple of files you’re working on, and then see a graph of all revisions?

Ending thoughts

I’m the opposite of a power user of SVN. I used TortoiseSVN so that I didn’t have to keep a reference chart handy. It made it easier to read output. Now that I’m trying to dig deeper for ideas I see a couple things in it that I really like.

• It’s nice to have hyperlinked features, like being able to diff from within a commit window.
• It’s also nice to speed up a few things like creating patch files from a diff.
• Using the shell lets the user set his or her own preferred view in an already familiar environment. Can we make Basie’s browser let the user select different views? Is it worth it? Should we mimic conventions present in Windows, OS X, KDE, Gnome?
• Visualizations of changes are great. Maybe we ought to be able to do it by groups of files rather than commits, though.