About

Welcome to Panela, Matt Harrison's take on mostly Open Source, Linux, Python, innovation in those areas, other buzzwords and Dick Proenneke. It comes complete with the illustrations as needed. Note the opinions expressed here are merely my opinions and not the opinions of my employer.

about Matt

Calendar

««May 2008»»
SMTWTFS
     123
45678910
11121314151617
18192021222324
25262728293031

My Top Tags

                                       

Mailing List

My RSS Feeds








Latest Entries

pyExcelerator (xlwt) cheatsheet (create native Excel from pure python)

2008.04.26 Sat 11:37 P GMT-07

If you are looking to generate/create/write (but not read) excel spreadsheets from native python (read on linux or macos or even Windows!), then xlwt (a fork of pyExcelerator) is your friend. This library, (and the ancestor) is somewhat unknown, but quite good. The only real problem with using it is that documentation (and the bug I mention below), (though this blog post is pretty good) is somewhat spotty. But you can usually get around dearth of docs by reading the code that reveals that the internal model for .xls documents is quite good, and that is supports most features (sans graphing). The original MIA author did quite a good job.

I've made a cheatsheet(pdf) for these tools. Inkscape still has some issues exporting this but here's the source for it (and the formattings bad, but I thought I'd just get it out there). It's an executable cheatsheet, so if you run it (the .py not the .pdf), it will create a spreadsheet. Hopefully this is useful to some folks. I've also tried to embed docs into the code that will serve as useful hints.

Also, I'm using a factory to reuse styles and fonts, as it seems that if you define too many of them, Excel has problems and crashes (note that OOo won't manifest them and happily plugs along on the non-conforming files). Please send any feedback or comments on how this could be more useful.

Also note, this is the .xls format, not the newer xml/ISO/.xlsx stuff

Lasers/Wiimotes/Python presentation at Utah Code Camp

2008.04.25 Fri 8:19 A GMT-07

For those in the Utah vicinity, and who want to see a cool presentation about python, wiimotes and lasers (or who really like MS products, since most sessions seem dedicated to that), my brother is presenting at Utah Code Camp. It should be a little cooler than the Pycon version, since some more stuff has been added.

Oh, and I guess there is one other python talk amidst the sea of MS technologies, Kevin Kubasik is talking about twill and lxml.

[Pycon 2008] Lasers, Webcams, Wiimotes and Python Video up

2008.04.25 Fri 8:15 A GMT-07

My brother gave a talk about using python to detect lasers with webcams and also demo'd a prototype of a 3d game with wiimote headtracking. For those who missed the preso, there's now a video up.



Laser dudes from Utah, originally uploaded by BrianLy.
tags:                

GSOC Project for Branch Coverage reporting for Python

2008.04.02 Wed 12:43 P GMT-07

As there appears to be some interest in branch coverage and the cool testing/metrics/quality tools that can be built upon it, I've proposed a GSOC project to start the ball rolling. (I assume it will start rolling either way, but allowing someone to devote more time to it is always nice).

I put a short writeup on the Python SOC wiki page. Here's a little more info. There's only so much one can do with line coverage, but that's the best python has at the moment because the one can only trace at a line level. If tracing were enabled to happen at a condition level, then branch (and path) coverage can be reported. For people who want to go hog wild and try to do full path coverage, they can go ahead. For those who want a little more than branch coverage, there is Structured Testing. (For people who don't want to go that far, perhaps you have a bunch of tests that take a while to run, if we have path data, it would be able to tell you which tests were redundant and you could cut your testing time).

Here's my thoughts on what the project might include:

  • Update sys.settrace to enable branch coverage (or add a new function)
  • Create library to implement branch coverage using above tool
  • Create command line app using said library to implement branch tracing (aggregating subsequent runs) and dump/pickle data
  • Create command line app to load and report on data
  • (Test above)
  • Bonus stuff
  • Calculate flow graphs
  • Calculate basis paths through graphs
  • Take branch reports/basis paths and determine missing paths/tests
  • Create visualization for flow graphs/extend Ned's work
  • Create visualization for basis paths (in pygame?) that links code to flow graphs and tests to executed paths
  • Redundant test finder
  • .... cool stuff you think of

I'm willing to mentor this. If you are interested please or have suggestions please let me know. Comments? Other ideas?

iSight? We don't need no stinking iSight

2008.03.25 Tue 9:41 A GMT-07

My brother and I have put up a blog discussing our messings with lasers, webcams, wiimotes and python. If you saw his talk at Pycon, and want to play with the code, you can find it there. I recently made the laser stuff work in Linux (hence the title of this post) (should be straightforward to port for windows folks). For those who didn't see his talk or BoF session, there's some videos there now explaining exactly what the games do. Enjoy!

tags:              

Undeleting photos on (gentoo) linux/windows/mac

2008.03.24 Mon 3:54 P GMT-07

This post is preceded by a public service announcement to back up all important data, often. (So far it seems the best way to back up photos is to get them printed/developed).

Had a scare with a half gig compact flash card acting up on me, that contained photo documentary of my daughter's birthday. Enter Gentoo Wiki to the rescue. (Note that this site contains information that could be useful for other distributions as well, like say fedora or ubuntu....).

Here's a nice link explaining different steps to take to recover photos.. A few notes, are that

dosfsk
runs in place (seems obvious), but when I have a wacked out drive/partition, I really want to copy it (using dd or something) and work on the copy, not on the original. Anyway, dosfsk claimed to fix/rename a bunch of files. Then I started trying to use
jpg-recover
. I had issues with it (it's slow and it seemed to hang my computer). Plus I don't think just recovering jpg's is sufficient these days, when you have cameras that now double as camcorders.

So here's the most useful advice. Use PhotoRec which also recovers movies. It uses a (semi-confusing) wizard interface. The User guide walks one through the steps. The most confusing issue was selecting the partition table (what is an Intel partition?). The Canon (fat16?) format was recognized when I selected "None" as the partition table type, but not when I tried Intel partition. After that it ran (really quickly) and recovered most/all of the files.

The homepage of photorec says that it runs on windows and macs too. So if you find yourself in need check it out!

[Pycon 2008] Linky linky

2008.03.21 Fri 10:20 A GMT-07

Here's my notes on Pycon this year. I'm going to throw out a collection of links that may or may not be useful to you. If you weren't able to make the conference, hopefully this list contains a link or two that are new or interesting to you.

  • elasticwulf slides
  • ipython1 is branch of ipython for running parallel jobs.
  • elasticwulf is a set of python tools for controlling beowulf clusters on ec2.
  • mpi4py is python bindings for mpi.
  • EC2 (according to one session, but not according to another) has some problems, look for the delicious tag: ec2+gotcha (which appears to be empty, perhaps there aren't problems http://del.icio.us/tag/ec2+gotcha )
  • pyke is an expert system written in python. (Attempt at rewriting prolog in python?)
  • gdata-python-client provides tools for reading/writing google data from python (programatically update spreadsheets, etc)
  • Firefox EC2 Extension allows one to interact with ec2 instances from a browser.
  • s3organizer extension allows one to interact with s3 from a browser.
  • boto is a python interface to amazon's webservices, s3, ec2, etc.
  • Use paramiko for scripting ssh sessions with ec2 instances.
  • Normal accident theory if you call echo "">/dev/null on too many instances they will crash!?!?
  • Trestle is a nose plugin for writing doctest like tests for webapps.
  • Thrift is RPC for facebook (NIH). It has python bindings.
  • haufe.eggserver allows for local pypi repositories. Could be useful when building your projects that depend on wheels that are constantly moving.
  • Python metaweb/freebase client code allows for querying the metaweb online database (like wikipedia, but graph based).
  • Ella is a cms for django that announced their open sourcing? at the conference.
  • Use argparse for parsing command line scripts if you don't mind pulling in one more library and optparse isn't quite doing it for you.
  • Orbited can be used as a comet server (server push instead of client pull).
  • If you forward emails to plans@tripit.com you will get back an itinerary
  • Wiimote head tracking/laser/webcam blog very cool demo code for wiimote head tracking in 3d enviroment and webcam/laser detection in python. (Disclaimer: this is my brother's blog, I've had a small hand in some of this)
  • Traits is an event system for python
  • Pystage (my google-fu doesn't turn up a good link here), is a python based animation engine. Presenter used it to propose to fiance. Sadly his laptop appeared to have been attacked by powerful magnets....
  • Five dash is python based open source accounting software (think quickbooks).
  • Geodjango extension to django to help with geospatial data. Jacob said this will go into the trunk soon.
  • Openlayers js library to do googlemaps type interfaces.
  • snpedia allows you to upload your dna and tells you your traits (either good or bad) has a social/wiki aspect to it.
  • Facebook is hiring and doing some funky php/python wrapping, that quite frankly scares me.
  • geopy allows one to find coordinates of addresses, cities, etc.
  • pvote lightweight (<500 loc) voting engine, that should be easily certified as "secure".
  • TGPPL is a new license that morphs from GPL to BSD overtime. Thought to follow same ideas as IP/Patent protection.
  • pyprocessing is an attempt to get around the GIL by making a library that uses processes that has an interface very similar to the threading interface.
  • pprocess is another GIL hater.

[Pycon 2008] Thoughts and Observations

2008.03.17 Mon 11:22 P GMT-07

Here's my thoughts regarding Pycon 2008.

Benefits of conference attending

  • Face time with virtual friends - Some people/IT shops invest in established technology provided by big vendors. I've been somewhat disenchanted by the products (and underlying lockin/intentions) of many vendors. Yet open source (for the most part) has provided wonderful cutting edge solutions and tools. As you invest in these tools such as these, you enter this "onion of open source" (by onion I mean consider a sliced onion, where the layers correspond to moving deeper within a project, ie: lurker, downloader, user, mailing lister, bug reporter, patcher, ircer, committer, ...).

    Just as you invest (money) in big companies for their tools, with open source, you invest time. But the paybacks for developers can be great. Rather than interacting with a huge faceless entity, oftentimes you can come mingle with the primary developers directly.

    It's very nice to be able to say "Thanks for your excellent software, and your help" in person.

  • Get up to date - I don't have time to read blogs all day long, so sometimes it's nice to get a firehouse of cool stuff that people are doing, that may or may not be directly related to your work or interests. A conference such as Pycon or OSCON can provide this.

  • Network - I don't consider myself to be a good networker, but if you are interested in something (like say python), it's good to be in an environment surrounded by people who probably share some subsets of interests with you.

  • Get a job - Perhaps this is related to the previous (but since I'm not looking for a job, I'll separate it). Vendors come to hire, and they realize that people attending a conference are probably more "into" the subject of the conference than those who don't attend, so it's one way to get a "foot in the door". Though, the older I get the more I realize that good job opportunities will probably find me via my network, than less effective means (such as me spamming my resume).

Food

Food quality went down this year. I guess the box lunches scale better, but the taste goes down. Also, previous years have had talks during lunch. This year lunch was the hallroom track. At least put vendor/lightning talks during that time.

Session quality

Disclaimer: I gave a session and convinced my brother (who was a python newbie at the time) to submit a session (which he gave).

Having attended a few open source conferences (OSCON, PyCon, Foss.in, Mysqluc, OSBC, LinuxWorld, Mashupcamp, UTOSC) and user group meetings, I realize that sessions are a mixed bag. You've got vendors, who want to tell their story to any eyeballs they can. There's the community figureheads who want to get their point out. There's community members who think they've got something interesting to say. Then you consider the level of the talk: beginner, newbie, advanced, requires 3 PhDs to understand. There's also the nature of the talk itself, is it more breadth or depth oriented?

I can usually tell within the first five minutes whether the talk will be interesting or valuable. The value derived from a talk can come in different ways. Sometimes, a URL or two are the most important parts of the talk. Sometimes the whole talk is interesting. Other times the title sounds interesting, but the content is unrelated or presented at a level below or above what was indicated.

Regarding Vendor/Sponsor talks. I'm comfortable with them. What I'd actually prefer would be a lightning session dedicated to them. Ask the vendors to describe the work they are doing, and where they are located. Don't tell me how nice the office is, or the view, or the size of your monitors. If I'm interested in working with you (it will probably be because your work is interesting), and I'll come and find you and then you can try and sell me on the other benefits (monitor/office size etc). Or preferably if you know who I am and what benefits I can provide, you can contact me. (Perhaps the conference registration could allow job seekers to note their skills/blog/projects and if they are interested in being recruited).

Regarding keynotes. I think there are too many. Rather than two 20 minute talks and an hour talk, just give me one or the other. And then have another session or two, that would give some room for another 24 talks. (Which seem to have about the same hit/strike ration as the keynotes)

Regarding talk quality. That's a hard one. You want the good consistent speakers, but also don't want to exclude other's who might be good, just not known. Perhaps a preparation checklist for speakers might be nice. I try to practice at the local user group previously. Also, I like to state the level of the talk and what I'll be covering up front. That way if there are two talks you are split between (and strangely there seemed to be a few sessions where the tracks seemed to overlap quite a bit), and mine isn't what you really thought it would be you can skip out and try to make the other one.

Another thing that is useful (but is after the fact) is letting the speakers actually view they videos of themselves. Then they can see how they look to the audience. Personally I can say that it is a scary and humbling thing. But the only way to get better at speaking is to practice, and make note of what you do good and what you could do better.

Here's another hint: Don't read #pycon before you have to talk, its somewhat ruthless nature (towards speakers) can be quite intimidating ;)

Regarding talk level. If you want to grow the popularity of python (admittedly that is not the goal of everyone, some like to be exclusive with their tools), then you need to cater to all segments, beginner, intermediate and advanced. Perhaps some (good) beginner talks will recur year after year (or every other year) because they are timely, yet useful. Perhaps some advanced topics can span two sessions. For example, I've programmed in python since 2000, but have never ventured into the c guts. A session (or two) on how to do that might be nice (it could also help find new core contributers), and it could be useful year after year to a new crowd. I'm pretty sure an Intro to Twisted would be packed about every year....

Regarding talk content. I can and have spoken about many different python subjects. But when I apply to speak, I try to speak on something that I think is interesting or useful, but not necessarily what the audience thinks is useful. I think speakers like to speak on new or novel things (more academic). But there are talks at all levels that could be quite repetitive but still useful. Perhaps for a month or two previous to the call for speakers the organizers can publish what topics people want to see. Or perhaps there could be specific tracks (ie testing, web, scientific, gui, automation, etc).

Regarding open spaces. Some have suggested that pycon should be taken back to the community and become an unconference. I don't know that that scales to 1000+ people. (Or that it would attract that many people. It's a risky proposition to spend 3 days at a conference that could talk about anything). When I attended mashupcamp (a pure unconference) that was supposed to have the who's who (I was sitting in for one), I found 70% of the sessions extremely boring. It was probably worse than most conferences. But I guess with the Bofs/open space rooms available all day anyway, this could be/is done already. Note that I have found Bofs useful, I'm just not sure I want 100% Bofs.

Vendor Swag

With ultimate (frisbee) leagues and conference attending, I have more than enough t-shirts. But I must say that the conference shirt and the Leapfrog Online shirts were quite funny. Good job.

It's always nice to get something the kids might like. But please, if your swag has the following warning:

Sonic Rocks are powerful magnets.  Always keep them at a safe
distance (at least 20 inches) from magnetic storage media ... such
... as ... hardisks ... and laptops.

Please tell me not to put it in a bag right next to my laptop. (There were quite a few laptop/presenter glitches this year .... possibly .... related???)

Laptop/software thoughts

Apple appears to be the product of choice with about 2/3rds of the market here. The remaining third again split amoung linux/windows. Saw about 10 eee's, 2 macbook airs.

Most presentations were done in keynote or s5. S5's lack of image scaling is bothering me. Me thinks that clutter or bruce need needs an rst converter for slideshows....

Students (that I talked to) seem to be almost 90% on linux. Not sure if it is the geek factor, lack of windows fee (laptop already comes with windows so....), or they are just forward thinking/wishful.

Attendance

Last year had 40% growth to 600. This year saw 1000+ in attendance. I'm not sure the venue will scale to next year. The halls were pretty crowded.

Saw DHH briefly in attendance on Saturday. Weird.

Conclusion

No conference is perfect, but I had a great time this year. I highly recommend PyCon for developers as well as companies that are using python (not to pitch us, but to interact with the community and to form a sybiotic relationship instead of leaching one).