The city’s usage metrics show Portland’s muni-network is delivering value to the residents who are using it but performance metrics, gathered by two Portland State University faculty members, show spotty coverage.The city’s usage metrics show Portland’s muni-network is delivering value to the residents who are using it but performance metrics, gathered by two Portland State University faculty members, show spotty coverage.
So, what is going on? The question is becoming increasingly crucial as more muni networks come online. Effective evaluation was a key issue at our Dallas conference and both
and have discussed it at length.What is interesting about Portland’s case is that these reports offer two different perspectives on “success.” One measures user activity to determine the value the network delivers, the other measures technical performance.
Last week, I spoke with Logan Kleier, Portland’s IT director, about the metrics the city is using. Developed with MetroFi, the city’s service provider, the metrics attempt to monetize the number of hours users are spending on the network and what that means in terms of value delivered. Based on current usage, the city projects that by the end of 2007, the free network will save city residents $5.4 million from what they would have spent on competing commercial services.
I also talked via e-mail with Caleb Phillips, one of the two PSU faculty who did their own performance analysis of the downtown pilot. Phillips and Russell Senior presented a preliminary report on their findings this week to Personal Telco, a Portland user group that operates its own mesh of free hot spots in the city’s downtown. Phillips said the two based their study primarily on the list of requirements in the city’s RFP.
“This includes several metrics for performance and coverage,” he told me. “Since our test was done without access to the MetroFi infrastructure (or any more access than any citizen would have), we don’t have access to statistics about usage. However, both MetroFi and the city have been posting some monthly statistics.”
The city’s usage statistics can be found at
.More on Phillips and Senior’s study can be found at
, including links to press coverage about it and the two’s response to which challenged their objectivity. Phillips and Senior had bid to do the official performance evaluation for the city. That job went to Uptown Services.A detailed summary of Phillips and Senior’s presentation can be found at
; it also reports on activities of NetEquality, the Portland non-profit that is bringing free wireless to low-income residents in the city and discusses the timetable and tasks ahead for Metrofi in Portland.Uptown Services, the group that the city hired over the two PSU faculty members to evaluate performance of the downtown pilot, is scheduled to deliver its report on performance April 12. Phillips said he and Senior expect publish their full report before that time.
Performance in Portland is getting no shortage of attention. Novarum, which publishes
, has not reported on Portland’s performance but plans to include it in its next published report.Stay tuned…
One correction: Russell and I are not PSU faculty. Our bid for the original testing IRFP had two PSU faculty on the bill (as the chief bidders), but they haven’t been directly involved in our recent tests. Of course, they worked with us to establish our testing methodology, but aren’t responsible for the current work. I am, a student at PSU, where I hold a part-time research position in the mobile computing lab. Russell has extensive experience as a scientific data-analyst, but isn’t employed by PSU – he was on the bid primarily because of his experience mapping wireless networks in Portland.
Thanks for the write-up Carol! I’m sorry if I was misleading
.