Philly Wireless – Third Times the Charm

Last week Philadelphia announced it hopes to buy the wireless network from a group of local investors who bought it from EarthLink about a year ago. This on the same day the Feds announced the first batch of stimulus grant winners.

As I finish edits to the revision of my 2005 book on broadband strategy that focuses heavily on Philadelphia as a model for effective broadband needs assessment and planning, it warms my heart to be able to bring the Philly story around full circle for readers. I’m contemplating the role city-owned broadband networks could play in the national broadband strategy plan. The Gates Foundation’s call to wire all of the U.S. institutions (hospitals, libraries, schools) set the stage for this government connection.

But first, let me put to bed one of the criticisms I’m already seeing in a few places. Why buy a network with old (erroneously assumed to be obsolete) infrastructure? I called Tropos, whose access points hang all around Philly to verify my assumption that new 802.11n infrastructure and the City’s fiber lines they plan to integrate with the network will enable it to perform very well.

“The City can seamlessly integrate the 802.11n-based routers from Tropos as well as our latest mobile routers with the existing network’s Tropos 5210 router,” states Marketing Director Denise Barton. “The software is designed to be backwards compatible. Since the network will be mainly for municipal applications, we’re not talking about the general public using it. The old routers will be just fine.”

History and many muni successes favor Philly’s move

The other myth that won’t die is municipalities have failed, so why is Philly doing this again? Practically every muni wireless network project that failed was run by dyed-in-the-wool, free market lovin’ private companies. This includes the original network in Philly. Conversely, dozens of city-owned and run broadband networks used for local government purposes have proven to be highly successful.

Santa Monica, CA switched from its slow moving, expensive communication systems to fiber and saved $750,000 in the first year. With the ongoing savings, they fund wireless video streaming to police vehicles, connect all of traffic signals and traffic cameras, link parking structure signs to the network to tell people if spaces are available and build hotzones for free wireless access.

Oklahoma City and New York City both have citywide government-use networks that were launched supporting one or two applications, and each now supports over 200 applications that collectively save money, increase revenue and improve government service delivery.

Corpus Christi estimates they’ll get $1.6 million in savings over 20 years with their smart meter reading application, and the traffic signal control system is cutting costs, saving time and making it possible to universally manage more traffic lights.

When Houston conducted pilot projects for its smart parking meter application running over the City WiFi network, they saved $20/meter in carrier costs and increased revenue to the City by 30%.

Providence, RI estimates its building inspectors save about 2.5 hours per inspector per day by being able to complete paperwork in the field with its government-use network. You can read about these and other cities’ benefits derived from their networks in my Municipal Broadband Snapshot Report.

Philadelphia has over 2,000 mobile workers. Take the average worker’s hourly salary  (wages plus benefits) and assume these workers become only half as efficient as Providence’s workers, so they save only an hour a day. That’s 500,000 hours of personnel time Philly can keep in the field at a time when the city is hard pressed to do more with less.

This is probably not new to many of you. But we’ll likely need to trot these stories out repeatedly as the naysayers come dropping out of the woodwork. What you may not have thought about, though, is the role these communities can play in the national broadband plan.

Bill Gates, the FCC and local government anchor tenants

In October, The Bill & Melinda Gates Foundation sent a note to the FCC advocating for the wiring of all of our institutions. You can read my complete analysis here, but the main thrust is:

  • this makes sense because it’s easier to extend fiber from hospitals, libraries and key facilities to businesses and other paying customers than building infrastructure out to these places from scratch;
  • many of these institutions (except possibly libraries) represent significant potential on-going revenue as anchor tenants and/or investors to help financially sustain the network, especially if you can cover institutions’ upfront buildout costs;
  • the fiber network at institutions gives communities a foundation infrastructure upon which they can build wireless networks;
  • these institutions can be a highly influential catalyst driving broadband adoption by giving constituents content and other reason to go online to interact with instituions; and
  • local governments are the mother of all institutions and they definitely need to be added to the list.

Since Philly is one of the country’s largest cities (thus high visibility), it has the wireless network and it has fiber, they make the perfect testbed for the Gates institution strategy. Extend some of that Foundation grant money to Philly that was recently announced. And if the broadband stimulus grant proposal for public computer centers in the city wins, throw that in the mix too. Then watch or stage-manage how the elements come together to provide coverage for those who need it the most, even if the city itself is not driving broadband adoption.

Urban areas are likely to continue to be left out of the lion’s share of the broadband stimulus money. And unless someone really fights hard for them, there’s a chance they may not benefit much from the FCC’s efforts to reform the Universal Service Fund either. Therefore, this proposed exercise may be doubly valuable as a means of tackling the digital divide in urban America.

You can read some of my other analysis on the state or broadband and potential next developments in our march to better broadband at my blog, Fighting the Next Good Fight.

Comments

  1. Any laptop, desktop, server, smartphone, or Wi-Fi networking gear three+ years old is obsolete in my view. It’s Moore’s Law.

    If the plan is to continue with Tropos, upgrading firmware and deploying their 802.11N gear, has sufficient testing been done in Center City, the very core of Philadelphia, where the deployment of Wi-Fi would be most challenging?

    The Tropos gear is all over Philly, except in Center City, which was left for last. Now that the mission/plan has shifted from providing affordable/even free Wi-Fi to providing government/muni services first, getting it right in Center City is going to be crucial. Is continuing with Tropos the right choice?

    Tropos claims that the current gear should be sufficient to support muni applications, which will not tax the network like public Wi-Fi would. But as soon as you have a resource like this, it will be used. How future proofed is this, especially since we are told that the network will be built out with 17 million between 2011 and 2015?

    I agree with Craig that this was a canny acquisition for Philly, and that wireless muni services will bring new efficiencies. I ask though what has happened to addressing the digital divide in Philly, and whether especially given the requirements and the timeframe whether Tropos is / will be the best way to go from an infrastructure standpoint.

  2. I agree Marshal! The “it’s only the City” comment really translates to it will be slower than Christmas and will require huge amounts of cash to be brought up to any usable speed at all. Of course with federal stimulus money, it can be done. The question really is, why? The city could deploy other technologies better suited technologically and for less money. Rebuild the network from the ground up guys. Don’t be a Tropos guinea pig for a solution you will spend millions of dollars to fix then outgrow in less than 6 months from the go live date. Let common sense prevail!!!!

  3. To be fair re: the Philly deal abandoning those on the other side of the digital divide, Craig says elsewhere in PC Magazine http://www.pcmag.com/article2/0,2817,2357395,00.asp

    “For the most part, the broadband stimulus program has screwed the urban poor,” Settles said in an e-mail message. “However, city ownership of broadband could be a powerful new element to the national broadband strategy plan that the FCC is writing, and a path to broadband adoption for low-income citizens.”

    Given today’s budgetary realities, if you can make the case that something will save money, it’s a lot easier to fund. While increasing municipal efficiencies gets funded, making internet available and affordable doesn’t in part because the benefits are much harder to quantify.

    The NTIA would love to know how exactly digital divide initiatives move the needle — more jobs, higher graduation rates, less crime? — we need data. It’s a lot easier to say ‘we’ll need fewer meter maids.’ We may know intuitively that putting people on the grid makes them more employable, more informed, but this needs proof.

  4. I’d like to clarify an apparent misunderstanding by several of the folks who commented on this blog previously. To reiterate — the Tropos routers in Philadelphia will work just fine. There is not a need for the City to purchase new routers at this time to make this asset a high value resource for municipal services.  The Tropos 5210 broadband mesh routers continue to be purchased by customers today.  Where are they being used?  Oklahoma City, OK; Tucson, AZ; Corpus Christi, TX; Lompoc, CA..and the list goes on.  802.11n-based products are the latest and greatest and highest performance and Tropos makes these available as well, however, they are not needed everywhere to build a high-performance network.

    To give you a perspective, the network in Oklahoma City covers 555 square miles and all mobile city operations use the network for communications in the field.  This includes public safety (PD, FD, EMS); building inspectors; public works; animal control; and more.   In addition, machine-to-machine types of City applications utilize the same network infrastructure – traffic light controls, weather monitoring sensors, video cameras.   The network handles over 4 terabytes of traffic each day – check out this press release for more details.

    You can check out the Tropos website for more information on how customers are successfully using our wireless broadband mesh routers — 802.11 a/b/g/n today at http://www.tropos.com

  5. James Hay says:

    Denise,

    I’d be interested to know regarding the current routers hanging in Philadelphia how many radios are in each, what the unit’s throughput is, and what the path throughput would be if, say, I had to traverse 3 or 4 wireless routers from my location back to a wired point.

    Thanks,
    Jim

  6. Yes it can be done, and done on the “Cheap”, when compared to others gear.

    The Tropos are a fine product, for the Muni end of it.
    But to provide service to the end user (John Q. Public) you need to look at http://meraki.com/

    The products they sell can roll over that digital divide, it will be just a small speed bump.

    Inexpensive… not “Cheap”… well built mesh networking gear for the bugdet minded.

  7. A few comments:

    As the Chief RF Engineer for EarthLink on the Philly project, I can say that I am very happy to see the city take over the network and keep it alive. Had the city adopted the attitude of being an anchor tenant and and paying EarthLink for the use of the network, EarthLink would have never bailed out of the project and it would have complted the total buildout. The existing Tropos radios can deliver 6 megs to an end user as they are installed now, we tested that in real time, and on that version of firmware it seemed to be the maximum we could achieve. If there is extensive use of fiber to backhaul the traffic, performance should be maintained. I have not worked with any of the upgraded Tropos firmware, but I have no reason to believe that it won’t perform even better.

    There were many lessons learned on this project. From a technical perspective the network worked as designed. What didn’t work as planned, was the financial model. Early on, EarthLink failed to hire true wireless engineering talent and therefore went on very poor assumptions of how many transmitters per square mile (around 30) would be required to build a working system. The actual deployed design required around 52 per square mile. That was due to taller buildings in residential neighborhoods than assumed, and dense tree cover. This increased the deployment and operational costs significantly, which of course caused the original financial models to fail. Lesson learned, don’t trust all the marketing hype to be used in for financial forecasts….

    An interesting side note, I had conducted some GIS and demographic studies for a round one broadband stimulus applicant. We looked at census block level data for broadband adoption rates below 40%. When I mapped this information, it matched almost exactly where the EarthLink network was deployed and on air. Now either adoption rates were still poor, or the data company was asking the question “Do you SUBSCRIBE to Broadband Services?”. If the later is true I can see why the adoption would be low, people were getting free internet from the system. If the former is true, then the point should be made that “Even if you build it they will not come”. It certainly is a data point that should be looked in to further as far as the consumer aspect of a network like this is concerned.

    AT&T should be courted as a potential client. Given the fact that they recently stopped selling IPhones in NYC, I can imagine they are running in to similar capacity problems in other metro markets. If they were to update software in the phone for Philly customers to automatically roam on to the Wi_fi network, that would relieve the stress on the cellular network.

    There is certainly a place for Muni networks in the national broadband plan. The original notion that commercial companies would build extensive (and expensive)infrastructure and government would get to use it as a free ride, is finally gone by the wayside. Those of us who have actually built telecom infrastructure knew that idea would not work out. Government officials should have known this too. Now that there is actually some serious cost savings and cost benefit analysis being done, things don’t look so bad on paper even if the government has to contribute some costs to the project. While these wireless networks will never be able to deliver capacity that fiber can, they don’t need to. They can serve specific functions and be an important adjunct to the world of fiber to the curb internet. There will always be a place for the wireless mobility components.

  8. All the deployments you mention are in places where it’s low and flat. Whereas there are Tropos nodes all over Philly, the 1.5 square miles that make up Center City, the heart of Philly, where all the tall buildings, stores, and municipal buildings are, do not have any Tropos nodes.

    Could the new N Series products work in this environment of echoes, shadows, and wireless interference? Maybe. But that needs to be proven before the city goes and buys, or if possible upgrades the nodes it may have in stock.

    Depending on when this rollout of muni Wi-Fi is supposed to happen in Center City — 2010? 2012? — the answer may or may not be Tropos, depending on what is happening elsewhere in the industry.

    True, having Tropos there as the install base argues for staying with them, should they meet the requirements. But as we do not know the timeline and the requirements in a field that is evolving rapidly, it is hard to say now what the Philly network will eventually be.

    I am sure that is exactly what Philly’s CTO is trying to figure out now — what the rollout plan is. This is to be a six year project.

  9. In response to the question from Mr. Hays, regarding the performance of the current routers in Philadelphia — the network was never fully deployed nor optimized by EarthLink. Stating performance of the network in its current state would be inaccurate and not representative of its performance potential.

  10. Craig’s points about the proven value of digital mobile services to reduce operating costs and increase efficiency are compelling.

    However, you must have a network that works in order to begin to realize those benefits. The new Philly experiment will be interesting … but it WILL be limited to only a few of the possible applications.

    Novarum tested Philly several times during its construction, from the early mid-30 nodes/square mile up to the end at over 50 per square mile. As our analysis then showed – good performance on mesh networks with laptop clients DEMANDED greater than 50 nodes per square mile coverage.

    An 802.11g legacy network (such as Philly’s), deployed at 50 nodes per square mile, can provide for competent offloading of city services to high power mobile 802.11g clients with better than 95% coverage – but will likely only offer about 80% coverage to standard low power laptop clients – materially below the 90-95% coverage that modern 3G networks provide. So not just any client will work.

    Further – an 802.11g network (even at 52 nodes/square mile) will be expensive to maintain … with node costs 2x the list price of newer 802.11n systems such as Ruckus or Meraki or others.

    An 802.11g network … will only be partially useful for offloading smartphone traffic – at this node density, our measurements show that 802.11g smartphones (with lower TX power and poor antennas) get less than 50% coverage. So extended hot spot coverage is possible … but likely only with additional investment to make sure of adequate service.

    I hope it delivers value to Philly.

    Our recent testing of 802.11n networks strongly shows:

    1. A decrease of about 50% in capital costs over the previous 802.11g generation (under $2k list price access points vs $4K).
    2. Dramatically improved coverage for low power devices such as laptops and smartphones.
    3. Dramatically improved performance – that will equal or exceed future LTE or WiMax performance.
    4. Improved indoor coverage. 802.11n laptops as clients to 802.11n access points have better exterior wall penetration than legacy 802.11g CPE.

    I would recommend

    1. other cities that a modern 802.11n network can probably be deployed today with MUCH better coverage, MUCH better performance at better than half the capital expense of the Philadelphia network when first deployed.

    2. Philadelphia build out City Center with 802.11n product that will have superior coverage in the high multipath environment of urban canyons. The Tropos 802.11n product is a very good product – though not as attractively priced as competitors from Ruckus and Meraki.

  11. Dennis Holmes says:

    Denise,

    I’m betting you work for Tropos. I do not see any value in delivering service with an obsolete system when there is funding available to provide robust, reliable, and faster service with multiple radio units. Assuming that Philly really wants to make a impact in delivering service then they should upgrade the network. Yes, I know the other cities that you mention and I know the amount of bandwidth they consume (miniscule). In today’s world application vendors and users require true broadband connectivity. AMR, traffic control, and similar low volume aps can run on anyone’s network but when you factor in things like video streaming and video surveillance you had better provide adequate bandwidth. As for 802.11n, you really don’t gain alot from using true 11n. Packet aggregation performs poorly in most cases when attempting to provide wireless backhaul over multiple hops (adaptive routing protocols just can’t handle it). Spatial multiplexing is virtually impossible and finally, anything you gain on the client side you will ultimately lose on the backhaul side. If we are talking MIMO then all you really can gain from is MRC diversity and beamforming. This is the only real gain from any of the 11n technologies and is provided by almost all vendors now. Philly needs to take the approach of designing the network for capacity, reliability, and speed. If that requires replacement of obsolete “routers” then so be it. I agree with Ken and Phil’s studies that the use of 11n features like MIMO and client beamforming will make a huge difference at the client device but stating that you will get true 11n speeds is ridiculous. 150meg from a mesh node? Come on, really? I am anxiously awaiting the rebirth of Philly and hope that some common sense approaches will prevent the mistakes of the past. Tropos, you’ve got way too much to lose if you let this go south on you. If this goes wrong then we (stakeholders in municipal wifi) will all suffer from the bad press. Just like before. Please get it right Philly! Not just what’s right for Tropos. Good luck Philly! Please keep us posted on your progress.

  12. Esme Vos says:

    Dennis,

    Yes, Denise Barton is the marketing director of Tropos.

  13. In message 9, what Denise said is true. I worked for Motorola on the deployment on another city for Earthlink and then went to work at Earthlink providing service support to all the markets. I spent lots of time in New Orleans and even more in Philly in mid 2007.

    I documented many, many moves and changes to radio locations that needed to be done to give better service- especially around Temple University where the gateway (wired node) capacity was severely lacking. I do know that for whatever reason, Earthlink never was able to get the bucket trucks back in to make those changes. They were busy in the North part of town working on deployment.

    I did lots of drive testing with specialized diagnostic gear and I knew where the holes were and what needed to be done.

    In about October ’07, most of us were laid off when EL began pulling out of the business, but I had (and still have) high hopes for Philly.

    Muni wireless is NOT dead. Here it is Octobet 2010 and I still do it for a living (design/deploy/troubleshoot) and know of many systems that are working well. The ones that work well are by Tropos and a small few others. The ones that do not work well are the ones by Cisco.

    The Meraki gear is neat to play with, but is really a toy compared to what real mesh can do. In my experience, the Meraki (and the variants of it) are not suitable for a really controllable network. The dashboard/map is cute, but the fact that Meraki/BATMAN/ROBIN/Open Mesh, etc. present all the clients as looking as if they are behind one gateway does not lend itself to true carrier class control. It makes it difficult if not impossible to control. The system should be able to get IP addresses from an external controller(Mikrotik, Colubris, etc) and let that controller deal with payments, bandwidth control, AAA, etc.

    Yes, I played with WiFiCPA and some of the other “accounting systems” that have been designed for the Meraki type gear, but none of it was carrier class. I would not have run my business using it!

    Anyway- hello to all my old friends and I will go back into my hole now lol.

Trackbacks

  1. […] On MinuWireless broadband specialist Craig Settles offered his take on the recent news that the city purchased the rest of the municipal wireless network. […]

  2. […] – Third Times the Charm Posted on January 6, 2010 by phillyrocketman An article posted on Muniwireless – December 21 2009 Written By: Craig […]

  3. […] Wireless – Third Times the Charm An article posted on Muniwireless – December 21 2009 Written By: Craig […]