Tales from the towers, Chapter 1: ISP’s experience building large Wi-Fi networks

Several people have asked me how to build lower cost municipal Wi-Fi systems that cover anything from a very small area to hundreds of miles. To be honest, we haven’t yet deployed a muni Wi-Fi network that covers hundreds of miles, but in the course of my work as a wireless ISP over the past few years, I have learned many lessons from our small and medium-scale deployments that may be useful to people who are thinking of rolling out large scale muni Wi-fi projects.

As a wireless Internet Service Provider (WISP), our smallest system is 15 clients and the largest system is about 270. I have designed and managed systems that handled up to 3000 users per day and supported about 30,000 users per year through pay per use servers. My designs are based on systems installed and deployed over the last several years that typically didn’t follow the standard cookie-cutter models. Since then I have engineered mesh systems for public safety and municipal utilities for video surveillance/analytics over 100 square miles, Major League spring training baseball facilities, traffic systems, airports, city facilities, point-to-multipoint (PTMP) WISP systems, and point-to-point (PTP) links up to 1Gbps. The proposal I set out below for a city-wide system is based on my experience and the lessons I learned with these systems and concepts.

Basics of a municipal Wi-Fi network

A municipal wireless network is basically a bunch of hot spots connected through some type of wired or wireless backhaul system using a cluster of Access Points (APs). The network eventually funnels all traffic back to a central location which may utilize an authentication system to allow various users on the system. It’s pretty simple stuff with the devil in the details. However, the details are the difference between whether this network becomes financially and technically feasible or not. I am not going to cover that in one article. However, I will show how we developed and deployed different techniques over the last few years to create the foundation for a low cost, scalable muni Wi-Fi network.

Before designing a system, determine the following basic issues:

  • Rules for deliverable product;
  • Potential upgrade path for future growth;
  • Calculate costs/income if applicable.

Simplest case: wireless Internet in a building

Let’s take the first case which is the simple idea of providing Internet in a building. I’m going to spend a little time on it because it is a microcosm of a very large system. Consider it a miniature muni Wi-Fi network in terms of number of users and the foundation for one of our industries. It’s also where we came up with the concept of eliminating fiber by using VDSL converters and using WDS for backhaul for AP to AP hopping. Later, we decided that Wireless Distribution System (WDS) was also preferable to mesh for our outdoor designs due to cost issues, the fact that mesh radios typically only connected to the radios before and after them meaning no change in the mesh structure, and that some of the mesh manufacturers weren’t using load balancing to determine mesh paths, only signal level and modulation.

Inside a building or in a heavily populated area, there are two problems: density and connectivity. The inside of a building is basically a controlled environment. Outside in a populated area, you also have interference, but I will deal with that in a later article. I will also introduce a central server managed design based on these same concepts that will also greatly reduce the cost of in-building Wi-Fi systems.

Lessons learned from a hotel Wi-Fi deployment

The first building we did – the Stratosphere Hotel in Las Vegas – was a pay system. Keep in mind that this was before WDS or mesh was used as a standard for deployments. At the time, we realized that we needed at least 50 access points to cover 2500 rooms on 24 floors with the ability to support about 200 users simultaneously. Due to building codes, all Ethernet cables for each radio would have to be in conduit. In addition, the cable runs from each AP would be well over 300 feet. One hallway alone was 270 feet long. Conduit and cabling costs would run several hundred thousand dollars. The only existing asset we had was a few pairs of telephone wires in a vertical riser conduit that we could use. The vertical riser conduit was already full and there was no way additional cable could go in there.

The vertical asset problem was the easiest. We used Ethernet VDSL converters to push bandwidth from the data room several hundred feet away to a single Access Point to every other floor. These units support 10-15Mbps half-duplex for distances up to a mile across a twisted pair of wires with current versions supporting up to 100Mbps. VDSL also eliminated fiber which would have raised the cost significantly because of the fiber switches. That’s more than enough bandwidth for an 802.11b access point. Basically we traded $250,000-$500,000 worth of conduit work and fiber equipment for about $4000 worth of VDSL converters, a VDSL switch, and WDS, which I will discuss later. We eventually did the World Market Center in Las Vegas about a mile away with many of the same ideas.

Now that we had bandwidth on every other floor to one AP, the second problem was that the hallways were sequential, meaning end-to-end. The hotel is in the shape of an S when seen by planes that fly over. That’s when we came up with the idea of using WDS hops from AP to AP while using the same radios in AP mode simultaneously. Since there were 2 radios in each AP, the WDS path went in one radio and out the other, thus minimizing the 1/n problem. There were 5 radios on every other floor. On each radio, we used custom directional antennas to aim down the hallways for greater range and better room penetration. At the time, the Vivato APs were the only units we could find that could support AP and WDS mode on the same radio simultaneously and were reasonably inexpensive at $350 per radio.

Total cost of equipment for the entire installation was less than $40,000 with labor at about $10,000. It cost $3000-$4000 per month for bandwidth and tech support. We generated over a million dollars worth of revenue over the next 4 years from that system. The reason I use this example is because the techniques we learned became the foundation for almost every future system.

There was a book written called “All I really need to know I learned in Kindergarten” written by Robert Fulghum. We found that the Stratosphere was our Kindergarten. The VDSL units we used later saved another client $100,000 in their initial installation. The roaming spammers gave us the opportunity to test various methods of blocking them and handling high volumes of traffic. Hackers scanning through the system forced us to micromanage connectivity and to be proactive instead of reactive. The system was later upgraded to go from 200 simultaneous users to over 1000 over the next few years by simply adding more radios and VDSL entry points.

The main problem with large scale Wi-Fi systems

This is also where we learned the hard lesson as to what we believe caused many of the problems of future industry municipal systems. Our APs were 200mw. Our antennas were directional and approximately 12dBi. Laptops are 30-100mw.

Because we couldn’t get approval to bring in more access points, there were some areas where clients could see the wireless signal, but couldn’t connect. They weren’t happy. Before I get hammered on why we didn’t have closer access points, the first radios that went in were at the end of the halls for coverage and backhaul. We were going to add additional radios later to cover poor signal areas but a change in management at the Stratosphere held up the second phase of the installation. Therefore, we had to live with hitting laptops at distances up to 135 feet through several rooms for quite a while.

Nobody designs a point-to-point (PTP) system with 2 radios of dissimilar power levels, but that is how many municipal Wi-Fi networks are implemented. Outdoor WiFi AP’s are typically 100mw to 1W. The end result is users thinking that if they see bars on their computer, they should be able to connect. This differential causes huge problems for municipal wireless deployments with APs several hundred feet away trying to pick up laptop signals through brick walls, trees, and other obstacles. The solution is to ask users to purchase high-power indoor units (repeaters) to amplify the signal and reduce the differential. However, that dampened enthusiasm and created additional interference issues. Another option would be a truck roll of an outdoor radio installed by a technician but that idea was never adopted by any of the bigger companies.

Adapting this idea to a municipal wireless model means that early 2.4GHz single radio systems should have been designed with the highest gain aesthetically pleasing antenna connected to APs set to about 100mw. That design would have been more productive, lower cost, and would have reduced the number of connection problems for clients. Ideally, the simplest, least expensive, and best AP would be about 20dBm with a 16dBi omni-antenna. This will deliver the longest range and best penetration bi-directionally with low power clients. Keep in mind I’m not addressing radio specifications, density, interference, or any of the higher tech ideas currently being deployed, just a basic single AP setup. Since we are after a cost effective system, that’s where our focus will be.

Muni wireless systems, like any other wireless network are made up of 3 parts:

  • access points;
  • transport;
  • ingress/egress.

I am going to analyze each one of these for the cost/performance benefits of each phase over the next few articles and mix in some of our other installations to demonstrate real-world applications of these components in action. The goal is to show that it is cost-effective for almost every city to have some type of system in place. Not every city needs a muni wireless network that can do everything and with budgets where they are, it may not be realistic. However, defining the needs and expectations rigorously will help the project stay within the financial scope of most cities.

The overwhelming response to Google’s fiber project shows that hundreds of cities are interested in upgrading their infrastructure. Yes, fiber everywhere would be the ultimate option, but as an intermediate and realistic step (and considering Verizon is backing away from FIOS), not only can the muni wireless system I have outlined above fill that need, it can be built starting at about $3000 per square mile. It can also be deployed as a point-to-multipoint system for less than $300 per AP location and $150 per household for more remote areas while integrating with the muni wireless network later. The ultimate design is an integrated muni wireless/PTMP structure.

What’s really funny is that cell phone companies that clearly can’t get enough bandwidth on a tower to support all the new smart phone applications, haven’t jumped on this concept nationwide to offload some of the bandwidth needs. In some cities, the cable companies are already doing that. In others, the cell phone companies are supporting hot-spot integration. They can do 25 square miles with Gigabytes of capacity for less than it costs to put up 1 tower. That’s a discussion for a future article.

NEXT CHAPTER: Tales from the Towers, Chapter 2: Wireless Access Point Secrets You Didn’t Share with your Parents

* * * * * * *

About the author

Rory Conaway is president and CEO of Triad Wireless, an engineering and design firm in Phoenix. Triad Wireless specializes in unique RF data and network designs for municipalities, public safety and educational campuses. E-mail comments to rconaway at triadwireless.net. Rory writes regularly for MuniWireless.com.

Previous articles by Rory Conaway:

A Primer on Network Security for Municipal and Public Utility Networks (24 March 2010)

New Generation of Low-Cost Municipal Networks (18 March 2010)

About Rory Conaway

Rory Conaway has been in the IT and Wireless Industries for the past 25 years as an author and consultant. He currently operates a growing WISP operation in Southern Arizona. He consults with investors, manufacturers, and WISPs, and develops financial business models for startups. In addition to writing articles in industry publications such as Mission Critical Magazine, Mr. Conaway also writes the series “Tales from the Towers” that can be found on various such as www.triadwireless.net and www.muniwireless.com. He has also engineered several wireless designs such as S.P.I.R.I.T. and Guerilla Wireless as well as building integrated wireless and video surveillance for airport security, municipal and critical infrastructure, SCADA systems, and hotel/MDU deployments.

Comments

  1. Sathees says:

    nice article….very useful…

  2. Oh my … thanks for the Vivato mention.

    And much of what I learned about high capacity, scalable systems that recognize both the reality of interference and the demands of user satisfaction – came from the networks we built at Vivato.

  3. I was right there with you. Unfortunately, I couldn’t get the minions working in your marketing department to realize how good the AP/firmware was on the smaller AP’s. When you finally came out with the outdoor unit and it was $3K, it was overpriced. They told me the radio needed extra cooling. I had several of them in NEMA enclosures on roofs with temperatures hitting at least 140 in the boxes and only lost 1 radio in 4 years. IMHO, If you had hit the $999 price point I was pushing your people with, you would have owned the market. I have 4 of those units still running handling WiFi in Suprise although they are getting upgraded Friday

  4. Dear Rory,

    Thanks for the war stories, and for pointing a way forward. Wi-Fi is a fickle beast, but now it is maturing as a technology. There will be 500 million Wi-Fi enabled smart phones sold in 2014. One for every 14 people on the planet. That’s excluding netbooks, laptops, up from an already impressive 141 million in 2009. Demand for mobile data is growing at 129% through 2015. 802.11N is here. The gear has just gotten so much better too. There is no comparison between what we can do now versus five years ago in terms of coverage, throughput and reliability of signal. Add to that the flood of mobile content, cloud computing, geolocated services, mapping, local search, and then you have a compelling model for municipal Wi-Fi.

    Have that network as a ‘community intranet’ delivering educational, medical, civic, and merchant information, have it interfacing with public safety gear; have it act as backhaul for wireless electrical monitoring sensors.

    The cost justifications for communities to build municipal systems are numerous and growing, as a wireless web integrated into our very environment emerges.

    On the technical front, I found with my experiences building Wi-Fi in 10 NYC parks at my previous company, that the weak link was the low power of the radio in the devices. In many cases — this was 2006 and these were first generation Smart Phones — and the radio was poorly housed.

    The devices could see our amped up APs all right, but couldn’t broadcast back reliably. I only solved that problem with gear from a company called Altai Technologies. I lit up Times Square for Yahoo! with them, and Union Square NYC as well. Coverage and traffic went up ten fold overnight at Union Square when I switched Altai in because their algorithms and antenna technology negotiates the RF environment and retains the connection.

    It’s why I go point to multipoint. I don’t know who else, aside from Altai’s customers (they are in 50 countries, but not so much in the U.S.)is taking this approach. It’s all mesh or some variant such as your architecture, because of that basic power imbalance between devices and access points. What if that was solved with smarter, better antennas. MIMO, N. We have the best approach for voice, video in PTMP. If PTMP is doable, why go with mesh, which is always going to be challenged by high bandwidth/multimedia?

    Mixed topologies will probably be where this goes, with a PTMP core and mesh segments to fill out.

    Looking forward to the next installment!

  5. I addressed the high-power AP issue in this article and the most basic solution. I’ll cover that in more detail in my next article.

  6. The real description of mesh is PTMP to PTMP. It’s just dynamic. Since most mesh AP’s can rarely see past the AP’s coming in and going out, it’s basically static once it’s set up. In addition, the low failure rate of AP’s makes it difficult to justify the cost of mesh over something as simple as WDS.

  7. The power mis match between the access point and possibly a laptop in definitely an important consideration. We have implemented WDS successfully also.

  8. You said something about pay per use, what type of servers and software where you using?

  9. Thanks for give useful information. I want to start my own wireless internet service in my city. How much expensive it. & Is required any legal permission to start this business. One can guide to me for this business? Please Contact me.