Detailed analysis of EarthLink-San Francisco muni WiFi contract

This is a detailed breakdown of the EarthLink – San Francisco proposed muni WiFi contract.

This is the longest post I have ever done on Muniwireless, but I think it may also be one of the most important ones on this site. I have spent the better part of a rainy Amsterdam afternoon and evening reading through 48 pages of dense legalese and writing my comments.

This analysis is very interesting for anyone who cares about citywide Wi-Fi, but especially for municipal officials, city council members, mayors, lawyers, service providers, community activists and prospective customers of EarthLink’s service in SF. You need to know what they will be doing with your personal information. I invite people to post their comments below. Let us know if you’ve found other disturbing provisions or disagree with my interpretation of the contractual terms.

If my analysis is harsh in some parts, it is not meant to be a criticism of EarthLink or the SF folks who negotiated the contract, or their consultants. I know everyone has worked very hard to get this done and every contract is a give and take. The hard parts are yet to come: getting the Board of Supervisors and the Public Utilities Commission (PUC) to approve it, building the network and getting people to use it. I applaud San Francisco for their effort to get free basic wireless Internet access throughout the city. It’s a wonderful start.

Attention cities and counties: please do NOT copy the provisions in this contract and apply them to your situation. I know many of you copied the SF RFP. The terms in this contract are specific to San Francisco’s needs and to its own special brand of politics.

Download the contract from here, print it and follow the analysis below.

We’re not done yet

Nearly every press outlet carried news of the SF-EarthLink contract and from reading them, you’d think it’s a done deal. Not so fast. The contract has to be approved by the Board of Supervisors and the Public Utilities Commission (PUC). The Board has criticized the mayor and the Department of Telecommunications and Information Services (DTIS) for pushing this deal through, and not looking hard enough at alternatives, such as a city-owned network.

Indeed, this may not go through at all. Under section 6.2, if the Board and the PUC do not approve this contract within 180 days after its execution, EarthLink can pull out of its obligations, unless the failure of the Board or PUC to approve it was caused by EarthLink or some event of Force Majeure (defined in 14.3).

The city can terminate the contract without the network even going up if EarthLink does not accomplish “Final Network Acceptance” (defined in 5.1.12) 18 months after “Network Implementation Authorizations” (defined in 6.5). Several things have to happen before EarthLink gets these Network Implementation Authorizations (I am naming just a few below):

  • the city must obtain approvals from PG&E (the electric utility) to enable the PUC to provide electricity to the wireless nodes on light poles (see 6.3.4);
  • EarthLink must enter into an agreement with PG&E for the use of PG&E light poles and electricity (6.5.2).

Why am I picking out these particular clauses? Because electricity utilities have been blocking other cities’ efforts to set up citywide Wi-Fi. This may not be a problem for EarthLink in SF if there are not that many PG&E poles or if PG&E decides to play nice.

Network Services (Section 11)

EarthLink must offer at a minimum five kinds of service:

(1) Premium service: fee-based, term is one month or more, 1 Mbps “minimum average” symmetrical service. This means some will have slower than 1 Mbps service. The good thing is that when you sign up, you go to a capture portal that shows the other service providers on the network, so you don’t have to sign up with EarthLink. You can choose. Verizon and AT&T could even offer service on this network. Very good for open access.

(2) Occasional Use service: for temporary users, less than one month, again the “minimum average” 1 Mbps symmetrical service.

(3) Digital Inclusion Product: 3200 lucky households, to be nominated by the city, can get 1 Mbps (minimum average) access for $12.95 per month. EarthLink will make CPE available to these households but for a price which the city will pay, whichever is less: $100 per unit or the cost of CPE plus fulfillment.

(4) Roaming Service (defined in 1.77): access service with a minimum average symmetric throughput of 1 Mbps which is temporarily extended to subscribers of a service provider.

(5) Basic Service: “minimum average” symmetric throughput of 300 Kbps best efforts only (this isn’t very good at all) for all 16 years of the contract (if it lasts that long). But in all likelihood it won’t stay at 300 Kbps because EarthLink is required to raise the speed every year to the greater of: (a) 300 Kbps or 15% of the “advertised speed” of the “Best Selling Wireless Broadband Product” defined in 1.11 as the Access Service having the largest number of combined wholesale and retail Subscribers for the six-month period for which data are available immediately preceding the date on which the calculation is made, using the same data EarthLink uses to determine the number of Subscribers for its products. So at least the free service will keep up with the service most in demand by users.

EarthLink’s payments to San Francisco (Section 4)

EarthLink is paying the city a right of way (ROW) fee every quarter equal to 5 percent of “Gross Access Revenues”, defined in 1.36 as total revenues received by EarthLink (minus taxes) from providing retail and wholesale access, not just on plain old Internet access but also VOIP services and advertising.

EarthLink will make an upfront non-refundable prepayment of $600,000 (similar to the Minneapolis deal with Minneapolis ISP, US Internet) with a maximum amount to be accrued and applied as a credit against ROW fees owed by EarthLink each quarter — up to $18,750 per quarter until the $600,000 is used up (32 installments — 8 years). $600,000 is 5 percent of $12 million.

EarthLink will pay the $600,000 in three installments: (a) $200,000 within 15 days from the Effective Date, i.e. after the Board and PUC approve the contract; (b) $200,000 within 15 days of Proof of Concept Acceptance (defined in 5.1.6); and (c) $200,000 within 15 days of Final Network Acceptance (defined in 5.1.12).

So EarthLink stands to lose $200,00 if they never get beyond the Proof of Concept Acceptance if, for instance, the city failed to get approvals from PG&E to enable the PUC to provide electricity. EarthLink will have lost $400,000 if after Proof of Concept Acceptance, EarthLink fails to enter into an agreement with PG&E.

Another wrinkle: the $600,000 prepayment can be reduced by $6000 for every $1 that the pole fee is below $99.48 (on the date of PUC approval of the contract), but the prepayment fee can never be reduced below $300,000. The example provided in the contract (4.1.4) says: if the pole fee at the time of PUC approval is $89.48 ($10 below $99.48), EarthLink pays $60,000 less ($10 x $6000) in ROW prepayment fees.

Network Implementation (Section 5): here comes neighborhood politics

The city requires EarthLink to set up a proof of concept: a limited implementation of 2 square miles or more to be agreed upon by the parties, together with the acceptance test criteria (outdoor and indoor coverage up to 40 feet, throughput, availability, reliability).

This is where the politics come in: which neighborhood gets to go first? The city has to make sure it’s in a neighborhood where people will be “breaking” the network, i.e. using it intensively, pushing the network’s bandwidth, reliability and latency limits. Otherwise, the city will be giving EarthLink an easy time.

Suggestions for neighborhoods? Post your comments below.

What about network upgrades? EarthLink is required to update the network with “industry standard technology” whatever that means, but this obligation does not apply during the last 18 months of a Term. Upgrades must be similar to upgrades performed in comparable cities where EarthLink has a Wi-Fi network.

Term of the contract (Section 7)

The contract’s initial term begins on the Effective Date (when the Board and PUC approve it) and ends 4 years after satisfaction of Network Implementation Authorizations (defined in 6.5). The contract renews automatically at the end of the initial term for three 4-year terms (additional 12 years). This is in essence a 16-year contract.

If it is not renewed, the contract continues for an additional 18-month period (Transition Period), presumably to allow EarthLink and the city to find another service provider.

Service Level Agreement ( Section 8 )

It’s nice to see the city negotiate a service level agreement with EarthLink that covers outdoor/indoor coverage, availability, throughput, and latency. But the clause does not have any teeth because section 8 says “failure of the network to satisfy the SLAs shall not be a Termination Default.” EarthLink will have to make reports on the service level criteria, which we hope will be made public.

Where a service level agreement will have any teeth is between EarthLink and a service provider who is going to deliver Internet access using the EarthLink network (one of EarthLink’s wholesale customers). Assume Verizon or Comcast decides to buy wholesale access from EarthLink to bring VOIP and wireless Internet access to their customers. I am certain their agreement with EarthLink will have stricter performance criteria than these, including termination clauses if certain levels of service are not met.

Network Policies (Section 9): net neutrality, limitations on wholesale access and other creepy clauses

I am happy to see that the contract requires EarthLink to open up its network to other service providers. Section 9.1 says EarthLink must offer other service providers wholesale “access” within 30 days after satisfaction of the Network Implementation Authorizations (defined in 6.5).

Exactly what type of wholesale “access service” does EarthLink have to offer to other service providers? Only the type of access that EarthLink itself offers to the public. So if you, a service provider, have more advance products than EarthLink, this contract does not require EarthLink to offer you wholesale access to the San Francisco network.

Moreover, this openness does not apply to Digital Inclusion Products, Occasional Use Products and Basic Service:

9.1.1. Wholesale Access Required. Beginning no later than thirty (30) days after satisfaction of the Network Implementation Authorizations, EarthLink shall offer to any Service Providers wholesale access to any Access Service of the Network that EarthLink offers to the public (except for Digital Inclusion products, Occasional Use products, and Basic Service) on nondiscriminatory terms and conditions.

Digital Inclusion Products refer to the $12.95 per month 1 Mbps service that 3200 low-income households in SF will be getting. Occasional Use products refer to Internet access for less than one month — so hourly or weekly subscriptions. Basic Service refers to the free 300 Kbps service.

So, this means that if T-Mobile wants to offer one-hour or one-day wireless broadband service via EarthLink’s network, the contract does not require EarthLink to provide T-Mobile with wholesale access to offer this type of service. EarthLink could do it, but it is NOT REQUIRED to do so. I am assuming that EarthLink thinks the “occasional use” market is so lucrative that it wants to have it all to itself. Not surprising given that EarthLink is also spending a lot of money to build the network. But we cannot call it a true open wholesale access network.

The contract does not specify what level of wholesale access fees EarthLink can charge. If they charge a lot of money on a wholesale basis, then those service providers cannot offer competitive Internet access or other type of service (VOIP) to end users.

Now for the creepy part. There is a non-discrimination clause in 9.2.1 which starts out very well. But watch out for the terms “subject to”, “lawful”, “legal” and “harm the Network”:

9.2.1. Non-Discrimination Required. EarthLink “will promote the open and interconnected nature of the public Internet and the neutral and non-discriminatory treatment of consumers in the following respects:
9.2.1.1 By allowing users to run applications and use services of their choice, subject to the needs of law enforcement.
9.2.1.2. By allowing users to access the lawful Internet content of their choice.
9.2.1.3. By allowing users to connect their choice of legal devices that do not harm the Network.
9.2.1.4. By fostering competition among broadband providers, application and service providers and content providers.

What does “subject to the needs of law enforcement” mean? If the police tell EarthLink to block you from using a particular application (say, VOIP) or a service (www.relakks.com, a Swedish service that allows you to surf the Internet anonymously), simply because it fits law enforcement’s “needs” as they see fit, EarthLink can simply do so without violating this contract. There is NO requirement for EarthLink to seek a court order requiring the police to PROVE that the police are indeed entitled to have EarthLink block you from using that application or service.

Here’s another one: EarthLink can block you for accessing Internet content if they deem it to be “unlawful”. What exactly is that? And how do you know what’s lawful or unlawful? Do you keep up with federal and state court cases?

Finally, EarthLink can block you from connecting the device of your choice if they think it’s “illegal” and “does not harm the network”. What does harming the network mean? If you upload and download a lot, is that harming the network? Maybe. Then they can just disconnect your home PC or your Wi-Fi enabled gaming device. What is illegal? A device approved for use in Japan which you bought on your shopping trip in Tokyo, but is not approved by the FCC? Technically illegal, isn’t it?

At the end of Section 9, it says the parties shall encourage similarly-situated providers of broadband services to abide by these principles. A bad idea when it comes to the vague clauses I point out above.

Privacy (Section 10): even creepier

This is the section which users – basic, occasional and premium – have to read very carefully. The EarthLink Privacy Policy applies ONLY to fee-paying EarthLink subscribers. For you lucky Fee Subscribers, EarthLink promises not to use your Protected Personal Information (defined in 1.70 as any information which personally identifies the person to whom such information pertains . . . including but not limited to name, address, phone number . . . medical profiles, social security number and credit card information. Section 1.70 goes on about Location Information and Unique Information based on MAC address).

I am quoting the full clause 10.3.1.1 (with my emphasis in bold) because you should read it carefully (and out loud):

10.3.1.1 Sharing of Protected Personal Information. EarthLink will not share Protected Personal Information with any person or entity without the voluntary, affirmative consent of the user, subject to the following exceptions:

a. EarthLink may share Protected Personal Information with EarthLink’s Third Party Suppliers to deliver or promote EarthLink’s services, provided that users may opt out of receiving marketing communications from EarthLink or EarthLink’s Third Party Suppliers using Protected Personal Information obtained from use of any EarthLink Fee Service.

b. EarthLink may share Protected Personal Information with Third Party Suppliers for purposes of processing payments, collections, and order fulfillment and service delivery.

c. EarthLink may share Protected Personal Information with law enforcement in accordance with Section 10.3.1.2.

d. EarthLink may share Protected Personal Information with other persons or entities in connection with civil legal proceedings in accordance with Section 10.3.1.3.

e. EarthLink may share Protected Personal Information with entities that jointly promote EarthLink’s service to their customers, provided that users may opt out of receiving marketing communications from such entities or EarthLink using Protected Personal Information obtained from use of any Fee Service.

This is what it means to me:

(1) EarthLink will not share your Protected Personal Information except with “Third Party Suppliers” (defined in 1.87) but you have to opt out. Who are Third Party Suppliers? The definition is very broad: Vendors or partners that provide products or services to EarthLink or the Subscribers of Fee Services on behalf of EarthLink. These can include anyone who advertises on the EarthLink network or anyone indeed who provides products and services to any Fee Subscriber on the EarthLink network.

(2) The opt out is only for “marketing communications” whatever that means, from EarthLink or Third Party Suppliers. What about opting out of having your information used for purposes other than marketing communications? The contract is silent. It means the third party suppliers and EarthLink can still use your personal info for other purposes, just not to send you marketing pitches.

(3) They can share it with Third Party Suppliers for the purpose of processing payments, collections, order fulfillment and service delivery.

(4) They can share your personal information in connection with civil legal proceedings and with law enforcement agencies.

(5) They can share it with companies that jointly promote the EarthLink service to their customers. But your opt-out is limited to opting out of receiving marketing communications, not opting out of these people using your information for other purposes.

Let’s take number 5. Assume you are already an EarthLink Fee Subscriber on the San Francisco network. Clearly, you don’t need to receive any marketing communications from third parties, say the local supermarket or a credit card company, who are promoting EarthLink’s service in SF. So why would EarthLink share your personal info with the local supermarket and the credit card company? Even if you opt-out of receiving such useless pitches, you’re not safe. Because you’ve opted out only from receiving the marketing pitches. You have not opted out of these EarthLink partners using your info for other purposes. You can’t even tell EarthLink NOT to share your personal info with these parties in the first place!

The clauses that deal with EarthLink providing your personal information for the purposes of criminal investigations and national security (10.3.1.2) and civil proceedings (10.3.1.3.) are troublesome. EarthLink can disclose your personal info, without prior notice to you, to law enforcement if EarthLink believes in “good faith” (whatever that means) that giving away your private into will satisfy the legal process, help them investigate potential violations of terms of service, or protect against imminent harm to the rights, property and safety of EarthLink and its users.

What about the use of Location Information, defined as information that enables a Service Provider or a third party to identify the physical location of a device connected to the Network? You can opt out of EarthLink’s use of Location Information but you have to be an EarthLink Fee Subscriber. No protection for those who get free access (you will have to rely on the firm providing the Basic Service). Even if you opt out, EarthLink can still provide your Location Information to law enforcement or to civil proceedings that involve you (10.3.1.4).

For those of you who opt for the free service, the provider (perhaps Google but they are not mentioned anywhere in the contract) can disclose your personal information to law enforcement and in connection with civil proceedings (similar to those outlined above). With regard to other types of opting out, EarthLink leaves it up to the Basic Service provider.

Future Products (section 11.8)

This is interesting: by the time of the Final Network Acceptance, EarthLink will offer point-to-point fixed wireless 3 Mbps service (but it’s on a best efforts basis, which to me is meaningless). What could this be? WiMAX?

Future Municipal Network Services (section 11.9)

The city will consider EarthLink if it wants wireless service for city departments but there’s nothing definite in the contract. Unlike Minneapolis, which has an agreement with US Internet for municipal use of the network, San Francisco decided not to negotiate this right now. It would have been great to see the city commit to using wireless broadband to enable its employees to work more efficiently, e.g. for public safety, inspection, parks and maintenance, etc. This is a large part of the value of a citywide Wi-Fi network.

Assignment (Section 12)

EarthLink has to get the city’s approval if it wants to assign its obligations under the contract, except if someone buys EarthLink or the division running the SF network. The assignee must agree to assume all of EarthLink’s obligations, in particular the open access requirements.

Termination Default (Section 14.5)

The city can terminate the agreement if EarthLink:

  • fails to accomplish Final Network Acceptance within 18 months after Network Implementation Authorizations;
  • does not pay the right of way fees;
  • abandons Basic Service (the free service) for 15 days consecutively (with some exceptions like Force Majeure).

There are many other provisions in the contract that deserve your attention but I am stopping here.

I welcome your comments especially if you have other interpretations of the contract’s clauses. Feel free to disagree with my reading of the contract.

 

Comments

  1. Thanks, Esme, your analysis, especially of the privacy implications of this, is very helpful!

    The lack of defined service levels at the outset is very troublesome. There’s no requirement that Earthlink provide sevice to every neighborhood in San Francisco, as far as I can tell. This means that they could very well redline the same neighborhoods as AT&T and Comcast already do. And as you point out, any agreement as to service is meaningless, since it’s not a “termination default”.

    It’s also worth emphasizing the slow speed of the Basic Service. Isn’t 300kbits well behind the curve?

    I also don’t think the “Digital Inclusion Product” is cheap enough to make a real difference, unless the city further subsidizes it, which would pretty much wipe out any money Earthlink would pay the city.

  2. [...] Update: See also a more detailed and thoughtful analysis by Muniwireless’s Esme Vos. [...]

  3. Thanks for the good work, Esme.

    I suspect that provisions for City use are not now part of the contract so that the City can continue to say this network comes at no cost to the City. This is a ruse. As you point out, if the City does not use the network, it is a lost opportunity to improve efficiency. If the City built a wireless network for its exclusive use in the future (as NYC is doing now – half a billion dollars for a public safety network that is closed to the public), it is a wasteful redundancy. And if the City does purchase services via the network after the contract is completed, it will negotiate prices from a weak position, lacking any real leverage over a company that has, as Kimo Crossman has pointed out, a de facto pole monopoly for the next 16 years.

    Between that and the substantial privacy concerns, there is enough to justify second thoughts from San Francisco residents and the Board of Supervisors. At minimum, they should delay taking a position on the contract until the pending studies of publicly owned wireless and fiber are complete.

  4. This is a fantastic analysis, Esme, really great job. I was heartened to read that certain issues that I’d been raising with folks (for example, the need to tie basic service speeds to the premium speeds) have been implemented. Along with the numerous concerns you’ve raised (particularly about privacy rights, length of contract, and numerous potential areas for abuse by Earthlink) there’s one issue that really stands out in my mind. And that is, “What is the definition of ‘minimum average’ speed?”

    I found the use of this term odd — especially since I couldn’t find an operational definition anywhere. The reason I bring this up is that I’m deeply concerned that it opens up a loophole you could steer the Titanic through. For example, if “minimum average” is simply the average speed over time — what does it mean for users?

    Here’s a quick example to raise some questions for folks:

    Let’s say the network offers basic users speeds of 1mbps during the hours of, say, 1am and 7am (i.e., when most people are sleeping. This means that 1/4th of every day is averaging 1000kbps (give or take). With a quick equation, it’s easy to see that the rest of the day (i.e., from 7am until 1am) might have seriously reduced speeds while still ensuring that the average speed over the course of the day is still 300kbps.

    If the average hourly rate over a 24-hour period has to be 300kbps and 6 hours are averaging 1000kbps, then the rest of the 18 hours of the day need only average 67kbps to still average out to 300kbps speeds over that 24-hour period.

    As every network administrator knows, there’s a huge lull in network use during about 1am and 7am — so the numbers I’m using are not only possible, but probably quite low — which really demands an answer to the question, “How is ‘minimum average’ speed defined?”

    –Sascha

  5. Esme, as most readers to your site will already know, I have tremendous respect for the important role you have played in the municipal wireless movement. Your blow-by-blow reporting as the market has taken shape over the past few years has been critical, allowing information, lessons and ideas to flow between communities.

    That said, I have to tell you how terribly disappointed I am with your coverage of this important agreement. In fact, I wouldn’t call what you have provided as analysis, but rather a biased, cynical, cherry-picking of terms of the agreement that have been the most controversial, contentious and headline-grabbing. You failed to consider the agreement in any sort of overall context.

    Before I continue, and in the interest of disclosure, I will point out that Civitium served as the advisor to San Francisco throughout the negotiation of the agreement. So, I openly admit my own bias.

    Most of your readers know that I rarely, if ever, speak, blog or write openly about projects that we advise on; I will break that rule to defend an agreement that I believe translates to more community value per resident than any that has been established in the market, including the Philadelphia-EarthLink agreement (which Civitium also served as the advisor to.)

    Let me call attention to just a couple of examples of the bias that is so apparent in your post:

    • You say to “Let us know if you‚Äôve found other disturbing provisions..” Would you also like those who find provisions that are not disturbing, or even encouraging to let you know?

    • You say “..[minimum average] means some will have slower than 1 Mbps service.” Esme, average also means that some will have faster than 1 Mbps service.

    Rather than continue to dissect your post, I will leave things there for the evening. There will be plenty of opportunities for all of us to opine about this agreement in the coming days, weeks and months.

    If I could provide advice to your readers, it would be to look deeper than those headline-grabbing parts of this and other partnerships. It would be to consider the delicate balancing act required to conceive of, and reach agreement on, more than 100 discrete terms of such a complex agreement. It would be to stop defining a partnership as “one or both sides getting all they can get.” And it would be to consider just for a moment what the collective strengths of the public, private and nonprofit sectors could accomplish for communities if they were only aligned.

  6. Regarding Ms. Vargo Dagget’s post.

    First, let me say that your organization’s charter to promote “local self reliance” is something that no-one could disagree is a good thing. I have great respect for ILSR’s enthusiasm for public ownership.

    In fact, the argument that communities should have the right to choose public ownership is something that I have fought and argued for over the past 3 years. But this is where our opinions begin to differ. I support the right of public ownership; you prescribe it as the only way.

    For example, your website at http://www.newrules.org states “ILSR believes that only public ownership of a city’s information infrastructure can guarantee citizens a controlling voice in the design and operation of those systems.”

    The key word you use is “only” and I respectfully disagree, both with the tactic of a Minneapolis-based nonprofit prescribing what is right for all communities, and to the the premise that public-private-partnerships are not a viable option for communities to consider in meeting their needs. I have no affinity for public or private ownership, only for the specific, unique goals of each.

    In your post, you refer to a lack of anchor tenant commitment as “a ruse.” Now, this is just naive and misguided. A city not being required to commit to large anchor tenant commitments in a public-private partnership such as this is in fact a statement of 1) strong negotiating by a city and 2) respect for a level playing field for all providers to compete for city telecommunications business going forward. Had the city committed to millions of dollars in anchor tenancy, I imagine that you would have argued that was “a ruse” as well, the city granting a monopoly to its business, preferential treatment.

    What if a new provider deploys “broadband over tapwater” throughout the city in year 2 of the Wi-Fi agreement, and this service can meet certain municipal needs at a lower cost than Wi-Fi. Would you prefer that the city be locked-in in such a way that the tapwater provider couldn’t compete, or that taxpayers be paying higher prices for services than they should? What you call “a ruse” I would call responsible government.

    And regarding your citing of Mr. Crossman about “a defacto pole monopoly,” I can only say two things. 1) I have learned that pole attachment law is such a complex area that even trained experts have trouble at times interpreting it, and I’m not sure that Mr. Crossman should be held up as an expert here. And 2) with only an estimated 750 city-owned poles being used (a tiny fraction of the number of poles in the right of way) and a clear precedent that attachment rights must be granted to subsequent entrants “on a competitively-neutral and non-discriminatory basis, at terms that are no less favorable or more burdensome than those granted to the initial entrant”, your argument that there is some kind of monopoly is just, well.. rablle rousing.

    I have and will continue to admit my bias. I believe this is a fantastic agreement, and the community, as it should, will decide whether they agree or disagree.

    I accept ILSR’s comments as lobbying instead of expert analysis. And another statement on their website, in my opinion, makes this clear. It says “We’re [ILSR] working with key officials in a half dozen cities to foster publicly-owned information networks.‚Äù Ms. Vargo Dagget, we’re [Civitium] working with key officials in more than 50 cities to help them acheive their goals.

  7. Sacha’s post above highlights one of the best examples for how a complex agreement like this must be viewed in an overall context, rather than deal-term by deal-term. On the one hand, he refers to “the slow speed of the Basic [Free] Service” and then on the other, he expresses concern about “money EarthLink would pay the city.” What he fails to realize is how those two deal-terms (and even others) are inter-related; inseparable. Let me explain.

    I contend that any increase in the speed of a free tier of service into a market will correspond to a decrease in the uptake-rate of paying subscribers for the paid-tier. While we might argue about whether the decrease is linear, how it’s related, etc. I don’t think many people will argue with the basic assumption. Unfortunately, the market knows very little about the elasticity of these products, certainly not on a large scale, so let‚Äôs admit that there‚Äôs lots of risk in experimenting with a model that has so little historical data.

    Now, there is a clear and overwhelming precedent for how much-needed revenues for digital inclusion programs (for computer purchase, education, etc.) are derived in these public-private partnerships. It is through revenue sharing, and the percentage is often in the 5% range. Now, if I increase the speed of the free tier, will I not decrease paying subscriber revenues, and therefore generate less money for digital inclusion? I believe the answer is yes.

    But what about ad revenues you say? Well, based on the number of large-scale muni Wi-Fi implementations, across major urban areas, that rely on an ad-sponsored revenue model, and the limited amount of data about whether they are sustainable long-term, I say those revenues are speculative at best. I hope the free, ad-revenue model is wildly successful in the long-run, but I can’t, in good conscience, recommend policy or business terms for a deal that assumes it.

    But this might change over time you say? Yes, and a short, 4-year initial term is quite powerful to have if you are the recipient of the revenue share, no? Now, I have demonstrated how yet another term of the agreement is inter-related to the first two; the length of the contract.

    There is another important consideration here. Let’s use a toll-road analogy (even though it has some negative connotations in an Internet-context, I admit).

    Isn‚Äôt it the case that, in many communities where there is a large daytime population (typically commuters) and visitors (typically tourists and business travelers,) those “non-residents” rightly contribute to the tax revenues of the city through things like toll roads and probably other ways? In other words, people who don’t live in my city, but access its infrastructure and business-base, compensate my city, and contribute to its residents and businesses.

    Now, let’s consider that, in light of the San Francisco free-tier issue, isn‚Äôt it true that San Francisco has a daytime population that swells far beyond its permanent population? Isn‚Äôt it true that San Francisco is the number one tourist destination in the country? I believe it is.

    Don’t you think that many of those non-residents have both the means and the motivation to pay for a nomadic, occasional-use broadband service while visiting the city? I know I routinely do this, and am quite happy to do so. But, if I had a high-speed free tier of service wouldn‚Äôt I be less likely to make that contribution to the community? And wouldn‚Äôt that decrease revenue shares to the city, and wouldn‚Äôt a digital inclusion funding dollar be lost? I believe it would.

    The point here is that there‚Äôs a delicate balance between providing a basic, free service that lowers the barrier of entry for Internet adoption for digital inclusion, while generating revenues to support the non-access barriers to entry (education, computer ownership), while not artificially destroying revenue-potential that exists in a market — particularly when that revenue potential contributes back in a circular way to the social goal of the program. Whew, that was a mouthful.

    Now, I am not commenting on whether toll roads are good, or whether disincentives (if you want to call them that) for commuters or visitors is good or bad policy. I am no expert in those things. I am only demonstrating the impact they have on a model like this.

    Almost everyone agrees that the goal is how to accrue maximum benefit to the community through a free tier of service.. and that a critical goal of the entire program was to tackle the digital divide.. and that this requires money for things other than building a network. My argument is that pulling this off will require much deeper thinking and analysis than just jumping to a conclusion that “300 Kbps is behind the curve.”

  8. The issue of first mover advantage and access to the most favorable vertical assets of the city is real and I became aware of it in my tour of Tempe’s WAZMetro rollout with their network architect in 2005.

    The first wifi franchise will soak up most of the revenue in the city for their wireless service, raising the financial bar for any competitor. Additionally no one may place additional transmitters on the poles that Earthlink is using that cause their equipment to not work (limited electrical power, spectrum interference or additional weight/wind stresses).

    Mr. Richardson is arguing for balance, but isn’t it really the San Francisco officials who should be explaining the tradeoffs?

    BTW where is the Coverage in the agreement? Remember it was supposed to 95% outdoors, 90% indoors based on the RFI and RFP.

    This website I believe is to gather best practicies – and learn from others: good and bad – It is quite acceptable to point out flaws in this agreement – I can assure you they will come up during the local political process here in San Francisco.

    I think our city can do better than an effectively 16 year agreement with a best effort starting avg speed of only 300k that has privacy concerns.

  9. Bruce Wolfe says:

    Esme, thanks for staying close to the SF concerns that we, as residents and users here have. This is an extremely important step the city is taking on a macro level. But, I would hope that you would support more of the residents of a city whose taxes are going to pay for many things including the salaries of the people they elect to be more self-determining of what services and who services them. As you know, San Francisco has been rife with unmanageable franchises and overarching mega-opolies for decades.

    Plus, why should we settle for arguments of speed at 1 megabits per second when we could be getting well over a minimum of 4 mb/s per user tied into a fiber network with wireless access points at the last mile? We all know technology is only going to become more innovative and exponential very soon that will require far more bandwidth. Why not prepare for the jump to the future with as much bandwidth as possible?

    San Franciscans could use their own fiber network that is already installed.
    Why isn’t this on the table?

    Yes, let’s look deeper than those “headline-grabbing parts” which are really the elements that take away rather than giving to the residents of San Francisco.

    Let’s be real here.
    San Francisco has been fleeced more often than not when it comes to giving away municipal services to private enterprises or outsourcing. Using your favorite search engine, please check this out for yourself. Public-private partnerships are good when the public doesn’t give up its self-determination and control over the partnership. I am not saying there shouldn’t be a free-market to provide such services but there also should be good competition of these essential services and where there is none that the citizens retain control over their destiny.

    Electricity: SF owns its own hydro-electric plant caused by the O’Shaughnessy Dam of the Hetch-Hetchy Valley in the Sierra Mountains. But, because of a surreptitiously agreed to contract with PG&E, it is not allowed to provide that FREE power to its residents but only city facilities. This has been going on for over 90 years despite federal law, the Raker Act of 1913. Ratepayers have been gouged for decades without the ability for the City to break free. PG&E has spent millions, if not, billions of taxpayer dollars to prevent city after city from self-determination in creating public municipally-owned power and districts as recently as this past year in Davis, CA. Also, know that post-Enron, California taxpayers bailed PG&E out of bankruptcy at a cost of $8 billion that was reserved to repair every public school in the state. The taxpayers got nothing back for that favor except increases in rates, fires in the power substations and transformers exploding under people’s feet in high traffic areas.

    Telephone: We all know the monopoly situation of the 60’s-70’s with AT&T and now they are back. With regards to Internet access, the state just passed a law allowing telephone companies to push television and other media programming through their DSL networks using franchise agreements made with the state. CA state legislature has taken away the rights of local municipalities to self-determine what services it wants in their locale without regulation. This also cut out the powerful cable industry that is obliged by regulation under federal rule.

    Cable: SF has been dealing with this debacle for decades, too. The steering wheel has been handed over to company after company without any ability for the city to make its own determination because of franchise agreements. The cable network has changed hands at least three times in the past 20 years without public input. Plus, city officials have been forced to extending their franchise agreements. The city has the right to public input under powerful Sunshine laws but unscrupulous dealing have left the residents of SF out in the cold with high prices for minimum service and horrible customer support as reported.

    Fiber: WE OWN IT! Yes, San Francisco has a large bundle of unused fiber. How much could be used to deliver the entire city with fiber-to-the-premises? From what professionals have said, about 8-10 pairs of strands. From testimony in hearing, that is about 1/20th of the total bundle. We own it, its all installed and with ongoing undergrounding of electrical wires and sewer retrofits, it is not hard to lay fiber wherever the streets get dug up.

    Water: We own it! From the beginning.
    Waste water treatment: We own it! From the beginning.
    Public transportation: We own it! From the beginning.

    Providing Internet access as a city service has got to be the cheapest service and at the highest quality a city can provide next to more difficult services like public transportation and water delivery. What is the resistance? Why should we continue the trend of giving these services away when they serve the public good?

    Solutions:
    Hybrid network: Trying to push all that data across the city over a created “cloud” may cause serious and additional interference with all kinds of existing networks plus interfere with other devices. We won’t know really until it is installed in such a small foot print of a city that is 7×7 miles square. Plus, it is unnecessary. Why not use this city-owned fiber network and build it out using neighborhood wireless access points? Somehow, this seems an obvious solution especially when it already exists. In addition, San Francisco has difficult terrain. There has not been any studies like other cities have done to see of this particular WiFi network will work here. Mountain View is very different in terrain than San Francisco.

    Again, why should we settle for arguments of speed at 1 megabits per second when we could be getting well over a minimum of 4 mb/s tied into a fiber network with wireless access points at the last mile?

    Plus, creating the infrastructure would be fairly painless compared to the other city services, at least here in SF, we have a dearth of great expertise in the field. Many cities are able to get their projects up and running with ease partnering with local technology experts. Take a look at Tim Pozar’s of BARWN & United Layer and Ralf Muehlen’s of SFLAN submissions to create excellent infrastructure, truly free access to the city’s residents and create ample revenue streams to boot to support ongoing maintenance and digital inclusion for everyone. Even Esme felt they were commendable and viable.
    http://muniwireless.com/municipal/bids/862/
    Submitted plans:
    http://www.muniwireless.com/reports/docs/BARWN-SF-RFI.pdf
    http://www.muniwireless.com/reports/docs/sflan_rfi_response.pdf

    What is the resistance to using the fiber anyway? It’s inert. No radiation. It’s passive. Easy to manage. Secure. Very fast. Already available. No one knows as the City’s Department of Telecommunications and Information Services (DTIS) does not reveal any of its future planning not even to the elected Board of Supervisors. They have been left completely out of the loop which means the entire city’s residents are being led into a deep dark pit once again (now for 16 years) without any public input or oversight.

    We also know that there are many reports that Google is buying up dark fiber wherever it can. Plus, where business can be created it will and now there are whole companies whose sole mission is to find and broker that dark fiber. Do a Google-search on that.

    Plus, why is Google so interested? They are not an Internet service provider. They are a search engine and advertising machine and a very good one at that. Why expand into creating provider networks except for the same purpose as the one in Mountain View, CA, which was set up for market research where they installed a WiFi network to provide access so they could achieve that purpose? In fact, they had to partner with Earthlink in order to gain access into these markets.

    With the Google-Earthlink plan here in San Francisco, we are dealing now with (yet another) utility coming into existence and wishing to install 15 year old technology when all this extremely fast fiber lays dark and dormant, owned by the residents of SF but very far from its reach to use it. When I attended Esme’s Muni-Wireless Conference in Santa Clara last year, I was approached by several Google employees, technologists, that truly understand the nature of what Google wants to do and how to do it.

    The question to me was, what are your thoughts on using Wi-Fi to deliver broadband access to the residents of SF? At first, I thought I should be coy because my position was very different from most of the fee-based services that were attending there. I decided not to hold back as the part of being self-determining is to be outspoken as equally important.

    As an amateur radio operator and WiFi enthusiast, I replied that this topology of using radio to push data over a free spectrum at such low power was truly backwards thinking. That it was more like trying to use a baby monitor down the block from your house because without a HAM radio license that is all that is available at reasonable cost. If we were so forward-thinking to install so much fiber, why not use a fraction of it for the residents? I remember in the mid-80’s when packet switching was just getting underway and the first wireless chat rooms emerged over the HAM radio networks. Just seeing a ping from the other side of the country was very exciting.

    The Google employees agreed with me. They felt that this Google-Earthlink for SF plan was a fait accompli that it would be obsolete within a year and that it would not provide access except to those who had line-of-sight to any access point mounted on a street lamp pole (btw, for difficult access, DTIS says that you could buy for ~$200 external equipment and must subscribe to the $20/month pay service. That is not free, affordable and accessible WiFi). They went on to say that WiFi was never meant to be used outside of a closed room and using it as a city-wide solution when there is available fiber already to use was fairly stupid.

    Well, I was floored at their agreement and confirmation. In fact, their support to keep up the opposition to this submitted plan. There is something to say about how and why city governments make their decisions when it comes to providing services. There is a trend and understanding that not the best or even good services are decided upon thus leaving many municipalities with “white elephants.” That was the word the Google employees used to describe this WiFi plan for SF.

    I would urge that readers seriously read Tim Pozar and Ralf Muehlen’s entries during the RFI phase of this project. Their plans are solid. There are similar plans that other cities are doing. Also, inquire with the Mayor and Board of Supervisors as to why we are not using our city-owned fiber network to provide true high-speed access and digital inclusion.

  10. Bruce Wolfe says:

    Regarding Mr. Richardson’s view of the pole issue:
    ” And 2) with only an estimated 750 city-owned poles being used…”

    Actually, if you have been following it, Earthlink asked for 750 but then Google wants to piggy-back 1000 more.

    Ultimately, this all can be easily solved if the city just adopts a municipally-owned model using its own fiber network and then contracts with others to provide last mile services. It doesn’t even need to be with just one company. This way the city as it continues its undergrounding in outlying neighborhoods can replace the WiFi with direct fiber-to-the-premises.

    WiFi will not be with us in its current form forever. WiMax or pre-N (802.11n) hasn’t even been approved by the FCC yet. By the time it is, there will be another innovation and still yet another. This is just the beginning and to lock the residents into something that will change in six months and then six months again is ludicrous.

    To think that this system will serve the city over the next 16 years in its current form without a reasonable sense of update or complete upgrade is really being naive.

  11. [...] It appears that the debate over municipal wireless in San Francisco is far from over, despite the fact that the city negotiated a contract to operate a private-public partnership with EarthLink less than a week ago. [...]

  12. Frank Robbinson says:

    I agree with Bruce on his desire to build out fiber in SF. But you are talking apples and oranges. Muni owned fiber could make total sense. No new wired technology is going to top fiber in the near future.
    You have a long time to recover your investment.

    In some Cities, fiber was built out with the help of the Cable companies money. In Denver the City is stuck with a non-compete clause where they are not allowed to offer ISP services to consumers that could compete with the Cable companies product. Could be the reason why in SF they can’t resell it to EarthLink.

    If the fiber is available for EarthLink/Google and is lit then I would imagine they will be negotiating to have access to it. They need backhaul…Rates on fiber are fairly standard. If the fiber is dark then
    it’s probably too expensive to use for a Muni-WiFi build.

    Unfortunately, Bruce, fiber is not mobile. I can’t plug into fiber using my mobile device on the go. No one is proposing Muni-WiFi is going to compete with fiber on speeds.
    Muni-WiFi fills a short term gap for fixed access, until the City or a private company offers an alternative product cheaper. Approx. 30 % of residents of SF do not use broadband today. Some for financial reasons, others because of choice. Muni-WiFi may be good enough for them. One account you can use mobile and fixed all around town.

    On the wireless front, most Municipal projects business cases are based on 10+yrs usefulness of the product; roads, water projects, bridges etc. When it comes to new technology like Muni-Wireless and competition from 3G, WiMax shortly, LTE, then I recommend that large Cities stay well out of the business. Corpus Christi recognized this one. Others will follow. It’s not that they can’t do it, it’s just that they probably have more pressing issues to deal with than maintaining a large network.

    Smaller cities like St.Cloud, FL can be successful as they will not have the volume issues a SF will have. It’s a long story…but equipment to handle 20,000 customers is much cheaper, as the market for these devices is a lot larger than the big stuff. Since less people buy the bigger stuff…less demand if you will….it’s expensive, very expensive.

    Can you imagine the arguments of when to upgrade the network in SF City Hall, if the City owned it? By the time they get around to agreeing to invest in an upgrade, many users will have churned off.

    I doubt the pro-Muni owned SF Muni-WiFi contingent fully understand what it costs to maintain a large subscriber based system. You have to market it, even if it is free, you have tons of back office, core network equipment, that needs to be maintained and upgraded more often than the radios on the street, and you need a call center to be able to handle large volumes of calls. A Muni the size of SF would have to outsource the maintenance, which gets you right back into the whole ‚Äúwho is in control” issue. There are many many horror stories of Muni’s having to pay exorbitant fees for simple change controls with outsourced service providers. Contrary to many lawyers thinking that they can predict all potential changes and can lock it down in legalese, you can‚Äôt.

    Call centers. As much as the integrators will tell you they can support large consumer networks (millions of users) they generally don’t. They support large Enterprise networks, not consumer ones. Big difference! Since call centers are a very political issue here in the US, the cost of a US based call center to support a partially free based consumer broadband access system is financially and politically suicide.

    People also forget that you will have to advertise this network. The free gig will only get you play for a little while. Then you will have to pay some serious advertising dollars to drive up subscription and offer specials to reduce churn. Keep in mind WiMax, 3G, and others are going to compete and challenge the competitive advantage of Muni-WiFi. You will have to defend it. Can you imagine City Hall funding a $3 – $5million dollar advertising campaign for a partially free network?

    As far as the speeds go…..you seem to have forgotten about competition, or at least don’t trust it. Ultimately in the Mobile world, real time Video experience is the killer app and most likely the unique advantage that Muni-WiFi has over 3G and WiMax. The Muni-WiFi operators out there will be exploiting this aspect…and Video requires speed, hence your “minimum speeds” will be ok. Last I checked Google is still influential, they will make darn sure that the speeds increase appropriately over time to deliver their cool apps.

    Privacy. A lot of talk on this one. All I can say is chill out. EarthLink has a long history of defending privacy, even taking on the NSA years back. Other ISP’s will join the network, if you don’t like EarthLink’s policies choose another ISP, it’s an open network. However, if you want to take advantage of location based services then you will have to give in on privacy for it to be useful.

    Esme & Sascha let’s get real here, opt out clauses are just paper. Rather than looking for legal ways to protect your privacy, get smarter. You are already doing it. The “system” only knows what you put in to it. Like in life you have your real name and then your “bar name” or internet profile. Start using your “bar name”. If you are doing something illegal…stop it. If you are paranoid about your information…stop giving out so much.

    As Greg says we don’t know all the details. This looks like a really good deal for SF. 2 major companies and a critically important City putting it on the line to build a world class network free for its citizens. No other city in the world has this deal.

    It gets the city what it wants without being tied up in tons of “anchor tenancy” issues. De-railing this project now will set back SF 2yrs+ on Muni-WiFi.
    That would be a crying shame and a reason to move!

  13. [...] The report finds a bunch of problems with the Earthlink contract, including slow speeds, expensive “digital inclusion products”, reliance on one vendor (Google, presumably) for the free service, that Earthlink would essentially saturate the entire series of bands at which wi-fi runs, and the privacy problems so thoroughly addressed by Esme Vos. It doesn’t, however, mention the lack of guaranteed coverage, which I think is a serious omission. This report doesn’t suggest we ought to burn our free Earthlink t-shirts in the street or anything, but it does suggest the city (and more importantly, its residents) is being short-changed by the narrowly-visioned, rushed process that got us to where we are. [...]

  14. The reason Google wants access to every municipality is for “Attention Data.” Attention data is the most valuable thing on the internet. It’s the digital gestures that represent everything you do, everywhere you surf online, every email you send, every IM, every interest you show in anything. You don’t need it to market to people directly. You need it to create the marketplace that is AdSense. It’s the basis for the market pricing of all the keywords Google sells.

    Right now, Google has only your search data, maybe email or IM (gmail), maybe your image or video interests, but it doesn’t know everything you do.

    Google can cement their ownership of the AdSense marketplace by owning data about everything you do.

    In the Web 1.0 world, Microsoft owned your machine. That was the unnetworked world, where owning everything you did on the desktop was most important. Eventually, after 15 years of this, they were sued for monopolistic practices under antitrust law in the US and Europe.

    In the Web 2.0 world, Google owns your Attention Data. It’s the most important thing in the networked, online world.

    By partnering with all the wifi providers, Google gets access to a much richer cache of Attention Data, than they have with just search and your visits to their websites and services. Now, they have everything.

    How long, as Google spreads through the ecosystem, before people realize this (5 or 10 years?) and then, how long ’til they are sued under antitrust law? And then, how long until the world changes again, and Attention Data isn’t most important, but something else is, and the antitrust suit is happening, but it doesn’t matter, because Google has made their money over 20 years, left SF and other cities in the dust, and is slowly imploding as Microsoft is now?

    We are just at the beginning, and SF is giving away what, $500 million in Attention Data? Maybe billions of $$$$ in Attention Data, for almost free in comparison to it’s value, over the next 16 years.

    Besides the direct users of the service, who will likely have mediocre service at some parts of the day, or whatever, though Google will have incentive to make the networks faster.. so my guess is the service won’t be that bad. The real travesty here is that SF, and all the other municipalities, don’t even know what they are really delivering for essentially “free” is data they should have control over, and rights to sell, aggregated with individual users personal information attached, in the AdSense marketplace, for it’s true value.

    anna

  15. Bruce Wolfe says:

    With regards to Frank Robinson’s response, I agree with him totally. The concern we have is to make sure that the fiber network is under city governance and maintained ownership, that it is considered as an element of the vision in the future of the city and that it could be used in some capacity now.

    Using the fiber net now as the backhaul is correct but the Google/Earthlink plan doesn’t even ask for it which leads many to believe there is some ulterior motive. 60% of it has been sitting there dark for three years.

    Yes, the most obvious plan is for a hybrid system that is what many tech professionals have been asking for since the public hearings started. There have been 12 since. The same message goes out for using the fiber net as the backhaul and expand using WiFi now and as the city does its sewer renovations and above-wire undergrounding to start FTTP installations (the hardware will be much cheaper by then) with additional support for wireless as needed.

    But, this Google/Earthlink plan doesn’t even look to the future. It is simply a quick install of an access point over the entire city that is rife with terrain and building issues that are not taken into account.

    If the rest of the city were allowed some involvement from the beginning instead of this being behind closed doors, we would definitely have a different outcome.

    There are many critics of the so-called critics abound. The so-called critics are not that. They are the tech professionals of this city that are in abundance unlike other major capital projects like fresh water, waste water, electricity, etc. There is plenty of expertise to go around. This is not rocket-science especially when there is significant wireless transmission existing already.

    I am not being naive about this. Get a hybrid system going. There will be people who already can afford the hardware for fiber installs. Let them pay for it out of their pockets but make fiber access available. For those who can’t afford it now, build out a WiFi network on top of the fiber net but spread the wealth and allow competition among many WiFi vendors that could or would provide WiFi access plugged into our backhaul. Get the variety which offers plenty of test cases; contracts would be short-term so customizing the network as more FTTP moves in and wireless hardware becomes obsolete is best. Let the WiFi companies deal with that obsolescence. Maintain taxpayer-owned assets, the fiber net.

    This is a very fluid situation and time in wireless technology now. We have seen companies like Buffalo Tech get sued by the Australian government over 802.11a & g on patent infringement. Intel owns 802.11n.
    http://www.techdirt.com/articles/20061116/001301.shtml
    http://news.zdnet.com/2100-9595_22-6137372.html

    This is no time to secure ownership of white elephants.

  16. [...] Those of you who are closely following this debate will be interested in Esme’s detailed analysis of the Earthlink contract (click here to read it; it includes a link to the contract) and in the San Francisco budget analyst’s feasibility for a municipally-owned network (click here¬†for my report which includes a link to the study). [...]

  17. Concerned Citizen says:

    And better than that, EarthLink and discriminate against its employees and violate their rights and the city of San Francisco will continue to do business with them. See Brundige vs. EarthLink and Coker vs. EarthLink.