Loading...
HomeMy WebLinkAboutAGENDA REPORT 2022 0316 CCSA REG ITEM 09CCITY OF MOORPARK, 
CALIFORNIA City Council Meeting of March 16, 2022 ACTION APPROVED STAFF RECOMMENDATION. (ROLL CALL VOTE: UNANIMOUS). BY A. Hurtado. C.Consider Meet-Me Room Market Study. Staff Recommendation: 1) Receive and file the Meet-Me Room Market Study; and 2) Direct staff to continue supporting the County of Ventura and the State of California efforts to construct fiber-optic networks through Moorpark, and direct staff to continue monitoring and pursuing funding opportunities that may be used toward a Meet-Me Room and data center. (Staff: Brian Chong, Assistant to the City Manager) Item: 9.C. MOORPARK CITY COUNCIL AGENDA REPORT TO: Honorable City Council FROM: Brian Chong, Assistant to the City Manager DATE: 03/16/2022 Regular Meeting SUBJECT: Consider Meet-Me Room Market Study SUMMARY In October 2020, the City Council adopted the City’s Broadband Strategic Plan, which recommended actions to accelerate the deployment of broadband infrastructure and adoption throughout the community. One of the recommendations was to commission a market study to determine the viability of a potential Meet-Me Room to be constructed as part of the Moorpark City Library project. In March 2021, the City retained Magellan Advisors, LLC (Magellan) to prepare the market study, which is now complete. It is recommended that the City Council receive and file the market study and, as recommended in the market study, further pursue a Meet-Me Room at such time as either the County of Ventura or the State of California funds their regional broadband networks that are currently proposed to be constructed through Moorpark within State Route 23 (Moorpark Avenue). BACKGROUND On October 21, 2020, the City Council adopted the City’s Broadband Strategic Plan, which contained numerous analyses and actions by which the City could advance the deployment of broadband infrastructure throughout the Moorpark community. One of the Plan’s recommended actions was to determine the viability of a Meet-Me Room to potentially be built as part of the new Moorpark City Library project. The City Council appropriated $18,900 for the market study on February 17, 2021, and approved an Agreement with Magellan to complete the Market Study. Magellan previously prepared the City’s Broadband Strategic Plan, as well as similar plans for the Cities of Oxnard and Ventura, and is currently also providing similar services to the County of Ventura and numerous other clients across the country. Item: 9.C. 35 Honorable City Council 03/16/2022 Regular Meeting Page 2 A Meet-Me Room houses servers that act as a hub for telecommunications service providers, from which expanded services across the community and region could commence. Users of Meet-Me Rooms would rent space within it, which would represent a revenue stream for the City that may completely offset the costs of constructing and operating it. The location of the new Moorpark City Library building on High Street was identified as a strong location candidate for a Meet-Me Room because of its close proximity to numerous long-haul fiber-optic lines located within the railroad right-of-way and central location within the developed portions of Ventura County. To inform the market study, Magellan prepared, and the City issued, a Request for Information (RFI) designed to assess interest in a Meet-Me Room in Moorpark. The RFI yielded four responses (from Ridge Communications, Smart Fiber Networks, Onward, and AT&T). In addition, Magellan proactively reached out to numerous other telecommunications and broadband providers and infrastructure owners to assess their interest. Ultimately, the market study was informed by discussion with a dozen different industry respondents. The market study also incorporates what is currently known about major federal broadband investment (as part of the American Rescue Plan Act) and major state broadband investment (approved as part of the state budget in mid-2021). However, the flows of these funds and investments are currently still being finalized by the federal and state governments at this time, and the City will need to track the resultant federal and state programs going forward. DISCUSSION The market study affirms that there is interest in utilizing a Meet-Me Room in Moorpark, especially when combined with a data center facility, but recommends that the City not pursue constructing a Meet-Me Room until interest reaches a critical mass when either the County of Ventura or State of California funds their regional broadband networks that are currently proposed to be constructed through Moorpark within State Route 23 (Moorpark Avenue). A summary of the market study’s conclusions are as follows: 1.A Meet-Me Room in Moorpark would significantly advance the City’s broadband- related goals. A Meet-Me Room in Moorpark would: increase the access, affordability, and reliability of broadband services in Moorpark; facilitate deployment of wireless technologies; support Smart City applications; produce cost savings for telecommunications services for City operations; and promote economic growth by attracting and retaining key industries and by creating jobs. 36 Honorable City Council 03/16/2022 Regular Meeting Page 3 2.There is significant interest in a Meet-Me Room, especially when paired with a Tier 3 Data Center. To improve the viability of a Meet-Me Room, the study recommends that the facility include a Tier 3 data center. Data centers house servers, storage, and networking systems that serve as off-site backups for users, such as large employers. Because they have similar physical requirements as Meet-Me Rooms, there is significant synergy between collocating them. The study recommends a Tier 3 (of 4 tiers) facility based on the current and future businesses located in Moorpark and Ventura County. Higher tiers have less downtime, additional power outage resilience, and higher costs. 3.A combined Meet-Me Room and data center facility should be approximately 5,000 square feet in size, ideally expandable to 10,000 square feet, and detached from other buildings. To maximize the viability of a facility, the study recommends 5,000 square feet of floor area that is ideally expandable to 10,000 square feet. 4. The estimated cost to construct a combined 5,000 square foot Meet-Me Room is $5,000,000. This estimate is based on an estimated $1,000 per square foot to construct a Meet-Me Room. This cost does not include the cost of the land on which a Meet-Me Room would be constructed. 5.Public-Private Partnership (PPP) opportunities exist for the marketing and day-to- day operation of a Meet-Me Room. The study confirms that there are private entities able and willing to partner with the City to operate a Meet-Me Room on a day-to-day basis. This includes marketing and sales of space inside the Meet- Me Room/data center, building access, and 24/7 downtime/threat response (such as if the temperature gets too hot inside). In such a PPP relationship, revenues generated by the Meet-Me Room would be shared between the City and its private partner, at negotiated rates. A PPP approach also decreases some of the risk of constructing a Meet-Me Room because the City would have reduced operating expenses to keep it running. 6.A Meet-Me Room is likely not viable until either the County of Ventura and/or State of California commits to constructing their broadband networks through Moorpark. The value of a Meet-Me Room increases as more providers install infrastructure inside and can thereby interconnect their networks. If an AT&T cell phone user calls a Spectrum land line, for example, the interconnection between the AT&T and Spectrum networks need to occur somewhere. A Meet-Me Room makes this interconnecting simple and economical. While there is some current demand to occupy a Meet-Me Room in Moorpark, it would be unlikely to generate sufficient revenue to offset the costs to construct and operate it. However, the market study states that critical mass to make the Meet-Me Room less risky would occur when either the County of Ventura or State of California commits to constructing their own regional broadband networks. If those 37 Honorable City Council 03/16/2022 Regular Meeting Page 4 networks are constructed through Moorpark, there will be many telecommunications network operators seeking to locate inside Meet-Me Rooms along those network paths. Moorpark’s central location in Ventura County makes it especially desirable for those network operators to utilize a Meet-Me Room in Moorpark. The original concept of a Meet-Me Room in Moorpark did not incorporate a data center and its additional square footage and construction cost. The recommended 5,000 square-foot size of a Moorpark Meet-Me Room renders it physically inappropriate to be constructed as part of the Moorpark City Library project; the site is too small. Just as the square footage increased severalfold, so too does the construction cost – now estimated at $5,000,000 – which would in turn be offset by higher revenue potential. Scaling up the project also increases the risk in constructing it, since more initial capital investment is required. Based on the above considerations, the market study recommends that the City not immediately and actively pursue construction of a Meet-Me Room. However, the factors that currently drive this recommendation are in rapid flux and could change dramatically in the coming months, such that a recommendation to actively pursue design, construction, and tenanting of a Meet-Me Room would be appropriate. First, both the state and federal governments have committed to investing tens of billions of dollars into broadband infrastructure through multiple existing and new programs. Staff is aware of 18 pools of broadband funding, some of which have published eligibility requirements and some of which have not yet published eligibility requirements. Of those programs that have published eligibility requirements, the City is not eligible for most (such as grants to rural communities, unserved communities, or tribal communities). However, tens of billions of dollars still remain on the table with not-yet-published eligibility requirements, and grant funding and/or favorable lending programs remain a distinct possibility. If the City is able to secure part or all of the funding for a Meet-Me Room through such financing programs, the cost to the City (and the associated risk) is greatly diminished. Second, construction of a regional broadband network through Moorpark by either the County or the State would greatly increase the desirability of a Meet-Me Room in Moorpark and therefore demand for space inside it. That translates to higher revenues generated by the Meet-Me Room and also decreases the risk to the City. As part of its multi-billion dollar investment in broadband, the State of California has announced its intent to construct a fiber-optic network across the State, largely within state highway right-of-way that the State already owns. The first draft of the State’s network includes a fiber-optic cable within State Route 23 between Fillmore and Moorpark. If the fiber-topic cable is actually constructed within the SR 23/Moorpark Avenue right-of-way, the demand and revenues for the Meet-Me Room increase greatly. It is expected that numerous network operators would take advantage of the State’s 38 Honorable City Council 03/16/2022 Regular Meeting Page 5 entire network and lease fiber-optic capacity from the State. These network operators will need to establish Meet-Me Rooms and data centers somewhere along the network, and a Moorpark Meet-Me Room located adjacent to State network and in the geographic center of Ventura County’s population would be highly desirable. In September 2021, the City sent a letter to the California Public Utilities Commission advising that the City is considering construction of a Meet-Me Room in Moorpark in an effort to leverage the network for our mutual benefit. Meanwhile, the County of Ventura has also developed a plan to construct a “Middle- Mile” network that would interconnect all Ventura County communities. The network is denser than the State’s network and includes redundant circular routes within Ventura County to improve reliability and resiliency. The County network also includes fiber- optic cables installed within Moorpark inside SR 23 (Moorpark Avenue, toward Fillmore), SR 118 (Los Angeles Avenue, toward Somis and toward Simi Valley) and Tierra Rejada Road/Moorpark Road (to Thousand Oaks). Like the State network, it is anticipated that numerous network operators would take advantage of the County network and similarly increase demand for a Moorpark Meet-Me Room and data center. The County of Ventura is actively seeking grant funding to begin constructing its network, and the City has been supporting the County effort as established by the City Council in its current set of Goals and Objectives. If the County obtains funding and is committed to constructing the portions of its network through Moorpark, then the market study recommends that the City pursue construction of the Meet-Me Room and data center. FISCAL IMPACT None. COUNCIL GOAL COMPLIANCE This action is consistent with City Council Goal 3, Objective 3 (3.3): “Support regional and local broadband deployment issues.” STAFF RECOMMENDATION 1.Receive and file the Meet-Me Room Market Study; and 2.Direct staff to continue supporting the County of Ventura and the State of California efforts to construct fiber-optic networks through Moorpark, and direct staff to continue monitoring and pursuing funding opportunities that may be used toward a Meet-Me Room and data center. Attachment: Meet-Me Room Market Study 39 C I T Y O F M O O R P A R K , C A M M R F I N D I N G S W W W .M A G E L L A N - A D V I S O R S . C O M 1 City of Moorpark, California Meet-Me Room Study March 2022 ATTACHMENT 40 C I T Y O F M O O R P A R K , C A M M R F I N D I N G S W W W .M A G E L L A N - A D V I S O R S . C O M 2 Moorpark Meet-Me Room Study Contents 1. Introduction ............................................................................................................................ 3 City of Moorpark Broadband Strategic Plan ............................................................................ 3 Ventura County Broadband Network ...................................................................................... 5 California Open Access Middle Mile Network ......................................................................... 6 MMR Recommendation ............................................................................................................. 6 What is an MMR? ........................................................................................................................ 7 Power .......................................................................................................................................... 7 Customers of The Moorpark MMR ........................................................................................... 8 2. RFI Response Summary ......................................................................................................... 9 Service Provider Outreach Summary ..................................................................................... 10 3. Early Research Conclusions ................................................................................................. 12 What are Data Centers? ........................................................................................................... 12 Colocation Data Centers.......................................................................................................... 13 Co-location Data Center Customers ....................................................................................... 15 Public-Private Partnership Opportunities .............................................................................. 17 Benefits of having a DC/MMR in Moorpark ........................................................................... 18 Other Data Centers Near Moorpark....................................................................................... 22 Financials .................................................................................................................................. 23 4. Conclusions & Next Steps .................................................................................................... 23 Next Steps ................................................................................................................................ 23 41 C I T Y O F M O O R P A R K , C A M M R F I N D I N G S W W W .M A G E L L A N - A D V I S O R S . C O M 3 1. Introduction Historically, the City of Moorpark has been a hub of Ventura County through two periods of industry growth. The first was during the railroad era when tracks were laid coast-to-coast and the second was the automotive era with the interstate highway system. Today, we are in the third wave, the digital wave. Here again, given the geographical location of Moorpark, the City can be the digital hub of Ventura County. Industries of all types rely on the cloud, the internet and vast gigabit connectivity to survive and thrive in today’s global economy. The amount of data enterprises are transmitting and receiving continues to increase exponentially. Recent trends also include a hybrid or remote work environment where employees and customers are accessing data from home or from the coffee shop. It’s not surprising that businesses and consumers have made the availability and affordability of gigabit broadband a key driver for decisions about where companies locate their headquarters and remote facilities and where people choose to live. Given these trends, as well as regional and state broadband projects occurring in light of the shift to online work and commerce, the City of Moorpark can leverage its central location to attract investment and bring in additional broadband options by building a digital interconnection point known as a meet-me-room (MMR) in the City. Investment in such a facility would bolster the broadband environment in Moorpark, meeting many of the goals and objectives of the City’s Broadband Strategic Plan. Findings indicate that demand for such a facility will largely depend on the success of other regional projects, including the Ventura County broadband network and the State of California’s middle-mile broadband network. Therefore, Moorpark’s investment in a MMR should be contingent upon these plans moving forward with connections that will position Moorpark as a critical location for broadband interconnections. CITY OF MOORPARK BROADBAND STRATEGIC PLAN Magellan Advisors has worked with both the City of Moorpark and with Ventura County on fiber and broadband infrastructure projects. Moorpark and Magellan Advisors began working together in December of 2019 to develop a comprehensive Broadband Strategic Plan. The original goals of the project were: a. Increase Access, Affordability, and Reliability of Broadband Services in Moorpark b. Facilitate the Deployment of Wireless Technologies Citywide c. Streamline Processes and Workflows and Remove Silos for Coordinated Broadband Efforts d. Develop Strategic Partnerships with Private and Public Sector Entities 42 C I T Y O F M O O R P A R K , C A M M R F I N D I N G S W W W .M A G E L L A N - A D V I S O R S . C O M 4 e. Support Smart City Applications f. Produce Cost Savings for Telecommunications Services for Internal City Operations g. Promote Economic Development h. Understand the Needs of Businesses i. Attract and Retain Key Industries j. Encourage Economic Growth k. Become a Gigabit City l. Create a Tech Ecosystem The final Broadband Strategic Plan developed by Magellan Advisors for the City of Moorpark was provided to the City in October of 2020. The Master Plan made the following recommendations.: 1. Incorporate broadband and technology into all planning considerations, especially the City’s 2050 General Plan update. 2. Implement policies related to the deployment of small cell telecommunications facilities including an MLA (Master License Agreement), Small Cell Design and Deployment Standards, an updated telecommunications ordinance, and a Dig Once ordinance and practice. 3. Include broadband in development agreements with considerations for impact fees and mitigations for the placement of conduit. 4. Set aside funds of $100-150k to install conduit and/or fiber infrastructure during CIP projects. 5. Incrementally and opportunistically build connections for a Citywide fiber network. 6. Engage with potential partners about agreements to use City assets for increasing service offerings. 7. Track and participate in the Ventura County broadband project and other local initiatives to promote a regional broadband vision and increase interest in the region. 8. Provide space for a meet-me room in the design plans for the new Civic Center. Moorpark should consider releasing an RFI or RFP to determine the feasibility of building an MMR. 9. Continue tracking grant opportunities for funding expansion of broadband assets including EDA, RDOF, and CASF 10. Should the City desire to further understand the cost and revenue potential of the network identified in this Plan, it should contract for a full financial analysis based on further outreach to the business community. 43 C I T Y O F M O O R P A R K , C A M M R F I N D I N G S W W W .M A G E L L A N - A D V I S O R S . C O M 5 VENTURA COUNTY BROADBAND NETWORK Magellan Advisors began working with Ventura County in May 2020, when the County’s administration began to plan for a county-wide open access middle-mile fiber network that would bring increased connectivity to the entire region. In addition to connecting key county facilities, public safety agencies, and other anchor institutions such as city halls, healthcare facilities and schools, the Ventura County network would also interconnect municipal networks and facilitate last-mile fiber deployments by competitive service providers. The middle-mile network owned by the County would provide bandwidth at wholesale pricing to improve broadband services to all users in the County. Cities, such as Moorpark, that participate in the project will be able to use the County’s network to connect back to internet points-of-presence (POP), which could save the City the cost of having to lease these connections from a private telecommunications carrier. It should be noted that there will be a limited number of locations where cities and service providers will be able to connect to the county network. The City of Moorpark has an advantage of being centrally located in the County and can leverage this to become a hub for the network. The proposed Ventura County network developed by Magellan Advisors is shown below. Figure 1. Proposed Ventura County Network 44 C I T Y O F M O O R P A R K , C A M M R F I N D I N G S W W W .M A G E L L A N - A D V I S O R S . C O M 6 CALIFORNIA OPEN ACCESS MIDDLE MILE NETWORK The State of California announced a $6 billion broadband infrastructure plan on July 12, 2021. The focus of the funding is un-served and under-served communities. It includes $3.25 billion to build, operate and maintain an open-access state-owned middle mile network. The middle mile is critical as it connects each city’s central office, or fiber aggregation point, to the big city inter-connect facilities. Figure 2. Proposed California Open Access Middle Mile Network – Ventura County MMR RECOMMENDATION The recommendation for Moorpark to explore a Meet-me Room (MMR) was based on the premise that telecommunication service providers will need to interconnect with the future County and state-wide fiber networks and that the City of Moorpark would benefit from having the interconnect location within the City. As illustrated in Figures 1 and 2 above, both proposed networks go through Moorpark on the 118 from the west and then go north on the 23 to the 126. They both traverse Moorpark near the rail alignment along with the private long-haul fiber companies. CenturyLink (Lumen), Hudson Fiber, Level 3 (Lumen), Sprint/T-Mobile and Zayo all have fiber running through this area. Due to Moorpark’s central location, the County and State proposed fiber networks and the presence of these long-haul providers along the rail alignment, it was surmised that the City would be an ideal place to locate an MMR. This central location would further attract investment in additional digital infrastructure within the City, putting Moorpark at the center of the broadband crossroads of Ventura County. 45 C I T Y O F M O O R P A R K , C A M M R F I N D I N G S W W W .M A G E L L A N - A D V I S O R S . C O M 7 WHAT IS AN MMR? Fiber optic and wireless networks are deployed by a wide array of companies. These include telephone companies (such as AT&T), cable TV companies (such as Spectrum), long- haul fiber companies (Zayo), middle-mile fiber companies (such as Crown Castle), mobile network operators (such as Verizon, AT&T, T-Mobile), wireless internet service providers (such as Geolinks), and other retail and wholesale providers of active and passive infrastructure such as fiber and bandwidth. It is critical that service providers interconnect, or peer, with each other to exchange traffic. For example, users need to be able to call an AT&T cellphone user from a Spectrum landline phone. The interconnection between these networks can only occur at a small number of locations where each providers’ network is terminated. An MMR makes this interconnecting simple and economical. In the MMR, a service provider wanting to interconnect, or “peer,” with another provider would physically connect a wire from their equipment to the other provider’s equipment. A separate wired connection would be required to connect to each provider. The number of connections increases rapidly as more service providers participate. This connection matrix is solved by the MMR. One of the largest MMRs in the world is at One Wilshire Boulevard in Los Angeles. It has over 250 services providers interconnecting amongst themselves. The figure below provides a diagram of how an MMR would connect networks in Moorpark. Figure 3. Meet Me Room Diagram POWER The power system merits its own section due to its importance, complexity and both high capital expenditures (CAPEX) and high operational expenditures (OPEX). The power system takes in electric power from redundant connections to the power grid and distributes it throughout the facility, including to the tenants’ electronic equipment. It also needs to include an on-site backup power generation system. 46 C I T Y O F M O O R P A R K , C A M M R F I N D I N G S W W W .M A G E L L A N - A D V I S O R S . C O M 8 Communications is critical infrastructure for governments, enterprises, and communities. Redundancy and resilience are expected. Thus, the power system is a critical component that requires dedicated expertise. CUSTOMERS OF THE MOORPARK MMR The key success factor to any facility, such as an MMR, is whether it can attract sufficient paying customers to be financially sustainable. Having a diverse customer base reduces the risks and improves the likelihood of achieving financial breakeven or better. There are many entities in the City and Ventura County that would benefit from having the ability to interconnect to the future county and state networks and to each other. The value of a local MMR includes: Reduced costs of transport Increased flexibility Improved performance and attraction of multiple service providers to serve local businesses. Customers and users of the MMR shown in Figure 3 include the following: The Ventura County Network is the proposed county-wide fiber network connecting county anchor institutions and providing wholesale middle-mile bandwidth. Private Long Haul Fiber Providers are companies that connect facilities in major cities in a long-distance point-to-point network. The Moorpark MMR would enable them to connect to the future county and state network and to other providers in the City and county. California Open Middle Mile Network is the planned $3.25 billion middle mile fiber network. This network will be based on many parameters including the availability of interconnect facilities. Residential Providers such as Spectrum and AT&T could use the Moorpark MMR to interconnect to the county and state networks and to other providers. Mobile Network Operators, AT&T, Verizon and T-Mobile could use the Moorpark MMR to connect to the county and state networks and other providers such as Spectrum and Comcast. Private Metro Fiber Companies that serve local businesses could use the MMR to connect to the county and state networks and other providers. The City of Moorpark could use the MMR to connect to the county and state networks and other providers. 47 C I T Y O F M O O R P A R K , C A M M R F I N D I N G S W W W .M A G E L L A N - A D V I S O R S . C O M 9 2. RFI Response Summary In June 2021, The City of Moorpark issued a Request for Information (RFI) to judge interest in the Moorpark MMR. Two locations were proposed as possible sites for the MMR as indicated in the Figure below. Four companies responded. They are: Ridge Communications (San Ramon, CA) Smart Fiber Networks (San Francisco, CA) Onward (Rancho Cucamonga, CA) AT&T (Dallas, TX) The first three expressed a high level of interest in actively participating in the Moorpark MMR project. Their proposals are complementary and could represent future partnership opportunities. AT&T responded that they are not interested and would prefer that the City work with them for their communication needs. The RFI responses are summarized below. Ridge Communications o Ridge Communications is interested in engineering, permitting, constructing, and maintaining fiber and cables, power, and stand-by power for the MMR. Smart Fiber Networks (SFN) o Smart Fiber Networks is interested in assisting the City with building and supporting the operations of the MMR and they expressed interest in 48 C I T Y O F M O O R P A R K , C A M M R F I N D I N G S W W W .M A G E L L A N - A D V I S O R S . C O M 10 deploying a city-wide full fiber-to-the-premises network using the facility as a central office. Onward o Onward (formerly Inyo) proposed a PPP (Public-Private Partnership) to construct, manage and operate an MMR. They proposed that the City to designs/builds/owns the building and Onward designs/finances/owns the telecom equipment. AT&T o AT&T prefers to use their own infrastructure and partner with the City for favorable treatment and concessions to build out residential fiber-to-the- home (FTTH). SERVICE PROVIDER OUTREACH SUMMARY After reviewing the RFI responses, Magellan Advisors implemented a service provider outreach program to gain insight from regional providers into their interest in participating in the Moorpark MMR. This section highlights the interviews conducted with select service provider representatives. The results are as follows: i. Southern California Edison 1. MMR usefulness depends on what happens with the Ventura County network. 2. It is a good location and would consider using it if everything in the County falls into place. 3. Initially, would use it just for interconnect and as things grow, would consider it as a data center. ii. T-Mobile 1. Would find the MMR useful, not immediately, but in the future. 2. T-Mobile doesn’t run its own fiber; they use Crown Castle. 3. Would be helpful to some of T-Mobile’s fiber partners. iii. Geolinks 1. Likes Moorpark location. 2. Interested if providers that they purchase IP Transport from co- locate there. 3. Also want to connect with content providers including Netflix and YouTube. 4. Must have carriers: Zayo and Crown Castle. AT&T would be good to have. 5. Interested if there’s a robust datacenter as part of facility. iv. Key Information Systems 1. If the City built the building “shell” they would be interested in discussing operating it as a co-location facility. 49 C I T Y O F M O O R P A R K , C A M M R F I N D I N G S W W W .M A G E L L A N - A D V I S O R S . C O M 11 2. They would want to build around the MMR and the Data Center by adding executive suites, shared office space, etc. 3. Ideally, would prefer 10-12K square feet of space. 4. Their current Agoura Hills facility is at 80% capacity. v. Astound/Digital West/Wave Communications 1. Liked the location and indicated they would be interested in using it. 2. Could see using the facility to expand their last mile network in the City and county. vi. Google Fiber 1. Would be interested in an MMR/data center in Moorpark if they made a commitment to serve the entire county. 50 C I T Y O F M O O R P A R K , C A M M R F I N D I N G S W W W .M A G E L L A N - A D V I S O R S . C O M 12 3. Early Research Conclusions Our early research discovered that MMRs tend to be part of larger datacenters, and not stand-alone facilities. In many cases, the MMR is the driver to build the data center. Here, data center users can interconnect with a large array communication service providers very cost effectively. The architecture of the proposed facility is illustrated in Figure 4. Figure 4. MMR/Data Center Architecture WHAT ARE DATA CENTERS? Organizations of all types need systems to run their business and operations. These servers, storage and networking systems need to be physically located in a safe, secure, and environmentally resilient facility. This can be in an on-premises room in the basement, in a cage in a local data center, or on a public-cloud’s server thousands of miles away. More likely, it includes all the above and more. The term “data center” encompasses a wide range of facilities and capabilities. They range from co-location data centers to hyperscale data centers. In all cases, data centers are critical infrastructure-grade facilities. For many reasons, including business continuity, disaster recovery and economics, most organizations are looking to locate all, or part, of their critical IT infrastructure in off-site 51 C I T Y O F M O O R P A R K , C A M M R F I N D I N G S W W W .M A G E L L A N - A D V I S O R S . C O M 13 data centers away from natural and civil hazards such as hurricanes, earthquakes, and wildfires. When the entity chooses to rent space and bring their own equipment, they need a Colocation Data Center. At the other end of the spectrum are hyperscale data centers. Hyperscale data centers can be massive campuses that are measured in hundreds of Megawatts of power. One of the largest in the world is the Citadel campus in northern Nevada. It supports 650 Megawatts of 100% renewable energy and has over 7 million square feet of data center space. Google, Microsoft, Amazon Web Services, Facebook, and Apple are the top hyperscale companies. These are where “The Cloud” resides. The Cloud has grown from a remote disk drive to store photos to the mainstream storage, compute and application platform driving the world’s economy. All aspects of personal and corporate life are being cloudified or are going cloud native. Cloud providers need proper facilities to put their electronics (e.g., servers, switches). Traditionally, these locations were at or near the big centers such as LA, New York, and Ashburn, VA. For many technical and economic reasons, the edge is expanding outward from these big Internet edge cities to smaller cities and towns closer and closer to the end users. As cloud companies desire to expand their edge locations closer and closer to commercial and residential customers, they need robust facilities. Wherever large amounts of fiber is aggregated, such as a central office or a meet-me room, it is an ideal location for the edge data center to reside. These trends of everything moving to the cloud and the desire to move the edge of the cloud closer to end users makes having the MMR/Data Center connected to multiple fiber networks and providers in Moorpark even more relevant for the next many decades. COLOCATION DATA CENTERS Co-location data centers sell space and power. Space is available in a “cage”, a rectangular steel structure where the customer brings their own equipment and sets up their data center within the ‘caged’ space. Space is also available in a “rack” and “half-rack” basis. Data center customers bring their own equipment and bolt it into their rack space. In addition to providing space to their co-location customers, the data center operator is also responsible for power, security, cooling, and regulatory compliance. Security o This is focused on physical security and access to the building and to the cages and racks within the data center. This could include cameras, biometric locks, 52 C I T Y O F M O O R P A R K , C A M M R F I N D I N G S W W W .M A G E L L A N - A D V I S O R S . C O M 14 and strict access control. Data center customers are responsible for their own cyber security of their servers and systems. Power o In addition to measuring data centers in terms of area (ft2), they are also measured in terms of available electric power. Large data centers are measured in tens and hundreds of megawatts. Thus, power is a critical subsystem of the data center. In coming feeds from the power company (companies) must be redundant to prevent an outage should one get interrupted. In addition, on-site back-up power in the form of diesel generator increases both capital expenditure (CAPEX) and operational expenditures (OPEX) costs and on-going complexity. Cooling o Modern servers run exceptionally fast and that translates into running hot. This heat needs to be removed to protect the equipment. Thus, the data center must have sophisticated and redundant cooling systems to remove the heat from the environment. Regulatory Compliance o Regulatory compliance is a complex issue for data centers. The regulations that apply depend on the types of users within the facility. Healthcare companies, financial services companies and other businesses often deal with sensitive, confidential information and they need to adhere to specific compliance rules. They need infrastructure that is in accordance with federal and state regulations. For example, if a health care provider locates their corporate servers in a data center, HIPPA regulations will be in effect for the facility. Thus, even though the customers of the data center are setting up their own equipment, the facility itself must also be compliant. Data centers are classified by the level of redundancy, resiliency, and security in four categories called “tiers”. The requirements of each tier will drive both the construction costs and on-going operational costs. Most enterprises will not consider using a Tier 1 or Tier 2 facility, with Tier 3 being the most common. The requirements of each tier are summarized in the table below. 53 C I T Y O F M O O R P A R K , C A M M R F I N D I N G S W W W .M A G E L L A N -A D V I S O R S . C O M 15 Table 2 : Tiers of Data Centers Tier Minimum Requirement Tier 1 Data Center •No more than 28.8 hours of downtime per year. •Zero redundancy •99.671% uptime per annum Tier 2 Data Center •No more than 22 hours of downtime per year. •99.741% uptime per annum •Partial cooling and multiple power redundancies Tier 3 Data Center (RECOMMENDED) •No more than 1.6 hours of downtime per year. •99.982% uptime. •N+1 fault tolerance. •No maintenance related downtime. •72 hours of protection from power outages. Tier 4 Data Center •No more than 26.3 minutes of downtime per year. •99.995% uptime per annum. •2N+1 “fully redundant” infrastructure. •Zero single points of failure •No maintenance related downtime. •96 hours of protection from power outages. CO-LOCATION DATA CENTER CUSTOMERS The key to The City of Moorpark MMR/Data Center achieving financial stability is having a stable and long-term customer base with multi-year leases. As previously discussed, organizations of all types need data center space. This includes the City itself and local and regional businesses. As the cloud expands outward, companies interested in serving businesses and residences with high bandwidth low latency services could likely find Moorpark an attractive location as well. The convergence of the state network, Ventura County network, and the private long-haul providers at the MMR will greatly increase the attractiveness of the proposed facility. Thus, there is a diverse customer base of the potential co-location data center which reduces the economic risk of the proposed investment. Potential data center co-location customers include: 54 C I T Y O F M O O R P A R K , C A M M R F I N D I N G S W W W .M A G E L L A N - A D V I S O R S . C O M 16 o City IT Infrastructure The City of Moorpark needs a data center to house their IT infrastructure. The co-location facility would be an ideal location. o Regional/Local Business Businesses are looking for an alternative solution to having their critical IT infrastructure located on-site. They are looking for facilities that are secure and resilient. This is even more effective when the off-site data center is located within a short drive from the company headquarters. Additionally, businesses of all types want to connect to a variety of public and private cloud providers that have a presence in One Wilshire area facilities. The proposed facility would enable them to access them economically by connecting the Moorpark facility and then interconnecting to the abundant fiber to One Wilshire. o LA Data Center Users One Wilshire Boulevard in Los Angeles is often described as the most connected location in the US. This facility has also resulted in numerous data centers and carrier hotels being built across the street and in the near vicinity. These facilities are mission critical for many entities and they are looking for expansion and back-up locations to ensure business continuity. Given the increased demand for redundancy and disaster recovery, many entities are adding Las Vegas and Phoenix data centers to their infrastructure. A data center in Moorpark would give these customers the redundant infrastructure they demand in a facility that’s a commutable distance from LA. o Cloud and Saas Companies Software-as-a-Service (SaaS) has emerged as the dominant method for enterprises and residential users to consume software. Salesforce.com and Microsoft Office 365 are SaaS offerings. Traditional software was loaded on each PC and executed on the PC’s CPU (Central Processing Unit). With cloud-based SaaS, the software resides on a server in a remote data center and is executive remotely as well. The closer the server is to the end user, the lower the latency and the better the user experience will be. Thus, SaaS companies looking for low latency solutions to serve Ventura and Los Angeles Counties could find Moorpark an attractive option. o Service Providers Communication service providers including local, metro, and long-haul providers need data centers to house their networking equipment and servers. All participants in the MMR are likely to lease space to support 55 C I T Y O F M O O R P A R K , C A M M R F I N D I N G S W W W .M A G E L L A N - A D V I S O R S . C O M 17 their interconnections as well to locate servers for both internal network management purposes and to improve the customer experiences. PUBLIC-PRIVATE PARTNERSHIP OPPORTUNITIES Building and operating a Tier 3 Co-location data center with a Meet-me Room is a substantial endeavor. The three primary functions to design and operate within the proposed facility are the co-location data center, the MMR and the power sub-system. The City would have the choice of participating in all or none of these functions. This architecture creates a number of Public-Private Partnership opportunities that will reduce the risk to the City. Based on the RFI replies and Magellan’s provider outreach, we identified three private companies that expressed interest in participating in the long-term operations of the proposed facility. They are shown in Figure 5 and include: Key Information Systems runs a Tier 3 co-location data center in Agoura Hills and they would be interested in replicating their operations in the proposed facility. They would prefer that the City builds the building shell and then they would make a long-term commitment to operating the co-location data center. Onward proposed a PPP (Public-Private Partnership) to construct, manage and operate an MMR. Ridge Communications proposed several partnership opportunities including designing and operating the Power Sub-System. Smart Fiber Systems expressed interest in using the facility for a “central office” and entering the City of Moorpark as a “full fiber” overbuilder. 56 C I T Y O F M O O R P A R K , C A M M R F I N D I N G S W W W .M A G E L L A N - A D V I S O R S . C O M 18 Figure 5. Public Private Partnership Opportunities BENEFITS OF HAVING A DC/MMR IN MOORPARK The entire globe is going through explosive growth in fiber and wireless connectivity and cloud-based services. There are no indications that this growth will not continue for decades. There are substantial benefits to communities like Moorpark and throughout Ventura County to creating a Meet-Me Room and co-location data center in the City. Benefits in terms of the original goals of the City’s Broadband Master Plan include: 1. Increase Access, Affordability, and Reliability of Broadband Services in Moorpark a. The MMR/Data Center should enable current and future broadband providers to better serve the city. New competitive service providers such as Wave, Zayo, Onward, American Dark Fiber, Geolinks and Smart Fiber Systems would have competitive options for middle-mile connectivity to major interconnect facilities to the south in Los Angeles and north to San Jose and San Francisco. They could also leverage the facility as a central office or fiber aggregation point for local fiber deployments. 2.Facilitate the Deployment of Wireless Technologies Citywide a. To achieve the stated performance of 5G mobile networks, operators such as AT&T, Verizon and T-Mobile must densify their networks. This means they must deploy more macro-sites (big towers) as well as small cell sites. Small cell sites provide coverage over a smaller geographic area and are compact enough to be mounted on structures such as streetlights. All 5G sites benefit from having fiber to connect to their network. Each mobile network operator has finite resources, and they will 57 C I T Y O F M O O R P A R K , C A M M R F I N D I N G S W W W .M A G E L L A N - A D V I S O R S . C O M 19 prioritize their upgrade locations based on many factors including the actual costs and effort. b. The proposed MMR/Data Center will attract regional and long haul private and public providers of fiber which should create a vibrant marketplace in Moorpark for fiber and bandwidth services. These new 5G sites will be served by many of the fiber and connectivity providers that reside in the new facility. This will simplify and reduce the costs of the densification efforts of AT&T, Verizon and T-Mobile and ensure Moorpark is in the first wave of 5G and future upgrades. 3.Streamline Processes and Workflows and Remove Silos for Coordinated Broadband Efforts a. The goal is not addressed by the proposed MMR/Data Center. 4. Develop Strategic Partnerships with Private and Public Sector Entities a. The proposed MMR/Data Center will be the digital infrastructure center of Moorpark and Ventura County for decades. Thus, it will undoubtedly attract many private and public sector entities. A primary driver for the MMR/Data Center is the convergence of the public fiber networks being planned by the county and the state and the long- haul providers. Additionally, based on the initial RFI replies there are private companies interested in working with the City on long term public-private partnerships. 5. Support Smart City Applications a. The proposed MMR/Data Center will greatly facilitate Smart City applications of all types. Smart City Applications encompasses a wide range of use cases. Many of these require remote sensors (Internet of Things) connected to a centralized server. The sensors require some form of connectivity to reach the server. Depending on the application and local options, this can be fiber or wireless. Both options should be available from providers leasing long-term space in the new facility. b. The MMR/Data Center would benefit the computation side of the Smart City application too. Those applications that require a centralized server and data tools such as artificial intelligence and machine learning will benefit from having a cost- effective co-location data center in the city as well as high-speed connections to public cloud servers. Any latency sensitive application will be greatly enhanced by having the edge server in the City itself. 6. Produce Cost Savings for Telecommunications Services for Internal City Operations a. The proposed MMR/Data center should attract numerous public and private telecommunication service providers. This will create a competitive marketplace that should reduce the costs of telecommunication services for the City’s operation needs. If the MMR/Data Center attracts a full-fiber competitor to the city, it would also provide the City with additional competitive dark and lit fiber options for their internal networking needs. 58 C I T Y O F M O O R P A R K , C A M M R F I N D I N G S W W W .M A G E L L A N - A D V I S O R S . C O M 20 7. Promote Economic Development a.The economic development benefits are intertwined with many of the other objectives discussed in this section, starting with becoming a Gigabit City. The foundation of a Gigabit City is comprised of competitive gigabit connections to all locations and full 5G coverage by AT&T, Verizon, and T-Mobile. b.The more specific economic development benefits start with the initial construction jobs and continue to on-going facility operations jobs. In addition, it is expected that the City of Moorpark will benefit by attracting and retaining businesses to the area. Within the overall exploding digital economy there are many types of “gigabit” businesses that will find the new facility and local ecosystem attractive. The co- location data center and easy access to the fiber backbones of public and private providers creates a competitive marketplace for businesses connectivity needs. 8. Understand the Needs of Businesses a.The recommendation of a Tier 3 co-location data center with MMR is based on meeting the needs of current and future businesses located in Moorpark and Ventura County. 9. Attract and Retain Key Industries a.The proposed MMR/Data Center will benefit all local businesses directly and encourage them to remain in the area. First, it will provide a local co-location data center to base their digital operations. Second, it will provide them with competitive high speed connectivity solutions from multiple providers. 10. Encourage Economic Growth a.The proposed MMR/Data center both promotes economic development (Original goal 7) and will encourage economic growth for the same reasons. It will create construction jobs as well as on-going operational jobs and it will retain and attract businesses of all types. It will also attract gigabit businesses and entrepreneurs to enhance and diversify the Moorpark economy. 11.Become a Gigabit City a.The proposed facility supports the stated goal of ensuring Moorpark becomes a Gigabit City. We believe that there are three key components of local digital infrastructure to become a Gigabit City. 59 C I T Y O F M O O R P A R K , C A M M R F I N D I N G S W W W .M A G E L L A N - A D V I S O R S . C O M 21 Components of a Gigabit City The first is having competitive gigabit broadband networks. Today, Spectrum is offering “up to 1 gigabit” services and we expect them to offer more as they upgrade their systems to the newer standard of DOCSIS 4.0. AT&T has not upgraded their copper DSL to fiber-to-the-home and has decided to end upgrades to the technology. Smart Fiber Systems has expressed initial interest in overbuilding the city with a full fiber architecture and they noted that the new MMR/Data Center facility will directly impact their financial models by reducing the costs to enter the Moorpark market. The second component is ubiquitous mobile coverage by the three US mobile network operators (MNO) (AT&T, Verizon, and T-Mobile). The proposed MMR/Data Center will provide the MNOs with a location for equipment and with interconnection to their regional and national networks and connectivity partners. The third component of a Gigabit City revolves around connecting to the edge data centers. Cities can either ensure they have hundreds of Gbps of bandwidth to the edge or they must bring the edge to them. The proposed facility will do both. It will facilitate high speed connections to LA edge data centers and it will create the foundation for the edge to migrate to Moorpark. 12. Create a Tech Ecosystem a. The availability of the proposed facility should attract high-tech businesses to the City. The availability of competitive gigabit access and co-location space with easy access to a multitude of fiber providers are two key components of a high-tech ecosystem. b. The City could also accelerate the creation of the high-tech ecosystem if they expanded the facility to include spaces for a start-up company incubator, a “maker space” and other shared offices and workspaces. 60 C I T Y O F M O O R P A R K , C A M M R F I N D I N G S W W W .M A G E L L A N - A D V I S O R S . C O M 22 OTHER DATA CENTERS NEAR MOORPARK Magellan Advisors also researched the regional data center ecosystem to determine the availability of facilities near Moorpark. In Los Angeles, 40 miles from Moorpark, there are two clusters of data centers. The first is One Wilshire Boulevard and a large group of data centers around it. This is one of the most critical data centers and interconnect facility in the country. It’s also considered “full”. The second cluster is located near the LAX airport. There are also four data centers in and around Ventura County. They are as follows: Cyxtera has a data center in Burbank. Their focus in the local media industry and they are connected to both LA data center clusters. IH Networks has a web hosting data center in Chatsworth. Level 3, now Lumen Technologies, has an interconnect facility in Santa Barbara. Key Information Systems operates a co-location data center in Agoura Hills. They also have a fiber ring around Ventura County on leased fiber from Crown Castle. Their data center is currently 80% full. CoreSite has recently opened a new data center next to their current data center at One Wilshire Boulevard in Los Angeles. This new one, LA1, has 200,000 square feet and 18 MWatts of power available. Coresite also has LA2, a 500,000 square foot data center in LA’s Chinatown neighborhood. Figure 6. Regional Data Centers 61 C I T Y O F M O O R P A R K , C A M M R F I N D I N G S W W W .M A G E L L A N - A D V I S O R S . C O M 23 FINANCIALS The budgetary costs of building a data center in California is $1,000/square foot. Thus, a 5000 square foot data center would cost approximately $5 Million. A detailed financial analysis and market assessment of demand is recommended. 4. Conclusions & Next Steps There are many tangible and intangible benefits to the City and county of having the MMR/Data Center in Moorpark. It must, however, be balanced with any financial risk to the City. Moorpark can leverage its geographic location to be the digital hub of Ventura County with the MMR interconnecting the proposed County and State open access middle-mile networks, long-haul service provider networks and existing and future last mile networks. Opportunities exist for a Moorpark MMR to be an aggregation center brokering high-speed connections to One Wilshire Boulevard and other POPs and resources in the region. The potential MMR/Data Center facility supports the goals and objectives of the Moorpark Master Broadband Plan. The MMR/Data Center facility is long-term digital infrastructure and will generate income for many decades. There are interested private partners that would consider participating in the design, creation, and on-going operations of the facility. This will reduce overall operational project risk to the City and provide a number of in-kind business transactions between the City and service providers. A primary value proposition of the MMR/Data Center facility would be to interconnect the proposed county and state fiber networks with long haul fiber providers along the rail alignment. If one, or both, the county and state networks, are not built, the interconnect value proposition of the proposed facilities would greatly decrease thereby reducing the value of a co-location data center. If the county network is not built, or is not built at full scale, the demand to interconnect to the county network is reduced. Thus, the value of an MMR in Moorpark is greatly reduced. A key consideration is whether there is sufficient co-location data center demand from city and county businesses to support a 5,000 square foot Tier 3 Data Center, that is expandable to 10,000 sq. ft. with a small MMR. NEXT STEPS Given the potential upside of the proposed facility balanced against the risks, we propose the following strategy. 1. Representative(s) from Moorpark should actively participate in, and monitor, the progress of the Ventura County Network project. 62 C I T Y O F M O O R P A R K , C A M M R F I N D I N G S W W W .M A G E L L A N - A D V I S O R S . C O M 24 2. If Phase 1A, 1B AND 2 of the Ventura County Network are approved and funded, the City should go forward with further exploration of building the MMR and initial Data Center. These phases include: a.Phase1A – along Hwy 126 from Santa Clarita to Ventura b.Phase1B – Ventura to Moorpark to Simi Valley c.Phase 2 – Ventura to Oxnard to Thousand Oaks 3. If the Golden State Network approves and funds the network along Highway 126 from Interstate 5 and south on Highway 23 to Highway 101 the City should go forward with further exploration of building the MMR and initial Data Center. 4. If either, or both, of the events listed in Step 2 and Step 3 occur, the City should initiate the project by: a. Identifying the property. b. Publishing RFPs to identify construction and operational partners in the facility. c. Publishing RFPs to identify architecture and builders. d. Identifying an agent to act as the City’s broker in interconnect agreements. e. Initiating discussions with private fiber companies. 63