The CHEPREO cyberinfrastructure effort has directly supported several networking, cluster computing and grid computing activities.  These include (1) an optical network infrastructure (WHREN-LILA) linking the U.S. to education and research activities in Brazil;  (2) local computing cluster facilities at FIU that support CMS research and education projects; (3) FIU's participation in grid initiatives; and (4) deployment of new cyberinfrastructure tools. These are described briefly below.

Optical Network Infrastructure: CHEPREO deployed two Cisco ONS 15454 optical multiplexers in Miami and Sao Paulo that allowed for the growth, management, and monitoring of international bandwidth to support HEP, as well as other activities. CHEPREO funds established the circuit for less than 12 months, but were insufficient to sustain it for the remaining 3 years of the project. The successful proposal from FIU to the NSF IRNC program sustained the link capacity, then provided incremental growth to the current link capacity of 2.5 Gbps between Miami and Sao Paulo to continue to keep pace with the needs of the CMS collaboration. Several of these efforts, and the groups at the Tier-2 centers in Brazil, have leveraged the NSF-funded UltraLight (winner of the 2007 Internet2 IDEA award) efforts in advanced networking and the optimization of the Tier-2 center configurations for rapid data transfer by utilizing FAST and FDT servers deployed to Brazil in Sao Paulo. Network switches donated by Cisco Systems to UltraLight are used to connect the Latin American Tier-2s to CHEPREO-WHREN-LILA infrastructure in Sao Paulo.

AMPATH: AMPATH [AMPATH] the International Research and Education Network Exchange Point in Miami is providing CHEPREO's connectivity requirements to U.S. and non-U.S. production and research networks.  Through AMPATH, CHEPREO members access the Internet2 and NLR networks that interconnect large research projects, such as collaborations involving FermiLab and U.S. LHC Tier-2 and Tier-3 sites.  For network research and testbed applications, CHEPREO has access to the UltraLight network. The AtlanticWave [ATLANTIC] was established in 2006 to facilitate international peering and transport along the Atlantic rim among U.S. and international networks. The CHEPREO community is now able to utilize the AtlanticWave to peer with ESnet and NLR and to establish peering relationships to other networks for production and research applications.

Figure 5 is a physical diagram of the AMPATH International Exchange Point, showing Brazil's distributed Tier-2 (SPRACE and HEPGRID), the regional network of Latin America (RedCLARA), ANSP and RNP, accessing production connections (Internet2 network and AtlanticWave) and research connection (UltraLight) through the Cisco ONS 15454. Access is provided across the WHREN-LILA 2.5Gbps international circuit from Miami to Sao Paulo (described below).  Sao Paulo hosts one of the CHEPREO Cisco ONS 15454 multiplexers of the shared cyberinfrastructure connecting RedCLARA, ANSP, RNP, SPRACE and HEPGRID.

WHREN-LILA: Since 2005, support for CHEPREO's international bandwidth requirements has been provided by the WHREN-LILA [WHREN] (Western Hemisphere Research and Education Network – Links Interconnecting Latin America) project. The WHREN-LILA project is an NSF International Research Network Connections (IRNC) program award # OCI-0441095 to FIU.  It is coupled with financial support from Brazil, through the State of Sao Paulo's FAPESP [FAPESP] award to the Academic Network of Sao Paulo (ANSP [ANSP]), Award #2003/13708-0.

Figure 1: AMPATH IXP Physical Diagram and Connectivity

A notable achievement has been the integration of CHEPREO and WHREN-LILA to create a single cyberinfrastructure shared by diverse communities in the western hemisphere, facilitating new collaborative opportunities. The CHEPREO-WHREN-LILA cyberinfrastructure connects the CHEPREO community to several Latin American research and education networks: ANSP (the academic network of Sao Paulo), RedCLARA (the regional network backbone connecting the NRENs of Latin America), and RNP (the NREN of Brazil).  It also provides improved access and increased bandwidth and growth for the distributed Tier-2 Grid facility in Brazil, consisting of HEPGrid UERJ in Rio de Janeiro and SPRACE at USP/UNESP in Sao Paulo.



Figure 2: Open Science Grid Sites

FIU computing facilities: Integration with CMS and Open Science Grid: Open Science Grid [OSG] is a consortium of approximately 75 institutions and organizations that operates a large-scale shared distributed cyberinfrastructure supporting U.S. science and engineering (see Fig. 6). Funding is provided by NSF and DOE of $30M for five years starting in 2006. Because all resources are owned by the participating institutions, OSG funds only support personnel for tasks involving the common facility (software integration, operations, security). P. Avery of UF is an OSG co-PI and serves on the Executive Team as Resource Manager. All U.S. institutions within the CMS worldwide cyberinfrastructure are part of OSG.

All CHEPREO institutions are OSG resource providers: two CMS Tier-2 Centers (UF and Caltech) and two Tier-3 Centers (FSU and FIU) whose participation in OSG was facilitated by CHEPREO. Both Tier-2 institutions are supported by the US-CMS research program and have extensive CPU and storage resources that are used in CMS physics production simulation exercises. Caltech, FSU and UF have 10 Gbps campus connections to the wide area network, while FIU has a presence in the NAP of the Americas with 10 Gbps, permitting very fast transfer of CMS datasets. All four sites support CMS users through local grid-enabled interactive login servers and computing clusters, but their resources also serve other research communities because of their participation in OSG.

CHEPREO funds enabled UF and FIU to create the Florida Grid Operations Center (fGOC) in 2006. The center supports the Florida CHEPREO sites as well as Florida Tech (FIT), which collaborates with CHEPREO through its participation in CMS, and UF's High Performance Computing (HPC) center at UF, whose very large CPU and disk resources are used heavily by OSG and CMS. The fGOC provides a robust framework (Trouble Tickets, security, mail lists, documentation, software downloads) that is well-integrated within the OSG hierarchical support model.  Through the fGOC, members can resolve installation issues and answer questions relating to the OSG software stack and operations.

CHEPREO funded FIU's Tier-3 computing facility, which currently consists of a single grid-enabled cluster of 20 dual-processor nodes and 1.5 TB RAID storage. The facility is being reorganized in preparation for new hardware (from current funds) and an increase in local CMS computing activities due to increases in the number of local CMS graduate students, the arrival of new faculty member Jorge L. Rodriguez and the commencement of data taking at CMS. The facility also supports the computing needs of CHEPREO's Physics Education Research program. The reorganized facility will support all these activities and provide the foundation on which to add computing resources for CHEPREO and CMS.
Deployment of Cyberinfrastructure Tools: CHEPREO-sponsored interactions and funding make it possible to deploy advanced tools and services such as the following:

• MonaLISA: The MonaLISA [MONALISA] monitoring system developed by Caltech provides a distributed service for monitoring, control and global optimization of complex systems including the networks and sites managed by the CHEPREO project. It is deployed at more than 340 sites.
• ROOTlets: The ROOTlet service makes ROOT, a widely-used HEP analysis framework, available to lightweight client environments. The service is latency tolerant and scalable to the worldwide grid, allowing for independent execution of ROOT code by authenticated and authorized users.
• FDT: Caltech's Fast Data Transfer (FDT) is capable of reading and writing at disk speed over wide area networks using standard TCP. At Supercomputing 2006, Caltech led an HEP consortium in an FDT demonstration that achieved a peak throughput of 17.77 Gbps using a single bidirectional 10-Gbps link. Work is underway with FNAL and developers at DESY on integrating FDT with the CMS storage system dCache to enhance dataset transfer performance.