General topics related to Grid computing Ad Emmen 2009-05-20T08:38:06Z 2009-05-20T08:38:06Z

Discussion document download. The [i]Production Desktop Grids - Standardisation Requirements. Discussion document.[/i] is available at: [url=http://www.edges-grid.eu/c/document_library/get_file?p_l_id=29093&folderId=23974&name=DLFE-1213.doc]Production Desktop Grids - standardisation requirements[/url] doc format. [url=http://www.edges-grid.eu/c/document_library/get_file?p_l_id=29093&folderId=23974&name=DLFE-1214.pdf]Production Desktop Grids - Standardisation Requirements.[/url] PDF format.

Ad Emmen 2009-05-20T08:38:06Z Ad Emmen 2009-05-16T08:04:43Z 2009-05-16T08:04:43Z

Please note that this, indeed, is a very first draft of the document. It is just provided to give a general idea of the contents we intend to cover. Please send comments to one of the authors.

Ad Emmen 2009-05-16T08:04:43Z Etienne Urbah 2009-05-15T20:45:25Z 2009-05-15T20:45:25Z

Desktop Grids is a general term for a Grid where the Grid resources are not under control of the user or Grid operator. A typical Desktop Grid is a volunteer computing Grid that can use resources from people at home, or a company Desktop Grid that uses idle time of office machines for statistical calculations. The main reason to look at these Grids from an eScience perspective is the vast amount of resources. The biggest volunteer computing Grid has more that 2 million PCs connected as resources. Desktop Grids in large enterprises can easily include tens of thousands of processors. Volunteer computing Grids depend on keeping the resource owners satisfied: otherwise they disconnect from the Grid. As research has shown [DNA2.2], it is mainly providing feedback about the applications they run and the ability to choose, that make them choose for a specific Grid. Having no or little control over the resources means that for instance security mechanisms that are used in Service Grids do not work. For a volunteer Desktop computing grid, you can never be sure a resource will deliver a result in time, and whether that result has not been tampered with. So the challenge from the Grid operator's point of view is how to build an SLA style computing service for users out of untrusted, volatile computer resources. The resource owner (for instance a person at home), must be absolutely sure the Grid application he runs will not harm his machine or software configuration, and is indeed the application he volunteered to run. For enterprise Desktop Grids, it is often a bit easier, as computers are administered by a single authority. During the past years, there has been tremendous progress in operating production Desktop Grids that provide a reliable computing resource to users. However, most of the Desktop Grid operators and Desktop Grid middleware provider have solved problems on their own each in a slightly different way. They do no yet use or apply Grid standards in a rigorous way yet. A number of Desktop Grid operators have connected their Desktop Grids to Service Grids. Again, mostly with own solutions. In the following document, we want to provide background information on production Desktop Grids and interoperability between production Desktop Grids and Service Grids. The goal is to better understand the requirements for standards from these type of Grids, identify possible standards that could already be applied, and identify potential areas for standardisation.

Etienne Urbah 2009-05-15T20:45:25Z Etienne Urbah 2008-07-24T19:26:18Z 2008-07-24T19:26:18Z

[b]List of the EDGeS Activities (Work Packages) with Lead participant and Short description[/b] (from EDGeS_DoWB_v4_7_final.pdf) : I do NOT mean this text to be the start of a discussion thread, but to be a WIKI page. So : – If there is any comment, I will directly update this text. – As soon as I have write access in the EDGeS Wiki, I will transfer this text there. Abbreviations : – DG = Desktop Grid (Resource Scavenging Grid powered by a middleware such as [url=http://boinc.ssl.berkeley.edu/]BOINC[/url], [url=http://dghep.lal.in2p3.fr/spip.php?rubrique13]XtremWeb[/url], ...) – SG = Service Grid (Institutional Computing Grid such as [url=http://www.eu-egee.org/]EGEE[/url], [url=http://www.nordugrid.org/]NorduGrid[/url], [url=http://www.deisa.eu/]DEISA[/url], ...) [i]Beware that EACH EU-funded project has its own list of NA1..., SA1..., JRA1... activities. Therefore, as example, the JRA1 activity of EDGeS is completely different from the JRA1 activity of EGEE.[/i] [b]EDGeS NA1 – Project administrative and technical management - [url=http://www.lpds.sztaki.hu/]MTA SZTAKI - LPDS[/url] (Hungary)[/b] This activity is vital for the success of the project. It should provide a lightweight and efficient management of the consortium by putting in place management structure, decision and communication mechanisms that enable the effective administrative and technical management of the consortium. It also should represent the project vision and mission in European and international initiatives. [b]EDGeS NA2 – Knowledge Services (dissemination, training and consulting) – [url=http://almeregrid.nl/]AlmereGrid[/url] (Netherlands)[/b] This activity focuses on providing expert consulting and technical support to new communities in the adoption and customization of the EDGeS services. It will disseminate the knowledge produced by the project through the organization of focused events, and will train scientists in developing applications for the combined EGEE-DG e-infrastructure maintained by EDGeS. NA2 will also train system administrators on how to establish and run local and public DGs, and how to integrate these systems into the combined EGEE-DG e-infrastructure. This activity has also got the crucial responsibility of collecting and understanding user requirements, and identifying new user communities that need the integrated EGEE-DG e-infrastructure. In order to achieve this, it will include the organization of a User Forum. Besides, it will organize an Industry Forum too, in order to attract companies who are willing to connect their local DG based enterprise grids into the combined EGEE-DG e-infrastructure. [b]EDGeS NA3 – Standardization procedures – [url=http://dghep.lal.in2p3.fr/spip.php?rubrique5]CNRS IN2P3 LAL[/url] (France)[/b] This activity will focus on collecting and spreading the new standards proposed by this project while aggregating the deepest communion. It will also open the widest possible audience from public and private entities. It will call for experience and expertise sharing in order to define a definitive standard to enable SG and DG resource sharing by unifying a set of standard and immutable interfaces as well as services and protocols. The process of standardization will iterate until an agreement is found within the obtained audience. NA3 will highly interact with JRA1 since this collaboration will help to iterate through the definition of the standard. [b]EDGeS SA1 – Combined EGEE-DG Service Management – [url=http://www.lpds.sztaki.hu/]MTA SZTAKI - LPDS[/url] (Hungary)[/b] This activity has to establish and maintain the combined EGEE-DG e-infrastructure, a production service that combines EGEE and existing local and public DG systems. SA1 will also establish as part of the combined EGEE-DG e-infrastructure two public DG systems (one based on BOINC and one based on the XtremWeb technology) that will be 100% devoted to run EGEE applications. The service will also include the creation of an application repository where validated and certified applications that are able to run on the combined EGEE-DG e-infrastructure will be stored. [b]EDGeS SA2 – Support infrastructure Service Management – [url=http://www.ceta-ciemat.es/es/]CIEMAT[/url] (Spain)[/b] This activity has to support the effective usage of the production EGEE-DG e-infrastructure. It will provide an application testing and validation infrastructure and will provide the testing and validation procedure as service for the various user communities. SA2 will also establish and maintain a training and application development infrastructure (a small version of the production EGEE-DG e-infrastructure) that will be used by the NA2 and SA3 activities. [b]EDGeS SA3 – Application Support Service – [url=http://www.cpc.wmin.ac.uk/cpcsite/index.php/Main_Page]University of Westminster - Centre for Parallel Computing[/url] (UK)[/b] This activity has to support current and future user communities in developing their applications for the combined EGEE-DG e-infrastructure. SA3 will establish two application development centres, where users will be provided with direct help in gridifying and boincifying their applications. Following the successful implementation of the reference applications, SA3 aims to raise awareness of the developed and operated platform, and extend the user community by selecting and porting applications to the infrastructure based on invitation. SA3 will develop and publish the selection methodology and criteria, and will support the selected user communities to port their applications to the platform up to the phase of a working and approved prototype. [b]EDGeS JRA1 – SG-DG Bridges Technologies – [url=http://www.lri.fr/projet.associe.php?prj=15]INRIA - Grand Large[/url] (France)[/b] This activity has to develop the bi-directional SG-DG bridges technologies, including the integration of EGEE with public and local DGs based on BOINC and Xtremweb technologies. JRA1 will solve the seamless integration of SG and DG grids hiding the technology difference at the user level. This research activity will include tackling the issues of task scheduling across the different platforms and solving the security issues. [b]EDGeS JRA2 – SG-DG Support Technologies – [url=https://www.dei.uc.pt/weboncampus/]Universidade de Coimbra - Departamento de Engenharia Informática[/url] (Portugal)[/b] This activity will produce tools and APIs by which the user can adapt their applications to the integrated EGEE-DG infrastructure. JRA2 will further develop the necessary application development tools according to the users’ feedback. It also provides monitoring and benchmarking tools to support the interaction between EGEE and the DG Environments, and assess the operation of the EGEE-DG bridge. JRA2 will integrate the already existing monitoring tools of EGEE, BOINC and XtremWeb, providing a consistent view of the system, while the real novelty comes from the bridge and the extended possibilities that it brings. JRA2 will extend the existing monitoring tools where necessary, in order to cover both the resource and application monitoring areas. JRA2 will also develop the necessary benchmarking mechanisms by which the performance of the integrated EGEE-DG e-infrastructure can be measured and tuned. [b]EDGeS JRA3 – SG-DG Data Access – [url=http://www.cs.cf.ac.uk/research/dcc.php]Cardiff University - CSCS - Distributed Collaborative Computing[/url] (UK)[/b] This activity has to solve the data management problems of the integrated EGEE-DG e-infrastructure. First, JRA3 will focus on extending the capabilities of the EGEE-DG Bridge (to be developed in JRA1), to provide mechanisms to transfer large data sets from EGEE resources and repositories to the DG environments. Second, it will extend and integrate current Peer-to-Peer data distribution technologies to adapt to the challenges of a Desktop Grid environment. Work will focus on building a secure and reliable network that enforces data integrity and self-organizes to provide a scalable network with an adaptable topology and innovative transit mechanisms such as file swarming. Third, JRA3 will focus on enabling current Desktop Grid applications access to the data distribution network.

Etienne Urbah 2008-07-24T19:26:18Z Etienne Urbah 2008-07-11T12:39:16Z 2008-07-11T12:39:16Z

Since you assess that 'Desktop Grid' is a term that is widely used, and more or less understood by anyone, I accept it as a shortcut for 'Resource Scavenging Grid'. But I would be very cautious including notebooks, because notebooks are NOT designed for continuous 100% CPU usage, and are at risk of overheating. Etienne URBAH

Etienne Urbah 2008-07-11T12:39:16Z Ad Emmen 2008-07-10T16:04:37Z 2008-07-10T16:04:37Z

"Desktop Grids" is a term that is widely used. It is more or less understood by anyone in the same way: It is a Grid that consists of a number of Desktop computers: i.e. not of servers or clusters in a computer centre. See the definition in the EDGeS Wiki: [url=http://www.edges-grid.eu/web/11219/2/wikipage/11978/WhatIsGridComputing]EDGeS Wiki[/url]. Personally I include notebooks too (I am typing this on a notebook on my Desktop-_- ) Desktop Grids are not only uses for voluntary computing projects, but also widely for enterprise Grids.

Ad Emmen 2008-07-10T16:04:37Z Etienne Urbah 2008-06-13T16:23:51Z 2008-06-13T16:23:51Z

Thanks to Gilles for his reply. 'SG' and 'DG' are not words, but abbreviations. 'Service Grid' and 'Desktop Grid' are VERY ambiguous expressions. Even if the partners have started the EDGeS project with the European Commission as being 'Integrating Service and Desktop Grids', EDGeS is a research project, and it is our duty to replace ambiguous expressions by clearer expressions. After more research, I strongly suggest to rename : - 'Service Grid' as 'Institutional Computing Grid', - 'Desktop Grid' as 'Resource Scavenging Grid', - 'Integrating Service and Desktop Grids' as 'Integrating Institutional Computing Grids with Resource Scavenging Grids towards a Global Computing Grid'. I attach here the updated content of following file : svn+ssh://cvs.lpds.sztaki.hu/var/lib/svn/edges/na3/docs/misc/Edges-Glossary.html Etienne URBAH

Etienne Urbah 2008-06-13T16:23:51Z Etienne Urbah 2008-06-13T16:01:46Z 2008-06-13T16:01:46Z

On 09 June 2008, Gilles FEDAK answered : Choosing the correct words to describe our projects is a difficult task. We already had discussion at the time of the writing of the proposal. The name SG and DG results from a consensus between the partners. At some points it is now too late to change this points. I appreciate your effort to write this glossary. It is very useful to explain the outside world what we are doing. But it is also very important to stick to words which are used in the community. For instance, I have never seen the word "Grid of Cycle Stealing" and the definition of Cycle Stealing that you give is wrong. Please check project like GLunix, Condor for instance and many papers written in the 90's. My 2 cents advice : don't use Wikipedia but IEEE Xplore or ACM Gilles FEDAK

Etienne Urbah 2008-06-13T16:01:46Z Etienne Urbah 2008-06-06T19:57:06Z 2008-06-06T19:57:06Z

Inside our EDGeS project, we will have to make the protocols for cycle stealing computing and peer to peer storage traverse the firewalls protecting the computing clusters. This can be predicted to be a nightmare : - Security and network engineers with refuse to let the necessary ports open. - More and more, hole punching, SYN spoofing, ... will be detected and blocked by the firewalls. On Monday 02 June 2008, in OGF23 at Barcelona, I attended a BoF about 'Firewall Virtualization for Grid Applications' described at http://www.ogf.org/gf/event_schedule/index.php?id=1296 This BoF was very interesting, as it completely addresses the problem, and propose following clean solution : 1) Once : Install a secure Authorization Server, which has to be trusted by all firewalls to be traversed. 2) Each time a connection is needed : With a certificate, kindly ask the trusted Authorization Server to request each firewall to open the necessary ports. 3) When the connection in not needed anymore : Kindly ask the trusted Authorization Server to request each firewall to close the ports. See in particular slide 5 of attached 'FVGA-WG-BOF.ppt' presentation. The authors of this BoF are preparing the creation of the 'fvga-wg' OGF working group. I think that we can not do much without their work, and I strongly suggest to encourage them.

Etienne Urbah 2008-06-06T19:57:06Z Etienne Urbah 2008-06-06T18:13:21Z 2008-06-06T18:13:21Z

I am afraid that the terminology/vocabulary that we are using to describe our work is NOT understood outside Edges insiders. That could explain why our 'Integrating Service and Desktop Grids' BoF at OGF23 had a limited number of attendants. In order to help everybody to understand our subject of study and our work, I strongly suggest that we define and publish on the Edges public web site a glossary of the terminology/vocabulary that we are using, with examples. My suggestion for the Edges Glossary, which I have already checked using Wikipedia, is to be discussed, improved, and updated in following file (also attached in this post) : svn+ssh://cvs.lpds.sztaki.hu/var/lib/svn/edges/na3/docs/misc/Edges-Glossary.html As a result of this work, I would suggest, for example, to rename 'Integrating Service and Desktop Grids' as 'Integrating Grids of Cluster Computing with Grids of Cycle Stealing'.

Etienne Urbah 2008-06-06T18:13:21Z José Luis Guisado 2008-05-26T19:14:08Z 2008-05-26T19:14:08Z

Hi, Here I have included a short presentation describing the CALD (Cellular Automata-based Laser Dynamics) application, which I find interesting to be migrated to Desktop/Service Grid. I have also sent it to Ad Emmen for the 1st EDGeS User and Industry Forum Meeting. Best regards, ================================================= José Luis Guisado Departamento de Arquitectura y Tecnología de Computadores - Computer Architecture and Technology Universidad de Sevilla Escuela Técnica Superior de Ingeniería Informática Avda. Reina Mercedes, S/N. 41012-Sevilla, SPAIN Despacho I0.72 Tel.: (+34) 954 55 62 41 Fax: (+34) 954 55 28 99 E-mail: jlguisado@us.es Web: http://www.atc.us.es/~jlguisado =================================================

José Luis Guisado 2008-05-26T19:14:08Z Ad Emmen 2008-03-15T22:34:38Z 2008-03-15T22:34:38Z

This Forum can be used for discussions about all topics related to Desktop Grid computing.

Ad Emmen 2008-03-15T22:34:38Z