In the absence of an effective and affordable vaccine and non-curative abilities of current antiretroviral therapies, behavioural and psychosocial prevention with the goal of limiting sexual risk behaviours remains central to the efforts to decrease sexual HIV/STI transmissions among MSM [8]. Further, while antiretroviral therapy treatments have tremendous life-saving potential, they are expensive and carry debilitating side-effects for some people [9]. Behavioural and psychosocial HIV/STI risk reduction interventions to reduce unprotected sex among MSM range from individual-level interventions and group level-programmes, to community-level interventions [10,11]. Such interventions will continue to be vital in the battle against HIV/STI, and therefore it is important to find out whether they help, harm or are ineffective.

The effectiveness of HIV/STI preventive interventions targeted at MSM has been assessed in various publications. Most recently, in 2008 Johnson et al. systematic Cochrane review evaluated the effects of behavioural interventions to reduce risk for sexual transmission of HIV among MSM [12]. The review included 58 randomised controlled trials (RCT), of which almost three quarters were from the United States (US). The review concluded that behavioural interventions reduced UAI by 27% compared to minimal or no HIV preventive intervention. A few other reviews have been published about the effectiveness of HIV prevention interventions, but most of these are not specific to MSM. When the target population has comprised MSM, MSM in Europe have not been the focus. Further, the majority of reviews have neither utilised a comprehensive search strategy nor clear inclusion criteria, and many of the reviews are out of date, having been published before or shortly after the year 2000. Therefore, there is a need for a systematic review that incorporates explicit inclusion criteria and that updates the current knowledge base about HIV/STI preventive interventions targeted at MSM in Europe. The objectives of the systematic review were to

1. Identify and describe outcome studies evaluating the effectiveness of HIV/STI prevention interventions on UAI for MSM living in Europe.     
2. Summarise the effectiveness of HIV/STI prevention interventions for MSM in reducing unprotected anal sex, and, if available and possible, HIV/STI infections.
3. Identify intervention characteristics associated with effectiveness.
4. Identify gaps in a) subpopulations targeted, b) intervention characteristics incorporated, c) outcomes evaluated, d) methodological matters.

Methods

The completion of the systematic literature review was in accordance with the Cochrane Collaboration standards [13].

Search methods for identification of studies
The primary method of study identification was electronic searches. Under the guidance of the author, a research librarian designed and executed the electronic database search. References in obtained reviews and included primary studies were scanned to identify new leads and included studies were looked up in ISI Web of Knowledge in order to identify further studies.

We applied the population, intervention, comparison, outcome (PICO) model described by Sackett et al. with respect to criteria for considering studies [14]. Concerning population, the intervention had to be received by MSM, who resided in the European region. We introduced the regional specification to ensure the included studies were clearly relevant for European-based research and intervention activities. We enforced no other limitations on participant characteristics. All forms of behavioural and psychosocial interventions designed to promote safer sexual risk behaviours among MSM were eligible for inclusion. There were no restrictions in level or mode of delivery. Regarding types of comparisons, we accepted no intervention, minimal intervention, placebo psychotherapy, standard treatment, or other active HIV/STI preventive intervention condition. We viewed studies in scope if they included measurement of sexual behavioural or biological outcome indicative of HIV/STI transmission risk.

With respect to study design, eligible studies were RCT, controlled clinical trials (CCT), and controlled before-and-after (CBA) studies. Lastly, only publications written in English, German, or one of the Scandinavian (Danish, Norwegian, Swedish) languages were included. To ensure that all research included was relatively new, we included only publications that were published in or after the year 2000.

The screening of literature was carried out in a three-stage procedure (screening of title, abstract, full text) whereby each level consisted of increasing scrutiny of the studies based on the inclusion and exclusion criteria of the review, as described above.

Data extraction and analysis
Data from each included study were extracted using a pre-designed data extraction form. All data were entered twice and the accuracy of all data extracted by the main reviewer was checked, including data in tables, before analyses were initiated. With respect to quality of the evidence, we used the Cochrane Collaboration's tool for assessing risk of bias [13]. Two reviewers discussed and agreed about the adequacy of risk of bias for six domains by assigning a judgement of 'yes' indicating low risk of bias, 'no' indicating high risk of bias, and 'unclear' indicating unclear or unknown risk of bias. Criteria set by the Cochrane handbook and adapted to the health promotion field were used to make these judgements. We estimated effects of interventions in two ways for binary outcome measures. One, by the adjusted absolute risk difference (ARD) in which the pre-post change score (in percentage points) in the control group was subtracted from the pre-post change score in the intervention group, and two, by the risk ratio (RR) and 95% confidence interval (CI) (95%CI) based on the post intervention data. We also decided, a priori, to perform meta-analyses to estimate intervention effect. We used Mantel-Haenszel random effects meta-analyses because it was assumed that the intervention effects would vary across studies. All analyses were conducted using RevMan5 [15]. Where there were several follow up times, we analysed them separately.
 
Results
     
The literature search resulted in 2,199 potentially relevant records (Figure 1).

Figure 1. PRISMA flow diagram of the literature reviewing process. Literature review on the effectiveness of behavioural and psychosocial HIV/STI prevention interventions for men who have sex with men in Europe. Berg R, 2009

 

We excluded 2,166 records at title or abstract level which were clearly outside the scope of this systematic review (e.g. descriptive studies), leaving 33 potentially relevant records which were read in full text. We included six studies presented in seven publications for our evaluation [16-22]. One study is unpublished but results were made available [23].

Description of studies
Four of the included studies employed a randomised controlled design, including two cluster RCT, and the remaining two included studies were CBA studies [16,21-23]; [16,23]; [17,19] (Table 1).

Table 1. Description of included studies (n=6). Literature review on the effectiveness of behavioural and psychosocial HIV/STI prevention interventions for men who have sex with men in Europe. Berg R, 2009

 

The included studies involved a total of 4,111 participants at entry (range 50-2,276) from four different European countries: Russia and Bulgaria, the Netherlands, and the United Kingdom [16];[23];[17,19,21,22].

The studies targeted gay and bisexual men of various ages (one study also included 55 women) [16]. One study specifically targeted young MSM, another aimed to promote sexual health in HIV-positive MSM [16];[23]. In the four studies that reported information about ethnic background, the populations were predominantly white (about 90%) [17,21,22,23].     

The self-help and motivational enhancement intervention of van Kesteren et al. was individual-based and consisted of a self-help guide, a face-to-face motivational interview, and a motivational interviewing telephone call [23]. Two interventions consisted of group sessions; one covered various aspects about sadomasochistic sex [21], while the other was a cognitive behavioural workshop [22]. The remaining three studies were community-based and modelled after the popular opinion leader interventions developed by Kelly et al. and Kegeles et al. [24,25];[26,27]. In sum, not two interventions were identical, but the three peer-led, social behavioural interventions were similarly modelled and all but one of the interventions were theory-based [16,17,19];[21]. With respect to intensity and duration (dose) of the interventions, this was not clearly ascertained from the texts, but the programmes appeared to have 'intervened' from one peer conversation of about ten minutes duration to about 28 hours of education. Primary mode of delivery was in person, generally one-on-one.

Only one category of comparison was used in the six included studies: Minimal to no intervention. Imrie et al. used standard treatment at an STI clinic as comparison [22]. Three studies placed the comparison group on a waiting list to receive the intervention after the study [16,21,23].

With respect to outcome measures, all the included studies had collected self-report data about UAI with men. One study included a biological measure of new sexually transmitted infection [22]. Follow up ranged from two months post intervention to 18 months [21];[17]. Several studies incorporated multiple follow ups [16,17,21,22].

Risk of bias in included studies
The risk of bias assessment comprised six domains and we judged that there was 'high' or 'unclear' risk of bias in one or more of the assessed domains in all studies (Figure 2).

Figure 2. Result of risk of bias assessment. Literature review on the effectiveness of behavioural and psychosocial HIV/STI prevention interventions for men who have sex with men in Europe. Berg R, 2009

Briefly, with respect to sequence generation, there was insufficient information in all studies, except one, to judge whether it was adequate [22]. The situation was similar for allocation concealment and blinding. The issue of incomplete outcome data was adequately addressed in two studies [16,22] and unclear or insufficiently addressed in the remaining four studies. All studies were judged to be free of selective reporting. Lastly, we judged other risk of bias, including intervention exposure, which varied greatly among the studies. It was lowest in the gym-based study – 3% of the participants reported having spoken to a peer educator during the intervention [17].

Effects of HIV/STI prevention interventions for MSM
A priori we decided to focus our effectiveness analyses on UAI because it is the most epidemiologically pertinent behaviour for MSM in an HIV risk context, and likely to be included in most studies [28]. UAI was reported as a dichotomous outcome, thus, we calculated ARD and RR with 95% CI based on the post intervention data. Two texts did not provide data in sufficient detail for us to include them in analyses (requested data from the authors were not received in time for inclusion in the analysis) and we reproduced the results of their significance tests [21,23]. With respect to sexual risks, we could calculate effect estimates for six outcomes (multiple assessment points) across four studies (Table 2).

Table 2. Sexual risk behaviour outcomes at baseline and follow up, and effect estimates for included studies. Literature review on the effectiveness of behavioural and psychosocial HIV/STI prevention interventions for men who have sex with men in Europe. Berg R, 2009

 

At study level, four of the six included studies reported null effect. While all ARD results indicated that the interventions had positive effect, inspection of the effect estimates show that three quarters of the outcomes failed to reach significance.

We used Mantel-Haenszel random effects meta-analyses to estimate the intervention effect of the four interventions for which we obtained data. Collectively, the four interventions that were measured against minimal to no HIV prevention intervention appeared to reduce the probability of gay-or bisexual identified men engaging in UAI (Figure 3).

Figure 3. Forest plot of effect sizes, main effect and subgroup analyses for unprotected anal intercourse. Literature review on the effectiveness of behavioural and psychosocial HIV/STI prevention interventions for men who have sex with men in Europe. Berg R, 2009

The pooled effect estimate of the four interventions suggested that MSM who participated in HIV/STI prevention initiatives were 10% less likely to report UAI (RR 0.90, 95%CI 0.83-0.96). The total MSM sample in these four interventions was 3,777. One study included 2,380 MSM, and consequently, the study contributed disproportional weight (84.9%) to the pooled effect estimate [19]. In subgroup analyses the pooled effect estimate showed that the result of the two interventions with design of highest internal validity (RCT) became non-significant (RR 0.81, 95%CI 0.65-1.03), while the result of the CBA studies was significant (RR 0.91, 95%CI 0.84-0.98), with one study contributing disproportional weight (94.6%) to the pooled effect estimate. Similarly, the pooled effect estimate of the three community-level interventions reached significance (RR 0.90, 95%CI 0.83-0.97), with one study contributing disproportionate weight (90.4%).                 
 
In subgroup analyses, the pooled effect estimate for the short-term effects (three-six months) of the two RCT with least risk of bias suggested that MSM participating in HIV/STI interventions were 34% less likely to report engaging in UAI (RR 0.66, 95%CI 0.52-0.82). The effect was not significant at medium-term follow up (12 months) (RR 0.81, 95%CI 0.65-1.03).

The evidence base was insufficient to examine characteristics of interventions most closely associated with magnitude of effect and to draw solid conclusions about unique gaps in the evaluation literature on HIV/STI interventions for MSM in Europe.

Discussion

This is the first systematic review to summarise and assess the effectiveness of HIV/STI prevention interventions for MSM living in Europe. The main finding of the review is the dearth of HIV/STI prevention interventions for European MSM which have been evaluated in such a way as to enable reliable conclusions about effectiveness. Among the six studies identified and included the proportion of information from studies at high risk of bias was sufficient to affect the interpretation of results.

Effectiveness of HIV/STI prevention interventions for MSM
The meta-analysis results of four studies showed that one pooled effect size is most valid. The subgroup analysis for the short-term effects of the interventions by Amirkhanian et al. and Imrie et al. suggested that MSM participating in HIV/STI interventions were significantly less likely to report engaging in UAI than MSM in the control groups at short-term follow up [16]; [22]. An effect size associated with significant reduction in UAI was not found at 12 months follow up. The findings mirror other high-quality reviews showing that effects of non-US interventions are limited and become attenuated over time [18,12,29]. In stratified analyses of rate ratios for small group and individual-level interventions, a recent Cochrane review found that while studies performed in the US yielded a net reduction of 22% in unprotected sex, studies performed elsewhere in the world showed a much smaller net reduction that was not statistically significant [12]. Nonetheless, the findings in the current systematic review give cause for guarded optimism. The controlled studies included in this systematic review demonstrate that it is possible to successfully conduct rigorous HIV/STI prevention trials for MSM in Europe, and there may be some effect of interventions aimed at reducing HIV/STI risk behaviour among this population.
 
Gaps
It is not presently possible to know which unique gaps in the evaluation literature on HIV/STI interventions for MSM in Europe exist. However, it should be noted that all but one of the six included studies are from Western Europe; four of them were set in the United Kingdom. Further, the samples included mainly white MSM. Non-white MSM appear to be underserved. Only one study included a biological outcome measure.
 
Implications for future behavioural HIV/STI interventions for MSM                  
Almost thirty years into the HIV epidemic, it is disheartening to find so few behavioural HIV/STI prevention interventions that have been rigorously evaluated for MSM in Europe. The paucity of controlled studies demonstrates the need for research in this area: more and better outcome evaluations of HIV/STI prevention interventions for MSM living in Europe are warranted. While there is no other reliable substitute for evaluating the effect of interventions than controlled trials, other designs such as interrupted time series designs can also be used [30,13,31]. Researchers who are concerned about the ethics of allocation to experimental groups can use waiting list controls whereby the control group receives the potential beneficial intervention post data collection. The drawback is the difficulty of establishing long-term effectiveness of the intervention [32]. It also remains important to integrate process assessment into the evaluation design in order to learn about feasibility, acceptability, practical constraints, and related issues. Implementation and adherence are typically difficult to measure in multi-component intervention programmes, but provide critical information [33]. For example, Elford et al. process evaluation helped explain the likely reasons for lack of programme effectiveness [34]. Researchers and journal editors should strive to disseminate also null findings and related issues in intervention research [35].

As far as possible, prevention professionals should incorporate clinical HIV/STI outcomes, and not just rely on self-reported changes in cognitions and behaviours. Cognitive processes are not necessarily pre-requisites for behaviours and as self-reported behavioural outcomes, tend to overestimate intervention benefits [31,36]. Further, because risk assessment for HIV transmission by self-report covers a wide range of behaviours it would be important to specify UAI according to partner type and partners' serostatus, as done by two of the included studies [17,19]. One alternative suggested by Newman et al. is to use new technology, such as computer assisted self-assessment, to improve the truthfulness of self-reported sexual behaviours [37]. Biological outcomes reliably assess potential harms as well as benefits. Of the six included studies in this systematic review, only one included clinical outcomes and it found that incidence of STI significantly increased in the intervention group compared to the control group. Imrie et al. state that screening of asymptomatic infection was not part of the original study protocol because they believed it would affect recruitment, but the return of specimens by post worked well [22]. Lastly, multiple follow up assessments allow for an evaluation of the longevity of effectiveness and should be attempted. Several of the included studies in this systematic review did, but the longest follow up was 18 months. Ideally, since incidence of HIV/STI infections is the most important and reliable outcome and changes cannot be reliably measured in a short time period, long-term follow up of several years is desirable.

Strengths and limitations of this review
This systematic literature review was conducted according to the Cochrane Collaboration standards [13]. A further strength is that controlled studies were evaluated, i.e. studies that can reliably say something about effects of interventions. Additionally, meta-analyses were conducted to synthesise independent and diverse studies to derive an overall estimate of effectiveness of interventions, allowing also for an exploration of differences across studies. However, findings must be viewed within the context of the limitations of the systematic review. The reviewer was not at any screening level blinded to the authors or other information about the publication when assessing the studies. Only recent publications in five languages were included in the literature search because of resource limitations. While it is possible that the resulting search may have excluded relevant studies, this does not seems likely because the reviewer inspected 14 related literature reviews, which had no publication year- or language restrictions (see literature list) and no other behavioural HIV/STI outcome evaluations for MSM in Europe were identified.

Conclusion

The main finding of this systematic review is that despite 30 years into the HIV epidemic, rigorous outcome evaluations of any form of behavioural HIV/STI intervention for MSM in Europe are scarce. Evaluating the effectiveness of interventions poses significant challenges to the scientific community, but if one were to have evidence-based policies and practices to prevent HIV/STIs among MSM in the future, additional behavioural interventions with accompanying outcome evaluations should be implemented. Interventions should target both individuals, groups, and communities, strive for biological outcomes alongside behavioural measures, and include multiple follow up assessments. Evidence from this systematic review demonstrates that it is possible to successfully conduct rigorous HIV/STI prevention studies for MSM in Europe which meet these criteria, they indicate sexual risk behaviour change, and such studies should to a greater extent become part of a comprehensive continuum of behavioural and biomedical HIV/STI prevention and care.
        
Acknowledgements
The study author is grateful to Dr van Kestern and her colleagues for making their study available for this systematic review. The author gratefully acknowledges the expert contributions provided by information search specialist Sari Ormstad, statistician Jan Odgaard-Jensen, and senior researcher Geir Smedslund at the Norwegian Knowledge Centre for the Health Services in the preparation of this systematic review. The systematic review would not be possible without the support of Dr Ronny Tikkanen.

Financial support
The project was funded by the European Centre for Disease Prevention and Control, publication reference: OJ/2009/06/03-PROC/2009/022. No financial or other conflict of interest is reported by the author.

References [16-23] are to the studies included

1. European Centre for Disease Prevention and Control/WHO Regional Office for Europe. (2008). HIV/AIDS surveillance in Europe 2007. Stockholm, European Centre for Disease Prevention and Control.
2. Joint United Nations Programme on IV/AIDA (UNAIDS) and World Health Organization (WHO) 2006. 2006 AIDS epidemic update: special report on HIV?AIDS: December 2006. Geneva: UNAIDS/WHO; 2006. Available from: http://data.unaids.org/pub/EpiReport/2006/2006_EpiUpdate_en.pdf
3. Catania JA, Morin S F, Canchola J, Pollack L, Chang J, Coates T J. (2000). U.S. priorities - HIV prevention. Science. 2000;290(5492):717
4. Elford J, Sherr L, Bolding G, Serle F, Maguire M. Peer-led HIV prevention among gay men in London: process evaluation. AIDS Care. 2002;14(3):351-60.
5. Johansen JD, Smith E. Gonorrhoea in Denmark: high incidence among HIV- infected men who have sex with men. Acta Derm. Venereol. 2002;82(5):365-8.
6. MacDonald N, Dougan S, McGarrigle CA, Baster K, Rice BD, Evans BG, et al. (2004). Recent trends in diagnosis of HIV and other sexually tranmitted infections in England and Wales among men who have sex with men. Sexually Transm Infect. 2004;80(6):492-7.
7. Simms I, Fenton KA, Ashton M, Turner KM, Crawley-Boevey EE, Gorton R, et al. The re-emergence of syphilis in the United Kingdom: the new epidemic phases. Sexually Transm Dis. 2005;32(4): 220-6.
8. *Herbst JH, Sherba RT, Crepaz N, Deluca JB, Zohrabyan L, Stall RD, et al. A meta-analytic review of HIV behavioral interventions for reducing sexual risk behavior of men who have sex with men. J Acquir Immune Defic Syndr. 2005;39(2):228-41.
9. Conant M. Losing the war on AIDS. J Am Acad Dermatol. 2004;51(1 suppl):S47-8.
10. Elford J, Hart G. If HIV prevention works, why are rates of high-risk sexual behaviour increasing among MSM? AIDS Educ Prev. 2003;15(4):294-308.
11. *Herbst JH, Beeker C, Mathew A, McNally, Passin WF, Kay LS, et al. The effectiveness of individual-, group-, and community-level HIV behavioural risk-reduction interventions for adult men who have sex with men: a systematic review. Am J Prev Medicine. 2007;32(4 Suppl):S3867.
12. *Johnson WD, Diaz RM, Flanders WD, Goodman M, Hill AN, Holtgrave D, et.al. Behavioral interventions to reduce risk for sexually transmission of HIV among men who have sex with men. Cochrane Database of Syst Rev. 2008;(3):CD001230.
13. Higgins JP, Green S. (editors). Cochrane handbook for systematic reviews of interventions. Version 5.0.1 (updated September 2008). The Cochrane Collaboration, 2008. Available from: http://www.cochrane-handbook.org/
14. Sackett DL, Straus SE, Richardson WS, Rosenberg W, Haynes RB. Evidence based medicine: how to practice and teach EBM. New York: Churchill Livingstone,2000.
15. Cochrane Collaboration. Review Manager (RevMan) [Computer program]. Version 5.0. Copenhagen: The Nordic Cochrane Centre. The Cochrane Collaboration, 2008.
16. Amirkhanian YA, Kelly JA, Kabakchieva E, Kirsanova AV, Vassileva S, Takacs J, et al. (2005). A randomized social network HIV prevention trial program with young men who have sex with men in Russia and Bulgaria. AIDS. 2005;19(16):1897-905.
17. Elford J, Bolding G, Sherr L. Peer education has no significant impact on HIV risk behaviours among gay men in London. AIDS. 2001;15(4):531-8.
18. Supplemental publication to Elford et al. (2001): Elford J, Sherr L, Bolding G, Maguire M.,  Serle F. Peer-led HIV prevention among gay men in London (the 4 gym project). Intervention and evaluation (pp. 205-230) in J. Watson & S. Platt (2000) Researching health promotion. London: Rutledge.
19. Flowers P, Hart GJ, Williamson LM, Frankis JS, Der GJ. Does bar-based, peer-led sexual health promotion have a community-level effect amongst gay men in Scotland? Int J STD  AIDS 2002;13(2):102-8.
20. Supplemental publication to Flowers et al. (2002): Flowers P, Frankis J, Hart G. Evidence and the evaluation of a community-level intervention (pp. 102-24), in J. Watson. Platt S. (2000) Researching health promotion. London: Rutledge.
21. Harding R, Bensley J, Corrigan N, Franks L, Stratman J, Waller Z, Warner J. Outcomes and lessons from a pilot RCT of a community-based HIV prevention multi-session group intervention for gay men. AIDS Care. 2004;16(5):581-5.
22. Imrie J, Stephenson JM, Cowan FM, Wanigaratne S, Billington AJ, Copas AJ, et al. (A cognitive behavioural intervention to reduce sexually transmitted infections among gay men: randomised trial. BMJ. 2001;322(7300):1451-6.
23. van Kesteren NM, Kok HJ, van Brukelen G, Kok G. Evaluation of a self-help and motivational enhancement intervention to promote sexual health in HIV-positive men who have sex with men. [In submission]
24. Kelly JA. St Lawrence JS, Stevenson LY, Hauth AC, Kalichman SC, Diaz YE, et al. (1992). Community AIDS/HIV risk reduction: the effects of endorsements by popular people in three cities. Am J Public Health. 1992;82(11):1483-9.
25. Kelly JA, Murphy DA, Sikkema KJ, McAuliffe TL, Roffman RA, Solomon LJ, et al. Randomised, controlled, community-level HIV-prevention for sexual-risk behaviour among homosexual men in US cities. Lancet. 1997;350(9090):1500-5.
26. Kegeles SM, Hays RB, Coates TJ. The MPowerment Project: a community-level HIV prevention intervention for young gay men. Am J Public Health. 1996;86(8):1129-36.
27. Kegeles SM, Hays RB, Pollack LM, Coates TJ. Mobilizing young gay and bisexual men for HIV prevention: a two-community study. AIDS.1999;13(13):1753-62.
28. O'Leary A, DiClemente RJ, Aral SO. Reflections on the design and reporting of STD/HIV behavioral intervention research. AIDS Educ Prev. 1997;9 (suppl 1):1-14.
29. *Kalichman SC, Carey MP, Johnson BT. Prevention of sexually transmitted HIV infection: A meta-analytic review of the behavioural outcome literature. Annals of Behavioral Medicine. 1996;18, 6-15.
30. Gold RS. Evaluation of interventions for gay men: two desiderata. AIDS Care. 2002;14(3): 425-9.
31. *Stephenson J, Imrie J, Sutton SR. Rigorous trials of sexual behaviour interventions in STD/HIV prevention: what can we learn from them? AIDS. 2000;14 Suppl 3:S115-124.
32. *Oakley A, Fullerton D, Holland J. Behavioural interventions for HIV/AIDS prevention. AIDS 1995;9(5):479-86.
33. Armstrong R, Waters E, Doyle J. (editors) Chapter 21: Reviews in health promotion and public health. In JPT Higgins & S Green (editors). Cochrane Handbook for Systematic Reviews of Interventions. The Cochrane Collaboration, 2008.
34. Elford J, Bolding G, Sherr L. High-risk sexual behaviour increases among London gay men between 1998-2001: What is the role of HIV optimism. AIDS, 2002;16(11):1537-44.
35. *Kegeles SM, Hart GJ. Recent HIV-prevention interventions for gay men: individual, small group, and community based studies. AIDS. 1998;12 Suppl A:S209-15.
36. Stephenson J, Imrie J. Why do we need randomised controlled trials to assess behavioural interventions ? BMJ. 1998;316(7131):611-3
37. Newman J C, Des Jarlais DC, Turner CF, Gribble J, Cooley P, Paone D. (2002). The differential effects of face-to-face and computer interview methods. Am J Public Health. 2002; 92(2):294-7.

References below and those marked with an asterisk in the above reference list are to literature reviews which literature were screened for leads

• Choi KH, Coates TJ. (1994). Prevention of HIV infection. AIDS. 1994;8(10):1371-89.
• Ellis S, Barnett-Page E, Morgan A, Taylor L, Walters R, Goodrich J. HIV prevention: A review of reviews assessing the effectiveness of interventions to reduce the sexual risk of HIV transmission. 2003. Retrieved 20 July, 2009, from http://www.nice.org.uk/niceMedia/documents/HIV_review.pdf
• Johnson BT, Carey MP, Chaudoir SR, Reid AE. Sexual risk reduction for persons living with HIV. Research synthesis of randomized controlled trials, 1993 to 2004. J Acquir Immune Defic Syndr.2006;41(3):642-50.
• Oakley A, Oliver S, Peersman G, Mauthner M. Review of effectiveness of health promotion interventions for MSM. 1996. London: London University Institute of Education, EPPI Unit.
• Vergidis RI, Falagas ME. Meta-analyses on behavioral interventions to reduce the risk of transmission of HIV. Infectious Disease Clinics of North America, 2009;23:309-314.
• Weinhardt LS, Carey M, Johnson B, Bickham NL. Effects of HIV counselling and testing on sexual behaviour: A meta-analytic review of published research, 1985-1997. Am J  Public Health. 1999;89(9):1397-1405.
• Wolitski R J, McGowan R, Higgins D, Jorgensen C. (1997). The effects of HIV counselling and testing on risk-related practices and help-seeing behavior. AIDS Educ Prev. 1997;9 (s3 Suppl):52-67. 

Additional references

• Fennema JS, Cairo I, Coutinho RA. [Substantial increase in gonorrhoea and syphilis among attenders of the Amsterdam Sexually Transmitted disease clinic].Ned Tijdschr Geneeskd. 2000;144(13):602-3.