Accessibility of Municipal Websites

1. The Information Accessibility Observatory at Discapnet

In 2004, the Discapnet Project, co-financed by the ONCE Foundation of Spain and the European Regional Development Fund (ERDF), started up the Info-accessibility Observatory to generate and publicize information on website accessibility, by both analyzing specific sectors as well as comparing across sectors to monitor their development over time. The reports on the accessibility of university websites and the general State Administration’s e-services are a result of that line of work [1].

The purpose of the reports by the Discapnet Info-accessibility Observatory is to inform on and highlight not only the degree of compliance with current norms, but also the good features and main obstacles on the websites, including assessments from user feedback. It is hoped that a deeper understanding of the strengths and weaknesses identified by web experts and users alike will lead to a better understanding in webmasters and web designers of what constitutes web accessibility, what tools and services are available. Greater interaction, then, will lead to improving the ever-increasing usefulness of such sites.

The Observatory employs an innovative methodology designed by Technosite. The methodology follows on the W3C/WAI [2] guidelines for combining the technical analysis of accessibility with an assessment of the usability and accessibility based on feedback from the users’ own experiences.

• Evaluation of the technical aspects takes the Web Content Accessibility Guidelines 1.0 from the W3C/WAI web [3] as a framework and synthesizes them in a set of indicators applied to a sample of web pages by website. Verification is carried out by professionals running automatic and manual checks.
• Assessment is done by a panel of users of varying functional abilities who undertake a set of tasks and then answer a questionnaire on their perception of each site. This procedure helps identify both the barriers and the aids in using each site, check the “information architecture” (i.e., how the content is organized, how to navigate around the site, perform searches, etc.) as well as determine how individual users interact with the websites.

The combination of both approaches provides information that is relevant, systematic, and qualified regarding accessibility in the sectors subject to this study. With it, insight is gained into how to correct and improve the Internet medium.

Back to the top

2. Selection of the sample

This study has chosen to examine the official websites of 18 city governments in Spain, all of which are provincial capitals. They were selected such that there would be one from each Region plus one of the Autonomous Cities. The sample was also chosen to include a balanced representation of large, medium, and small city governments according to population. The websites under analysis were as follows:

1. Madrid.
2. Barcelona.
3. Málaga.
4. Murcia.
5. Palma de Mallorca.
6. Sta. Cruz de Tenerife.
7. Oviedo.
8. Pamplona.
9. Santander.
10. San Sebastián.
11. Castellón de la Plana.
12. Albacete.
13. Logroño.
14. Cáceres.
15. Lugo.
16. Ceuta.
17. Soria.
18. Teruel.

On each of the 18 websites under study, we examined 5 representative pages of the following characteristics:

1. Home page.
2. Standard page.
3. A page with tables.
4. A page with forms.
5. Results of doing a search (the search engine did not work on the Soria and Teruel sites).

Back to the top

3. Technical evaluation of web accessibility

In order to evaluate the technical aspects of accessibility, twelve aspects were used which synthesize most of the Web Content Accessibility Guidelines on the W3C/WAI 1.0 website (WCAG 1.0) corresponding to levels A and AA. The experts at Technosite, who led the study, consider the WCAG criteria able to provide a synthetic view closely matching the degree of accessibility of websites and web-based services. Included most are priority 1 aspects, though in some cases those of priority 2 were also used. The points of verification, itemized further on in the section on the analysis of the results, are as follows:

1. Validation of W3C technologies (priorities 1 and 2 in WCAG 1.0).
2. Frames (priorities 1 and 2 in WCAG 1.0).
3. Forms (priorities 1 and 2 in WCAG 1.0).
4. Text-only alternatives to multimedia elements (priority 1 in WCAG 1.0).
5. Headers (priority 2 in WCAG 1.0).
6. Units in Style Sheets (priorities 1 and 2 in WCAG 1.0).
7. Understandable links (priority 2 in WCAG 1.0).
8. Contrast (priority 2 for images in WCAG 1.0).
9. Semantic use of colors (priority 1 in WCAG 1.0).
10. Alignment of content in tables for layout (priority 2 in WCAG 1.0).
11. Data tables (priority 1 in WCAG 1.0).
12. Scripts (priority 1 in WCAG 1.0).

The technical analysis of accessibility was carried out on the sample in the month of May 2006.

Back to the top

4. Classification by Percent Success on the Technical Evaluation of Accessibility

To compare how the city hall websites fared on the technical evaluation of web accessibility, Table 1 ranks each city hall’s overall score in percentages, from highest to lowest.

It should be noted that successful scores may not fully reflect the global accessibility of a site: not all of the criteria for verification as set forth in WCAG 1.0 were used in the present study.

Even a cursory reading shows that the scores on the technical evaluation leave much to be desired. The overall rate of successful compliance for the whole set barely reaches the 25% mark. While this sets them above the scores from the Regional Government Websites study, they fall sharply below the scores from the e-services on General State Administration websites.[4]

Table 1.- Classification of Spanish provincial capital city websites, by % success in correctly applying the criteria analyzed in the technical evaluation of web accessibility.

Websites	% Success
Pamplona	57.14
Ceuta	56.52
Castellón de la Plana	52.27
Barcelona	44.19
Murcia	31.11
Lugo	28.26
Madrid	24.49
Logroño	20.83
Teruel	20.51
Palma de Mallorca	20.45
San Sebastián	18.18
Málaga	17.39
Albacete	13.46
Total	13.33

Individually, only the top three websites scored higher than 50% on the technical analysis. These three were Pamplona (57.14%), Ceuta (56.52%), and Castellón de la Plana (52.27%). In contrast, there were 12 sites that did not even reach a 25% success rate. Among them, and at the bottom of the list, was the Oviedo municipal website, which scored a mere 1.89% for having only one criterion for accessibility verified on only one of its pages.

Graph 1 displays how each of the 12 criteria for accessibility scored on the technical evaluation. The highest-scoring criterion was that of alignment in tables for layout (65.91% successful application). Even though it is recommended to avoid tables for layout purposes, their use was found to be widespread: indeed, they were found on each and every page in the sample. Furthermore, though table alignment scored higher than the other criteria, we detected a considerable number of pages with errors of that kind (30 pages out of 88 under review). The widespread use of tables for layout purposes instead of other techniques based on style sheets may be due to the fact that Content Management Systems (CMS) often make use of such tables. None of the remaining criteria attained a 50 % score of successful verification. At a distant 2nd place, scoring almost 28 percentage points lower, is the criterion of correct use of color contrast in images (38.10% successful application). Ranking near the 33 percent success mark were understandable links (34.09%) and scripts (32.25%).

[D]

The remaining criteria scored lower than 30% for successful compliance. In the 10-30% success range were frames (25.00%), text alternatives for images (24.42%), valid code (12.50%), and forms (11.54%). Under the 10% success mark were another 4 criteria: data tables, at 9.52%; headers, at 7.95%; style sheets, at 1.15%; and semantic use of color, at 0.00%. Semantic use of color was sampled on only 4 pages, which in and of itself reflects a poor showing given that some people may need to rely heavily on it for accessibility.

Back to the top

5. Assessment of Accessibility from User Feedback

Just as WAI and Technosite suggest, scores on technical evaluations should be accompanied by feedback from users themselves. The aim is to evaluate each website’s ability to be used by a variety of real people of different physical abilities to provide an imminently practical point of view. This approach is based on the ISO 924 quality standard which defines usability as "the effectiveness, efficiency, and satisfaction with which a product lets specific users reach specific objectives in a specific context of use."

User-centered assessment (and design) also makes it possible to check the "information architecture" of the website, i.e., how well the information is organized (classification and labeling), how it is retrieved (systems for surfing, searching, and guiding), and how users are to interact with it (definitions of the interaction process with the system).

The customary technique for registering the experience is a User Feedback Test, applied here as a self-administered questionnaire. It consisted of a set of tasks and follow-up questions to be answered by the user following previously given instructions. Users could thus complete the questionnaire on their own, without the need for proctors or interviewers. The questions were on simple, common tasks to be carried out on the websites under study.

The users were given the following instructions on how to perform the assessment of each of the 18 websites:

1. Surf the web and find the indicated sites.
2. Perform 5 tasks for each web service under evaluation.
3. Write down the answer to each task and the time and steps it took to accomplish it.
4. Note any defeats when accessibility problems caused the user to give up before completing the requested task.
5. Answer a satisfaction survey of 10 multiple choice questions (each having 4 alternatives) and explain why for each answer.

User profiles are shown in Table 2. They include people with a wide range of functional characteristics, a variety of technical aids, and differing degrees of skill a using the internet

Table 2.- User profiles from the assessment.

Impairment.	Technical aids employed	Internet skill level
Greatly reduced movement.	Use of head wands.	Medium.
Deafness.	None used.	Medium.
Blindness.	JAWS 5.0 screen reader.	Medium.
Severe visual impairment.	Screen magnifier and high contrast.	Advanced.
Blindness.	JAWS 6.20 screen reader.	Medium.
No notable impairment.	None used.	Advanced.

The results were interpreted and tabulated by an accessibility expert at Technosite.

The purpose of the tabulation is to merge and weigh the user feedback with the objective scores from the task test done for each website in terms of effectiveness, efficiency, and satisfaction.

The user feedback tests were carried out in May 2006.

In addition to the self-administered user feedback tests, a discussion group was also held for the users to share their experiences.

Back to the top

6. Results of the User Feedback Assessment

Let us look first at the number of successes, errors, and defeats made by the users who undertook the assessment for each website in the sample.

Table 3 shows the scores of the 6 users for each of the 5 tasks carried out on the 18 websites sampled in the study.

Of a total of 540 tasks undertaken by the 6 users on the 18 websites of the sample, 399 (73.89%) were successfully accomplished, while 64 (11.85) were erroneous. The number of defeats—giving up the task due to accessibility or usability issues—came to 77 (14.26%), making it the lowest percentage of defeats on any of the Observatory studies to date.

Table 3.- Successes, errors, and defeats on the user assessment tasks, in absolutes and total percent.

Websites	Success	Error	Defaul
Teruel:	29	1	0
San Sebastián:	27	2	1
Castellón de la Plana:	27	3	0
Palma de Mallorca:	25	3	2
Murcia:	25	3	2
Oviedo:	24	2	4
Albacete:	24	6	0
Málaga	23	3	4
Soria:	22	2	6
Madrid:	22	2	6
Ceuta:	21	6	3
Cáceres:	21	3	6
Barcelona:	21	8	1
Logroño:	20	1	9
Pamplona:	19	5	6
Lugo:	19	6	5
Santa Cruz de Tenerife:	18	5	7
Santander:	12	3	15
Total	399	64	77
%	73.89	11.85	14.26

An examination of the results displayed in Table 3 leads to the following observations:

1. The percentage of defeats in this study is the lowest found on any study done by this Observatory so far.
2. The highest-ranking website in this section belongs to the City of Teruel, where 29 of the 30 tasks were completed correctly and no defeats were recorded. Two other websites, from Castellón de la Plana and Albacete, also registered no defeats.
3. At the other extreme, the highest number of errors on tasks carried out fell to Barcelona’s website, at 8 errors.
4. The website registering the most defeats (15) corresponds to that of the City of Santander, which also had the fewest number of successfully completed tasks.

Although the results of this study cannot provide a direct link between the complexity of a website (sites from more populous cities tend to have more pages, more online services, and more complex contents), we have noticed that complexity may be an indirect cause of some of the unfavorable results. For example, based on the remarks from the users’ feedback, we have found that bilingual websites may not have handled the various official languages properly, resulting in unwanted changes from one language to another, or when searches can only be performed in one of the languages. Errors are caused as a result, as has happened on the Barcelona and Lugo sites. However, the high number of errors and defeats on the monolingual Santander site can only be explained by issues with usability.

Next we shall look at the scores for each city hall website in the study based on the ad hoc questionnaire filled out by all of the users after completing the assigned tasks.

Each user was asked to answer 10 questions on aspects specifically related to his/her experience in undertaking the tasks. The questions all involved accessibility and usability issues, and were to be rated on an increasing scale of satisfaction by choosing one of the 4 statements provided. Additionally, users were requested to comment on the reason(s) underlying each answer. The questions on the questionnaire were as follows:

1. Were you able to accomplish the task as requested?
2. Were you able to move around the website?
3. What was your overall impression of the website?
4. Did correct labeling with text alternatives for graphics help you understand the contents better?
5. Did the written text on the site help clear up how to accomplish the tasks at hand?
6. What is your opinion of how the site is organized overall?
7. Did the link names clearly indicate their targets?
8. Do you think the page titles were helpful indicators of the page contents?
9. Would you use this website for its ease at finding and getting the information and services it provides?
10. Do you think it is a good website?

Graph 2 shows the resulting scores converted into percentages for the set of 6 users who took the assessment. It should be recalled that 5 of the 6 users had some degree of functional impairment whereas one did not.

[D]

An examination of the data in Graph 2 reveals the following highlights:

1. The mean percent score on the user satisfaction survey was 55.77%. this average is notably higher that that of the study on Spanish regional government websites (47.03%) but falls short of the average on General Public Administration sites (60%).
2. The highest-rated website for user satisfaction was that of Castellón de la Plana (69.44%).
3. Second on the list is Teruel (67.22%), followed closely for third place by Madrid, Pamplona, and San Sebastián, all tied at 65%.
4. In the 50-60% range there are 90 city websites: Barcelona and Ceuta (58.89% each), Málaga (58.33%), Murcia (57.78%), Albacete and Oviedo (57.22% each), Cáceres (55%), Palma de Mallorca (52.78%), and Soria (52.22%).
5. Four city hall websites failed to reach the 50% mark: Lugo (45.56%), Santa Cruz de Tenerife (45%), Logroño (42.78%), and, lagging at more than 12 percentage points below, Santander (30.56%).

By and large, almost all of the city hall websites under study received a rating of mild satisfaction from the users in out study. Moreover, the scores fall within a relatively narrow range of 39 percentage points, which would shorten to a 27-point spread if the bottom-ranking site were not included.

Back to the top

7. Combined Score Assessment

As has been witnessed on earlier Observatory studies on website accessibility [5], the scores from the user feedback survey were higher than the scores obtained on the technical evaluation.

Table 4.- Comparison of percent scores from both assessments, ranked highest to lowest in compliance or satisfaction.

Technical evaluation			User feedback
Website	%		Website	%
Pamplona	57.14		Castellón de la Plana	69.44
Ceuta	56.52		Teruel	67.22
Castellón de la Plana	52.27		Pamplona	65.00
Barcelona	44.19		Madrid	65.00
Murcia	31.11		San Sebastián	65.00
Lugo	28.26		Ceuta	58.89
Madrid	24.49		Barcelona	58.89
Logroño	20.83		Málaga	58.33
Teruel	20.51		Murcia	57.78
Palma de Mallorca	20.45		Oviedo	57.22
San Sebastián	18.18		Albacete	57.22
Málaga	17.39		Cáceres	55.00
Albacete	13.46		Palma de Mallorca	52.78
Santander	13.33		Soria	52.22
Cáceres	10.87		Lugo	45.56
Soria	10.81		Sta. Cruz de Tenerife	45.00
Sta. Cruz de Tenerife	10.00		Logroño	42.78
Oviedo	1.89		Santander	30.56
Average	25.00		Average	55.77

Table 4 offers a side-by-side view of the results from both the technical evaluation of compliance and the user feedback assessment comprising the study. Once again, each and every website scored higher on the satisfaction survey than on the technical analysis.

A comparison of the two tables reveals the following noteworthy information:

1. Three city hall websites scored better than 50% on both tests: Pamplona, Castellón de la Plana, and Ceuta. They are also the three that exhibit the smallest difference between both scores.
2. Four city hall websites failed to reach the 50% mark on either test: Lugo, Logroño, Santa Cruz de Tenerife, and Santander.
3. The City of Oviedo website shows the greatest spread, at more than 55 percentage points, between scores, and jumps from last place on technical compliance to a mid-range score on user satisfaction.

Graph 3 shows the combined scores from the two dimensions of our study. To understand the content displayed in the graph, the following points should be noted:

1. The vertical axis shows the percent satisfaction from the user feedback.
2. The horizontal axis shows the percent score from the technical analysis.
3. The graph displays a four-quadrant map reflecting accessibility (technical analysis) and usability (user satisfaction).
4. Reading clockwise, the upper left quadrant shows the most usable websites according to user feedback, but not very accessible according to the results of our technical analysis. The upper right quadrant shows the most usable and accessible sites. The lower right quadrant shows the websites that are not very usable but more accessible, while the lower left quadrant shows the websites that are neither usable nor accessible.
5. The diagonal crossing the graph from bottom left to upper right, denotes the point where both assessments would be if the site were equally accessible and usable. Scores above the line (which in this case are all of them) indicate sites considered more usable (user feedback) than accessible (technical evaluation). Under the diagonal (none in this case) would be the other way around. This reveals that all the scores in our study give a higher rating from user feedback than from their compliance to the norms of accessibility. In some cases such as that of the Oviedo municipal website, that difference can be quite large.
6. Three municipal websites (Pamplona, Castellón de la Plana, and Ceuta) lie in the upper right quadrant, where scores reflect greater usability and accessibility. However, the fact that they lie on the lower left side of the quadrant indicates that their success is only moderate.
7. The highest concentration of websites fall within the lower half of the upper left quadrant, which suggests that the websites overall (12 websites) are “discreetly usable” according to the users, but hardly very accessible according to their degree of technical compliance.
8. The websites of the cities of Lugo, Logroño, Santa Cruz de Tenerife, and Santander appear in the lower left quadrant displaying the worst results from both usability and accessibility. They stretch across the upper region of the quadrant, suggesting that they scored considerably worse on accessibility than on usability.
9. Again we point out the large difference between the scores on each analysis seen for the Oviedo municipal website. Its extremely poor score on compliance with criteria for accessibility (1.89%) sharply contrasts with the degree of satisfaction expressed by the users (57.22%). No clear explanation for such a large gap is evident. We may hazard that the accessibility problems may be ameliorated by the user’s skill and the use of compensatory strategies and devices.

Legend: This graph displays the combined scores from the technical analysis and the user feedback assessment on a quadrant table.

• Upper left (yellow): technical analysis below 50% and user satisfaction about 50%.
• Upper right (green): technical analysis and user satisfaction both above 50%.
• Lower left (red): technical analysis and user satisfaction both below 50%.
• Lower right (blue): technical analysis above 50% and user satisfaction below 50%.

The diagonal line crossing the table marks the dividing line above which fall sites scoring higher on the user feedback survey and below which are sites having higher scores on the technical evaluation.
The following list shows the abbreviations used and the percent scores obtained by each website on the technical analysis and user feedback, separated by a slash:

• AB: Albacete (13.46 / 57.22).
• BA: Barcelona (44.19 / 58.89).
• CC: Cáceres (10.87 / 55.00).
• CE: Ceuta (56.52 / 58.89).
• CP: Castellón de la Plana (52.27 / 69.44).
• LO: Logroño (20.83 / 42.78).
• LU: Lugo (28.26 / 45.56).
• MA: Madrid (24.49 / 65.00).
• MG: Málaga (17.39 / 58.33).

Back to the top

8. Conclusions.

The purpose of the studies carried out by the Discapnet Info-accessibility Observatory is to show the current state of accessibility of Web content. At the same time, they hope to provide information for improving websites and better matching them to the needs of their users. To that effect we present here our main conclusions from this study:

1. The municipal websites of the Spanish provincial capital cities analyzed in this study do not comply with the provisions set forth in the fifth additional disposition of Law 34/2002 on Services of the Information Society and E-Business. Indeed, the highest ranking website on the technical analysis barely reached 57% of the criteria for evaluation. The generalized incompliance is especially worrying considering that, even though the accessibility requirements were made know three years earlier and now several months since the law took effect (on December 31, 2005), there is still a great deal to be done. Some of the websites make accessibility statements that we have been unable to verify. Other have expressed their plans to renovate their websites to make them more accessible. Whatever the case, the data from our study reveal that minimum standards have yet to be reached.
2. As on earlier studies, the assessment from users was more positive than the results of the technical evaluation. On all of the websites under study, the percentage obtained on the satisfaction assessment from user feedback outscored the results of correct application of the criteria on the technical analysis. The websites with the best technical compliance were the ones to show the least variance with their user assessment ratings (Pamplona, Castellón de la Plana, and Ceuta). In other cases, the gap between the two scores is considerable, with the extreme being the Oviedo municipal website’s spread of nearly 55 percentage points. The explanation behind such differences can most likely be found in the adeptness which users with some degree of functional limitation learn to develop to overcome barriers, as well as the use of specialized devices and programs for that purpose.
3. Out of the twelve criteria used for the technical analysis of the sample, the only one to receive a percentage score above 50% was table alignment (at 65.91%). The rest of the criteria fell far short of the mark, with three in the 30-40% range (color contrast 38.10%, scripts 34.09%, and understandable links 32.95%). All the remaining sites scored below 25%.
4. Special mention should be made of the still frequent use of frames for layout purposes and their poor fit with the guidelines for accessibility: only 25% of the pages using frames did so correctly.
5. We also note the low success rate for the criterion of code validation (at 12.50%), particularly since code can be validated automatically and with very precise indications on how to solve any errors that may have occurred.
6. In addition to the abovementioned, it is important to note that images without alternative text (success: 24.42%) restrict and limit web use by the blind; forms (success: 11.54%) should observe the criteria for accessible design if they are to be understood and filled out by users surfing with specialized devices; data tables (success: 0.52%) not designed for accessibility are an encumbrance to those who cannot see them visually; and that headers (success: 7.95%) included and correctly used can greatly facilitate navigation and content comprehension.
7. It is of great concern to us that the use of style sheets following criteria for accessibility scored a mere 1.15% on our technical analysis, since they control how the page is to be displayed to the web visitors, and should thus allow users to fit the display to meet their own needs.
8. Finally, the semantic use of color (success: 0%) was hardly even attempted, which forgoes the chance to help visitors who need clues to understand the contents fully. We point out that the few times semantic used was tried were unsuccessful at complying with the guidelines for accessibility.

To end this report, we wish to remind webmasters that the provisions set forth in the fifth disposition of the Law on Services of the Information Society and E-Business are fully in effect. As of January 1, 2006, all Public Administrations are obliged to make all the information they put on the net accessible. Public Administrations have had sufficient time to take the necessary steps to modify and update their sites to comply with the provisions of the law. Thus, there is no excuse for making it unduly cumbersome for some citizens to access web-based public services. The necessary resolutions must be made immediately so that this situation is not protracted any longer. Fortunately, there are now a number of businesses offering web-accessible design and maintenance services. There is currently also a sufficient number of training centers where Administration personnel who design, develop, and maintain web sites can learn to do so in full compliance with the criteria for accessibility.

Back to the top

---

Note: All the external URL´S are opened in a new window.

[1]These papers are available at:
http://www.discapnet.es/Discapnet/Castellano/Observatorio_infoaccesibilidad/observatorio04.htm

http://www.discapnet.es/Discapnet/Castellano/Observatorio_infoaccesibilidad/observatorio05.htm

[2]W3C/WAI: Web Accessibility Initiative of the World Wide Web Consortium. For further information, see http://www.w3.org/WAI.

[3]Available in English (http://www.w3.org/TR/WCAG10/), on the W3C/WAI website, and in Spanish (http://www.technosite.es/accesibilidad/recursos/documentos/index.html) on the Technosite website.

[4]Prior studies revealed average scores of 34.31% for General State Administration websites, and 24.48% for Regional Government websites. Further information on both studies is available at :
http://www.discapnet.es/Discapnet/Castellano/Observatorio_infoaccesibilidad/

[5]These studies are available online at:
http://www.discapnet.es/Discapnet/Castellano/Observatorio_infoaccesibilidad/

---

Back to the top