Done by:
Accessibility of e-services of the General State Administration.
We are approaching the deadline set forth in the Law on E-business and the Information Society to make public services on the web accessible to the disabled and the elderly. By December 31, 2005, all such public services must comply with “generally acknowledged criteria for accessibility to content.” The purpose of this study is to provide an overview on the “state of the art” by analyzing a sample of services and drawing conclusions that not only express the existent problems, but also will enable measures to be adopted for attaining the goal of making public websites accessible to EVERYONE.
Noteworthy points include our perception of overall improvement in the services under study, though still a far cry from reaching minimal standards of acceptability. Despite a few minor and easily solvable drawbacks, Social Security’s service for obtaining tax records sets an admirable example for other heads of maintenance and development of public websites to follow.
Table of Contents:
In 2004, Discapnet, a project co-financed by the ONCE Foundation of Spain and the European Regional Development Fund (ERDF), started up the Info-accessibility Observatory to generate and publicize information on website accessibility, by both analyzing specific sectors as well as comparing across sectors to monitor their development over time. The early results were published in a report on the accessibility of Spanish university websites [1].
The Observatory employs an innovative methodology designed by Fundosa Teleservicios. The methodology follows on the W3C/WAI [2], guidelines for combining the technical analysis of accessibility with an assessment of the usability and accessibility based on feedback from the users’ own experiences.
The combination of both approaches provides information that is relevant, systematic, and qualified regarding accessibility in the sectors subject to this study. With it, insight is gained into how to correct and improve the Internet medium.
In an attempt to integrate a variety of requirements while covering a broad range of services provided by the General State Administration, the following websites were selected to make up the sample for this study [4] (Note: All external URL´s open in a new window):
In order to evaluate the technical aspects of accessibility, twelve aspects were used which synthesize most of the Web Content Accessibility Guidelines on the W3C/WAI 1.0 website (WCAG 1.0) corresponding to levels A and AA. The experts at Fundosa Teleservicios, who led the study, consider the WCAG criteria able to provide a synthetic view closely matching the degree of accessibility of websites and web-based services. Included most are priority 1 aspects, though in some cases those of priority 2 were also used. The points of verification, itemized further on in the section on the analysis of the results, are as follows:
Upon analyzing each aspect, a data sheet is filled out in which compliance is marked as 1, while incompliance is marked as -1. A value of 0 is assigned when the aspect is not applicable.
However preliminary the web accessibility guidelines, they require carrying out manual or heuristic tests, which even the automated tools such as the Web Accessibility Test (TAW) [5] and the WCAG 1.0 point out. It is the only way to check compliance of the basic qualitative indicators such as understandable links, TITLE and ALT label appropriateness.
Web pages were examined using the most widely used browsers: Explorer 6.0, Netscape 7.0, and Opera 7.2. A few browser characteristics (such as scripts and CSS support) were disabled in order to review some of the points being tested.
The technical analysis of accessibility was carried out on the sample web pages during the first two weeks of March 2005.
As WAI suggests and Fundosa Teleservicios firmly concurs, the results of the technical evaluation on accessibility should be complemented with a consideration of the user’s own experience. The purpose is to assess each web-based public service’s ability to be used by persons with differing limitations from a strongly practical point of view.
The standard way to record the user’s feedback is by means of a User’s Test. For this study, a self-administered questionnaire format was chosen. This type of test consists of a series of tasks and questions answered by the users themselves after receiving prior instructions so that they may carry out the tasks on their own, without the presence of an outside observer or interviewer. Questions were asked regarding simple tasks that are commonplace in the websites visited.
The sample profile of the users, presented in Table 1, includes persons with different types and degrees of functional limitations, various kinds of technical aids, and differing degrees of technical mastery.
| Functional limitation | Technical aid employed |
|---|---|
| Highly reduced mobility in hands | Trackball; voice recognition program |
| Deafness | None used |
| Blindness | JAWS screen reader |
| Vision deficiency | Zoom Text screen magnifier |
| No discernible limitation | None used |
The results were interpreted and tabulated by an accessibility expert from Fundosa Teleservicios. The tabulation compares and weighs the users’ assessments against the objective results of the tasks specific to each service in terms of efficiency, effectiveness, and satisfaction.
The user feedback surveys were carried out in the second half of March 2005.
To complement the self-administered user feedback surveys, the users also took part in a discussion group to share their experiences.
This section gives general remarks on the results obtained on each of the twelve technical criteria used in the technical evaluation of the web accessibility of the seventy-four pages reviewed from the fifteen online General State Administration services under study.
It should be pointed out that the web pages of the services studied here undergo frequent changes. Therefore, the results presented here refer exclusively to their state during the testing period.
Priorities 1 and 2 in WCAG.
This section establishes that both the HTML code used in the web pages and the Style Sheet Code must be expressed correctly and validated by formal grammar, namely, in compliance with HTML and CSS2.6.
The results from ten out of the fifteen services studied were completely negative: errors were found in the HTML and CSS encoding on every single page. On the other hand, noteworthy among the five pages that did in fact obtain positive results was the Ministry of Public Administration’s website section on public job posts, in which every page successfully passed the validation test. Also of note were the results from the analysis of the Red.es website and that of Social Security, where 4 of the 5 pages analyzed were validated successfully. BOE, INE, and ID+Passport were the only other sites to obtain passing scores.
Priorities 1 and 2 in WCAG.
If frames are used, they should always be named in such a way that a user who can only access the text on the pages knows what they contain and how they are interrelated. Alternatives should also be provided for users whose browser does not support frames.
The sample pages were analyzed to see whether or not frames were used and, if so, whether they had fittingly descriptive name or title tags as well. The names and titles were further judged on their helpfulness for the user.
At nine of the fifteen web-based services under study, frames were used in the layout design of their pages (22 out of a total of 74 sample pages used frames, nearly 30%). The Ombudsman site was the only one whose pages all used frames, and the four pages in the sample did so correctly. Two other services—Public Administration and Social Security—made proper use of them on the single page that had frames. Five sites—CERES, AEAT, ID+Passport, INE, and Post Office—used frames incorrectly on every page that had them. Adminstracion.es used frames correctly on only one of their three pages; the other two pages used them incorrectly.
On the pages examined at the Ombudsman website, the frame for browsing options used both a name (“Tools”) and a title (“Toolbar”). Though we gave them a passing score, the choice did not seem fully suitable. It would have been clearer to call the “Browsing” and “Browsing bar” respectively.
Priorities 1 and 2 in WCAG 1.0.
Some users may have trouble dealing with forms because they are not sure what information to enter or choose in each field. That situation is most likely due to improper structuring of the form or an incompatibility with Javascript in some web browsers.
Further difficulty ensues if movement through the form with the tab key does not match up with the order of the fields in the form, or that the fields in lengthy forms are not grouped into orderly groups by concept or topic.
Fourteen of the services in this study used forms somewhere on the pages analyzed in the site (MAP being the only one to have none at all). A total of 42 pages (56.76% of all pages sampled), however, did use forms. The ones that worked the best were found on the Social Security website, where all five pages in the sample used forms that were correctly made in terms of accessibility. The ID+Passport service worked well, in which 3 of the 4 pages with forms were designed correctly. The only other service earning a passing score on some of its pages was the Post Office website, where, out of the 6 pages with forms, 2 passed the evaluation. The remaining 11 services failed validation of their pages in terms of form design.
Priority 1 in WCAG 1.0.
There are users who, for different reasons, are unable to see images: the blind, users of text-only browsers, those who stop downloads of pictures because their internet connection is slow, etc. For them, it is essential to be given an alternative to the images depending on their importance. For example, if no alternative text is available, both the screen readers used by the blind to access the net as well as text-only web browsers will only reveal the path of where the image is stored, which can become annoying for the user.
The vast majority of the sampled pages (71 out of 74, thus 95.95%) contained images. Social Security and MAP scored the highest in this regard, since all their pages in the study had the images correctly labeled. Also noteworthy were the Red.es and ID+Passport websites, where only one page on each site showed any error. At the other extreme were CERES, AEAT, BOE, DGT, INEM, ICEX, and the Ombudsman, all of whose pages with images were incorrectly tagged.
Priority 2 in WCAG.
An essential feature to have on any website is its headers (also known as “section titles”) to show how information is structured on each web page. These headers must also be named to fit the level of depth appropriately. For example, a level one header <h1> should not be followed by a level 3 header <h3>.
Our analysis of headers, then, examined the sample pages to check for both the presence of headers as well as their correctness.
Only one of the pages we analyzed (on the CERES website) failed the headers analysis completely. On the other hand, the Social Security website once again scored the highest, with correct use of headers on all five pages in the sample. Red.es failed on one of its pages but used them correctly on the other four, giving it an overall passing score. Widespread mistakes were found on a total of 10 different sites, while on three others (MAP, BOE, and AEAT), successful use was limited to only some of their pages.
Priorities 1 and 2 in WCAG 1.0.
Some people may need to be able to change the text size in order to access a website. The ability to change the font size requires using relative units of either em or percentage (%). Style sheets, however, should only use relative units, never absolute units.
Nevertheless, some web browsers can not interpret style sheets. Therefore, web developers should check that the content on the page can be interpreted correctly even without using style sheets.
All of the pages reviewed in our sample used cascading style sheets (CSS) for their layout on screen. The Ombudsman site was the only one to comply with the criteria on all four pages of their sample. Three pages each at Red.es, Social Security, and INEM also passed, and the MEC+UNED and MAP sites each had one correct page. None of the pages on the remaining 9 service sites (of the 15 surveyed) passed the criteria.
Priority 2 in WCAG 1.0.
From an accessibility point of view it is vital that text and/or image links be self-explanatory. In other words, the link should provide the user with a clear indication of what will be shown on the next page if s/he clicks on the link.
Having the right text or a helpful alternative description to an image in each link is essential because, for some users, it is the only guide they use to find the information they want. This is most frequent among users who navigate on small-screen devices, those who use text-to-speech screen readers, etc.
All of the pages surveyed used links, and more than half of them used them according to the accessibility guidelines. In four of the service sites (Red.es, BOE, Social Security, and INEM), the links were applied correctly on every page in our sample. CERES, ID+Passport, DGT, and MAP were at the other end of the spectrum: only one page on each site earned a passing score.
Priority 2 for images in WCAG 1.0.
Some people may not be able to see colors correctly. Contrast between an image and the background color becomes an important issue since the user cannot change the images. Thus, the web designer must be sure to provide clear contrast.
Some 67 of the 74 pages (90.54%) making up the sample were analyzed in terms of the contrast between the images and the background color. Once again, the Social Security website took first place, with all five pages sampled on its site receiving a passing score for color contrast. At the other end, Red.es, AEAT, MEC+UNED, and MAP failed the test on every single page in our sample.
Priority 1 in WCAG 1.0.
It is essential for any information that is conveyed through the use of colors to be equally available to the blind and the color-blind, not to mention anyone using a black and white monitor. It is not that color should not be used: many people with other types of impairments find color a great help to navigating through a website. Nevertheless, colors should be used wisely (by means of context or markers, for example).
Only 2 of the 74 pages in the total sample (2.70%) used colors semantically to convey information. Both belonged to the DGT website, though their usefulness met with uneven success on each.
Priority 2 in WCAG 1.0.
Using tables for layout can create havoc for screen readers (and thus for their users), which may not be able to render them correctly. This also holds true of text-only browsers. The WCAG 1.0 guidelines consider it essential that laying out content in tables be done correctly.
OF the 74 pages in our sample, 62 (86.11%) used tables for the layout of content. Ten of the web services using tables did so in such a way that every page using them was rendered properly. The worst score of all the services examined was on the Postal Service site, where 3 of the 6 pages using tables for layout did not render the content on the pages correctly.
Priority 1 in WCAG 1.0.
For the blind or otherwise visually impaired, something even harder than reading data gathered in a table is knowing when a piece of data falls under more than one category. To that end, it is vital that the tables have certain special features. For example, column and row headers should be properly marked as such; markers can be used to associate header cells with data cells in tables with two or more logical tiers of headers.
In our sample, we found 9 pages (12.16%) that used data tables. Though eight different web services used tables, only the Social Security website did so following W3C guidelines. CERES, AEAT, ID+Passport, INE (2 tables), DGT, ICEX, and the Ombudsman all failed to meet this criterion.
Priority 1 in WCAG 1.0.
Care must be taken so that there is no loss of functionality of scripts should the user for some reason not have them turned on in their browser. It is essential, then, to be sure to have some content alternative if scripts and other programming objects are not turned on or supported in a web browser.
A total of 46 pages in the sample of 74 pages in the study—62.16%--used scripts in designing the pages. Only the BOE site did not use scripts of any kind. All four pages on the Ombudsman website used scripts and did so correctly. The same was found of the 4 pages sampled on the Social Security site. As such, they both received a passing score. Also worthy of mention is the DGT site (3 of 5 pages) which used this programming element correctly. The only other sites to pass the test were Red.es, with its one scripted page, and ID+Passport, in one of its two scripted pages. The rest did not meet the prescribed guidelines.
It should be noted that, on the day the sample was taken, all 4 pages on the MAP website had script errors, but that 3 of them were subsequently corrected, not including the incorrect scripting on their homepage, which still persists.
For an overview of the scores received by each web service on the technical evaluation of web accessibility, we have listed the percentage scores in Table 2, from highest to lowest.
It must be noted that passing scores should not be interpreted as a guarantee of overall accessibility for any of the websites involved, since the present analysis is based on only some of the criteria set forth in the WCAG 1.0 guidelines. Further criteria listed therein should also be born in mind.
Topping the list is the web service for obtaining tax records from the Social Security website. Minor problems were found involving the use of absolute letter sizes on two of the style sheets; their correct use would have pushed their final score even higher than its praiseworthy 93.48% in this evaluation. The other error concerned the HTML mark-up for one of the pages sampled. These errors are easily corrected; we encourage their webmasters to fine-tune the pages and keep up with the good work done so far.
| SERVICE | % Success |
|---|---|
| Requesting Social Security tax records | 93.48 |
| Red.es, information on the information society | 61.11 |
| Filing a grievance to the Ombudsman | 52.63 |
| Searching public-sector job posts from MAP | 51.52 |
| Applying for unemployment at INEM | 43.33 |
| Consulting the BOE | 40.91 |
| How to apply for an ID card and passport | 34.88 |
| Sending mail from the Postal Service site | 24.49 |
| Consulting information from ICEX | 23.81 |
| Requesting financial aid from MEC and registering for UNED | 21.95 |
| Traffic reports and vehicle registration at DGT | 19.51 |
| Accessing Administracion.es Citizens’ web portal | 18.87 |
| Filing tax returns and quarterly VAT payments at AEAT | 16.33 |
| Checking data at INE | 16.33 |
| Obtaining a digital certificate from CERES | 15.91 |
| Average success rate (%) of services studied: | 34.31 |
Three other services ranked above the 50% success rate, which indicates a trend toward making governmental website services more accessible when compared with data from similar previous studies. The services sampled on Red.es, the Ombudsman, and the Ministry of Public Administration all show an encouraging improvement that, though still far from achieving full accessibility for everyone, suggests their efforts are beginning to pay off.
Nevertheless, it is discouraging to see that eight services (more than half of those in the study) failed to reach even a 25% success rate. Some of them have in fact heralded themselves as complying with a level of accessibility they are regrettably far from actually having, as this study reveals.
Most glaring among the failures is the poor showing (18.87%) of the Citizens’ Portal at Adminstracion.es, which is meant to be the gateway to access Public Administration services on the web for all users in Spain.
In last place, however, at a mere 15.91% success, is a website service considered essential for making any safe government transaction: obtaining a digital certificate through the Public Certification Authority of Spain, CERES.
The average success rate falls rather short at 34.31%, indicating that a considerable amount of works remains to be done for the web-based public services of the General State Administration to be accessible to everyone in the few months remaining before the legal deadline arrives.
The assessment of the General State Administration web services was carried out by 6 users, each of whom received a self-administered multiple-choice test with instructions on how to take the test.
Users were given the following instructions for assessing each of the 15 services:
The results were tabulated to draw comparable and measurable conclusions in terms of percentages.
After completing the survey, users met to discuss their experience to compare and contrast their overall impressions on the accessibility and user-friendliness of the services in the study.
Our first point of interest focuses on the rate of successes, mistakes, and defeats (i.e., giving up before completing the task) by the users carrying out the assessment tasks assigned for each service in the sample.
Table 3 shows the scores of the 6 users for each of the 5 tasks to be done in the 15 web services in our study sample. This amounted to 30 tasks for each service.
Of the 450 tasks done by the 6 users for the 15 services in the sample, there were 328 completed successfully (72.89%). Users made mistakes on only 30 (6.67%) of the tasks. However, the number of defeats in which the user gave up before completing the task due to accessibility problems reached 92 (20.44%). This failure rate is quite high, especially considering the fact that all the users had enough skill at using the medium (internet) to be able to undertake the tasks without supervision.
| Services | Successes | Mistakes | Defeats |
|---|---|---|---|
| Information on obtaining ID and passports | 27 | 3 | 0 |
| Obtaining a digital certificate from CERES | 27 | 1 | 2 |
| Filing a grievance with the Ombudsman | 25 | 1 | 4 |
| Consulting the BOE | 24 | 2 | 4 |
| Obtaining Social Security tax records | 24 | 1 | 5 |
| Traffic information and vehicle registration from DGT | 24 | 1 | 5 |
| Applying for unemployment benefits at INEM | 23 | 3 | 4 |
| Sending mail from the Postal Service | 23 | 0 | 7 |
| Financial aid from MEC and registration at UNED | 22 | 3 | 5 |
| Red.es, information on the information society | 21 | 1 | 8 |
| Data search at INE | 21 | 2 | 7 |
| Information from ICEX | 19 | 1 | 10 |
| Public-sector job posts on MAP | 19 | 6 | 5 |
| Administracion.es, Citizens’ portal | 15 | 2 | 13 |
| Filing income taxes and quarterly VAT payments on AEAT | 14 | 3 | 13 |
| Total: | 328 | 30 | 92 |
Worth mention here is the service for obtaining information on applying for an ID card and passport, which not only scored 27 successes out of 30 tasks, but also had no defeats and only 3 mistakes. Also awarded 27 successes, though with 2 defeats and one mistake, is the CERES site for obtaining a digital certificate.
[D]
At the other extreme, the poorest scores were found on services for filing taxes and VAT payments on the Tax Authority (AEAT) website, where success was met in only 14 cases, users gave up before completion in 13, and made mistakes in 3. In second-to-last place is the Administracion.es website for information on the Citizens’ portal, where only half of the tasks were carried out correctly (15), with 13 defeats, and 2 mistakes.
No mistakes were made on the Postal Service website for sending mail, though 7 tasks were given up before completion. The 6 mistakes made by users while assessing the Ministry of Public Administration’s civil service job posts gave it the highest number of mistakes of all the services polled, in addition to 5 defeats.
This section displays the results for each public service site in our study in terms of the “ad hoc” feedback written by each user-participant after performing the assigned tasks.
On the whole, the users were most satisfied with their experience performing the task of obtaining tax records from the Social Security website (79.44%). Close on its heels is the site for applying for an ID card and passport, which had the highest rating in the “successes, mistakes, and defeats” category of task assessment.
There were 5 public services on the web that did not reach 50% in their satisfaction rating: MAP, Red.es, Administracion.es, ICEX, and AEAT. The web service for filing income taxes and quarterly VAT payments at the Tax Authority website AEAT received by far the lowest score among the sites sampled (26.11%), mirroring their last-place ranking on the “successes, mistakes, and defeats” assessment of tasks by the users.
The results of the user feedback survey are higher than those of the technical evaluation. Namely, while 11 services failed to reach 50% on the technical evaluation, only 5 fell below the 50% mark on the user satisfaction ratings. It may be inferred that users, despite their functional limitations, develop alternate skills to overcome theoretical obstacles and complete the task.
Nevertheless, the Social Security website for obtaining tax records on the web, though ranked top in both categories, scored higher on its technical evaluation (93.48%) than on its user feedback assessment (79.44%). This may be due more to problems of usability rather than accessibility; the webmasters should therefore take heed of this difference.
Furthermore, it is curious to note the difference in scores on accessibility depending on which analytical procedure is applied. A good example is to be found in the scores for obtaining a digital certificate from the CERES website. This service received the lowest score on the technical evaluation (15.91%) yet was well positioned in the user feedback ratings (in fourth place at 73.33%). The discrepancy may be due to its having a high degree of usability.
[D]
Our last classification gathers the results from the two procedures used to analyze the sample of web-based services offered by the Spanish General State Administration. Chart 2 shows the average score in percentage points for each service. The purpose is to provide insight on the accessibility, including usability, of the services examined in the study by combining the criteria from the technical evaluation and the user feedback survey. It is not meant to be interpreted, however, as an absolute reckoning of the degree of accessibility.
At a glance, the results displayed in Chart 2 reveal the following points of note:
Although the overall success is far from resounding, our study does indicate that there has been considerable achievement made toward making the public services offered on General State Administration websites more accessible. While no similar prior study is known to have been done (i.e., focusing on the services rather than the home pages), reference can be made to other studies on the accessibility of Spain’s public administration websites [7].
Topping the list among the services reviewed in the present study is the Social Security website for requesting tax records. We emphatically encourage the people in charge of the website to fine-tune their site to make it “completely” accessible and for other webmasters to follow their noteworthy example.
Moreover, we notice a lack of concurrence between the statements some websites in the analysis have made regarding the level of accessibility they claim to offer and the results found in the present study. In our sample, 7 sites providing service make explicit mention of a level of compliance with the Web Content Access Guidelines 1.0 from W3C/WAI. Six of them claim to meet the priority 1 guidelines for accessibility (a WAI-A rating): adminstracion.es, DGT, the Postal Service, MAP, the Ombudsman, and Social Security. One site, Red.es, claims to fulfill the criteria for priority 1 and 2 (a WAI-AA rating).
The service with the lowest-ranking score in terms of technical criteria and user feedback alike was the AEAT website for filing income taxes and quarterly VAT payments. Oddly enough, this agency celebrated the European Year of the Handicapped in 2003 by underwriting an agreement with the Spanish Committee of Representatives of People with Disabilities (CERMI) in which they made the commitment to make their website services accessible to all. While the intentions of this public entity are no doubt admirable and their efforts well meant, the present study shows that few real gains have been made in that direction.
The most significant design errors, in terms of either sheer numbers or the negative effect they have on information accessibility, are summed up in the present study to be as follows:
On the positive side, we find the following points:
The new Law on Information Society Services and E-Business will soon take effect on December 31, 2005. This law requires that all Administration and governmental public services on the net comply with the “generally acknowledged” guidelines for web accessibility. If the deadline is to be met, effort must be made to review websites such as those covered in the present study, where considerable work remains to be done to meet even minimum standards of compliance. The Social Security website, despite its minor deficiencies, may serve as an example for webmasters and others in charge of developing public services on the web.
Note: All external URL´s open in a new window.
[1] See for the full report: http://www.discapnet.es/Discapnet/Castellano/Observatorio_infoaccesibilidad/observatorio04.htm
[2] W3C/WAI: Web Accessibility Initiative of the World Wide Web Consortium. For further information, see: http://www.w3.org/WAI
[3] Available in English (http://www.w3.org/TR/WCAG10/), on the W3C/WAI website, and in Spanish (http://www.teleservicios.es/accesibilidad/recursos/documentos/index.html) on the Fundosa Teleservicios website.
[4] Abbreviations in bold are used throughout the study to refer to each service.
[5] For more information on this program for automated evaluation of web page accessibility, see http://www.tawdis.net
[6] Web Accessibility Initiative / World Wide Web Consortium. See: http://www.w3.org/WAI/
[7] As a reference, see the study by Carlos Egea (1998) titled “Accesibilidad en los servidores de la Administración Pública”, in which none of the websites analyzed passed the requirements for accessibility (available at: http://usuarios.discapnet.es/disweb2000/art/admonweb3.htm).