Article Details

The Contribution of Diagnostic Substitution to the Increasing Prevalence of Autism

Categories: | Author: Matt O'Leary | Posted: 8/21/2009 | Views: 3175


This article is designed to evaluate the contribution of diagnostic substitution to the measured increasing prevalence of autism globally, but particularly in the United States. This is a very difficult task, largely because in order to do so, you have to identify what the actual prevalence of autism is over time. And, doing this is very hard because of the real difficulty in finding all or most cases of autism, a disorder that presents as a spectrum of dysfunction that is diagnosed exclusively by behavior.

A Preliminary Note - Use and Interpretation of Administrative Data
Administrative data, in this context, is data that is gathered pursuant to a governmental or other program designed to treat children with disabilities. The data was not gathered pursuant to a study or program that was designed to ensure accuracy in the gathering of the data. It was assembled for good reasons, but not for reasons that make this information particularly useful and accurate for studying things like changing prevalence of a disorder. Hence, researchers almost universally acknowledge that the use of administrative data in research studies looking at prevalence is often improper.
Despite this, the use of administrative data is very tempting for researchers. The marginal cost of assembling the data is minimal since someone already paid to assemble it for other purposes. And, the data often concerns populations that researchers are very interested in studying.
Special education data, one type of administrative data, provides an especially cost effective way to examine changes over time in the prevalence of ASDs. However, it suffers from the same problems other administrative data suffer from. For instance, special education data often includes only the child’s primary diagnosis.[1] Also, the IDEA[2] definition for autism is general enough to encompass all ASDs but state eligibility criteria can limit the extent to which high functioning children on the autism spectrum receive special education classifications.[3] These issues, and others, cause the number of cases identified in special education to be significantly below the true number of cases in the general population – problems with case finding.
Data from the Metropolitan Atlanta Development Disabilities Surveillance Program indicated that, although 91% of 6 to 10 year old children with identified autism in 1996 received special education services, only 48% of those had autism as a primary education classification. At the same time, virtually all age-eligible children with a special education classification of autism met the case definition criteria for autism in the MADDSP. This indicates that, in general, autism prevalence estimates based on special education data alone will be underestimates. [4]
Diagnostic Substitution is Obviously Occurring to Some Degree
Before talking in depth about diagnostic substitution (‘DS”) and autism, it will likely be helpful to describe what DS is. Coo et al suggest there are two primary types of DS: first, where children with multiple diagnoses are categorized differently over time, and second, when children are given a different diagnosis in the present than they would have been given in the past, due to changing referral and diagnostic practices. Coo et al suggest that both of these variations of DS may partially account for apparent increases in autism prevalence when using administrative data.
No serious commentator would argue DS is not happening in autism, or pretty much any condition, particularly those diagnosed primarily or exclusively by behavior and those diagnosed primarily in children who are constantly changing and presenting different behaviors. Diagnostic substitution occurs all the time. It happens at greater levels when a new disorder classification in introduced, as happened when the autism classification was first introduced into special education in the US in 1992 - some of the increase in prevalence should be attributable merely to local education agencies incorporating the new category into special education classification practices.[5]
As the Coo (2007) paper discussed below showed, DS happens in both directions: some kids were previously diagnosed with some other condition and then got an autism spectrum diagnosis. And, some kids with autism spectrum diagnoses are subsequently reclassified into another code. It is only the sum of the two measurements that can provide a direction and magnitude for the DS that is occurring.
Based upon all of the available evidence, it does seem substantially more likely that DS into autism is higher than DS away from autism. But, as Newschaffer said in his commentary piece to the Shattuck (2007) article:
assessment of potential diagnostic replacement should be rooted in a comparison of the magnitude, not just the direction, of classification prevalence. It is quite possible that small magnitude negative associations will be deemed statistically significant when denominators are large, as with the US census counts used here.
Conflicting Estimates of the Level of Diagnostic Substitution into Autism
An unanswered question (and a question that is difficult if not impossible to completely answer) is how much of the increase in prevalence in autism can be accounted for by diagnostic substitution from other classifications. The answers that have been offered vary widely.
            The Lower End of the Range
At the lower end of the range of estimates is a study by Gurney et al (2003) in Minnesota. In this study, the authors found that the prevalence of autism spectrum disorders among children 6 to 11 years of age increased from 3 per 10,000 in 1991/1992 to 52 per 10,000 in 2001/2002. Over the same time period the prevalence of other major special education disability categories also increased, with the exception of severe mental handicap, which decreased slightly from 24 to 23 per 10,000. Thus, the estimate of the magnitude of DS into autism was zero or close thereto.
The 2006 study by Newschaffer et al found that there was no substantial evidence of DS occurring related to autism. However, his 2007 response to Shattuck’s paper pulled back from that conclusion, basically agreeing that the logarithmic scale he used in his 2006 paper (along with his method of visually comparing distance between lines on the logarithmic charts to observe changing prevalence) obscured an apparent increase of DS into autism. Newschaffer noted:
I realize now that our choice of a logarithmic scale, although well-suited to our overarching descriptive goals, actually hampered our ability to determine if classification trends offset… The rescaled graphs demonstrated that, at least in the US aggregate data, the increases in autism classification prevalence were greater in absolute terms (the distance between lines is further) than the MR decreases, suggesting that any offset by MR alone would not be complete. In addition, at ages 10 and 11, autism prevalence still increases with successive birth cohorts, but MR prevalence no longer decreases consistently.
A visual inspection (admittedly by a lay person - me) of Newschaffer’s rescaled numbers (and the vertical distance between the plotted lines) indicates that decrease in MR cases, in absolute terms, is only half the magnitude of the increase in autism, leading to the implication that diagnostic substitution could account for no more than half of the additional cases of autism in this data.
While Newschaffer obviously made an error in interpreting the data he generated, I don’t think his error was intentional such that I would tend to disregard his opinion moving forward. There is a threshold question in DS analysis: whether the two conditions (or groups of conditions) that are being compared are moving in inverse directions – whether one is increasing in prevalence while the other is decreasing. Data that show both conditions (or groups of conditions) moving in the same direction likely indicate a relatively low level of DS between the two conditions. You don’t need intense statistical analysis to deal with such a situation. This is what Newschaffer thought he found related to autism and MR and hence limited his examination to a visual inspection of the graphed data; but he misunderstood the data related to DS because of the logarithmic scale he chose to use which obscured a significant pattern.
            The Upper End of the Range
At the upper end of the range of estimates of DS involvement in growing autism prevalence is the 2007 study by Shattuck. Shattuck, as with the studies discussed above, used administrative special education data from the US Department of Education in order to do his analysis. Like Newschaffer, Shattuck used national US data at the group level. There was no analysis of individual cases.
Shattuck did a complicated statistical analysis (one over my head) of prevalence over time of autism as well as numerous other conditions with special education classifications, including OHI (other health impairments - which includes ADHD), DD (generic developmental delay), TBI (traumatic brain injury), LD (learning disorder), and MR (mental retardation).
Shattuck’s administrative data showed an increase in combined prevalence of autism, OHI, TBI and DD of 12 per 1,000, as well as a decrease in the combined prevalence of LD and MR of 11 per 1,000. In Shattuck’s original article, he did not draw explicit conclusions other than noting that these changes were almost exactly the same absolute magnitude. [6] The implied conclusion, according to my reading, was that there hasn’t been any real statistically significant change in prevalence in any of these conditions – diagnoses have just been shifting around.
Shattuck was more explicit in his response to Newschaffer’s commentary that was published in Pediatrics. He states that in his study:
The observed aggregate MR prevalence among 6 to 11 year olds in special education decline by 2.8 per 1,000 from 1994 to 2003, whereas autism prevalence increased by 2.6 per 1,000. This indicates, in the aggregate, total decline in MR prevalence could have offset the total increase in autism prevalence almost 1 for 1 (the autism increase was from about 5/10,000 in 1994 to 31/10,000 in 2004 – see Figure 1 in Shattuck’s original article).
I have several problems with Shattuck’s analysis and conclusions. My first problem has to do with the language Shattuck uses. Shattuck’s paper is filled with complicated statistics. He obviously knows the principles. However, when he gets to the conversational areas of his paper and response, he uses fuzzy language that could easily be misinterpreted by the people reading this paper on a very controversial topic. Shattuck did not say that he had proved that the increase in autism was the result of diagnostic substitution from MR. However, his language offers, seemingly deliberately, exactly that implication. And, that is exactly the implication drawn from his language by the news reporters and most bloggers who were interested in the study – that the increase in autism could be explained away entirely by a decrease in MR. Shattuck should have known this would happen, and in my uninformed opinion, likely did. Moreover, Shattuck knew that he was drawing conclusions based upon unreliable administrative data and in fact acknowledged this in his study; however, this acknowledgement was in a different section of his original paper than his conclusions, and was not in his response to Newschaffer at all. The fact that administrative data is not reliable for this type of analysis was of course not reported in the news or the blogs. And, Shattuck likely knew this would be the case as well.
Second, and more substantively, when data show that the prevalence of two conditions, or groups of conditions, are moving in inverse directions, then this raises the possibility of substantial DS. However, it tells you only of a possibility; it does not establish causation. Shattuck’s data showed increasing levels of autism, DD, TBI, and OHI and decreasing levels of MR and LD. Despite all of the sophisticated statistical analysis he did, his paper basically comes down to him taking the position that it makes sense that DS can account for essentially all of the changing prevalence of these conditions - nothing is actually changing, other than the labels physicians apply to various behaviors / deficits kids display.
The problem with this conclusion is that the only evidence of causation is circumstantial. Yes, the statistical analysis provides evidence that not all of the change in prevalence can be happening by chance. But, it also doesn’t tell us that none of the changing prevalence is real – which is what this study is being taken to mean. The decrease in MR may, for instance, relate partially to reduction of fetal alcohol exposure or exposure to other teratogens. Also, part of the decrease in MR may reflect better diagnoses being given in other areas besides autism. It seems there should be a tendency to over diagnose MR because it requires performance up to a standard in order to avoid being labeled as MR. We know many autistic kids have been diagnosed MR who were not MR because of problems with communication. It is likely that other kids, say with dyslexia or other reading difficulties, have been improperly diagnosed as MR in the past who are now being properly categorized today. The decrease in LD may result, in part, from more sophisticated teaching techniques and the widespread use of special schools. The increase in TBI may result, in part, from higher numbers of accidents involving motorized vehicles, such as ATVs or motorcycles. I don’t know. There is all kinds of complexity that is being covered up in these numbers. What I do know, or think I know, is that drawing the conclusion that there is no real change in prevalence of a series of childhood mental conditions because those conditions that are increasing in prevalence are offset by those conditions that are decreasing in prevalence is a vastly oversimplified position that is similar in logic to the ‘correlation does not imply causation’ logical fallacy.
Third, attempting to explain essentially all of the increase in autism prevalence as a shift from the MR classification doesn’t make logical sense since many kids with autism, particularly the higher functioning ones who have only started being diagnosed in large numbers as awareness of autism has increased, are clearly not mentally retarded. Many autistic kids have scored as normal or superior in intelligence despite the inappropriateness of how intelligence tests have been administered to these kids with communications difficulties – in person. Significantly less than half of kids on the spectrum are truly mentally retarded. Now, we know that a significant fraction of them did get classified as MR in the past because of the testing problems. But, certainly not all or most of them were so misclassified.
Finally, the Coo et al study below demonstrates that untangling DS in mental / developmental disorders in children is extremely complex. The relatively equal change in prevalence of autism and MR in Shattuck’s dataset offers a convenient and simple explanation for the question of what is causing autism prevalence rates to skyrocket to those who are inclined to seize upon it. However, the world of childhood disorders is rarely simple. In the detailed, individual level analysis Coo et al did, the world they uncovered was extremely complex as I would expect. DS was happening constantly between all the various disorders, with DS going both ways. Kids with MR diagnoses were subsequently diagnosed with autism, and vice versa; same with all of the other potential diagnoses. Only 30% of kids who were classified with one diagnosis who were subsequently classified into autism originally had MR diagnoses. I have a hard time believing that Shattuck’s dataset is so vastly different from that used by Coo et al to allow for such a simplistic situation as Shattuck suggests.
What this tells me is that I don’t think you can rely on Shattuck’s analysis for much of anything. His group level administrative data is not appropriate for this type of analysis and for drawing conclusions. The implications he makes suggest a world that is simple and linear, which is something the Coo et al study (and simple logic) tells us does not reflect the real world. The relatively close magnitude of shifts between autism and MR should be interpreted at this time as no more than a partial, unquantified explanation for increasing autism prevalence, and doing otherwise involves falling into a typical logical trap.
            The Best Study to Date – The Middle of the Range
Probably the best study on the role of DS in increasing autism prevalence was performed by Coo et al. It is the best study because, while it still relies upon administrative data like the other studies (and suffers from similar case finding problems), it is the only study that looked into individual cases versus looking at group level data.
The Coo et al study was done in British Columbia on the west coast of Canada using special education data. This special education data seems to be somewhat different than US special education data. As far as I can tell, US special education data only comes from students enrolled in special education programs. In BC, officials from all schools (public and private, including home schooled) are required to provide their school district offices with enrollment and demographic information for each child in their respective schools, including special education classifications.
The study looked at kids who were 4 to 9 years of age during the years 1996 to 2004. That time period seems to have been chosen because the diagnostic world was pretty static during that time. DSM-IV had been introduced in 1994. In the US (and I think Canada), the IDEA law required schools to add autism on a mandatory basis to their classifications starting in 1994. Also, autism was treated the same way in Canada during that period - autism referred to a diagnosis of autistic disorder, thus excluding other autism spectrum disorders (“ASDs”). In 2004, the BC government introduced standards for the assessment of children with ASDs, thereby changing the diagnostic landscape. This careful selection of a time frame relatively free of events that would exacerbate apples to oranges comparison problems seems well considered.
The Coo et al study found that the point prevalence of autistic disorder increased by 30.8 per 10,000 over the study period, from 12.3 per 10,000 to 43.1 per 10,000 in 2004. Children with another special education code who were then assigned an autism code in the following year accounted for 16.0 per 10,000 of the total increase in prevalence. In contrast, children with an autism code in one year who were then assigned another special education code in the following year accounted for a decrease in autism prevalence of 5.9 per 10,000. Accordingly, the net contribution of DS to the increase in autism prevalence from 1996 to 2004 was 10.1 per 10,000, or 32.8%.[7]
Another third or so of new diagnoses of autism were accounted for through identification of previously undetected cases; however, the authors did not discuss what caused these cases to be previously undiagnosed. In discussing what might account for the remaining autism cases (not DS or previously unidentified cases), the authors noted that it is also possible that alterations in diagnostic and referral patterns, or an increase in the real risk of autism, may underlie the remaining increase. However, they were not able to examine any of these hypotheses with the data that was available. But, Coo et al were able to largely rule out broadening diagnostic criteria as a contributor. They indicated:
It is unlikely that changes in diagnostic criteria played a major role in children switching classifications during the study period, since the DSM-IV criteria were in effect throughout.
While the Coo et al study still suffers from the use of administrative data, its estimate (based upon individual levels data) that 32.8% (10.1 of the 30.8) of autism diagnoses result from DS is the best estimate that is available today. And, the Coo et al study gives us an additional reason to be even more cautious related to the speculations Shattuck made in his 2007 study. Coo et al showed that approximately 3 per 10,000 of the 10.1 per 10,000 cases involving DS were with MR. The other 7.1 per 10,000 cases of DS involved one of the many other conditions diagnosed in children. This is close to 1/10th of the rate of DS between autism and MR that Shattuck implied existed based upon the group level administrative data his study rested upon.
Recalculating DS Contributions to Autism Prevalence with Consensus Estimates of Autism Prevalence
 As shown above, there is an extremely wide range of estimates as to what percentage of the increase in special education autism cases can be accounted for by DS. The estimates essentially range from none to all. What is missing from this debate is a discussion of the fact that special education data do not include all cases of autism.
It is well established that relying upon one method of case finding generates dramatic underestimates of autism prevalence. Case finding improving when researchers 1) identify cases through health care settings, other educational settings, community settings, and otherwise, 2) avoid the use of case registries and administrative data that are established for purposes other than establishing prevalence, 3) work in countries with health care systems that offer universal coverage and aggressively seek to identify disorders in children (like Japan or Sweden), 4) have contact directly with the kids being screened, 5) control for the fact that in many settings cases are missed because children are only given one diagnosis when they would qualify for multiple conditions, and 6) use relatively small populations such that cases don’t slip through the cracks. The case finding that has occurred in the United States has been particularly suspect largely because of nature of American society and its health care and educational systems.
Even the best studies using administrative data have big problems with case finding. For instance, the excellent Coo et al study clearly has missed lots of cases of autism that exist in British Colombia society. This is because the Coo et al study used data assembled from only one source, special education administrative data, which was done for the convenience of the researchers. Also, the special education data did not include many cases of Asperger’s Syndrome or Pervasive Developmental Disorder, since those conditions were not diagnoses during the time studied (though it is likely that many kids with Asperger’s Syndrome or PDD were still diagnosed as autistic under the classifications that existed at the time).
If you look at studies that were specifically designed to estimate autism prevalence, are recent (the quality of the studies has improved dramatically recently), and engaged in aggressive efforts to find cases, consensus minimum estimates for autism spectrum disorders are 1 in 150 (66 per 10,000). However, there are numerous studies that support much higher estimates, and researchers are starting to recalibrate minimum estimates to 1 in 100 (100 per 10,000), or even higher.
What does this mean for the role of DS in autism prevalence? Data concerning prevalence constitutes the denominator in all of the measurements concerning DS. For instance, Coo based her estimate that DS accounted for about a third of the increased prevalence on special education data that showed that autism rates increased from 12.3/10,000 in 1996 to 43.1/10,000 in 2004; she compared the 10.1/10,000 number that could be explained by DS to this 30.8/10,000 increase generate her estimate of 32.8% (10.2/30.8) of the increase in autism prevalence being attributable to DS. But, her measurement of current autism prevalence based on special education data (43.1/10,000) is well below low and high-end consensus estimates based on all available data (66/10,000 and 100/10,000). If you substitute these two higher consensus numbers into the denominator of Coo’s fraction, you end up with much lower contributions of DS to the increase in autism prevalence, 18.8% (66-12.3=53.7;10.1/53.7=18.8%) at the low end and 11.5% (100-12.3=87.7;10.1/87.7=11.5) at the high end, with the role for DS from MR being only 30% of those recalculated totals. This is based on the assumption that there was no DS happening in the cases that are not included in special education data.
If you look at Shattuck’s numbers, the contribution of DS to increases in autism prevalence drops dramatically when you factor in the cases that are not showing up in the administrative data. Shattuck’s data showed an increase in autism prevalence from 5/10,000 to 31/10,000. He suggested that all of this increase (26 per 10,000) could be explained by DS from MR, which decreased by 28 per 10,000 in the same period. If you replace 31/10,000 with the low end and high end consensus estimates, the theorized 100% contribution of DS to increasing autism prevalence becomes 42.3% (66-5=61;26/61=42.3) at the low end and 27.3% (100-5=95;26/95=27.3) at the high end. Both of these percentages are substantially less than 100%, leaving open lots of room for other factors to contribute to the increasing observed prevalence of autism.
Evaluating the Estimates Made by Prometheus, a Pro-Science Autism Blogger
Prometheus, a pro-science autism blogger who blogs at photoninthedarkness.com, a site I read frequently, did an analysis of the possible contribution of DS to increases in autism prevalence called Five Simple Graphs. He used these graphs to illustrate that DS can account for a very large percentage (he didn’t say exactly what) of the newly diagnosed cases of autism. A few quotes from his post are below:
As you can see, the rise in autism is almost exactly paralleled by the decline in mental retardation (slope = 0.017, r = 0.999), indicating that the total prevalence of autism and mental retardation - as a fraction of all children served under the IDEA - has not changed significantly since 1993.
Does this mean that allof the rise in autism is simply a shift away from “diagnosing” children with mental retardation? Unfortunately, these data can’t make that determination - all they can show is that the “epidemic” rise in autism has been accompanied by an…. what would you call an inverse epidemic?… an anti-epidemic of mental retardation. However, a number of studies - looking at different data sets [Shattuck (2006) and Coo et al (2007) come readily to mind] have shown that diagnostic substitution is a large part of the “autism epidemic”.
All these data show is where the “hidden horde” might have been hiding.
There are a couple issues with Prometheus’ argument that are similar to some of the problems in the Shattuck paper.\
First, the magnitude in changes in autism and MR prevalence are not really all that close. According to a chart from Prometheus’s paper, the rate of autism rose from 5 per 10,000 to 50 per 10,000, a 45 per 10,000 increase, while the rate of MR fell from 112 per 10,000 to 86 per 10,000, a 26 per 10,000 decrease. That leaves a difference of 19 per10,000 or 42.2% of cases not explained by DS, which is a pretty big number that shouldn’t get swept under the rug.
Second, the source of Prometheus’s data is US Department of Education administrative data. As discussed above, administrative data is unsuitable for drawing conclusions about the cause of changing prevalence in a condition like autism. Yet, Prometheus is sure implying what he thinks about the data.
Third, like Shattuck, Prometheus is implying conclusions based on potentially coincidental associations between prevalence rates for autism and MR. While there is likely a substantial causal relationship between rising autism rates and falling MR rates, we have no evidence that falling MR rates account for all, most, or even a lot of the rising autism rates. The data simply does not speak to this.
Fourth, the USDE administrative data Prometheus is relying upon is not reliable data for establishing autism prevalence levels since it was collected for entirely different purposes and therefore lacked rigor in case finding. The most recent autism prevalence rates this data show are 50 pe 10,000 which are much lower than low-end (66 per 10,000) and high-end (100 per 10,000) consensus levels. If you recalculate Prometheus’ estimate that DS from MR to autism accounts for 57.8% of cases of increased prevalence based on these consensus prevalence estimates, the actual rates become 42.6% (66-5=61;26/61=42.6) at the low end and 27.4% at the high end (100-5=95;26/95=27.4). Both 42.6% and 27.4% are a lot less than 100% and leave open a lot of other possible explanations for what is causing autism prevalence to increase dramatically.
I don’t mean this paper to add fuel to the fire in the debate over whether there has been an autism ‘epidemic’ which is intimately tied into the question of whether vaccines cause autism. I personally believe that vaccines have little if anything to do with autism. However, I believe that there is a real possibility that autism prevalence has increased not only because of improved scientific practices but also because there are more cases which are caused by as of yet unidentified environmental factors.
I tend to believe that the scientific mainstream has an interest in disposing of the idea that autism rates are actually increasing (whether they are or are not) because such a conclusion would bolster the belief that any such increase would be caused by vaccines, and this is the last thing scientists want to be dealing with. Thus, a bias has been created (scientists are human too) in scientists to suppress the idea of an actually increasing autism caseload that is resulting in research slanted (probably unconsciously) towards an agenda. I am really only interested (I hope, and I think, but bias is always there) in figuring out what is really happening.

[1] Coo, 2007
[2] The Individuals with Disabilities Education Act in the US mandates state funded education for all children with disabilities and reporting associated with this education.
[3] Newschaffer, 2005
[4] Newschaffer, 2005
[5] Newschaffer, 2005
[6] Shattuck, 2007
[7] Coo, 2007
Print Bookmark and Share


Caitlinse Serx Says:
5/13/2011 1:12:11 AM
Evidence is beginning to come out that more children may have autism than are diagnosed. A study found that among random families in South Korea, more children exhibited signs of autism than were diagnosed with it. The implication is that many cases of autism go undiagnosed. I found this here: Autism rates could be higher than previously thought
Leave a Comment
Email: (not displayed)

Enter the code shown above:

Return to previous page