The Common Core Standards in Mathematics stress the importance of conceptual understanding as a key component of mathematical expertise. Alas, in my experience, many math teachers do not understand conceptual understanding. Far too many think that if students know all the definitions and rules, then they possess such understanding.

The Standards themselves arguably offer too little for confused educators. The document merely states that “understanding” means being able to justify procedures used or state why a process works:

But what does mathematical understanding look like? One hallmark of mathematical understanding is the ability to justify, in a way appropriate to the student’s mathematical maturity, why a particular mathematical statement is true or where a mathematical rule comes from. There is a world of difference between a student who can summon a mnemonic device to expand a product such as (a + b)(x + y) and a student who can explain where the mnemonic comes from.

A few of the “understanding” standards provide further insight:

Students understand connections between counting and addition and subtraction (e.g., adding two is the same as counting on two). They use properties of addition to add whole numbers and to create and use increasingly sophisticated strategies based on these properties (e.g., “making tens”) to solve addition and subtraction problems within 20. By comparing a variety of solution strategies, children build their understanding of the relationship between addition and subtraction. [emphasis added].

Note what I highlighted: understanding requires focused inferential work. Being helped to generalize from one’s specific knowledge is key to genuine understanding.

Daniel Willingham, the cognitive scientist who often writes on education, offers a more detailed account of the nature and importance of conceptual understanding in math (along with the other two pillars of mastery, factual knowledge and procedural skill) in his article from a few years ago in the AFT journal American Educator.

A procedure is a sequence of steps by which a frequently encountered problem may be solved. For example, many children learn a routine of “borrow and regroup” for multi-digit subtraction problems. Conceptual knowledge refers to an understanding of meaning; knowing that multiplying two negative numbers yields a positive result is not the same thing as understanding why it is true.

…knowledge of procedures is no guarantee of conceptual understanding; for example, many children can execute a procedure to divide fractions without understanding why the procedure works. Most observers agree that knowledge of procedures and concepts is desirable.

Willingham discusses the poor results for basic content and procedural knowledge, as revealed by trends in testing. However, he also notes -

More troubling is American students’ lack of conceptual understanding. Several studies have found that many students don’t fully understand the base-10 number system. A colleague recently brought this to my attention with a vivid anecdote. She mentioned that one of her students (a freshman at a competitive university) argued that 0.015 was a larger number than 0.05 because “15 is more than 5.” The student could not be persuaded otherwise.

Another common conceptual problem is understanding that an equal sign ( = ) refers to equality—that is, mathematical equivalence. By some estimates, as few as 25 percent of American sixth- graders have a deep understanding of this concept. Students often think it signifies “put the answer here.”

Here is a lovely paper expanding upon the issue of misconceptions in arithmetic, from a British article for teachers (hence the word “maths” and the spelling of “recognise”):

1. A number with three digits is always bigger than one with two
. Some children will swear blind that 3.24 is bigger than 4.6 because it's got more digits. Why? Because for the first few years of learning, they only came across whole numbers, where the 'digits' rule does work.

2. When you multiply two numbers together, the answer is always bigger than both the original numbers
. Another seductive 'rule' that works for whole numbers, but falls to pieces when one or both of the numbers is less than one. Remember that, instead of the word 'times' we can always substitute the word 'of.' So, 1/2 times 1/4 is the same as a half of a quarter. That immediately demolishes the expectation that the product is going to be bigger than both original numbers.

3. Which fraction is bigger: 1/3 or 1/6?
 How many pupils will say 1/6 because they know that 6 is bigger than 3? This reveals a gap in knowledge about what the bottom number, the denominator, of a fraction does. It divides the top number, the numerator, of course. Practical work, such as cutting pre-divided circles into thirds and sixths, and comparing the shapes, helps cement understanding of fractions.

4. Common regular shapes aren't recognised for what they are unless they're upright
. Teachers can, inadvertently, feed this misconception if they always draw a square, right-angled or isosceles triangle in the 'usual' position. Why not draw them occasionally upside down, facing a different direction, or just tilted over, to force pupils to look at the essential properties? And, by the way, in maths, there's no such thing as a diamond! It's either a square or a rhombus.

5. The diagonal of a square is the same length as the side?
 Not true, but tempting for many young minds. So, how about challenging the class to investigate this by drawing and measuring. Once the top table have mastered this, why not ask them to estimate the dimensions of a square whose diagonal is exactly 5cm. Then draw it and see how close their guess was.

6. To multiply by 10, just add a zero
. Not always! What about 23.7 x 10, 0.35 x 10, or 2/3 x 10? Try to spot, and unpick, the 'just add zero' rule wherever it rears its head.

7. Proportion: three red sweets and two blue
. Asked what proportion of the sweets is blue, how many kids will say 2/3 rather than 2/5? Why? Because they're comparing blue to red, not blue to all the sweets. Always stress that proportion is 'part to whole'.

8. Perimeter and area confuse many kids
. A common mistake, when measuring the perimeter of a rectangle, is to count the squares surrounding the shape, in the same way as counting those inside for area. Now you can see why some would give the perimeter of a two-by-three rectangle as 14 units rather than 10.

9. Misreading scales. 
Still identified as a weakness in Key Stage test papers. The most common misunderstanding is that any interval on a scale must correspond to one unit. (Think of 30 to 40 split into five intervals.) Frequent handling of different scales, divided up into twos, fives, 10s, tenths etc. will help to banish this idea.

From Teachers: January 2006 Issue 42 [UK] 

A definition of conceptual understanding. In light of the confusion about conceptual understanding and the pressing problem of student misunderstanding, I think a slightly more robust definition of conceptual understanding is wanted. I prefer to define it this way:

Conceptual understanding in mathematics means that students understand which ideas are key (by being helped to draw inferences about those ideas) and that they grasp the heuristic value of those ideas. They are thus better able to use them strategically to solve problems – especially non-routine problems – and avoid common misunderstandings as well as inflexible knowledge and skill.

In other words, students demonstrate understanding of –

1)   which mathematical ideas are key, and why they are important

2)   which ideas are useful in a particular context for problem solving

3)   why and how key ideas aid in problem solving, by reminding us of the systematic nature of mathematics (and the need to work on a higher logical plane in problem solving situations)

4)   how an idea or procedure is mathematically defensible – why we and they are justified in using it

5)   how to flexibly adapt previous experience to new transfer problems.

A test for conceptual understanding. Rather than explain my definition further here, I will operationalize it in a little test of 13 questions, to be given to 10^th, 11^th, and 12^thgraders who have passed all traditional math courses through algebra and geometry. (Middle school students can be given the first 7 questions.)

Math teachers, give it to your students; tell us the results.

I will make a friendly wager: I predict that no student will get all the questions correct. Prove me wrong and I’ll give the teacher and student(s) a big shout-out.

1)   “You can’t divide by zero.” Explain why not, (even though, of course, you can multiply by zero.)

2)   “Solving problems typically requires finding equivalent statements that simplify the problem” Explain - and in so doing, define the meaning of the = sign.

3)   You are told to “invert and multiply” to solve division problems with fractions. But why does it work? Prove it.

4)   Place these numbers in order of largest to smallest: .00156, 1/60, .0015, .001, .002

5)   “Multiplication is just repeated addition.” Explain why this statement is false, giving examples.

6)   A catering company rents out tables for big parties. 8 people can sit around a table. A school is giving a party for parents, siblings, students and teachers. The guest list totals 243. How many tables should the school rent?

7)   Most teachers assign final grades by using the mathematical mean (the “average”) to determine them. Give at least 2 reasons why the mean may not be the best measure of achievement by explaining what the mean hides.

8)   Construct a mathematical equation that describes the mathematical relationship between feet and yards. HINT: all you need as parts of the equation are F, Y, =, and 3.

9)   As you know, PEMDAS is shorthand for the order of operations for evaluating complex expressions (Parentheses, then Exponents, etc.). The order of operations is a convention. X(A + B) = XA + XB is the distributive property. It is a law. What is the difference between a convention and a law, then? Give another example of each.

10)  Why were imaginary numbers invented? [EXTRA CREDIT for 12^th graders: Why was the calculus invented?]

11) What’s the difference between an “accurate” answer and “an appropriately precise” answer? (HINT: when is the answer on your calculator inappropriate?)

12)   “In geometry, we begin with undefined terms.” Here’s what’s odd, though: every Geometry textbook always draw points, lines, and planes in exactly the same familiar and obvious way - as if we CAN define them, at least visually. So: define “undefined term” and explain why it doesn’t mean that points and lines have to be drawn the way we draw them; nor does it mean, on the other hand, that math chaos will ensue if there are no definitions or familiar images for the basic elements.

13) “In geometry we assume many axioms.” What’s the difference between valid and goofy axioms – in other words, what gives us the right to assume the axioms we do in Euclidean geometry?

Let us know how your kids did – and which questions tripped up the most kids – and why, if you discussed it with them.

A postscript for geeky readers of my blog, and for fans of E D Hirsch’s work who have been critics of mine in the past re: Knowledge:

I have been surprised to discover that there are a whole bunch of smart, literate, andlearned teachers who seem to deny that (conceptual) understanding even exists as a goal separate from knowledge – and by extension that my work and the work of many others is without merit. To them – as to E D Hirsch, it seems – there is only “Knowledge.” This, despite the fact that the distinction between knowledge and understanding is embedded in all indo-European languages, has a pedigree that goes back to Plato and runs through Bloom’s Taxonomy; the National Academy of Science publication How People Learn; and is the basis of decades of successful work in understanding by Perkins, Gardner, the research in student misconceptions in science, and the research on transfer of learning.

Some of my critics regularly cite Willingham’s summary of educational research, and a paper by Clark, Kirschner, and Sweller (discussed below) to make clear that direct instruction leading to knowledge is the only way to frame the challenge of both aim and means in effective education. As I will show, I believe they overstate what the research actually says and have little ground for suggesting that there is no meaningful difference between knowledge and understanding.

Wilingham on conceptual understanding in math. First, let’s look more closely at what researcher Daniel Willingham has to say about conceptual understanding in mathematics. His article is based on the idea that successful mathematics learning – presumably generalizable to all learning – requires three different abilities that must be developed and woven together: control of facts, control of processes, and conceptual understanding. And throughout the article he discusses not only the importance of understanding – and how it is difficult to obtain – but also notes that instruction for it has to be different than the learning of basic skills and facts. I quote him at length below:

Unfortunately, of the three varieties of knowledge that students need, conceptual knowledge is the most difficult to acquire. It’s difficult because knowledge is never acquired de novo; a teacher cannot pour concepts directly into students’ heads. Rather, new concepts must build upon something that students already know. That’s why examples are so useful when introducing a new concept. Indeed, when someone provides an abstract definition (e.g., “The standard deviation is a measure of the dispersion of a distribution.”), we usually ask for an example (such as, “Two groups of people might have the same average height, but one group has many tall and many short people, and thus has a large standard deviation, whereas the other group mostly has people right around the average, and thus has a small standard deviation.”). [emphasis added]

This is also why conceptual knowledge is so important as students advance. Learning new concepts depends on what you already know, and as students advance, new concepts will increasingly depend on old conceptual knowledge. For example, understanding algebraic equations depends on the right conceptual understanding of the equal sign. If students fail to gain conceptual understanding, it will become harder and harder to catch up, as new conceptual knowledge depends on the old. Students will become more and more likely to simply memorize algorithms and apply them without understanding.

Yet, for some reason, critics fail to accept this distinction – or see the inherent paradox, therefore, in education (discussed below). Novices need clear instruction and simplified/scaffolded learning, for sure. But such early simplification will likely come back to inhibit later nuanced and deeper learning - not as a function of "bad" teaching but because of the inherent challenge of unfixing earlier, simpler knowledge.

Perhaps part of the problem are the either-or terms that some researchers have used to frame this discussion. The essence of the false dichotomy is contained in Clark, Kirschner, and Sweller. Here is the introduction to the paper:

The goal of this article is to suggest that based on our current knowledge of human cognitive architecture, minimally guided instruction is likely to be ineffective. The past half-century of empirical research on this issue has provided overwhelming and unambiguous evidence that minimal guidance during instruction is significantly less effective and efficient than guidance specifically designed to support the cognitive processing necessary for learning.

The authors suggest, in other words, that evidence-based research shows that so-called “constructivist” i.e. "discovery" views of teaching are wrong on two counts:

1. The authors claim that those who use discovery/problem-based/project-based learning - all unhelpfully lumped together as one thing by the authors - have confused the cognitive meaning “constructivism” (a correct psychology theory of how minds make sense of data) with “constructivist teaching” (an unsubstantiated theory of how people best learn).
2. The authors claim that this inappropriate view of inductive pedagogy confuses the needs and traits of the expert with that of the novice:

Another consequence of attempts to implement constructivist theory is a shift of emphasis away from teaching a discipline as a body of knowledge toward an exclusive emphasis on learning a discipline by experiencing the processes and procedures of the discipline (Handelsman et. al., 2004; Hodson, 1988). This change in focus was accompanied by an assumption shared by many leading educators and discipline specialists that knowledge can best be learned or only learned through experience that is based primarily on the procedures of the discipline. This point of view led to a commitment by educators to extensive practical or project work, and the rejection of instruction based on the facts, laws, principles and theories that make up a discipline’s content accompanied by the use of discovery and inquiry methods of instruction. The addition of a more vigorous emphasis on the practical application of inquiry and problem-solving skills seems very positive. Yet it may be a fundamental error to assume that the pedagogic content of the learning experience is identical to the methods and processes (i.e., the epistemology) of the discipline being studied and a mistake to assume that instruction should exclusively focus on methods and processes.

In sum, those who promote "discovery" or "unguided" learning make two big mistakes, unsupported by research, say the authors: effective learning requires direct, not indirect instruction. And the needs of the novice are far different than the needs of the expert, so it makes little sense to treat novice students as real scientists who focus on inquiry. (Even though the authors offer the aside that “a more vigorous emphasis on the practical application of inquiry and problem-solving skills” is a good thing.)

But: huh? In 30 years of working with teachers I know of no teacher – secondary school or college – who rejects the teaching of “scientific facts, laws, and principles.” Indeed, science classes in HS and college universally are loaded with instruction, textbook learning, and testing on such knowledge.

Here is what the Clark et al. say in a follow-up article in American Educator:

Our goal is to put an end to the debate (about direct vs discovery learning). Decades of research clearly demonstrate that for novices (comprising virtually all students) direct, explicit instruction is more effective and more efficient than partial guidance. So, when teaching new content and skills to novices, teachers are more effective when they provide explicit guidance accompanied by practice and feedback, not when they require students to discover many aspects of what they must learn. [emphasis in the original]

What a curious definition of “novice”! The “novice” category is stretched to include “virtually all students.” This is surely a sweeping overstatement - much like the sweeping categorization of all non direct-instruction pedagogies as "discovery" that has been so criticized by others. We quite properly expect older middle and high school students, never mind college students, to do extensive self-directed and inductive work in reading, writing, problem solving, and research because they are no longer novices at core academic skills.

Indeed, later in the article, the authors strike a somewhat different pose about the complete repertoire of pedagogies needed by good teachers:

…[T]his does not mean direct, expository instruction every day. Small group work and independent problems and projects can be effective – not as vehicles for making discoveries but as a means of practicing recently learned skills. [emphasis in the original]

Though this properly expands the list of effective instructional moves, their framing is odd – and telling. The purpose of non-routine problem-solving, making meaning of a new text, doing original research, or engaging in Socratic Seminar they say is to “practice” recently learned “skills.” Hardly. These approaches have different aims, understanding-related aims, that are never addressed in their paper.

Indeed, this is just how conceptual and strategic thinking for transfer must be developed to achieve understanding: through carefully designed experiences that ask students to bring to bear past experience on present work, to connect their experiences into understanding. As Eva Brann famously said about the seminar at St. John’s College, the point of student-led discussion is “not to learn new things but to think things anew.” Indeed, Willingham’s warning about “not pouring concepts into a student’s head” when understanding is the goal is the important advice that is constantly overlooked by the authors and their supporters as the focus is overly-narrowed to teaching skill via direct instruction.

The authors even tacitly acknowledge this later in the article, in discussing why what works for novices doesn’t work for “experienced learners” in a subject – and vice versa:

In general, the expertise reversal effect states that "instructional techniques that are highly effective with inexperienced learners can lose their effectiveness and even have negative consequences when used with more experienced learners." This is why, from the very beginning of this article, we have emphasized that guidance is best for teaching novel information and skills. This shows the wisdom of instructional techniques that begin with lots of guidance and then fade that guidance as students gain mastery. It also shows the wisdom of using minimal guidance techniques to reinforce or practice previously learned material.

Well, which is it, then? Are “virtually all” students “novices” or not? When does a gradual-release-of-responsibility kick in? Just when is a student “gaining mastery” enough to use more inferential methods? We know the answer in reading: in middle school, based on the “gold standard” controlled research of Palinscar and Brown – that the authors mention in the citations!

Willingham in fact concludes his article by questioning the very novice-expert sequence laid out by Clark, Kirschner, and Sweller when the goal is conceptual understanding. After describing the “caricatures” in the math-wars debate of “process” vs. “conceptual” knowledge, he says:

Somewhat more controversial is the relative emphasis that should be given to these two types of knowledge, and the order in which students should learn them.

Perhaps with sufficient practice and automaticity of algorithms, students will, with just a little support, gain a conceptual understanding of the procedures they have been executing. Or perhaps with a solid conceptual under- standing, the procedures necessary to solve a problem will seem self-evident.

There is some evidence to support both views. Conceptual knowledge sometimes seems to precede procedural knowledge or to influence its development. Then too, procedural knowledge can precede conceptual knowledge. For example, children can often count successfully before they understand all of counting’s properties, such as the irrelevance of order.

A third point of view (and today perhaps the most commonly accepted) is that for most topics, it does not make sense to teach concepts first or to teach procedures first; both should be taught in concert. As students incrementally gain knowledge and understanding of one, that knowledge supports comprehension of the other. Indeed, this stance seems like common sense. Since neither procedures nor concepts arise quickly and reliably in most students’ minds without significant prompting, why wouldn’t one teach them in concert?

Indeed. Sequence in learning is not at all settled, as Clark et al profess, when the aim is understanding as opposed to basic skills to be learned the first time.

The key to understanding understanding: the ubiquity of persistent misunderstanding. Ultimately, a key lacuna in the everything-is-knowledge-through-direct-instruction view is its inability to adequately explain student misconceptions and transfer deficits that persist in the face of conventional direct teaching in science and mathematics. A glaring weakness in the Clark, Kirschner, and Sweller paper is their one-sentence treatment of student misconceptions: they suggest that misconceptions are the likely result of allowing students to discover concepts and facts for themselves!

This is surely a slanted view. There is a 30-year history of research in science and math misconceptions that shows conclusively that traditional high school and college direct instruction leads unwittingly to persistent misconceptions, and that a more interactive concept-attainment approach works to overcome them.

Multiplication is not repeated addition. The equal sign does not mean “find the answer.” Then, why is this a near-universal misunderstanding of these ideas? Presumably as a result of teachers not teaching for conceptual understanding and failing to think through the predictable misunderstandings that will inevitably arise when teaching novices the basics in simplified ways. teaching a concept as a fact simply does not work, as Willingham notes.

The paradox of education. What these examples beautifully indicate is the paradox of teaching novices that so many knowledge-centric educators seem to overlook. Yes, we must simplify and scaffold the work for the novice and make direct instruction clear and enabling – but in so doing we invariably sow the seeds of misconceptions and inflexible knowledge if we do not also work to attain genuine understanding of what the basics do and do not mean.

Indeed, the success of Eric Mazur’s work at Harvard and with other college faculties, and the Arizona State Modeling project in physics, both backed by more than a decade of research in college and high school science, cannot be understood unless one sees the connection between conceptual understanding and transfer, and the failure of transfer to occur when there is just factual and procedural instruction.

In fact, a telling comment made by Barak Rosenshine, a leader in direct instruction, that DI has a more limited use than Clark et al acknowledge:

Rosenshine and Stevens concluded that across a number of studies, when effective teachers taught well-structured topics (e.g., arithmetic computation, map skills), the teachers used the following pattern:

Begin a lesson with a short review of previous learning.

[emphasis added]

Rosenshine is far more careful than Clark et al to clarify the meaning of the term "direct instruction" which he claims has five different meanings that need to be sorted out. In fact, he notes that reading comprehension is a different kind of learning task than developing straightforward skills, and thus requires a different kind of direct instruction – instruction in cognitive strategies:

Even though the teacher effectiveness meaning was de­rived from research on the teaching of “well-structured” tasks such as arithmetic computation and the cognitive strategy meaning was derived from research on the teach­ing of “less-structured” tasks such as reading compre­hension, there are many common instructional elements in the two approaches.

In most of these studies students who received “direct instruction” in cognitive strategies significantly outper­formed students in the control group comprehension as assessed by experimenter-developed short answer tests, summarization tests, and/or recall tests.

(Note, therefore, that DI offers no justification for the kind of “direct instruction” done by ineffective high school and college teachers – i.e. too much teacher talk. DI is a method for learning and applying skills.)

Here we see the paradox, more clearly: no one can directly teach you to understand the meaning of a text any more than a concept can be taught as a fact. The teacher can only provide models, think-alouds, and scaffolding strategies that are practiced and debriefed, to help each learner make sense of text. Otherwise we are left with the silly view that English is merely the learning of facts about each text taught by the teacher or that science labs are simply experiences designed to reinforce the lectures. As I noted here, Willingham argues that teaching cognitive strategies are beneficial in literacy – in contrast to Hirsch’s constant and sweeping complaints about the lack of value in teaching such strategies and asking students to use them.

Interestingly, in an interview Rosenshine seems a bit insensitive to the problem of inflexible knowledge in less able students who need to rely on initial scaffolds for a long time:

Rosenshine: "Cognitive strategies" refers to specific strategies students can use to provide a support in their initial learning. For example, in teaching writing there is a cognitive strategy called the five-paragraph essay. The format for this essay suggests that students begin with an introductory paragraph containing a main idea supported by three points. These points are elaborated in the next three paragraphs, and then everything is summarized in the final paragraph.

After describing a lesson on Macbeth in which the essay template and DI are used, Rosenshine says:

The teacher told me he used this same approach with classes of varying abilities and had found that the students in the slower classes hung on to the five-step method and used it all the time. Students in the middle used the method some of the time and not others, while the brighter students expanded on it and went off on their own. But in all cases, the five-step method served as a scaffold, as a temporary support while the students were developing their abilities. [emphasis added]

I find this an ironic comment since I have often written about the English test item in Massachusetts in which 2/3 of all 10^th graders could not identify a reading as an essay because “it didn’t have 5 paragraphs.” It is precisely the paradox of the inflexibility and over-simplification of well-scaffolded novice knowledge that has to be aggressively addressed if understanding (what an essay is as a concept) and transfer (recognizing thatthis is an essay, even though its surface features are unfamiliar) are to occur.

How hard would it be to show weaker students a 3- and a 9-paragraph “essay” as well as a 5-paragraph essay, all on the same topic; and then ask them to explain what makes an essay an essay, regardless of surface structure? Indeed, this is just the kind of scaffold for inferring a concept that lies at the heart of teaching for understanding: concept attainment and meaning-making via examples, non-examples, and guided inferences – mindful of prior learning experience (and likely misunderstanding). Not at all the same as “discovery learning” and hit or miss “projects.”

Yes, the research is clear: direct instruction is better than "discovery learning" when the aim is brand new unproblematic knowledge and skill and when contrasted with "students discovering for themselves core facts and skills." But this is a very cramped argument. And it simply does not follow from it that all important learning occurs through direct instruction or that knowledge = understanding. Indeed, as Plato said 200 years ago, learning for understanding is “not what is proponents often say it is, that is the putting of sight into blind eyes. Rather, it is more like turning the head from the dark to the light…”

PS: Rosenshine offers a very different take on the issue that so motivated Clark, Kirschner, and Sweller, i.e. the link between the practice of experts and the pedagogy that supports developing expertise. He laments our failure to pursue the pedagogical question of how novices become experts:

Rosenshine: One very promising area of teaching research has been to compare the knowledge structures of experts and novices. For example, the experts might be professors of physiology and the novices might be interns or graduate students. Or the experts could be experienced lawyers and the novices were first-year lawyers.

What the researchers consistently found was that the experts had more and better constructed knowledge structures and they had faster access to their background knowledge. These findings occurred in diverse areas such as in chess, in cardiology, chemistry, and law. They also compared expert readers with poor readers and found that the expert readers used better strategies when they were given confusing passages to read.

A lot of expert-novice research was done from the mid-1980s until about 1992, but then it stopped. I would have hoped they would have gone on to ask questions such as, "What sort of education should novices go through in order to become like experts?" and "What does creating expert knowledge mean for classroom instruction?"

But, unfortunately, the research was never used to develop an instructional package for training experts. It was never used to establish instructional goals for classes to teach all children to be like the experts. Our goal should be to develop experts, and we're not doing it.