Roger Clarke's Web-Site

© Xamax Consultancy Pty Ltd,  1995-2024
Photo of Roger Clarke

Roger Clarke's Knowledge-Based Apps

Knowledge-Based Expert Systems:
Risk Factors and Potentially Profitable Application Areas

Roger Clarke

Version of January 1989

© Xamax Consultancy Pty Ltd, 1989

This document is at http://www.rogerclarke.com/SOS/KBTE.html


Abstract

Expert systems technology appears to offer a great deal of potential for many organisations. However, like any new technology, it is risky. Some organisations are by preference or necessity risk-takers, but for most managers it is important that some a priori economic evaluation of proposed expert system applications be undertaken.

This paper discusses risk factors which militate against the successful application of KBT to business problems , grouping them into questions concerning the existence and nature of the knowledge, the nature of the domain, the existence and characteristics of the domain specialists, aspects of knowledge engineering, characteristics of the user, and some extraneous factors. Characteristics of potentially profitable application are identified. The business need is seen to be for 'complementary' rather than 'artificial' intelligence.


Contents


1. Introduction

The class of software commonly called 'expert systems' has been attracting a considerable amount of interest in recent years. Better described as Knowledge-Based Technology, it comprises a set of tools and techniques whereby 'knowledge' is expressed directly to the machine, usually in the form of rules, rather than being first converted into data definitions and procedures. Not only can 'hard' knowledge be trapped into the computer, but so also can loose (but useful) knowledge such as 'heuristics' or rules of thumb. In addition, the problem-solver no longer needs to think down at the level of the procedures and data which underlie the knowledge, and can therefore cope with more difficult problem-domains.

The kinds of techniques and tools involved can be classified into knowledge acquisition (including rule-induction and other machine-learning models); knowledge representation (including various models of semantic networks such as order-attribute-value triplets and frames, together with production rules; inheritance; plausible reasoning; and logic programming); and inference procedures (including data-driven forward-chaining and goal-directed backward-chaining, depth-first and breadth-first search strategies, and non-monotonic reasoning).

This paper assumes that the reader already has an understanding of the technology (see, for example, Elam & Henderson 1983, Hayes-Roth et al 1983, EDP Analyzer 1985, Ford 1985, Harmon & King 1985, Luconi et al 1986, Clarke 1989). Exhibit 1 provides a model of the development and use of KBT-based software.

Many large organisations have experimented with expert systems and some have invested significant sums of money to develop expertise. A few applications embodying KBT have entered into operational use, particularly in the scientific and medical arenas, but also in business (Buchanan 1986, von Weissenfluh 1987). Software houses are beginning to offer packaged applications. Although many of these packages give the impression of being designed more to advertise the seller's expertise than as serious products, it does appear that the private sector is giving at least qualified support to claims that the technology is now commercially exploitable.

Whether KBT is really ready to be applied depends on the benefits which may be gained and the costs and risks involved. For a manager to reach a conclusion on KBT's applicability to him and his organisation, he needs a framework whereby he can subject specific proposals to financial assessment.


2. Economic Evaluation of KBT Applications

There is a conventional sequence through which technological development of any kind proceeds (seeExhibit 2).

Exhibit 2: Stages of Technological Development

Stage	  					Motivation

RESEARCH
Basic or 'Pure' Research			Because
by tertiary institutions	    		it's there

RESEARCH & DEVELOPMENT	
Strategic, Pre-Competitive R&D	   	Reasonable
by governmental research organisations,	hopes of future
contracted private research organisations,	commercial
and tertiary institutions, sometimes in	exploitation
association with innovative companies

INDUSTRIAL RESEARCH & DEVELOPMENT
Prototype Products and Trial Applications	Expectation
by governmental research organisations,	of a future
contracted private research organisations,	saleable
and by innovative private companies, often	product
in joint ventures with tertiary institutions
and/or government research organisations

PRODUCT DEVELOPMENT
Products and Practical Applications		Strong
by innovative private companies,		expectation of
sometimes inassociation with			a profitable
tertiary institutions or			product
government research organisations

It is debatable how far along this path KBT has progressed. However claims have been made, not just by marketing interests, but also by well-respected researchers, that it is quickly graduating into a commercially exploitable technology, and that at least some classes of real-world problem can already be practically and profitably addressed (e.g. Debenham 1983, Michaelsen & Michie 1983, Reitman 1984, Blanning 1985, Waterman 1985, EDP Analyzer 1987, Quinlan 1987. For more imaginative promotional literature, see Feigenbaum & McCorduck 1983 and some of Minsky's widely publicised interviews).

During the early stages of technological development, the criteria against which evaluation is undertaken are largely of a technical nature, and most such evaluation is undertaken ex post facto (e.g. Hayes-Roth et al 1983, Liebowitz 1986). This is entirely reasonable, since during the delicate youth of a new technology, narrow accounting criteria would unduly fetter the imagination.

The governments of all major countries provide substantial funding for research activity directed at problems of national strategic importance. Such projects are commonly justified by assertions of future benefits by one or more scientists or technologists in good standing. Given the uncertainties inherent in such activities, this funding process is effective and desirable. However, as KBT begins to be applied in earnest, the costs and benefits of an intended investment must be assessed in advance rather than after the fact, and the evaluation must be concerned less with the technical considerations themselves than with their economic implications.

Gaschnig et al (in Hayes-Roth et al 1983) considered that "only a few AI systems are beginning to reach [the product development] stage in systems evolution ... Ultimately, ... the marketplace judges the cost-effectiveness of a product". The approved capitalist mechanism is for companies pursuing successful strategies to enjoy (short-term) super-profits, and for the rest to encounter financial difficulties, in some cases of a terminal nature.

For a commercial organisation, the acquisition of KBT capability requires significant capital investment and elapsed time. Since most companies tend to be risk-averse, they prefer to gamble only at the margin, and seek information prior to making a significant commitment to a new, unproven technology. They have a responsibility to their shareholders to use the limited resources at their disposal in an effective and efficient manner, and must justify the use of new techniques. Companies can therefore be expected to increasingly subject proposed applications of KBT to some appropriate form of economic evaluation. Only by distinguishing its real advantages will C.I.O.s and Information Systems Managers be able to clearly identify priority applications.

These important matters appear to have attracted relatively little serious consideration in the literature to date (see, however, Johnson 1984, Smith 1984, Marchand 1985, Prerau 1985, Connell 1987, Connell & Powell 1987, EDP Analyzer 1987 pp.13-14 and Jenkins 1987). There is as yet no clear framework within which a priori economic evaluation of proposed applications of KBT can be undertaken. This paper addresses that need, by identifying, classifying and discussing firstly the major risk factors involved, and secondly the potential areas of profitable application.


3. Risk Factors for KBT Applications

Many difficulties have been encountered in the development of the new technology. Some of these have been overcome, some are being actively addressed, and many remain. This section presents a catalogue of warning signs which an organisation should consider when assessing a potential application. Some of them are of the nature of feasibility factors or disqualifiers, while others are cost factors, which tend to decrease the attractiveness of a technically feasible application. The contention is not that applications to which these reservations apply should not be undertaken, but rather that the investor should give explicit consideration to their implications for the project's feasibility and economic worthwhileness.

Clearly, since KBT applications are a sub-set of application software generally, many of the factors discussed below also arise in projects undertaken using conventional technology. In many cases, however, the risks involved are different, or are exacerbated. The factors are considered in several groups, and are summarised in Exhibit 3.

Exhibit 3: Risk Factors in the Application of KBT


3.1 The Nature of the Knowledge

The first cluster of factors have to do with the knowledge about the problem-domain. It is a fundamental requirement of a potential application that knowledge exist. So, for example, telepathy might not be a suitable area of application. In addition, the knowledge must be of a kind that can be coped with by digital computing technology. Such expertise as food- and wine-tasting cannot be supported by contemporary technology, because crucial data and/or criteria are defined only in human sensory or entirely 'subjective' terms. In the context of KBT, an operational meaning for 'objectivity' might be "expressibility in terms of an external cardinal, ordinal or at least nominal scale". An explicit nominal differentiation is sufficient for a conditional rule to be specified.

In addition, the knowledge must be able to be captured into a knowledge-base. One common pre-processed knowledge-form which is at least human-readable, and increasingly readily machine-readable, is encyclopædias, text-books and other materials comprising (type-)written text, diagrams and pictures. However, experience to date suggests that such sources will not be by themselves a usable source of knowledge until firstly automated understanding of natural language is achieved, and secondly a machine can be provided with a broad understanding of fundamental concepts about the world to which the knowledge relates.

Another form in which knowledge may present itself is as a collection of cases. Such information must comprise both descriptive data, and also interpretive information whereby the outcome (diagnosis, prognosis, recommendation, etc) can be associated with the descriptive data. The automated inference of rules from case data is addressed by 'knowledge acquisition' theory and 'rule induction' software.

KBT currently offers a variety of approaches to developing deterministic models of problem-domains which have previously resisted solution. Digital's much-quoted R1/XCON is a major instance. Well-defined mathemetical systems which evidence the property of invertibility, such as Dendral and Macsyma, can be implemented with confidence, because a strong theory underpins them. On the other hand, the mathematics of uncertainty, or 'plausible reasoning' (e.g. O'Neill, 1987) itself remains uncertain: consider for example the long debate over the elegance and the efficacy of Mycin's 'certainty factors'. Non-deterministic models are therefore an important risk factor, and this despite the fact that human knowledge in many domains is so uncertain as to demand them.


3.2 The Nature of the Domain

The nature of the problem-domain also poses difficulties for KB technologists. It is necessary that the domain be sufficiently well-defined and bounded that the knowledge engineer be able to commence with, or at least develop, a clear conception of the classes of problem which are to be addressed, and equally importantly those which are not to be addressed. For example, medical disorders can be addressed at several levels of abstraction, based respectively on physiology, biochemistry and molecular biology; and rather different knowledge-bases are required to support the evaluation of loan applications received from employed individuals, unincorporated businesspeople, partnerships, and family companies. At the other extreme, since the strengths of KBT lie in symbolic reasoning rather than numeric computation, most structured decision-making is probably better addressed, at least at this stage, by conventional data processing technology.

It is necessary that there be a sufficient degree of stability of the domain phenomena. For example some marketplaces are sufficiently dynamic that each new decision is undertaken on largely pragmatic grounds, and these change too quickly for a computer-based system to keep up. Similarly, KBT may not be of much assistance in dealing with a quickly-mutating flu virus, because the delay in collecting and interpreting new data may be such that by the time the knowledge has been formulated, its brief period of usefulness has expired. Information Economics provides a basis for analysing such circumstances.

It is now conventional to distinguish between 'surface' and 'deep' knowledge. 'Surface' or shallow knowledge is derived by correlation or from intuition and experience, and the rules tend to incorporate very limited semantic content and explanatory capability. To be able to reason causally and instrumentally, a KB application must be based on a systemic model. At least for important applications, surface knowledge-bases will therefore need to be supplemented by 'deep' knowledge, embodying what might be variously termed 'causal', 'structural' or 'systemic' reasoning. Only in this way can the heuristics in the surface knowledge-base be subjected to critical consideration.

Another risk factor is domains in which 'common sense' plays a significant role. Some KBT applications are capable of reaching decisions which a (trained) human would regard as laughable, because although they may be theoretically sound, they are not practical or sensible. In addition to domain knowledge, such knowledge-bases must also embody some amount of general-purpose 'world knowledge'. However, some authors argue that even this is not sufficient, and that much human common-sense involves not only general knowledge of the world, but also skills in the recognition and application of patterns.

The domain knowledge must also be expressible, in the sense of being able to be conveniently and accurately mapped onto some formalism. The main research languages, LISP and Prolog, offer enormous scope for expressing knowledge, but they demand considerable technical expertise, and involve significant development and maintenance productivity penalties. The enormous growth in specialised languages and so-called 'expert systems shells' promises much higher productivity and modifiability, at the cost of flexibility and expressive power.

A final consideration is entry barriers against new domain-specialists. The conventional 'market forces' argument is that shortages for any commodity tend to be overcome by the market price being bid up, and the higher price attracting new entrants into the field. A variety of factors can cause market forces to work very slowly, or can prevent them working altogether. Some of the entry barriers relevant to KBT are further considered in section 4.3 below.

Entry barriers may be a risk factor for KBT applications if they are either to high or too low. At one extreme, barriers which are effective against individuals may also be effective against KBT. On the other hand, if they are too low, then training more people may be a more effective solution than developing a KBT application, or, alternatively, late entrants may steal the market from the pioneers, by developing superior, 'second-generation' KBT applications.


3.3 The Domain Specialist(s)

In contemporary KBT, the most common approach to knowledge acquisition is the collection and codification of knowledge directly from humans. In many cases this is the prime, or even the sole, source. In others it may be a secondary source, used to clarify, expand or test knowledge which was acquired in some other way, such as from written rules (e.g. statute law or a corporation's standard 'procedures') or by applying inductive inference procedures to case data.

Clearly, the collection of human knowledge is predicated on the existence of one or more humans with expertise in the given domain, but there are a number of other necessary conditions for a domain to be appropriate for KBT. One is that the relevant human domain-specialist(s) must be available to the team developing the software. A common circumstance in which this may not hold is where he or she is too heavily committed to other activities, commonly to the application of the very knowledge which is to be captured into a knowledge-base. Other possible difficulties might be that the person is no longer employed by the organisation (or never was), and declines the invitation to participate.

There may well be legal difficulties in requiring a person to divulge his expertise (Clarke 1988). This applies whether the person is an employee, a contractor or a consultant, and even if the knowledge has been developed entirely in the context of one particular organisation.

It is important that the domain-specialist(s) be eager, willing, or at least amenable to take part in the project, not just in principle, but also in practice, through the long haul of preliminary discussions, education of the knowledge engineer, assessment of the rules the knowledge engineer formulates, and testing of the knowledge-base against known cases. Behavioural difficulties will arise. In the early days of KBT it appears that most (reported) domain-specialists have taken it as a compliment that their knowledge is sufficiently important that the organisation is prepared to invest heavily in a highly paid, highly academically qualified person to capture it. As KBT becomes routinised, this 'fishbowl effect' will wear off, and some domain-specialists will be variously daunted by the prospect, or insulted by the time it took before their knowledge was deemed important enough to capture.

Beyond being amenable, the domain specialist(s) must be able to express the knowledge in a manner which the knowledge engineer can understand. This is no trivial matter, since most specialists have hitherto sensed or imaged a great deal of their knowledge, and never expressed it outwardly, especially in the exhaustive and precise manner in which the knowledge engineer seeks to capture it.

There is also the question of the stability of the domain specialist's knowledge, by which is meant the extent to which the same question or case, presented on different occasions, elicits the same response. There may also be a problem of internal inconsistency among the rules expressed by the specialist or adduced by the knowledge engineer. For example, particularly in large knowledge-bases, several rules may be expressed in such a way that they lead to different conclusions, which are (or logically should be) mutually exclusive.

In the development of 'community knowledge bases' (Bobrow et al 1986 p.893), where more than one domain specialist is involved, inter-specialist inconsistency may arise. This may be because the knowledge of a single domain is shared across several people (typically because each has experience of a sub-set of, for example, the customers, geographical areas, or product groups). Or it may be because the application is intended to straddle two related domains, and essentially requires the knowledge of two specialists to be combined. In less common cases (at least as far as can be seen in the literature to date), it may be the intention of the organisation commissioning the application to reconcile the, in part conflicting, expertise of two or more specialists in the same domain. A tempting, though probably futile, application would be macro-economic policy formulation.

Frequently, apparent instability and inconsistency arise from a perceived difference in context, and careful engineering results in rule refinement, or the detection and expression of additional rules. However some instances of instability and internal inconsistency are evidence of error and/or illogic in the knowledge of the (at times all too human) expert. And some inter-specialist inconsistencies are evidence of irreconcilable differences of opinion between experts.


3.4 Knowledge Engineering

A further cluster of potential problems relates to the process of knowledge elucidation and capture. One is the shortage of suitably skilled knowledge engineers (the so-called 'Feigenbaum bottleneck'). Members of this fledgling profession require a wide variety of both technical and human skills. They must also have the patience and commitment to bring potentially very long projects through to completion, since the loss of a knowledge engineer is likely to have serious repercussions on the whole project. In the early stages of the technology, computer science has been crucial educational background for knowledge engineers. However, as general-purpose tools are developed to support the knowledge-base implementation phase, other skills are becoming of greater importance, and the field may quickly merge into a re-defined systems analysis profession.

To be useful in solving business-related problems, a knowledge engineer needs a broad business-related education, and knowledge and experience in such techniques as interviewing, knowledge acquisition, knowledge representation and testing. In addition, he or she must develop an adequate understanding of the problem domain. Given an effective combination of domain specialist and knowledge engineer, this need not be an insuperable barrier. However, it does demand the investment of time, firstly by the knowledge engineer to familiarise himself with the fundamental concepts and terminology of the area, and later by both knowledge engineeer and domain specialist, and therefore raises the costs of knowledge-base development. In some cases, of course, the same inherent difficulties of the problem-domain which cause the shortage of domain specialists in the first place will cause considerable problems for the knowledge engineer.

A further problem is whether an appropriate formalism exists to deal with the particular constellation of data and entities that exist in the particular problem-domain under study. In principle, some language features map onto one another, although with varying degrees of convenience, efficiency and reliability (e.g. O-A-V triplets and frames). However some capabilities are distinctly different from one another (such as forward- and backward-chaining; monotonic and non-monotonic reasoning; and the various approaches to the handling of uncertainty).

Given that an appropriate knowledge representation technique exists, the question arises as to whether the knowledge engineer will recognise it, and whether he will have, or be able to acquire, the requisite skill in using the formalism.

A further question is whether a machine-translatable language which implements the formalism exists, and if so, whether it is available in a suitable hardware and systems software environment.

The testing of KBT applications has attracted little consideration in the literature until very recently. It is a sign of the discipline's immaturity that the naive terms 'validity' and 'verification' are currently more commonly used than such software engineering terms as 'testing' and 'quality control'. In many circumstances it will be very difficult to assess the application. As a minimum it might be expected that such cases as were used in inferring the rules would be run, and the results compared against expectations. In many cases of course, given the complexity of the knowledge-base, the result may not be the same as was documented, and an examination of the explanation would be important. In other cases the result might be the same, but the manner in which the software reached the conclusion might contain surprisal value.

Another area of concern relates to knowledge-base maintenance. In many applications, the need for significant modifications will arise, sometimes because of environmental change, in other cases as a result of ongoing learning by the domain-specialist or the organisation as a whole. During the development stage of a KBT application, the integrity of the knowledge-base design is protected by the professionalism of the knowledge engineer. However after the software is put into operation, the person-dependent nature of knowledge about the knowledge-base ('meta-knowledge'?) means that understanding about it quickly dissipates. Changes may be made in an inappropriate manner because "in the editing process, one isn't always sure what the rationale for all the functions are" (Soleway et al, 1987). Rather than rationalising existing rules, new knowledge may be piled in on top of the existing base, with unpredictable results. The result is what Brooks referred to as 'integrity degradation' - the quality of a KBT application may deteriorate over time, with resultant costs in erroneous decisions or advice, or unanticipated exposure to contingent liabilities to third parties.

An investor must take into account the immaturity, and highly dynamic nature of KBT. The increasing recognition of the importance of knowledge-base maintenance is resulting in more attention being given to standards, to needs-driven rather than tool-driven methodology, and to 'software engineering' factors like knowledge-base modularity, structure mapping, and life-cycle support tools. Hofstadter (1985) has gone so far as to claim that AI and KBT have an inadequate scientific basis, and are built merely on a number of recipes whose success has been limited to domains whose boundaries are not yet understood.


3.5 The Users

The extent to which an investment in software will be profitable depends very heavily on the application's usability. A variety of aspects of the use of KBT-based software may undermine a project's attractiveness. Fundamentally, there must be some doubt regarding the acceptability of KBT-based software by workers, both as a matter of principle, and in practice. At the extreme, there may be Luddite reactions, with workers expressing serious fears about job destruction. The extent to which these concerns can be dealt with will depend on the industrial relations climate and mechanisms which prevail, including the extent to which trade union officials and informal workplace opinion-formers recognise and understand the technology's implications.

If the claims of computer scientists are to be believed, then the concept of humanness is being once again challenged, just as clockwork, steam technology, Darwinian evolutionary theory and Skinnerian response conditioning changed our self-conception (e.g. Bolter 1984). Some employees may therefore feel a deep-seated moral revulsion against the notion of machine intelligence.

Experience with conventional application software has shown that people are more likely to react favourably to it if they have been involved in its development. There is some evidence that the most superior specialists in many domains are pleased to delegate much of their thinking to a machine. However difficulties may arise in convincing other people to apply the resulting product. At the clerical and operational level, staff may accept 'orders from a machine' just as readily as they do from human managers, particularly if there is at least some nominal opportunity for them to argue with it, put counter-proposals to it, or over-rule it. However, organisations should anticipate at least some negative behavioural consequences (e.g. Laulan, 1985).

The greater difficulties are likely to arise with professional and managerial staff. Professional people (actuaries, financiers, insurance assessors, engineers, architects, solicitors, public accountants, etc) generally regard the exercise of their professional judgment as being critical to their occupation and to their self-esteem. Genuinely expert applications are likely to find little success in such circumstances, and even adviser applications are highly likely to be resisted. Approaches to avoiding such reactions are the (at least symbolic) involvement of all affected people in the design process; depiction of the knowledge-base as a professional colleague, or as an assistant which has been trained by a colleague; and delicate handling of contentious rules, and of differences of opinion among domain specialists.

Significant advances have been made in user-interfaces for conventional transaction-data processing and MIS software, but the challenges are far greater with inherently less structured KBT applications. Coping with human differences may be critical to the success of many KBT-based applications. Differences of particular importance include the discipline, profession or trade in which the user has been educated and trained, the person's socio-economic level or class, his or her language(s), regional and occupational dialects, the ethnic cultures from which the person comes and in which he or she lives, and separation between the development team and users both in time and space. "Designing software for persons outside the research laboratory imposes a discipline on AI that it has not had to face in its early, formative years" (Buchanan, 1986, p.38).

Beyond the user interface lie questions of the user's knowledge of the the problem domain. A first issue is user recognition as to when the application should be consulted, and, also importantly, when it should not. This requires that the user have an adequate appreciation of the software's scope and limitations.

A related problem is that the provision of case-specific data to the KBT application is dependent on an effective dialogue between user and software, to ensure comprehension by the user of prompts, conclusions (including the force of any qualifications), and explanations. This requires the user to have a domain-model and vocabulary consistent with that of the development team, even though, for many classes of user, these must be greatly simplified. "Current systems often exhibit ... brittleness ... because they make strong assumptions about the context in which they will be used, the types of users, the vocabulary, the 'reasonableness' of other lines of reasoning, and so forth" (Buchanan, 1986, p.39).

A poorly designed adviser application, or one used by a person inadequately trained, or insufficiently sceptical of the tool, will adopt the status of a de facto genuinely expert application. Conversely, if a decision made by genuinely expert software is intercepted by a human and considered rather than being acted upon, then the software is being used as a de facto adviser. In both cases, poor quality decision-making should be anticipated.


3.6 Additional Factors

This section identifies some further factors which may have a negative effect on the return on investment in KBT. As with with any new investment requiring time to mature, the commitment of senior management is essential, to the technology as a whole, and to individual projects. Otherwise, as problems arise, delays occur and expectations are lowered, there is a risk that individual projects and even the entire investment may be abandoned just as the breakthrough is being neared.

There is also the question of property rights in the KBT application. Intellectual property protection for software is fraught with difficulties and uncertainties. Copyright, the most relevant form, generally accrues to an employer in respect of work performed by an employee. However, copyright in work undertaken by contractors may, depending on the terms of contract, accrue to the contractor rather than the sponsoring organisation. It is also possible that a domain specialist might have claims.

Such concerns are predicated on the assumption that software is subject to copyright. Since copyright conventions are very broadly expressed, each country's copyright laws are their own affair, and hence software, or KBT applications, may be treated differently in each country. Only in the U.S.A. has it been established in the courts that software is a suitable subject for copyright, and that was only in 1982-83 and based on a 1980 amendment. In Australia, amendment legislation was necessary in 1984 to reverse a negative finding by the ultimate appeal court, while in the U.K. an amendment was passed in 1985. In neither case has the amended statute been considered by a court, and the question of origination of the software in a form not visible or otherwise perceptible to a human (e.g. using word processing software, a text editor or a graphics-based CASE product) has not yet been brought before the courts. In most major countries software may reasonably be assumed to be copyrightable, but some doubt remains.

There is also the question whether KBT applications are software. Where a KBT application is implemented using conventional languages and data management software, it probably enjoys the same level of protection as other software. However, where a knowledge-base is created as a separate entity, it is not software in the conventional sense of a sequence or set of 'instructions', but rather a set of 'rules' expressed in a manner suitable for processing by a particular kind of run-time interpreter called an 'inference engine'. It is not clear that courts will construe knowledge-bases to be software for the purposes of copyright. If not, then they might be construed to be some other type of work to which copyright applies, or they might be found to be uncopyrightable. These matters are considered in greater detail in Clarke (1988).

Consideration must also be given to the implications of errors resulting from KBT applications. Subject to important qualifications concerning adequate care in both development and use, it does not appear likely that the use of an adviser application need materially alter an organisation's liability in law. However, an intended or a de facto 'genuinely expert' application could give rise to new liabilities to third parties. Examples might be inappropriate quality control during development or maintenance resulting in unreasonable refusal of credit, rejection of a commercial transaction such as car hire, refusal to employ, dismissal from employment, denial of a government benefit, causing of an industrial accident, or failure to prevent an industrial accident.

Prima facie, such a third party liability would appear to be to the cost of the organisation using the software. It could be, however, that that organisation might have a case against one or more people or organisations who supplied a product or provided a service to them, in particular as a knowledge engineer, but perhaps also as a domain specialist. These matters are dealt with in greater detail in Clarke (1988).


4. Characteristics of Profitable Applications

The previous section dealt with a wide variety of factors which may decrease the attractiveness of a potential application of Knowledge-Based Technology. Despite these risks, carefully selected and developed KBT applications will prove to be profitable for pioneers, and late adopters will subsequently be forced to apply KBT in order to counter their competitors' advantages.

This section comprises firstly a number of general observations, and secondly a discussion of particular circumstances in which KBT applications are particularly likely to be profitable. These points are summarised in Exhibit 4.

Exhibit 4: Characteristics of Potentially Profitable Application Areas

Some business functions have not been particularly well-served by conventional application software techniques, and many claims have been made that KBT can enable problems to be addressed which have previously resisted solution. However, these assertions are seldom accompanied by any more useful analysis than a reference to Digital's R1/XCON application (which configures VAX computers to fulfil customer orders) as an example. In the absence even of any practical 'rules of thumb', such claims should not be regarded as particularly credible.

A more positive observation is that higher levels of return on investment are to be expected from applications which address one or more of the organisation's Critical Success Factors, and/or capitalise upon existing organisational strengths.

Another is that proper economic analysis demands that a long-term view be taken. Many applications will initially offer only qualitative gains (such as speed, accuracy and auditability of decision-making), and will only provide quantifiable benefits when sufficient time has elapsed for the application to be absorbed, and for requisite changes to occur in the organisation and organisational processes.

In some areas, KBT-based software may be successful in head-to-head competition against humans. Software can have advantages over humans where reliability, pedanticness and auditability are particularly important, whereas humans retain the upper hand when imagination, flexibility and adaptability are crucial factors. The main opportunities for KBT should accordingly be anticipated in well-defined, relatively independent and stable problem-domains.

The remainder of this section considers the dimensions which, on the basis of the limited existing literature, appear to be relevant in assessing the scope for profitable application of KBT. These are:


4.1 The Economic Sector

The potential of KBT applications may be dependent on the particular business sector. Von Weissenfluh (1987 pp.71-81) identified and analysed the business applications recently reported in the literature as being in operation or in prototype. Nearly 60% of operational applications (43/74) were in a limited range of segments of the manufacturing sector. However, the focus which exists in the manufacturing sector was not apparent in the service sector, where 20 applications were scattered across many different segments. Several applications spanned both manufacturing and service sector activities, a further 5 were in the mining and mineral and petroleum search sectors, and a couple in the agricultural and pastoral sector.

Although some gross conclusions might be drawn, it does not appear that significant differences in applicability are likely to exist at this level, and finer analysis is necessary.


4.2 The Business Function Supported

In the manufacturing sector, von Weissenfluh found that 28 of the 43 applications which were claimed to be operational fell into the classifications of equipment configuration (12), maintenance (9) and process control (7). This relatively high concentration appeared to be related to the existence of at least one famous application in each segment, such as R1/XCON. Since process control applications would appear to be functionally similar to traffic control, there may also be implications for the service sector.

In the services sector itself, there are many prototypes in the banking, insurance and other information industries (e.g. Carter & Catlett 1987). In addition, projects have been commissioned to interpret and apply legal regulations, e.g. of government benefits, immigration and building. There are also many opportunities in health care, particularly in the third world and in remote areas of more advanced countries which are relatively poorly serviced by medical professionals.

One clear guideline for selecting appropriate business functions is that knowledge should exist, or be readily expressed, in the form of rules. Cautious organisations will also watch both the literature and the market-place in order to be aware of leading applications, not only amongst competitors, but also in functionally related areas.


4.3 Entry Barriers

According to the the market forces argument outlined in section 3.2, shortages in human domain specialists will tend to be satisfied in due course by the emergence of more human domain specialists, attracted into the area by the financial (or perhaps other) rewards. Unless entry barriers prevent the emergence of sufficient human specialists, the opportunity for KBT applications is limited to areas in which they have a demonstrable advantage over a suitably trained human. However, entry barriers which deter humans will also tend to make KBT payback harder to demonstrate and harder to achieve. The following sub-sections discuss particular entry barriers which are of especial interest.

(a) The Impossibility of Training Additional Humans

Some problem-domains are so intrinsically difficult that very few people are ever able to master the volume of information and/or the complexities. It is in precisely such areas that many of the Leitbild applications were undertaken, such as Caduceus, Dendral, Glaucoma, Mycin, Prospector and Puff. As research projects, these have been most important. Some have seen use as teaching aids. However, few of these applications are reported as being in consistent use either as adviser or genuinely expert applications, and the others do not appear to be used as anything other than KBT research tools (e.g. Hayes-Roth et al 1983, Buchanan 1986, von Weissenfluh 1987).

There are therefore considerable risks involved in such heroic undertakings. The argument is not that extraordinarily difficult problem-domains cannot be fruitful avenues for investment. For some organisations, the chance of using a technological breakthrough to achieve a significant competitive or strategic advantage, the projection of an exciting corporate image and/or intellectual satisfaction and community service will be sufficient justification to invest significant product development expenditure. Many organisations, however, will defer such challenges until they have established a track-record.

(b) The Impracticability of Training Additional Humans

The inherent difficulty of some problem-domains falls short of the impossible, but remains a serious impediment. Another condition which may make it impracticable to train more humans is infrequency of opportunities to develop expertise. With rare diseases and injuries in humans, the institutionalised (and very effective) solution has been for humans to specialise into very precise areas, and for all cases that do arise to be referred to them. The economic mechanism necessary to support such an approach has been more difficult to achieve with diseases in animals (e.g. foot and mouth disease) and plants (e.g. rust in coffee plantations). Other instances of infrequent learning opportunities include fires in oil-wells and various kinds of vegetation, and emergencies in environment and process control systems.

Many organisations are heavily dependent on the expertise of a small number of people, and considerable value may be placed on capturing the knowledge of such employees, particularly prior to their retirement. It may not be necessary for the knowledge-base to be able to perform, or directly assist a less skilled person to perform, the particular function. The payback may arise from transmission of the knowledge to future generations of humans, by providing them with vicarious experience.

A special class of this entry barrier is where existing domain-specialists combine to protect their conditions of service, their ability to charge high fees for their services, and/or the standards of the profession. A professional body may constrain members from communicating their expertise to others (e.g. by requiring a long apprenticeship), their case material may not be accessible or understandable without their assistance, and/or legislation and licensing regulations may preclude non-members of the clique from practising the arcane art. Although constructed as a protection against human competitors, such barriers may be equally as effective against computer-based expertise.

(c) The Impracticability of Using Human Expertise

The first two sub-sections dealt with barriers relating to the learning or development phase. In addition, computer-based applications may have advantages over humans at the point at which the expertise is used.

This may arise because of the dangerousness, unpleasantness or remoteness of the working conditions. It may not only be difficult to find suitable people willing to work in the conditions, it may even be illegal to put them there. The enhancement of industrial controls is therefore likely to be attractive, particularly in relation to unpleasant and/or continuous chemical processes. Examples include concrete mix specification (Miller 1985) and lead/zinc smelting (Brew & Catlett 1986).

Opportunities in hazardous conditions can be readily found, such as the cleaning-up operations at industrial disaster sites such as Windscale, Seveso, Three Mile Island, Bhopal and Chernobyl, or any number of chemical effluent accidents and oil spills. Emerging and speculative possibilities include mining, defense and scientific research applications in deep waters, the antarctic and space. Clearly, such uses will be most effective where undertaken in conjunction with robotic technology.

It may be no accident that the only effective non-research use of many of the pioneer KBT applications has been in education. As society's dependence on complex technology increases, so does the sheer volume of post-secondary education, training and re-training. The availability of practitioner-teachers soon encounters practical and economic bounds, which the various forms of Computer-Based Training have not yet satisfactorily overcome. As a result, Intelligent Computer Aided Instruction offers potential as an application area for KBT, always assuming that the people wanting the services have access to sufficient funds to pay for them.

Another kind of impracticability is where the circumstances of decision-making are such as to place excessive demands on a human, e.g. because of the very short time available to make a decision, the decision's gravity, or the number of decisions which need to be made in a short time. A particular class of circumstances where the use of humans may be impracticable, emergency-handling, is treated in the following separate section.

(d) Emergencies - A Special Case?

Software to deal with emergencies must have particular characteristics, including execution-efficiency, and a high standard of quality assurance. It must be integrated with the management information system which deals with normal operations, and have adviser capabilities, in order to support a human decision-maker during the initial stages after a problem has been recognised. However, it must also have genuinely expert capabilities, since in some circumstances it may have to act in the absence of any human instructions. This may be because there is no human present, or the human is overcome by the enormous importance of the decision, or overloaded by the flood of information, and might as well not be present. Alternatively it may be because the situation has 'gone critical' and 'real-time' action is the only remaining chance of avoiding disaster, even if, by the very nature of KBT, there is no guarantee that the application will be able to cope.

Examples of such opportunities are in the control of potentially hazardous or pollutive industrial processes, environmental control in buildings and mines, traffic control (e.g. diverting traffic around the scene of an accident), and public health, (e.g. coping with outbreaks of life-threatening disease, particularly in remote areas). The challenging aspect of such applications is that they must comply with high standards of design, construction and implementation, both as adviser and as genuinely expert applications.


4.4 Frequency of Opportunity

Sometimes a course of action is vastly superior to the alternatives, either because it generates a far greater stream of income or benefits, or because the financial or other costs are much lower. Detecting these occasional opportunities is vitally important to an organisation's success, and a KBT application which promises to be instrumental in doing this represents a high return-on-investment proposition.

Such circumstances are uncommon. However, there are many instances where opportunities for low levels of payback arise sufficiently frequently that the cumulative value of that payback may be sufficient to justify the investment. This arises particularly where operations must be performed routinely in continuous, around-the-clock operations (e.g. equipment fault diagnosis) and in dispersed locations (e.g. the interpretation of statutory regulations). Similarly, there are many repetitive tasks performed by clerks processing paper-flows, and counter-clerks processing verbal enquiries and applications. It appears that the large majority of current business applications seek their payback via cumulative benefits from continual use.


4.5 Class of Person Supported

Each KBT application must be targetted at a particular kind of user. The following classes can be distinguished, although the boundaries are to some degree arbitrary:

An analysis of von Weissenfluh's data into these classes is informative. It suggests that over 80% of published applications fall into the professional and operational areas, with applications for middle management being the largest remaining classification. The vast majority of applications appear to be used as advisers, with only 2 of the 192 considered claimed to be used exclusively as 'genuinely expert' applications. The following sub-sections consider each of these classes in greater detail.

(a) Members of the Public

Several applications are already in operation, advising members of the public on such matters as the applicability of agricultural products and government benefits, and bus, train and plane networks and timetables. Such applications will naturally be used as 'promotional gimmicks', but at least some will have more direct commercial value. When members of the public come to an organisation's front-counter or telephone service, the better-informed they are, the less time they need from the organisation's employees, and the higher their perception of the quality and speed of service the organisation offers.

Such applications may be delivered via the increasingly available public terminal network (such as Automated Tellers, Point-Of-Sale devices and in particular library catalogue access points), and readily integrated with home information service networks such as teletext and cable television.

Broader educational uses at primary, secondary and tertiary level can also be readily envisaged. Such services would represent worthwhile promotional opportunities for organisations who wished to establish and maintain the image of being 'as close as your television set'.

(b) Operational Workers

Several distinct areas of application have been attempted. One is particularly applicable to industrial processes which are poorly understood. This may be because the underlying physical science is not sufficiently advanced and/or because investment in better knowledge (such as chemical analysis of low-cost materials) cannot be cost-justified. Such KBT applications involve the capture of the knowledge of one particularly skilled worker, or the pooling of the knowledge of several, and the provision of the knowledge-base to all workers.

Maintenance activities may be supported in a variety of ways. One approach is to augment existing approaches to monitoring processes in, for example, industrial plants, power production and traffic. Related to this is the intelligent analysis of exceptional conditions, and the prompting of the human controller in the investigation of the problem.

As implied in sections 4.3(c) and 4.5(a) above, training of operational staff can also be addressed by KBT. Some organisations will find value in training staff in normal operational tasks, while others will achieve greater payback from the simulation of exceptional conditions.

(c) Professionals

Section 3.5 suggests that there are difficulties in developing KBT-based software to support professionals. However, an analysis of von Weissenfluh's data suggests that close to half of both operational and propotype applications are intended for use by professionals. About half of those applications were intended for engineers of various kinds, while the remainder address the problems of a wide variety of professionals including petroleum and mineral search, medical, legal, law enforcement, application software, finance and insurance specialists and aircraft pilots.

Such applications provide payback partly be ensuring that fewer mechanical and computational errors are made, but mainly by relieving domain-specialists of some of their more tedious tasks, and leaving them free to concentrate on tasks where their experience and judgment cannot be codified. Where the creativity of these domain-specialists is a Critical Success Factor for the organisation, this contribution may be highly (if unmeasurably) profitable.

Luconi et al concluded that "for many of the problems of practical importance in business, we should focus our attention on designing systems that support expert users rather than on replacing them" (1986, p.9). Those authors used the term 'Expert Support Systems' (intended as a hybrid of Decision Support Systems and Expert Systems), but this risks compounding the confusion that already exist surrounding those terms. The term 'Professional Support Systems' would be less pretentious, more descriptive, and less liable to confuse.

Another author refers to the 'progressive restructuring' inherent in KBT (Sviokla 1986). This is the ongoing process whereby the development and use of KBT applications alters the nature of the underlying task and the organisational mechanisms which support it. The tendency is to move the problem along the 'structuredness' dimension, away from dependence on expert or professional judgment, toward better-understood, more teachable and more programmable decision-making.

(d) Middle Management

Many of the comments made in the immediately preceding sub-section also apply to middle management, although the knowledge of middle managers is by definition more generalised than that of professionals. Of the 10-15% of applications reported to date as supporting middle management, the majority are in production planning. Scope also exists for integration with model-based simulation of the likely effects of alternative tactical measures.

Another major area that appears to be attractive is 'Intelligent MIS', referred to in the Decision Support Systems (DSS) literature as 'information attenuation'. This involves the review of potentially vast quantities of transaction-based data processing reports, in order not just to detect the manifold exceptions but also to assess their significance in the current context, and present them for consideration in priority order.

(e) Top Management

Semi-structured management decision-making has already been subjected to study under the heading of DSS and recently under other rubrics such as Executive Information Systems. A few authors have considered the relationship of KBT to the existing body of DSS theory (see Elam and Henderson 1983, Ford 1985, Holsapple & Whinston 1985, Sen & Biswas 1985 and Pfeifer & Lüthi 1987). One well-known commercial publication even went so far as to characterise expert systems as 'intelligent decision support systems' (EDP Analyzer 1985).

However, KBT's applicability to the support of senior management decision-making is subject to some important qualifications. For example, KBT cannot yet deal convincingly with uncertainty, nor with perceptual data, nor with "subtle evaluations of people's competence and motivation" (Luconi et al, 1986, p.9), nor with competitive and strategic thinking. That very few operational products or even prototypes have been reported in this area is therefore no surprise.

However, there is scope for KBT-based applications to make some important contributions in the development of DSS theory. Moreover, even where no software product results, or is not even likely to result, the techniques of KBT may still have considerable value for senior management. As with free-wheeling discussions and brain-storming, knowledge-acquisition techniques are interaction mechanisms which help to unlock individual and group creativity, and elucidate structure and knowledge that was previously hidden or did not exist. KBT application prototyping can provide prompt feedback to the individual or group, and lead to further rounds of suggestions and insights. KBT may therefore become a valuable 'instrument of discovery' in the emerging area of Group Decision Support Systems (GDSS).


5. Implications

This paper has provided a structured list of risk factors militating against KBT application, and a discussion of key characteristics of KBT applications which are likely to determine their profitability.

Further research is required in the following areas:

Such research is presently hindered by coyness on the part of many corporations which are applying KBT. To the extent that they are undertaking genuine Product Development, they are justified in hindering publication of the results until they have gained their desired competitive advantage. However, where they are undertaking Industrial Research and Development, their community and hence they themselves stand to gain from mutual publication of results, both of successful and unsuccessful projects.


6. Conclusion

Knowledge-Based Technology will find profitable application in a variety of circumstances. However the payback from KBT applications is constrained by many factors, and most organisations will give careful consideration to both the promise and the risks before committing large sums of money to applying the technology to business problems.

The major advantages which computer-based expertise can bring is where reliability, consistency and pedanticness are important considerations. It will be commercially advantageous to deflect the focus away from the somewhat unnecessary notion of 'artificial intelligence', toward the concepts of 'complementary intelligence' and 'silicon workmates'. We will be commercially (and probably also philosophically) better served by conceiving our machines to complement human strengths and weaknesses, rather than to compete with them.

The long list of risk factors suggests that there may be many circumstances in which the capture of human expertise is of intellectual interest, and may be of assistance in education and training, but offers little or no real prospect of profitable application. Naturally, further developments in KBT may well, over time, qualify, and to some extent even invalidate, this conclusion.

One of the most significant contributions of KBT is to enable domains to be addressed which have resisted previous analytical approaches. The new analytical techniques, developed to support the construction of knowledge-bases, may prove to be a more important contribution than the knowledge-bases themselves.


Acknowledgements

The assistance of colleagues at the Australian National University and the Institut für Wirtschaftsinformatik, Universität Bern was greatly appreciated, as were the comments of attendees at seminars at the Universities of Southhampton, New York, Denver, Minnesota and Zürich.


References

Barr A. & Feigenbaum E.A. (Eds.) 'The Handbook of Artificial Intelligence' Vols. 1 and 2 Kaufman 1981, 1982.

Blanning R.W. 'Expert Systems for Management: Research and Applications' J of Info Sci 9 (1985) 153-162.

Bobrow D.G., Mittal S. & Stefik M.J. 'Expert Systems: Perils and Promise' Comm ACM 29,9 (September 1986) 880-894.

Bolter J.D. 'Turing's Man: Western Culture in the Computer Age' Penguin 1986 Orig. published by North Carolina U.P. 1984.

Brew P.J. & Catlett J. 'SA: An expert system for troubleshooting a smelter' Proc. Aust. AI Conf., NSWIT, Sydney, May 1986.

Buchanan B.G. 'Expert Systems: Working Systems and the Research Literature' Expert Systems 3,1 (January, 1986) 32-51.

Carter C. & Catlett J. 'Assessing Credit Card Applications Using Machine Learning' IEEE Expert, Fall 1987 71-79.

Clarke R. 'Legal Aspects of Knowledge-Based Technology' J. of Info. Technology 5,1 (March 1988)

_______ 'Knowledge-Based Expert Systems' Working Paper 1989

Connell N.A.D. 'Development of a Corporate Strategy for the Selection of Appropriate Application Domains for Expert Systems' ESRC Project Report, Dept. of Accounting & Management Science, Uni. of Southhampton, May 1987.

Connell N.A.D. & Powell P. 'Success and Failure in the Commercial Application of Expert Systems' Proc. O.R. Soc. Conf., Edinburgh, September 1987.

Davis R. 'The Dipmeter Adviser: Interpretation of Geological Signals' Proc. 7th IJCAI, Vancouver, 1981 846-849.

_______ 'Expert Systems: Where Are We? And Where Do We Go From Here?' AI Mag Spring 1982.

Debenham J. 'Practical Applications of Expert Systems' Proc. 10ACC, A.C.S., Melbourne, September 1983.

Dreyfus H.L. & Dreyfus S.E. 'Mind Over Machine' Blackwell 1986.

_______ 'Why Expert Systems Do Not Exhibit Expertise' IEEE Expert Summer 1986.

EDP Analyzer 'What's Happening with Expert Systems?' 23,12 (Dec. 1985).

_______ 'A Three-Year Strategy for Expert Systems' 25,3 (Mar. 1987).

Elam J.J. & Henderson J.C. 'Knowledge Engineering Concepts for Decision Support System Design and Implementation' Info. & Management 6 (1983) 109-114.

Feigenbaum E.A. & McCorduck P. 'The Fifth Generation' Michael Joseph, 1983.

Ford F.N. 'Decision Support Systems and Expert Systems: A Comparison' Information & Management 8, 1985, 21-26.

Harmon & King 'Expert Systems' Wiley, 1985.

Hayes-Roth F., Waterman D.A. & Lenat D.B. (Eds.) 'Building Expert Systems' Addison-Wesley, 1983.

Hofstadter D.R. 'Metamagical Themas' Penguin 1985.

Holsapple C.W. & Whinston A.B. 'Management Support Through Artificial Intelligence' Human Systems Management 5, 1985.

Jenkins P.G. 'Quantifying and Assessing Potential Applications' in KBS'87 Online Publications Pinner, U.K. 1987.

Johnson T. 'The Commercial Application of Expert System Technology' Ovum, London 1984.

Laulan A-M. 'Social and Cultural Conditions in the Acceptance of New Technologies' in Bernold T. (Ed.) 'Expert Systems and Knowledge Engineering' Proc. Conf. Gottlieb Duttweiler Institut, Rüschlikon, Zürich April 1985 pp.81-94.

Liebowitz J. 'Useful Approach for Evaluating Expert Systems' Expert Systems 3,2 (April 1986).

Luconi F.L., Malone T.W. & Scott Morton M.S. 'Expert Systems: The Next Challenge for Managers' Sloan Management Review, Summer 1986, 3-14.

Marchand H. 'On Commercial Expert Systems Projects' in Bernold T. (Ed.) 'Expert Systems and Knowledge Engineering' Proc. Conf. Gottlieb Duttweiler Institut, Rüschlikon, Zürich April 1985 pp.73-80.

Michaelsen R. & Michie D. 'Expert Systems in Business' 29,11 Datamation Nov. 1983.

Miller A.J. 'DESIGN - A Prototype Expert System for Design Oriented Problem Solving' Aust. Comp. J. 17,1 (February 1985).

O'Neill J. 'Plausible Reasoning' Aust. Comp. J. 19,1 (February 1987) 2-15.

Pfeifer R. & Lüthi H-J. 'Decision Support Systems and Expert Systems: A Complementary Relationship?' in Sol H.G. et al (Eds.) 'Expert Systems and Artificial Intelligence in Decision Support Systems' Reidel, 1987, 41-51.

Prerau D.S. 'Selection of an Appropriate Domain for an Expert System' AI Mag Summer 1985.

Quinlan J.R. (Ed.) 'Applications of Expert Systems' Addison-Wesley, 1987.

Reitman W. (Ed.) 'Artificial Intelligence Applications for Business' Ablex 1984.

Roszak T. 'The Cult of Information' Pantheon 1986.

Sen A. & Biswas G. 'Decision Support Systems: An Expert Systems Approach' Decision Support Systems 1, 1985.

Smith R.G. 'On the Development of Commercial Expert Systems' AI Mag 5,3 (Fall 1984) 61-73.

Soleway E., Bachant J. & Jensen K. 'Assessing the Maintainability of XCON-in-RIME: Coping with the Problems of a VERY Large Rule-Base' Proc. AAAI'87 Seattle, July 1987 824-9.

Sviokla J.J. 'Planpower, XCON and Mudman: An In-Depth Analysis into Three Commercial Expert Systems in Use' D.B.A. Dissertation Harvard G.S.B. 1986.

von Weissenfluh D. 'Überblick über Einsatzgebiete betrieblicher Expertensysteme' ('Overview of the Application Areas of Operational Expert Systems') Arbeitsbericht Nr. 15 Institut für Wirtschaftsinformatik Uni. Bern (December 1987).

Waterman D.A. 'A Guide to Expert Systems' Addison-Wesley 1985.

Winograd T. & Flores F. 'Understanding Computers & Cognition' Ablex 1986.



xamaxsmall.gif missing
The content and infrastructure for these community service pages are provided by Roger Clarke through his consultancy company, Xamax.

From the site's beginnings in August 1994 until February 2009, the infrastructure was provided by the Australian National University. During that time, the site accumulated close to 30 million hits. It passed 65 million in early 2021.

Sponsored by the Gallery, Bunhybee Grasslands, the extended Clarke Family, Knights of the Spatchcock and their drummer
Xamax Consultancy Pty Ltd
ACN: 002 360 456
78 Sidaway St, Chapman ACT 2611 AUSTRALIA
Tel: +61 2 6288 6916

Created: 27 July 2000 - Last Amended: 27 July 2000 by Roger Clarke - Site Last Verified: 15 February 2009
This document is at www.rogerclarke.com/SOS/KBTE.html
Mail to Webmaster   -    © Xamax Consultancy Pty Ltd, 1995-2022   -    Privacy Policy