1 THE METHOD THE METHOD USED IN THIS MASTER THESIS HAS BEEN SATISFYI...

9.1 The method

The method used in this master thesis has been satisfying. There were several aspects

contributing to choosing what method to use, but the most important was the

opportunity to perform a lot of work in the users right context, at their place of work,

and the contextual design method was therefore naturally given. One aspect contributing

to the choice of creating prototypes was that there is no similar system with an

Operations Monitor existing today, and to reduce the cognitive load of the users, trying

to imagine one, it had to be visualized. It was also the best way to communicate the ideas

to the people at Carmen, the members of the Descartes team. The greatest benefit from

creating prototypes as a design method is that it facilitates the cooperation between

designer and user, creating a greater understanding for each other’s problems and assets.

It was important to be able to create and to demonstrate scenarios, for the

designers to understand the type of tasks there is and for the users to recognize their

work. Discussions concerning what functions are needed to solve a problem came up

during tests, and without these discussions important aspects related to reality would

probably not have been considered. It is important to discuss how the result will be used,

since it is difficult to predict how the future users will actually use the system and how it

will be experienced.

A certain degree of criticism has been involved when evaluating the results from

tests and workshops in accordance with literature and previous work, since the

prototypes and recommendations evolved from these are nothing but guidelines and

experiments. A weakness with this method is that the scenarios are typical and realistic

problems, but never enough since each task for a controller is always unique, and this

complexity affects the result. The time constraint is always an important factor, and that

is also a weakness to this method, it is difficult to perform realistic tests in the right

context with all the complexity concerning the work in the operations control of an

airline company.

In the design of the prototype, the users have been involved for test purpose. The

designers have decided all decisions concerning the design, but the users have been

consulted along the way. It would of course have been optimal to have crew controllers

cooperating in all stages of the design process according to participatory design, ensuring

all details to be optimal. Since this was not possible, the design process became iterative,

with tests performed along the way. Users were involved when creating the foundation

for the work, demonstrating their way of working today, as well as in testing our general

ideas, the information presented and the interaction. The creation of all the graphical

details was entirely the designers’ choice; there was no possibility to focus on this too

during the tests.

It is a great advantage of having a prototype to communicate concept and ideas,

but one has to be aware of that it can also limit the tests. The focus tends to be directed

to the details of the appearance, making it harder for the test person to see what is

missing and what to change. When testing the third prototype, letting the test persons be

aware of that it was not graphically completed, the focus were more directed towards the

functionality, interaction, concept and what was missing.

User studies have been conducted to study how the users interact with the systems or

prototypes available in different stages of the design process. At first the purpose was to

study how the controllers use the available systems today, to gather requirements and

information. Later on user studies was conducted to evaluate the work, the reactions, the

ideas, and to investigate whether the prototype was enough for achieving the goals we

had set up in advance.

The contextual inquiry worked very well for observations as well as for testing.

Problems with conducting this kind of interviews became very obvious, since the users

are all experts on what they do, it is their every day tasks, and they summarize their

performance down to nothing. This is why the observations are important, while

studying when the work is actually performed; the complexity of the problems as well as

of the work procedure becomes obvious. Details, problems and hidden structures would

not have come to our knowledge if we were not to observe the work in the right context.

Performing the interviews in the control room while they were working, gave us a lot of

useful information about to what extent they take advantage of divided attention within

the entire room, while focusing on a task. The fact that these interviews and observations

were performed at two different airline companies contributed to even more concrete

and detailed information. Unfortunately the days for the visits have been really calm days,

without any major disturbances. This provided more time to discuss and interview, but

less knowledge of how the work is performed, how problems are actually solved, which

has been a great lack throughout the work.

Sometimes problems occurred with keeping focus during the interviews, even

though there was a pre-decided goal. To keep focus turned out to be difficult, since the

environment and the problems were totally unfamiliar, especially in the beginning. As

soon as we started to collect the information, it was realized that the unfocused parts of

the interviews contributed with a lot of important facts too, since this was the

controllers’ personal opinions and stories.

The number of performed tests was satisfying, even though more tests would

have contributed to even more aspects and opinions to consider. The selection of test

persons was satisfying in one sense; they were a great difference in ages and therefore

experience. To become as company independent as possible, tests would have to have

been performed in several cultures, meaning in other parts of the world. More aspects

would probably have come up if all test persons were crew controllers, even though this

actual variation in professions contributed to a wider range of subjects were to be

discussed.

Is it possible to test and evaluate a graphical user interface separated from the rest of the

system? This question has been vital during the entire design process, and several

answers are to be found. It is possible when performing tests with test persons that are

familiar with the Descartes concept; the focus is much more on just the Operations

Monitor, its concept and functions. When performing tests with users that have not

heard of the type of decision support system that is Descartes, then it is almost

impossible to separate them. In that case, it is preferable to test only the traditional way

of solving problems, not involving the solvers or the new way of working. This way, the

purpose of the monitor would be to support the users in their existing systems and just

be an add on, which is not quite correct, and still the trap of the general idea of an user

interface. When using a system, the user does not make a distinction between the

interface and the underlying functionality. How is it possible to measure the usability of a

system if that underlying interaction is not satisfying, or even existing?

How the results in this thesis could be used in the future has to be discussed. The

analysis is a good basis for future work concerning the crew controllers, as well as the

recommendations. Specific details in the design should not be considered as optimal in

any sense, since this was not the main purpose of the thesis. Since this concept of an

Operations Monitor has been abstracted from the rest of the Descartes in many levels,

then it is recommended to create new analysis based upon how the work will change the

routines for the users, what reorganizations are meant to be, and so forth, if the

Operations Monitor is to be a part of a larger system.

It was good practice to really have to understand the users work in detail, since it

provided the possibility to practice our capability to make them express exactly what they

are doing, every single step, even though it might be totally obvious and natural to them.

It is not easy to make someone aware of actions that they have in their backbone, to

make them verbally express their silent knowledge, especially when you are not familiar

with the environment yourself. This might be complicated, but never the less necessary.