Our user observations were made up of users from both a college background and those who are not currently in college, IT literacy for all was reasonably high, which reflects our personas level of literacy. They were all given the set core task of logging into Blackboard and sending an email with some carrying out minor extra tasks. We also carried out a user survey of IADT students to obtain further feedback on issues and desired functionality.
We carried out a total of five user tests to identify issues with our chosen problem, the email system within blackboard, and any possible features that could be retained. At the same time we where also conscious of the secondary issues which had been identified in our earlier problem identification phase, confidence, calendar and navigation and considered them during the user observations.
The following is an overview of the of the participants taking part in the study.
Types of tasks.
Users where asked to preform a core task of creating and sending a email. In some cases additional tasks where added such as adding an attachment, sending email to one recipient and creating a contact.
User observation 001: Create contact and send an email
User observation 002: Create contact and send an email
User observation 003: Create contact and send an email
User observation 004: Send an email
User observation 005: Send an email
In all cases the additional two steps of having to select a course and user types before being able to compose an email created a large amount of confusion. There is clearly a mismatch here between the designers model, the system image and the users model which leads to the overall conceptual model been poorly understood, if at all (Preece et al., 2015). The user quotes support this observation.
- “I don’t know what any of this is!”,
- “The group thing is confusing,”
- “Ah this is just too confusing….”
- “List of courses have no connection to sending an email. No explanation as to why you have to select a course”
In cases where participants where given the additional task of creating a contact before sending an email, the level of confusion and frustration was higher and appears to be as result of having to interact with the core Blackboard navigation and the fact that creating a contact in the address book is not reflected in the possible email recipients in the email system. Again there is a breakdown in how the conceptual model is understood by the user.
- “I’m going to go back to the home page again in case it’s in there, that’s not the home page I was expecting!”
- “Oh my god! There is no option to send a contact an email”
- “Hopefully there’s some way I can use this to send an email”
- “Is there no way I can do it from here no?”
When creating an email most users although they understood how to use the recipient selection list where at first confused to see as an option for selecting the recipient. This was also the point at which the disconnect between this and address book was understood by those users who had used the address book, causing some to recheck the address book to confirm they had actually added a contact.
Users also pointed out that they were unable to add or edit recipients at this stage.
- “Why cant I add groups in the email”
- “Groups could be part of email I should be able to write the email before selecting the group”
You can find more detailed information on our user observations in video transcripts and observation notes at the following links.
To aid us in further clarifying pain points within Blackboard Learn’s email system and identify possible useful new features based on our user observations, competitor analysis and initial review we created a short survey and sent it to IADT students. In total twelve students completed the survey with two of these being from courses outside the UX Masters.
The full survey responses can be viewed here
The table below is an overview of the results
|How frequently do you use Blackboard’s email system?||Rarely ever: 83%
Once a week: 17%
|How satisfied are you with the email system?||Very Unsatisfied: 73%|
|How likely are you to use it for college over your personal email?||Very Unlikely: 75%|
|How would you rate your experience of Navigation||Miserable: 45%
Very Poor: 45%
|How would you rate your experience of Selecting Recipients||Miserable: 27%
Very Poor: 45%
|How would you rate your experience of Formatting Emails||Miserable: 9%
Very Poor: 27%
|How would you rate your experience of Adding Contacts||Miserable: 27%
Very Poor: 45%
When users where asked for further information on some of their choices the following replies where received.
- No inbox. Have to check separate email account
- Can only send not receive. Navigation is a bit weird, you have to select from a certain group first then you get to the email edit page. It’s just quicker and easier to use slack or personal email
- No inbox/outbox within Blackboard. Have to check my IADT email account which I rarely use.
- I sent emails to 2 members and I have yet to get an acknowledgement that the emails were received
- I can’t see the email addresses of individuals. I don’t trust it. I also don’t know if the message I sent will be saved somewhere in a thread.
- Keeps personal and College life separate
- I find it hard to understand if I’ve sent or received anything
The survey results and the respondents comments further supported our original identification of the problem areas, of confidence, calendar and navigation. As well as corroborating the user observation feedback around the email pain points, group creation, unnecessary steps, adding recipients, user feedback in the form of an inbox and out box notifications.