Project

The goal of this project is to read about a specific area of Usable Security and Privacy in more depth, think critically about existing gaps in knowledge. and plan a study that could theoretically help fill the identified gap. The project has two feedback-only deadlines and then a final deadline which results in a project grade.

I recommend reading through the full document to get an understanding of what will be required as the end state. The two feedback deadlines should help students make progress in that direction.

You will be picking a subject topic of your choice within the realm of Usable Privacy and Security. The proposed study needs to be about usability, but the study does not need to directly involve humans. A study looking at pre-existing datasets (i.e. password breaches), or content analysis (i.e. Stack Overflow posts) would also work. As would more traditional approaches like a lab study looking at encryption software usability.

For this project you only need to demonstrate the ability to consider some existing literature, identify an interesting gap in knowledge, and plan a study to address that gap.

Feedback Deadline 1: Rough idea

Deadline: July 2nd

For this deadline you only need to specify a rough idea for the project in 1-2 paragraphs. You need to state what the subject area is you will be focusing on. What research questions or directions you have in mind. And how you are considering answering them in a study.

Feedback Deadline 2: Initial plan

Deadline: July 11-18th - I’ll start giving feedback on the 11th, but it can be turned in as late as the 18th.

For the second deadline you need to have a more revised plan.

  • A short description of the problem. Think abstract-type explanation.
  • Brief summary of prior work that cites at least two papers. This summary need not be comprehensive, but it should show some engagement with what others have done.
  • Research Questions or Research Goals - These should be clearly stated. There could be only one such research question, or there could be several, it depends on what is being researched.
  • Methodological approach - What methodology are you considering for the study. For example, it could be an interview study, a survey, a document analysis, or many other options.
  • What you plan on measuring in the study.
  • Plan for analysis. How are you planning on turning your raw data or observations into a final result. This could be the statistics you plan to use, or the analysis methodology. It can even be descriptive statistics (mean, median, mode) if those are most appropriate.
  • List of appendix documents you will be creating.

Final deadline: Research plan

Deadline: July 28th

The final report is essentially a plan to conduct a research study. You do not need to conduct the study or even a pilot of it. Only the written plan is required.

Use the ACM one column article template either in Microsoft Word or in LaTeX. I will not be strictly marking template, so minor variations from the official template are fine.

Plan structure

The final document should have the following sections in this order. It is ok to add sub-sections below these to structure the document as needed. Below I state what needs to be in each section.

Briefly explain the issue that is to be studied. The introduction should answer questions like:

  • What topic area is being studied?
  • Why does this topic matter? Or who cares about the topic?
  • How will this study advance knowledge above and beyond existing research?

The introduction should cite at least two papers related to your proposal. Unlike a real paper, it is not necessary to provide a comprehensive related work. But at least some related work needs to be considered and planned.

Research Question(s)

State the research questions or research goals of the work.

Planned methodology

This section should read very similar to a methodology section in a research paper. It should explain the methodology you plan on using, reasoning behind methodology choices, and how you plan on analyzing the resulting data.

Bibliography

This section should follow the ACM template.

Research materials - appendices

This section is most similar to an appendix in a research paper. There is a trend in science towards replicatable research. In other words, modern research papers are encouraged to include enough of their materials that a future researcher could re-run the study.

This section should contain the research materials necessary to run the study. These materials will differ depending on the methodology used, but here are some examples of what might be included:

  • Interview script for an interview study.
  • Questions asked for a survey.
  • List of content sources for a content analysis and a list of possible initial qualitative codes if coding is planned.
  • The list of “correct” steps to be analyzed for a cognitive walk through.
  • Interfaces tested - UI interfaces shown in a lab study. If software is tested, key UI interfaces are often shown in the appendix to help readers understand what users saw. For a planned study these can be included if the software exists, or mocked-up if the software does not exist yet.

Not needed - You do not need to include any of the following documents in this section:

  • Recruitment materials - things like advertisements, email posts
  • Code - You do not need to write or provide any data analysis code or code you would use to create any of the study setup.
  • Implemented surveys - The questions should be included (see above) but there is no need to imput them into functional survey software.

Research plan grading guide

The project will be graded on the following points.

Study framing (30 points)

The introduction and related work should:

  • Make it clear what is being studied and why.
  • Explain how the planned study fits in with the chosen related work.
  • Study motivation or methodology decisions need to be linked to related work.

Research questions (20 points)

Research questions or research goals need to be clear and precise enough that they can be evaluated in a study. They also need to align with the goals stated in the framing.

Internal validity (40 points)

Internal validity means that the study is actually testing what it is meant to test. Most of the assessment will be if the methods and research materials sections make sense internally and if they match the research questions.

Overall presentation (10 points)

Overall presentation and flow. Issues like spelling, layout, and other presentation-related issues are marked here.