Chapter 7 Piloting

In this video we are going to talk about the importance of piloting. You might think that since people have already done these kinds of studies in the past, then you don’t have to do piloting yourself. I’m sorry to say – this is simply not true: You need to make sure that the community with whom you are working approve of your methods, feel comfortable, and have a chance to give you feedback.

What do we mean by community? We mean a superset of people you are going to sample from. This depends on your study. Often, this is defined by your inclusion criteria; and sometimes also by how you recruit. For instance, if you intend to have a sample that is representative of the population of a given city, this will mean trying to reach out to parents with a wide range of educational backgrounds. Or in other cases your project will call for a more homogeneous sample, perhaps including only parents who have a lower level of education. In any case, you should contact at least 2 families who come from the population that you are going to sample. Note that in some cases, this may exclude your friends or family. You can start early piloting by asking friends and family, but if they do not belong to the community you aim to sample from, then their answers might be different, and therefore misleading for you.

What do you want to discuss with the community? You can start asking them how they feel about the clothing – some communities prefer clothing that is flashy (so that everyone knows which child is being recorded) and others prefer clothing that “blends in” (so that the child does not feel singled out). Also, you can ask about the device you intend to use. We’ve seen communities tend to have strong preferences about the device they prefer their child to wear. Some families don’t like the USB device because they feel like it can be easily lost, resulting in an additional worry for the parents. Other families have a preference for the USB device, especially when used for extended periods of time and during the night, because they think the child will be more comfortable with this lightweight option.

Ask also for their opinion on how you aim to analyse their recordings. It’s always very useful to talk about this process with the families, and get their impressions on what it means when manual as opposed to automated methods are used. We gave found that some communities do not like the idea of human annotation: they worry about what the people who listen to the recordings are going to think of them. This is something you can address in your consent form and/or other explanatory material. For instance, you can specify that listeners are going to be professional and respectful about it, and act with awareness of the unique situations each family is in.

Some of the complaints and worries we got about annotation are concerned with automated annotation, but they are very very few and usually they come from very informed parents that come from the tech industry. In this case, parents might know the use of automated analysis for purposes that are close to surveillance or that are being used in discriminatory manners (e.g face recognition, since algorithm are trained with faces taken from the internet, which is biased with faces of White people, these algorithms might work against people who are not white). I personally am concerned about “algorithm bias”, the possibility that an algorithm systematically misrepresents data from one type of community. So I think it’s worthwhile to still bring up the topic of automated annotations and ask families what they feel about it, so you can make sure to address this in your consent form or other explanatory material, as well as in your annotation process (for instance, by retraining the software).

The next thing that should definitely be among the topics that you bring up is mandatory reporting. Some of you will live in a State or Country where you are compelled to report anything illegal (e.g drug consumption); any suspicion of child-directed violence; or any suspicion of developmental delays. If this is your case, first ask yourself is something in your analysis process could reveal one of these cases. If you are relying purely on automated processing, then you probably won’t be able to find illegal events or violence, but you could find evidence suggesting a developmental delay. If you are using human annotation, all three could be discovered. Next, make a plan about how you are going to report it. Finally, tell your pilot families about this, and ask them what they think about this plan. Remember that since you will have to specify this anyway in your consent form, it is important to know whether this will result in certain families opting out of the study as a result, and therefore your study ending up with a biased sample of the community you were hoping to represent.

What else should you ask? Honestly - everything! It’s always good to involve communities in every step of your study, which will be enriched by this contact. So take a moment to discuss all the stages of your study with the representatives of the community you are working with, because you want to minimize the surprises you find down the line.

Start with what the goals of your study are, asking families what these goals mean to them. Sometimes when you have a very narrow, technical goal, parents might not really care about it, and that’s fine. You just want to check if they have any strong feelings about it.

A related question is if there are other areas or research goals they want you to be thinking about. For example, something that happens very frequently is that certain communities are interested in research about bilingual or multilingual children development. Some communities speak a language that is not spoken by many and they are afraid their children will lose it. Yet another question they may have is how this participation will help their children’s academic outcomes. This doesn’t necessarily mean that you’ll change your research goal for this study, but it could help you strengthen your research by incorporating these ideas in a future study. Or if you are well connected in the community, perhaps a community member will be interested in exploring these idea via data reuse – as we just discussed. Piloting is a the natural time to look into this, so that you can already foresee it in your ethical request.

When talking about goals with the families, you’ll also want to discuss about what could happen in the context of data reuse. We will talk about data reuse in another video, but we think that these data are really valuable, so they shouldn’t be used for a single purpose, right? People have gone through a lot of effort to collect them (including the participating families), so they should just be thrown away after one use. Piloting is a great time to talk with families about the options for donating data and reusability of data – since this needs to be specified in your ethical request, and in consent forms and explanatory material. When having these discussions, don’t think just of your research interests. We recommend that you also mention other work people are carrying out with these studies, such as automated recognition of child and parent emotion, computational modeling of language learning (or training of artificial intelligences), etc. You can then make your questions more precise, and see if parents are ok with data being reused in different settings.

Finally, this is the perfect time to ask families (and the community as a whole, if this is relevant) about the feedback they want, whether they are interested in having some of your resulting data and in which form. You can then see whether this is logistically and ethically possible for you to do. For instance, we do not recommend sharing back the whole recordings with them unless you know for a fact that third parties are not recorded anywhere – and even if it’s only the family, as these data could be e.g. used in divorce cases (which was not a purpose mentioned in the consent form). So keep an open mind, trying to accommodate the families’ wishes, but also being careful about the ethical and legal implications of such choices.

7.1 Resources

• Cychosz, M., Romeo, R., Soderstrom, M., Scaff, C., Ganek, H., Cristia, A., … & Weisleder, A. (2020). Longform recordings of everyday life: Ethics for best practices. Behavior research methods, 52(5), 1951-1969. pdf
• Levin, H. I., Egger, D., Andres, L., Johnson, M., Bearman, S. K., & de Barbaro, K. (2021). Sensing everyday activity: Parent perceptions and feasibility. Infant Behavior and Development, 62, 101511. pdf