The Testing Show: Exploring Roles in and Around Testing

The Testing Show: Exploring Roles in and Around Testing

How well do we know the work that we do as testers? Do we understand what it is we do? Really understand it? Jon Bach thinks we can do better at figuring out what it is that we do in our roles as testers and in the roles that support and offer service to people in our organizations. Much of what we do is implicit, and carries responsibilities, expectations and even contracts for what we do and how we act. In today’s episode, Jon helps us break down both traditional and not so traditional roles that we may find ourselves in, and ways that we can leverage both explicit and implicit knowledge of what we do, and maybe what we can stop doing.

Also, in an unconventional news segment this go around, friend of the show Anna Royzman tells us about Test Masters Academy and a fresh take on testing conferences geared towards testing leaders (the Testing Leadership Conference is May 1-3, 2017 in New York City) and emerging topics and technologies at the New Testing Conference coming this fall to New York City). It’s a wild ride!





Any questions or suggested topics?
Let us know!



MICHAEL LARSEN: Hello and welcome to the testing show. I’m Michael Larsen, your show producer, and today we have our regulars. Justin Rohrman.



MATTHEW HEUSSER: Good morning or good time zone.

MICHAEL LARSEN: Welcoming back Anna Royzman.


MICHAEL LARSEN: And new to the show, we’d like to welcome Jon Bach.

JON BACH: Hi, everybody.

MICHAEL LARSEN: I guess in classic Ed McMahon style… Here’s Matt!

MATTHEW HEUSSER: Thanks, Michael. The big news that’s happening, there’s a couple of things that… and disrupt is a big word. Digital destruction, OK, so were getting rid of DVDs and we’re moving to Netflix. Like, what’s the big deal? But there’s two interesting things happening. For our News segment today, we brought in Anna Royzman. Friend of the show, I think she was on the old show a couple of times [Software Test Professional’s “This Week in Software Testing”].


MATTHEW HEUSSER: Anna has transitioned. She’s changed her role from test manager to executive of a non profit at…

ANNA ROYZMAN: Global Quality Leadership Institute.

MATTHEW HEUSSER: Which has a couple of events now, and the events are kind of… they’re a different take on a test conference, I guess I’d say. One is Test Masters Academy and there’s a management event.

ANNA ROYZMAN: Yes, there is a Test Leadership Congress,which is May 1-3 in New York. Some of our roles are changing, so we sometimes no longer see the QA management per se, but I’m seeing there still needs to be a leadership component. Understanding how to manage and lead testing is really essential skills, no matter what your role is called right now.

MATTHEW HEUSSER: When’s Test Masters Academy?

ANNA ROYZMAN: Test Masters Academy is the initiative that runs those conferences. There is another conference in the fall; it’s called New Testing Conference. It’s about new, emergent trends in software testing.

MATTHEW HEUSSER: Cool. Great. So can you tell me, and I have my own opinions from the outside. I now Justin has been to one of these events.

JUSTIN ROHRMAN: Yep, I went last September.

MATTHEW HEUSSER: I don’t want to inject my own opinion. From your position as the organizer, how do you think that these events are differentiated from, I don’t know, every other testing conference on the planet? How’s your stuff different?

ANNA ROYZMAN: Test Leadership Congress is totally different. I have not seen anything like that. I was looking for an event like that for many years when I was a manager. Usually, the testing conference is “one size fits all”. Everybody who’s coming, they kind of search for the sessions that they feel they belong to. What I notice in the big testing conferences, it would be too broad or it would be too narrow. I think that managers and leaders have special needs, special skills and special tasks that are different from software testers per se. Components such as facilitating, coaching, motivating, inspiring people, growing people. I feel that leads and managers deserve a space where they can confer among themselves. You want to bring them to the next level, to give people new ideas and expose them to emergent trends and what’s going on in the industry.

MATTHEW HEUSSER: I like the split between leadership and test excellence. You’re bringing in International speakers. Jon is coming in from California. You had a speaker in Finland, Maaret [Pyhäjärvi]. Justin came in.

ANNA ROYZMAN: Yes [laughter]

MATTHEW HEUSSER: Most of the attendees, I think… it’s kind of Northeast focused. Anybody can go, but I have the impression that you’re very likely to meet someone at lunch that you could have a follow-up coffee with the following week and they’re not from, I don’t know, Florida or South America.

ANNA ROYZMAN: I have international attendance at my conferences. I advertise globally. I usually have some people from Europe, not just from North America, but the thing is, if you’re in New York, then use the opportunity.

MATTHEW HEUSSER: There are enough Northeast people that you can do some serious networking. Some of these big shows, every person you meet is from somewhere else, and then it’s hard to make a connection.

ANNA ROYZMAN: All of my speakers are all over the world. It’s a really good networking opportunity to meet people who are not local. This is the reason that I want my speaker line-up to be as diverse as possible, because I want people to be exposed to so many different ideas.

MICHAEL LARSEN: So first off, for those who want to apply for the next go-around, what are some of the things that you are looking for or that you would encourage people to include if they want to present?

ANNA ROYZMAN: So I usually invite speakers to this conference, I don’t have a call for proposals for this one. However, for the next one, for the New Testing Conference, we do have call for proposals, which will be opening right after this conference, so I say like May 1, the call for proposals will be open. What we are looking for is to look at areas which are new in testing. What’s emerging? How your role is changing? How your tasks are changing? What are the new tools that you are using? What are the new methodologies?


ANNA ROYZMAN: Yes [laughter].

MATTHEW HEUSSER: What are some of the new tools and methodologies?

ANNA ROYZMAN: We’re going to find out [laughter].

MATTHEW HEUSSER: We’ve got one of the key speakers, Jon Bach, on the line and he’s been very patient. Before we dive into his session, though, tell us about some of the folks that you’ve got lined up for the conference and what they’re going to be talking about?

ANNA ROYZMAN: Sure. Jon, of course. I invited him because what him and James discovered recently and the things that they teach about the roles, tasks, and actors is really going to help testers and test managers to fit in the changing world of our industry. I invited Dave Snowden. He’s the creator of Cynefin, which is an environment framework, and this is his first exposure to the testing community. There are several things that are applicable to us. One of them is the theory of complexity that he is describing, how certain things cannot be predicted. We as testers always work in a world of complexity, the world of uncertainty. There are certain, I would say, expectations from that type of environment. I also have Angie Jones. Angie is kind of a rising star in our community. She’s an adjunct professor and she’s an automation specialist. She was talking about how to create inventors from your testers, and I think that’s amazing. If you want to motivate your people, develop their creative potential. She’s doing also advanced automation framework for Agile. Then I have several speakers who are local. Lauren Langsner, head of QA at Rent the Runway and today she’s the head of Engineering. They have a very unorthodox way how they find and hire their talent. They’re growing very rapidly, and Lauren has stories about how they find their talent for their QA team and now their Engineering teams. Carl Horned, from Spotify, he’s an Agile coach, I believe, and he’s kind of a test coach in Spotify. They really value James Bach’s work. They created some processes where they do exploratory testing with the whole team. Carl’s role is really to facilitate and coach people to do this. He’s going to do a hands-on session on using his product to teach people how to coach the whole team to do this type of investigative exploratory testing. We have six tutorials, one of them is done by Fiona Charles -I think everybody knows who Fiona is- a tutorial on how to lead by example and how to develop a testing strategy. There is also a DevOps for Testers tutorial. Martin Hynie, Frankie Botero and Joe Lopez, and we’re going to take it from A to Z, starting with containers and new environment, to kind of system testing approach in DevOps environment. And then I have a very special tutorial which is on machine learning and this is emergent and Matthew you’ve been asking me about what’s new, so machine learning is something new, and Artificial Intelligence is kind of taking the world by storm and we’re not involved in that. It’s a very interesting industry. Because machine learns itself, so you need to develop and design the testing experiments to know what the machine does and how does it do it. We need to be there. We as a community need to be on top of that thing.

MATTHEW HEUSSER: So speaking of which, one of the -you mentioned him several times- he’s right here on the show. Jon, what are you going to talk about?

JON BACH: Well, thanks for having me on the show. I want to talk about roles. I want to talk about testing roles. In fact, my role on this call as I understand it, is subordinate guest, andI’ll say subordinate because I’m yielding to Anna, who I deeply respect and has invited me to come to this conference because of what she heard in material I wanted practice with and to present, so my role is to let Anna go first, and talk about what she talked about. As I see it, my contract, and that is implicit in every role, is to listen to what Anna had to say and figure out how I can contribute to what she had to say but also fit it in to the theme of your show. So, we didn’t talk about this, all of us behind the scenes. It was just kind of implicit, and I am fascinated by implicit and explicit contracts as they pertain to testing roles. Especially when companies are changing and evolving and downsizing and upsizing and back-shoring and all this kind of stuff. I’ve got a framework that my brother and I have created to help people talk about their roles at their companies as testers and test managers and as those roles evolve into things like… well, Anna called it years ago a “quality leader” at a conference, and I was blown away. I’m like “She was the first person I heard use that term, quality leader” and she did a whole talk on it. So, you’ve mentioned things on this call already, like uncertainty. You used the word expectations. You used the word outcomes. I’ve been making some notes and they’re right in line with the things I want to talk about. I have an experience where people talk about their roles to someone else in class, but not only talk about it but diagram it and see what frameworks emerge from that. What implicit contracts, what explicit contracts emerge from that? How do you know what your role is? What if it changes? What if you leave? Who does your role? There’s a rich vein of dynamics inherent in testing roles. I want to help people explore that.

MICHAEL LARSEN: I think what we are starting to see, at least this has been my experience, is that the days of just saying “I’m a tester and I work on the test team and I just do testing”, that’s not reality anymore. I’ve shifted into a number of different roles. In fact our team has shifted into a number of different roles due to necessity. All of us take a stab at doing automation, or doing systems maintenance or, in my regard, trying to make it so that our AWS environments, I’m responsible for managing the AWS AMI roots that we use when we set things up. I’m the person that actually rolls our releases and gets them out to our customers, but yet, it’s not a full time job, it’s not my primary role. I still test, but there are some weeks where I don’t set my hands on testing anything so to speak or depending on how you look at the equation, I’m testing all the time, but I’m testing other components. I’m testing “are our Jenkins runs working? Is our CI behaving the way we expect it to? Do we actually have the pieces for our servers in place?”

JON BACH: Wht you just described is what I want people to talk about with each other at the conference, and here’s what I heard already. I heard a contract in there. I heard that there’s expectations of you to behave a certain way or perform some service. I heard a person, you, with capabilities, behaving in certain ways and performing services. I heard tasks and duties. Activities that solve some kind of problem for the organization, and I heard a boundary object, which is a term for… it’s an artefact that serves as a medium of exchange between people or roles. So what is the thing that gets transferred between you and someone else? Maybe confidence or a status report/confidence report or a risk matrix that you, as the AWS guy, provides. The surest way to figure out, Michael, how valuable you are in that role is to not do it anymore and see what emerges [laughter]. That’s a risky thing to do, but I want to do this thought experiment at the conference. So you step into the coaching role, maybe for a day or two, that provides service to the organization. Even if you do that, there’s an implicit contract. There’s the person, which is you, and behaving in certain ways and performing services. There’s tasks and duties in that little coaching stint that you do with that new person. And there’s a boundary object. You give them the last two podcasts that are germane to their role as they start learning the new thing. I’m just fascinated by the story you just told of your own role and I think that we all need to come together, or could come together, and talk about these and go “hey, you know, I do that too. You mean I don’t have to do that anymore? Maybe I can get somebody else to do this, and that frees me up to be more of a tester?” Then I’d say “Yeah, let’s talk about how we can do that.”

MICHAEL LARSEN: So I want to add a little bit of additional context here, if I can. In Socialtext, like every organization, there are silos and there are level of expertise. We want to encourage everybody to learn and understand as much as possible and if they feel like they want to take on a part or do something, they can, so we have a rich “How To” environment, what we call our “Dev Guide”, and our Dev Guides are… we consistently go through and we try to green them every chance we get [Editors note: our How To pages. I realized after recording that this could be interpreted as individuals guiding others in real time, though that happens, too. –Michael]. I’ve written out every single step that has to happen to be able to do a release. I don’t want to have to say “Look, I’m going to be at Scout Camp during the middle of July, and we want to schedule a release. Don’t not release because I’m not going to be there. Have somebody else do it, and please, walk through.

JON BACH: I’m fascinated by this role, Michael, and see what you’ve done? You’ve proved that, if we talk just even for a little bit, we find these things that we’ve invented, ways that we’ve reinvented ourselves, that could help other people go “Oh what? Tell me more about that. I want to do that in my career next and see if I can hand of all this busywork and do what I’m good at. I want to be a Dev Guide now. Tell me more.

MICHAEL LARSEN: We have a tester on our team, newer than myself. One of the things that we’ve been working on is, as part of the Dev Guide and Writing the Dev Guides, and being able to make sure that they work is the implicit knowledge that we have and that we cover is being transferred. I realized that I wrote up this whole thing about “Oh hey, here’s how you do the certs on the new servers, so you can make sure you have a real verified cert on a test server, because we bring them up and tear them down all the time. This tester called me up and said “hey, can we go through the process here” and as we were walking through it, “OK< so what do I do at this point?” I had said “Oh, copy the cert over to your server”, but where was the cert? How did I do it? If I have it attached to this HowTo, how do I get it there? And I completely took it for granted that she would automatically know how to do that. So I said “Oooohhhh… this is good!” I had this piece of knowledge, this implicit piece, that I figured everybody knew. “What, you don’t know how to use scp?” Well, no. Maybe somebody hasn’t had to do that, or they don’t happen to have a mac, they’re on a PC. So that opened up my eyes for me to say “Hey, this is a good opportunity for me to re-evaluate what I’ve already written and I think I’m transferring this information over well”… ehhh, not so much.

JON BACH: Yeah, a lot of dynamics, isn’t there?

MATTHEW HEUSSER: To spin off that, when I was at Socialtext, it was not uncommon for “I’m struggling with X” and a developer points me to a page and I’m like “yeah, I’m on that page, I don’t know how to do X” and he’s like “look harder”. “No, it doesn’t tell you how to do that.” “Look better!” We’re a physically distributed company, right? You’re just over chat. The temptation to assume that documentation is going to save you, or that the problem of knowledge; you know how to do it, so you can’t imagine someone not knowing how to do it.

JON BACH: Yeah. Well, we all know this with test cases. If I give you a test case, Matt, and I watch you do it, I’ll bet you would do it differently, even though I wrote it all down. That fascinates me. I’ve actually leveraged that for exploratory testing when, in something that I call “Open Book Testing” where I give kind of a vague charter of what I want someone to explore, knowing that they would misunderstand it, in a meaningful way perhaps, because I can give three people the same thing, like “Find the most expensive thing on eBay” and watch what they do, and they each interpret it a little bit differently in a meaningful way that may find a bug that somebody else doesn’t find. So although I’m a fan of precise language, there are times where I can let the language be ambiguous purposes.

MATTHEW HEUSSER: I do want to mention that your talk at Test Leadership Congress, the workshop, the beta of that, I believe with the role-grams, the images and the diagrams… most of the people on the show today except for Michael were there for the beta of that on Orcas.

JUSTIN ROHRMAN: We did a show at Orcas.

MATTHEW HEUSSER: Perze, who’s another regular for the show, he was out there, too.

JUSTIN ROHRMAN: So actually, I have a quick question going back to role-grams. It’s kind of loaded. Why should people care about the ability to diagram a role? I understand that it’s interesting to study contracts and test agreements and explicit agreements between how people work, but how does it change how people work? Does it make it easier for me to do things like Michael was talking about? Mediating tasks between people when we didn’t know who was doing what? How do you see this being applied.

JON BACH: I love the question. I don’t make any guarantees that it’s going to help you do your role, or your perception of your role. Any good testing reveals dimensions and risks and problems and capabilities. In my experience, we don’t do enough evaluation or even description of what our role really is. It’s more of an elucidation and exploration of what we think our role is. Especially when we try to describe it to someone else. I think all of us maybe are familiar with the teddy bear effect, of trying to work out a problem just by talking to an inanimate object. What emerges from that (I guess it’s kind of Rogerian Therapy as well) is realization that when you have to make your brain formulate the words and sentences to describe and convey an idea to somebody else, there something interesting that happens about that that role grams or the polygons, literally, that James and I came up with to describe these dimensions. I heard a contract, as Michael spoke about his role. I would diagram that as a square and say “OK, there’s a contract.” I would make it have dotted lines if it was an implicit contract. If I heard a task, I would say “Oh, there’s a triangle,” and I would draw a triangle for every task I heard, and I would use a dotted line if that task I thought was conditional or optional, and a circle is a person. How many people are involved in your role? How many people can do your role when you’re not there, so Michael can go to the Scout leadership session. So circles, squares, triangles; it’s meant to be an exercise where you really take a look at the assumptions that really are the foundation of why you do what you do, what implicit rewards you get from that, and what if you didn’t do that anymore? What are the risks and consequences. So think of it like a test map, a test diagram, or an architectural diagram of a system. All we testers sometimes can do to be really effective, even if we don’t know the whole diagram, is point to any place on that whiteboard and say “what if this fails?” Just point to any object and ask a question, and that’s really the spirit of the role gram.

MATTHEW HEUSSER: Cool. I was going to say, the components of the role gram are… list them for me.

JON BACH: Well there’s what we call tasks and duties, which are triangles. Contracts, which are squares. People, which are circles, and then we have one last polygon, so four total, and that’s the hexagon, and that’s the boundary object. If Michael produces a report in his role as the AWS guy, I would draw a hexagon, saying “Ah! You produce this report, don’t you? That’s interesting. What if it didn’t get produced, or what if it got lost? How many people does this report go to? Then we can have conversations about the value of the work that he produces. If that work is implicit, is he just expected to do that? Maybe he’s the only one who thinks he’s supposed to produce a report, and after a little bit of talking, he realizes that “you know? No on reads this report. I’m going to stop doing it and see what happens.” When we take the time, when we got people talking about their role and drawing it pictorially with triangles and circles and squares and hexagons to delineate dimensions of their role, we had people realizing different aspects of the role that they hadn’t realized before or different interactions they hadn’t considered, and moreover, if they’re thinking about reinvention or even forced to reinvent or highly pressured to reinvent their role, like “You’re not a Test Manager anymore, now you’re a Scrum Master”, how does their role as a test manager change? How do they want it to change and how should it change in relationship with their business? When we get people to do this, it was profound, in my opinion, and I’d like to try it again.

MATTHEW HEUSSER: Also, all of those things that you described can have… there’s three different ways you can do it. You can do it as the boundaries. They can be straight lines, squiggly lines or dotted lines. Tell us about those.

JON BACH: So let’s say, as Michael was talking about his role, there’s a tacit contract. A contract means a square, but I don’t have to draw the square with a solid line. I can draw it with a wavy line. It’s an unspoken or vague agreement, or an informal protocol. I can draw it with dotted lines and that’s an open contract, which is an agreement between us on the call, even though I may disappear. So let’s say James is on the call with me. If I had to go ten minutes ago, he could easily step in and do that role for me. That’s fascinating because in the Agile parlance, anybody should be able to do anybody else’s role, which has merits, but also has drawbacks because I like being a specialist. Where this comes in, if I drew an open contract, a dotted-line square, it would mean “Oh, you know, anybody could really do this role.” It’s an agreement with actors who can come and go. Does that make me feel safe, or does it make me feel freed up to do something else, and that’s important. There’s the solid line, the wavy line, and the open line or the dotted line. Those are the boundaries you can draw.

MATTHEW HEUSSER: Now the reason I wanted to get at that, now the audience understands conceptually nouns and verbs, the verbs being the kind of line, at Orcas, my table had… Carol Brands is a tester at a company, and at my table was a dev manager at her company, and they both drew out what they thought that the role of the tester was and then they compared them, and I think that’s incredibly powerful, in a way that no job description is going to be. From there, they could say “Ooh, this is the way testing is, this is the way testing could be”, and then what are the gaps between those two? I would say “Bring your manager to Test Leadership Congress” and now you’ve got something to talk about at dinner. Or, and this might be even more powerful, if they took the course, did the role grams, kept them in a drawer, brought them home and then had the tram do the role grams, and compared, and then brought out theirs, “this is what I found, what did you find?” and then use that as a jumping off point, define testing in a more meaningful way and talk about how it might change.

JON BACH: Yeah, it’d as simple as managing change. If someone leaves the company, got a better offer or decides to retire, what happens to their role? How easily can it be absorbed? What are the tasks and duties? What are the contracts, implicit and explicit? What are the interactions, the boundary objects that now get orphaned? The org just seems to go “well, we’ll figure it out”, and they will, but it would be interesting to have this diagram at the disposal of the people who now have to take on these new duties and roles to figure out how it may work, and that’s how I’ve evolved over at eBay over six years starting as a test manager and now I’m an Agile transformation coach. How does that relate to testing? How does that relate to quality advocacy, and evangelism and good testing practice? That’s all part of what I brought to each role, and figured out how I’m going to add value to eBay.

MATTHEW HEUSSER: Yeah, I want to hear it now!

ANNA ROYZMAN: You need to come to the conference, Matt. [laughter]

JON BACH: Yeah. That’s a teaser.

MATTHEW HEUSSER: All right, I’m going to price some plane tickets. I think it’s time for shout outs and updates, so if you want to talk about the show, want to give us feedback, want to be on the show, have a question to ask a guest, have a question to ask the regulars, our email address is “TheTestingShow(at)QualitestGroup(dot)com” and of course you can go to and you can get all the show notes in plain text, everything, all the links you can just click through it and go to any web page. Other stuff going on… I’ll be at Agile and Beyond May 4-5, I might be at Test Masters Academy, we’re going to have to see.

MICHAEL LARSEN: CAST is in August, and I and Claire Moss have signed on to be part of the social media team. We’ll explain that once it’s ready to go live.

MATTHEW HEUSSER: Not only that, you mentioned Claire Moss. Right now, this instant, Claire Moss is available for test work in Atlanta. She won’t last very long, so if you are looking to hire, she’s an Agile tester, test leadership stuff but I would say social media and JavaScript and UI/Web Testing is what she’s been doing laely. Multi-skilled. Multi-talented. Got a CS…

JUSTIN ROHRMAN: Scrum Master, Agile Coaching.

MATTHEW HEUSSER:  Yeah, yeah. Got a CS degree from Georgia Tech. Wicked smart and she’s available.

ANNA ROYZMAN: Conference speaker!

MATTHEW HEUSSER: One of us, is how I think I’d put it.


MATTHEW HEUSSER: OK, thanks for being on the show everybody.

MICHAEL LARSEN: Thanks very much.

ANNA ROYZMAN: Thank you [laughter].

JON BACH: Thanks