June 7, 2017
For many testers, Selenium is a well known tool and a familiar friend. For others, it may be something you are curious about but haven’t had a chance to do much with yet. All of our panel has had some level of experience with Selenium, and Brian Van Stone visits us again to tell us what he’s been up to and how his Adventures with Selenium have informed his automation processes and overall testing.
Also on the Selenium and testing tools front, what is up with the VC community making big bets on software testing tools? Is it Silicon Valley business as usual, or is there something else going on here? We investigate, or pontificate, or at least we offer an opinion or four.
- Zephyr Raises $31M in Funding
- Software testing platform QASymphony raises $40 million
- Insight Venture Partners invests $165 million in automated testing startup Tricentis
- Sauce Labs sops up $70 million round
- SauceCon: The first Sauce Labs User Conference
- Cross-browser Testing (in your Browser!) with Sauce Scout
- Appium: Mobile App Automation Made Awesome
- Sauce Labs Acquires TestObject to Expand Real Device Mobile App Testing Platform
- Compatibility between Selenium WebDriver and the major web browsers
- Applitools: Automated Visual Software Testing
- How To Run Your Tests Headlessly with Xvfb
- Running Selenium With the New Headless Chrome
- Sauce Connect Proxy
- Stack Exchange
- The latest trends from the Agile Alliance Technical Conference
- NYC Testers
MICHAEL LARSEN: Hello, everybody. Welcome to The Testing Show. I’m Michael Larsen, your show producer, and today we have our standard guests. Let’s welcome back Perze Ababa?
PERZE ABABA: Good morning, everyone.
MICHAEL LARSEN: Jessica Ingrassellino?
JESSICA INGRASSELLINO: Hey there.
MICHAEL LARSEN: Matt Heusser?
MATTHEW HEUSSER: Good to be here.
MICHAEL LARSEN: For a definite throwback to early days of the show, let’s welcome back Brian Van Stone?
BRIAN VAN STONE: Hey, guys. It has been a while, hasn’t it?
MICHAEL LARSEN: Yes. So, one of the neat things here is that when we recorded the pilot episode and the first couple of episodes, Brian was on to help us get our sea legs and we’ve gone on to talk to a lot of other people, but we’re coming back to Brian. More to the point, I’m going to get out of the way so that Matt can continue doing what he does, which is moderate the show. Matt, over to you.
MATTHEW HEUSSER: Hey, thanks. Welcome, everybody. Even if you’re a long‑time listener, you might know Brian well. Brian is actually a Solutions Architect with QualiTest—the sponsors of the show. It’s been a while. What have you been up to, Brian?
BRIAN VAN STONE: Oh, you know. All work, no play. No, I’m just kidding. It’s been fun. We’ve got lots of stuff going on with lots of technologies—Selenium not the least among them—which I guess we’ll get to a little bit today. I’m just trying to figure out how we can apply more technology to do more awesome things every day.
MATTHEW HEUSSER: Well I don’t know if you want it on the show, but you’re going to be a dad again, right?
BRIAN VAN STONE: That is true. I have another one coming. June 23rd is the due date, so it’s creeping up. I better get ready.
MATTHEW HEUSSER: Well, we’re happy for you. We will keep a light on for you when you come back. The first thing I want to talk about for news, besides the new baby, is all of the investments that are happening in the test-tool space right now. For example, last year Zephyr raised $31 million Series C round of investment. Zephyr is a test management company—test case management. You create test cases in your tool, and they also have some neat integrations to different automation and reporting. So, you can see one report between both your tooling and your human that is running test cases. QASymphony just raised $40 million in a similar round of investment. Tricentis just raised $165 million in a similar round of investment, and SauceLabs just raised $70 million. By the time this recording is out, they’re going to have done their first conference—their first SauceCon. Usually, when a company raises money, I say, “There’s no story there. So what? Another company raises millions of dollars,” but these are all in the test-tool space in a reasonably close period of time. But, I wanted to throw it out to our guests. I’ll start with Michael, as he is in Silicon Valley, the land of the big Venture Capital Firms, and might have a better feel for what’s going on. How do you interpret all of these Investments, Michael?
MICHAEL LARSEN: Well, I think that it’s pretty clear that because of the way that software, at least now, is being developed and the fact that there is a lot of investment in Cloud architecture, there’s a lot of investment in continuous integration, continuous delivery, continuous testing, deployment, and your favorite fill-in-the-blank “continuous”, I guess, [LAUGHTER]. The idea is that you want to be able to have infrastructure that you can pull up and break down as you need it and not have to rely on that infrastructure being in your building, so this is just a continuation of that. These places have been around for a long time. We’re familiar with them. I think what we’re seeing, though, is that more and more companies are actually seeing the benefit of having this pool of resources that they can use when they need them, and not use them when they don’t. Just an example Sauce Labs, in addition to just being able to have your machines run up and use a bunch of them, you can also do straight-up exploring. You can create machines anytime you want to with an operating system and a browser, and you can test on just about anything. When you’re done, you destroy it and it goes away. Just that benefit alone, I think, is where a lot of venture capitalists are seeing benefits in the sense that there’s money to be made and people are willing to say, “We don’t want to shoulder the machine investment any longer. We want to be as lean as possible and as quick as possible.” Just the way that Agile development has now become much more commonplace, there’s a lot of space for pools for being able to use these type of tools, especially in a Cloud environment, at least that’s what I think is happening here and why I think a lot of people are paying more attention to it.
MATTHEW HEUSSER: Yeah. I think that that’s definitely part of it. I wonder if the fear would be that, “There’s just too much money swimming around venture capital circles and there’s good money chasing after bad ideas.” I don’t think that’s the case here. Dollar amounts are huge though. I remember when Socialtext would get $5 million in financing. I think the total VC investment over the life that I was at that company (and before I even came) was, maybe, $25 million. These numbers are just huge, so I’m hoping they do something really good with that money.
BRIAN VAN STONE: I’m a little curious, because a lot of these companies, they’ve been around for a while, if we look at the way the money has flowed for Sauce, for example, this $70 million is more than they’ve gotten in the past 4-or-5 years, and they’ve been well established as kind of a global leader in the Cloud/Selenium Grid space for a while now. That’s nothing new to us. So, I’m wondering, as much as the “continuous-X” argument behind this makes so much sense to me, some little part of me wonders if there’s some other discreet event that’s making some more of this money move around, might require a little more research.
MATTHEW HEUSSER: Yeah, $155 million for Tricentis seems like a lot. Now, what I would say is for what I expect them to do with that, I would want them to put it in product development and then tie into CI natively better and tie into version control natively better and tie into something like AWS. So the tool will do more than push testing through it, but you’ve actually got connectors so that developing a whole CD pipeline is easy, which, right now, it isn’t. It’s kind of like this weird, bizarre Black Art that you hire some expert to do, because it’s super hard. So, I’m hoping the tools advance in that way. I should say that, the companies I listed have been clients of Exelon at one time or another. We’ve done work with them, whether it’s R&B or marketing or research or writing. I think that’s whatever, you can call it a “conflict of interest,” it’s worth disclosing, if you’re listening, but it allows us to fund the sorts of research and thought experiments that I think do end up helping the community, just like this podcast. So, QualiTest is a Sauce Labs partner. That’s worth disclosing too. Lots of good stuff going on. To speak to Brian’s point, Sauce Labs, from what I can tell, is already break even. They’re already making money. So, this investment should allow them to put more into R&D, I would think, to build better solutions faster, because I think they’re already there. This should just be putting the accelerator down.
BRIAN VAN STONE: Yeah. I think what we may actually see as a side effect, with the money flowing into Sauce, is more investment into Appium as a tool, which, as much as I love Appium, it can use a little more investing. It’s not quite where Selenium is, so I’m hoping to see some better capabilities in the mobile space as a result.
PERZE ABABA: As you can see, that’s actually starting now. I mean, earlier this year, Sauce Labs has announced that they acquired a company called “TestObject,” which is sort of the equivalent of Selenium. I don’t know about real devices, but it’s on the Cloud. You’re shifting, really, from the browser space into the mobile space. They do have an existing solution for that right now where they have native emulators that you can spin up and tear down and run tests, but the challenges that I’ve seen personally, when it comes to native emulation solutions, is that it is an emulator. We have libraries now that require a dedicated GPU. For example, if you use WebGL or if you use OpenGL, for example, that requires a dedicated GPU processor, and none of those will be testable or at least you won’t be able to interact with that particular system with real devices. On the Cloud, that is something that they have a solution for, you know, moving forward. The other thing that I’m also seeing is that in the evolution of any of the products that you’re dealing with, you start with the genesis of that product, you shift into something that’s custom-built, then you turn it into a rental product, and then finally it becomes a utility. So, you see the amount of work that people do to get their systems up to speed to be able to use that product gets cheaper as it moves from genesis towards a commodity space, so I believe a lot of these tools right now, the majority of them, are still in between custom built as well as product and rental. The value that I’ve seen—I’ve worked with Sauce Labs since 2009, I still work with Sauce Labs now, and I guess that’s going to be my disclosure. I’m actually part of the Sauce Labs Customer Advisory Board—I have the chance and the ability to be able to talk to other customers within Sauce Labs and learn and glean on the solutions that they’ve built; but, when you jump between a custom-built products as well as into the rental space, things do get a lot easier. Again, it’s not just that but we also can glean into the analytics of, “What is going on with our tests?” You know, “What is the primary cause of failure? Is it on our side? Is it on their side?” But, all of this data is now available to you. When, before, you had to create a completely custom solution for you to do that, and that’s all something that you, as a company, had to pay for. Either that’s one or two people so that you can get to that level of analytics or information that you want to get out of the tests that you’re performing, but all of that is provided for you by Sauce Labs or probably even with these other companies now.
MATTHEW HEUSSER: Eventually this will get to a realistic expectation of what this technology can deliver and for who, and then we will be onto the next thing. Welcome to software. Speaking of a “realistic expectation” and what this technology can do for you, QualiTest has been talking about Selenium this month, which is an open-source, probably‑needs‑to‑be‑programmed tool for driving a browser and checking the values that it finds. Maybe Brian should tell us about some of the successes and not so much successes that he’s had using Selenium?
BRIAN VAN STONE: Ugh. I’ve had a complicated relationship with Selenium over the years. No, I’m just kidding. Selenium’s been pretty good for us, all things considered. When it comes to driving web browsers for the automated-tester mindset, it’s one of the cleanest and most intuitive tools to use. It’s doesn’t come with a lot of bells and whistles; and like you said, there’s programming activity that needs to take place. We don’t want to lie about any of that. You need to write some code when you’re going to work with Selenium, if you want to be successful with it, and that’s just kind of a fact of the tool. I think what kind of sets it apart from a lot of the other things in the test-automation world is that you don’t have all those bells and whistles. You don’t have built-in reporting and integrations to all the other tools in your suite—like your release engineering and build-automation platforms and things like that. You’ve got to build some of that yourself; but, at the end of the day, when you just want a nice, clean API for driving web browsers to automate your web apps, it’s exactly what you need and nothing more, and in a lot of ways that’s what I really love about Selenium. That being said, more recently, I don’t know if you guys have been following all the changes to the Firefox Browser of late, they’ve really cracking down on a lot of their security implementations and things like that. There’s also been some problems with Selenium compatibility with Firefox[x], just so that we’re not painting too pretty of a picture. There have been struggles along the way too; but, all things considered, we’ve had great success with Selenium at QualiTest. Most of the time, it comes by way of us building automation frameworks around it; because, when you work with Selenium, you need to be ready to supplement some of that extra functionality that you don’t get, you want to integrate with your test management tools. A lot of it’s device integrations, but not all of it is device integrations. You want good reporting. You want good logging. A lot of that, you’ve got to be ready to put in place; but, once you get there, you end up in a pretty happy zone.
MICHAEL LARSEN: I can definitely account to that. We have been using Selenium for—oh, I think, honestly—probably close to the entire time that Socialtext has been a company. Actually, that’s not entirely true. But, as long as I’ve been there, we’ve had a very solid and robust suite of hundreds—I guess you could even say thousands—of Selenium tests that allow us to do our continuous integration pipeline and testing that we do and every story has tests that we update and that we modify. We are currently running on a slightly older version of Selenium because of exactly what you said. Selenium provides a great API. It provides an option to run and drive browsers. But, to get it to do other things and to get it to do them effectively, to integrate with your product, you do have to code, you do have to write your own extensions or other additional pieces to it, and we’ve done that extensively. There are benefits to that; but, at the same time, there is a challenge in the sense that when new features come out, there’s a lag into when you’re going to be able to use them, because either you have to throw a lot of stuff out and re-tool or you have to write up a bunch of stuff to be able to utilize the new features.
BRIAN VAN STONE: In some ways, though, with Selenium, you kind of have the freedom to implement the way you want. When you go to some of the other test tools on the market, you’re kind of forced into the paradigm of test automation that is dictated by the tool; but, Selenium being just an API to drive browsers, allows you to decide what paradigm you want to live in once you get on top of that tool. Because a lot of the struggles that we see with some of our clients is, they want to be more continuous. They want more test automation artifacts, but maybe they don’t necessarily have all the skills in their QA organization to be writing code and producing these kind of tests. Well then, you know you have the opportunity to sit back and say, “Okay. What kind of design approach do I want to take to my test automation around Selenium? Do I need to enable business users to be able to get involved in this automation, or is it all developers that are going to be doing it? What kind of skills do I have available, and how can I put a solution in place that allows me to best utilize the expertise that I already have in my organization?” To go along with the kind of native complexity that you get from something that requires those lower-level skills, you also get a lot of flexibility around what kind of solution you can deliver at the end of the day.
MATTHEW HEUSSER: Let’s get into specifics for just a couple of minutes. So, if your audience hasn’t used Selenium before, right? It’s a code library. So, in any object or any language you say, “Hey, give me a new browser. Hey, browser. Point yourself to www.amazon.com. Hey, browser. There should be a text box there called, ‘thing to search for,’ right?” Hey, browser. There should be a button called, ‘search,’ right? Hey, browser. Type in, ‘Lessons Learned in Software Testing,’ into the ‘thing to search for,’ and then click the button. Hey, browser. There’s a book called Lessons Learned in Software Testing. It’s a link. It should appear on the screen now, right?” Every time I said, “Right,” it’s going to search. You’re probably going to use it with some other framework that can do something with that as search, like send it to Visual Studio or send it to Jenkins or send it to the command line or keep it in memory; and then, when you’re done, “I passed 50 out of 60 checks.” Then, “Here are the ones that failed.” The beauty of that is, whether you’re writing C# or Ruby or Python. I think the Perl Bindings don’t work anymore; but, at one point, Perl worked pretty well with Selenium. What other popular languages? Java.
BRIAN VAN STONE: I haven’t, no. But, in the past, I’ve done a little bit with PhantomJS, which is designed exclusively to be kind of like a Headless lightweight.
MATTHEW HEUSSER: Right.
BRIAN VAN STONE: JS Implementation.
MATTHEW HEUSSER: Right.
BRIAN VAN STONE: So, that one is really quick too.
MATTHEW HEUSSER: Michael, you haven’t used Headless Chrome?
MICHAEL LARSEN: Not as of yet, no. Most of the bindings that we have, as you were talking earlier about, are the “Perl Bindings,” and I would say that up until recently the person who was most responsible for that Perl Binding worked at Socialtext, does not anymore.
MATTHEW HEUSSER: Yep.
MICHAEL LARSEN: So, that’s one of the things that’s getting us to look at how we can update and modernize our implementation and part of it probably will mean that, like you said, even though a lot of our previous implementation was done in Perl, some of our future stuff is going to be done in something else.
MATTHEW HEUSSER: Now, Jess has done a lot of REST API testing. At least, I’m trying to remember. That’s what you were doing at Bitly, right?
JESSICA INGRASSELLINO: We were doing some through Selenium. Largely, though, we would try to be verifying different things that would happen as we shortened the length and making sure that we were getting the responses back that we expected and that sort of thing.
MATTHEW HEUSSER: Every now and again, I’ll get asked, “Can you use Selenium for REST API testing?”
JESSICA INGRASSELLINO: Yeah.
MATTHEW HEUSSER: You could; but, if you’ve got plain text going across a wire, why not just grab that plain text, use something like Perl, get the response, then put in a string, and then just parse it and figure out whether it’s right or not? Selenium is just going to slow you down, I think, in that case.
JESSICA INGRASSELLINO: To me, in the tooling, in all the companies I’ve worked at, I’ve used Selenium, except my first, where I didn’t know how to write code. So, [LAUGHTER], and I think that the value proposition that I’ve seen from it is my ability to—I like watching the tests, actually watching the browser screen—match up what’s happening on the screen with the code or the test that I’m watching in my terminal and if I have a log window open at the same time. I like monitors. So, I use a lot of those. Using those different tool sets all at the same time, I can quickly figure out if there’s an error happening in my test, “Oh, I can see that it’s happening,” because, “Oh, the element is on the screen.” But Selenium says, “It’s not finding it.” So, “How do I fix my test?” Because I know it’s a test error, because that element is showing for the user. I think I use it a little differently than most people, [LAUGHTER], do. I’ve definitely used it for the general purposes of running it against the CI system, running it on deploys, and that kind of thing. But I’ve also used as an accompaniment to exploratory testing or to regression testing, in terms of actually watching the browser so I can kind of see how things are behaving over time.
MICHAEL LARSEN: Matt has previously mentioned this. He talked about a little feature, which is actually a series of tests that we put together, called “Slideshow,” and the benefit to Slideshow is exactly that. It’s meant to be watched; but, one of the things that I have found of value with the Slideshow implementation that we used at Socialtext (and still use at Socialtext) is, “What if we used a different metaphor?” Instead of saying, “We have a train, that we can order the cars any way we want, but we’re still going from point A to point B. What if we used our automation as a taxi cab instead?” I think that’s also where we’re getting with a lot of the investments in automation, you know, as we tie back into our news segment earlier, is the idea that if we can set up our automation as a taxi cab, one of the nice things about using Selenium and driving the browser and getting it to work is that you can get into the nooks and crannies of your application over an extended period of time; and, by doing that, you can actually look at state conditions that you… manually it’s going to be too longwinded for you or you’re going to feel like, “Ugh. Really, I don’t want to have to do all of this. I’ve already got all the steps set up. Let’s go through all of these iterations.” So that what I get to after loading 15,000 users and after doing some verification with those users, now I want to go take a look at something and, “Oh, hey. That looks interesting. What just happened there?” Pause test, and now you’ve got your browser open and you can go poke around with a whole lot of stuff that’s been manipulated on your system and some interesting state changes that you might not, frankly, get to unless you’ve spent a lot of time setting up and exploring manually. There are benefits to being able to explore; but, to explore, first off, sometimes you need a good guide.
PERZE ABABA: Gleaning on what everyone else has said before, “Selenium is just the tool that can control a browser. If you want it to do other things, then you have to programmatically include a ton of other modules or libraries that are available out there so that you can perform more stuff.” I think the key that I’ve been hearing from everyone is that, “Look, you have the tool. You can pretty much build the framework on top of that tool that can perform the things that you want it to perform, but you do require skill for the processes around it. So, how do we look at this tool so that it’s still easily maintainable?” One of the things that we did was really looking at a more comprehensive way to be able to use Selenium as a tool because of the problems that we had. Not everybody has the skillset to be able to write code, so you have to look for people who can actually do that or you can hire some developers to maintain that for you and you can just lean on your existing testing team to be able to perform the actual testing and write some automation and plug that through a continuous delivery pipeline. If you have a framework that’s easy to install and you can write tests pretty straightforwardly or you have custom flows or setting some weird vales on your browser session IDs or your cookies or stuff like that where you have to write very specific code and how to get that done, if that’s taken care of, then your testers can just go ahead and write these tests themselves. Because, as soon as we introduce a new future, we can just ask our software development engineer in Test to create an equivalent step so that you can write a corresponding automation script within your tests. The reporting is not that good. It doesn’t come “out of the box”, so there’s a ton of frameworks out there that you can use, that you can easily ingest the results, and then look at the differences in the failures and you can introduce logic on how to deal with those failures. The tests that I’ve really seen where we’re able to use Selenium effectively is mainly because we understood where we’re going to be able to use it for and we’ve established those limits; and, if those limits are challenged, like for example if you use Sauce Labs it has a feature called “Sauce Connect,” which gives you the ability to set up a reverse proxy between the Sauce Labs infrastructure as well as behind your firewall. That gives you the ability to test internal projects without compromising your security, without opening your tests to the public. I think it’s the small things like these that help you with, you know, how to use the tool more effectively. I think for the folks who have been using it for a long time, they do have a lot of horror stories, and I do enjoy listening to, [LAUGHTER], Michael about this, when he talks about the challenges especially, because you’re dealing with a custom binding. The moment that binding support is gone, it gets more difficult to maintain your test as well. So, that’s where the language conversation is on, “Where can we pivot?” Like, “What is the scripting language that we’re going to be using moving forward, and how are we going to transition to that?” But, for me, the story that I really have for Selenium is that it’s been a very effective tool, mainly because I’ve been working in the web space and that’s the majority of the things I’m doing. Now, we’re slowly shifting into native applications, and you have Appium which uses the same JSON wire protocol as the existing Selenium API. It’s quite an easy switch from a test-writing perspective, but the whole set up around it kind of changes. Because it’s no longer just a browser that, you know, you have to deal with you. You’re dealing with permissions on your native device, your permissions on your local machine, etcetera, and etcetera. For me, as an encouragement to those who are struggling, there’s a lot of help that you can get out there. There’s a lot of knowledge that you can glean in especially, [LAUGHTER], on stack exchange, if you want to look into how to simplify, how to make things maintainable and scalable as you perform more tests.
MATTHEW HEUSSER: Yeah. Appium is a whole different beast in terms of configuring and setting it up, because it’s designed to make your mobile devices feel like as easy as running a browser. It’s not. So, hopefully these new rounds of investments are going to help Sauce continue to develop that, make it easier, which is going to drive adoption. The easier these things are to use, the more people are going to use them. So, it’s a virtuous circle. With that, we’re getting tight on time. So, before we go, the podcast is going to be coming out in June. Is there anything folks are doing that’s new and exciting, anything you’ve talked about? I’ve got out an article. I went to the Agile Alliance Technical Conference in April and then I went to Agile and Beyond in May. I’ve got two articles on that, and we will put that in the Show Notes. Anybody else have anything new and exciting that they want to talk about.
BRIAN VAN STONE: Well, I think it’s probably appropriate throughout that QualiTest is going to be hosting a Webinar on June 28th, which is actually going to be focused on, utilizing page object model design patterns with selenium to solve some of the more common automation challenges and tricky objects that you run into in web applications. So, if anybody’s interested, definitely look for the materials and announcements around that. Yeah. June 28th look for a Webinar from QualiTest.
MATTHEW HEUSSER: Great. Well, put a link in the Show Notes if we can. Anybody else? Last call?
PERZE ABABA: I just wanted to shout out to the NYC Testers Organization, and Jess is actually part of that. The reason why I wanted to bring this up was, in our last MeetUp, a couple of weeks ago, one of the participants said, “Hi” to me and wanted me to extend greetings to everyone at the show, because she is an avid listener to our podcast. So, Shereen Bashar, wherever you are, thank you for listening, and we hope you continue listening.
MICHAEL LARSEN: Thank you, again.
JESSICA INGRASSELLINO: Yeah.
MICHAEL LARSEN: That’s fantastic.
JESSICA INGRASSELLINO: Woo!
MICHAEL LARSEN: Yay! [LAUGHTER]. Also, we want to definitely make a point that, if you have questions, comments, concerns, or frankly, if you’d just say, “I’d love to be on the show,” we encourage you to contact us at email@example.com, and we will answer your questions. We will be more than happy to encourage you if you want to be on the podcast. If you have something you want to say, let us know, and we want to be able to include you in the conversation.
MATTHEW HEUSSER: Yeah. The testing world suddenly just got a whole lot more interesting with some of these major announcements. So, we will see what the next six months will bring, and we will cover it here live. I hope you enjoy listening to us on your commute or at work or in the gym. Thanks for coming, everybody. We will see you again in two more weeks.
MICHAEL LARSEN: Thank you.
PERZE ABABA: Thanks, everyone.
BRIAN VAN STONE: Thanks for having me guys.
JESSICA INGRASSELLINO: Thank you.
[END OF TRANSCRIPT]