Following my previous post about test videos where I asked why don’t people record them, I opened several discussions at several QA communities across the web regarding the subject:
- Software Testing Club
- SQAForums
- Israeli QA Forum (in Hebrew)
The people have spoken. All of the above discussions were great and yielded some powerful feedback, so I’d like to say thanks to everyone who participated.
Below is a summary of the most common opinions about recording test and bug videos following the above discussions. Even though Steve Swanson beat me to the punch with his summary about recording test videos I would like to present my own. Here we go!
Most people don’t record test videos for tests and bugs
- Text and screenshots are sufficient.
- Video file sizes are too big and take up too much space on our servers and databases.
- It takes too much time and hassle to take videos of tests and bugs.
- Videos can’t be searched.
Some people do record test videos
- As an aid to bug reports, not a replacement.
- When doing Exploratory Testing.
- For hard to reproduce bugs.
- For complicated scenarios.
- For bugs that are graphical by nature.
Conclusion – videos of tests and bugs are useful in certain cases, but aren’t necessary in others. They really help as an addition to a textual bug report, but cannot completely replace it.
Therefore, we shouldn’t expect to see test and bug videos being recorded all the time, and maybe even not most of the time. Yet according to our statistics we rarely see test videos being recorded!
Some people seemed to have a certain resistance to testing videos mainly due to technical constraints. However, most of them simply aren’t true.
- Text and screenshots are sufficient
As quite a few people noted in the discussions, there are cases when text and screenshots aren’t sufficient. Such is the case with hard to reproduce bugs, complicated bugs, graphical bugs, and more. - Video file sizes are too big
We use our own custom and top secret compression algorithm to make test and bug videos really small. Download the Testuff Video Player and check out an example bug video of a silly bug in Testuff 1.0 with the forgot my password feature (to be fixed in Testuff 1.1
. The 11 second video only takes about 74 KB. With most bug videos being around 10 seconds, there shouldn’t be a worry about file sizes. - Video files take up too much space on our servers and databases
Testuff videos for tests and bugs are actually uploaded to the Testuff servers, so it’s our worry if they take up too much space. - It takes too much time and hassle to take videos of tests and bugs.
When running a manual test in Testuff, the test runner already includes the video recorder controls. All you need to do is target a window or area to record, and hit the record button. If some step fails, the video is automatically attached to the bug report. See the video recording help section for more info.
Of course, there is a whole lot more that could be done with testing videos to make it even better and simpler, and we have plenty of interesting ideas. Still, assuming that we did solve these technical constraints in a decent manner, we should see more videos being recorded.
The only conclusions that I can draw from the above is that either people are way too lazy to make videos, or the idea is still too new so that only a few stray sheeps are using it while the rest of the herd continues along the known path.
Well, if you aren’t recording test and bug videos today and haven’t even tried, you might be missing on something. There isn’t a need to record a video for every bug, but quite often they come in very handy. Give it a try. Besides, recording videos can be fun and magical too. I highly recommend you to watch Be Kind Rewind and see why:
blogstuff
October 15th, 2008 - 7:46 am
Pretty offensive conclusions you draw from that – I would certainly take offence at being labelled ‘lazy’ or one of the sheep
Thing is, a lot – maybe the majority – of testers have much bigger problems to deal with than whether to log their bugs with video or text. Problems such as persuading management to take testing seriously, to get properly trained testers, get time, equipment yadda yadda yadda
October 15th, 2008 - 9:50 am
Thanks for your comment. Up to now no one has said explicitly that they don’t record testing videos because they have bigger fish to fry. However, I don’t see what that has to do with it. The things you mentioned are problems in human and working relations. Recording testing videos is a technical tool to improve your bug reports or your exploratory testing. Sounds like apples and oranges.
It’s definitely way easier and much less time consuming to take testing videos than convince your managers that QA is important. In fact, testing videos may even help you a little to do just that. When people see a bug happen in front of their eyes they seem to treat it more seriously than just reading or hearing about it. That would be quite an experiment to see if testing videos improve your status as a tester in your company.
October 15th, 2008 - 5:06 pm
First time commenter/reader Innocent question:
If a defect is well written outlining the impacts to the user and system listing the exact steps and data to recreate, why is a video even necessary?
Perhaps if the steps to get there are involved can I see the usefulness but I get plenty of attention and responsiveness when I log highly detailed bugs with supplemental screenshots and datafiles and the essential steps to replicate. When I list a bug I never leave out the impact to the user and system at large.
Media is cool but I tend to agree with pkirkham in the challenges to get that in play for a standard. I know my servers for our tools are hand me downs from 1991 with disk space challenges so our storage issues on a tool such as this would be a maintenance drain. Like the starship enterprise and the warp drive maintenance. We need more disk space as it is. So are you saying that this utility compresses the video files to take up much less space than text and screenshots and data files?
October 15th, 2008 - 5:19 pm
Teenbopper, please reread my post as you seem to have missed some important points.
For one, the conclusion I drew out of the discussions, that test and bug videos aren’t always necessary, but sometimes they can be very useful. This depends on the nature of the bug and the AUT.
In your case, isn’t it possible that you could open new bugs faster with videos and a little bit of text rather than elaborate bugs with lot’s of text and screenshots? A few people even talked about this in the discussions. Isn’t it also possible that it’s easier and faster for the developers to see a 10 second video than taking a few minutes to read a bug report?
Secondly, I also mentioned that at least with Testuff the video files are uploaded to our (Testuff) servers, not to yours. So you don’t need to worry about disk space, that’s our concern.
I’m sorry to hear the servers you are using are obsolete. If they’re that old, why doesn’t your company upgrade? Disk space actually comes cheap these days in unbelievable quantities. Even my Gmail account is more than 7GB big, more than I’ll ever need for my emails, and that’s nothing these days.
October 15th, 2008 - 7:42 pm
RE: uploading to your servers – I think that in terms of uploading details of our proprietary software by way of video file to a 3rd party source would be a violation of our company disclosure policy. No way to sell that point to my IT teams.
Secondly I agree on the point for cheap space. Always has been a ponderable in terms of what is valued for resources and tools for QA. But in many operations QA budgets are tight and they make due with the leanest implementations possible before money is granted for upgrades. Money is poured into the dev environments first then QA is given what dev used to use.
Understand your points on the view being useful timesaver on the video playback. Hope that you understand the challenges many QA face in terms of just cutting over to a new vendor.