Musings of a Tech Transfer Enthusiast
In the past three weeks we have collected anonymous usage data from developers (with their knowledge and permission!!) using our search tool in Visual Studio. Learning from the failed Usage Data Collection effort in Eclipse, which perhaps had too broad a scope, our data collection is tightly focused on improving search, and nothing else. Thus, after reviewing even three weeks of data we have several useful conclusions. Primarily, that developers who utilize recommendations (shown above) when searching code are more satisfied with the results and execute 20% less failed queries than those not using recommendations.
As you can see in pane (b) above we collected data for 25 days. We collected 204 log files from 44 unique users (excluding all Sando developers). Interestingly, as a comparison, Eclipse's Usage Data Collection collected only 453 new users' data over a 14 day period in 2008, even though they were potentially collecting from all Eclipse users.
During this 25 day period we collected anonymous metrics about 363 user queries executed on local projects via Sando. As seen in pane (a) 90 utilized the recommendations provided by Sando. Of those 90, 41 were utilized pro-actively (e.g., by selecting an entry from a dropdown) and 49 were triggered automatically (e.g., when the user searched for a word that did not exist in the code base). Of the 41 active queries 25 were "lookup"-style searches, such as using the autocomplete to help remember the name of a method, and 16 were exploratory-style searches, such as searching for phrase like "open file".
Using the data from these 363 queries we tried to gauge user satisfaction level using a few different metrics. First, we measured the query failure rate. We considered a query to fail if a user did not review any of the results, as indicated via UI events. As you can see in pane (e) queries that utilized recommendations had a 20% lower failure rate, which certainly leads to higher user satisfaction.
We also used both the short-click and long-click satisfaction measures, as are often used in web search. The intuition is that if a user clicks and views a result and then immediately returns to the result list that result was not helpful (a short-click), where as if a user clicks and views a result for a long period before (if ever) returning that result was helpful. As you can see in pane (d) queries with and without recommendations provided about the same rate of long-clicks, which is a positive metric, whereas queries with recommendations showed a significantly lower short-click rate, which is good as short-clicks are a negative satisfaction indicator.
As I mentioned at the beginning of this post, we only collect information aimed at helping us improve code search. So what's useful about the information I've presented today? Well, in the short term, Sando users should start using recommendations, immediately. They will be more satisfied with the results and fail 20% less. While we intend to do our part to improve the recommendation UI, possibly the most useful thing about this data is that it gives Sando users field evidence to support feature adoption. And when you think about it...how often have your other software tools given you evidence that a new feature actually makes you better? ;)
When using code search tools, such as Find in Files, developers don't want to waste mental energy crafting the perfect search query, they just want to find relevant code. Previous versions of the Sando Code Search Extension helped by providing users with conceptual autocompletion. Even with this help we found users creating ill-advised queries (including very smart, experienced developers). For instance, users would perform literal searches (e.g., document.Add) without adding the required quotes (e.g., "document.Add"). Or they'd enter a (previously) unsupported query format, like a wildcard query (e.g., custom*Document). To make it more natural for developers to interact with Sando we expanded our query interpretation code in today's release. Developers can enter a keyword query (e.g., open file), a literal query (e.g., File.Open), or even a wildcard query (e.g., Document*Create) and expect Sando to take care of the details of returning relevant results.
Please give Sando's new query interpretation code a spin and let us know what you think! If you like this new direction we'll look into even better query interpretation in the future.
Here's an example search using FindInFiles (left) and Sando (right):
Make it work with 2012! :) -naspinski, from reddit
Well, the people have spoken, and we've (finally) delivered. The Sando code search extension is now available in VS2012 (in addition to VS2010), thanks in large part to Kosta Damevski. As we release this version I realize that not all readers are familiar with Sando's Raison d'être. For those of you new to Sando here's the top few reasons why we've spent the last year creating it.
Above I've quickly described a few reasons that we expect Sando to perform better than available regex-based tools and I've used a few scenarios to explain why. However, its important to know that Sando is not primarily based on my personal insights. Sando is built upon the huge body of code search research, started by Andrian Marcus's thesis work and so ably continued by researchers like Denys Poshyvanyk, Dawn Lawrie and David Binkley, Lori Pollock, Emily Hill, and many others. Thus, you can download and use Sando, assured that it's providing you with high-quality search results influenced by cutting edge advances in software engineering research.
Sando is available as a Visual Studio extension for VS2010 and VS 2012
When fixing bugs most software developers start with a search. I know. I've watched them. They read the bug report, reproduce the problem (sometimes), and then search for relevant code. This is when it gets ugly. Developers either wander semi-randomly around the Solution Explorer (Roomba Style), struggle to use Find in Files with increasing levels of anger (William Foster Style), or I've even seen one user step through almost the entire program in the debugger (Slowskys Style). Please, for goodness' sake, next time you search try Sando Style. Our newly released version of Sando (0.3), is a free, open source extension for VS2010 that makes searching your C#, C, and C++ code as easy as using Google.
VS Gallery Download: http://bit.ly/SandoDownload
Getting Started With Sando
After installing Sando the main view will appear in your Visual Studio Studio, as shown to the left. The UI consists of the main input box (upper left), the search button (upper right), the results view (bottom center) and the normally folded advanced options (middle center). To use Sando simply open a solution, type in a search term, and press search (or hit return).
Once you search, you'll see results like those shown to the left. Notice how the method OpenFile is shown first, because it matches both search terms, whereas the lower ranked terms only match one of the two terms. Each result is shown with a representative icon that communicates both the element type (e.g., method, class, or plaintext) and the accessibility level if relevant (e.g., public or private).
See a result you like but not ready to commit to opening it? No problem, after the search is executed just press the down arrow to scroll through results, expanding each result to view a partial snippet when it is selected. In the picture to the left OpenFile has been selected and you can view the first few lines of this method. To open the method in the VS editor simply hit return or double click on the result.
Getting Fancy with Sando
So far we've discussed the very basics of how to use Sando. However, the entire Sando team dog-foods Sando (i.e., we use it everyday ourselves) and so it's got a few advanced features baked in, which I'll discuss here.
Halfway through your task and decide you need to search again? No need to pickup that mouse, Sando is designed to allow you to work mouse-free. Hit Alt+Shift+S and the Sando view will appear with the cursor in the text box, ready to search. Using this shortcut I find Sando to be a great way to open any file or method I'm thinking of, and thus I almost never open the Solution Explorer.
One of the things we noticed when using Sando is that sometimes you just don't want to see certain results. For instance, we have huge text files of words in our test directories, which are indexed by Sando, but sometimes we don't want to see these results. In the advanced section of Sando you can deselect any program type, such as a method or a comment, or any other type, such as a text line. Doing so means that deselected types won't appear in your results. This provides a quick way to eliminate unwanted items from your current search.
You may have noticed that Sando often changes the text just under its main input box. This is the status line and it is used to give you feedback on your query. If your query has hits it will tell you how many, and if it doesn't Sando may give you tips on how to improve your query. You may also notice the message that "Sando is still performing its initial index...". This occurs when you open a new project that Sando has not analyzed before. This only happens once and, when analyzing Sando itself, takes only ~1 minute to complete on a 1.7GHz machine with no SSD. For a machine with SSD I've seen this index happen almost instantly for sizable projects. Note that this indexing is necessary for Sando to be able to return search results virtually instantly on even *very* large Solutions.
Finally, while we've done our best to choose good defaults for most users, you may want to customize a few things. To do this, visit the Sando options page (Tools > Options) where you can configure the directory where Sando stores its index as well as the maximum number of results returned, which is by default set to 20. I recommend resetting this to 50, but I'm crazy about search so I didn't force this default on the masses...
Search With Style
Hopefully this post gives you a good idea of the functionality that Sando provides. My team and I have spent many a late night improving Sando based on early adopters' feedback and we truly think this release represents a big step forward in usability and robustness. If you have been using Sando now's the time to start recommending it to your friends. If you have been considering using Sando, you should now enjoy a much improved experience. Next time you search, I hope it's Sando Style!
VS Gallery Download: http://bit.ly/SandoDownload
Video Introduction (dated): http://bit.ly/SandoVideo
The IDE is a software developer's toolbench. For the most part it's an ordered chaos, with all of a developer's favorite windows and extensions at the ready. Unfortunately, the IDE's complexity helps to hide some of its glaring shortcomings -- it's easy to focus on the shiny wrenches across the front while ignoring your Grandfather's 1970s disc sander that's about to melt down from misuse. Like Grandpa's sander, code search* is a tool that's in dire need of an overhaul.
* In this article code search refers to the act of searching over the code base that one is currently working on, not searching over repositories of code.
Code Search Sucks.... Really?
Developers, having grown up on awful search tools, have become numb to the problem. Many may not even believe it is a problem! In order to illustrate the sorry state of code search let's explore an analogy based on web search...
I've got a new web search service that I'd like for you to try, called Gaagle. It's incorporated all of the key features of modern code search. Let's try searching for "Visual Studio" using Gaagle:
Ohps! I misspelled "Visual Studio", but Gaagle doesn't fix my mistake (or even point it out). Gaagle simply obediently reports that there are, in fact, no matches for the search terms "Vizual Sudio". Thanks Gaagle!
Having corrected my search terms to "Visual Studio" I search using Gaagle again. This time I get lots of hits! However, the hits are visualized as a list of paths, leading to an exciting game of "what's behind that link!" After clicking on each link in the list I have a pretty good idea (and a sore mouse-finger).
But wait, there's been an upgrade to Gaagle! Instead of returning results in an ordered list Gaagle now returns unordered results. That means that each and every web page in the world that contains the terms "Visual Studio" are returned to me for my search. No worries! I start wading through the thousands of (mainly irrelevant) hits until I find what I'm looking for.
Not ready to switch to Gaagle? Well surprise... you're already using it! If you're using Visual Studio 2010 or earlier your code search (i.e., Edit > Find and Replace > Find in Files) operates almost exactly like Gaagle.
Join Us in Fixing Code Search!
Visual Studio, so powerful in many other ways, has neglected code search. This January we're starting a project to modernize code search, called Sando. If you long for a modern code search experience please join us at http://sando.codeplex.com/. We're looking for great developers (with C# or Java experience) but also for any suggestions or requests that can help shape our search tool.
David Shepherd leverages software engineering research to create useful additions to the IDE.