David C. Shepherd
  • Blog
  • Colleagues
  • Contact
  • About

Transferring Tech

Use Recommendations to Avoid Code Search Failure

9/16/2013

 
Picture
Drop-down recommendations
Picture
Auto-corrections with links to related recommendations
Picture
Tag cloud recommendations
In the past three weeks we have collected anonymous usage data from developers (with their knowledge and permission!!) using our search tool in Visual Studio. Learning from the failed Usage Data Collection effort in Eclipse, which perhaps had too broad a scope, our data collection is tightly focused on improving search, and nothing else.  Thus, after reviewing even three weeks of data we have several useful conclusions. Primarily, that developers who utilize recommendations (shown above) when searching code are more satisfied with the results and execute 20% less failed queries than those not using recommendations.  

Details

Picture
Summary of the captured data
Usage Data
As you can see in pane (b) above we collected data for 25 days. We collected 204 log files from 44 unique users (excluding all Sando developers). Interestingly, as a comparison, Eclipse's Usage Data Collection collected only 453 new users' data over a 14 day period in 2008, even though they were potentially collecting from all Eclipse users.
Query Breakdown
During this 25 day period we collected anonymous metrics about 363 user queries executed on local projects via Sando. As seen in pane (a) 90 utilized the recommendations provided by Sando. Of those 90, 41 were utilized pro-actively (e.g., by selecting an entry from a dropdown) and 49 were triggered automatically (e.g., when the user searched for a word that did not exist in the code base). Of the 41 active queries 25 were "lookup"-style searches, such as using the autocomplete to help remember the name of a method, and 16 were exploratory-style searches, such as searching for phrase like "open file". 

User Satisfaction
Using the data from these 363 queries we tried to gauge user satisfaction level using a few different metrics. First, we measured the query failure rate. We considered a query to fail if a user did not review any of the results, as indicated via UI events. As you can see in pane (e) queries that utilized recommendations had a 20% lower failure rate, which certainly leads to higher user satisfaction.

We also used both the short-click and long-click satisfaction measures, as are often used in web search. The intuition is that if a user clicks and views a result and then immediately returns to the result list that result was not helpful (a short-click), where as if a user clicks and views a result for a long period before (if ever) returning that result was helpful. As you can see in pane (d) queries with and without recommendations provided about the same rate of long-clicks, which is a positive metric, whereas queries with recommendations showed a significantly lower short-click rate, which is good as short-clicks are a negative satisfaction indicator. 

So What?!

As I mentioned at the beginning of this post, we only collect information aimed at helping us improve code search. So what's useful about the information I've presented today? Well, in the short term, Sando users should start using recommendations, immediately. They will be more satisfied with the results and fail 20% less. While we intend to do our part to improve the recommendation UI, possibly the most useful thing about this data is that it gives Sando users field evidence to support feature adoption. And when you think about it...how often have your other software tools given you evidence that a new feature actually makes you better? ;)

Comments are closed.

    Author

    David Shepherd leverages software engineering research to create useful additions to the IDE. 

    Archives

    May 2018
    May 2017
    November 2016
    May 2016
    October 2015
    September 2015
    August 2015
    January 2015
    September 2014
    May 2014
    March 2014
    February 2014
    January 2014
    October 2013
    September 2013
    August 2013
    July 2013
    May 2013
    April 2013
    March 2013
    February 2013
    December 2012
    November 2012
    October 2012
    July 2012
    June 2012
    May 2012
    January 2012

    Categories

    All
    Abb
    Extensions
    Grants
    Ide
    Software Engineering
    Software Engineering Research
    Software Tools
    Visual Studio
    Visual Studio Extensions

    RSS Feed

Powered by Create your own unique website with customizable templates.
  • Blog
  • Colleagues
  • Contact
  • About