Atlassian Confluence blogs atWikiConsulting.com

JIRA Studio is our latest hosted offering, which combines JIRA, Confluence, Fisheye, Crucible and SVN via a number of plugins to provide a seamless development environment to the end user. Search is an area where the integration between products has been lacking so far. Each application provides powerful search mechanisms, however there was no global mechanism to search all applications.

Implementing global search in Studio proved to be one of the more interesting features I’ve worked on so far. The requirements were:

• Quick search (from navbar) results in global search by default
• User can select specific application search from the quicksearch drop-down
• selecting a specific application takes you to that application’s search (with results)
• Studio search results: current app takes top position and returns the most results
• Ie, if you search from JIRA, Issues is on top, if you search from Confluence, Wiki is on top
• Current app returns 5 results
• All other apps return 3
• Clicking on “More Results’ takes you to that application’s search screen (eg. Issues ‘More Results’ takes you to JIRA search with same criteria)

After doing a technical spec I estimated this piece of work would take 81 hours.

Design

There were a few design choices that had to be made:

• Search via each of the applications RPC interface
• Search using Sage
• Build a new Search API in SAL and add a search aggregator to Applinks

A search needs to be executed using the currently logged in user’s privileges in each application to ensure only results that a user is permitted to see are returned.

Searching via RPC was somewhat problematic because there’s no support for trusted application calls yet, so we would have had to add this ourselves to the RPC interface in JIRA, Confluence and Fisheye (actually I never got as far as investigating Fisheye so I’m not sure if this is true). The only other options would have been to make authenticated calls to the RPC interface using the existing token mechanism (which would have required using a user’s existing login credentials).

Sage was even worse because it would have had the same trusted application problem as RPC, plus it would have required us to run yet another application with Studio. We really did not want this option.

Finally the third option was chosen: Adding a simple search API in SAL which was then implemented for JIRA, Confluence, Fisheye and Crucible.

Implementing the SAL search API

After discussing a few options we chose to implement a very simple API, that would work for us now, rather than trying to come up with an API that would allow for all sorts of app-specific searches in some uber generic way. In the end each application’s SAL layer needs to implement this SearchProvider (comments stripped for conciseness):

public interface SearchProvider
{   
    SearchResults search(String username, String searchQuery);
}

SearchQuery can in fact contain additional parameters by appending them to the original query e.g.: searchString&maxHits=20. Perhaps a little too simple but it serves our purposes for now. The search will return a SearchResults object, which may contain errors, a list of matches, total time taken for the query, as well as the total number of hits available for a particular query.

A SearchMatch is defined as:

public interface SearchMatch
{
    /**
     * Absolute URL to reach this search match.
     * @return Absolute URL to reach this search match.
     */
    String getUrl();

    /**
     * Title of the search match
     * @return Title of the search match
     */
    String getTitle();

    /**
     * An excerpt of the search match.  For example this could be a summary of the Wiki page.
     * @return excerpt of the search match. May be null.
     */
    String getExcerpt();

    /**
     * Contains more information about the source of this match.
     * @return The source resourceType.
     */
    ResourceType getResourceType();

}

Implementing these search providers was an interesting excercise in exploring each applications internal Search APIs (or lack thereof). From most difficult to least difficult to implement:

1. Fisheye
2. JIRA
3. Crucible
4. Confluence

Confluence was by far the nicest and easiest to implement (and this is coming from a JIRA developer!). We have a lot of work to do though in general to make our internal APIs more developer friendly!!

Search Aggregation

Now that SAL supports search I had to implement search aggregation to actually get some results to the user. For this I built a simple search webwork plugin in JIRA, that would call a SearchAggregator in the applinks layer. This SearchAggregator would call to the local SearchProvider directly (which is available via dependency injection) and call to all remote apps registered in applinks via a RemoteSearchProvider implementation. The RemoteSearchProvider simply makes a REST call to a Search Servlet I added. This gives us trusted apps for free, since we already have all sorts of utilities in Studio to make trusted app calls to REST services.

The SearchAggregator aggregates results in parallel using a threadpool, which means the overall result will be returned as soon as the slowest resources returns. Threadlocal’s proved to be a bit of a problem when it came to detecting the currently logged in user, which is why the SearchProvider interface above requires a username to be specified.

The User Interface

As usual the user interface is what took the longest. The actual search results page was quite quick and easy since our UI team already provided a HTML mockup including CSS (you rock!). The majority of time was spent with the quicksearch dropdown, allowing users to select which application to search in:

Implementing this dropdown in one application was easy, but implementing it in JIRA, Confluence and Fisheye for all the browsers we support took a LONG TIME!

Results

Here’s the results screen of running a search in Studio (scaled down slightly from original size):

Search will be rolled out to users with the next release of Studio. Before this can happen there are still a couple of things we need to cover however. There’ll be a blitz test to ensure there are no XSS attacks or other weird encoding issues with studio search. We’ll also have to focus on any UI quirks in the different apps in different browsers with the quicksearch dropdown. I’ll also be doing a load test on a real instance to ensure search isn’t too slow.

According to JST-636 I spent 84.5h on development, plus there’s still quite some work outstanding for the load and blitz tests. I went quite a bit over the original estimate due to the difficulties encountered implementing the quicksearch dropdown in the various apps and browsers. The lesson is: Don’t forget to add LOTS of padding for any non-trivial UI tasks in Studio!

• Comments Off

I suspect it comes as no surprise to any developer if I say that Facebook has become extremely popular (not to mention extremely valuable!) over the last year.

The feature on Facebook that has most intrigued me is the activity feed.

For those that don't know, Facebook's feed shows you activity (any form of action taken really) of all your friends across various different functions in a single place. Or alternatively, it looks like this:

Why can't you do something similar for a development team?

I played with this in Fedex VI with my Atlasbook project. While it was fabulously interesting (and could generate all manner of activity feeds), it is a little heavy weight for a normal development team and relied on replicating too much data to be practicable.

So how could we build something similar into JIRA itself?

With our upcoming JIRA Studio service, I've been championing a new feature that shows a stream of activity across various development applications in a single place. We're calling this the "Stream plugin".

I thought I'd blog about it to get some feedback and let you know a little more what we're up to.

How does it work?

Basically, right now we're streaming activity from a series of different applications to be displayed in a series of different places.

The initial activities we're tracking in the plugin are:

• JIRA / Issues - issue creation and commenting, issue modification, file attachment and issue resolution
• Confluence / Wiki - any form of content creation, commenting and editing
• Fisheye / Source Code - source code commits and their comments
• Crucible / Code Review - review creation, comments and closure

To start with, the plugin displays this stream in a number of different ways:

• A project portlet - showing you the stream for a project that you can add to your JIRA dashboard
• A project RSS feed - an RSS feed of updates for a project across all applications
• A project tab panel - a permanent tab on the project's homepage showing the latest updates

Beyond this, in the future we're looking to add per user streams (for a user's profile page for example), Confluence macros (to put an up-to-date stream into any wiki page) and issue focused streams (see the updates across all applications focussed on a single issue).

What does it look like?

OK, enough text - here's what the portlet looks like at the moment (with a little amusing test data):

Where can I get it?

Unfortunately, the answer is you can't quite get it yet - but it's coming very soon.

Firstly, it will be included in the upcoming JIRA Studio service for all customers - already configured, setup and streaming all your project information across all applications.

Secondly, the entire feature is built as a plugin - so behind the firewall Atlassian customers will be able to install it themselves.

The plugin degrades really nicely to handle whatever applications you have. For example if you just have JIRA, it will only show you issue data - but this is still an extremely useful way to keep up with a JIRA project.

As soon as you install the simply awesome JIRA Fisheye plugin (if you're not showing your code next to each issue, you're really missing out!) and point that to your Fisheye / Crucible instance, presto! Commits and reviews will start streaming.

This plugin should be available for behind the firewall customers some time this quarter I imagine.

I hope this has piqued your interest in the potential for viewing your team's development activity stream. If you have any ideas for improvements (or you think the entire idea needs work!), I'd love to hear from you in the comments.

• Comments Off

As part of the upcoming 3.12 release of JIRA and 2.7 release of Confluence, a neat little feature will be added that allows the two applications to communicate in a trusted way, such that it is possible for Confluence to request information from JIRA on behalf of the currently logged-in user, but without having to re-authenticate the user on the JIRA end. Sounds intriguing? Read on!

The Problem

Confluence has a nifty macro called the JIRA Issues Macro that allows users to embed issues into a page straight from JIRA. While the macro works well, it is not entirely secure as you have to store JIRA user credentials right there in the macro. The reasons we require the user credentials are clear. Firstly, your JIRA instance might not be public, and enabling an anonymous account to access your issues is not an option. Secondly, you might have security restrictions on your issues, and so you don't want to allow someone to leak issue data from your "Top Secret" project by using the JIRA Issues Macro.

The Solution

(click for more...)

In order to satisfy these requirements, but keep our credentials safe, we decided to "look up" to establishing trust on the application level, and in doing so, we get trust on the user level for free (or close to it). Our functional requirement now becomes: "If I (JIRA) can trust that you (Confluence) are who you say you are, then I'll authenticate the user you give me without their password". The benefit of taking this approach is that trust only has to be established once between the two applications, and must be approved by a JIRA system administrator. Once trust has been established, it is entirely transparent to the users of Confluence - they no longer have to provide their credentials to the macro, as we simply send the user name of the currently logged in user.

The Implementation

The problem of application trust is not a new one, and there are several established protocols which help solve these problems. When designing our solution, we examined a few of these protocols to see if we could simply implement them, or at least borrow ideas from them.

OpenID was one of the protocols examined. In a nutshell, OpenID is about sharing identities in a decentralised fashion, so that users can just refer an application to a single existing identity rather than having to manage an identity per application.

While OpenID is cool and helps us generally with the problem of identity management between Confluence and JIRA, it unfortunately doesn't give us any benefit when trying to conduct trusted communications between the two applications, as it is a identity/user-centric solution, not an application-centric one. It would also introduce additional dependencies on OpenID servers, which is something we can't assume when devising a solution for the general user base.

OAuth is a younger protocol which was designed to grant consumer applications access to a user's protected resources held by a service provider, without having to give away the user's service provider credentials to the consumer application. This is achieved by redirecting the user between the applications and requiring them to authenticate themselves on both ends. Once authentication has been set up, the applications communicate with a shared token that encapsulates all the details of the request for resources.

At first glance, this protocol looks like a pretty good fit for our problem:

• consumer application == Confluence
• service provider == JIRA
• protected resources == issues.

OAuth also does not require any additional infrastructure to work; the applications simply talk to each other, periodically requiring some user input. This however highlights one of the downsides of the protocol for our situation. It is very well suited to one-time requests for protected resources, where you want user supervision of the authentication for peace of mind. But in the context of Confluence and the JIRA issues macro, you wouldn't want the user to be redirected to JIRA to authenticate themselves every time they viewed a Confluence page that contained the JIRA issues macro.

One way to counter this is to again "look up", and treat the entire consumer application as a single user, so that the one-time request for resources sets up the trust between Confluence and JIRA indefinitely, and then all requests made by the JIRA issues macro will just include the currently logged in user. This could work, but the effort required to make OAuth work didn't seem worth it when we could roll our own solution. There was also a problem that, at the time when we were discussing implementation details, no Java libraries for OAuth were available to utilise (but this appears to have been remedied).

Details of the Trusted Applications Authentication (TAA) protocol

So after checking out what was already on offer and not finding anything suitable, we decided to roll our own protocol. In order to minimise engineering effort, it's design is fairly straight forward:

Setting up trust between JIRA and Confluence

1. JIRA sysadmin requests a trusted application authentication certificate from a Confluence instance by providing the base URL of the instance. The certificate contains Confluence's Trusted Application ID and Public Key (generated specifically for use with the TAA protocol).
2. JIRA validates the certificate and asks the sysadmin for a few extra details about the trust relationship, such as Name, Timeout, Allowed IP Addresses and Allowed Request URLs.
3. JIRA stores all this information in the database.

Making a trusted request from Confluence to JIRA

1. Confluence sends a web request to JIRA, appending additional headers to the request, including:
   • Timestamp (nonce) of the request + user name of the currently logged in Confluence user, encrypted with a symmetric key (generated on the fly)
   • The symmetric key, encrypted with Confluence's private key
   • Confluence's application id (as stated when trusted was established)
2. JIRA attempts to decode the encrypted headers, using the stored information about the relationship. A couple of checks are conducted to validate the request:
   • The trusted application id refers to a valid trusted application
   • The user name specified exists in the JIRA user base
   • The agreed timeout length has not expired
   • The request originated from a trusted IP address
   • The resource being requested matches those specified in the URL Match list
3. If any of these checks fail, a response is sent to Confluence, indicating the reason for failure. Otherwise, JIRA will authenticate the specified user for the duration of the single request, and respond with the resources (i.e. the JIRA issues)

Limitations and Risks

As the protocol has a simple design, there are inherent limitations and risks to be considered. Firstly, this protocol can only work if the Confluence and JIRA user bases are the same. Because Confluence automatically appends the currently logged in user to the request header, JIRA will always return an error for that request if the same user does not exist in JIRA. There is also a more subtle issue whereby the user does exist in JIRA, but they do not have the correct level of permissions that you would expect (and hence do not get the correct set of issues returned by the macro). Of course, if you are using a single user base solution, such as Crowd, you wouldn't have this problem.

Secondly, the protocol is not immune to certain security attacks such as man-in-the-middle and replay attacks, though it is considerably hard to pull these off successfully. For the man-in-the-middle scenario, the attacker needs to be present during the trust relationship establishment, so that their public key is registered with JIRA instead of the intended client's. But since the JIRA sysadmin manually enters in the base URL of the client application, this means that the attacker needs to have poisoned the JIRA instance's DNS so that the domain specified (e.g. confluence.atlassian.com) points to the attacker's domain (evilconfluence.attacker.com). If this is the case, then I'm afraid you have slightly more pressing concerns than unauthorised access to your JIRA instance. To successfully pull off a replay attack, the attacker would have to be able to spoof the original client's IP address, and they would also have to be able to forward their spoofed requests before the timeout window on requests closes. Security risks such as these will not likely be an issue for most Confluence and JIRA customers however, as we expect the common scenario to be that both instances are running inside a trusted network.

"Trust no one unless you have eaten much salt with him" — Cicero

Finally, remember that the protocol is all about trusted relationships. I'm sure if Cicero was still around today (probably died of heart failure from eating too much salt), he would tell you that while this feature will allow you to do some cool things between Confluence and JIRA, as a JIRA admin, it's your responsibility to ensure that you really can trust the Confluence instance you're talking to.

• Comments Off