Friday Pre-Fosdem events

Floss Community Metrics Conference

On Friday, I attended the Floss Community Metrics conference. We arrived at about 3:20 so missed a lot of the talks. Here are my notes on the talks we attended.

“GitHub users per city in Spain” by JJ Merelo (Granada University)

The main takeaway of this talk is that studying communities affects them.

“Communities work in different ways., studying them has an impact”.

The specific example used was the impact of publishing rankings of github contributions from Spain, on a city by city basis. JJ demonstrated that publishing the data changed the behaviour of hackers in the area, for example, many more people declared their city in their github profile who had not previously done so.

At phpList we have been actively trying to get people to state some “personal” information on their community profiles, as it allows us to thank them for their work more publicly, for example, on twitter.

One interesting question raised after this talk addressed “Quality not quantity”. The fedora project issued badges for number wiki edits, however, some community members started abusing the system to get badges without making helpful edits. The solution described was to use something like code reviews not commits, ie: community validated metrics should be used for rewards/badges etc rather than just actions.

You can see the talk slides here.

“Gerrie: A crawler for Googles code review system Gerrit” by Andy Grunwald

Gerrit is a code review tool used by big projects such as ebay and android. Review data is very large and complex. Gerrie is a tool used to crawl gerrit and extract data which can be analyzed

Slides here, and code here.

“Measurements for success – Building a community around Juju” Charles Butler (Canonical)

I have to state first of that I dislike the name JuJu, I have no idea what possessed Canonical to name software after something relating to a religious/spiritual belief. I find it rather offensive and maybe a little racist? Anyway – onward.

This talk provided a useful list of metrics to consider examining, and a nice toolkit list too. It’s development focused, and the tools are pretty technical.

First level metrics described as:

Downloads
Number of commits – Reviewed / Open
Bugs Filed / Triaged / Opened / Closed
Emails to the mailing list
IRC Engagements
(JuJu project doesn’t yet collect info on the last two)

These are described as being Easy to “slime”, and also they provide an incomplete picture of the community.

Second Level Metrics (Combine first level metrics)

Activity by Time Zone
Contributors by sub-project
Repeat commits vs drive-by commits
Sentiment Analysis to rank positive/negative
Engagements

Toolkit

iPython Notebook
R
NodeXL – it’s free software but built Microsoft excel?
D3.JS
Metrics Grimoire
Data Science Toolkit
Apache Storm & Apache Spark

I have personally used an R script while working with User Prompt, but I can’t wait to have access to more user friendly graphical tools.

Slides here (pdf) or on SpeakerDeck here (is that proprietary software? Can’t seem to find any licence info!)

“How metrics motivate” Ioana Chiorean (Mozilla)

Takeaway from this talk is that cash is one possible motivation but personal performance-related recognition is more important

A community wants to feel part of a project and see they make an impact.

• Project metrics are not the same as as personal metrics
• Personal metrics – why is an individual community member working for you?

So, to exemplify, as a community manager I might collect data on how many people are contributing, but that won’t necessarily motivate the community: we could also collect and show metrics about community/individual targets.

To help get personal metrics which motivate the whole community you need to put community in driving seat when deciding on what metrics to use

You can visualize, gamify (positive, no yelling) review and reward individuals work

Slides are here

“Quantified living environment: metrics in hackbases” David Potocnik (Hackbase Lanzarote)

Odd but cool talk which discusses, in some respects, the extreme of community metrics – measuring all aspects of a “live in hack space” (a hackbase) to improve efficiency and happiness of the people living there. Link to slides.

“Experiences building a dashboard from community data” Federico Capoano

A great looking, small project which collects data from github, mailman, analytics’s of docs (from google) and puts the out put in grafhana which looks VERY good. Uses influx db. The code is here and the nodeshot project being monitored is here.

There are no slides, but here is a screenshot:

“Choosing a community related key performance indicator” (KPI) (Mozilla)

This talk had a great takeaway about “the funnel” and community metrics: the funnel is where you have a lot of community members making occasional contributions at the top, some of those become more and more involved until you end up with a small group of core members. To grow your community you need people to come through the funnel, it needs to be easy to get more involved.

In this case (MDN) the funnel is described as: readers > editors > core

You should chose a community-related KPI that measures the bottom (the core) of your funnel as well as the top, especially if the gap is big and the core do most of the work.

“Feedback loops” Kadir Topal (Mozilla)

The take away from this talk is great:

“Focus the communities attention on what really matters and let them see the impact of their work.”

This talk was especially powerful because of a particular graph (below) showing the impact of a successful change in strategy for Mozilla community help forums (I think?) which lead to all questions being answered within 72 hours, 100% (ish) 24/7.

The idea was to give the community on-line feedback as to their process which in itself becomes a motivator (like Ioana said earlier, metrics can motivate), specifically there is a status bar which shows how many questions are left.

In addition, the goal became more realistic which keeps motivation and the chances of success as high as possible – specifically, questions which were not answered are removed from the queue and given to a specific team.

Also mentioned this book  “influence, science and practice” which seems worth a read.

Slides here.

“How to move towards analyzing software development with free software” Open Discussion

A great chance to discuss the issues of community metrics in more detail. I am still pretty concerned with how much projects are focused on development, rather than other aspects of community such as users or documenters. I think a lot can be learned from the UX research field, with talks just as “Björn Balazs on The Problem of Representativity”, which gives chance to see our users-only group as a contributing part of the community.

In addition, I asked how many of the projects measuring their development community actually build community metrics measurement capacity into the projects they are managing. As a community manager my biggest challenge has been getting data out of the community tools, for example translation tools or bug trackers. I think it highly likely that, say, theoretical “bug tracker project X”, are examining their own development community with metrics, while the bug tracker product itself provides no capacity for metrics when it is eventually used as in situ by external projects. These projects will then be, like me, battling to get data – scraping and whatnot.

My hope is that in the future, all tools used by communities will have community metrics facilities built in. The best way for this to happen is for Community Managers on these projects to suggest and use the features.

Fosdem

Fosdem was awesome right? So many great talks and so many people, plus a vegan food van! Hurray!

Saturday

Karen Sandler on Identity Crisis: Are we who we say we are?

An insightful keynote, which has helped me realize I am not the only person to experience some confusion around allegiances/hats/work-v-social etc.

In Free Software “we” say “we” a lot, but it means a lot of different things. People even say “we” in the same sentence twice meaning two different things.

As a lawyer she always needs to be thinking who’s lawyer she is at any given time. Who is the client? The Company? Individual? Charitable or Non-profit element?  She says this does not happen in other, non Open Source related situations. In Free Software employees often have many other affiliations, for example, if she is representing a non-profit each board member will usually have employment at a company which has vested interest in the non-profit.

She noted that the IRS has a keyword search for things like “Tea Party” – and also on “Open Source” in tax returns, where the return will receive more scrutiny. She understands the confusion, because “we” is so complex

Friends and Work

She wanted to understand why someone would contribute to Ubuntu (esp with licensing things) so she went to Ubuntu conference and met a woman who was not canonical staff but worked hard to contribute to the project. The main reason (of many) was

“This is where my friends are”

In the lecture hall nearly 50% of people worked from home. She described working from home, in pj’s on sofa, talking to friends – but it’s work none-the-less. I can very much relate to this. Also, often the company changes but the people remain the same (this is very much the case for me and Sam Tuke, we have worked together on perhaps four separate projects now, with more in the pipeline – the working relationship with each other is sometimes better than the relationship “we” have with the project itself!).

This blurring of home and work – personal and professional is confusing. Karen suggests this is a component of out of hand behaviour at conferences.

“We are confused” in many ways, not just roles but mentalities, even personally defining attributes, for example we could be “evil capitalist AND the ideological free software guy” at the same time.

People find the need to balance, and sometimes step back.

One huge confusing/conflict she describes is when the gnome woman’s outreach program (paid internships for women) was transferred from gnome into the care of the Free Software Conservancy to become “Outreachy” (she has worked for both, as paid and as volunteer). She is on the board of all three organizations and is lawyer to all three too! She simply could not participate in the legal work for the move (she cant negotiate with herself!) and also realized she couldn’t even decide what she wanted, because it was so confusing.

We need policies about default positions in community discussions

She mentions

• Better use of email addresses and aliases (personal domain for personal, business for business)
• Do the right things, worry, don’t worry.

Jonas Öberg on Automating Attribution

I only got the end of this, but I was really glad to see Commons Machinery speaking. The talk was about their new project elogi.io – which is a plugin that can identify photos and tell you the author and the licence. they have indexed all of wiki commons already!.

It’s super dooper! Find out how you can support them below.

Sylvestre Ledru and Lukas Blakk on A new version of Firefox is available

I didn’t get to hear all of this talk, however, it was a very good overview of how Mozilla Firefox are managing to keep releasing new versions every few weeks. They summarized the train method, see below, of releasing which sees each version go through

Nightly > Developer > Beta and > Release.

It would be a good watch for an organization with a big team looking to speed up releases, and contains details to bear in mind for a smaller organization with regards to efficiency.

The slides are here.

Anthonia Ghalamkarizadeh on Mozilla ID – Developing and protecting a living brand

A really fantastic talk about Mozilla ID, the project which allows the Mozilla logo to be varied and hacked by community users. The talk gave a good summary of how very flexible branding can still work, and then discussed whether or not this is a good/safe/sensible option in Open Source.

She described the current notions of living brand / fluid trademaks which include things like the Google doodles – the word and colours are trade markd, the design varies but the process is very controlled still.

Similar to absolut vodka bottle, where the bottle shape and typeface are copyrighted, but colours and design are varied, include artists commissions etc. AOL and MTV keep wordmark by vary the rest. Sometimes a brand, like Chiquita, Nutella and Perrier, can be so strong that they can do away with the wordmark too: the brand can still be visible without it by using she shape of a label or bottle.

In all these cases, the changes are very much controlled and vetted. With Mozilla id the control will be with the community. The result will be a constantly refreshed look and appeal, which can react to current events etc.

But how does this fit into trademark protection?

Normal trademarking is tricky anyway, it’s very static. Straying from core, etc, make it hard. Even Uniliver (BOO) “U” change from blokish to floral was seen as too radical a shift to maintain the original trade mark.

There is a slide on living brands (will try and find it) about best practices is good!

Do we want to use brand protection (trade mark) to protect the brand of an Open Source project

She says no conflict between 1) open source software and 2) protected trademark. The protected trademark is as much for the community as it is for the company.

When there is inadequate protection the community are open to exploitation as well as the company.

For example subscription traps, where you download what you think is Mozilla for free but you get billed for two years. Just one example did 170,000 invoices per week and made 16 mil euros profit. Also modified malware, bundling unwanted software and fraudulent surveys which forwarding info to scam sites or bullying advertising sites happen.

Mozilla community actually actively report misuse of the trade mark at “protect the fox”:  people don’t expect weakness on this issue they expect Mozilla to enforce it’s trademark and protect users from scammers.

So Mozilla ID is about allowing the community to participate in a fluid brand, but at the same time Mozilla can enforce backbone of brand (the workmark).

Donnie Berkholz (from redmonk) on Is your community healthy? Metrics on the top CM software – was replaced by panel discussion with Kara Sowles, Mary Thenvall, Dawn Foster, Nathen Harvey, Manrique Lopez and a dog called Ember.

Donnie was totally jet-lagged and wasn’t well enough to speak, so there was an impromptu panel discussion.

The notes are a bit random!

Code stuff is easier to get, one reason why metrics on developer aspect of community are easier to get. Also, industry had/has attitude of code is most valuable. Meego had translation software metrics. Community management camp? Bitergia – still developer focused but does also include mailing list discussion etc. Is diversity a good measure of community health? Your criteria, eg, number of commits, will shape the community too. Do people stay when they come? Or do they leave? Anti-community: eg patch is made, disliked by others, do you lose that person? Sentiment analysis? On irc? Do you publish your metrics? Puppetlabs do Public metrics, quarterly blog most, more on recognizing contribs, highlight key contribs.  Group in Madrid measured how long from topic on mailing list to actual commit. Pupetlabs measures people who take action not just join. Identify what are the meaningful metrics for your specific community. Having docs in github is good way to track community and get people in same space. Finding the data, eg local meetup groups might be assessed on meetup.com How to track data on tech u don’t own, eg blogs not on your site? Paid tracking of social media and blogs inc sentiment tc mostly the scope of marketing. How does new comer or exiting member see the community? Numbers show stay or leave but talking to users is better info. If metrics show lack of diversity, what tools re-dresses this. Watch a new user, what do they do, assess resources etc. Increasing this is manual, hard work, eg provide scholarships for female speakers, provide resources. Some things cannot be measured, eg, conflict and resolution etc. Own diversity and work to correct imbalance without excuses. A lot of people don’t want to be involved to become a metric (on gamification of diversity) – a space to exist is better. Resources over incentives.

Sunday

Björn Balazs on The Problem of Representativity: Challenges of User Centred Work in FLOSS projects

This talk was a lead up and pre-launch of the User Rights project.

“Good software is about us – the users. We hereby ask the communities of the applications we love and use to respect our right to improve the software. To exercise this right, we need a new category of user relationship tools that bridge the gap between users and developers.”

He explains that a persona is a representative, but artificial description of “a user” and that user centred design can create products much better. Users make a product great, they are the reason for the project to exist even – and there are many users and they differ a lot. The sheer size of the userbase is why we have personas.

Many users > “abstraction level” = artifacts, personas to represent the user base.

We can then see the project through the coloured lens of the artifact. When designing a new feature you can ask which group is most likely to be using a feature? eg, tech-shy group would require a different style of design than a power users group.

In Free Software we usually don’t know how many users a project has. If we knew, we could drill-down from there but it’s hard to actually reach the users.

Without knowing the users, you can’t create artifacts, so the methodology is broken: artifacts are not research driven when we make them without knowing our userbase.

A bug tracker is not an effective tool for communicating with most users, we need a “user relationship management tool”

User Relationship Management

This software should:

• Seduce users to participate (to a non coding user, the freedom to change the software is worth very little) the program  should to draw the user in.
• Bring users into the idea of contributing “your first contribution”
• If you can find what version they are using, survey them, then you get bench-marking results on each version.
• Offer adequate options for participation
• Engage users further in community work. So far in this idea it’s “us asking them”, next we need the user to start initiating the relationship
• Gamify, the idea is to fill our artifact with good, real info.
• Give the user task which are specific to them, eg, first language is Hebrew, we note that the Hebrew translation needs work
• You could get the user to choose wich persona represents them most, gamify improving persona.
• Manage and create personas/artefacts – we all need these

Part of the issue is we have biased sampling, we only talk to the inner circle. One clear indicator of this is the very low level of female users responding to survey requests at KDE (and phpList)

Discussion

IRC is a big block to getting contributions/growing the community. WordPress moved all conversations from IRC to slack. Within one day, they had 2000 people singed in and they stayed. So easy to use, very natural to them. “anything is better than IRC for users” WordPress now have more people, even though it seemed like “a lot of hassle” compared to IRC.

Users and developers have a different view of what is or is not “hassle”.

Easier for users might be more effort initially, eg signing up. and less effort in the long term.

Christopher Webber and Deb Nicholson on User-land and developer-land chat

This was a fantastic, laugh-a-minute talk comprised of a conversation between a Mediagoblin user and developer.

Bugs

Won’t-fix and no-fix responses are frustrating for users because they don’t understand, but they work well for developers.

Dev’s get annoyed with there is not enough info/feedback, however, users may work hard to give the right kind of info, but vocabulary is a problem. “recursive knowledge” is an issue. Even when dev’s and users are working super-hard to communicate well, there is still a barrier.

People don’t come back when issues are closed without communication. Tell a user how to discuss it more and why/what you are doing.

Talk which assumes that users are stupid is bad, politeness is good. Resources for how to write a better bug report are important.

User interface

Changes to the user interface can really annoy users, because they think they did something wrong and spend hours trying to solve what they think is a problem or a bug or an install issue – but actually is just a change.

We don’t want to be like “hey look at Free Software, it looks like the 90’s here”

Plug-in systems can allow transitions between old and new interfaces, so the old interface does not need to be maintained in parallel. Also “advanced” sections for old features.

Involvement

“When your favourite free software project has a little spinach in it’s teeth and you need to tell them”

Tango style-guide (this?) design style guide worth a look Styleguides help keep design consistent if/when designers leave (bus factor)

“What does it DO” – developers take to long to describe this. They say what language it’s written in before they explain the purpose.

Collaboration tools tend to be developer oriented. etherpad is okay, email is good. emacs is….

Ali Spivak on Maintaining & growing a technical community

Ali is the manager of the Mozilla Developer Network, a project “documenting the web”. They have 5 paid writers and many volunteers.

They break contributors info those who edit weekly, monthly, once in 6 months etc. Diversity comes from community, including diversity of topic coverage.

Contribution is not the same thing as community.

Community tends to be made from contributors, however, there are many contributors who are not part of the community. Community is a set of contributors. Also community people who don’t contribute in the “normal way” – eg run booths at conferences but don’t make edits to wiki.

You can measure breadth and/or depth of community.

Slides here.

Motivations

Intrinsic: autonomy, learning, curiosity, belonging

Extrinsic, badges, money, fear of failure, gold stars and reward points

If you reward kids for drawing, kids stop drawing out of that context. It removes the fun.

Transparency, community based processes promote participation, accountability and trust.

Be open: people have to find where and how you are open to contributions. How do you make sure you know where/how to contribute to a community?

Listen, be open, admit mistakes, move on and do better.

Find new ways new things for people to get involved, to get new perspective, new diversity, the users they don’t consider because you are in your own space.

Ask, how can you expand your borders?

Slides here.

Pablo Cúbico on Every pixel hurts

This was a great talk about User Experience (UX) in Free Software. Pablo defined UX as the experience a user has while using a product, and noted that the product doesn’t need to be a gui smart phone app, but could be a sys-admin command line tool just as easily.

User Experience

• Can be measured
• Can be improved
• Can be developed and designed.

UX is not a process, you are talking about an ATTRIBUTE of your product: it is not illusive, it can be measured and understood.
So, UX design is a process, and UX itself is an attribute. Additionally, UX is not beauty: people often trust spyware because it’s beautiful…

Design is not art, it is form and function together.

A common myth is that Free software UX sucks, commercial software does it better? However, he gave several examples where this is clearly not the case, including Instagram.

However, Free Software design does have some issues, which he lists in his slides. These include:

• Lack of participation – not enough designers involved
• No clear paths for contribution
• Disconnected lone ranger contributors
• Style guides not made often enough
• Everyone is a designer

He suggests a good path way to getting involved, which could be a useful resource for Community Managers looking at designing processes for involvement.

Many projects lack identity, font, colours etc need to be consistent.

“trying to please everyone, that’s the road to failure”

Slides here, and code here.