Drupal 8 fireside chat: A D8 PM Perspective

For the last 6 months or so I've been a volunteer project manager for the Drupal 8 core development process. I've been helping (as much as time allows), with the regular calls held bi-weekly to go over progress and with fleshing out some details on initiatives. One of my goals is to bring transparency on the calls for the community. I realize an update is overdue, and it will be coming soon. In the mean time, I wanted to share some of my perspective with you.

Take it with a grain of salt, I know many of you have more years of experience than I do in this group, and I hope to hear from you to enlighten me where necessary.

The Drupal Dream: The good, the bad & the ugly

The promise of Drupal as I see it, is that web people from all over the world, who speak different languages, and hail from varied cultures come together to build an amazing web development toolset that ultimately benefits the web community at large. It's all very kumbaya, kittens (live ones), rainbows and unicorns. Sounds great right?

Now the bad news: Drupal is a complicated beast as open source web tools go. Other web toolsets don't have the same environment issues, and I'm not talking about servers. Drupal decisions are made by community consensus. Dries and several other key delegates help to resolve conflict, but ultimately, the community decides. I've heard from Drupal community members that this is what sets Drupal apart from other platforms. It's also what makes Drupal decisions so hard to make.

That's where it can get ugly. This group is full of passionate, talented and knowledgeable people and for that, I'm grateful. On the other hand, there are many egos to balance, many itches to scratch and many many people to *try* to please. This is a creative environment that many label as a "Do-ocracy", meaning, "You want it, do it!", but at times it seems to me like a "Do-crazy".

Tensions run high occasionally, and that's to be expected. What is unfortunate, is that I sometimes hear about people impeding progress because they disagree, and that doesn't seem to go along with what Drupal is all about.

The Fireside Chat that Dries held a few weeks ago brought up some questions, that deserve discussion. But, I think keeping the big picture in mind is important. The initiative leaders are very dedicated to the community, and they're donating an incredible amount of time to making it better. I really admire their efforts and applaud them for all they are giving. That goes for the rest of the community of contributors as well -- you lot are pretty awesome.

Drupal 7 Lessons Learned

During the meeting, Dries and Angie shared some slides about lessons learned from the Drupal 7 release that I wanted to put in here in case you missed it (http://blip.tv/dries-buytaert/chat-with-dries-drupal-core-development-pr...). I commend the community for taking a beat to reflect on the past (More here: http://buytaert.net/drupal-7-development-process-retrospective).

So as we can see in this image, some formalization and structure are becoming a part of the process. But issues around Drupal version development remain, check out these next two slides:

What does this mean?

Most of these last 2 slides seem to revolve around the decision-making process (Initiative leader authority, community empowerment, release modeling, communication, prioritization, the list goes on). I know, you're going to say "The buck stops with Dries.", but realistically with all the decisions to be made, I think we can agree that he can't be everywhere at once.

From where I'm sitting, it sounds like we're facing a problem that is essential to Drupal's growth and success: How best to suss out, convey and support the community consensus?

Dries has chosen to implement initiatives in this release cycle to help solve this problem.

What I like about initiatives:

1. We're setting goals.
Goals are good and bring focus to our project. I believe goals are the basis for any project's success, and I've said so in previous blogs (http://www.bluesparklabs.com/blog/10-ways-rock-your-project-planning). If you don't know where you're going, you'll never get there. As a Project Manager on the D8 initiatives, I've seen some changes that are rattling cages, like #2 below.

2. We're being specific.
Instead of making a sweeping statement like, "we're going to make x better", we're trying to get a clearer vision of what exactly we are going to make better, how and why. We haven't time boxed anything at this point, but it doesn't seem to be an issue yet since we don't have a code freeze date. Another thing we lack, is specific assignment of tasks to resources, but that's the nature of the beast in a community of volunteers. However, steady help is here, and that is #3.

3. We've got leaders.
I don't think it's realistic to expect Dries, Angie or Catch to do all the work. I love the idea that people have stepped up and said, "I'll help!". That's the Drupal spirit! Furthermore, we need it. We need people who are willing to lead the charge and follow the details. It also means more outlets for the community's voice. Which leads me to #4.

4. We're listening to the community.
I've heard some think decisions are being made behind closed doors, but from what I've seen as a part of the Drupal 8 project, that is not the case. On every scrum call we've had, I hear all the initiative leaders pointing out suggestions, improvements and disagreements between their proposals and the community.

Why are these things bothering people?

1. The process is changing.
I'm definitely hearing push-back about the initiative process and version release model coming from different groups in the community, and even initiative leaders themselves. This is still evolving from what I'm seeing, so it's hard to know exactly what the end-game will be. Having lieutenants and initiatives is a departure from the "standard" method used in previous Drupal versions, and it puts some pressure on the leaders and their helpers to get things done.

The organic creation method is a fine way to go, but at some point, I really believe some organization is necessary. I think Drupal 8 has reached that point because these initiatives need support, more than just one person can handle. When we're talking about multiple people, multiple goals, thousands of issues to track, etc -- I think there is some real merit to adding more structure to the mix.

Definition of success, prioritization, planning can be powerful tools to employ to rally support and direct it where it's most needed. I'm not sure if the community feels like this is necessary or not, but I'm keen to know. Are they getting enough information? Is it digestible? Could you use more organization?

My gut and past experiences tell me that this stuff is useful and helps projects succeed.

2. The community wants more control.
Discussions I've had at and after Drupalcon London that lead me to think the community doesn't feel like they have chosen the initiatives. Imho, Dries asked for feedback and compiled it into a useful summary (http://buytaert.net/drupal-7-development-process-retrospective-summary), but the format of comments is not the greatest for determining consensus.

Not everyone who has a strong opinion necessarily wants to take the time to read comments and reply back. I think the IRC chats and fireside chats are a better public platform where people can just chime in, it's more organic.

I think there's a second problem here… the feedback was compiled, and the decisions were made, but I'm wondering how. Was the community making that decision? Should they? Should Dries? Should the initiative leaders? I don't have the answer to that question.

Maybe we should expose all of the arguments for each initiative online (pros, cons, who benefits, why, how, etc), and create a poll module to tally community votes on where their priorities lie?

3. Momentum is slow, and there are complaints about process.
People are expressing that the development process (having project managers, phase gates and critical bug thresholds, documentation, etc) is slowing down progress.

I think the initiative-style management of this release is going to help us nail down what/why/how/when/who, and may even save time in the long run. Sometimes good planning goes a long way.

In the Fireside Chat, Angie mentions that people feel like Drupal 8 momentum is slow due to a focus on Drupal 7 clean up before version 8 really got rolling. The question was put to the community, "are we doing this right", but I don't think they got an answer.

Angie said feels hopeful that momentum will improve, and I agree with her. We've only just begun the Drupal 8 journey, it seems preemptive to discount the community's momentum and moral after only 6 months! (Started during Drupalcon London, August 2011). Lots of things are happening, though they are in flux, as we've seen with the WSCCI initiative and the big direction change that Crell has recently proposed (http://groups.drupal.org/node/198538).

4. The community feels they can't/shouldn't work on non-initiative issues.
With all the hubbub about initiatives, there is a general feeling that other work doesn't matter, or worse, shouldn't be worked on.

These initiatives are important, yes, but that's not all there is to do in Drupal 8. Angie pointed out to me her surprise that people felt they couldn't work on things that aren't initiatives, and as much as we want to achieve these goals, we don't want to stop the contributors from scratching their own itches.

5. Distrust of Acquia.
Some in the community feel that Acquia is influencing these initiatives to serve their own agenda.

Based on everything I've said above, it's obvious that I don't believe that and I strongly urge the community to take into account my testimony as a non-Aquian.

What's next?

I don't think that the fireside chat solutioned the topics on its agenda, but it's a good conversation starter and upholds Drupal's ideals. Change is hard to manage, and communication is key to adoption. More of these chats, maybe one per topic with a goal to resolve it, perhaps with a poll/vote system to get consensus, would be a great way to keep the community involved in the decision-making process.

There are still some questions about the release model, and I hope more discussion on that will follow. I think the topic from the fireside chat "What if there is no consensus" is a valid one that also merits a good think. (ie: Does Majority rule in case of non-consensus?)

I realize setting public, specific goals will be stressful if they aren't met, but we all know that Drupal 8 Initiatives are dependent on volunteers, so imho, it's not going to be any one person's fault, but rather a community failure if something doesn't get done in time. And so what if it doesn't? Drupal 9 will need initiatives too ;)

My point is, we all do what we can, and a great bunch of people are doing a lot. These problems aren't going to be resolved overnight, or by any one person. It's going to take some time to hash out a comfortable process, and we should embrace change if it helps us create a better, stronger community and product.

What do you think about all this? Any ideas to help us resolve the problems that Dries points out? Any reactions to my reaction?

Resources:
Fireside Chat: http://blip.tv/dries-buytaert/chat-with-dries-drupal-core-development-process-5839273
Drupal 8 on D.o: http://groups.drupal.org/drupal-initiatives
Drupal 8 write up from Crell on WSCCI: http://groups.drupal.org/node/198538
Drupal 7 Development Process Retrospective: http://buytaert.net/drupal-7-development-process-retrospective
Drupal 7 Development Process Retrospective summary: http://buytaert.net/drupal-7-development-process-retrospective-summary
Core Gates: http://drupal.org/core-gates

10 Comments

Chris Weber's picture

Thank you Shannon and everyone is taking the time to make our favorite CMS even better. Herding cats would be easier, since there would be fewer variables to handle.

Right now, the Drupal 8 development phase we seem to be weighed down by these early days where the path to the end is unclear. I think we're asking the right questions so far and I think the the initiative leaders are doing a decent job of communicating back and advocating for their ideas of a better future for Drupal.

Over the past six years I've been either watching, loosely involved, or actively engaged with the Drupal community, I have been thoroughly impressed by the personable conversations and helpful nature of the community. That above all else made me feel welcome to participate in the community.

Shannon Vettes's picture
Shannon Vettes -

Thanks for your thoughts Chris, I think your comment about path to the end being unclear is right on the money. But, clear definition of an end game also has its drawbacks :P Thanks for your involvement, I too agree that this community has some personable & professional people in it.

Bojhan Somers's picture

One of the things I think we should evaluate, is whether the lack of momentum is because of initiatives.

You mention that setting goals creates expectations, and if these aren't met its our own fault. It is important to remember here, that this puts an incredible strain on a number of highly involved contributors to finish the roads that where taken. Because we simply cannot release D8, when significant parts are broken from a UX and DX perspective. By setting too ambitious goals, creating too many initiatives, getting the moment going too late - we risk repeating Drupal 7's year-long bug squashing. I know this is in the hearts and minds of the leadership of these initiatives, but it would also be great to get an understanding on how we approach these risks.

Dries raised an important point, we cannot expect certain things to happen without dedicated resources. The reason we still have no UX or any significant end-user change in D8 is because of this issue.

The fireside chats are a nice way for solving communication issues around initiatives, I wish there was a summary. I also still wish there was a "overall" page of initiatives (showing status, news, goals and those involved) that is kept up-to-date. Currently initiative communication is too distributed, and focused on solving through g.d.o (which is our only hammer).

Shannon Vettes's picture
Shannon Vettes -

Hi Bojhan,
WRT stuff being broken, I think a lot of time was invested in fixing core for D7 so we don't have to do so much in D8. That's surely affected the momentum. Safe to say you think that's a good idea, or iyo is it too much of a risk to slow down D8? I'm not passing judgement, just wondering what you think :) I'm also curious to know which initiatives you think are too ambitious, and which one's you'd trim off the list if you had to focus on some instead of all?

About UX, this has been a point of discussion on some calls, I'm sure things will pick up in that dept soon. I've talked with Angie about it as recently as Friday in fact :)

I'll mention a summary, maybe I'll write one up if no one else is game. I too wish for the overall page, pretty sure Angie has some good ideas for that, but finding time to develop it is tricky for someone as in-high-demand as she is :) Agree w/ you about d.o, hopefully we'll have a solution for that soon.

Thanks for reading/commenting!!

Ray's picture
Ray -

For what it's worth:

1. Drupal has become so complex, your average unpaid developer can't keep up or has already quit out of frustration. It seems to me the average "user" (non-developer) has been left out of the equation completely. Either you are a Drupal expert, or get the (expletive) out. You can't take that attitude AND expect to be #1 at the same time - it's incongruent.

2. Finish what you start: Please show me anything that's stable. We want stability. Website owners need to be able to count on the next update, not try it out and discover that all 6,000 comments went off into the wind because someone "oops" made a coding mistake. And here we are talking about D8. How about getting D7 even close to right, first? Clearly, why bother - we'll just roll right into D8, right? The problem with that is the 6 months to a year or more before the absolutely necessary 3rd party modules catch up...if ever.

3. In the end, 3rd party modules are what makes Drupal better. Unfortunately, there are more abandoned projects than anyone can count. And some of these contain rather critical and expected functionality. This aspect of Drupal needs to get reigned in. Every day I see new projects started that no one cares about, while the modules we need and rely upon languish in a state of "not my problem" mostly because those developers were smart enough to not bother with D7 yet.

4. You (they) rolled out and pushed D7 like a drug, and WAY too early. It's STILL not in a stable release state. Will it ever be? Or is D7 the Windows Vista version?

Sorry, really want to be "into" it, but it's difficult to "believe" the rhetoric when in practicality D7 has been a huge leap backwards. NOTHING works right. Fix core. Get the developers looking for something to do on track with what really needs to be done instead of spinning their wheels. Until these things happen, any other conjecture, especially involving D8, is a complete and utter waste of time.

benjamin melançon's picture

To Shannon, thank you greatly for your perspective. Drupal's goals are now big enough -- even in core alone -- that major project management seems necessary, and you are absolutely right that this will be a transition.

To Ray, to reply to your points in turn:

1. Drupal is still accessible to people who first get into development with it. Get the Definitive Guide to Drupal 7 for some paths into that (that's totally self-serving but it also serves as the full disclosure to responses to your remaining points...)

2, 3, 4. Please list specific problems because contrary to what you say most things work right in Drupal 7 and many people have been building all their sites with it for six months. Your over-the-top claims repeated in each point discredits what might be a valid critique of some particular area, because "no modules, nothing works" for Drupal 7 has no basis in reality in 2012.

Shannon Vettes's picture
Shannon Vettes -

Wow, Ray I'm so glad you commented, this is really interesting. Can I ask what perspective you're writing from: drupal developer (experienced or new to D), D user or D client?

As I mention in the post, a lot of work is going into D7 clean up, and I know Angie is working round the clock to keep improving D7 as co-maintainer. It sounds like this isn't being communicated enough and some kind of "This is what was fixed in D7 this week or month" post would be kinda cool to have to show that things are still very much buzzing over there, it's not all D8 business. (I think some others commented about D7 things, and I'm no expert, so I'll let them do the talking about stability.) I'm intrigued by your thoughts on D7, and like Benjamin am anxious to get some more information before we dismiss D7 as a platform that doesn't work. In my experience however, lots of things work in D7 on my projects :) Any borked things you'd like to share so maybe they can get some face-time here?

Any abandoned but very useful modules you're thinking of in particular? We could raise awareness about them together -- maybe get some wheels turning. What you describe is a risk of open-source, lots of people going in lots of different directions, but it's also another reason why I think these initiatives are useful!

I think D8 is happening no matter what the current state of D7 is -- so it's not a waste of time to commit energy to something that will be released and used imho.

Thanks for commenting & reading!

Ray's picture
Ray -

First off, I don't want to come off as a curmudgeon, but things definitely aren't all wine coolers and fuzzy puppies in D7-land. If I take the perspective that Drupal was built as a (very open) CMS, then one should not have to be a developer to use it. That's the whole point of a CMS, isn't it? My perspective is that of a user, sadly that's all I am so therefore whatever opinion I hold is baseless and clueless - right? (this comes from years of being beaten about the head and shoulders with blunt heavy objects by Drupal developers...)

When you say cleanup - it actually kind of trivializes it. There are many of us (I know for a fact I'm not alone) that are following hundreds of issues in the queue hoping and praying that any one of them might get fixed anytime soon. Granted, most are 3rd party, but lest I remind you there have been right around 15 critical drupal core issues since the first release. Then there are hundreds of major issues (at least some of which truly are critical, but who am I to split hairs?) Critical means it's a showstopper. Dead in the water. If you haven't or aren't experiencing any of them, that's fantastic! If you do run into a critical but can get around it without any help, that's fantastic too!! However, some of us are just downright stuck....waiting. I know, I know, I should be a developer!!! Some of us just aren't going to get there from here. But, we appreciate Drupal for what it can do when it works correctly.

I'm all for looking ahead, but seeing where things will get fixed in D8 and then "maybe" backported is a huge concern to me and others. It takes years to never for 3rd party modules to catch up after a major core release. In retrospect, it's no wonder so many developers have yet to upgrade to D7. There's no incentive, really. I wish I'd known better myself.

Borked? Where to start. Let's just say the upgrade path from D6 to D7 remains thrashed. There's more, much more, but that is bad enough.

There's a very long list of very useful modules that have no working D7 version. Weblinks module (just a random example) provides some rather key functions. The D7 version doesn't work. And there's no working alternative. So, you want to make a pile of links to other sites and actually manage them? Please, let me know how. Or how about having a nice gallery? In D6 I was using Gallery Assist - if you've never seen it, it is a rather incredible module. However, it now appears to be abandoned. But that's OK, with all of this talk about how great Media Gallery is, it doesn't matter. Except Media Gallery wants to pile every uploaded photo in the same directory. You just haven't lived until you've filled one directory with so many images that they can't be displayed by the operating system. These are just two very small examples. There's plenty more where that came from.

The article reminded me of some of the rhetoric I read early on last April and upgrading seemed like a good idea. Here I am going on 9 months later and I'm still regretting that decision. So, I guess the takeaway for this is simply that some of us read the cheerleading articles like this and while that's all well and nice, it just simply doesn't provide a realistic perspective. The hell is unfortunately in the pesky details, that when rolled up get lost, trivialized and soon forgotten.

effulgentsia's picture
effulgentsia -

Ray, it sounds like you had a webiste you were happy with in Drupal 6, but for some reason, felt like you needed to upgrade it to Drupal 7, and have not been happy with that choice. Man, that sucks. To the extent that your decision to upgrade was driven by drupal.org or 3rd party blog pages that made it seem like you had to upgrade, or implied a level of contrib module readiness that wasn't reality, please share those links. One thing that I agree we should do better as a community when Drupal 8 is released is to set the correct expectations on what it means for core to be released: that core alone isn't enough for most websites, that some time is needed for contrib modules to catch up and stabilize, and that therefore, most people should wait and follow contrib progress before using the new core version for their websites. According to http://drupal.org/project/usage/drupal, there are still more websites using D6 than D7, and that's fine. It'll probably flip within the next couple months.

But I disagree with your implication that development work on the next Drupal core version should wait until some threshhold of contrib modules are mature on the current version. It takes time to make all the improvements in Drupal core that are necessary to keep up with a rapidly evolving web industry. Drupal 6.0 was released in February 2008. Work on Drupal 7 core started immediately after that, and it took 3 years to get to a 7.0 release. Back then, as now, it took 1-2 years for contrib modules to stabilize to the then latest version. For example, taking the two modules you mentioned, the first stable release of Web Links for D6 was in June 2009, and the first stable release of Gallery Assist for D6 was October 2009. Given that there was 3 years worth of work needed on Drupal 7, what would have been the benefit of waiting a year and a half to start it? Similarly, we're roughly half a year into the multi-year project that is Drupal 8. It's okay if you personally are not interested in the technical and team coordination details of that right now, but for the people working on that project, including Shannon, the author of this blog post, it is most certainly not a "complete and utter waste of time". In a few years, when 8.0 is released and contrib modules have caught up to it, you'll hopefully enjoy the benefits of the work that is happening now.

Ray's picture
Ray -

Thanks Effulgentsia. First off, OK - I hear ya. I'm not a luddite, and I understand we need to look and work ahead of the curve. So, I do need to take back my parting shot of "complete and utter waste of time". That is an unfounded remark and I hereby retract it wholly. My bad. As you know, Drupal can become an emotional thang.

Having said that, I still can't seem to reiterate enough...please be careful in how D8 is rolled out and framed. I will definitely post links to the articles I read that highly encouraged upgrading to D7. These were linked to right off the main Drupal page. More than anything - saying it is ready and putting links to those downloads first really does carry a lot of implication and weight. For those of us who do not live and breathe Drupal 24/7, we do take those articles as meaningful - again, I'm not "involved" in Drupal development, so I didn't really "know better" and assumed it was time for an upgrade. WOW was I ever wrong. And to be clear, I know it is my fault that I wrecked all of my websites. But I do have a demanding job that's NOT Drupal, a family, etc etc. I do what I can when I can. And when there is a rah rah rah siss boom bah all over the Drupal site about D7 being the next best thing to sliced bread...well....I do what i can when I can. Clearly, hindsight is 20/20.

So, whether this was the right place to sound off or not, I don't know. Seriously, I am just trying to help provide some "outsider" perspective, in particular to Shannon. I do appreciate the original article, I do get what she's trying to do and where she's trying to go. My main caution is that promoting D8 and putting resources against it may be premature for the following reasons:

1. The current "greatest thing" has a great number of issues that really do need attending to. Many are months overdue from a "seriously? this was reported in May" standpoint. Those have got to get fixed, mainly because...

2. D7 has managed to alienate a large number of developers. Alienate might be a bit strong, but...you really do need the 3rd party developers to get behind it. Some are, it appears many are not. You've got to get them IN to it, or D8 becomes pie in the sky. You need the many, you need the majority. You don't appear to have either. I'm hoping you win them over sooner than later, but you probably won't until you solve #1.

In summary - D8, no matter how good, won't be useful to most until the 3rd party developers do their thing. I maintain simply that if they aren't even on D7 yet, then ...well...we are all in trouble. I can only recommend some focus on the issues I've presented, and I thank everyone for their time and consideration.

Long Live Drupal.

Add Your Comment

Case studies

Roomify — Powerful online booking. Open source comes to travel.
Italy Magazine — Italian lifestyle, travel, and culture.
Integrative Nutrition — 25,000 websites on a single platform
Sandusky Radio — 11 Radio Station Websites in 6 Months, within Budget

Subscribe