Nixys > Blog > Multi-threaded work experience, or How to be a DevOps for multiple development teams

Multi-threaded work experience, or How to be a DevOps for multiple development teams

  • Jan 12th 2022

Although Nixys mainly works in an outsourcing or project format, in this case we interacted with the customer, a large manufacturing company, for more than one year in the outstaff format

I would like to share my experience in the digitalization of a large company. Under the cover, I will talk about the peculiarities I encountered when interacting with a large number of development teams. On the basis of these, in the second part of the article we will consider what principles helped to cope with and optimize the work in such conditions.

A large production consists of many different stages, and each stage is handled by a separate department or team. In order to carry out digitalization, it is necessary to create a special application for each individual stage with unique internal logic.

So, the client company had multiple development teams and a separate team of DevOps engineers working on deploying the applications to a common cluster. In the context of this article, the term “project” can be equated with the term “application”. Each of the engineers, including myself, was assigned a different list of projects. Hence the need to interact with a large number of teams. At one time, the number of projects for which I was responsible exceeded 20, but in the normal mode, I had to interact with an average of 10 teams. Often this work is intensive and very interesting due to the large number of communications with different people.

There are many of them and they are all so different.
A great number is variety, a great variety is chaos.

“What we call chaos is just patterns we haven’t recognized. What we call random is just patterns we can’t decipher.”

― Chuck Palahniuk

It was the many inevitable differences between the development teams that created the most confusion at the beginning of our cooperation. I will talk about these “differences between” below.

Team membership

  • People in teams may be totally different. Some are specialists in development, others are development managers who know a lot about the production process, but have little understanding of the development process they lead.
  • The same situation with developers. They can be experienced or not understanding some basics of DevOps team.

Role distribution

  • Some teams may have a clearly defined role for the lead. Then all decisions are coordinated with him, and there is no confusion; in others, one of the developers takes responsibility, or the process is managed by the PM. In one team you get used to some rules of interaction, while in others things are completely different. It’s hard to predict before you come across a specific situation. You just have to be more flexible and look for your own approaches in each case.

Work coherence, interaction

  • If the work on a project is coordinated and is actively progressing, and the PM has questions about why “nothing is still working”, then the problem is in communication. Sometimes you learn about this from a complaint of management.
  • One of my favorite (now) memories is when in the early days of work I got a call from a stranger. He complained in raised tones that they were “always breaking down” and demanded to know why things were so bad. Then I found out later that this person was the head of the project.
  • But in contrast to this case, there are always teams with full understanding, each sees their own tasks and the progress of the process. It is very pleasant to work in such an environment.

Different pace and activity on projects.

  • Some teams need to meet a specific deadline, so they use weekends as a working time (hello, Sunday deplays!). Others, on the contrary, are willing to hold off checking for a few days to properly dedicate time to it, having completed scheduled tasks beforehand.

The number of messengers and chats.

  • Their number is unbelievable. Some projects close, others open, developers write in private messages, chats pile up. Often you spend a couple of minutes flipping through a person’s correspondence to find out who’s writing about what project. If you need to write to someone, there is a risk of spending a lot of time trying to remember which messenger you communicated with him or her in.
  • The key point is to find an approach and communicate effectively with each team. And to do this, you have to isolate and run through all of these parameters. Gradually you start to keep all these points in mind. Sometimes using all the subtleties greatly speeds up the task.
So how to cope with working in this mode?
Of course, at first (as on any new project) it was hard.

For example, the usual situation is a pause in communication with a person. During this time there are several other conversations going on. At the moment of returning to the first chat, you no longer understand what is happening, who is writing it, and what you’re doing here in general.

The principles that I’ll list below helped me to optimize my work in this mode. They did not appear all at once. Each at one time helped to improve the work process. Sometimes it was resolving conflict situations, sometimes it was speeding up the understanding of what was going on, improving the quality of communication, or simply organizing my own peace of mind.

Write everything you do.

  • Since communication with developers takes place in dialogs, it’s easy to lose track of exactly what you were doing on the current day in the enormous flow. Especially at first. Such details should be written down. This will help you with future tasks:
  • Describe in detail the workload at the end of the day;
  • Not lose a single task, even if it stretches over several days or weeks (weeks);
  • Create a detailed history of the project.

Keep a task history of each project.

  • A continuation of the previous point. Keeping the statuses of 10 projects in memory is unrealistic (in the first few months). Why is it useful?
  • Describe in real time the status of the project.
  • Share tasks by projects inside the team.
  • Since there are several engineers working on a project, there have been situations of uneven workload when colleagues are bored and I have an overflowing backlog ( or the opposite). At that point, it’s easy for them to plug in to help me if there’s a detailed description of the project history.

Use our own task system.

  • I haven’t thought about it much, since there was no such need, the communication with the teams mostly took place outside our task tracker, in different chat rooms. But after my other colleagues from Nixys joined the projects of that company, the situation changed. The board in trello was suitable for our purposes. It helped to conveniently organize the two items described above. At the very beginning of a project, a list was created with its name and description at the start. When I received a task from any project, I created a card in my personal list, and after solving it, I moved it to the end of the list of the specific project, forming a history. Besides, it was convenient to collect links to constantly updated important resources in the general workspace.
  • This item, coupled with the previous ones, reduced the 4-5 hours per week spent on organizational stuff to 1 hour.

Always check the inputs.

  • A point that I and colleagues have crashed about many times. If a “change this value to this value” task comes in, the most incorrect decision is to go and do exactly what was asked. Because of varying degrees of immersion, familiarity with technology, and simply inattention, very often incorrect data is provided in such requests. Russian characters in Latin text, misspelled host, missing digits in the IP address – the list can be endless. To speed up the process, before you do the task itself, it is worth to add verification that the data you were given is correct. Otherwise, you might spend a lot of time searching for the error in completely different places.
  • This rule helped me save at least 2 hours per week on searching for self-generated errors.

Insist on simple and clear solutions.

  • At times when the task comes to implement some atypical solution, it is important to pay attention to its detailed justification, to ask clarifying questions. As described above, sometimes developers have incomplete information, especially about the CI/CD process. Often there is a simpler and more competent solution, it is important to just offer and justify it.
  • And the main thing about this point is to insist. Even if you understand very well some complicated way of doing a project, it will be hard for your colleague to grasp its specifics (remember about backlog transfer), and it will be hard for you yourself to grasp it again after a few months. So if there are more common alternatives – it is better to use them to make life easier for yourself and others in the future to support the project.

Designate a deadline in time.

  • Above I described the emergence of situations with too In this regard, a simple task (15-30 minutes) can take much more time. And in this situation, it is better to designate in advance, in what time approximately it will be solved. Because the second time PM in anger may go straight to the heads of the management to complain about ignoring their most important project! This way we immediately give an understanding of the situation to the authors of the task and reduce the possibility of conflicts in the future.

Describe important things in the general chat.

  • Even if the decision was discussed during the call with the developer, a summary should be written about the decision. It helps a lot to keep colleagues (and especially PM) informed of what’s going on, even if they’re not directly involved in the decision. Also, if the decision affects someone else, they can voice their opinion, ask questions, and make adjustments before it is implemented and the next problem occurs.

Attend daily meetings of major projects.

  • Active and large projects should go to dailies. This greatly improves the understanding of project status and task prioritization by all involved. The criteria for the need for this are recurring situations with a lack of understanding with the project team, as well as the presence of many parallel tasks from different authors.

Conclusion

All the principles described above were gradually formed by encounters with real situations, and I had time to fully experience their effectiveness. Some of them I managed to deduce on my own, others I received tips and recommendations from my senior colleagues, and I thank them for that.

My work experience in this mode turned out to be very interesting and intense. Above all, it is about interaction and developing communication skills. If I were asked whether I would recommend experiencing something similar, unequivocally yes. And if you do happen to experience it one day, I hope the tips described here will help you not to rush headlong into the situation, but to approach it more prepared and aware of what to expect and how to act.

Now that time has passed, I realize that I have participated in digitalization, a huge, resource-intensive process, and have contributed to the convenience and efficiency of the production work environment. Like huge buildings, big processes are also inspiring and enchanting.