Why I did my first Engineer-ication, and why I think you should do it too
The benefits of observing first hands your codebase, processes and team interactions even when you're a senior leader in a tech organisation.
Between July 17th and July 21st 2023 I took my first “Engineer-ication” with one of our teams here at Nansen, and it's been a great investment of my time.
But wait, what the hell is an “Engineer-ication”?
This is a concept I discovered when listening to an episode of the Larry's podcast where hew was interviewing David Singleton, Stripe's CTO. I don't know for sure whether the concept of engineer-ication has been invented at Stripe, but that's where I learnt it from.
The concept in itself is very simple and applies to hands-off engineering managers and engineering leaders. During a few days you block your calendar and agenda rejecting meeting invites and delegating certain tasks to other people, as if you were to go on vacations.
Instead of spending that time resting and having fun in an exotic location, you join one of your engineering team as Individual Contributor, and work on one or more software engineering tasks.
Why would you want to do that?
The main purpose is to get a higher definition / higher fidelity understanding of the software engineering processes in your organisation, what are the key friction points that every engineer goes through every day as well as getting a more realistic and less abstract understanding of the underlying platform and architecture.
I must admit that as soon as I heard the concept I had one of those “why didn't I think about it myself?” moments, and quickly decided I'll try out this approach.
What follows is a summary of that first experience, and some suggestions for other people that might want to try this out.
Choosing a time and a place
Once I've aligned with my CEO that this was a good investment of my time - he agreed without discussion, the first decision to make was around timing and teams.
They both turned out to be quite easy.
From a timing perspective, I wanted to pick a period where chances would be high that things would be on the quieter side. As I was planning to take my real holidays towards the end of July, I picked the 3rd week of that month for my engineer-ication, knowing many other people will already be on holidays.
Choosing the team required me to consider multiple variable, starting from my own background (Backend / Infra), the bandwidth and availability of more senior engineers that could help me in each team, as well as any impending deadlines that I surely didn't want to put at risk with my silly questions.
Taking all of that into consideration, and after a few chats with people across the org, we agreed I'd join our “Integrations team” for a week. This team focuses on integrating with Blockchains and other Web3 projects to bring their data into our raw datasets, on top of which all of our analytics is build. This teams deals with high volumes of data, pipelines and quite a lot of automation. It's also a team that is quite mature, going through an important technological migration from BigQuery to ClickHouse as our main data storage solution, and is planning some non trivial architectural evolutions as a consequence of that.
Setting goals for the week
I had two set of goals for the week: one set by myself, and one set by the acting manager for the team (the official EM was on vacations).
From my side, I wanted to come out of the experience achieving two objectives:
Learn more about our codebase, standards and practices. There is nothing that beats direct experience when you need to really understand something. Without that you can just have an accurate ‘overall idea’ about how things are in reality
Identify what are potential issues and bottlenecks in our developer experience. By going through the motions myself, I'll be able to really feel both the joy and the pain of developing in our current environment.
Those were my main motivators to engage in this whole experience.
At the same time, I wanted to do something meaningful and useful for the team, as such I simply asked the manager to assign me tasks. I was very clear in my intention to not request a special treatment due to my job title, and as such I didn't want to express any preference as not to influence the task allocation beyond what the manager deemed both useful for the team and achievable for someone that has been away from professional software development for quite a few years.
The manager decided to put trust in me (positive, as I expect us to do the same with every new joined) and assigned me the responsibility for migrating from BigQuery to ClickHouse the data for the different type of entities we track for one specific blockchain. And to be clear, I had absolutely zero prior exposure to this specific chain.
These tasks would require me to familiarise myself with both our bach and streaming infrastructure for data synchronisation.
I was both exited and a bit anxious.
Will I be able to deliver on those expectations after all these years, or will I just embarrass myself?
The onboarding experience
The onboarding experience was very smooth. Besides a couple of issues with permissions that came up a couple of days into the week, I was able to get up to speed coding from the first day. Three actions from the team greatly helped with making me effective quickly:
The acting manager prepared a document for my onboarding, and we went through it together on the Friday prior to me joining the team. This gave me an overview of the repos I needed to work on, their relative workflows and an opportunity to ask some basic questions. He told me I will have a buddy for the entire week, someone that was very familiar with the systems I'll be interacting with.
On Day 1, my buddy and I had a video call where he literally hand held me step by step through my first simple Pull Request (PR) to add support for synchronising an entity via the batch system. The code change was very simple, but having someone guide you to the flow of how to make a PR in a new software you've never looked at before was invaluable. It gave me a great sense of confidence for the work ahead.
On Day 3, I had another session with the acting manager, where he gave me a deeper overview of the streaming architecture and setup, the repo and the workflow.
Besides that, I was encourage to ask any question in the team's slack channel, and I received great and constructive answers to each one of them, no matter how simple some of them turned out to be.
I do hope every single person that joins Nansen will have such an encouraging and effective onboarding experience.
Dealing with Timezones and Context Switches
All the team members that could answer my questions and review my PRs were on the Asian timezone, while I'm in CEST. This situation is not uncommon in fully distributed teams, and in my experience it caused both positive and negative side effects.
During the week I found myself working on code changes through the day, asking questions as early as possible in the process - sometimes literally batching questions in the early hours of the day - and then spend the time in the afternoon typically finalising the work on a PR, then pushing it for review before the end of the day.
The experience of waking up the day after and find comments and suggestions on the PR was very positive, as it allowed me to jump straight into addressing any issue with virtually zero delay, as things happened while I was off duty.
The flip side of this situation is quite obvious: you can't always control / plan for when you'll be blocked either because you need and answer to a question or because you need someone to review your PR. When that happened and I found myself blocked on the coding side, I switched my attention to my “previous life” activities: following up on slack threads, emails, working on documents and other deliverables.
When this happens occasionally it is manageable, and a price worth paying for the flexibility we all benefit from by working remotely. When this happens too often though, it can lead to very frequent context switches that will undermine productivity.
As a manager I'm used to frequent context switches. Jumping from one topic to another is quite common and actually required in such a role.
When dealing with complex work such as writing software, the cost of context switching is significant. Loading in your mental cache information about a system and the changes you are trying to introduce can take multiple minutes, in the double digit.
Another compounding factor to this I observed is when I know there is a meeting coming up, and I only have 30 minutes to one hour to get stuff done. I've observed my level of attention to be more disrupted in these cases as I would constantly check the time to avoid being late. This led me to be less focused on the work at hand, and therefore I ended up making more stupid mistakes, ultimately wasting time.
The fun fact is that we've known this for decades.
Observing this phenomenon again after years AFK, it reminded me of an old but good article from Paul Graham. He wrote it in 2009, but what he describes is still largely the same today.
Based on my short observations during this week, what I found being the most productive schedule for the day was basically the following:
Morning
If any PR had received comments while I was away, I'd focus on addressing them and pushing changes to production as the first thing. This would allow me to completely forget about that open task one the PR was closed.
If no PR was open or once the open PR was handled, I'd pick up the next task. Instead of jumping into coding I'd familiarise myself with the repo, make sure I had all the required access to deploy (when needed) and looked at the code to figure out what I should do. That process would typically trigger questions that I'd promptly ask in the slack channel, knowing people would be around to answer them quickly
Afternoon
With all the information collected in the morning, I'd then sped as much time as needed writing and testing the actual code until eventually I'd be able to open a PR.
Once the PR was pushed for review and I didn't have any coding related pending task waiting for me, I would then switch my mental context to all the other stuff from my “previous life” that required my attention: slack, emails, notion documents, etc
What I found effective with this approach is that it limited the amount of major context switches to 2:
When starting my day I would switch my attention to the PR / coding task waiting for me.
Once the PR was ready for review, I would switch my attention to non coding related tasks
I found this approach to be the most productive and the least mentally taxing, and I wonder how other people working in heavily distributed teams are optimising their schedule for productivity and mental energy.
I'd love to hear your input on this in the comment section!
The peace of mind of automated releases
This is definitely one of my favourite piece of the whole experience: working with repositories that have a robust and simple to use continuous deployment setup. This was the case for the batch synchronisation project.
There is plenty of literature about the benefits of CI/CD for productivity and code quality, but as a “new joiner” I found the well documented benefits to be compounded by a few other observations:
Having Continuous Delivery (CD) in place makes it easy for everyone to ship to production, which contributes to both a sense of safety, as a machine is doing it for you, and a sense of fulfilment, as you can push to production on your first day at work.
It reduced the amount of cognitive load on me, as I didn't need to learn about all the exact details of how things were set up in the different environments to be able to ship, allowing me to focus my attention to the actual business logic, and less on the infra side. Of course there is benefit of eventually learn the inner details of how everything work, I just believe it doesn't make sense to start from there. This part reminded me of one current of education called top down teaching model that is being more and more used in software engineering related topics, such as the Fast.ai.
It eliminate a lot of sources for potential errors or mistakes. I didn't have to think about whether or not I was executing commands in the right order or in the right environment (ever pushed changes to prod when you only wanted to test stuff in dev instead?). I also didn't have to care whether I had the right version of tool A or tool B while shipping to production.
Overall, repos with proper CD setup greatly improve the overall onboarding experience, ultimately leading to better productivity.
On the flipside, at the time of my Engineer-ication, the streaming service was still lacking a fully automated deployment setup, which required me to fiddle with terraform commands. Nothing overly complex, but due to that when I was working on this system I had to deal with additional cognitive load, which took away mental energy and bandwidth from the main focus on adding a certain functionality.
So, did I meet my goals?
I'd say the answer is a well resounding yes.
To be completely honest, when I embarked in the initiative I was afraid I might struggle to be productive due to the lack of context, lack of expertise on data systems and the amount of years that have passed since I wrote production-grade code.
The reality though turned out to be much more encouraging. Thanks to the good onboarding, the well defined scope, the good level of documentation and the support from the team I was able to finish all the tasks that were assigned to me.
So, definitely a 👍 for the goals that I was assigned by the team.
What about my “own” goals? I can say they were also a hit.
This experience allowed me to gain a much more detailed understanding of a relatively small but highly crucial part of our architecture. It also gave me a good exposure to different levels of maturity around testing and release management. definitely a ✅
I could clearly identify areas that are suboptimal and that will require some adjustments and improvements going forward, and some of them have already been addressed since I went through the experiece.
Last but not least: it was a really fun and enjoyable experience, and it allowed me to sprinkle some green colour on my otherwise empty and boring GitHub history!
One of the biggest realisations though has been something else.
During this week I experienced again what is often lacking in a manager's life: long stretches of focused work. Beyond the well known positive effects of being in flow state regularly, I could observe clear signs on my general wellbeing.
At the end of each day I was feeling tired but not mentally exhausted. I felt I had more creative energy left than it usually happens when my days are a sequence of short interactions and frequent context switches. This is something I'd like to bake in again into my day-to-day routine as a Manager, knowing that there is an inherent requirement for me to be able to switch context that it'll be difficult to completely avoid.
It also reminded me why I got into software engineering to begin with, and it gave me a lot of motivation to get back to it on a more regular basis. I might try to do small PRs every now and then just to get to understand better some parts of our platform, and keep experiencing the power of building things just with your fingertips.
Will I do it again?
Absolutely yes! 😎
The amount of information and knowledge I got out of this week of investment - minus some hours that I had to dedicate to my previous life - is invaluable.
As a byproduct, this experience also offered an opportunity to interact more with some members of the team I don't usually interact with a lot. It helped with building new connections and cementing existing ones. I feel now somewhat part of the Integrations team forever, and I hope the team feels the same way.
In my role I need to keep connected to the reality of what it looks like to ship software on a daily basis, and this type of experience is a very good way to achieve that.
Of course my job is to lead people rather than write code, not matter how much I still love it. This means I need to find a good balance between doing this often enough to stay relevant, and not too often to the point that I'll distract myself from my main duties.
For that, I'll settle for now on taking an engineer-ication twice a year, every six months roughly. The goal is to do it with a different team every time to offer more context and more diverse input as I'm pretty sure every team has its own set of strengths and challenges.
And you, are you considering taking an Engineer-ication anytime soon? In part 2 of this article I'll share a set or recommendations on how to get the best out of it!
Stay tuned, and in the meantime I'd love to read your thoughts in the comments section!