This is a crosspost from VanessaSaurus, dinosaurs, programming, and parsnips. See the original post here.
Whether you are a graduate student thinking of your first job, or a veteran wondering about the grass on the other side, you might have asked the question:
What’s it like to be a research software engineer in a national lab vs. academia?
The problem is, of course, you can’t really compare apples and oranges until you’ve had the opportunity to taste both. So what do you do? You probably seek out people from both places and ask them a lot of questions. And then you try to compare answers. But we still have a problem - it could be the case that the people you talked to have never experienced the other side.
I’ve been working at a national lab for almost two months now, and it feels about the right time to start reflecting on the differences. I’m in a rare position of having experienced both, and I realize that this comparison could be valuable to someone that is trying to decide. I’ll start with some caveats, and then move into categories of things that I’ve found fundamentally different.
First, of course, let’s start with some caveats. My reflection today can only be based on my life experience. You could, in fact, have frequented the same academic institutions and national lab as me and still have a different experience. My reflections are also based on the specific people and teams that I’ve interacted with. It could be that the experience would be entirely different on a different team at the same institution. You can take my experience with a grain of salt if you disagree, or treat it as one data point in a larger dataset that remains to be collected. It’s also the case that I am not reflecting on being any kind of student or role on a traditional academic path (e.g., a postdoc). I have experience as a student, and can solidly state that the goals, wants, or needs of that path are fundamentally different than the path of being a software engineer. Finally, when talking about academia, since I have personal experience along with knowing the setup of other institutions, I will do my best to generalize my comments. I have had experience at several academic institutions as staff, and experienced others as a third party, and my thoughts here are a culmination of these experiences.
Let’s start off with an easy thing - titles! Your title is what your institution calls you, which in a way influences how other people view you and what you do.
At most academic institutions, the title of “Research Software Engineer” (RSE) is not a real one. When I say “real” I mean it is not a position title officially recognized by the university. It could be that you sit in a group or lab and call yourself an RSE, but your official title is something else. Although there are efforts such as the US Research Software Engineer Association working on this, it remains the case that the role of an RSE is not an established piece of the academic fabric. Because of this, we still don’t know what the qualifications, training, or credentials of an RSE look like. And because of that, if someone is aware of your self stamped title of RSE, they might question your skill as an engineer. Truly, if anyone can just self-proclaim the title, it can’t be that hard to get it, right? But let’s focus on the positive here, and also state that this point only matters if you care about how others view you. This also won’t be our reality forever. If work continues to establish these roles, there will be a day when the title draws more respect from the academic-sphere. It will be known to be important, substantial to achieve, and valued in the ecosystem. Knowing this already about RSEs that are in the community, I already have this sense of value for my colleagues, and want to see it realized.
Now let’s talk about the national labs. While some national labs have groups that hint of a research software engineer (e.g., a group that studies software engineering or titles like “Scientific Software Developer”), I was surprised in my case to learn that my lab does not have the title of RSE. Instead, the folks that design, write, and innovate in software development are called Computer Scientists. This likely goes back to (possibly historically) the tendency to name different roles as different kinds of scientists. If you study materials, you are a Materials Scientist. If you study computers, you are a Computer Scientist. That makes sense, right? This was my opinion, superficially. But once I dove into the actual work of the lab, I realized that the title was fitting.
In reality, you probably shouldn’t. Titles aren’t that important. But having an official, valued title has unexpected trickle down effects. It suggests that your institution values you, which means that you might have more job security, and attention or training for your role. Let’s discuss this learning next.
Learning can encompass two broad areas – the training and expectations that exist in the role, along with how you might pick up new knowledge while you are in it. Both are important, and the path that you take hugely depends on how you best learn. For example, if you learn best by taking courses, doing problem sets, and exams, you’ll maximize your learning by starting early and pursuring something like a Computer Science major. But not pursuring this path does not mean that the roles of RSE or Computer Scientist are forever closed to you. If you learn better by way of immersing yourself in work, although the path might be harder or longer, you can get there. It all comes down to where your curiosity is, and how you best learn.
Curiosity is the driver for many things. Combined with stubborness, it’s the motivation that has you staying up late because you “just want to try one more thing” or asking questions that don’t have answers. But notably, where your curiosity can reside can vary for different people. I ultimately realized that I was a terrible academic researcher because I was not driven and curious about science. I was skeptical of studying the human brain because it didn’t feel like I was really making a difference, or trying to answer questions that I cared about. A member of my defense committee noted to me directly that I had accomplished and built a lot, but that I wasn’t very scholarly. That felt like a dig at the time, but I also sort of agreed with him. It was just one of many small indicators that I wasn’t fit for the traditional academic path. I didn’t know that before I started graduate school, but I knew it by the end. My heart and curiosity were just elsewhere. I wanted to built systems, spend all my time programming, and learning the intimate details of technologies. I felt valuable if I was able to build anything that I could dream. Many people hinted that this other place wasn’t hugely important in the world of research, and this was reflected in the lack of RSEs at institutions. But I disagreed, because research would not be possible without the technical aspects. It was not that RSEs weren’t needed, but rather that a traditional academic ecosystem didn’t have a place for them yet. The burden of doing the research, including writing the software, was always on the shoulders of the researcher. I share this story because I want to point out that we can differ in where our curiosity lies. The motivation that you have to learn will depend on if your curiosity lines up with what you need to do for your role. It will influence how well you learn.
Before we talk about different kinds of training, expectations, and learning environments in academia and at national labs, you should think about how you best learn. Do you learn best from an academic course? Do you feel that you need to know details before starting working on something? Do you learn best when you are interacting with others, or by yourself? Do you learn more from reading and thinking, or doing? Asking yourself these questions, and forming an understanding of how you best learn will make it easier to make decisions about the right environment for your growth. In my case, I learn quickly and easily by just jumping into a new problem and trying things. It doesn’t always come quickly, but usually with exposure over time, and lots of ruminating runs and good naps things eventually click. You may not need to take a course or read a book until you’ve reached a point where you have a specific thing to look up. Someone else might need the course or book first before trying anything. If you don’t need to learn things befor doing, you’ll find it easy to be in environments where learning or finding direction is your responsibility. You can do well without a lot of structure because you create it for yourself. You can find yourself presented with an abstract problem and break it into concrete things to try. Personally speaking, for these reasons I did very well in academia. I also think this kind of learning strategy will translate well to a national lab. Your learning style doesn’t have to match others, or even be common, to be successful.
It can be hard to introspect about how you learn, and you might even be unaware that other people learn differently from you. But I bet if you look around, there are people that listen to a lot of podcasts, and can remember things easily through sound. There are likely others that read a manual from front to back. YouTube, or more generally instructional videos or courses, are really helpful for others. There are some like myself that need to engage with a task directly for things to click. The fact that there are so many different kinds of learning styles makes training quite a challenge. But both academic institutions and national labs make their best effort, and also have expectations for you. Let’s talk about that next.
For both academia and national labs, we can broadly state that you are expected to perform the duties of your job. But the expectations for your experience and training can be different.
In academia I could become an RSE mostly with experience as a researcher, and nobody checked if I had any experience as a software engineer. The expectation was that I had come from some kind of a research background, and having a PhD was evidence of that. The expectation for becoming an RSE was loosely that I had some technical background, experience with writing software in open source or in labs, and could independently solve problems that would help the user base of the university. You can actually view the job role that we put together for my position. It’s very oriented toward supporting researchers with software, which was my goal at the time. For my first staff RSE role after college and before I had a PhD, I didn’t have this background, and was called a “Research Technician II.” I didn’t know much about programming, HPC, or anything beyond Psychology (my undergraduate major) at that point. Although I wound up teaching myself to program and writing a lot of code, I don’t think it was a hard requirement for the position, which also had a lot of scheduling, running behavioral or imaging protocols, and working with scientific software graphical user interfaces. For each of these roles, it was surprising (but expected) to not have any whiteboard type interviews. As I mentioned earlier, not knowing what constitutes being worthy of the title of RSE is likely because we don’t really know. We are sort of a hodge podge mix of skills in research and programming. This also means that there isn’t training out there for the role. In my academic roles, I noticed that the IT branded training tended to be for using word processing or data entry applications, or (later in my career) simple courses for what was trendy at the time, things like machine learning or introduction to Python. There weren’t guides about software best practices, so I tried to put together my own. The only required training was expected things like HIPAA, which I took once every few years. This kind of environment has huge benefits for someone like myself because it meant that I could transition. I didn’t have the “right” background for being a software engineer, but I realized my love for programming in my first research-oriented role, and was on a slow path to get closer to it. Being able to get the title of RSE with a combined researcher and informatics background without anyone asking questions was a blessing in disguise. I could then, by way of a combination of work and personal projects, prove myself. I learned new concepts and standards, along with programming languages. I engaged heavily with open source projects to further my growth. I built entire production infrastructures and services on my own. By the end of almost 5 years, I most definitely earned that title, even if it was just a feeling and not a concrete list of attributes I had gained. So this is something to keep in mind. If you love programming, or love contributing to research but don’t want to pursue a traditional academic path, an RSE role at an academic institution might be a good fit for you. You can grow as an engineer and then pursue a more engineering heavy role, or stay happily right where you are.
I would suspect that the expectation of an RSE (whatever the title is) at a national lab is some combination of education (e.g., PhD) and real world experience. I think if you can prove yourself to be a hard working person that has potential to contribute, you can be a good candidate. The barriers to entry aren’t as stressful as what you’d imagine for an industry software engineering role (white board interviews), but you are still assessed on your knowledge and experience. In terms of training, there is quite a lot. The large majority of new hire training is what you would expect – catching up on protocol and security best practices. But (I’ve heard) that trainings for technical things come around every once in a while. For example, a C++ course is something that I’d dive into, and is expected to be around at some point. While the credentials for a “Computer Scientist” aren’t written down, I get the sense that others in my role have also proven themselves to get here. Whether they started as interns and proved their worth to work up to the title, or were impressive coming from another academic institution or industry, they had to earn it. There are quite a lot of PhDs, and people with actual computer science backgrounds.
The interview process is another difference to think about with respect to national labs and a different group that we aren’t really talking about – industry. I didn’t have any whiteboard interviews (which is a huge benefit because I get very anxious in that environment) but rather had many group interviews and gave a talk to assess my ability to communicate with others, and probably my potential to contribute. It was positive experience, and a breath of fresh air to actually be assessed for what I’d done, and not how I’d perform in a 60 minute block of time on a superficial task. But I am still very aware that I don’t have a traditional computer science background. I did take introduction courses early in college, but writing for loops on an old school computer in a basement lab never made sense to me. It never clicked why it was useful, and I didn’t pursue it further. That means that later, it was much harder to catch up, or at least it was on my shoulders to catch up. I still am acutely aware of the sheer amount of things that I don’t know, but maybe that’s the case for everyone, and it’s okay. If you are still a student, this is something to consider as you choose a major. I’ll probably spend the next few years proving to myself that I have earned my new title.
With respect to training and expectations, I would say that both academia and national labs are going to give you flexibility. You can come with a traditional background and fit right in to either place, but you can also come with a non-traditional background and have the flexibility to learn on your own. In either case, you have to prove yourself in some respect, whether it be what you’ve already done, or the education that you’ve attained. Getting a graduate degree (a PhD) is a really useful credential for both these paths, so it’s something I’d consider if you are interested but undecided about a path. I also don’t think you should worry about not having the right background, because people learn very differently. But do know that not having the right background can make it harder for you. I would generally say that if you are a passionate, and motivated person, you can be successful, eventually.
Have you ever stopped to wonder if different institutions have different mindsets? A minset for an individual can broadly be described as how you think about about yourself and what you are capable of (e.g., “I am capable of learning new things”), and for an institution, can broadly be described as a culmination of the organization’s beliefs and priorities. The distinction between the two is that we have control over our own mindsets, and less or no control over the mindset of an institution. Looking closely at academia and national labs, while my mindset is fairly consistent, I’ve found very different organizational mindsets. Let’s break these down.
As an academic research software engineer, although you might be doing a hefty amount of development, at the end of the day you are providing a service. Whether you are exposed to the financial bits of needing to justify the expense of hiring RSEs or not, your entire existence is to build software for labs. This may look like having a well established RSE group with a manager, and then getting assigned to projects, or it might look like working independently and having to recover your funding. It might also be that you are an RSE (even with a different title) that sits within a lab and works on software for the lab. There are also groups that provide resources as services (e.g., clusters or cloud accounts) or consulting for software. For all of these cases, if you step back to think about it, there is an unstated hierarchy. The science, and the scientists, are the top priority. Another priority is teaching. And given that you’re at an academic institution that is a research institution, this is probably how it should be. The only exception is for departments like Computer Science, where you can in fact do research around more generalist topics, and as long as you find funding, you can get by. However, this means that you really aren’t a software engineer, you are a Principle Investigator that runs a lab, and you are on the side of the researchers.
I can reflect on my own experience. When I was a graduate student, I was so grateful that there were IT service and Research Computing groups that I could email for help and get a quick response. I never really stopped to think what their life experience was like. I never stopped to think that they were there helping people, maintaining all of the services that I relied on, 24/7. But that means that there is a strong service mindset, with researchers as customers. The work of the researchers, and even the students, is really important. As a staff member that works on software, your entire purpose is to support that work. Thus the academic mindset, probably as it should, places the scientist as the priority, and software engineers as support staff. This is neither good nor bad, but something that will have different implications depending on your goals as an engineer.
What does this academic mindset trickle down to, in practice when it comes to learning? It means that as staff at an academic institution, it’s more likely that your personal growth and development are not a priority. If you do get projects that challenge and further these areas you are lucky. From experience, I’d say that you learn a lot early on, and then subsequent work tends to all start looking the same. In an academic environment, as a staff RSE that supports scientists, your learning is entirely dependent on what they ask or need you to do. It’s unlikely they will want to switch to a drastically new kind of technology or language, or ask you to allocate time to work on fundamental, generalist research software. The reason is because they are following what their grants are funding, which probably is related to a domain science. If you might be interested in getting your own funding for technical pursuits, it could be that your group, as a technical group, is not allowed to apply for their own grants. If you are in this boat and you want to grow, you will need to take responsibility for your own learning. This might mean engaging more with different kinds of open source projects, figuring out ways to collaborate with other groups that might offer a new kind of task, or just doing your own stuff on the weekends. Personally speaking, I did this for a few years in my staff role, and was very happy. But that happiness was contingent upon me realizing that it was up to me to find what I needed, and push for what I thought was needed but not realized.
Now let’s switch over to the different mindset that I’ve found at a national lab. To understand this mindset, let’s first dabble in the question of “What is research software?” Indeed I’ve thought about this a lot. I’ve always considered myself a “generalist” RSE, meaning that I get excited about developing base or core technologies for research. In practice this means container technologies, standards, APIs, or (more recently) package managers and solvers. Although this has been my definition of research software, what I’ve come to realize (especially if you look at what gets published in the Journal of Open Source Software) is that a lot of people consider research software to be very domain oriented. It’s the software that comes out of the labs that, at best, might have support of academic RSEs to make it reproducible and sustainable. Software that quickly gets labeled as “research software” is generally not something like a general database or container technology, but something that directly answers a scientific question. Or perhaps it preprocesses data specific to a particular domain. The more you stray from that, the more that someone will hesitate to call it research software. And it follows that research software engineers are the ones working on this research software, which is directly related to the goals of the research lab the software is for.
I bring this up, because what we might consider research software engineering at a national lab is an entirely different animal. This isn’t to say that there aren’t very scientific domain oriented research groups at labs (there are!) but the cluster of work I’ve found myself immersed in is a dream come true for someone with generalist interests like myself. I’m not just writing software, I’m doing research around software development. Although we have an applied tool that we want to work on and this will trickle down to users, we are innovating to get there. We aren’t just writing for loops and if statements to match a spec, or following instructions to implement something for a scientist, we are rethinking an entire model for how a task has been done in the past. We are dreaming up ideas that don’t exist, and taking risk to try new things.
For these reasons, what happens at a national lab, at least in my specific role, and in my short experience, has what I’d call a software innovation mindset. I never experienced this mindset at an academic institution unless I was collaborating on an open source project unrelated to an academic group directly. When I made software for labs, the general goal was more to “make it work!” and there wasn’t a team of people thinking hard about innovation of the software itself. Mind you, there is indeeed a mindset of innovation for the scientific research at academic institutions, but less so the software. And perhaps my experience would have been different if I were an RSE rooted in a computer science department. But I was not. Ironically, this kind of innovative software work is what I always considered to be research software engineering before I discovered other RSEs in the US-RSE. In my first taste of the role, when I wasn’t aware of any other RSEs, I figured out ways to rationalize having my focus be on container technology. In all fairness, it was easy to justify because the technology was so badly needed by the academic community. The other work that I did in support of labs I always considered separate. It wasn’t super challenging, but it was necessary to keep me funded. I would eventually learn that it would feel empty to not have challenging projects alongside this work. So although I was disappointed to realize that there weren’t many other RSEs with a generalist technology view like myself in academia, I was joyous to realize that there were at a national lab.
We talked about learning in an academic staff RSE role, now what about at a national lab? I previously described an unspoken hierarchy where the staff software engineers are there in support of the scientists, and I haven’t felt this same hierarchy at the lab. Although there indeed are scientists at national labs and different groups to support services and software, I’ve found that I don’t feel like my work is secondary to someone else’s. I might be doing work that will trickle down to support scientists and further their work, but my work is equally important. I am not just a service agent performing a function. I am developing and innovating research software. This naturally translates to a whole other ballpark for learning. Since my work is, by default, something that is not known, the default state is having challenge. If we already knew how to do it, we would not be working on it. But also, having complex problems requires a lot of expertise in many areas, which means it’s highly unlikely you will know everything when you start a new project. As an example, for a project I’m currently working on, I’m working with a complete lack of knowledge about three technologies that I’ve never touched before, including:
And all of these three things have to come together, in some respect for me to do my work. You might think this is terrifying, but it’s not. Firstly, if you have a growth mindset, you wouldn’t even notice, because you accept that there are many things in the world that you don’t know, and are eager to find new challenges to increase your scope of skill and knowledge just one tiny bit. I didn’t actually realize how little I knew about these core topics until someone pointed it out to me. As long as you don’t throw your hands in the air and give up, there really is no failure state. Rather there is learning a little bit more, trying something else, or taking a different direction. Secondly, you have a team. It’s probably the case that everyone on your team is so smart you wonder how you are present in the same meeting with them. The point is that challenge and thus learning are the default state. Your job as an RSE at a national lab is to be a good learner, meaning that although you don’t know everything, you have strategies for learning. You understand how you learn. I could go into an entirely different post on this idea, but learning how to learn is an effort in and of itself. It’s (at least what I think) what the real goal of a college eduction is. All in all, my learning and growth feels important. It’s not an extra bonus of some function I am performing as a service, but the main focus. And because of an overwheming support structure, I don’t feel afraid to try new things.
You can probably get the sense from the above descriptions that although organizational mindsets are separate from how we feel about ourselves, they are intimately related. If I find myself in an environment that supports my learning and growth, it is easier for me to feel comfortable taking risks, and not be perfect. A positive energy and growth mindset can also rub off on an organization, especially if you are in a role that has influence. The kind of organizational mindset that you surround yourself with, whether it be service and support focused or more challenge focused, is again up to you. There is no better mindset, but rather one that works with your goals as a person and engineer. As an interesting aside, the mindset of an institution actually has implications for what it means to be a research software engineer, discussed next.
Although I can’t rebrand the term “research software engineering,” I’m biased now that what happens at a national lab matches what I think is more true to the term, if we read the words, verbatim:
it says nothing about needing to be for a specific domain of science, and everything about innovating in software engineering. With the above, research software engineering is not performing a service directly, but doing research around software, which of course requires expertise as an engineer. Do you see the difference? In academia, an RSE is not in charge of doing any research or trying to innovate, at least directly. They are following the needs of the scientists and creating software for the research. Let’s describe this other definition of an RSE in academia:
The above says, logically, that research software engineering is software engineering explicitly for supporting research. This is what I’ve seen to be the case in academia. We are creating software to support research, and we aren’t doing research around creating software. It’s subtle, but it’s different. Is there a right one? I don’t think so, or rather I think it’s up to you. Whichever you choose to embody for your role, that is the right answer for you. If you are undecided and trying to decide between the two, an easy question to ask would be:
Am I more fulfilled helping people, or innovating software development?
Arguably, you get a bit of both in both areas. But I’d say that academia will have you more directly, regularly helping people as your primary role, and national labs will give you much more opportunity to be an innovator. I don’t mean to say that this experience is universal, and I don’t mean to tie one or the other to a specific place. It’s likely the case that you could be an RSE that focuses on user support in academia or at a national lab, and you could be an RSE that is innovating software at either. The above categorizations reflect my personal experience.
I would summarize communication as including how people are organized and thus able to talk to one another, how easy it is to ask for help, and then meetings and interactions with managers, who can really make or break your experience.
At an academic institution, communication is important because you typically have one set of people that have needs (researchers), and another than aims to meet them (staff). But if you are in the staff role, unless you are sitting within an academic lab, it can be very hard to figure out the needs across research labs at the university. The degree to which communication happens probably comes down to how groups are organized, and by what means they use to talk. For example, at an academic institution your units come by way of labs and service groups. Within the silo of a lab, or a service group, there is a lot of communication. But spanning those two silos is a challenge in and of itself. Although these two entites might talk by way of support channels (e.g., a forum, slack, or email) and then the occasional event where both parties are present, the degree to which the researchers truly communicate their needs to the engineers that might help is limited. It could be that the best signal is a survey at the end of the year, or a meeting between higher ups in both groups that then relay the information to the separate teams. This is probably why the model of an RSE being embedded in a lab works so well. The communication is there!
In my personal experience, coming to my staff role after being a graduate student I had a very clear vision of needs (containers! workflows! tools for reproducible research!) but the longer I was in my role, the more this vision dimmed until I felt fairly disconnected. I could not have anticipated this. I had a vision for a group of RSEs that might sit across a university, acting as a layer to connect labs and afford communication, but it was beyond me to realize it, especially as a remote employee. But I did try anyway. I still think this future could be a reality, but I wasn’t the one to do it.
Now let’s consider the structure of a national lab. Although I can’t do justice to describe this organization, I can say that it’s often described as a grid – where you have divisions overlapping with projects. You get to work with people with different expertise by way of joining new projects. For discussion channels, you have a mixture of everyone, and they aren’t scoped to specific groups that have private discussions. We have a daily morning meeting in a channel, where you feel connected to even groups that are very unlike your own that are giving updates. It’s okay if you are still in your pajamas, there is no video. And you don’t feel like you are left out of any specific channel. This is strange to describe, but it contrasts my academic experience, and even my experience in open source communities, because I’ve felt like I didn’t belong in certain channels. Either my role wasn’t quite right, or there were people that would dominate the channels with posting politics or similar. What do I do in situations of discomfort? I talk less. I stop looking. If I try expressing myself and feel like I don’t belong, I’m naturally going to stop doing that. This is how the physical and digital structure can influence communication. It either encourages you to speak more, or speak less. Thus, I’ve found the organization at a national lab to be flatter in the sense that I can freely communicate with and learn from others without worrying about my role, and more integrated because I don’t really notice when different groups are interacting. The borders exist, (and there are way too many acronyms for teams and groups that I’ll never know them all), but they feel fuzzy.
Organization has implications beyond how people communicate. An institution with well-established paths for working together across groups means that both sides can easily communicate about needs, and then they can be addressed. This is another trickle down effect that makes it easier to be needed and thus valued. Having a well-established organization with paths to funding and working together means that you likely have more job security. I haven’t had one night in my new role staying up and worrying about where my funding is coming from next. I can look around me and see so many people working together, and so many projects to collaborate on, that I have no worries about my current project finishing up. We have a long way to go, but when that time comes, I’ll be able to add value elsewhere. Communication, organization, and your well-being are intimately tied. This leads us to our next topic: asking for help.
It’s really important to not be afraid to ask for help. And the degree to which you are comfortable doing so really depends on the culture of your team.
As a graduate student I wasn’t afraid to ask for help. Whether it was from an administrative person in my department, my advisor, or a labmate, the understanding of being a student is that the structure of the institution is there (to some degree) for your learning. But as a staff in some of my academic roles (not all of them) I was afraid to ask. I was afraid of the deafening silence of a Slack channel where my question would be overridden with later links to the latest news or other kind of social media-esque post. This wasn’t always the case, but it felt more common than I would have liked. I was afraid of being judged because other people didn’t seem to be excited about what I was excited about. I questioned myself for having my energy and aspirations, because I felt out of place. This of course, totally depends on your institution or teams, but generally I’d imagine if you are a lone wolf research software engineer, either in a service group or a lab, you feel like a bit of an outsider. As a result, communication suffers. Over time you become less likely to want to ask for help. You feel more of a burden to figure things out on your own. And this indeed was the case – I even had holiday weekend days where I was the only engineer responding to a production issue, hoping that texts or emails would at least reach others that might offer advice or support. In retrospet, it helped me to grow and mature. In the moment, it felt terrible.
This lone wolf environment can really help you thrive if you are introverted and independent. Some people just learn better on their own. Some people are more hybrid and can adapt to either case. It surprisingly worked very well for me, because of that. I learned that I could reach out to open source communities with questions, or even something as broad as Twitter, because most of my questions were related to some open source technology or similar. But I still always wondered if there was a place, somewhere, for me.
Now let’s switch over to a national lab, and I’ll start with a story. In one of my early days as a national lab employee, I got very anxious about something administrative that I had to do. It was late at night, and I posted to a community channel. I was expecting to have to manage my anxiety until the next work day. To my surprise, not only did multiple people respond within minutes, but they escalated the issue to higher ups in the lab that could make decisions. Within the hour I received emails that they had discussion, and by the next day there was a solution. I was seriously amazed, because there was this overwhelming sense of “We are here for you, and we value your well-being, not just functionally but emotionally.” On my run the next day I ruminated on the previous evening a lot. I was overwhelmed with a sense of wanting to give back. My colleagues really cared about me. I could openly communicate with them and they listened, responded, and had my back. And after these two months, because of tiny interactions like this, I care deeply for my colleagues. This has never happened before, a work colleague has always just felt like a superficial acquaintance. But now, I also care about their well-being. I want to have their back. In a way, it feels much more like being part of a real community. I now understand why I’ve seen others express on Twitter that they are sad to leave their teams. The community is really there. It’s funny how this all came down to communication – feeling like I’m heard, and valued.
For explicitly asking for help, the funny thing is that sometimes, I haven’t needed to. If there is even a sense of having something to discuss, or perhaps we finished a meeting and I’m tasked to work with something and my colleage wants to help, it’s very natural for to say “Let’s jump on a call.” This is scoped not just to technical things, but for help with navigating different administrative tasks. Even in my first week, I had a colleague that wanted to talk on the phone to tell me all about a file format. It was pretty awesome and surprising. And I’ve also asked for help explicitly. There have been several instances of posting questions on communication channels, and having them lead to discussion, sharing of links, and ideas. It’s made me realize that because people are so eager to help, I should be absolutely sure that I need it first. I’ve asked questions several times to quickly figure out the answer a few minutes later. That’s when it’s easy to feel silly for asking too soon. Overall, when people are filled with knowledge and want to help you, it’s a great environment for growth.
I’ll remind you of the caveat at the beginning of this post. Your mileage may vary. I suspect there are many academic groups with staff engineers that really look out for one another. But in a lot of cases, because of the ephemeral nature of the role, you probably will feel like you are in survival mode, and be mostly looking out for yourself. Communication then suffers. I do think (and hope) that this will change.
Meetings, meetings, meetings! I never imagined that regular communication and meetings could be a good thing. I’m one to (historically) really dislike them, because you don’t always have control of the time, and it interrupts you from something you were working on. It’s also hard to get back in the zone after. How do they differ between academia and a national lab?
In academia, both as a student and staff, unless it was a 1:1 with my advisor, meetings felt really long and not super engaging. Labs give a lot of talks, and group meetings as a staff member tended to be very service oriented, practical things. However, in my later years I just didn’t have a lot of meetings, and the ones that I had were usually with just one more person, so I had a lot of control to decide when to have it. The meetings were usually service oriented (e.g., meeting with a resercher that I did work for), check-ins with a manager, or group update meetings where I could mostly be silent if I really wanted to. It can be lovely to have that freedom with your time, but these kind of meetings also make you feel a bit lonely that you aren’t working on a team.
This is probably the biggest surprise for me at a national lab. In meetings, although there are administrative things, along with the sense of comradeship that I described previously, I learn things. I especially learn things in 1:1s. And in group meetings, we have short talks just for the purpose of sharing knowledge. Yes, mind blown! Due to the sheer size of the institution, and the number of people that have been there for a long time, it’s pretty much the case that everyone is more experienced than I am. This makes just about every learning experience a good one, and it’s even challenging at times to take it all in. This kind of learning focused meeting is probably not typically the case for academic software engineering roles, because you don’t have as many people with that depth of experience. If you are an RSE, you already know the story. The salaries are not competitive with industry, the career ladders are non existent, and thus people leave more frequently in search of greener pastures. Your colleagues on your team probably won’t have been there for 10, 20, or even 30, years.
When I look back on my two months, I haven’t had a single meeting yet where I haven’t learned something, and because of all these factors coming together, I’ve learned more in these two months than I learned in years as a staff member in academia. Of course, another important note is that this judgment totally depends on what you want to learn. If you want to learn science, then maybe being rooted in an academic lab is where that kind of learning happens. But if you are like me and want to learn new programming languages, binary formats, and technologies, I suspect a national lab is going to have much more exposure.
Managers can really make or break your experience. I’ve heard horror stories, and although I had some stressful experiences with different lab rotations, I generally have been lucky. So let’s first state that for all of your managers, regardless of academia or a national lab, they wouldn’t be in their role unless they had expertise. But here’s the thing – it may not be in an area that you are hungry to learn. Let’s discuss the differences that I’ve seen.
I’ve been lucky to, in my first staff role, and then since the last few years of graduate school, consistently have good managers while I was in academia. I’ve had people that I can talk to easily, and get along with. I’ve also thrived in and enjoyed different roles that have managers with a largely hands off style. I’m able to independently get done what I want and need to do, and then update my managers about it. Why was it this way? I suspect that if your manager is a researcher or PI, he or she is investing most of their time in the success of the lab. This means meeting with students, writing grants, and doing administrative work. There is time allocated for you, but it must fit into a weekly, or even monthly, meeting slot. If your manager is head of a group that leads a service, it might be a similar phenomena. The success and organization of the group is top priority, and you will be very helpful if you can mostly direct yourself, and then provide updates. This is what I got used to in academia (and I won’t address a scatter of bad experiences that I had, because these can happen anywhere) and I found to be very different at a national lab.
It’s been surprising that my managers, meaning my group lead and technical lead on the project I’m working on, care a lot about what I’m doing. They want to regularly check in, make sure that I’m not blocked, and I’m doing okay. As I alluded to in the previous section, what has really blown my mind are the regular occurrences of teaching. Oh, I don’t know how to parse that binary? Or I’m not familiar with that in C++? Let’s open a terminal or a file and start talking about it. I even feel guilty sometimes because the other party is investing more time in their day to give me random knowledge. These interactions have made me realize that although I’ve had very skilled and experienced managers, I’ve never had a manager that has the kind of expertise that matches what I want to learn. A manager that doesn’t have experise that you want to learn means that it will be harder for you grow in the areas that you want. It will be up to you to find other ways to learn. I don’t mean to say that the responsibility of learning is on the manager, but it doesn’t hurt if they can contribute. Surprisingly, an article came out recently that makes exactly this point. The article has a bit of a click-baity title saying “Your boss could do your job,” but it’s really saying “It’s much more likely you’ll be fulfilled if your boss has expertise that you want to know.” I think this is probably true.
In my experience, having a manager with expertise that doesn’t match what I want to learn was not necessarily a bad thing. As long as my managers were supportive and kind I could be successful and happy. But on the other side of the galaxy, to have a manager that has their head full of cool things (in my opinion) that I want to learn, and wants to teach me them? Just totally, wow. It’s absolutely wonderful. I look forward to these sessions because I feel like I’m going to grow. My managers are not just check-ins to ask what I’m doing and where money is coming from, but people that are my teachers. This comes back to the idea that the kind of manager that will be best for you depends on what you want to learn. A manger that doesn’t have technical expertise can indeed teach you things about people, and help you discover things about yourself. A manager that is hands off can give you a lot of room to grow, on your own. This has definitely been the case for me, and it’s a different kind of growth that is also valuable. But having technical leads that want to invest time and energy in me has a strange trickle down effect – it makes me want to do the same. I have this desire to figure out how I can spend my time to help my colleagues. I’m not in a mindset of surviving, I’m in a mindset of being in a clan and figuring out how we can work together.
We’ve discussed communication, titles, mindsets, training and expectations, meetings, and people. And here comes the resounding “it depends!” The kind of work that you want to do, the way (and frequency) that you want to learn, are totally dependent on your personality and goals. I could easily imagine a version of myself that fell in love with the service mindset, and was more fulfilled by helping others directly. I would have been a perfect fit for an academic service-oriented role. I could also easily imagine a verison of myself in 30 years that doesn’t want so much to be smacked with figuring out things I don’t know, but would rather be a technical lead for a group, and teach the bucket of knowledge that has accumulated in my head after decades of stumbling. Your best fit depends not only on your goals at a specific time, but also your goals as you change over time. I suspect many of us will change, whether we are aware of it or not.
I’ve done my best to review the differences that I see between working at a national lab vs. in academia. Keep in mind that this is based on my experience, and might be very different than yours. It turns out that working at a national lab is a better fit for me, at least based on my particular goals and my assessment of these last two months. I was scared to make the transition, but now am grateful that I took the risk. Keep in mind that this might not be the case for you. I am aware that I’ve only been in my new role for two months, and things could change. Everyone is working from home to create a “remote first” atmosphere, and this could entirely change when everyone goes back (and I do not). I want to encourage others to also think about themselves in context of their institutions. Think about what you value and want to learn, and if that is afforded by your environment. If we continue to do this kind of introspection, we won’t just be happier engineers, we will be in a place where we are maximally matched to contribute back to the world.