Everything you want to know about working at ConvertKit as an engineer
Recently there was a post on Hacker News about reverse interview questions, or questions the interviewee should ask about the company they are interviewing for. I think we have interesting answers to those questions, so I've provided our answers here.
What's the on-call plan/schedule?
Currently we share a primary and secondary rotation between all engineers. The on-call is 1 week. We've been talking recently about sensible ways to modify our on-call, since nowadays our SREs handle pager events most of the time.
We also take advantage of the fact that we're a 100% remote company. On-call shifts are for 12 hours only, during your waking hours.
What are the tasks I would do on a usual day?
You'd primarily work from your squad's kanban board. The backlog is usually a combination of project tasks and sometimes customer issues. Occasionally you'd prepare a lunch and learn, or help with chapters you find interesting.
What's the junior/senior balance of the team? (and are there plans to change it)
Right now the junior/senior ratio is very low. We only have one junior engineer on the team. This was borne out of necessity since ConvertKit needed a solid base of senior engineers to start with, given our philosophy about team size. The junior engineer we hired was our first foray, and it went very well. We found someone extremely ambitious and she's been an amazing contributor since we've hired her.
Going forward we will be planning on changing this ratio. I intend to hire many more ambitious junior engineers, probably at a 2:1 ratio.
What does the onboarding look like?
Our onboarding has been described as wonderfully relentless. That’s pretty accurate! We know that at a remote company, it can be hard to feel like part of the team when you first start - especially when you’re just sitting at your desk in your own home. So, we load up your first two weeks with lots of face-to-face (Zoom) meetings so you can get to know team members, learn about our systems, programs, and projects to help you dive into your area of work. Our goal is for you to comfortably know your way around ConvertKit when you’re done, and make it possible for you to contribute in a meaningful way right out of the gate.
We also continually monitor and tweak our onboarding process, based upon feedback from new employees. We want to ensure it's as effective and efficient as possible.
What are the expected/core work hours?
We don't have expected or core work hours. Most of the engineers are in the US or Europe, so we naturally shift our days to try to get as much overlap with our teammates as possible. But, that's not a requirement.
We strive to have engineers respect their own time and work 40-50 hours a week. We also recognize that it's easy to work more, and sometimes that's what needs to be done. Being a remote company, it can be hard to put a problem down and turn off "work mode". But it's an important skill to learn in order to be a successful remote employee. We frequently talk about honing your "skill" of working remotely.
What is your definition of success for this role?
Our definition of "success" would mean that we've found a candidate who matches closely with the ideal person we're looking for:
- We're growing and are facing scaling challenges. You should not be afraid of big, meaty problems and thinking out of the box for solving them.
- We're a small but very talented team. We're all passionate about learning and honing our craft. We'd expect the same hunger and drive from you.
- Ideally you're a creator yourself, and have a developer blog you maintain.
- We're a remote company, so you should be autonomous, self-driven, and remote-disciplined. Remote work is not for everyone, and you should reflect and understand that it's a skill and commitment to do remote work effectively.
What is your tech stack?
The ConvertKit app is built using Ruby on Rails, with a MySQL backend. Additionally we use a Cassandra database for emails and email events, and ElasticSearch for reporting / aggregrations in the app. On the frontend, we use React.
How do you use source control?
We use Github and pull requests into the
How do you test code?
rspec for testing, which gets run for every pull request via CircleCI. Additionally we have a dedicated QA engineer who will help manually test when needed. Our QA engineer is also working on an automating these tests with Cypress.
How do you track bugs?
We use Github Issues to track bugs.
How do you integrate and deploy changes? Is it CI/CD?
Our infrastructure engineers created a slack bot lovingly named StoP Bob (Straight-to-production bob) for deployments, and deploying to production is as simple as running
/stop deploy convertkit.
Specs get run continually for both branches and master. At some point, we will automatically deploy green builds to production.
Is your infrastructure setup under version control / available as code?
Yes. We're firm believers in Infrastructure-as-Code. In 2018, we migrated our entire app from Heroku to AWS. Since our AWS setup was essentially greenfield development, we took the time to ensure that everything was set up according to best practices.
All of our self-managed data stores are baked using Packer. When we need to upgrade a data store we create a new AMI, update our terraform, test it on staging, if everything is ok we move it into production. This ensures all of our data stores are immutable, modular, and easily replaceable/upgradeable.
All of our autoscaling groups are set up using terraform modules. Creating new autoscaling groups or updating old autoscaling groups is as easy as modifying parameters in the module. Because we do this, we are capable of upgrading an deploying new autoscaling groups as well as old autoscaling groups while also maintaining immutable infrastructure.
If any service or data store node breaks all we have to do is terminate the instance and wait for the autoscaling group to replace the instance.
Looking forward, we are embracing containerization and our app should be run via Kubernetes by the end of 2019.
How do you prepare for disaster recovery?
We run daily backups of our MySQL database. Since MySQL is in RDS, we can utilize point-in-time recovery, so we can recover our database at any point in the time. We run hourly incremental backups of our Cassandra cluster and ElasticSearch cluster.
For each of our data stores (including MySQL), we run them in multiple AZs and are working to get multi-region recovery in place for our C* cluster.
Additionally, we have a completely separate AWS account that we replicate snapshots to. This is to prevent the case where our production AWS account gets hacked, and the attacker decides to delete everything. Instead of it being a company-ending event, it would merely be a giant headache with a maximum of 60 minutes' data lost.
For everything else, losing a server or 2 is a not an emergency, or even a pager event. Since all of our infrastructure is Terraformed and runs under auto scaling, replacing instances is standard procedure and does not affect the performance of our app.
Is there a standardised development environment? Is it enforced?
We have a docker-based development environment that nearly all engineers use. It isn't enforced, but there are clear benefits to having things set up like everyone else - namely, there are ways and resources for handling common issues. It's a time saver!
How quickly can you set up a new local environment?
Given our docker-based development environment outlined above, the answer is probably 30 minutes. Less than that if you already have
docker-compose set up locally.
How quickly can you respond to security issues in the code or dependencies?
Depending on the severity of the issue and the impact, it can be as little as 1 hour. Deploys are automated and take less than 10 minutes, so the main time would be how long it takes to think about and actually remediate the issue at hand.
Are all developers allowed to have local admin access of their computers?
Yes. Aside from some simple security measures that we require of every employee and contractor, you are free to have your computer set up the way you want.
How is the work organized?
For product work, we create Github Milestones, and use Github Issues to capture the tasks needed to complete the milestone. We strive to keep a single task as less than a day's work for a single engineer.
How does the intra/inter-team communication typically work?
We rely on Slack for all coordination / short-term conversations. Because we're remote, Slack is truly the heart of our company. For long-form updates and planning, we use Basecamp. Our AWS bill breakdowns are written in Basecamp.
Ironically, for being an email company, we don't really use email for communication within ConvertKit.
How are differences of opinions resolved?
It depends on the scope of the problem. If it's a minor issue (like a discrepancy for how to move forward in a pull request), usually engineers will get on a zoom call and hash out the different aspects of how to proceed. No one at ConvertKit is a jerk, and we encourage everyone to be able to speak their minds freely, without fear of recourse.
For large decisions, we utilize CoinBase's decision making framework or the DMF, as we like to call it. Anyone in the company can invoke the DMF if they feel it is warranted. The main reason the DMF is great is that it allows everyone who's interested to be heard. It nullifies the issue of the "loudest person in the room" getting their way.
Who sets the priorities / schedule?
The schedule is agreed upon between the Product Manager and the squad leads. There is always a healthy discussion about ensuring we are retiring technical debt vs moving the product forward and achieving company goals.
What happens after pushback? ("this can't be done in the projected time")
We'll usually talk about what scope we can cut in order to have something ship in the projected time.
What kind of meetings happen every week?
We have an all-team meeting that's on Thrusdays at noon EST. We also have a "big standup" on Mondays and Thursdays, which is almost exclusively "parking lot" discussions for topics that cut across engineering squads. If there are no items to discuss, we don't have the meeting.
Each squad has their own individual standups as well, usually on an as-needed basis.
What happens after production incidents? Is there a culture of blameless analysis?
ConvertKit does have a culture of blameless analysis. We've had production incidents in the past and instead of blaming anyone, we try to find a way to improve the process that lead to the incident in the first place.
Aside from a culture of blamelessness, we do our very best to keep our egos out of the decisions we make. We strive to find the best possible solution to all problems through collaboration rather than listening to the loudest person in the room.
Excerpt from Kris, one of our Infra engineers: "This is one of my favorite things about ConvertKit honestly and I told Ivan (our other Infra engineer) that at the retreat. Every day, I look forward to working with him because any time there is a fire there are no arguments or disagreements. We acknowledge what happened, and move forward and do our best to make sure they don't happen again."
What are some ongoing challenges the team is experiencing that you are yet to resolve?
We still have far too many exceptions occurring in our production environment. We've established a process for handling these, and overall it's trending in the right direction. But, we still need to devise ways to sink time into resolving these.
One thing we'll be trying in the upcoming quarter is a 6-week / 2-week cycle as documented in Basecamp's fantastic Shape Up book. By giving engineers dedicated time to fixing lower-priority issues that arise, we should see this number go down significantly.
Most of the exceptions arise from older areas of the codebase, where code / architecture was shipped before we had a mature software delivery process.
Is there a conference/travel budget and what are the rules to use it?
Yes. You may attend one conference per year, and we'll pay all the expenses.
What's the promotion process? How are requirements / expectations communicated?
Promotions occur during our twice annual performance review process. Managers keep track of an engineer's performance via continuous one-on-one feedback. If an engineer is outperforming their role for more than 2 consecutive quarters, they are considered for a promotion.
Is there a separate tech and management career path?
Yes. This is still a new concept at ConvertKit, given our size. But we will have separate roles for engineering managers vs more senior engineers.
What's the status of / view on diverse hiring?
We have a viewpoint and it's on our careers page. Having a diverse team is something that I (Grant) personally care deeply about, and our leadership cares about as well. We'll have more to say in future posts about our stance on diversity in tech, and how ConvertKit will be tackling the issue of diversity in tech.
Are there any company-wide resources for learning available, like ebooks subscriptions, or online courses?
Every person at ConvertKit gets a company card. We encourage engineers to buy the books, training courses, etc that they need.
What's the maturity stage?
I'd say we're firmly in the "small business" stage. We have a product/market fit, we have over 25k paying customers, and we're profitable.
Can I contribute to FOSS projects? Are there any approvals needed?
ConvertKit encourages contributions to FOSS projects, and we host a few ourselves.
Are you profitable?
Yes. Our financials are public, so feel free to peruse.
How long is your runway?
We don't have one. We're self-funded, profitable, and we have no investors.
How do you make money?
Via paid subscriptions to ConvertKit. The amount of money we charge depends on the size of your subscriber list.
What's preventing ConvertKit from making more money?
I'd say the main themes are:
- Broader awareness of the ConvertKit brand, and how we help creators earn a living.
- Itentional self-restraint to avoid a “grow at all costs” mentality. It's important for us to stay small and efficient.
- Competition in a crowded industry. There are no shortage of email marketing companies out there. We just happen to be the best for creators.
What's the ratio of remote to office workers?
NaN. The entire company is remote, and we don't have a central office.
Does the company provide hardware and what's the refresh schedule?
We provide a $2500 equipment credit every 2 years.
Are extra accessories/furniture possible to buy through the company? Is there a budget for them?
You can use your equipment credit to buy anything you like to make remote working comfortable and possible.
Is there a budget for co-working space or internet access?
Not at this time, but it's something we're actively evaluating.
If you have a bonus scheme, then how are bonuses determined?
ConvertKit shares a significant amount of profits with employees every 6 months.
Do you have a 401k or other retirement plan? If so, is there a company match to plan contributions?
Yes, we have a 401k. We do 4% matching.
How much PTO is offered?
3 weeks in your first year, going to 4 weeks after that.
Are sick time and vacation time separate or do they come from the same pool?
We have 3 pools. One for sick time, one for vacation, and one for parental leave. We give 5 days of sick time per year. The vacation time rolls over, but the sick time does not. We also give 8 holidays per year.
Can I use vacation time before it's accrued, effectively going into a negative PTO balance?
Yes, but it'd be at the manager's discretion.
What is the policy on unpaid leave?
We do allow unpaid leave. It's a conversation you'd have with your manager.
And that's it! Finding the right place to work is a big decision, and we encourage candiates to ask more questions. ConvertKit is unique in a lot of different ways. For some it might be a perfect fit. If, after reading this, you're interested in working here, then please check our careers page.