Top Berkeley Jobs

Introduction

The evaluations project at the Alignment Research Center is a new team building capability evaluations (and in the future, alignment evaluations) for advanced ML models. The goals of the project are to improve our understanding of what alignment danger is going to look like, understand how far away we are from dangerous AI, and create metrics that labs can make commitments around (e.g. 'If you hit capability threshold X, don't train a larger model until you've hit alignment threshold Y'). You can learn more about the project at this linked post on the Alignment Forum.

We have filled this position, but we may hire again for it in the future. You can express your interest should we open the role back up, but we will not actively review submissions for now and may not ever get back to you.

Job Mission

- Find, evaluate, recruit, manage, incentivize and supervise skilled contractors to generate high quality data on challenging and engaging tasks. Tasks include things like 'using only bash commands, research the most capable open-source language model and set it up on a new server'

- Ensure instructions are clear and incentives are appropriate

- Help contractors feel motivated and part of a team

- Set up training, mentoring or pairing to improve contractor performance

- Understand and fix the biggest bottlenecks to high quality data generation

- Evaluate different data generation contracting services or strategies, and make high level decisions about when to use different out-sourcing services to scale up data generation.

Example Projects

- Manage a team of skilled contractors to extend a small set of expert-produced gold-standard ratings of complex ML trajectories (e.g. 100 examples) into a large dataset, making sure to maintain rating quality close to the gold standard.

- Talk to researchers about how labels should be generated, and then produce clear instructional materials for contractors which help them understand the task.

- Do a small comparison of different contracting services for a data generation task, and decide which, if any, could produce data that meets our bar.

Skills and Experience

- Successfully organized fast-moving projects with many people involved 

- Understanding of LLMs, including finetuning with human data

- Data analytics

- Familiarity with bash, ability to engage with the tasks and produce high-quality data yourself

Introduction

The evaluations project at the Alignment Research Center is a new team building capability evaluations (and in the future, alignment evaluations) for advanced ML models. The goals of the project are to improve our understanding of what alignment danger is going to look like, understand how far away we are from dangerous AI, and create metrics that labs can make commitments around (e.g. 'If you hit capability threshold X, don't train a larger model until you've hit alignment threshold Y'). You can learn more about the project at this linked post on the Alignment Forum.

We expect to keep this posting up, but not necessarily evaluate applications actively at all times. It may be some time (months) before we get back to you.

Job Mission

- Effectively finetune LLMs to demonstrate different behaviors or capabilities

- Make informed guesses about how much a finetuning run will improve model performance at various tasks

- Plan what data we need to collect, and prioritize this given tradeoffs in ease of collecting different types of data

- Structure experiments to rapidly get information about how well our approach is working

- Anticipate and prevent costly mistakes in finetuning efforts. Create and track metrics for finetuning success. 

- Keep up-to-date on the latest techniques, datasets or tools for eliciting or measuring model performance and capabilities, and ensure we're getting an accurate picture of model capabilities

- Design and run experiments in a replicable, scientific manner, producing highly trustworthy results

- Identify novel hypotheses about the capabilities and behavior of large foundation models, and design experiments and data collection strategies to evaluate them.

- Forecast future progress in ML, including how we'll need to change our evaluation strategies, and how we might incorporate future models to speed up our workflows

Skills

- Conceptual alignment thinking: help generate and evaluate ideas for how to probe alignment, deception, agency and other conceptually slippery properties.

- Dealing with the engineering details of large models

- Selecting or improving model architectures

Introduction

The evaluations project at the Alignment Research Center is a new team building capability evaluations (and in the future, alignment evaluations) for advanced ML models. The goals of the project are to improve our understanding of what alignment danger is going to look like, understand how far away we are from dangerous AI, and create metrics that labs can make commitments around (e.g. 'If you hit capability threshold X, don't train a larger model until you've hit alignment threshold Y'). You can learn more about the project at this linked post on the Alignment Forum.

We have filled this position, but we may hire again for it in the future. You can express your interest should we open the role back up, but we will not actively review submissions for now and may not ever get back to you.

Job Mission

- Take ownership of designing and maintaining data pipelines that minimize friction in generating, labeling, quality-checking, filtering, searching and compiling ML datasets covering many aspects of model behavior.

- Take responsibility for the smooth gathering and processing of data at the organization as a whole, and take action to proactively address potential problems.

- Allow researchers to quickly understand properties like quality and balance of the data we've gathered

- Run experiments to discover the most effective prompting or finetuning strategies

- Ensure all results are reliable and easily replicable

- Produce scaling laws and other investigations of how results vary according to model properties

- Keep your eye on the ball of ensuring accurate assessments of model capabilities, be scrappy and willing to move beyond your job description, be part of a fast-moving team.

Example Projects

- Set up an efficient data pipeline and workflow for checking contractor agreement on data labels.

- Set up data pipelines for model behavior trajectories which allow easy searching, filtering and rating.

- Work with different evaluation projects to convert datasets into suitable formats.

- Work with researchers to identify data hypotheses, and set up experiments and visualizations of the data to investigate them

- Work with ML researchers to generate and process data to analyze scaling laws for different model capabilities.

Skills

Essential Skills

- Prior experience with finetuning LLMs or working with natural-language data.

Bonus Skills

- Expertise in particular domains relevant to model takeover, e.g. cybersecurity or ML engineering, in order to help assess model competence on these tasks.

- Design + data-viz skills: Generating novel ideas for improving usability of evaluation tools and providing an interaction experience that unlocks insights about the data or the model

- Experience with large-scale data or distributed systems

- Dealing with the engineering details of large models

Voleon is a technology company that applies state-of-the-art machine learning techniques to real-world problems in finance. For more than a decade, we have led our industry and worked at the frontier of applying machine learning to investment management. We have become a multi-billion-dollar asset manager, and we have ambitious goals for the future. 

Your colleagues will include internationally recognized experts in machine learning research as well as highly experienced finance and technology professionals. You will be working with strategies that are at the forefront of machine learning and statistical trading. The strategies have been carefully designed to generate non-correlated returns. Our firm and its strategies are the product of many years of meticulous research and craftsmanship, and you will lead the way in bringing them to investors.

In addition to our enriching and collegial working environment, we offer highly competitive compensation and benefits packages, unparalleled access to our experts and founders, a beautiful modern office, daily catered lunches, and more.

As a Staff Accountant, you will be a key member of the management company finance team. This team has a broad mandate and is responsible for managing the books, records, and budgets of both US and International businesses and managing tax filings with external service providers. You will report to our Finance Director and be a vital contributor to the success of the finance team, working with our senior executives and partners to support critical accounting and budgeting processes and growth initiatives across the firm. 

Responsibilities

• Assist with annual audit and tax preparation processes for both partnership and C-Corp entities
• Lead areas of general financial operations, including budgeting, research and development tax credit studies, and partnership accounting
• Maintain and reconcile partnership capital accounts and capital expenditure trackers across various entities
• Perform valuations and analyses to drive business decisions, including modeling multi-year month forecasts for various functional groups
• Assist group heads within finance in developing growth and financial metrics (and OKRs) for each functional group
• Collaborate with our legal team on research and implementation for entity structuring, both domestic and international
• Support cross-functional initiatives, including HR, Facilities, and Fund Finance
• Oversee and manage accounting for LCI, a C-Corporation entity, including various state-by-state and local requirements
• Lead bespoke projects to improve our management company finances and our finance effort overall
• Work with external advisors to prepare and file taxes

• A minimum of three years of experience in public or financial accounting
• Understanding of US GAAP and financial accounting and its application to partnerships and/or investment management
• Excellent Excel skills
• Experience working with enterprise-level accounting software; some possible examples include QuickBooks, Oracle EBS, Sage Intacct, and SAP
• Understanding of U.S. tax regulations and partnership accounting
• Active CPA
• Bachelor’s degree in accounting, economics, or related discipline.

• Exposure to international businesses including accounting and tax

The base salary range for this position is $100,000 to $125,000 in the location(s) of this posting. Individual salaries are determined through a variety of factors, including, but not limited to, education, experience, knowledge, skills, and geography. Base salary does not include other forms of total compensation such as bonus compensation and other benefits. Our benefits package includes medical, dental and vision coverage, life and AD&D insurance, 20 days of paid time off, 9 sick days, and a 401(k) plan with a company match.

“Friends of Voleon” Candidate Referral Program

If you have a great candidate in mind for this role and would like to have the potential to earn $15,000 if your referred candidate is successfully hired and employed by The Voleon Group, please use this form to submit your referral. For more details regarding eligibility, terms and conditions please make sure to review the Voleon Referral Bonus Program.

Equal Opportunity Employer

The Voleon Group is an Equal Opportunity employer. Applicants are considered without regard to race, color, religion, creed, national origin, age, sex, gender, marital status, sexual orientation and identity, genetic information, veteran status, citizenship, or any other factors prohibited by local, state, or federal law.

Vaccination Requirement

The Voleon Group has implemented a policy requiring all employees who will be entering our worksite, including new hires, to be fully vaccinated with the COVID-19 vaccine.  This policy also applies to remote employees, as such employees will be asked to visit our offices from time to time. To the extent permitted by applicable law, proof of vaccination will be required as a condition of employment. This policy is part of Voleon’s ongoing efforts to ensure the safety and well-being of our employees and community, and to support public health efforts.

What is ARC’s Theory team?

The Alignment Research Center (ARC) is a non-profit whose mission is to align future machine learning systems with human interests. The high-level agenda of the Theory team (not to be confused with the Evals team) is described by the report on Eliciting Latent Knowledge (ELK): roughly speaking, we’re trying to design ML training objectives that incentivize systems to honestly report their internal beliefs.

For the last year or so, we’ve mostly been focused on an approach to ELK based on formalizing a kind of heuristic reasoning that could be used to analyze neural network behavior, as laid out in our paper on Formalizing the presumption of independence. Our research has reached a stage where we’re coming up against concrete problems in mathematics and theoretical computer science, and so we’re particularly excited about hiring researchers with relevant background, regardless of whether they have worked on AI alignment before. See below for further discussion of ARC’s current theoretical research directions.

Who is ARC looking to hire?

Compared to our last hiring round, we have more of a need for people with a strong theoretical background (in math, physics or computer science, for example), but we remain open to anyone who is excited about getting involved in AI alignment, even if they do not have an existing research record.

Ultimately, we are excited to hire people who could contribute to our research agenda. The best way to figure out whether you might be able to contribute is to take a look at some of our recent research problems and directions:

- Some of our research problems are purely mathematical, such as these matrix completion problems – although note that these are unusually difficult, self-contained and well-posed (making them more appropriate for prizes).

- Some of our other research is more informal, as described in some of our recent blog posts such as Finding gliders in the game of life.

- A lot of our research occupies a middle ground between fully-formalized problems and more informal questions, such as fixing the problems with cumulant propagation described in Appendix D of Formalizing the presumption of independence.

What is working on ARC’s Theory team like?

ARC’s Theory team is led by Paul Christiano and currently has 2 other permanent team members, Mark Xu and Jacob Hilton, alongside a varying number of temporary team members (recently anywhere from 0–3).

Most of the time, team members work on research problems independently, with frequent check-ins with their research advisor (e.g., twice weekly). The problems described above give a rough indication of the kind of research problems involved, which we would typically break down into smaller, more manageable subproblems. This work is often somewhat similar to academic research in pure math or theoretical computer science.

In addition to this, we also allocate a significant portion of our time to higher-level questions surrounding research prioritization, which we often discuss at our weekly group meeting. Since the team is still small, we are keen for new team members to help with this process of shaping and defining our research.

ARC shares an office with several other groups working on AI alignment such as Redwood Research, so even though the Theory team is small, the office is lively with lots of AI alignment-related discussion.

What are ARC’s current theoretical research directions?

ARC’s main theoretical focus over the last year or so has been on preparing the paper Formalizing the presumption of independence and on follow-up work to that. Roughly speaking, we’re trying to develop a framework for “formal heuristic arguments” that can be used to reason about the behavior of neural networks. This framework can be thought of as a confluence of two existing approaches:

- Mechanistic interpretability: uncertain and defeasible, but not machine verifable

- Formal proof: machine verifable, but strictly confident only

- Formal heuristic argument (our approach): uncertain and defeasible and machine verifiable

This research direction can be framed in a couple of different ways:

- As a formalization of mechanistic interpretability: Mechanistic interpretability is a research field seeking to reverse-engineer the weights of neural networks into human-understandable programs. A number of the field's central concepts, such as a “feature”, are currently defined informally. Putting the field onto more of a formal footing could bring clarity to the methods and goals of the field, remove the need to have humans or human-like systems in the loop, and elucidate how interpretability could be applied to solve downstream problems.

- As a way of dealing with out-of-distribution generalization failures: We think that a formal heuristic argument that explains a neural network’s training set performance could be used to flag new datapoints that trigger unusual behavior inside the model. We have been calling this approach “mechanistic anomaly detection”, since it can be thought of as a way to detect anomalies in the model’s internal activations at inference time. Further details are given in this blog post.

Hiring process

Our current interview process involves:

- 3-hour take-home test involving math and computer science puzzles

- 30-minute non-technical phone call

- 1-day onsite interview

We will compensate candidates for their time when this is logistically possible.

We will keep applications open until at least the end of August 2023, and will aim to get a final decision back within 6 weeks of receiving an application.

Employment details

ARC is based in Berkeley, California, and we would prefer people who can work full-time from our office, but we are open to discussing remote or part-time arrangements in some circumstances. We can sponsor visas and are H-1B cap-exempt.

We are accepting applications for both visiting researcher (1–3 months) and full-time positions. The intention of the visiting researcher position is to assess potential fit for a full-time role, and we expect to invite around half of visiting researchers to join full-time. We are also able to offer straight-to-full-time positions, but we anticipate that we will only be able to do this for people with a legible research track-record.

Salaries are in the $150k–400k range for most people depending on experience.

Further information

If you have any questions about anything in this posting, please email hiring@alignment.org.

If you want to provide any feedback, you can use this form: https://forms.gle/DndeoBekS6ViyifW6