# publications

Below is a list of my journal and conference publications and preprints in reverse chronological order. You can also check out my Google Scholar profile.

## 2023

- The Scope of Multicalibration: Characterizing Multicalibration via Property Elicitation
*Georgy Noarov*, and Aaron Roth*40th International Conference on Machine Learning (ICML)*2023We make a connection between multicalibration and property elicitation and show that (under mild technical conditions) it is possible to produce a multicalibrated predictor for a continuous scalar distributional property Γ if and only if Γ is elicitable. On the negative side, we show that for non-elicitable continuous properties there exist simple data distributions on which even the true distributional predictor is not calibrated. On the positive side, for elicitable Γ, we give simple canonical algorithms for the batch and the online adversarial setting, that learn a Γ-multicalibrated predictor. This generalizes past work on multicalibrated means and quantiles, and in fact strengthens existing online quantile multicalibration results. To further counter-weigh our negative result, we show that if a property Γ1 is not elicitable by itself, but is elicitable conditionally on another elicitable property Γ0, then there is a canonical algorithm that jointly multicalibrates Γ1 and Γ0; this generalizes past work on mean-moment multicalibration. Finally, as applications of our theory, we provide novel algorithmic and impossibility results for fair (multicalibrated) risk assessment.

- Batch Multivalid Conformal PredictionChristopher Jung,
*Georgy Noarov*, Ramya Ramalingam, and Aaron Roth*11th International Conference on Learning Representations (ICLR)*2023We develop fast distribution-free conformal prediction algorithms for obtaining multivalid coverage on exchangeable data in the batch setting. Multivalid coverage guarantees are stronger than marginal coverage guarantees in two ways: (1) They hold even conditional on group membership -- that is, the target coverage level \(1-\alpha\) holds conditionally on membership in each of an arbitrary (potentially intersecting) group in a finite collection \(G\) of regions in the feature space. (2) They hold even conditional on the value of the threshold used to produce the prediction set on a given example. In fact multivalid coverage guarantees hold even when conditioning on group membership and threshold value simultaneously. We give two algorithms: both take as input an arbitrary non-conformity score and an arbitrary collection of possibly intersecting groups \(G\), and then can equip arbitrary black-box predictors with prediction sets. Our first algorithm (BatchGCP) is a direct extension of quantile regression, needs to solve only a single convex minimization problem, and produces an estimator which has group-conditional guarantees for each group in \(G\). Our second algorithm (BatchMVP) is iterative, and gives the full guarantees of multivalid conformal prediction: prediction sets that are valid conditionally both on group membership and non-conformity threshold. We evaluate the performance of both of our algorithms in an extensive set of experiments.

## 2022

- Practical Adversarial Multivalid Conformal PredictionOsbert Bastani, Varun Gupta, Christopher Jung,
*Georgy Noarov*, Ramya Ramalingam, and Aaron Roth*36th Conference on Neural Information Processing Systems (NeurIPS)*2022**Selected as an Oral Presentation**We give a simple, generic conformal prediction method for sequential prediction that achieves target empirical coverage guarantees against adversarially chosen data. It is computationally lightweight -- comparable to split conformal prediction -- but does not require having a held-out validation set, and so all data can be used for training models from which to derive a conformal score. It gives stronger than marginal coverage guarantees in two ways. First, it gives threshold calibrated prediction sets that have correct empirical coverage even conditional on the threshold used to form the prediction set from the conformal score. Second, the user can specify an arbitrary collection of subsets of the feature space -- possibly intersecting -- and the coverage guarantees also hold conditional on membership in each of these subsets. We call our algorithm MVP, short for MultiValid Prediction. We give both theory and an extensive set of empirical evaluations.

- Online Minimax Multiobjective Optimization: Multicalibeating and Other ApplicationsDaniel Lee,
*Georgy Noarov*, Mallesh M. Pai, and Aaron Roth*36th Conference on Neural Information Processing Systems (NeurIPS)*2022**Selected as an Oral Presentation**We introduce a simple but general online learning framework, in which at every round, an adaptive adversary introduces a new game, consisting of an action space for the learner, an action space for the adversary, and a vector valued objective function that is convex-concave in every coordinate. The learner and the adversary then play in this game. The learner's goal is to play so as to minimize the maximum coordinate of the cumulative vector-valued loss. The resulting one-shot game is not convex-concave, and so the minimax theorem does not apply. Nevertheless, we give a simple algorithm that can compete with the setting in which the adversary must announce their action first, with optimally diminishing regret. We demonstrate the power of our simple framework by using it to derive optimal bounds and algorithms across a variety of domains. This includes no regret learning: we can recover optimal algorithms and bounds for minimizing external regret, internal regret, adaptive regret, multigroup regret, subsequence regret, and a notion of regret in the sleeping experts setting. Next, we use it to derive a variant of Blackwell's Approachability Theorem, which we term "Fast Polytope Approachability". Finally, we are able to recover recently derived algorithms and bounds for online adversarial multicalibration and related notions (mean-conditioned moment multicalibration, and prediction interval multivalidity).

- Online Multivalid Learning: Means, Moments, and Prediction IntervalsVarun Gupta, Christopher Jung,
*Georgy Noarov*, Mallesh M. Pai, and Aaron Roth*13th Conference on Innovations in Theoretical Computer Science (ITCS)*2022We present a general, efficient technique for providing contextual predictions that are "multivalid" in various senses, against an online sequence of adversarially chosen examples \( (x,y)\). This means that the resulting estimates correctly predict various statistics of the labels y not just marginally -- as averaged over the sequence of examples -- but also conditionally on \(x \in g\) for any \(g\) belonging to an arbitrary intersecting collection of groups \(G\). We provide three instantiations of this framework. The first is mean prediction, which corresponds to an online algorithm satisfying the notion of multicalibration from Hebert-Johnson et al. The second is variance and higher moment prediction, which corresponds to an online algorithm satisfying the notion of mean-conditioned moment multicalibration from Jung et al. Finally, we define a new notion of prediction interval multivalidity, and give an algorithm for finding prediction intervals which satisfy it. Because our algorithms handle adversarially chosen examples, they can equally well be used to predict statistics of the residuals of arbitrary point prediction methods, giving rise to very general techniques for quantifying the uncertainty of predictions of black box algorithms, even in an online adversarial setting. When instantiated for prediction intervals, this solves a similar problem as conformal prediction, but in an adversarial environment and with multivalidity guarantees stronger than simple marginal coverage guarantees.

- Computing Approximate Equilibria in Weighted Congestion Games via Best-ResponsesYiannis Giannakopoulos,
*Georgy Noarov*, and Andreas S. Schulz*Mathematics of Operations Research*2022We present a deterministic polynomial-time algorithm for computing \( d^{d+o(d)} \)-approximate (pure) Nash equilibria in weighted congestion games with polynomial cost functions of degree at most \(d\). This is an exponential improvement of the approximation factor relative to the previously best deterministic algorithm. An appealing additional feature of the algorithm is that it only uses best-improvement steps in the actual game, as opposed to the previously best algorithms, that first had to transform the game itself. Our algorithm is an adaptation of the seminal algorithm by Caragiannis at al. [2011, 2015], but we utilize an approximate potential function directly on the original game instead of an exact one on a modified game. A critical component of our analysis, which is of independent interest, is the derivation of a novel bound of \( [d/W(d/\rho)]^{d+1} \) for the price of anarchy (PoA) of \( \rho \)-approximate equilibria in weighted congestion games, where \(W\) is the Lambert-W function. More specifically, we show that this PoA is exactly equal to \(\Phi_{d,\rho}^{d+1} \), where \( \Phi_{d,\rho} \) is the unique positive solution of the equation \( \rho (x+1)^d = x^{d+1} \). Our upper bound is derived via a smoothness-like argument, and thus holds even for mixed Nash and correlated equilibria, whereas our lower bound is simple enough to apply even to singleton congestion games.

## 2021

- Binary Scoring Rules that Incentivize PrecisionEric Neyman,
*Georgy Noarov*, and S. Matthew Weinberg*22nd ACM Conference on Economics and Computation (EC)*2021All proper scoring rules incentivize an expert to predict

*accurately*(report their true estimate), but not all proper scoring rules equally incentivize*precision.*Rather than treating the expert's belief as exogenously given, we consider a model where a rational expert can endogenously refine their belief by repeatedly paying a fixed cost, and is incentivized to do so by a proper scoring rule. Specifically, our expert aims to predict the probability that a biased coin flipped tomorrow will land heads, and can flip the coin any number of times today at a cost of c per flip. Our first main result defines an*incentivization index*for proper scoring rules, and proves that this index measures the expected error of the expert's estimate (where the number of flips today is chosen adaptively to maximize the predictor's expected payoff). Our second main result finds the unique scoring rule which optimizes the incentivization index over all proper scoring rules. We also consider extensions to minimizing the ℓth moment of error, and again provide an incentivization index and optimal proper scoring rule. In some cases, the resulting scoring rule is differentiable, but not infinitely differentiable. In these cases, we further prove that the optimum can be uniformly approximated by polynomial scoring rules. Finally, we compare common scoring rules via our measure, and include simulations confirming the relevance of our measure even in domains outside where it provably applies.